Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220126025722.GD17638@fieldses.org> <CAPt2mGP2guMMf1C9VoQ0AvZ819jPuz0vDoEzJJhtL8q5DJ300A@mail.gmail.com>
 <CAPt2mGNXq==1KUskF3U6-CDeoX57=d7NW4Qn_esDqarf9bTBaw@mail.gmail.com>
 <20220211155949.GA4941@fieldses.org> <CAPt2mGOx0qNTWoY9vmyVBtZ3gxdbv5qQ-2qVbtqWW9FiZFRhEg@mail.gmail.com>
 <164517040900.10228.8956772146017892417@noble.neil.brown.name>
 <CAPt2mGMLQCEPqsYGeaMd3BPGRne4F4h-2-pzqm1a8nwfKqv1ug@mail.gmail.com>
 <CAPt2mGMt3Sq66qmPBeGYE0CASTTy7nY2K_LjQK6VZx-uz2P-wg@mail.gmail.com>
 <20220425132232.GA24825@fieldses.org> <CAPt2mGMtBH=jzK0cTT7+PTbX-iR-iSx1RmF2beCDxBjXY5sj8A@mail.gmail.com>
 <20220425160236.GB24825@fieldses.org> <CAPt2mGPR9c9=rh4p_D7RPo+4S=DgH7VNpqvOKryKsYwaCAtnJA@mail.gmail.com>
 <165093700757.1648.16863178337904278508@noble.neil.brown.name>
In-Reply-To: <165093700757.1648.16863178337904278508@noble.neil.brown.name>
From:   Daire Byrne <daire@dneg.com>
Date:   Tue, 26 Apr 2022 13:29:20 +0100
Message-ID: <CAPt2mGPVWuut=ESWicSw0Ser2PGTeuyb+ACL41N6p_FAAuOUwg@mail.gmail.com>
Subject: Re: parallel file create rates (+high latency)
To:     NeilBrown <neilb@suse.de>
Cc:     "J. Bruce Fields" <bfields@fieldses.org>,
        Patrick Goetz <pgoetz@math.utexas.edu>,
        linux-nfs <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, 26 Apr 2022 at 02:36, NeilBrown <neilb@suse.de> wrote:
>
> On Tue, 26 Apr 2022, Daire Byrne wrote:
> >
> > I'll stare at fs/nfsd/vfs.c for a bit but I probably lack the
> > expertise to make it work.
>
> Staring at code is good for the soul ....  but I'll try to schedule time
> to work on this patch again - make it work from nfsd and also make it
> handle rename.

Yea, I stared at it for quite a while this morning and no amount of
coffee was going to help me figure out how best to proceed.

If you are able to update this for nfsd then I'll be eternally
grateful and do my best to test it under load in an effort to get it
all merged.

The community here has been so good to us over the last couple of
years and it is very much appreciated. It has helped us deliver (oscar
winning) movies!

To give some brief context as to why this is useful to us (for anyone
interested), we utilise NFS re-export extensively to run our batch
jobs (movie frame renders) in various remote DCs. In combination with
fscache and long term attribute caching, this works great for exposing
our (read often) onprem storage to the remote DCs (with varying
latencies).

But batch jobs have a tendency to start related tasks on many clients
at the same time with their results or logs being written to big
common directories. And by writing through the re-export server, we
often hit this limitation with parallel file creates which slows
everything down. We have tried to avoid large directories where
possible, but it's hard to catch and fix all the cases.

Using an NFS re-export server works 95% of the time for our workloads
(after much help from this community), so we are just trying to pick
away at the last 5% of edge cases. One of the disadvantages of the
re-export server in the middle, is that we lose some of the natural
parallelism that directly connected clients would otherwise have. And
this becomes very noticeable once the latency goes above 20ms.

Cheers,

Daire


> > It's also not entirely clear that this parallel creates RFC patch will
> > ever make it into mainline?
>
> I hope it will eventually, but I have no idea when that might be.
>
> Thanks for your continued interest,
> NeilBrown