Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 25 Apr 2022 12:02:36 -0400
From:   "J. Bruce Fields" <bfields@fieldses.org>
To:     Daire Byrne <daire@dneg.com>
Cc:     NeilBrown <neilb@suse.de>, Patrick Goetz <pgoetz@math.utexas.edu>,
        linux-nfs <linux-nfs@vger.kernel.org>
Subject: Re: parallel file create rates (+high latency)
Message-ID: <20220425160236.GB24825@fieldses.org>
References: <20220126025722.GD17638@fieldses.org>
 <CAPt2mGP2guMMf1C9VoQ0AvZ819jPuz0vDoEzJJhtL8q5DJ300A@mail.gmail.com>
 <CAPt2mGNXq==1KUskF3U6-CDeoX57=d7NW4Qn_esDqarf9bTBaw@mail.gmail.com>
 <20220211155949.GA4941@fieldses.org>
 <CAPt2mGOx0qNTWoY9vmyVBtZ3gxdbv5qQ-2qVbtqWW9FiZFRhEg@mail.gmail.com>
 <164517040900.10228.8956772146017892417@noble.neil.brown.name>
 <CAPt2mGMLQCEPqsYGeaMd3BPGRne4F4h-2-pzqm1a8nwfKqv1ug@mail.gmail.com>
 <CAPt2mGMt3Sq66qmPBeGYE0CASTTy7nY2K_LjQK6VZx-uz2P-wg@mail.gmail.com>
 <20220425132232.GA24825@fieldses.org>
 <CAPt2mGMtBH=jzK0cTT7+PTbX-iR-iSx1RmF2beCDxBjXY5sj8A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAPt2mGMtBH=jzK0cTT7+PTbX-iR-iSx1RmF2beCDxBjXY5sj8A@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk

On Mon, Apr 25, 2022 at 04:24:50PM +0100, Daire Byrne wrote:
> On Mon, 25 Apr 2022 at 14:22, J. Bruce Fields <bfields@fieldses.org> wrote:
> >
> > On Mon, Apr 25, 2022 at 02:00:32PM +0100, Daire Byrne wrote:
> > > On Mon, 21 Feb 2022 at 13:59, Daire Byrne <daire@dneg.com> wrote:
> > > >
> > > > On Fri, 18 Feb 2022 at 07:46, NeilBrown <neilb@suse.de> wrote:
> > > > > I've ported it to mainline without much trouble.  I started some simple
> > > > > testing (parallel create/delete of the same file) and hit a bug quite
> > > > > easily.  I fixed that (eventually) and then tried with more than 1 CPU,
> > > > > and hit another bug.  But then it was quitting time.  If I can get rid
> > > > > of all the easy to find bugs, I'll post it with a CC to you, and you can
> > > > > find some more for me!
> > > >
> > > > That would be awesome! I have a real world production case for this
> > > > and it's a pretty heavy workload. If that doesn't shake out any bugs,
> > > > nothing will.
> > > >
> > > > The only caveat being that it will likely be restricted to NFSv3
> > > > testing due to the concurrency limitations with NFSv4.1+ (from the
> > > > other thread).
> > > >
> > > > Daire
> > >
> > > Just to follow up on this again - I have been using Neil's patch for
> > > parallel file creates (thanks!) but I'm a bit confused as to why it
> > > doesn't seem to help in my NFS re-export case.
> > >
> > > With the patch, I can achieve much higher parallel (multi process)
> > > creates directly on my re-export server to a high latency remote
> > > server mount, but when I re-export that to multiple clients, the
> > > aggregate create rate again degrades to that which we might expect
> > > either without the patch or if there was only one process creating the
> > > files in sequence.
> > >
> > > My assumption was that the nfsd threads of the re-export server would
> > > act as multiple independent processes and it's clients would be spread
> > > across them such that they would also benefit from the parallel
> > > creates patch on the re-export server. So I expected many clients
> > > creating files in the same directory would achieve much higher
> > > aggregate performance.
> >
> > That's the idea.
> >
> > I've lost track, where's the latest version of Neil's patch?
> >
> > --b.
> 
> The latest is still the one from this thread (with a minor update to
> apply it to v5.18-rc):
> 
> https://lore.kernel.org/lkml/893053D7-E5DD-43DB-941A-05C10FF5F396@dilger.ca/T/#m922999bf830cacb745f32cc464caf72d5ffa7c2c

Thanks!

I haven't really tried to understand that patch--but just looking at the
diffstat, it doesn't touch fs/nfsd/.  And nfsd calls into the vfs only
after it locks the parent.  So nfsd is probably still using
the old behavior, where local callers are using the new (parallel)
behavior.

So I bet what you're seeing is expected, and all that's needed is some
updates to fs/nfsd/vfs.c to reflect whatever Neil did in fs/namei.c.

--b.

> My test is something like this:
> 
> reexport1 # for x in {1..5000}; do
>     echo /srv/server1/touch.$HOSTNAME.$x
> done | xargs -n1 -P 200 -iX -t touch X 2>&1 | pv -l -a >|/dev/null
> 
> Without the patch this results in 3 creates/s and with the patch it's
> ~250 creates/s with 200 threads/processes (200ms latency) when run
> directly against a remote RHEL8 server (server1).
> 
> Then I run something similar to this but simultaneously across 200
> clients of the "reexport1" server's re-export of the originating
> "server1". I get an aggregate of around 3 creates/s even with the
> patch applied to reexport1 (v5.18-rc2) which is suspiciously similar
> to the performance without the parallel vfs create patch.
> 
> The clients don't run any special kernels or configurations. I have
> only tested NFSv3 so far.
> 
> Daire