Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
From:   "NeilBrown" <neilb@suse.de>
To:     "J. Bruce Fields" <bfields@fieldses.org>
Cc:     "Patrick Goetz" <pgoetz@math.utexas.edu>,
        "Daire Byrne" <daire@dneg.com>,
        "linux-nfs" <linux-nfs@vger.kernel.org>
Subject: Re: parallel file create rates (+high latency)
In-reply-to: <20220125212055.GB17638@fieldses.org>
References: <CAPt2mGOaRsKOiL_wuSK_D5oYYnn0R-pvVsZc5HYGdEbT2FngtQ@mail.gmail.com>,
 <20220124193759.GA4975@fieldses.org>,
 <adce2b72-ed5c-3056-313c-caea9bad4e15@math.utexas.edu>,
 <20220125212055.GB17638@fieldses.org>
Date:   Wed, 26 Jan 2022 11:02:16 +1100
Message-id: <164315533676.5493.13243313269022942124@noble.neil.brown.name>
Precedence: bulk

On Wed, 26 Jan 2022, J. Bruce Fields wrote:
> On Tue, Jan 25, 2022 at 03:15:42PM -0600, Patrick Goetz wrote:
> > So the directory is locked while the inode is created, or something
> > like this, which makes sense.
>=20
> It accomplishes a number of things, details in
> https://www.kernel.org/doc/html/latest/filesystems/directory-locking.html

Just in case anyone is interested, I wrote this a while back:

http://lists.lustre.org/pipermail/lustre-devel-lustre.org/2018-November/00817=
7.html

it includes a patch to allow parallel creates/deletes over NFS (and any
other filesystem which adds support).
I doubt it still applies, but it wouldn't be hard to make it work if
anyone was willing to make a strong case that we would benefit from
this.

NeilBrown


>=20
> > File creation means the directory
> > "file" is being updated. Just to be clear, though, from your ssh
> > suggestion below, this limitation does not exist if an existing file
> > is being updated?
>=20
> You don't need to take the exclusive i_rwsem lock on the directory to
> update an existing file, no.
>=20
> (But I was only suggesting that creating a bunch of files by ssh'ing
> into the server first and doing the create there would be faster,
> because the latency of each file create is less when you're running it
> directly on the server, as opposed to over a wide-area network
> connection.)
>=20
> --b.
>=20
> >=20
> >=20
> > >
> > >So, it's not surprising you'd get a higher rate when creating in
> > >multiple directories.
> > >
> > >Also, that lock's taken on both client and server.  So it makes sense
> > >that you might get a little more parallelism from multiple clients.
> > >
> > >So the usual advice is just to try to get that latency number as low as
> > >possible, by using a low-latency network and storage that can commit
> > >very quickly.  (An NFS server isn't permitted to reply to the RPC
> > >creating the new file until the new file actually hits stable storage.)
> > >
> > >Are you really seeing 200ms in production?
> > >
> > >--b.
> > >
> > >>
> > >>If I start 100 processes on the same client creating unique files in a
> > >>single shared directory (with 200ms latency), the rate of new file
> > >>creates is limited to around 3 files per second. Something like this:
> > >>
> > >># add latency to the client
> > >>sudo tc qdisc replace dev eth0 root netem delay 200ms
> > >>
> > >>sudo mount -o vers=3D4.2,nocto,actimeo=3D3600 server:/data /tmp/data
> > >>for x in {1..10000}; do
> > >>     echo /tmp/data/dir1/touch.$x
> > >>done | xargs -n1 -P 100 -iX -t touch X 2>&1 | pv -l -a > /dev/null
> > >>
> > >>It's a similar (slow) result for NFSv3. If we run it again just to
> > >>update the existing files, it's a lot faster because of the
> > >>nocto,actimeo and open file caching (32 files/s).
> > >>
> > >>Then if I switch it so that each process on the client creates
> > >>hundreds of files in a unique directory per process, the aggregate
> > >>file create rate increases to 32 per second. For NFSv3 it's 162
> > >>aggregate new files per second. So much better parallelism is possible
> > >>when the creates are spread across multiple remote directories on the
> > >>same client.
> > >>
> > >>If I then take the slow 3 creates per second example again and instead
> > >>use 10 client hosts (all with 200ms latency) and set them all creating
> > >>in the same remote server directory, then we get 3 x 10 =3D 30 creates
> > >>per second.
> > >>
> > >>So we can achieve some parallel file create performance in the same
> > >>remote directory but just not from a single client running multiple
> > >>processes. Which makes me think it's more of a client limitation
> > >>rather than a server locking issue?
> > >>
> > >>My interest in this (as always) is because while having hundreds of
> > >>processes creating files in the same directory might not be a common
> > >>workload, it is if you are re-exporting a filesystem and multiple
> > >>clients are creating new files for writing. For example a batch job
> > >>creating files in a common output directory.
> > >>
> > >>Re-exporting is a useful way of caching mostly read heavy workloads
> > >>but then performance suffers for these metadata heavy or writing
> > >>workloads. The parallel performance (nfsd threads) with a single
> > >>client mountpoint just can't compete with directly connected clients
> > >>to the originating server.
> > >>
> > >>Does anyone have any idea what the specific bottlenecks are here for
> > >>parallel file creates from a single client to a single directory?
> > >>
> > >>Cheers,
> > >>
> > >>Daire
>=20
>=20