Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CAPt2mGOaRsKOiL_wuSK_D5oYYnn0R-pvVsZc5HYGdEbT2FngtQ@mail.gmail.com>
 <20220124193759.GA4975@fieldses.org> <CAPt2mGOCn5OaeZm24+zh92qRcWTF8h-H2WXqScz9RMfo4r_-Qw@mail.gmail.com>
 <20220124205045.GB4975@fieldses.org> <CAPt2mGPTGgXztawDJfAKsiYqnm6P_mn1rtquSDKjpnSgvJH1YA@mail.gmail.com>
 <20220125135959.GA15537@fieldses.org>
In-Reply-To: <20220125135959.GA15537@fieldses.org>
From:   Daire Byrne <daire@dneg.com>
Date:   Tue, 25 Jan 2022 15:24:05 +0000
Message-ID: <CAPt2mGMKqPDaH3U=frrONX0V0aae0A=KDQhUd2GYO78u8FNyJQ@mail.gmail.com>
Subject: Re: parallel file create rates (+high latency)
To:     "J. Bruce Fields" <bfields@fieldses.org>
Cc:     linux-nfs <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, 25 Jan 2022 at 14:00, J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Tue, Jan 25, 2022 at 12:52:46PM +0000, Daire Byrne wrote:
> > Yea, it does seem like the server is the ultimate arbitrar and the
> > fact that multiple clients can achieve much higher rates of
> > parallelism does suggest that the VFS locking per client is somewhat
> > redundant and limiting (in this super niche case).
>
> It doesn't seem *so* weird to have a server with fast storage a long
> round-trip time away, in which case the client-side operation could take
> several orders of magnitude longer than the server.

Yea, I'm fine with the speed of light constraints for a single process
far away. But the best way to achieve aggregate performance in such
environments is to have multiple parallel streams in flight at once
(preferably bulk transfers).

Because I am writing through a single re-export server, I just so
happen to be killing any parallelism for single directory files
creates even though it works reasonably well for opens, reads, writes
and stat (and anything cacheable) which all retain a certain amount of
useful parallelism over a high latency network (nconnect helps too).
Each of our batch jobs all read 95% of the same files each time and
they all tend to run within the same short (hour) time periods so
highly cacheable.

> Though even if the client locking wasn't a factor, you might still have
> to do some work to take advantage of that.  (E.g. if your workload is
> just a single "untar"--it still waits for one create before doing the
> next one).

Yep. Again I'm okay with each client of a re-export server doing 3
creates per second, my problem is that all N instances together do 3
creates per second aggregate (in a single directory).

But I guess for this kind of workload, win some lose some. I just need
to figure out if I can engineer it to be less of a loser...

Of course, Hammerspace have a different approach to this kind of
problem with their global namespace and replicated MDT servers. And
that is probably a much more sensible way of going about this kind of
thing.

Daire