Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 25 Jan 2022 16:59:42 -0500
From:   Bruce Fields <bfields@fieldses.org>
To:     Patrick Goetz <pgoetz@math.utexas.edu>
Cc:     Chuck Lever III <chuck.lever@oracle.com>,
        Daire Byrne <daire@dneg.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: parallel file create rates (+high latency)
Message-ID: <20220125215942.GC17638@fieldses.org>
References: <CAPt2mGOaRsKOiL_wuSK_D5oYYnn0R-pvVsZc5HYGdEbT2FngtQ@mail.gmail.com>
 <20220124193759.GA4975@fieldses.org>
 <CAPt2mGOCn5OaeZm24+zh92qRcWTF8h-H2WXqScz9RMfo4r_-Qw@mail.gmail.com>
 <20220124205045.GB4975@fieldses.org>
 <CAPt2mGPTGgXztawDJfAKsiYqnm6P_mn1rtquSDKjpnSgvJH1YA@mail.gmail.com>
 <20220125135959.GA15537@fieldses.org>
 <F7C721F7-D906-426F-814F-4D3F34AD6FB1@oracle.com>
 <42867c2c-1ab3-9bb6-0e5a-57d13d667bc6@math.utexas.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <42867c2c-1ab3-9bb6-0e5a-57d13d667bc6@math.utexas.edu>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk

On Tue, Jan 25, 2022 at 03:50:05PM -0600, Patrick Goetz wrote:
> On 1/25/22 09:30, Chuck Lever III wrote:
> >>On Jan 25, 2022, at 8:59 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> >>On Tue, Jan 25, 2022 at 12:52:46PM +0000, Daire Byrne wrote:
> >>>Yea, it does seem like the server is the ultimate arbitrar and the
> >>>fact that multiple clients can achieve much higher rates of
> >>>parallelism does suggest that the VFS locking per client is somewhat
> >>>redundant and limiting (in this super niche case).
> >>
> >>It doesn't seem *so* weird to have a server with fast storage a long
> >>round-trip time away, in which case the client-side operation could take
> >>several orders of magnitude longer than the server.
> >>
> >>Though even if the client locking wasn't a factor, you might still have
> >>to do some work to take advantage of that.  (E.g. if your workload is
> >>just a single "untar"--it still waits for one create before doing the
> >>next one).
> >
> >Note that this is also an issue for data center area filesystems, where
> >back-end replication of metadata updates makes creates and deletes as
> >slow as if they were being done on storage hundreds of miles away.
> >
> >The solution of choice appears to be to replace tar/rsync and such
> >tools with versions that are smarter about parallelizing file creation
> >and deletion.
> 
> Are these tools available to mere mortals?  If so, what are they
> called.  This is a problem I'm currently dealing with; trying to
> back up hundreds of terabytes of image data.

How many files, though?

Writes of file data *should* be limited mainly just be your network and
disk bandwidth.

Creation of files is limited by network and disk latency, is more
complicated, and is where multiple processes are more likely to help.

--b.