Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CAPt2mGOaRsKOiL_wuSK_D5oYYnn0R-pvVsZc5HYGdEbT2FngtQ@mail.gmail.com>
 <20220124193759.GA4975@fieldses.org> <CAPt2mGOCn5OaeZm24+zh92qRcWTF8h-H2WXqScz9RMfo4r_-Qw@mail.gmail.com>
 <20220124205045.GB4975@fieldses.org> <CAPt2mGPTGgXztawDJfAKsiYqnm6P_mn1rtquSDKjpnSgvJH1YA@mail.gmail.com>
 <20220125135959.GA15537@fieldses.org> <F7C721F7-D906-426F-814F-4D3F34AD6FB1@oracle.com>
 <42867c2c-1ab3-9bb6-0e5a-57d13d667bc6@math.utexas.edu> <20220125215942.GC17638@fieldses.org>
 <7256e781-5ab8-2b39-cb69-58a73ae48786@math.utexas.edu> <CAPt2mGNMGjq+i=k_6oYBYPFPCTR2UdeEtWfyeTU9uUC0OC=T4w@mail.gmail.com>
 <a5627c80-4b03-29f2-1432-6e0f0b5197ef@math.utexas.edu>
In-Reply-To: <a5627c80-4b03-29f2-1432-6e0f0b5197ef@math.utexas.edu>
From:   Daire Byrne <daire@dneg.com>
Date:   Tue, 25 Jan 2022 23:25:33 +0000
Message-ID: <CAPt2mGMsd1q1B42bgdKciaWjB9O+9wezr_vU9JRpzzYEc6m+3g@mail.gmail.com>
Subject: Re: parallel file create rates (+high latency)
To:     Patrick Goetz <pgoetz@math.utexas.edu>
Cc:     Bruce Fields <bfields@fieldses.org>,
        Chuck Lever III <chuck.lever@oracle.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, 25 Jan 2022 at 23:01, Patrick Goetz <pgoetz@math.utexas.edu> wrote:
> On 1/25/22 16:41, Daire Byrne wrote:
> > On Tue, 25 Jan 2022 at 22:11, Patrick Goetz <pgoetz@math.utexas.edu> wrote:
> Thanks for this suggestion! This option didn't even occur to me.  The
> only downside is that this server gets really busy during image
> processing, so I'm a bit worried about loading it down with dozens of
> simultaneous rsync processes. Also, the biggest performance problem in
> this system (which includes multiple GPU-laden workstations and 2 other
> NFS servers) is always I/O bottlenecks.  I suppose the solution is to
> nice all the rsync processes to 19.

Yea, that's always the problem with backups - when do you slow down
production because backups are more important? :)

You could also have another NFS client close (latency) to the server
which would free CPU but it's hard to limit IO. You can still use
rsync+rsyncd for the higher latency hop.

> Question: given that I usually run backups from cron, and given that
> they can take a long time, how does msrsync avoid stepping on itself?

I guess you would need a lockfile or pgrep for a running instance.
TBH, I wrote a simpler version of msrsync in bash that was more suited
to our environment but the principle is the same.

Daire