Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 24 Sep 2021 15:03:52 -0400
From:   "J. Bruce Fields" <bfields@fieldses.org>
To:     Kazi Anwar <kazia@srf-ext.com>
Cc:     linux-nfs@vger.kernel.org
Subject: Re: nfs4err_delay
Message-ID: <20210924190352.GD13115@fieldses.org>
References: <CAGw7ksJxiKkOf2F9FUDCa_mSAkoU6U=vCXFcupsu+izKuEE1WA@mail.gmail.com>
 <20210924163948.GB13115@fieldses.org>
 <CAGw7ksJGjG0PUZqr+U+ssq_6YHgJ4cga40MCdpm8b_rYn1a5OA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAGw7ksJGjG0PUZqr+U+ssq_6YHgJ4cga40MCdpm8b_rYn1a5OA@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk

On Fri, Sep 24, 2021 at 12:54:30PM -0500, Kazi Anwar wrote:
> Yes, both clients and server are centos 7.6. And the NFS4ERR_DELAY is
> a response to a REMOVE.
> I will need to check on the locks the next time it happens. Can you
> share what you are thinking?

Offhand the only reason I can think a server would return DELAY is that
there's a delegation on the file being removed, and the delegation
recall and return isn't working for some reason.

If that's the case, it should succeed after about 90 seconds.  Also, you
can workaround the problem by turning of delegations and leases with
"echo 0 >/proc/sys/fs/leases_enable".

--b.

> 
> thanks,
> Kazi
> 
> On Fri, Sep 24, 2021 at 11:39 AM J. Bruce Fields <bfields@fieldses.org> wrote:
> >
> > On Wed, Sep 22, 2021 at 03:56:23PM -0500, Kazi Anwar wrote:
> > > We are running nfs v 4.1 on centos 7.6.
> > > We are seeing an NFS issue where when files/dirs are deleted from a
> > > client they are occasionally stuck at unlinkat system call(according
> > > to strace its stuck for 100.5 secs every time). Can anyone explain
> > > this behavior?
> > > Running tcp dump shows nfs4err_delay status sent from the server to
> > > the stuck client.
> >
> > Client and server are both centos 7.6?
> >
> > Is the NFS4ERR_DELAY a reponse to a REMOVE?
> >
> > Does /proc/locks show a delegation held on the file the client's trying
> > to remove?
> >
> > --b.