Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 9 Oct 2019 16:19:50 -0400
From:   Scott Mayhew <smayhew@redhat.com>
To:     "J. Bruce Fields" <bfields@fieldses.org>
Cc:     chuck.lever@oracle.com, linux-nfs@vger.kernel.org
Subject: Re: [PATCH] nfsd4: fix up replay_matches_cache()
Message-ID: <20191009201950.GG8791@coeurl.usersys.redhat.com>
References: <20191009191137.28007-1-smayhew@redhat.com>
 <20191009195121.GA23703@fieldses.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20191009195121.GA23703@fieldses.org>
User-Agent: Mutt/1.11.3 (2019-02-01)
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On Wed, 09 Oct 2019, J. Bruce Fields wrote:

> On Wed, Oct 09, 2019 at 03:11:37PM -0400, Scott Mayhew wrote:
> > When running an nfs stress test, I see quite a few cached replies that
> > don't match up with the actual request.  The first comment in
> > replay_matches_cache() makes sense, but the code doesn't seem to
> > match... fix it.
> 
> Thanks, I'll apply.  But I'm curious whether you're seeing any practical
> impact from this?  I don't think it should matter.

Yes, the client is occasionally getting tied up into knots.  It appears
to always be a REMOVE request getting a cached OPEN reply, and that
loops over and over.  It seems like a client bug because when it
happens, the client sends an OPEN and immediately sends a REMOVE using
the same slot (bumping the seqid) without waiting for the OPEN reply.
The server replies with NFS4ERR_SEQ_MISORDERED, and the client
decrements the seqid and re-sends the REMOVE request.  Then the server
sends the reply to the original OPEN and sends the cached OPEN reply in
response to all the subsequent REMOVE requests.  I haven't had much luck
in tracking it down though...

-Scott

> 
> --b.
> 
> > 
> > Fixes: 53da6a53e1d4 ("nfsd4: catch some false session retries")
> > Signed-off-by: Scott Mayhew <smayhew@redhat.com>
> > ---
> >  fs/nfsd/nfs4state.c | 15 ++++++++++-----
> >  1 file changed, 10 insertions(+), 5 deletions(-)
> > 
> > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> > index c65aeaa812d4..08f6eb2b73f8 100644
> > --- a/fs/nfsd/nfs4state.c
> > +++ b/fs/nfsd/nfs4state.c
> > @@ -3548,12 +3548,17 @@ static bool replay_matches_cache(struct svc_rqst *rqstp,
> >  	    (bool)seq->cachethis)
> >  		return false;
> >  	/*
> > -	 * If there's an error than the reply can have fewer ops than
> > -	 * the call.  But if we cached a reply with *more* ops than the
> > -	 * call you're sending us now, then this new call is clearly not
> > -	 * really a replay of the old one:
> > +	 * If there's an error then the reply can have fewer ops than
> > +	 * the call.
> >  	 */
> > -	if (slot->sl_opcnt < argp->opcnt)
> > +	if (slot->sl_opcnt < argp->opcnt && !slot->sl_status)
> > +		return false;
> > +	/*
> > +	 * But if we cached a reply with *more* ops than the call you're
> > +	 * sending us now, then this new call is clearly not really a
> > +	 * replay of the old one:
> > +	 */
> > +	if (slot->sl_opcnt > argp->opcnt)
> >  		return false;
> >  	/* This is the only check explicitly called by spec: */
> >  	if (!same_creds(&rqstp->rq_cred, &slot->sl_cred))
> > -- 
> > 2.17.2