Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <CAK3fRr_Yh9Ko+E3-=z6g8p0vvUA9QW+PAoQ+ct3EWya9_oxZ3w@mail.gmail.com>
 <9932cd04adc20b73040d82c605acb0b5a24e3855.camel@hammerspace.com>
In-Reply-To: <9932cd04adc20b73040d82c605acb0b5a24e3855.camel@hammerspace.com>
From:   James Pearson <jcpearson@gmail.com>
Date:   Wed, 25 Mar 2020 13:53:26 +0000
Message-ID: <CAK3fRr9BcGbKRj4PcOdQVabPS2MkUzyEXyOq+XsTjGN4VxmaCA@mail.gmail.com>
Subject: Re: Stuck NFSv4 mounts of Isilon filer with repeated
 NFS4ERR_STALE_CLIENTID errors
To:     Trond Myklebust <trondmy@hammerspace.com>
Cc:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On Wed, 25 Mar 2020 at 12:22, Trond Myklebust <trondmy@hammerspace.com> wrote:
>
> On Wed, 2020-03-25 at 10:30 +0000, James Pearson wrote:
> > We're seeing a number of Linux (CentOS 7.5) clients getting nfs:
> > server isilon not responding, still trying'  from various exports
> > from
> > a Isilon
> >
> > I appreciate we're using a vendor's Linux (out-of-date) kernel and a
> > third party filer, but if anyone can give me any pointers of how to
> > debug this issue, I would be grateful (we also have a support case
> > open with the Isilon vendor)
> >
> > Running tshark on a client when this issue happens (taken several
> > hours after the issue happened), we get repeating:
> >
> >   1   12:18:11 10.78.201.95 -> 10.78.196.184 NFS 194 V4 Call RENEW
> > CID: 0xde68
> >   2   12:18:11 10.78.196.184 -> 10.78.201.95 NFS 114 V4 Reply (Call
> > In
> > 1) RENEW Status: NFS4ERR_STALE_CLIENTID
> >   4   12:18:16 10.78.201.95 -> 10.78.196.184 NFS 194 V4 Call RENEW
> > CID: 0xde68
> >   5   12:18:16 10.78.196.184 -> 10.78.201.95 NFS 114 V4 Reply (Call
> > In
> > 4) RENEW Status: NFS4ERR_STALE_CLIENTID
> >   7   12:18:21 10.78.201.95 -> 10.78.196.184 NFS 194 V4 Call RENEW
> > CID: 0xde68
> >   8   12:18:21 10.78.196.184 -> 10.78.201.95 NFS 114 V4 Reply (Call
> > In
> > 7) RENEW Status: NFS4ERR_STALE_CLIENTID
> > ...
> >
> > My knowledge of NFSv4 is sketchy, but from my (partial) reading of
> > rfc7530 shouldn't the client be sending a SETCLIENTID in response to
> > a
> > NFS4ERR_STALE_CLIENTID - which doesn't appear to be happening here?
> >
> > Although the server hasn't rebooted since the client mounted the file
> > system - so not sure what might be going on ?
> >
> > We are upgrading clients to the latest CentOS (RHEL) 7.7 to see if
> > that 'fixes' the issue - but would appreciate any other pointers
> >
>
> WAG: the clients all have the default hostname 'localhost.localdomain'
> and are using that to identify themselves in the SETCLIENTID call? If
> so, that would cause them to cancel each other's leases by declaring
> client reboots of the client with name 'localhost.localdomain'.

All the the clients have their hostname set as something like
hostname.our.domain

The Isilon has the NFSv4 domain set as 'our.domain'

Is it possible to find out (after the fact) what the host/domain name
the client used to connect to the server?

Thanks

James Pearson