Received: by 2002:a25:6193:0:0:0:0:0 with SMTP id v141csp468166ybb; Wed, 25 Mar 2020 03:31:16 -0700 (PDT) X-Google-Smtp-Source: ADFU+vsd4myfRZPdn3mkjXesqg40L/Gbekopm7BRqWLOV0hwWTPCIwq1dLs0gXurmj0wm2rzSNZ9 X-Received: by 2002:a4a:2a47:: with SMTP id x7mr823477oox.23.1585132276256; Wed, 25 Mar 2020 03:31:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1585132276; cv=none; d=google.com; s=arc-20160816; b=uDGv/pSaFD8xB7As389GYyi2Gk8UMZld8PCITSFUXs8439hEQaxufQo0BWovkXOif5 Ek+WR4R+aWYZglE2YULtPNNWsbitSyckkfPcF0ig2nhMk1TpBfvRNaxEjA1/sBbeVjtx rtsP3z3/ZdgaCtbCy+AKTruE0+8yzH6S42qsZB2yCkZ06l+D8+lhG35sDFFyoEnIPcIV ADpw2512cMSM8miymCmo1+n/HCNVNqqvSaKOOl4mb0nipfSFVSQ6HNpWyNH8/qpwtEW4 vX/mtHOL5EMJrSeOO9WuTM+g++dra+BaG67K6gG784ed6Zvm29meK1GU0/wmQ8eGH3c4 IA5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:to:subject:message-id:date:from :mime-version:dkim-signature; bh=2vx7juo2SrcWwCvw0JhtUEkwXLe4JTpokN+5rvlXIBQ=; b=uxI7fWkZ08Vpji8N/ljemyBh+UZYZSF+SYyxZWz7OSGOLSIlVg2FHPtpK+hBsEkRIX qWd5ER5AmJSmTcDHeEauqwh7lwPHvNdbdwUCOyTPxTR/Wi865pp6j3k0Cbk12WGLbbPH m589IuXM91f6V1JT05gUsV1B1fx2/NbRtCK4xFtvKmvqq9mDp7EOnkVLLbERdmhDNmYC Qe9CfurUrvD9jNxH8WBIFklGjS5arIJwkFCtH9IrUELSNcgmUsrXnt/oKaQnQFAifHhM h4If4BDBKeosEr4emmedK1EPnPVLgDbUB97qXYtLOFaKNTjVFK8xMqAGoucQMC/VH9kC 1cmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JqK87J+g; spf=pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m8si982313oim.26.2020.03.25.03.30.54; Wed, 25 Mar 2020 03:31:16 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JqK87J+g; spf=pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726225AbgCYKav (ORCPT + 99 others); Wed, 25 Mar 2020 06:30:51 -0400 Received: from mail-lj1-f181.google.com ([209.85.208.181]:40475 "EHLO mail-lj1-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726139AbgCYKav (ORCPT ); Wed, 25 Mar 2020 06:30:51 -0400 Received: by mail-lj1-f181.google.com with SMTP id 19so1833969ljj.7 for ; Wed, 25 Mar 2020 03:30:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=2vx7juo2SrcWwCvw0JhtUEkwXLe4JTpokN+5rvlXIBQ=; b=JqK87J+gCehU5QntYxpZZ1q0wjOWXZxZ+Mx+Qa27DFkNfrimjV9L7ZmCVcoiDI7Ql+ fR+aykLygUWkQY71DuvLMFDS/nLFqMKT4q//T9Kum/e2k0Te7UCO7FWVCFo1xve5putk mQTt3nHeIL9cdOcwLGCBGCARuDO/E4OKClT204i+KIV4JiVwQ62HfsTC/3PK+s23qwWn Xudvw9enFQFwaYQjOSx6CSlGMMSRncBf34YikTP+jrQdgyc26RNhEX0BmyQhd9hasBu7 a4stwNd93dfRV7SAfbllkTnEA10fuZGShc0TmvHOxq5iZVSVF2Z0JjpIb+nrXaXRGbHW xWSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=2vx7juo2SrcWwCvw0JhtUEkwXLe4JTpokN+5rvlXIBQ=; b=sBVDg+WtzUUuZiLu5VgL0C3Fjveto9FlzO3+dpswz5FF5XORaYphOPQuCSqJBbNltT vt0L/5kl4dSkyCRwe8C5ksvXk9TM4aAbz5+FUI0MGhfWh5p+RBA54gbqktj0xUHIfx/R cqdzS97hCLkFY9Y9r8OMPnQ2px+OFflf7oDgFtFjLPx+7cK9TcbpYrsNEyJnfT9nX+jy N2gV8HQuF65wqKgnHOhNlsfViYmFnV/QoqY4e3TVC9qOO6GhE1U00KxF8bh+zugOKJr+ Drruh8pOp4HQbw0Fuq8GU+WN0BrUDglJRTRcnoy9NimSUVpiGmzse0tmz2+ZyOF5MOz4 LxeA== X-Gm-Message-State: AGi0PuaXYbME81mtLyYowe3xRGOzbyuk1Bp9RBffqvCsagiKk9kl2cD2 HyhRlIe51LkkH6wdVr7boUJ3RYBll/Z1OehKQisquCJ+ X-Received: by 2002:a2e:964e:: with SMTP id z14mr1626886ljh.44.1585132248069; Wed, 25 Mar 2020 03:30:48 -0700 (PDT) MIME-Version: 1.0 From: James Pearson Date: Wed, 25 Mar 2020 10:30:36 +0000 Message-ID: Subject: Stuck NFSv4 mounts of Isilon filer with repeated NFS4ERR_STALE_CLIENTID errors To: linux-nfs@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org We're seeing a number of Linux (CentOS 7.5) clients getting nfs: server isilon not responding, still trying' from various exports from a Isilon I appreciate we're using a vendor's Linux (out-of-date) kernel and a third party filer, but if anyone can give me any pointers of how to debug this issue, I would be grateful (we also have a support case open with the Isilon vendor) Running tshark on a client when this issue happens (taken several hours after the issue happened), we get repeating: 1 12:18:11 10.78.201.95 -> 10.78.196.184 NFS 194 V4 Call RENEW CID: 0xde68 2 12:18:11 10.78.196.184 -> 10.78.201.95 NFS 114 V4 Reply (Call In 1) RENEW Status: NFS4ERR_STALE_CLIENTID 4 12:18:16 10.78.201.95 -> 10.78.196.184 NFS 194 V4 Call RENEW CID: 0xde68 5 12:18:16 10.78.196.184 -> 10.78.201.95 NFS 114 V4 Reply (Call In 4) RENEW Status: NFS4ERR_STALE_CLIENTID 7 12:18:21 10.78.201.95 -> 10.78.196.184 NFS 194 V4 Call RENEW CID: 0xde68 8 12:18:21 10.78.196.184 -> 10.78.201.95 NFS 114 V4 Reply (Call In 7) RENEW Status: NFS4ERR_STALE_CLIENTID ... My knowledge of NFSv4 is sketchy, but from my (partial) reading of rfc7530 shouldn't the client be sending a SETCLIENTID in response to a NFS4ERR_STALE_CLIENTID - which doesn't appear to be happening here? Although the server hasn't rebooted since the client mounted the file system - so not sure what might be going on ? We are upgrading clients to the latest CentOS (RHEL) 7.7 to see if that 'fixes' the issue - but would appreciate any other pointers Thanks James Pearson