Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 18 Jun 2019 18:07:46 -0600
From:   Alan Post <adp@prgmr.com>
To:     Benjamin Coddington <bcodding@redhat.com>
Cc:     linux-nfs <linux-nfs@vger.kernel.org>
Subject: Re: User process NFS write hang in wait_on_commit with kworker
Message-ID: <20190619000746.GT4158@turtle.email>
References: <20190618000613.GR4158@turtle.email>
 <6DE07E49-D450-4BF7-BC61-0973A14CD81B@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <6DE07E49-D450-4BF7-BC61-0973A14CD81B@redhat.com>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On Tue, Jun 18, 2019 at 11:29:16AM -0400, Benjamin Coddington wrote:
> I think that your transport or NFS server is dropping the response to an
> RPC.  The NFS client will not retransmit on an established connection.
> 
> What server are you using?  Any middle boxes on the network that could be
> transparently dropping transmissions (less likely, but I have seen them)?
> 

I've found 8 separate NFS client hangs of the sort I reported here,
and in all cases the same NFS server was involved: an Ubuntu Trusty
system running 4.4.0.  I've been upgrading all of these NFS servers,
haven't done this one yet--the complicity of NFS hangs I've been
seeing have slowed me down.

Of the 8 NFS clients with a hang to this server, about half are in
the same computer room where packets only transit rack switches, with
the other half also going through a computer room router.

I see positive dropped and overrun packet counts on the NFS server
interface, along with a similar magnitude of pause counts on the
switch port for the NFS server.  Given the occurences of this issue
only this rack switch and a redundant pair of top-of-rack switches in
the rack with the NFS server are in-common between all 8 NFS clients
with write hangs.

-A
-- 
Alan Post | Xen VPS hosting for the technically adept
PO Box 61688 | Sunnyvale, CA 94088-1681 | https://prgmr.com/
email: adp@prgmr.com