Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Fri, 28 Jun 2019 12:33:24 -0600
From:   Alan Post <adp@prgmr.com>
To:     linux-nfs <linux-nfs@vger.kernel.org>
Subject: Re: User process NFS write hang in wait_on_commit with kworker
Message-ID: <20190628183324.GJ4158@turtle.email>
References: <20190618000613.GR4158@turtle.email>
 <6DE07E49-D450-4BF7-BC61-0973A14CD81B@redhat.com>
 <20190619000746.GT4158@turtle.email>
 <25608EB2-87F0-4196-BEF9-8AB8FC72270B@redhat.com>
 <20190621204723.GU4158@turtle.email>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20190621204723.GU4158@turtle.email>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On Fri, Jun 21, 2019 at 02:47:23PM -0600, Alan Post wrote:
> > Verifying this is the problem could be done by setting up some rolling
> > network captures.. but sometimes it can be hard to not have the capture
> > fill up with continuing traffic from other processes.
> > 
> 
> I did go ahead and set up a rolling capture between this NFS
> server and one rack of clients--I hope I can catch the event as
> it happens.  Time will tell.
> 

I've run this rolling capture and did catch four candidate events.
I haven't confirmed any of them are real--I don't really know
what it is I'm looking for, so I've been approaching the problem
by incrementally/recursively throwing stuff out and manually
working through what's left.

As far as I understand it, for a particular xid, there should be a
call and a reply.  The approach I took then was to pull out these
fields from my capture and ignore RPC calls where both are present
in my capture.  It seems this is simplistic, as the number of RPC
calls I have without an attendant reply isn't lining up with my
incident window.

In one example, I have a series of READ calls which cease
generating RPC reply messages as the offset for the file continues
to increases.  After a couple/few dozen messages, the RPC replies
continue as they were.  Is there a normal or routine explanation
for this?

RFC 5531 and the NetworkTracing page on wiki.linux-nfs.org have
been quite helpful bringing me up to speed.  If any of you have
advice or guidance or can clarify my understanding of how the
call/reply RPC mechanism works I appreciate it.

-A
-- 
Alan Post | Xen VPS hosting for the technically adept
PO Box 61688 | Sunnyvale, CA 94088-1681 | https://prgmr.com/
email: adp@prgmr.com