Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 21 May 2019 13:22:54 -0600
From:   Alan Post <adp@prgmr.com>
To:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: User process NFS write hang followed by automount hang requiring
 reboot
Message-ID: <20190521192254.GN4158@turtle.email>
References: <20190520223324.GL4158@turtle.email>
 <c10084e889df77fc2b6a6c9a04b232faae3a80bc.camel@hammerspace.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <c10084e889df77fc2b6a6c9a04b232faae3a80bc.camel@hammerspace.com>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On Tue, May 21, 2019 at 03:46:03PM +0000, Trond Myklebust wrote:
> > A representative sample of stack traces from hung user-submitted
> > processes (jobs).  The first here is quite a lot more common than
> > the later two:
> > 
> >     $ sudo cat /proc/197520/stack
> >     [<0>] io_schedule+0x12/0x40
> >     [<0>] nfs_lock_and_join_requests+0x309/0x4c0 [nfs]
> >     [<0>] nfs_updatepage+0x2a2/0x8b0 [nfs]
> >     [<0>] nfs_write_end+0x63/0x4c0 [nfs]
> >     [<0>] generic_perform_write+0x138/0x1b0
> >     [<0>] nfs_file_write+0xdc/0x200 [nfs]
> >     [<0>] new_sync_write+0xfb/0x160
> >     [<0>] vfs_write+0xa5/0x1a0
> >     [<0>] ksys_write+0x4f/0xb0
> >     [<0>] do_syscall_64+0x53/0x100
> >     [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >     [<0>] 0xffffffffffffffff
> > 
> 
> Have you tried upgrading to 4.19.44? There is a fix that went in not
> too long ago that deals with a request leak that can cause stack traces
> like the above that wait forever.
> 

That I haven't tried.  I gather you're talking about either or both
of:

    63b0ee126f7e
    be74fddc976e

Which I do see went in after 4.19.24 (which I've tried) but didn't
get in 4.20.9 (which I've also tried).  Let me see about trying the
4.19.44 kernel.

> By the way, the above stack trace with "nfs_lock_and_join_requests"
> usually means that you are using a very small rsize or wsize (less than
> 4k). Is that the case? If so, you might want to look into just
> increasing the I/O size.
> 

These exports have rsize and wsize set to 1048576.  That decision was
before my time, and I'll guess this value was picked to match
NFSSVC_MAXBLKSIZE.

Thank you for your help,

-A
-- 
Alan Post | Xen VPS hosting for the technically adept
PO Box 61688 | Sunnyvale, CA 94088-1681 | https://prgmr.com/
email: adp@prgmr.com