Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20190520223324.GL4158@turtle.email> <c10084e889df77fc2b6a6c9a04b232faae3a80bc.camel@hammerspace.com>
 <20190524173155.GQ4158@turtle.email>
In-Reply-To: <20190524173155.GQ4158@turtle.email>
From:   Trond Myklebust <trondmy@gmail.com>
Date:   Fri, 24 May 2019 15:19:22 -0400
Message-ID: <CAABAsM7DPHJyrfsOuse-wv4Vxwbd5BZ4DXQnAOiV7jVJE6sp2Q@mail.gmail.com>
Subject: Re: User process NFS write hang followed by automount hang requiring reboot
To:     Alan Post <adp@prgmr.com>
Cc:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On Fri, 24 May 2019 at 13:32, Alan Post <adp@prgmr.com> wrote:
>
> On Tue, May 21, 2019 at 03:46:03PM +0000, Trond Myklebust wrote:
> > Have you tried upgrading to 4.19.44? There is a fix that went in not
> > too long ago that deals with a request leak that can cause stack traces
> > like the above that wait forever.
> >
>
> Following up on this.  I have set aside a rack of machines and put
> Linux 4.19.44 on them.  They ran jobs overnight and will do the
> same over the long weekend (Memorial day in the US).  Given the
> error rate (both over time and over submitted jobs) we see across
> the cluster this well be enough time to draw a conclusion as to
> whether 4.19.44 exhibits this hang.
>
> Other than stack traces, what kind of information could I collect
> that would be helpful for debugging or describing more precisely
> what is happening to these hosts?  I'd like to exit from the condition
> of trying different kernels (as you no doubt saw in my initial message
> I've done a lot of it) and enter the condition of debugging or
> reproducing the problem.
>
> I'll report back early next week and appreciate your feedback,
>

Perhaps the output from 'cat /sys/kernel/debug/rpc_clnt/*/tasks'?

Thanks
  Trond