Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 6 Apr 2023 17:33:12 +0200
From:   Christian Herzog <herzog@phys.ethz.ch>
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: file server freezes with all nfsds stuck in D state after
 upgrade to Debian bookworm
Message-ID: <ZC7mOH4I3roIM4xr@phys.ethz.ch>
Reply-To: Christian Herzog <herzog@phys.ethz.ch>
References: <ZC6oX7FxdJd86rF7@phys.ethz.ch>
 <6785EFE7-2CE1-45CD-8643-C40CCCDADEB8@oracle.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <6785EFE7-2CE1-45CD-8643-C40CCCDADEB8@oracle.com>
Precedence: bulk

Dear Chuck,

> > for our researchers we are running file servers in the hundreds-of-TiB to
> > low-PiB range that export via NFS and SMB. Storage is iSCSI-over-Infiniband
> > LUNs LVM'ed into individual XFS file systems. With Ubuntu 18.04 nearing EOL,
> > we prepared an upgrade to Debian bookworm and tests went well. About a week
> > after one of the upgrades, we ran into the first occurence of our problem: all
> > of a sudden, all nfsds enter the D state and are not recoverable. However, the
> > underlying file systems seem fine and can be read and written to. The only way
> > out appears to be to reboot the server. The only clues are the frozen nfsds
> > and strack traces like
> > 
> > [<0>] rq_qos_wait+0xbc/0x130
> > [<0>] wbt_wait+0xa2/0x110
> 
> Hi Christian, you have a pretty deep storage stack!
> rq_qos_wait is a few layers below NFSD. Jens Axboe
> and linux-block are the folks who maintain that.
are you saying the root cause isn't nfs*, but the file system? That was our
first idea too, but we haven't found any indication that this is the case. The
xfs file systems seem perfectly fine when all nfsds are in D state, and we can
read from them and write to them. If xfs were to block nfs IO, this should
affect other processes too, right?

thanks and Happy Easter,
-Christian