Received-SPF: pass (google.com: domain of linux-nfs+bounces-3015-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
MIME-Version: 1.0
From: "NeilBrown" <neilb@suse.de>
To: "Chris Packham" <Chris.Packham@alliedtelesis.co.nz>
Cc: "Chuck Lever III" <chuck.lever@oracle.com>,
 "Jeff Layton" <jlayton@kernel.org>,
 "Greg Kroah-Hartman" <gregkh@linuxfoundation.org>,
 "Linux NFS Mailing List" <linux-nfs@vger.kernel.org>,
 "netdev" <netdev@vger.kernel.org>,
 "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
 "stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: Re: kernel BUG at net/sunrpc/svc.c:570 after updating from v5.15.153
 to v5.15.155
In-reply-to: <141fbaa0-f8fa-4bfe-8c2d-7749fcf78ab3@alliedtelesis.co.nz>
References: <>, <141fbaa0-f8fa-4bfe-8c2d-7749fcf78ab3@alliedtelesis.co.nz>
Date: Fri, 26 Apr 2024 09:18:40 +1000
Message-id: <171408712052.7600.17665015984185673222@noble.neil.brown.name>

On Fri, 26 Apr 2024, Chris Packham wrote:
>=20
> On 25/04/24 11:37, NeilBrown wrote:
> > On Thu, 25 Apr 2024, Chuck Lever III wrote:
> >>> On Apr 24, 2024, at 9:33=E2=80=AFAM, Chuck Lever III <chuck.lever@oracl=
e.com> wrote:
> >>>
> >>>> On Apr 24, 2024, at 3:42=E2=80=AFAM, Chris Packham <Chris.Packham@alli=
edtelesis.co.nz> wrote:
> >>>>
> >>>> On 24/04/24 13:38, Chris Packham wrote:
> >>>>> On 24/04/24 12:54, Chris Packham wrote:
> >>>>>> Hi Jeff, Chuck, Greg,
> >>>>>>
> >>>>>> After updating one of our builds along the 5.15.y LTS branch our
> >>>>>> testing caught a new kernel bug. Output below.
> >>>>>>
> >>>>>> I haven't dug into it yet but wondered if it rang any bells.
> >>>>> A bit more info. This is happening at "reboot" for us. Our embedded
> >>>>> devices use a bit of a hacked up reboot process so that they come back
> >>>>> faster in the case of a failure.
> >>>>>
> >>>>> It doesn't happen with a proper `systemctl reboot` or with a SYSRQ+B
> >>>>>
> >>>>> I can trigger it with `killall -9 nfsd` which I'm not sure is a
> >>>>> completely legit thing to do to kernel threads but it's probably close
> >>>>> to what our customized reboot does.
> >>>> I've bisected between v5.15.153 and v5.15.155 and identified commit
> >>>> dec6b8bcac73 ("nfsd: Simplify code around svc_exit_thread() call in
> >>>> nfsd()") as the first bad commit. Based on the context that seems to
> >>>> line up with my reproduction. I'm wondering if perhaps something got
> >>>> missed out of the stable track? Unfortunately I'm not able to run a mo=
re
> >>>> recent kernel with all of the nfs related setup that is being used on
> >>>> the system in question.
> >>> Thanks for bisecting, that would have been my first suggestion.
> >>>
> >>> The backport included all of the NFSD patches up to v6.2, but
> >>> there might be a missing server-side SunRPC patch.
> >> So dec6b8bcac73 ("nfsd: Simplify code around svc_exit_thread()
> >> call in  nfsd()") is from v6.6, so it was applied to v5.15.y
> >> only to get a subsequent NFSD fix to apply.
> >>
> >> The immediately previous upstream commit is missing:
> >>
> >>    390390240145 ("nfsd: don't allow nfsd threads to be signalled.")
> >>
> >> For testing, I've applied this to my nfsd-5.15.y branch here:
> >>
> >>    https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
> >>
> >> However even if that fixes the reported crash, this suggests
> >> that after v6.6, nfsd threads are not going to respond to
> >> "killall -9 nfsd".
> > I think this likely is the problem.  The nfsd threads must be being
> > killed by a signal.
> > One only other cause for an nfsd thread to exit is if
> > svc_set_num_threads() is called, and all places that call that hold a
> > ref on the serv structure so the final put won't happen when the thread
> > exits.
> >
> > Before the patch that bisect found, the nfsd thread would exit with
> >
> >   svc_get();
> >   svc_exit_thread();
> >   nfsd_put();
> >
> > This also holds a ref across the svc_exit_thread(), and ensures the
> > final 'put' happens from nfsD_put(), not svc_put() (in
> > svc_exit_thread()).
> >
> > Chris: what was the context when the crash happened?  Could the nfsd
> > threads have been signalled?  That hasn't been the standard way to stop
> > nfsd threads for a long time, so I'm a little surprised that it is
> > happening.
>=20
> We use a hacked up version of shutdown from util-linux and which does a=20
> `kill (-1, SIGTERM);` then `kill (-1, SIGKILL);` (I don't think that=20
> particular behaviour is the hackery). I'm not sure if -1 will pick up=20
> kernel threads but based on the symptoms it appears to be doing so (or=20
> maybe something else is in it's SIGTERM handler). I don't think we were=20
> ever really intending to send the signals to nfsd so whether it actually=20
> terminates or not I don't think is an issue for us. I can confirm that=20
> applying 390390240145 resolves the symptom we were seeing.
>=20
>=20

Makes sense - thanks.
"kill -1 ..." does send the signal to *every* process including kernel
threads.  I'm glad you weren't depending on that to kill nfsd.
Hopefully no one else is.  I think the best way forward is to apply that
patch to 5.15-stable as Chuck plans to do.

Thanks,
NeilBrown