Received-SPF: pass (google.com: domain of linux-nfs+bounces-3005-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Thu, 25 Apr 2024 11:50:02 -0400
From: Chuck Lever <chuck.lever@oracle.com>
To: Chris Packham <Chris.Packham@alliedtelesis.co.nz>,
        Neil Brown <neilb@suse.de>
Cc: Jeff Layton <jlayton@kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        netdev <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: Re: kernel BUG at net/sunrpc/svc.c:570 after updating from v5.15.153
 to v5.15.155
Message-ID: <Zip7qnyExaCNZhSy@tissot.1015granger.net>
References: <b363e394-7549-4b9e-b71b-d97cd13f9607@alliedtelesis.co.nz>
 <0d2c2123-e782-4712-8876-c9b65d2c9a65@alliedtelesis.co.nz>
 <06de0002-c3c6-4f13-9618-066cb9658240@alliedtelesis.co.nz>
 <1235583F-8299-435B-A8C3-41DEB917D6CA@oracle.com>
 <5D19EAF8-0F65-4CD6-9378-67234D407B96@oracle.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <5D19EAF8-0F65-4CD6-9378-67234D407B96@oracle.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?Q3ArY1ljc3NiNk1KT1BOTnQ0V01GQTl4RnVzazl6bWJJOHlSNFpUTmxxWGFl?=
 =?utf-8?B?d05JTy95MVV0N1NIcVZYNzdPVkltYUxDeGFrdlU3WUZzVXJhdGUrcFhLQ3du?=
 =?utf-8?B?amNIM1lmdTFPb2VpYkowSHJzYkdURjU5S1p2ME1qWG1UQ2FsK1BDSzJwNlM2?=
 =?utf-8?B?TEMzdFZON3RSd0IvaUFmMENOU1pUS3RHeEJYYkE1MEZlTzZIWXNGNmNNSVJS?=
 =?utf-8?B?MUtDRWtDNXZmdEtJOWEvcVlmbGw2RFJZNVZDYURLUnVhMWlqQ3dIaFVGTkNG?=
 =?utf-8?B?b00rTWxOTWVFYXFhc0VaQU4weXV4cFVmWitMeEkrRW1naUoxN1pEczFwTnR3?=
 =?utf-8?B?Y2E1TmQ2Ym9Nci8zTW4wc0N4NEVOQ29tUzMrbFZzQ0xYSmlwOUpWaTdPTGJN?=
 =?utf-8?B?MHNLaVJUM1ljK2ZqS1poTHdzYWZIa1l5VHI0QjkxMzlJZTFzZE5tQW9NajF4?=
 =?utf-8?B?T1VOeHpvcU03RVVaUXhYNXBEYkRsUGpXSE85TjJRd1ozMUpuUjJRNzVCSTJs?=
 =?utf-8?B?OGJEeCttQW1QVmtLSXpFYldYZjQwVEFHc09pYmtzV2M1UUpyQ3dxYnY2RjM3?=
 =?utf-8?B?c0tjR00wTzFqM3ZjSUhDUXVlb3pleDZSNXNscVFhMXlKTWhtVFF5VnFveURK?=
 =?utf-8?B?NjAxSDBYeGozT1dIYWVqRllCeHNTTnZpM1dXVjMzUk8zNUlmQUV0eDRodG1r?=
 =?utf-8?B?bi85WWlINlNiMWk2SHhYTmxQOUV4bm1PQzlnQVdXd2Y3UGxUQ2o1c3JEQXkz?=
 =?utf-8?B?V292QjZoN1dyOWdDOXowZ25PODBjU3QrbFEvYzl5Sm5kOCtsZitNK2RqVFNX?=
 =?utf-8?B?ZUZSU0h3Y3JxY3MrMDNuQko4L3FmOWU2cDhYLzJYOFVYTHVsb01qUktydk95?=
 =?utf-8?B?NUJyTDRLaTBVdjl4ZHRUbzBLdUpYbUpOSGpXbk1hUUx0cXBGUWhJbi83THN3?=
 =?utf-8?B?VUdyNG0xY2hhZlpLK0lFcENqVFkzaFUxYmlsODk0UjBUblZyU0ZiMTlqZWN2?=
 =?utf-8?B?SjhySUtjUDh4NFpRMlpxQUlZMDd6WVI4eVpOZHRiZ3NBRzBnbDExQ2ZMc3JL?=
 =?utf-8?B?cEJwMmRpR0hOZkhJVlZIM205Qm1pU2ltcW0yRDdBQnVDVHhXajIrWmxOMlBl?=
 =?utf-8?B?ang4OHJWbGlQMVhMTDU3Q0FqTXI4b2kwQlJlcWVobTFPQmczRlo5QkFMTXBz?=
 =?utf-8?B?NGJXa2RjSU12elhkVXZlVFp6Vk1hOFNmT0ZnQk5ta2tUajhOeGo0S2VyZC9G?=
 =?utf-8?B?RUF6cGptNEowbmpLeEsvbGc2cXdvVEJaQy82MmJ4Snlodk9OeXo3SXNYR1B4?=
 =?utf-8?B?TXpiekFMK3FReWF1VG9jU0ZXdFFGZTBVZjJja1NyYi8wZ0Q0WDhoZXB1V2xW?=
 =?utf-8?B?UmZjcTh0VFJ2dFZMNGJJQm9lVTZ0dGs2YXFkaTBEejRjSEVGcFA4MldJay9I?=
 =?utf-8?B?Ym1NRmZQUzE0VWY3WVpxK3c4dzluZFBKSkk5ajlEMDFqSCt0eUNuVHltaHZH?=
 =?utf-8?B?bXZaTHl2UHc1eS85WW9oMnBJc3hmYkQrNnhCQWhSeURUOHpheTdQeUhJRWg0?=
 =?utf-8?B?V0E2djV0Q0RTMmk5MHpyL2FXR3V5ZnJJNDJJRUNJWHVJejhyQlNURUJScDdL?=
 =?utf-8?B?S3ZLYWlldVlyL1FnL2gzQWhhclgxbkhGVU9IY0RXbE1oa2ZFYTB3S2tsV2JH?=
 =?utf-8?B?MEV0ekxxOFlKMVNFSW5rSVFyL1gwSXkyK1BXOHFDbTVETE9mT05OTGFTY3ZT?=
 =?utf-8?B?enNQNTdXajZWeU9HcGlQcklQMnlCRzlBWENObnRPVXd6N2txaFE2OTBhQUxJ?=
 =?utf-8?B?d2l6WFV5UG5tR3BSOVlBaHJpWnp0YkpLMDZsekFRMmpOOEllSVczRVZvM0NV?=
 =?utf-8?B?ZDlZd3Y0SUFtUldwemV2VXR6ZlBTOE9kamI1REZHYTZUSTIxVFd3N3NJL0xi?=
 =?utf-8?B?N0JwMlhQOU1CeThiQk1KWm9HZmlsdzYzVEtsSmFPMXpqK2RzYVUyVFVXdm50?=
 =?utf-8?B?K1RXK0tacmFVekpVS0dPdWo1ZnpIRzdlMms2dTlWNk9mbEcwTlM5NXdBc3RU?=
 =?utf-8?B?Q09kL01qQ20wdFZrNm1PSXRVaDBSQ0JlUHRQUDJhaG5pbXRjcjRaM2NQaFM5?=
 =?utf-8?B?NStPdEVuWnQ1YWFBQlhUSXBCSzRSUkdKWmpaZi9rRkl5ZEV2Z2dyVVlvVUg4?=
 =?utf-8?B?VlE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	hBC+6FAXmUU/N7frih277YCF7+MpA9yGj7PNK/mBpwpjaOP5RJ3YslVwuV9jRDdDsmt5By8dOtQWcYIFbUC11WKJ/XWJn76ernfuN38JKyDZR4zT94332MTI6JtQOoypTZqc/LrZ4W/sGuoKnNgwusO9KeLmXp3n+gxzqmbO5E8YoNqVa2fZEEpAO75Rjt2tDEL896FU2XSw5a4QqfFDMscA5pqNtGHYeiAFUT/CcUDXVV2bT/4SVc0vMpgEyE8CPEWIdo7i5P1TrbxFyrlCxLUAK1/hg7pO5wtSfcPblajEgT9omb1yfJ9aUT9KwrOAJIj51ouBqlM/cLBA0do/yHm7XTbLF5yaR59bCLxDaRcSVuPoclbaNhGn/jqw4YutaiGj1I1JNg0gZYvzWsvSmVVy2wtkAUPcdvoChYDHOhaNP6VqaZSVDGAAiZIXcBFxNztT+d+jKu0BTuwnbUOCtK+LcQ12Q0Sly/AjuxzSXnqV9UoWt/504GcHZEN9N9RcXHTsczNAsSgIoV1hp+TvlmotmMvCpfunFWUThTgG5rPaUmUul5HioI3aknJvB9i7msQfH6x4MHjLLvNicRmTw1wNUFqYGnxeeUp+kj2O0nc=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6b90647b-c00c-4396-eef7-08dc653f6045
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Apr 2024 15:50:05.3784
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: M/Q3wMbbxi9VNLxAbREVnLWMDwMbOV44L1ckonGjqyP7pjMnUWPVjTAMzH38kZvT2ixS5y0n0LNoKqthGNVJYw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5582
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1011,Hydra:6.0.650,FMLib:17.11.176.26
 definitions=2024-04-25_15,2024-04-25_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0 mlxscore=0
 mlxlogscore=999 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000
 definitions=main-2404250115
X-Proofpoint-GUID: oB3S4kkn7U3q1ep52XWIDo3GmsK4Pr9R
X-Proofpoint-ORIG-GUID: oB3S4kkn7U3q1ep52XWIDo3GmsK4Pr9R

On Wed, Apr 24, 2024 at 02:03:22PM +0000, Chuck Lever III wrote:
> 
> > On Apr 24, 2024, at 9:33 AM, Chuck Lever III <chuck.lever@oracle.com> wrote:
> > 
> >> On Apr 24, 2024, at 3:42 AM, Chris Packham <Chris.Packham@alliedtelesis.co.nz> wrote:
> >> 
> >> On 24/04/24 13:38, Chris Packham wrote:
> >>> 
> >>> On 24/04/24 12:54, Chris Packham wrote:
> >>>> Hi Jeff, Chuck, Greg,
> >>>> 
> >>>> After updating one of our builds along the 5.15.y LTS branch our 
> >>>> testing caught a new kernel bug. Output below.
> >>>> 
> >>>> I haven't dug into it yet but wondered if it rang any bells.
> >>> 
> >>> A bit more info. This is happening at "reboot" for us. Our embedded 
> >>> devices use a bit of a hacked up reboot process so that they come back 
> >>> faster in the case of a failure.
> >>> 
> >>> It doesn't happen with a proper `systemctl reboot` or with a SYSRQ+B
> >>> 
> >>> I can trigger it with `killall -9 nfsd` which I'm not sure is a 
> >>> completely legit thing to do to kernel threads but it's probably close 
> >>> to what our customized reboot does.
> >> 
> >> I've bisected between v5.15.153 and v5.15.155 and identified commit 
> >> dec6b8bcac73 ("nfsd: Simplify code around svc_exit_thread() call in 
> >> nfsd()") as the first bad commit. Based on the context that seems to 
> >> line up with my reproduction. I'm wondering if perhaps something got 
> >> missed out of the stable track? Unfortunately I'm not able to run a more 
> >> recent kernel with all of the nfs related setup that is being used on  
> >> the system in question.
> > 
> > Thanks for bisecting, that would have been my first suggestion.
> > 
> > The backport included all of the NFSD patches up to v6.2, but
> > there might be a missing server-side SunRPC patch.
> 
> So dec6b8bcac73 ("nfsd: Simplify code around svc_exit_thread()
> call in  nfsd()") is from v6.6, so it was applied to v5.15.y
> only to get a subsequent NFSD fix to apply.
> 
> The immediately previous upstream commit is missing:
> 
>   390390240145 ("nfsd: don't allow nfsd threads to be signalled.")
> 
> For testing, I've applied this to my nfsd-5.15.y branch here:
> 
>   https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
> 
> However even if that fixes the reported crash, this suggests
> that after v6.6, nfsd threads are not going to respond to
> "killall -9 nfsd".

I cannot reproduce a crash when using "killall -9 nfsd" on the fixed
kernel above. On that kernel, the killall command does not terminate
the nfsd threads either.

Either we can apply 390390240145 to 5.15.y, or if the imperviousness
to "kill" is considered a regression, I can look into stripping out
dec6b8bcac73 and the subsequent commits that depend on it.


-- 
Chuck Lever