Received-SPF: pass (google.com: domain of linux-nfs+bounces-306-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <537b96d3-1d8a-4eaa-b271-e103f73e980d@oracle.com>
Date: Mon, 4 Dec 2023 11:12:52 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: kernel v6.6.3: nfsd hangs in nfsd_break_deleg_cb
Content-Language: en-US
To: Chuck Lever <chuck.lever@oracle.com>, Wolfgang Walter <linux-nfs@stwm.de>
Cc: Linux Nfs <linux-nfs@vger.kernel.org>
References: <e3d43ecdad554fbdcaa7181833834f78@stwm.de>
 <ZW37M7DOavddVpFd@tissot.1015granger.net>
From: dai.ngo@oracle.com
In-Reply-To: <ZW37M7DOavddVpFd@tissot.1015granger.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?YTgxaVJ1TVYzaStsckwzc2NIeEhKN1A1aUZDcmJOdllienRzZncwNWJVRFBs?=
 =?utf-8?B?eVlsNUdKU1YxTlg4QklBVDVlRWlPZmxEdDNnTndtQm1DNzdnUGFHdERzVlN4?=
 =?utf-8?B?c1hSc2g2dmN3akdoTTNyb0hwQXVEOTA1NFJFUjRrakpVSVRxTldhejRFakh2?=
 =?utf-8?B?TC9XUWpmVGlJS2F6MWY2SnZLcmNVZFVzZHhOSy9VNFMxZHI4cG8rNDVqUHRO?=
 =?utf-8?B?K3Vic0dXQ0paUnp1dU5nWnRlODB6M3lwdEcwekxaM0VTcno1OVhWSzRIQnd0?=
 =?utf-8?B?alRIN1o0V25lOGVJZFNNQUdFaTlYeU14aTU4cEs2TUw3NG4vZWNCcDJHaUxS?=
 =?utf-8?B?dThjamJJQmtWM1RaWldEL1d4WlFOUUVQYVFFNkl6Y0M3K3NidXRybVRKeGFk?=
 =?utf-8?B?SXprbit2SXF5RmtJQ29SNXB6azB4aTJMZDZuaXBCc0NlemFyZ3NxbWxpV0dx?=
 =?utf-8?B?TGp0ekdaYWNsa2xyOVExTGU1eExxajRocmxoaUxjWWVlZks5azY1amJzb1Nw?=
 =?utf-8?B?Sk51WGh4Nklsb0Rka0Vjcmd5R2N6MkZ0a1VGQ0Yydmw3VlJHZ3ZrTlIrN05E?=
 =?utf-8?B?T054RUw2eG9vNm9PdVhpTUIwL0xDZVFnalpyalJxYnJsTG92UVBON1dNcWx6?=
 =?utf-8?B?ZG9kamNkYWg0cXBRTnBDakpCQXlrSTIzNWlSRUR3bUpsQTdpWHo0aGF4dGxO?=
 =?utf-8?B?UUEzdFJ1Q1J4WkRxTTZkclUzTkw5cmRQOVFGSXFMZVdJWHRRMXFQUCszWkI2?=
 =?utf-8?B?T1JUQlQyMjhaWkl6RnZpWk44cmRJVVFIT1dRd1UxMSs0akpJTHlBODB4eWVP?=
 =?utf-8?B?bEl5Kytta1VjWnBUZHovRjNYekV6d3ArcVBBRklYZ1dqek5sVldJRnpFeGl2?=
 =?utf-8?B?ZEx4UE05MEs1ak9ubk5RelhwZ0Y2Ujh0ck1UNzVtdFZKTFBkSG5YWmhKSVNj?=
 =?utf-8?B?MHg5SGRVK3VpVllYQXU5U2pKM1lpMlpjQmtlMkF4S2ZGYzg5Ym4rdlRkcEZq?=
 =?utf-8?B?c1BWZ3lMQjFwa3hHQlBTeFhiT1E1Rlg4N3FOL1VvUnJwODU1M01VNGNsdFJI?=
 =?utf-8?B?RUhVRTB3enVVMHFXekc4WUpJZUl0TEFjdkhnUnBHT3VWZ0JjZ2lTNDZtUXlB?=
 =?utf-8?B?ZjBHT3F3eEtBRzA3UHpEanlsZDdQWHM1Mk5FVllIQ0JZYjB5ZGVGV2ZCeXFz?=
 =?utf-8?B?UlRPeHB4VTFHWVRNc1NrRGdhbFZPcEtGSStlV1hEMkdlRlN5QlBCdXRXRWFU?=
 =?utf-8?B?OFIrRCtGckxQWG45V3FLbThOQUsyZXBQbytCazQyeExtbHozMEVLVTNYRVBC?=
 =?utf-8?B?MzdiT3VnRjFZR2lsTzQ5TG5VRjRCcnZFUmlzUElQc0w4VzFTUldac2s5OE1V?=
 =?utf-8?B?UVhFZkRZYVhRTitEKzk3eHlwdFdKUGQrazhwTTB4T01YS284L3pFVUNZV254?=
 =?utf-8?B?YkxTa1JHNkN2cGlrMTl1UXJvbnl5S1ljNFVLTEJtK1I4dXhwVTRYZXJGekV4?=
 =?utf-8?B?TjE2RklVcll1bmg2Z2pOYmJHZThXd2pIRXhPcGZndWJraDFyaFJpWE1rSTRp?=
 =?utf-8?B?b3lTNGpmRjNMUDhnTzkzRlhLS3JnNTBML0YyTXdzUVhoWHQydkZFRURiVW1C?=
 =?utf-8?B?dkk1c3ZQb2wrMXMxV3RDMVRBZHZydUF0eEZuOVpDeWJtTStOYnFVOU43TEtW?=
 =?utf-8?B?QWMwOEx1RFJsb0VYSWgxdnFjYWthRTNTTkdMclpPUENyMWtTQWoySzNxYWhI?=
 =?utf-8?B?SlYxaldwNHZ5Uk9DaTY4QlpSNjIrM3hwbTBCVzEwcnB0d0tRenovbnFCb016?=
 =?utf-8?B?dVdXK1kwS0JGR2JxUGNmZ29DWlMrRm9JMmVzOU53by9YM29OaEF3TTNkbXpK?=
 =?utf-8?B?eGRXS0dvclNUejFJVDlybDdqdVBrSFFFTHpUZ0t2NWdYTkV2bGJLSVN0SWc4?=
 =?utf-8?B?bTdlVlhMWVZFZksrSjczZWhMcHVzS1NWTkJuTkJUc2pFdDZQeHoyZGtlNlRH?=
 =?utf-8?B?dWs3cjIvd1FhektqaHl4MVlJcmJ4TEplMnlwMUxPdndwYlg4MUJmY0xJYlBW?=
 =?utf-8?B?cmwrNlE4b21DQ2tIVDdob3RGMWdUTW5qUFQ3UmxSVERvbFhqZFMvMWhaMmZW?=
 =?utf-8?Q?gaohoOAv0+IVrDyWfQwuFsmMb?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	73gW4M1h4w87mO+//0E8IjGU709p3a137YZtnQ0UQrYsH9XxZE/j0rn21KFAKoRz93Gk8r40vIfRspNHFeiHriyZ18Fvb5RPzKkVpTmBhYpkfhJ+h5tfyvxtrjzPlHPZ0J2Lv3rIbiS/4ef0aPY5uwDSkM8MH/8/IOnWibACZVWeYUMzvaTF430jUejt2wqoCwL/VvBHwrFa0sXf+O1Z9BdCb4O9L1dlmpKWt56ciZJKAPgWchvfQIHT1L5oou3GYDRrKvS4FP/BkozU804E2PV6z55DcVABm2bqon+13ITkae+M6/yVUGvmOWKEAn5shO71zBp9aPScBaGV9ohPXHDT339BW658zeKxAbZj/aY6JEoa3Lniwbfuk9mkxHke9gS0HETML50S/7MXKuH7uuhJvPZl+RhCjWsHUwhvSZmhlUzSp6s0bC0UJCuzjXUdI8tHy+bqJVZnQADzmfHd4a9evSVof3v4JWeaenEcr0BfcOmzfToH1+U/EjW1M9DfhXrHJ+KQobf4gDlWYERs4EA9gAare29DmT9reB+mC/28qW30LU8wz2H9I7r29SJOqJgUb5HI/xFPA/7cBEPZl+wb7iYJ7hDb4tDQaPrshfjeedf+nhek5i8UJEn5mHYT4L4JORBGBgQdIqu2NqUtkZ26ztGacCxxSG43jKRtzXQQwVbIJ7pFZDRoa79sGydB8qZlV9yXNJ7o/KtlrOtgQM2NLJnXPQk06wnpuFeGKGP7aeiSlja761opyixjyEg9NO8FJU6GF2AVPRe9pyMOr9zOhGmY0ihWOYMkEDe+N20=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3a9e9424-e3ae-4a7a-a517-08dbf4fd04b8
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Dec 2023 19:12:54.7502
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ouo0ptVPIb4XJnrCqje5OL/kFnnPtce0i6O+ItMATrVqoDfeJU2kw3pTPz0OqKNdGjlcXDTCTrD2zGnbb/WeiA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR10MB6866
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-04_18,2023-12-04_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 mlxscore=0
 bulkscore=0 malwarescore=0 phishscore=0 suspectscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311060000
 definitions=main-2312040149
X-Proofpoint-ORIG-GUID: -NvV430LZVYbxZkRv_25hvC_TEbmurGy
X-Proofpoint-GUID: -NvV430LZVYbxZkRv_25hvC_TEbmurGy


On 12/4/23 8:15 AM, Chuck Lever wrote:
> On Mon, Dec 04, 2023 at 04:34:00PM +0100, Wolfgang Walter wrote:
>> Hello,
>>
>> after upgrading from stable 6.1.63 to stable 6.6.3 our nfs-server logged a
>> WARNING and then more and more clients hanged:
>>
>>
>> Dec 04 14:59:25 engel kernel: ------------[ cut here ]------------
>> Dec 04 14:59:25 engel kernel: WARNING: CPU: 17 PID: 8431 at
>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x174/0x190 [nfsd]
>> Dec 04 14:59:25 engel kernel: Modules linked in: cts rpcsec_gss_krb5 msr
>> 8021q garp stp mrp llc binfmt_misc intel_rapl_msr intel_rapl_common sb_edac
>> x86_pkg_temp_thermal intel_powerclamp coretemp kv>
>> Dec 04 14:59:25 engel kernel:  enclosure sd_mod usbhid t10_pi hid
>> crc64_rocksoft crc64 crc_t10dif crct10dif_generic ixgbe ahci xfrm_algo
>> xhci_pci libahci dca mdio_devres mpt3sas ehci_pci crct10dif_p>
>> Dec 04 14:59:25 engel kernel: CPU: 17 PID: 8431 Comm: nfsd Not tainted
>> 6.6.3-debian64.all+1.2 #1
>> Dec 04 14:59:25 engel kernel: Hardware name: Supermicro X10DRi/X10DRI-T,
>> BIOS 1.1a 10/16/2015
>> Dec 04 14:59:25 engel kernel: RIP: 0010:nfsd_break_deleg_cb+0x174/0x190
>> [nfsd]
>> Dec 04 14:59:25 engel kernel: Code: 02 8c a4 c2 e9 ff fe ff ff 48 89 df be
>> 01 00 00 00 e8 70 7c ed c2 48 8d bb 98 00 00 00 e8 b4 0e 01 00 84 c0 0f 85
>> 2e ff ff ff <0f> 0b e9 27 ff ff ff be 02 00 00 0>
>> Dec 04 14:59:25 engel kernel: RSP: 0018:ffffbd57227c7a98 EFLAGS: 00010246
>> Dec 04 14:59:25 engel kernel: RAX: 0000000000000000 RBX: ffff94a77356e200
>> RCX: 0000000000000000
>> Dec 04 14:59:25 engel kernel: RDX: ffff94a77356e2c8 RSI: ffff94b78cf58000
>> RDI: 0000000000002000
>> Dec 04 14:59:25 engel kernel: RBP: ffff94a0392b3a34 R08: ffffbd57227c7a80
>> R09: 0000000000000000
>> Dec 04 14:59:25 engel kernel: R10: ffff94a05f4a9440 R11: 0000000000000000
>> R12: ffff94b8e3995b00
>> Dec 04 14:59:25 engel kernel: R13: ffff94a0392b3a20 R14: ffff94b8e3995b00
>> R15: 000000010eb733cd
>> Dec 04 14:59:25 engel kernel: FS:  0000000000000000(0000)
>> GS:ffff94b71fcc0000(0000) knlGS:0000000000000000
>> Dec 04 14:59:25 engel kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
>> 0000000080050033
>> Dec 04 14:59:25 engel kernel: CR2: 00007f9ef8554000 CR3: 000000295e020003
>> CR4: 00000000001706e0
>> Dec 04 14:59:25 engel kernel: Call Trace:
>> Dec 04 14:59:25 engel kernel:  <TASK>
>> Dec 04 14:59:25 engel kernel:  ? nfsd_break_deleg_cb+0x174/0x190 [nfsd]
>> Dec 04 14:59:25 engel kernel:  ? __warn+0x81/0x130
>> Dec 04 14:59:25 engel kernel:  ? nfsd_break_deleg_cb+0x174/0x190 [nfsd]
>> Dec 04 14:59:25 engel kernel:  ? report_bug+0x171/0x1a0
>> Dec 04 14:59:25 engel kernel:  ? handle_bug+0x3c/0x80
>> Dec 04 14:59:25 engel kernel:  ? exc_invalid_op+0x17/0x70
>> Dec 04 14:59:25 engel kernel:  ? asm_exc_invalid_op+0x1a/0x20
>> Dec 04 14:59:25 engel kernel:  ? nfsd_break_deleg_cb+0x174/0x190 [nfsd]
>> Dec 04 14:59:25 engel kernel:  ? nfsd_break_deleg_cb+0x9a/0x190 [nfsd]
>> Dec 04 14:59:25 engel kernel:  __break_lease+0x25c/0x720
>> Dec 04 14:59:25 engel kernel:  __nfsd_open.isra.0+0xa9/0x1a0 [nfsd]
>> Dec 04 14:59:25 engel kernel:  nfsd_file_do_acquire+0x4ca/0xc50 [nfsd]
>> Dec 04 14:59:25 engel kernel:  nfs4_get_vfs_file+0x34c/0x3b0 [nfsd]
>> Dec 04 14:59:25 engel kernel:  nfsd4_process_open2+0x42c/0x15d0 [nfsd]
>> Dec 04 14:59:25 engel kernel:  ? nfsd_permission+0x63/0x100 [nfsd]
>> Dec 04 14:59:25 engel kernel:  ? fh_verify+0x42e/0x720 [nfsd]
>> Dec 04 14:59:25 engel kernel:  nfsd4_open+0x64a/0xc40 [nfsd]
>> Dec 04 14:59:25 engel kernel:  ? nfsd4_encode_operation+0xa7/0x2b0 [nfsd]
>> Dec 04 14:59:25 engel kernel:  nfsd4_proc_compound+0x351/0x670 [nfsd]
>> Dec 04 14:59:25 engel kernel:  ? __pfx_nfsd+0x10/0x10 [nfsd]
>> Dec 04 14:59:25 engel kernel:  nfsd_dispatch+0x7c/0x1b0 [nfsd]
>> Dec 04 14:59:25 engel kernel:  svc_process_common+0x431/0x6e0 [sunrpc]
>> Dec 04 14:59:25 engel kernel:  ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>> Dec 04 14:59:25 engel kernel:  ? __pfx_nfsd+0x10/0x10 [nfsd]
>> Dec 04 14:59:25 engel kernel:  svc_process+0x131/0x180 [sunrpc]
>> Dec 04 14:59:25 engel kernel:  nfsd+0x84/0xd0 [nfsd]
>> Dec 04 14:59:25 engel kernel:  kthread+0xe5/0x120
>> Dec 04 14:59:25 engel kernel:  ? __pfx_kthread+0x10/0x10
>> Dec 04 14:59:25 engel kernel:  ret_from_fork+0x31/0x50
>> Dec 04 14:59:25 engel kernel:  ? __pfx_kthread+0x10/0x10
>> Dec 04 14:59:25 engel kernel:  ret_from_fork_asm+0x1b/0x30
>> Dec 04 14:59:25 engel kernel:  </TASK>
>> Dec 04 14:59:25 engel kernel: ---[ end trace 0000000000000000 ]---
>>
>>
>> 6.1. did not show such a problem.
>>
>> Both are vanilla stable kernels (self-built).
> Thank you for your report.
>
> If you are able to bisect your server between v6.1 and v6.6, that
> would help us narrow down the cause.
>
> Dai, can you have a look at this?

The warning message indicates the callback work was not queued since
it was already queued. In this case we'll have taken an extra reference
to the stid that will never be put (see b95239ca4954a0), we should fix
this but I don't think this extra reference causing the client to hang.

It's hard to say what the root cause is without a core dump and/or some
network trace or a way to reproduce the problem. As Chuck mentioned, it's
best to bisect the server to help us narrow down the cause.

Wolfgang, could you provide some additional info such as how often this
problem happens, under load?, problem reproducible?, number of clients
involved, type of NFS activities, etc.

Thanks,
-Dai

>