Received-SPF: pass (google.com: domain of linux-nfs+bounces-695-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <f68e2a10-4bb0-4b46-966d-d7806a10faea@oracle.com>
Date: Mon, 18 Dec 2023 12:27:30 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 3/3] NFSD: Fix server reboot hang problem when callback
 workqueue is stuck
Content-Language: en-US
To: Chuck Lever <chuck.lever@oracle.com>
Cc: jlayton@kernel.org, linux-nfs@vger.kernel.org, linux-nfs@stwm.de
References: <ZXyvCnEuV9L18JSS@tissot.1015granger.net>
 <195ba461-0908-4690-85a9-a9d12ffbad90@oracle.com>
 <ZXzIGmhDZp7v87aZ@tissot.1015granger.net>
 <aef15e6d-20c2-461d-816b-9b8bc07a9387@oracle.com>
 <ZXz7nxzlPfJ+06QI@tissot.1015granger.net>
 <1a988fe4-a64c-48c2-9c2c-add414294e07@oracle.com>
 <ZX0gOlqGbIES5RtB@tissot.1015granger.net>
 <88802128-3ae8-4f91-aeeb-69693b137981@oracle.com>
 <ZYBtEs7JfMCi0TCl@tissot.1015granger.net>
 <11b384fc-413b-45e8-8592-1f736de6a3de@oracle.com>
 <ZYCZQYK9jM2dhiag@tissot.1015granger.net>
From: dai.ngo@oracle.com
In-Reply-To: <ZYCZQYK9jM2dhiag@tissot.1015granger.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?V2NHTTdsektEYzg0VVNkUlVXaCtsOXIyY242a08wSzJVaFprTzhFYU1pcld1?=
 =?utf-8?B?QUVJS1hodjJGMDViVW9iUTRnY1pSR0VWeUQvSGl5SGh1OXpVSmY0YjFUT0RW?=
 =?utf-8?B?dGZxeFFURUNnb0hVMlRSN0xSTVZqTlJaRkFjVkdpRlZmc3NmRmZtcDBSSDlX?=
 =?utf-8?B?aW9xN1FVZ0VQbzdCR1VpdElScmxCZDUwWDR1bDJLQW1nL3FtZnJLcWlFY1Nh?=
 =?utf-8?B?Z1BWWFp0UERJQ3cySFRGV1pzZnVyeWxycUREa0NtSUtuaURoVTEvTG1ETHdZ?=
 =?utf-8?B?NUEvU2o3Mjlud2NZVGgyUmhpdEd0WndXNFZ1NXBBVlFKYnFTMm9PcFYxWFJu?=
 =?utf-8?B?WE5HdVBUaVg5Z3hMVkk1SFdZaDZRRDJISXlmZlNQOFBISGVlUU1QZ0ZTVVJ6?=
 =?utf-8?B?YkFGbFVKZDNyS0RXQ1VVVjRxdWl2ME83WXNBVXZ4TldRUXhyM1NiR1YrbUl3?=
 =?utf-8?B?YXNFTmMrMm1qaU1sTUw5aWMrRDZ1VDFmLzhVdWpKN3hJeFRXeFNKSVpWTzRI?=
 =?utf-8?B?dmJGS2F5UGtkQ2RMR3NWdThzMmxIUGhnYU5qYnBBaEpLalZITEJtMXNzN0tE?=
 =?utf-8?B?V2IyMks1Vm5kYkowT0l3cFFQVnliYWhYNmxMTnh5UVkzM2FlQWlIKzREOFlS?=
 =?utf-8?B?SHZ2ZzR6MkxtSVAvM1pDTUZFVU1odDB4bmRscjBHMmx5YjAvMXZ1My9MVGFI?=
 =?utf-8?B?RTFJblEvK1M2UDREM1c1aTBWRC9BZ242eGNTMlVUaU9VcmJFOGtlbjE1a1Rn?=
 =?utf-8?B?Uys1TDE2NFUyRysvWU9uZEU3c21oYk1nT2pNazZKVk0zZFpGNXowS0tSWi96?=
 =?utf-8?B?aDRmVFVIQ3g1Mi9CMW9HMWZ0V1VDa3p5c0lLemJ1MWc3UGVjYVdhQ2wyQzBH?=
 =?utf-8?B?SHkxL0ZkaFJwZDg0cHNqSTJMZmdyV2ZPUUFERTJoTFBFM2I4em5ZRGNwbkVm?=
 =?utf-8?B?Yy9oZlNTak1IQVo3bm9RcHBKZEE3aUJYVUdXUGFRTTZVSDVQVFQ4STJKY2Rq?=
 =?utf-8?B?bzVzSjVta0hYMTJxZWZNUE5kR1QrUi80UU9TdjI0SVhjeXdvOE4yWklRQi9V?=
 =?utf-8?B?U0JhMi9UYTVkaTErdEVkM0JhSzNqWmRYcVNhZnpMd0s5Vks4Y2VpcERUak1n?=
 =?utf-8?B?UGM4alluYk1iTzZIYUFjYkJubnVhT3ZKS2pmUWlPZkM1QWdUZGdQdWdXcWhr?=
 =?utf-8?B?aEljcENGU1pncVo0bFFIQVc3REs5RXFOYS81bFBuOG5DRXFtd3haRjk1T0hD?=
 =?utf-8?B?eEdHVnRibjR1WUhkYjRoUlVHSVhxdGZDdEp2ZkROME9nNlJYYmZSSUpwWUlp?=
 =?utf-8?B?NHpqcUxTUjMxSGRqZTVaR3kxeEdmR0xkUW1yL3F3SUozZ2Z0dlRCbzlVbm9C?=
 =?utf-8?B?ck1XYlpyODVib1lqVkNwUmsyT1BsSlExTElnaFJKV2diUldpQkRQb2h0MzVF?=
 =?utf-8?B?SHNzZlg3RnZaYWxDdlpFOGxwU0lLLzllbzRtR3F4eEk5Y0JlNDV2Y1JRVWl5?=
 =?utf-8?B?ZkFmNHhBS25hUDVkZHYrNTdRVU1QYjBpVERZSkhFbVc2cVlvUGMzUjc1ZHEw?=
 =?utf-8?B?VjB2TE1oYWY0TzFwRE1aZjI3TnpkdzhOZXJXNkp4L2pNZ2EvUkhGMmtlOHU1?=
 =?utf-8?B?VVEyd1dWSDVzTXFzVzJnVnR1R2JFRFNjTG9oOE5YQ042TjJlNnB5cDN4cmpJ?=
 =?utf-8?B?aUNwd3BiN3diK0NqdEU0MWIyTDdrNmlhdEltaU44ZjNKdlFKdmdpOUdDM2FW?=
 =?utf-8?B?Q1dYQUg5QzVDakRUdXI2Z01lWnhhZ3RzK2xIR2ZIemtmNldrSDdWV1QzNnVQ?=
 =?utf-8?B?UElzYkNETmE0amNpTjZQZHo0ZE0yV3l6NzVNSDZVVGdycTVPbjhyT0NHWUNH?=
 =?utf-8?B?UmZIbTB1ckxVN1h6VHBnZTk0ckFaUm1ZU003RTB3VTYxb2txTGRrVG43ellE?=
 =?utf-8?B?UUxwZEN5RUl5eVFseS9EV083ZXRCZ2tUSmJaVVU4cDRydXlrTGVRMzNpMXlU?=
 =?utf-8?B?elYycDUrbWl2dXRNOVhPdW1jSnF6SU5ydmdtdVI5bHJidzE2c1I5ejllN1lo?=
 =?utf-8?B?NXBLOUI1ZHhIUGZlaGRYbExjbWJXV0VGek5Xc1VTWVF0VkFBOTFIQUE4VW8v?=
 =?utf-8?Q?xmFDY31hYiowm4xLgCFaPrMaH?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	RBj4beLwEeoyjOOsmug7aqGTIUV2h2er8MEAChsAHEzfK/55mavzhCrE/dfSF5oy0thXQAMgrO/vJvtLN75Uk4stDbaCYWlN5Tu+k7OL78zTl/PhZL89BOI8Lyj6iL8jc2XihePOzl/kVSPqDKo44a2/cmbn70h6u7vqLNMPfhGuLyf5wgB/4Ka9IA4NK/hkKwm+lXXJ6ZCzNlkglDrieNQqwuMaPow+XgVLc6AJpVZt+pSsXJ921G8OJ6ON+T+NWUfuCvDszmYMf4rTggo5TTfRcZjR88JWqge50wSYW9jyQssMScko3xYzBS1u9qD+nkMSr1dn0Ok6eqFx7C0kpiLD8G9ZjH2nM8RTeQ7Z8xVvR+E8kM+3gXS6Nni0FswNUQq9Rwpez9WBgHTVFT6SMBcNVJ8efVTtdDJPOxXhvHR0yQ2XqCBqt2CT2PaJFPs6983phhlHxZAZHZuVRDC3XakQ3o5bk3jXeLzVwjqPt/u8DLDwou8G+F0idrLmguUM5u5wRMxPAX5vEsZD6inNyFFKSvFZpap33DzvJ9FQduyGcxK1qhUnokzP6mX+gP14CAw9UbRwpEYYN+yoYAVUTjcaOy5vSIaw5v3ISRqCXbo=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 976ace8a-2076-4821-efb0-08dc0007c464
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Dec 2023 20:27:34.1527
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Qa7Q+udSY9zHmz6VykVmVGPVHqu5CxYmq22PwEezrVTuqn0IWxw3gZnbeqcptM4ULV00cJSJiMWzFx8vSjO6nA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR10MB5914
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-18_13,2023-12-14_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 malwarescore=0 phishscore=0 mlxscore=0 bulkscore=0 spamscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2312180152
X-Proofpoint-ORIG-GUID: x-mqpCmmgFSJZLpn-8MNo_hHFRe8y7tx
X-Proofpoint-GUID: x-mqpCmmgFSJZLpn-8MNo_hHFRe8y7tx


On 12/18/23 11:10 AM, Chuck Lever wrote:
> On Mon, Dec 18, 2023 at 10:17:49AM -0800, dai.ngo@oracle.com wrote:
>> On 12/18/23 8:02 AM, Chuck Lever wrote:
>>> On Sat, Dec 16, 2023 at 02:44:59PM -0800, dai.ngo@oracle.com wrote:
>>>> On 12/15/23 7:57 PM, Chuck Lever wrote:
>>> What we don't know is why the callback was lost.
>>>
>>> - It could be that queue_work() returned false because of a bug.
>>>     Note that there is a WARN_ON_ONCE() that fires in this case: if
>>>     it fired several days before the hang, then we won't see any
>>>     log messages for more recent misqueued work items.
>> The WARN_ON_ONCE came from nfsd_break_one_deleg which is a delegation
>> recall and not from nfs4_cb_getattr. I suspect this is because of a
>> possible bug in __break_lease as question for Jeff above.
> OK, so there's no indication at all if nfsd4_run_cb() fails when
> NFSD queues CB_GETATTR? No wonder it's a silent failure.

This patch adds a WARN_ON_ONCE just in case, but I don't this condition
will ever happen since we already had the test_and_set_bit on CB_GETATTR_BUSY
bit so the same CB_GETATTR will not be submitted to workqueue more than
once.

>
>
>>> - It could be that nfsd4_run_cb_work() marked the backchannel down
>>>     but somehow did not wake up any in-flight callback requests.
>>>
>>> Let's get more details about what's going on.
>>>
>>>
>>>>> I can add patches to nfsd-fixes to revert CB_GETATTR and let that
>>>>> sit for a few days while we decide how to move forward.
>>>> The simplest solution for this particular problem is to use wait with
>>>> timeout.
>>> The hard hang was due to an uninterruptible wait, which has now been
>>> reverted.
>>>
>>> Going forward, if there's no wait, there can be no timeout. The
>>> only approach is to handle errors properly when dispatching a
>>> callback.
>> not even wait for 30ms for well behave client, same as nfsd_wait_for_delegreturn?
> 30 milliseconds is acceptable. It's very brief and can never result
> in a shutdown hang. I just don't want a long timeout.

Thanks! I will submit v3 patch with timeout of 30 milliseconds.

-Dai