Received-SPF: pass (google.com: domain of linux-nfs+bounces-694-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Mon, 18 Dec 2023 14:10:57 -0500
From: Chuck Lever <chuck.lever@oracle.com>
To: dai.ngo@oracle.com
Cc: jlayton@kernel.org, linux-nfs@vger.kernel.org, linux-nfs@stwm.de
Subject: Re: [PATCH 3/3] NFSD: Fix server reboot hang problem when callback
 workqueue is stuck
Message-ID: <ZYCZQYK9jM2dhiag@tissot.1015granger.net>
References: <ZXyvCnEuV9L18JSS@tissot.1015granger.net>
 <195ba461-0908-4690-85a9-a9d12ffbad90@oracle.com>
 <ZXzIGmhDZp7v87aZ@tissot.1015granger.net>
 <aef15e6d-20c2-461d-816b-9b8bc07a9387@oracle.com>
 <ZXz7nxzlPfJ+06QI@tissot.1015granger.net>
 <1a988fe4-a64c-48c2-9c2c-add414294e07@oracle.com>
 <ZX0gOlqGbIES5RtB@tissot.1015granger.net>
 <88802128-3ae8-4f91-aeeb-69693b137981@oracle.com>
 <ZYBtEs7JfMCi0TCl@tissot.1015granger.net>
 <11b384fc-413b-45e8-8592-1f736de6a3de@oracle.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <11b384fc-413b-45e8-8592-1f736de6a3de@oracle.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?maUBG6/hhgQok1EeYDzIZei6RkMl1I4Ms1mq/sjS6xagamOxZan7Uu942Ttf?=
 =?us-ascii?Q?mdYZxVnppbdeVXYlwBJGDuKamnHXVVPTFPRxHeQS0hMUVYGN9CkTtaW18JAf?=
 =?us-ascii?Q?85CjNUP4Y5+sO58hNAO8/dqv+jy8tUftVUBlWwyNKVem6JkvZdC7dvjvaLll?=
 =?us-ascii?Q?wVnq3e7lMl3w7LZZRk3osTSgSOrbBy4rMNr+YPM7c64ENAul9H7JQdyK60O5?=
 =?us-ascii?Q?Mu8EudxE+2MSKVh3a8/M1wu3pMdZOGw2cUTYhljJ+lUrvdg8vvfjng3IqDNd?=
 =?us-ascii?Q?mZsn5MqhgwEW0IELrCd47NGqnCTrc/zVy7T2oIY2F9yaFiAYovJ4SAD7e42b?=
 =?us-ascii?Q?OpvneH6OYRdd2D/C4p7nTnCiGoIE/I+xvZKKzfxg/wx19UmP876uVuW1Gjnj?=
 =?us-ascii?Q?ykqPz50VFICI6S8yAqLa5QMmgC7A4Itj4f5utcE2RJ/Hc87IEi9jJAb8au+d?=
 =?us-ascii?Q?NhBCBsU+DiuWQ6pzUm9whi2qwcOm0RHhJ3xqdYXqzJsHJ3hlNLZIevKuxNVI?=
 =?us-ascii?Q?xYBPgnIEhLK61YSK0H3Vesc8dmQ0/OWDIFLKVAYz1wkC3Hp5E2MCCWKdjpUr?=
 =?us-ascii?Q?klRJHYKoQ2TLRcAh6SE5PjVbBWQUn11W+F1qQ6ZQ2fXOL/wPOflBw/HgxsX0?=
 =?us-ascii?Q?lU/rbtymXzWYSVVRRJctik9dAYDsuO28rATqvhiYozaTmz06+ytJfW+LnCZN?=
 =?us-ascii?Q?lr7jYZQYUKUgHy1Ip7e7ehobOEn/w15zv4bJry8Opp7uh/ZFpxToSaajzs9H?=
 =?us-ascii?Q?SUmZdBxOhV9nnKC1V5QT+pw+EzYzINVrLMkonwZ/lIaYQp5DRQEp5NRlx3ci?=
 =?us-ascii?Q?OaNk/lo/dBDpQlQUkF51M/Zpu0MtekLuSu8oQc1P9PYw7NBUGbpTtPU8i57M?=
 =?us-ascii?Q?lFKZO2qILQre+JE7wrtFTkeDbiHFqS+WylrvxaKUbzu2rO38OMS6kZGXKKes?=
 =?us-ascii?Q?aDBk4APQl9obn2zdJGWmmMpoV7enpx8aDewbR03wjzYVT2GBeVuXfdoHc/mx?=
 =?us-ascii?Q?AsLSBjVCyjzIrv2zIX0Y8nOwW9SD6jtqxFX2yiww/cLlxpfuO3Apot2BqCHL?=
 =?us-ascii?Q?z1LktrT8QPCZ5a3acRUHlMdQ9UDdPYWegFg51ijq1NWAVHiLVKkChhYJrDcF?=
 =?us-ascii?Q?zoVDawGUBfxVVxicPObGVDdya480A+sjiXz1+LufkPXusa8wBwgeQS59UoyK?=
 =?us-ascii?Q?zh0hA2JsGIJIcD43V50uAbrxgjYcBIMFLKjZNzF9Hf6eT4obZvnRb+Rc1GRB?=
 =?us-ascii?Q?gHmB5CBOhGXzlp58hkHwPoVh6dri4iqrFkz92mAy7i0o6lkBDR1GfBJLgr7x?=
 =?us-ascii?Q?6XdTSUAV+Yhzrdg41p/4UDt8C+M5I0yHLbfQs9CXU40AEy0CrWsE/tq/Xz9N?=
 =?us-ascii?Q?CC7c2R8HYEvz5gIqwWtBS6Fy90ZPE01f+ez4vVIv6ZIk/Fb4mTDFXo2FpxU9?=
 =?us-ascii?Q?MgtCSsymU6WST4d3XiktFMhGcPAkWNeHnPEmKtQLkclBKMO1WS+nu+dygL2h?=
 =?us-ascii?Q?O4JdR7Wa9NRfwqkAIXAlWJ8H55pxEJapq1GHywZrbIMtZ7HM2IqVmC5toYW6?=
 =?us-ascii?Q?3eIxpF4LagLSRhcvTLCx2lQvV7fkQhuflwvdK/v0vuMyTuijcF9W23l8jAzE?=
 =?us-ascii?Q?lg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	vZei7OdnYprUz4pyhl4gKCdbduh9SvQT8EDDCFLMJMJzp6PRWzcKyi2pMMTaXU+X8uZFUYBMSngNXSdur2rQlX3agOb+IbQX/5Yu+FrtQ393Ajl42EVdOhorxCHNEHk3rvRSRB//rjTTh6BsSAPOqbGsZaySf/Iqg3bmsgwof0Q11ePikvimZ8kWjO8xf8qcMjdMFlbYf5noSNL+zTod3PWIbjit86r2TKJNOUdqzaQfeDWC2J6C5QSnOMcD3YhiblFJul+0pMTBjP1AuZ3YiE92OwLSUjTG1dtUd6cd8XQ5+LmWU23p1bo7+orqHaxHoqQeK1VB5vxEUwhw8BWl/ScHou4p1D0RS3i9+Fe9TyfxxKAIehduIcHCmdEi9tZAVhSibC0UWKYR8GjUfFFLhHmalyo2GLrEJU08Sb+o2cQ5as469qSZl2aJ+IUoLTncocGE6hqkV96dgEI6SwT40dLhB8SbU4ZkTr/0qChtbhYwkuvuY+l1XJszT6nyRBuHYjBRC1CArRCxcf+uAwfggpZQtTefm9JGNSwUlC5smSPA4JOj6iYtGtyQjnr8XdyKZjlyqEXq89Vw2OvZ+avqCXuAtG2SdGERi6RkZHPtTV8=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 374f5b8d-74fc-49d9-1227-08dbfffd11f0
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Dec 2023 19:10:59.7423
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DX63wYbOd+rElhQ/NBp7ml/bqaiVtlo/BqkoEZSeT+xu7xraE5Cehw/0BMBJu0BfefamHn+kaIk73eXJVy7Gew==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR10MB7577
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-18_12,2023-12-14_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 mlxscore=0
 adultscore=0 suspectscore=0 spamscore=0 malwarescore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2312180142
X-Proofpoint-GUID: wx64pCx-qYqdn6EPykBGa2NeIFyUf5lM
X-Proofpoint-ORIG-GUID: wx64pCx-qYqdn6EPykBGa2NeIFyUf5lM

On Mon, Dec 18, 2023 at 10:17:49AM -0800, dai.ngo@oracle.com wrote:
> 
> On 12/18/23 8:02 AM, Chuck Lever wrote:
> > On Sat, Dec 16, 2023 at 02:44:59PM -0800, dai.ngo@oracle.com wrote:
> > > On 12/15/23 7:57 PM, Chuck Lever wrote:
> > What we don't know is why the callback was lost.
> > 
> > - It could be that queue_work() returned false because of a bug.
> >    Note that there is a WARN_ON_ONCE() that fires in this case: if
> >    it fired several days before the hang, then we won't see any
> >    log messages for more recent misqueued work items.
> 
> The WARN_ON_ONCE came from nfsd_break_one_deleg which is a delegation
> recall and not from nfs4_cb_getattr. I suspect this is because of a
> possible bug in __break_lease as question for Jeff above.

OK, so there's no indication at all if nfsd4_run_cb() fails when
NFSD queues CB_GETATTR? No wonder it's a silent failure.


> > - It could be that nfsd4_run_cb_work() marked the backchannel down
> >    but somehow did not wake up any in-flight callback requests.
> > 
> > Let's get more details about what's going on.
> > 
> > 
> > > > I can add patches to nfsd-fixes to revert CB_GETATTR and let that
> > > > sit for a few days while we decide how to move forward.
> > > The simplest solution for this particular problem is to use wait with
> > > timeout.
> > The hard hang was due to an uninterruptible wait, which has now been
> > reverted.
> > 
> > Going forward, if there's no wait, there can be no timeout. The
> > only approach is to handle errors properly when dispatching a
> > callback.
> 
> not even wait for 30ms for well behave client, same as nfsd_wait_for_delegreturn?

30 milliseconds is acceptable. It's very brief and can never result
in a shutdown hang. I just don't want a long timeout.


-- 
Chuck Lever