Received-SPF: pass (google.com: domain of linux-nfs+bounces-667-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Fri, 15 Dec 2023 22:57:46 -0500
From: Chuck Lever <chuck.lever@oracle.com>
To: dai.ngo@oracle.com
Cc: jlayton@kernel.org, linux-nfs@vger.kernel.org, linux-nfs@stwm.de
Subject: Re: [PATCH 3/3] NFSD: Fix server reboot hang problem when callback
 workqueue is stuck
Message-ID: <ZX0gOlqGbIES5RtB@tissot.1015granger.net>
References: <1702667703-17978-1-git-send-email-dai.ngo@oracle.com>
 <1702667703-17978-4-git-send-email-dai.ngo@oracle.com>
 <ZXyvCnEuV9L18JSS@tissot.1015granger.net>
 <195ba461-0908-4690-85a9-a9d12ffbad90@oracle.com>
 <ZXzIGmhDZp7v87aZ@tissot.1015granger.net>
 <aef15e6d-20c2-461d-816b-9b8bc07a9387@oracle.com>
 <ZXz7nxzlPfJ+06QI@tissot.1015granger.net>
 <1a988fe4-a64c-48c2-9c2c-add414294e07@oracle.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1a988fe4-a64c-48c2-9c2c-add414294e07@oracle.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?uxtn4PRFPPN16itNKVcurUEyiS0ZY/d7b+XWKTqc71BwagJXF9/BK3jmRzny?=
 =?us-ascii?Q?vPhqgqX+2PpVjbLNQvK5Lo+U0X3W21Zxe7Vt3nWfeueM3dlwhGwHb5LdkRu2?=
 =?us-ascii?Q?0vxoujiyIDPIb3hREOORSgbhalXKz6hWZJgWMGQeLA69BZs3S7dlqAbtEe/P?=
 =?us-ascii?Q?3yPWicg11UediPZO4o4VGV+WHA9IiSEfvE7aFJqyzlHOhEgZ9tLt+g1cZwMz?=
 =?us-ascii?Q?DJ1gFeIovB9yP72FkTs4+RspqAa6ehlSpKhxzBHcPsLOc9aCj2lbZDrT+u0y?=
 =?us-ascii?Q?N6fZ3sRUjfpcOHS6N91kRHlfMM+jUVk0A3ysSFZyJEpua9jpPXqiXQ2FTvWc?=
 =?us-ascii?Q?VYzr8Wd8mQT4tyXQcQrg3STmSdGZfu9JW5gSJiIWZKKQsLIqIKEsQ7yFghll?=
 =?us-ascii?Q?X/oyRvhl8+e3Qk/2fL9sWwMLPnYGNId2aEG5xneFRqWSSZ3AKUjTdgB/AQiQ?=
 =?us-ascii?Q?6z+stK6SwsAcccPXBc7dFlLH/FKO9cZtOa6ndZsSP7v5sUKSGoc3U1RsY56x?=
 =?us-ascii?Q?gx6ubwjY+RV03ddLOfOrvWGxoFfEWom5jS/NpZ7owDimTS6aqpiLeyY8FWPG?=
 =?us-ascii?Q?MlRNk+DEXIM8W+GOnw87+H0Fmq9KoLxLFFqQ+frIunh/9I7NVAeDRGZ8H2yT?=
 =?us-ascii?Q?FWpoknTgY265QoI9WnAO5mNB0PrcxlTocFmGe8tpvh11rZ/oZmcHPStVwN8p?=
 =?us-ascii?Q?ALcP6rtMVTrLHqmz3+LBBhIzpsIAB4xzajZc2XOJDMWqbNliu0Cimu4W5mFQ?=
 =?us-ascii?Q?zlWVRCP3TcEUHM95yaEwu4bOmuoAi0GgY4O3x+LsEOFeXOpO0PXamqAr5f+T?=
 =?us-ascii?Q?OxrFpMhMGI8pbHYbtmzVf+cH3vOim+mJEQGyprJmm7ddQpj/wd1uu8Ad8Q9w?=
 =?us-ascii?Q?BkuUJa3b3ZLpXUYyxxODC+3jI6Jl5AzlgLA//h7bTJVmz1uFS4eP6Enk49s6?=
 =?us-ascii?Q?y+M4qzeSHlOG55I8GRL6poFRllbbPLWB+FuVM0ne/seIVmjxhZWci6MKiIwP?=
 =?us-ascii?Q?9YsIqCT457AAB++pkRKwUHXmFxoaP5xP2cPkcGh+XpfdYcnerVvgN933qlJb?=
 =?us-ascii?Q?vefoDZqpaViLuvlAbjfof/7ylN9wv1ZPHoAH4cRjcCHcEjoxpyUU3pyM/k2C?=
 =?us-ascii?Q?uGf/y3p+6IEUEGe3AQguNR9YZG7gM+NwbZeu4irR7dKLb8HvVRxWnMV5dBpB?=
 =?us-ascii?Q?EiBi/8bVF6uDsARwSV1vrp+aTCtejmpGm+pzQuYOqyY9S8DR3G1/9EMwCWaO?=
 =?us-ascii?Q?EJomw99HuQwQ2YSDl/RPyQpPLK1rLl8w6Tbpa1iVpvRqJ6Bv2bQJh0rJfTlH?=
 =?us-ascii?Q?kBKiWdXZ7JTMYMR3x5C3J4Sc3Jfq5hXugaXae+d3cnLdV6HrY3mo6ThHKZmo?=
 =?us-ascii?Q?Dj/xAN0xMyJS6++HE/qJfjkvtS5dHnAJpngOdOCNXBrDVA5rkDExMfUt3FNT?=
 =?us-ascii?Q?vCb66XcUfD0086+GSr2BmHqxApjgG/5fpEYFzxqJduvgS4iwlBX2K7PA2oSm?=
 =?us-ascii?Q?cVqjbDQAVBsBPzD9R0POkGTrXQOwdXgzh5cGFCmIiyKPzgV142G7I6GLnde4?=
 =?us-ascii?Q?GVrqouRNsoQPaYkccNfyFRCWhtKRAIcj/KhSt4vKXKoK7VTWOKo196mxA7Dr?=
 =?us-ascii?Q?/g=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	avzU0xOSyjdLqH++XjuLNCiPG8OCoNIY6iJVKe39gM+2MXVuu0yL2Y6ZRRQp/E86eet4m/3yXIl+WVVIqqzLjmRP6NyWsb7vJ3EQpWg4k1GRKLAOlQ+EO9Ppv/SUVbRcoyf0vCozmX+Tm+7CAZE8wHBe+TIc9wnHOUkvJtunzCw6qinJ3hr3KNOJvFzXOsAks6J92Mt0PLp81tYWS4IgcVuQrtudLVrZRv2ntp01IVT/u2tWE+tEfZA5pMT4LrnVazeucOrfbxI3tfm3RkPi+ViMqr4j20/RreqIg3PH4pBRTq6V7NPoepkd2VxGkGG2j9+A+5zm5+xyeh5BnJ9u8D4y7o4UdoLGPEHMLLUZrwCJC2+23G6mUguRGmEzxaEX4ZeJP8BeonnM1hPWXBHDjaSQhZx2hknZXqZVLUN6otS3kiuYnNsnpnTtCJs9yRyD3zdY8LJVeBjve+FAY6lz1pI0ZXBzJ/hOeEK5C6ZUykz8Qiwddvrcr1FAqLpuJ9IMYpPKmAVJCDYfSlrzwLqCLhXoe1BbPRAsPer2ujkkKx1virAgOEjhqHuBUHmxqPBRlrmK+am1x9wWNP0ZNg4XTvRQ+Q5kyvfenCfpERLTVTQ=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 03c19ca2-47b1-494d-d869-08dbfdeb2b1e
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Dec 2023 03:57:48.6895
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: siQdefLUG51NdMlTiwPx3CzBBAzDJbwi+VB421J0g0e7Bf8QIC72KmXLjIwBGUfBW6pRbNEvXI/ATT83VDrefQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5389
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-16_03,2023-12-14_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 malwarescore=0 phishscore=0 mlxscore=0 bulkscore=0 spamscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2312160027
X-Proofpoint-ORIG-GUID: AYDExBKJ6AJK3jDtpOXLRYPqJo7x3nM4
X-Proofpoint-GUID: AYDExBKJ6AJK3jDtpOXLRYPqJo7x3nM4

On Fri, Dec 15, 2023 at 07:18:29PM -0800, dai.ngo@oracle.com wrote:
> 
> On 12/15/23 5:21 PM, Chuck Lever wrote:
> > On Fri, Dec 15, 2023 at 01:55:20PM -0800, dai.ngo@oracle.com wrote:
> > > Sorry Chuck, I didn't see this before sending v2.
> > > 
> > > On 12/15/23 1:41 PM, Chuck Lever wrote:
> > > > On Fri, Dec 15, 2023 at 12:40:07PM -0800, dai.ngo@oracle.com wrote:
> > > > > On 12/15/23 11:54 AM, Chuck Lever wrote:
> > > > > > On Fri, Dec 15, 2023 at 11:15:03AM -0800, Dai Ngo wrote:
> > > > > > > @@ -8558,7 +8561,8 @@ nfsd4_deleg_getattr_conflict(struct svc_rqst *rqstp, struct inode *inode,
> > > > > > >     			nfs4_cb_getattr(&dp->dl_cb_fattr);
> > > > > > >     			spin_unlock(&ctx->flc_lock);
> > > > > > > -			wait_on_bit(&ncf->ncf_cb_flags, CB_GETATTR_BUSY, TASK_INTERRUPTIBLE);
> > > > > > > +			wait_on_bit_timeout(&ncf->ncf_cb_flags, CB_GETATTR_BUSY,
> > > > > > > +				TASK_INTERRUPTIBLE, NFSD_CB_GETATTR_TIMEOUT);
> > > > > > I'm still thinking the timeout here should be the same (or slightly
> > > > > > longer than) the RPC retransmit timeout, rather than adding a new
> > > > > > NFSD_CB_GETATTR_TIMEOUT macro.
> > > > > The NFSD_CB_GETATTR_TIMEOUT is used only when we can not submit a
> > > > > work item to the workqueue so RPC is not involved here.
> > > > In the "RPC was sent successfully" case, there is an implicit
> > > > assumption here that wait_on_bit_timeout() won't time out before the
> > > > actual RPC CB_GETATTR timeout.
> > > > 
> > > > You've chosen timeout values that happen to work, but there's
> > > > nothing in this patch that ties the two timeout values together or
> > > > in any other way documents this implicit assumption.
> > > The timeout value was chosen to be greater then RPC callback receive
> > > timeout. I can add this to the commit message.
> > nfsd needs to handle this case properly. A commit message will not
> > be sufficient.
> > 
> > The rpc_timeout setting that is used for callbacks is not always
> > 9 seconds. It is adjusted based on the NFSv4 lease expiry up to a
> > maximum of 360 seconds, if I'm reading the code correctly (see
> > max_cb_time).
> > 
> > It would be simple enough for a server admin to set a long lease
> > expiry while individual CB_GETATTRs are still terminating with
> > EIO after just 20 seconds because of this new timeout.
> 
> To handle case where server admin sets longer lease, we can set
> callback timeout to be (nfsd4_lease)/10 + 5) and add a comment
> in the code to indicate the dependency to max_cb_time.

Which was my initial suggestion, but now I think the only proper fix
is to replace the wait_on_bit() entirely.


> > Actually, a bit wait in an nfsd thread should be a no-no. Even a
> > wait of tens of milliseconds is bad. Enough nfsd threads go into a
> > wait like this and that's a denial-of-service. That's why NFSv4
> > callbacks are handled on a work queue and not in an nfsd thread.
> 
> That sounds reasonable. However I see in the current code there
> are multiple places the nfsd thread sleeps/waits for certain events:
> nfsd_file_do_acquire, nfsd41_cb_get_slot, nfsd4_cld_tracking_init,
> wait_for_concurrent_writes, etc.

Yep, each of those needs to be replaced if there is a danger of the
wait becoming indefinite. We found another one recently with the
pNFS block layout type waiting for a lease breaker. So an nfsd
thread does wait on occasion, but it's almost never the right thing
to do.

Let's not add another one, especially one that can be manipulated by
(bad) client behavior.


> > Is there some way the operation that triggered the CB_GETATTR can be
> > deferred properly and picked up by another nfsd thread when the
> > CB_GETATTR completes?
> 
> We can send the CB_GETATTR as an async RPC and return NFS4ERR_DELAY
> to the conflict client. When the reply of the CB_GETATTR arrives,
> nfsd4_cb_getattr_done can mark the delegation to indicate the
> corresponding file's attributes were updated so when the conflict
> client retries the GETATTR we can return the updated attributes.
> 
> We still have to have a way to detect that the client never, or
> take too long, to reply to the CB_GETATTR so that we can break
> the lease.

As long as the RPC is SOFT, the RPC client is supposed to guarantee
that the upper layer gets a completion of some kind. If it's HARD
then it should fully interruptible by a signal or shutdown.

Either way, if NFSD can manage to reliably detect when the CB RPC
has not even been scheduled, then that gives us a fully dependable
guarantee.


> Also, the intention of the wait_on_bit is to avoid sending the
> conflict client the NFS4ERR_DELAY if everything works properly
> which is the normal case.
> 
> So I think this can be done but it would make the code a bit
> complicate and we loose the optimization of avoiding the
> NFS4ERR_DELAY.

I'm not excited about handling this via DELAY either. There's a
good chance there is another way to deal with this sanely.

I'm inclined to revert CB_GETATTR support until it is demonstrated
to be working reliably. The current mechanism has already shown to
be prone to a hard hang that blocks server shutdown, and it's not
even in the wild yet.

I can add patches to nfsd-fixes to revert CB_GETATTR and let that
sit for a few days while we decide how to move forward.


-- 
Chuck Lever