Received-SPF: pass (google.com: domain of linux-nfs+bounces-2511-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Thu, 28 Mar 2024 10:08:15 -0400
From: Chuck Lever <chuck.lever@oracle.com>
To: Dai Ngo <dai.ngo@oracle.com>
Cc: jlayton@kernel.org, linux-nfs@vger.kernel.org
Subject: Re: [PATCH 1/1] NFSD: cancel CB_RECALL_ANY call when nfs4_client is
 about to be destroyed
Message-ID: <ZgV5zwR0q/vjBAtI@tissot.1015granger.net>
References: <1711476809-26248-1-git-send-email-dai.ngo@oracle.com>
 <ZgMToHNkkGEHNb/y@tissot.1015granger.net>
 <69914825-e9d5-4859-a5a8-60d17e8e8bf6@oracle.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <69914825-e9d5-4859-a5a8-60d17e8e8bf6@oracle.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?TChtCRgK5VQb0NjvcYbaBLs8oxMDWvn6e91tH0Hp5NwkCq7ZuFDkr8Ta36Ac?=
 =?us-ascii?Q?50MHiN3fjekDxAvAnY8i3k2plFfzTEv9qdsfeAoFNAme0ZvlIJPMjOgdMP7p?=
 =?us-ascii?Q?RWPOL6NFhIHOAga86g1wYH3bHMwVRyUFp5t2yTkhR5ck4bx8BKlfW0XjNK73?=
 =?us-ascii?Q?OwEtY6FmKm+3T+PziNaMzEtXMZi2YVKF+HdhxtzO8htUhu5tXAivEGOqj2dr?=
 =?us-ascii?Q?nhDzcIMbdNSoB8nRLNBpHXzgsZ+a3flPkMzTP8eA3HNAagX5nW8FcGgHF1K3?=
 =?us-ascii?Q?N4iLDUJGNlp2/nH2qTq98VKdsRQcXfoQ6WTGtmZMPdj1IfgebFn7duL+dYgJ?=
 =?us-ascii?Q?ej5E9eq9Omomzyed6oDs95M+pabYODiwB2C1KsK6JJAak3i0KLdS3XF7qA7a?=
 =?us-ascii?Q?WwuBWNUZC1ZfVo4CgBtiGmWyfAR5eJ4Rqjm4nRCJFf5RcjVWHUC+kq1l9mnl?=
 =?us-ascii?Q?ZIs+xgLu16EymCdr8eT97M4ji2oZmjw9cs+lWji3mRHCkI5KyK5RHuXPhO0t?=
 =?us-ascii?Q?TistJKojfDpM1wfoVO09DqP+BcPPilbCLooPrQoz3JVNkm5iZNt7xnZiiNJj?=
 =?us-ascii?Q?4sgSGQ8T05Ut727/D+gI8qujZqXgsSGLMPLq6IzdD5dmVCTDiDx1hw7EXHpC?=
 =?us-ascii?Q?f3EYyST6K+Q8cj0YUtCSiS/WQgvxVgomYSODqOLmKH/jlKidf7ub7MtEhW0r?=
 =?us-ascii?Q?TInZZ7OyhrvC6AgPn9pZ8dcSevPw7luOtRXKj5kaTfu7XHvAj9gusR55jFJl?=
 =?us-ascii?Q?XqVsNy81xU+5mPb4IK4RHFGe2clhva8xh5MZM3hLDB38/19Eb5TNNadyDQxp?=
 =?us-ascii?Q?wkukPBs0FJeb1+PQMzYaLKcBMyaAkLYFGg2EsvTC4oKUpdLh7W+K8gmBwmNc?=
 =?us-ascii?Q?lAt6dcwyvxsutgw/A81jJp7HUo1T05vQWU+LbBaeNRoq7SlB9Ici+DWOOBWu?=
 =?us-ascii?Q?oE/VxzJbu/4Ddjnnk3ufxQG3jMYmjut1m9bMD8C2JyKVUmenwQpihT1pXgyE?=
 =?us-ascii?Q?ZXJw95HYxEYj9Oo2jmO+d7TZTx9FlWLofyK3OZtso4EfPbkYn0pQ5hBoAXct?=
 =?us-ascii?Q?D0wAN0iNFVm4L4510qLP3Q6zLuUctlCQIrhRtuc/uHytMo94El7xXUHIz10s?=
 =?us-ascii?Q?TmQMu66F8UpmBKHjg1K57Uj/SIokJmRD2Gjj8RDOUDh1ab1USpLdrUqnPevn?=
 =?us-ascii?Q?rONGt4Ef6TFO/JIuHugYXVASUC+G/UrQEizmjcKo7gx3WnjTGcWHxvh7y397?=
 =?us-ascii?Q?tKJ9lfkd409cfAIQUSGTPx6o9zbvsTC3r8CLl0MH29WWv/+2hnkj7p08YMQg?=
 =?us-ascii?Q?nTKNFqrnJxPH28O0OQGWVuKTzAsjDjmWsqe/Ix6ZXi0RpGoz48s+f6ldDLku?=
 =?us-ascii?Q?r1f30ScqAErJ58r3cAC5Qq6Biu38i+to5pDFtW/fFml4Qf+LmiA/gKzJr+HE?=
 =?us-ascii?Q?A6/nHAsMgeHGTF2veHUWNCf+DIRAlqLaNSN3/odZzrpnuRuZ3rOdxnGnUKkc?=
 =?us-ascii?Q?2gtzT0qAJ6wtJxp91u6BJOlmzJHQvy1itKntRiAj2wkP0VMwWLPsN4A7mkxA?=
 =?us-ascii?Q?l410JNFqzRp8Kn0soDKYoW7CWAJSs8ndKDGYaVW+T8eOnfCTbuK67FZloLbT?=
 =?us-ascii?Q?8Q=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	cfji9tN8sg8Fa8e7YwspDvVOKpS4EWCsCcMRRCkxPvoJo6Srpkr1m7KsHrxAfzstE6hcPpMjicEIlKE6kCjxJzJ/M3rJAlkxEn4OZl8wn5wa7z8V6Umwx0nmhTpJnjK1GeEeg51DVHI94G2tZhPx5qE3U6mc+Pgnc4eELQY1jcDfpKMsR3OAJaYXJl065JwamPW50BxGwykvLbtgzRccwI3SC2ZzKmkYbYxxv/EZY0e8r55SAe+DIyiWlw7usStmMYUNGJRYFmVBuTaVIUFeRvsR9ywhhdGCCDuvsqNeewgvEg8HS8E2Ova8QRyn9VYL0j77Mq0zJE4p1sL2fyFSVO5hBl7lKI1c4dDy86rTNLY7Pqi1/uCOh8zZapVR/eCauNp47RrJrIn5nwf2z9x4CprisE1okmXiRA03KggUHYYcRl59iApxUs7RX8OrV5WUNg8d5itAQ56WF1Vx3I39FYNbOm19uQyyA1sK+fAbVX9pSMUvEgTfs1rYb8QYEuoH6YZx5WX6q9T8WR/OABZnsgBLbrFSHU1eubr3SoSfl+lYdNhRRCe4qmsDYWtzglqWtzjj0mqErjxC0/EzopDZPiQjDiyL3dIpRUnlKgtYoB0=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 663d7af6-cd37-49f2-69cd-08dc4f3084d1
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Mar 2024 14:08:18.7485
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: k/A1bsauNUKxFYjt83aoO5b2dGb3vZ10X1lj/Xrm3JyljXpC16feggBHCNlJRiIMCL7XXw9wjriTpN+Aq4qmcw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB7378
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-03-28_13,2024-03-27_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0
 suspectscore=0 mlxlogscore=999 mlxscore=0 adultscore=0 spamscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2403210000 definitions=main-2403280096
X-Proofpoint-GUID: KdUE8kRjhKb4hGqQnbpjQBJpt5khBv-b
X-Proofpoint-ORIG-GUID: KdUE8kRjhKb4hGqQnbpjQBJpt5khBv-b

On Wed, Mar 27, 2024 at 06:09:28PM -0700, Dai Ngo wrote:
> 
> On 3/26/24 11:27 AM, Chuck Lever wrote:
> > On Tue, Mar 26, 2024 at 11:13:29AM -0700, Dai Ngo wrote:
> > > Currently when a nfs4_client is destroyed we wait for the cb_recall_any
> > > callback to complete before proceed. This adds unnecessary delay to the
> > > __destroy_client call if there is problem communicating with the client.
> > By "unnecessary delay" do you mean only the seven-second RPC
> > retransmit timeout, or is there something else?
> 
> when the client network interface is down, the RPC task takes ~9s to
> send the callback, waits for the reply and gets ETIMEDOUT. This process
> repeats in a loop with the same RPC task before being stopped by
> rpc_shutdown_client after client lease expires.

I'll have to review this code again, but rpc_shutdown_client
should cause these RPCs to terminate immediately and safely. Can't
we use that?


> It takes a total of about 1m20s before the CB_RECALL is terminated.
> For CB_RECALL_ANY and CB_OFFLOAD, this process gets in to a infinite
> loop since there is no delegation conflict and the client is allowed
> to stay in courtesy state.
> 
> The loop happens because in nfsd4_cb_sequence_done if cb_seq_status
> is 1 (an RPC Reply was never received) it calls nfsd4_mark_cb_fault
> to set the NFSD4_CB_FAULT bit. It then sets cb_need_restart to true.
> When nfsd4_cb_release is called, it checks cb_need_restart bit and
> re-queues the work again.

Something in the sequence_done path should check if the server is
tearing down this callback connection. If it doesn't, that is a bug
IMO.

Btw, have you checked NFSv4.0 behavior?


> > I can see that a server shutdown might want to cancel these, but why
> > is this a problem when destroying an nfs4_client?
> 
> Destroying an nfs4_client is called when the export is unmounted.

Ah, agreed. Thanks for reminding me.


> Cancelling these calls just make the process a bit quicker when there
> is problem with the client connection, or preventing the unmount to
> hang if there is problem at the workqueue and a callback work is
> pending there.
> 
> For CB_RECALL, even if we wait for the call to complete the client
> won't be able to return any delegations since the nfs4_client is
> already been destroyed. It just serves as a notice to the client that
> there is a delegation conflict so it can take appropriate actions.
> 
> > > This patch addresses this issue by cancelling the CB_RECALL_ANY call from
> > > the workqueue when the nfs4_client is about to be destroyed.
> > Does CB_OFFLOAD need similar treatment?
> 
> Probably. The copy is already done anyway, this is just a notification.

It would be a nicer design if all outstanding callback RPCs could
be handled with one mechanism instead of building a separate
shutdown method for each operation type.


> -Dai
> 
> > 
> > 
> > > Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
> > > ---
> > >   fs/nfsd/nfs4callback.c | 10 ++++++++++
> > >   fs/nfsd/nfs4state.c    | 10 +++++++++-
> > >   fs/nfsd/state.h        |  1 +
> > >   3 files changed, 20 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
> > > index 87c9547989f6..e5b50c96be6a 100644
> > > --- a/fs/nfsd/nfs4callback.c
> > > +++ b/fs/nfsd/nfs4callback.c
> > > @@ -1568,3 +1568,13 @@ bool nfsd4_run_cb(struct nfsd4_callback *cb)
> > >   		nfsd41_cb_inflight_end(clp);
> > >   	return queued;
> > >   }
> > > +
> > > +void nfsd41_cb_recall_any_cancel(struct nfs4_client *clp)
> > > +{
> > > +	if (test_bit(NFSD4_CLIENT_CB_RECALL_ANY, &clp->cl_flags) &&
> > > +			cancel_delayed_work(&clp->cl_ra->ra_cb.cb_work)) {
> > > +		clear_bit(NFSD4_CLIENT_CB_RECALL_ANY, &clp->cl_flags);
> > > +		atomic_add_unless(&clp->cl_rpc_users, -1, 0);
> > > +		nfsd41_cb_inflight_end(clp);
> > > +	}
> > > +}
> > > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> > > index 1a93c7fcf76c..0e1db57c9a19 100644
> > > --- a/fs/nfsd/nfs4state.c
> > > +++ b/fs/nfsd/nfs4state.c
> > > @@ -2402,6 +2402,7 @@ __destroy_client(struct nfs4_client *clp)
> > >   	}
> > >   	nfsd4_return_all_client_layouts(clp);
> > >   	nfsd4_shutdown_copy(clp);
> > > +	nfsd41_cb_recall_any_cancel(clp);
> > >   	nfsd4_shutdown_callback(clp);
> > >   	if (clp->cl_cb_conn.cb_xprt)
> > >   		svc_xprt_put(clp->cl_cb_conn.cb_xprt);
> > > @@ -2980,6 +2981,12 @@ static void force_expire_client(struct nfs4_client *clp)
> > >   	clp->cl_time = 0;
> > >   	spin_unlock(&nn->client_lock);
> > > +	/*
> > > +	 * no need to send and wait for CB_RECALL_ANY
> > > +	 * when client is about to be destroyed
> > > +	 */
> > > +	nfsd41_cb_recall_any_cancel(clp);
> > > +
> > >   	wait_event(expiry_wq, atomic_read(&clp->cl_rpc_users) == 0);
> > >   	spin_lock(&nn->client_lock);
> > >   	already_expired = list_empty(&clp->cl_lru);
> > > @@ -6617,7 +6624,8 @@ deleg_reaper(struct nfsd_net *nn)
> > >   		clp->cl_ra->ra_bmval[0] = BIT(RCA4_TYPE_MASK_RDATA_DLG) |
> > >   						BIT(RCA4_TYPE_MASK_WDATA_DLG);
> > >   		trace_nfsd_cb_recall_any(clp->cl_ra);
> > > -		nfsd4_run_cb(&clp->cl_ra->ra_cb);
> > > +		if (!nfsd4_run_cb(&clp->cl_ra->ra_cb))
> > > +			clear_bit(NFSD4_CLIENT_CB_RECALL_ANY, &clp->cl_flags);
> > >   	}
> > >   }
> > > diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
> > > index 01c6f3445646..259b4af7d226 100644
> > > --- a/fs/nfsd/state.h
> > > +++ b/fs/nfsd/state.h
> > > @@ -735,6 +735,7 @@ extern void nfsd4_change_callback(struct nfs4_client *clp, struct nfs4_cb_conn *
> > >   extern void nfsd4_init_cb(struct nfsd4_callback *cb, struct nfs4_client *clp,
> > >   		const struct nfsd4_callback_ops *ops, enum nfsd4_cb_op op);
> > >   extern bool nfsd4_run_cb(struct nfsd4_callback *cb);
> > > +extern void nfsd41_cb_recall_any_cancel(struct nfs4_client *clp);
> > >   extern int nfsd4_create_callback_queue(void);
> > >   extern void nfsd4_destroy_callback_queue(void);
> > >   extern void nfsd4_shutdown_callback(struct nfs4_client *);
> > > -- 
> > > 2.39.3
> > > 

-- 
Chuck Lever