Received-SPF: pass (google.com: domain of linux-nfs+bounces-2484-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Tue, 26 Mar 2024 14:27:44 -0400
From: Chuck Lever <chuck.lever@oracle.com>
To: Dai Ngo <dai.ngo@oracle.com>
Cc: jlayton@kernel.org, linux-nfs@vger.kernel.org
Subject: Re: [PATCH 1/1] NFSD: cancel CB_RECALL_ANY call when nfs4_client is
 about to be destroyed
Message-ID: <ZgMToHNkkGEHNb/y@tissot.1015granger.net>
References: <1711476809-26248-1-git-send-email-dai.ngo@oracle.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1711476809-26248-1-git-send-email-dai.ngo@oracle.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?PS5e7yeqgy4loV6rUZciT2V45qkmiiBzT7I7Je6k3tfc7ztLkifaG/oNv9de?=
 =?us-ascii?Q?eTCr3Gr/1Od0O60TJOe5+izXyCI6yt+uUyU7c2sXlKzx9UX//E/XZ3cXbm52?=
 =?us-ascii?Q?orOq3BKvXbcBrTB5fJNYDCwv47e7PtCi+a95+NkrwHmwii10y2GwxIZI4PUw?=
 =?us-ascii?Q?rBN4QA5eCYuqsvJY8Z2k8NDwxPNfYPbxIF5LgN1fYQUnrKo3n0gn1ZNey1+T?=
 =?us-ascii?Q?yrNm/BIQvmLtFm7CMvvhA9jlxt5XQlQLh9AXbVhS0pfYCbu/u/YyspFTs8dl?=
 =?us-ascii?Q?pkKjNjN8afkVcNkZd+40cGbMfrIIrVgo903EBfaEMO4J6qFPrScbJ+2Q1F81?=
 =?us-ascii?Q?KPrgI2aEZesTfKuKeJmV5hdqo2E0N92lHYbOszjKoPlyg1vvDhYLZDE09EK0?=
 =?us-ascii?Q?2pdmXbOQLFTXO3ravxP1WLmuRm4nsRl3xiu2FcqxT4S8BbjuTXc/YaTxJSyg?=
 =?us-ascii?Q?lAdLcUnOj85rXYjLtwBfLv/4DofCzavgT+BxOChVhxk0u8X/dO6CWczMLOTv?=
 =?us-ascii?Q?YrNxOyxBkXbNkc4bIZgWMXT/IJ1llOSaY7iipfyPVz/oCsSV9VbHFZPkOFsY?=
 =?us-ascii?Q?nzia9vZb2qiw3zQliErdHCbv6cdIF2KNHLLnYR0/YdxxL/LgWS4guMZeADTS?=
 =?us-ascii?Q?dXitLjIHCy9OWrciKBIy1eyso0lmKPCb3GL2zUpS5n8GbnOt/E2CgbUd3U4E?=
 =?us-ascii?Q?S7PzRZF0O4BjZ/6IDSm/6YklmnB2VmP7InSh9MJ22BTKWdII2mYWzVBZ7SEr?=
 =?us-ascii?Q?U7J8VcBHvqEiefxdDI8FeJ570MoeJ9CKL+TfOGFYLtU5oekt7Doc/O5SWvy3?=
 =?us-ascii?Q?LIu6erce/vaztK7s+qWVYGDjvioHhmwuHnt5qOBqGJkaIWQvDRc1MsioTVWA?=
 =?us-ascii?Q?QKBeNORWgIRfVhf68wadDhil6jLFkFY/ST6j8TsR573gxSc1Nm1bZ53HcUyH?=
 =?us-ascii?Q?QBwqvNnsFfSkQX2PEXEak7/Ggd2Yq9oKgM1/A9S1XAzkTCKaBPe2PIZexh8Y?=
 =?us-ascii?Q?5/tsfXGfZNvOKloHOT7BGm4KKAnHyEq5reamR3yaQEMRPapKypa1jo4fhi68?=
 =?us-ascii?Q?P1zSBU94iT7AoS31cCXO36FiBGButTap6SjRIjFQuW01ZFOuQcMEaggdiDT2?=
 =?us-ascii?Q?df2XCgFgiyc9rHN4XYNYXKr456j4GPlIczRPGxt0Fpi79DSPCrAxnLmh1fis?=
 =?us-ascii?Q?nSRgRIouzVf6Cvyy40t2tYpuT9nRquTOQDk7TVcoSCVzncCHThZRu45WkTI1?=
 =?us-ascii?Q?41D6Ylt9oBdb41TUvEbMAqS0KqpATslQRB29LBDrCfcGJRhKmBT2kqWTcZTP?=
 =?us-ascii?Q?eHrquaDsjD6ocJ/lg1CZ8bW99FrsTlTgNA6wXh0kYwhlXN5Lc0/lMOrxZcO3?=
 =?us-ascii?Q?njzSWXqMm21zzXI69KsjS2d2mQGJEQAPz4TnX0TlkqekDzQFVGo3vdeA6Tm+?=
 =?us-ascii?Q?ODWHUH2+1cbzxBtQtkBlt9olq7c63v3KV7wBhxVC1RKptWZNlxt1x91ehhgA?=
 =?us-ascii?Q?xtPJu6tyihz78bWEsYGfVPuS/60Iz5u+wH003MN/1U4ho8bi28RaIZI3n4UP?=
 =?us-ascii?Q?rHsd/eHM2DOS8z/KNoa5hK8M4yszGj+ZOGuO1OxkNtYp2iesZyyhLldDKBjY?=
 =?us-ascii?Q?EQ=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	TeUCIJB+MU9bwQBkzSloQlsyLLfLc+pjt7Gvec9zytDAHfWSZl7P0lkRTTpF7XR48rE0v7jzhumRQVil5xuz6D93AfxQGtSwJE+wQOtg2mJl8kTtI5sGFhXl2hahnEX+8B9mNUtupbNKFE+qFs1FsMABzeqBSJdg1P3H5FXmv2i6PaFRM34Xtn/LU7gKpYc5ef1ybA8/e9rdIin4S8AvQCzaBD+ENRcVzjAKGzXy4mh4eZftsoeg4kp3Ypf1yAKuRkwS3+yqRV+R7+Enu+ubb6VPTqIjl19IBDHTWYGsf6MjdzmlqPjHMrzasCSIQVMIHM5HyphdEVTrqvE+Fd1WLolZNATZQItbYwLxEa3ZzHTHMNWUZ6L2Fl04w0/+0L7Bjf+jeoC1YIAS0rhGpxWenNK+WUBtVxVaYRF7pEnAuzesrfPyRJ60Jzeyg70D0DsqvJrgpHKwQRk5IVD9WljHK0JsUfW9p26K5Ej6gMGbyp/G219GVE4dP8gSn56EvattNP135V/L9Ca2N+EM27T9o4oOU0PEhaaPXSgB36aYNS0BdYGVHrEWuV+OLAXbAQ3dQZk/NFzfwO5M1E2WxmIP+UEbZ7es2hpr2d+sfprlhmg=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b0e2a5f7-c592-49b5-10ec-08dc4dc26f55
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Mar 2024 18:27:46.7675
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: q98FW9SNoYIG0U70Dk7u6PIzv6LvlNMBdjPjG4wQrhPlZ9mexEghxdVs08iyjxTM9tyuO2+3s7xB1yJbaoVJQQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6587
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-03-26_08,2024-03-21_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxlogscore=999
 suspectscore=0 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2403210000 definitions=main-2403260132
X-Proofpoint-ORIG-GUID: djH64r9Hwz1w2hohTWw1_OPrHMj9b_A3
X-Proofpoint-GUID: djH64r9Hwz1w2hohTWw1_OPrHMj9b_A3

On Tue, Mar 26, 2024 at 11:13:29AM -0700, Dai Ngo wrote:
> Currently when a nfs4_client is destroyed we wait for the cb_recall_any
> callback to complete before proceed. This adds unnecessary delay to the
> __destroy_client call if there is problem communicating with the client.

By "unnecessary delay" do you mean only the seven-second RPC
retransmit timeout, or is there something else?

I can see that a server shutdown might want to cancel these, but why
is this a problem when destroying an nfs4_client?


> This patch addresses this issue by cancelling the CB_RECALL_ANY call from
> the workqueue when the nfs4_client is about to be destroyed.

Does CB_OFFLOAD need similar treatment?


> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
> ---
>  fs/nfsd/nfs4callback.c | 10 ++++++++++
>  fs/nfsd/nfs4state.c    | 10 +++++++++-
>  fs/nfsd/state.h        |  1 +
>  3 files changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
> index 87c9547989f6..e5b50c96be6a 100644
> --- a/fs/nfsd/nfs4callback.c
> +++ b/fs/nfsd/nfs4callback.c
> @@ -1568,3 +1568,13 @@ bool nfsd4_run_cb(struct nfsd4_callback *cb)
>  		nfsd41_cb_inflight_end(clp);
>  	return queued;
>  }
> +
> +void nfsd41_cb_recall_any_cancel(struct nfs4_client *clp)
> +{
> +	if (test_bit(NFSD4_CLIENT_CB_RECALL_ANY, &clp->cl_flags) &&
> +			cancel_delayed_work(&clp->cl_ra->ra_cb.cb_work)) {
> +		clear_bit(NFSD4_CLIENT_CB_RECALL_ANY, &clp->cl_flags);
> +		atomic_add_unless(&clp->cl_rpc_users, -1, 0);
> +		nfsd41_cb_inflight_end(clp);
> +	}
> +}
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 1a93c7fcf76c..0e1db57c9a19 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -2402,6 +2402,7 @@ __destroy_client(struct nfs4_client *clp)
>  	}
>  	nfsd4_return_all_client_layouts(clp);
>  	nfsd4_shutdown_copy(clp);
> +	nfsd41_cb_recall_any_cancel(clp);
>  	nfsd4_shutdown_callback(clp);
>  	if (clp->cl_cb_conn.cb_xprt)
>  		svc_xprt_put(clp->cl_cb_conn.cb_xprt);
> @@ -2980,6 +2981,12 @@ static void force_expire_client(struct nfs4_client *clp)
>  	clp->cl_time = 0;
>  	spin_unlock(&nn->client_lock);
>  
> +	/*
> +	 * no need to send and wait for CB_RECALL_ANY
> +	 * when client is about to be destroyed
> +	 */
> +	nfsd41_cb_recall_any_cancel(clp);
> +
>  	wait_event(expiry_wq, atomic_read(&clp->cl_rpc_users) == 0);
>  	spin_lock(&nn->client_lock);
>  	already_expired = list_empty(&clp->cl_lru);
> @@ -6617,7 +6624,8 @@ deleg_reaper(struct nfsd_net *nn)
>  		clp->cl_ra->ra_bmval[0] = BIT(RCA4_TYPE_MASK_RDATA_DLG) |
>  						BIT(RCA4_TYPE_MASK_WDATA_DLG);
>  		trace_nfsd_cb_recall_any(clp->cl_ra);
> -		nfsd4_run_cb(&clp->cl_ra->ra_cb);
> +		if (!nfsd4_run_cb(&clp->cl_ra->ra_cb))
> +			clear_bit(NFSD4_CLIENT_CB_RECALL_ANY, &clp->cl_flags);
>  	}
>  }
>  
> diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
> index 01c6f3445646..259b4af7d226 100644
> --- a/fs/nfsd/state.h
> +++ b/fs/nfsd/state.h
> @@ -735,6 +735,7 @@ extern void nfsd4_change_callback(struct nfs4_client *clp, struct nfs4_cb_conn *
>  extern void nfsd4_init_cb(struct nfsd4_callback *cb, struct nfs4_client *clp,
>  		const struct nfsd4_callback_ops *ops, enum nfsd4_cb_op op);
>  extern bool nfsd4_run_cb(struct nfsd4_callback *cb);
> +extern void nfsd41_cb_recall_any_cancel(struct nfs4_client *clp);
>  extern int nfsd4_create_callback_queue(void);
>  extern void nfsd4_destroy_callback_queue(void);
>  extern void nfsd4_shutdown_callback(struct nfs4_client *);
> -- 
> 2.39.3
> 

-- 
Chuck Lever