Received-SPF: pass (google.com: domain of linux-nfs+bounces-2305-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Fri, 15 Mar 2024 10:29:47 -0400
From: Chuck Lever <chuck.lever@oracle.com>
To: Muhammad Asim Zahid <muhammad.zahid@nokia.com>
Cc: "J. Bruce Fields" <bfields@fieldses.org>, linux-nfs@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH] nfsd: Make NFS client_id increment atomic to avoid race
 condition
Message-ID: <ZfRbWx8L9WJGKa_k@manet.1015granger.net>
References: <20240315124053.24116-1-muhammad.zahid@nokia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240315124053.24116-1-muhammad.zahid@nokia.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?bV0YODpRHrJwXilhWIYNXXiQ4qtE8ps4IJ9V3dbekd0v5vmipsN9cMPxpJGr?=
 =?us-ascii?Q?EQXcbogMq9RDutyajWum0N3v+K907u7T7rF1HRki5gpY/+VKH2BOWIhitZaW?=
 =?us-ascii?Q?rkW4YaPv4eH9thzWjPJlL6w8WUc8h7VSeQMiqCCz68OUerhcJg2izWiA61sY?=
 =?us-ascii?Q?0/yNfDjFzXJBS02Ens3EJJM5OFAYSYhtwX9UCTNMmw25Vci0o4rwpyfM0l6I?=
 =?us-ascii?Q?Miei6e09aXBQygkMYPuE32eUYnWeBRWD5Gb+80d9k88+ODKpX0RFuGszfh42?=
 =?us-ascii?Q?0nEiSUJD15MVoDHOa8cD2AJnBBLU4yyCHIe+OCmo0q9edfBaprSVjY7ATU8m?=
 =?us-ascii?Q?NLLaVbQmucvmGtNiC1kLTWSgJnBxHpumQ/5tUK4z7YyGveUqjxCH5RJYUWUg?=
 =?us-ascii?Q?JR8EXESASFhwa7Q98oxUlbCTu1k2cSaYAEp3tPvFPUUCnbKtqv7SONyl9eT5?=
 =?us-ascii?Q?r6Q5WXBwTkToycoyWA8LodziOle+n+Y4coD4pNlrAEN0bjjC191Ne+RfmJRE?=
 =?us-ascii?Q?E0aAf22vwj2mKuSk+VuoWcJ+b2Xy+nCceb+1yVNXAaGD7JA7bBGD+afG+p0C?=
 =?us-ascii?Q?6ImYVc8RapUfrCse3SkIA65uHSrjh3FAynkGirQzrB6uf3x6WAxWrVJ+e7x1?=
 =?us-ascii?Q?TYV3pOIhJ/2sBYUmXOBDoXSTJy6J4AVtTbGLnzHRpaM51J3My5QnDIyTWWOh?=
 =?us-ascii?Q?uD4ZQovP+txkGSpjqz3rqFj5WKLsdy+Kn3LZKk2GA9ltrO/dnEveNbC8Ab13?=
 =?us-ascii?Q?Z2lGDPrBC2uuqtWDomBAeOd/WmJkE5LCYAjXt9RsFkLUizVCGPkDI8X+PlGe?=
 =?us-ascii?Q?tVgIWBjFUF7US0JfxXNCoK7kkhbNiypJcDVWw+1Et3zZAhB02G6k0B+X5Tqp?=
 =?us-ascii?Q?embZGEqMSs3fMTjd9NLcOPr4XQbXFlG12vPyX+Y+TIyMo6Y+tQDyXWUZX9Qe?=
 =?us-ascii?Q?8n+KulOkAuBHmutwc6E5z/VmOhCQk1DxpL4wChjcwItEqXQ/bTHLTZ3jtJvb?=
 =?us-ascii?Q?cQ31Usn83M0rql3zKVC8w03E5R6OeDfMIFGOuKlW6FtHdKx3wOoVLew9o9E7?=
 =?us-ascii?Q?keQ+gKmkgMrz4/hlvXOgQOZ21/d3aMjABv1Mt/tauoMbW6kVckd6eKL6KVBy?=
 =?us-ascii?Q?h4NT3JmbKA987cYFtpG7V91qzuNCv++uc8ImJW5+KCgfj+Pj78YW6yW7krLD?=
 =?us-ascii?Q?URjR50jdfUGJp/KV70GBhlPRR27BOaNwY14XOnFziBGgxbUSpuXuBe8oavcW?=
 =?us-ascii?Q?LCRIIVW5euw2Mwx/58zIcy4dpQVSpLIfe6Y24R487N+JWYJ8z2YvVCcjyUVL?=
 =?us-ascii?Q?DPF1+15psSLYVytOj1uhIVqsaLoVVFaqpvCD91TlY2mYHG+cLMSXkWWKW+zV?=
 =?us-ascii?Q?MCsMalRGinEOB0GRDDAqWlv6cAITwGhkP1JDlDmCk61D1LfOsfKnpgpGJi/K?=
 =?us-ascii?Q?Q0A6rKXw1PIgGzUzkQGd0jBYL0JMVgmG08OCbcZQtRxiQVviutrsyP5udKBt?=
 =?us-ascii?Q?kSNUlT85B8fhCdBMHr+Vj7/0igpo1ll5qDz0aBf6nTOdnxn0XBbvFDjgasCk?=
 =?us-ascii?Q?3aSfkL0mmtPUtTDUXvCMYl5Zdbav2OkGHWmBulcsn3c3hFv6nJnxjj17Qr+Y?=
 =?us-ascii?Q?Bw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	XLcOgzi1fgaEezlN1r18I7oqXjndlH18Eocysn1X6dScTJRlkNTP0kaSYhscX04klrlNyiJmR4C92x3+O3BWww81Ob9rxVEV6a24MphhDl0MozD9z8eb4y+UkUoHL6P+1X9gmJAwYttdlMs1nyBS8paX6vvh9pPmdN6+M+G2nfeVXbT68ll2FsBSHEEZ+JVQNd/SD+1q/MdbqHjxBmB2Pb1gC0Sxxb5rEv5/DE+w+WI1I/Q4bT4eThwM7eeeas/Dsfqbm0aJDhHFYnQ/4YAprrgeqAljw65Dh2Ax1Myed5ECQ+sbxwjVzgJW7vOvnn1Oe9NW5c7emu0dv5HmKPAUOz3OVu4ErXxo+l8s6NpJssBwPeV0krnV1A3fd61vlwwoWkOotO+XuEWTuaXViJJE3xcs6PiJBuShulzgH54GNLTmlksqkp/h2xcaCq342oIURahhUZ6qhmhp8STy8sOBKk5/Y81qb6+IpgHhJ2xPk5HAQS+bjsx8paFbLpQiSeaM7+3YR8xJ/1SWTU+zN5xseSvZWCekx6rLmmH9bM+N+5C8Tr9cQZkbaXcNSsu612CCs1Mrxtblb+tXlQk6VUIV87i0GFnqhVtARbb0I2dR+tI=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f59a08b9-e47c-481e-a770-08dc44fc5f68
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Mar 2024 14:29:50.3597
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: tg6etQBLaCllO3C3X/eriHIiDgn+mjCQiEAPriqZJXJFzNUNsroCAgiQV+HnFrWnb8djiPdL5vpLicEtLoHl3A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB6097
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-03-15_02,2024-03-13_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 spamscore=0
 bulkscore=0 phishscore=0 malwarescore=0 suspectscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2403150117
X-Proofpoint-ORIG-GUID: 08T1DevPXfVGwpl6aw4OiuwDvzTqiXGf
X-Proofpoint-GUID: 08T1DevPXfVGwpl6aw4OiuwDvzTqiXGf

On Fri, Mar 15, 2024 at 01:39:57PM +0100, Muhammad Asim Zahid wrote:
> The following log messages show conflict in clientid
>        [err] kernel: [   16.228090] NFS: Server fct reports our clientid is in use
>        [warning] kernel: [   16.228102] NFS: state manager: lease expired failed on NFSv4 server fct with error 1
>        [warning] kernel: [   16.228102] NFS: state manager: lease expired failed on NFSv4 server fct with error 1
> 
> The increment to client_verifier counter and client_id counter is
> set to atomic so as to avoid race condition which causes the
> aforementioned error.

These client messages are in response to NFS4ERR_CLID_INUSE. This
condition is not because the server did not increment the client ID.
It's because there actually is another client (possibly more than
one) using the same clientid string and authentication principal.

Refer to:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/nfs/client-identifier.rst


> Change-Id: Ic0fa8c14a8bba043ae8882f6750f512bb5f3aac1
> ---
>  fs/nfsd/netns.h     | 4 ++--
>  fs/nfsd/nfs4state.c | 4 ++--
>  fs/nfsd/nfsctl.c    | 6 +++---
>  3 files changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h
> index 935c1028c217..67b5aa1516e2 100644
> --- a/fs/nfsd/netns.h
> +++ b/fs/nfsd/netns.h
> @@ -119,8 +119,8 @@ struct nfsd_net {
>  	unsigned int max_connections;
>  
>  	u32 clientid_base;
> -	u32 clientid_counter;
> -	u32 clverifier_counter;
> +	atomic_t clientid_counter;
> +	atomic_t clverifier_counter;
>  
>  	struct svc_serv *nfsd_serv;
>  
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 9b660491f393..d67a6a593f59 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -2321,14 +2321,14 @@ static void gen_confirm(struct nfs4_client *clp, struct nfsd_net *nn)
>  	 * __force to keep sparse happy
>  	 */
>  	verf[0] = (__force __be32)(u32)ktime_get_real_seconds();
> -	verf[1] = (__force __be32)nn->clverifier_counter++;
> +	verf[1] = (__force __be32)atomic_inc_return(&(nn->clverifier_counter));
>  	memcpy(clp->cl_confirm.data, verf, sizeof(clp->cl_confirm.data));
>  }
>  
>  static void gen_clid(struct nfs4_client *clp, struct nfsd_net *nn)
>  {
>  	clp->cl_clientid.cl_boot = (u32)nn->boot_time;
> -	clp->cl_clientid.cl_id = nn->clientid_counter++;
> +	clp->cl_clientid.cl_id = atomic_inc_return(&(nn->clientid_counter));
>  	gen_confirm(clp, nn);
>  }
>  
> diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
> index cb73c1292562..a9ef86ee7250 100644
> --- a/fs/nfsd/nfsctl.c
> +++ b/fs/nfsd/nfsctl.c
> @@ -1481,10 +1481,10 @@ static __net_init int nfsd_init_net(struct net *net)
>  	nn->nfsd4_grace = 90;
>  	nn->somebody_reclaimed = false;
>  	nn->track_reclaim_completes = false;
> -	nn->clverifier_counter = prandom_u32();
> +	atomic_set(&(nn->clverifier_counter), prandom_u32());
>  	nn->clientid_base = prandom_u32();
> -	nn->clientid_counter = nn->clientid_base + 1;
> -	nn->s2s_cp_cl_id = nn->clientid_counter++;
> +	atomic_set(&(nn->clientid_counter), nn->clientid_base + 1);
> +	nn->s2s_cp_cl_id = atomic_inc_return(&(nn->clientid_counter));
>  
>  	atomic_set(&nn->ntf_refcnt, 0);
>  	init_waitqueue_head(&nn->ntf_wq);
> -- 
> 2.42.0
> 

-- 
Chuck Lever