Received-SPF: pass (google.com: domain of linux-nfs+bounces-2467-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <2803859c-8849-4f1c-a75a-f1f0307bbb01@oracle.com>
Date: Mon, 25 Mar 2024 11:57:39 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: [External] : [PATCH v2] fs: nfsd: use group allocation/free of
 per-cpu counters API
To: Kefeng Wang <wangkefeng.wang@huawei.com>,
        Jeff Layton
 <jlayton@kernel.org>, Neil Brown <neilb@suse.de>,
        Olga Kornievskaia <kolga@netapp.com>, Dai Ngo <Dai.Ngo@oracle.com>,
        Tom Talpey <tom@talpey.com>, Dennis Zhou <dennis@kernel.org>
Cc: linux-nfs@vger.kernel.org
References: <20240325132139.113933-1-wangkefeng.wang@huawei.com>
Content-Language: en-US
From: Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <20240325132139.113933-1-wangkefeng.wang@huawei.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?MUpxajFhNkYxRVRPRVdXY2g0Q0tJOFNkQ243QjRoQ1VLZGdNREV4QmhZdU4w?=
 =?utf-8?B?cUdDUURLLytNTUYrTWh0SEJaVjVDaEVOTko4RTRxb204L1FkWnRMYTV0QlNx?=
 =?utf-8?B?NEtTZHo3amtWUkp1SkFqUnJBYjV3VzJkdHl5WEVpT2hQMU1qQkFBamZPczBk?=
 =?utf-8?B?akMwR3NuRVQxRkJuTnpLem9salZpWExwVWcrQzRoaXVhUXZUY3lhMlVad1lq?=
 =?utf-8?B?ZHNMTkxLZVo5VzRDbFFZTWZEcFVhNWZCRzFyTnQ4akd3ZnlqSVdSYlc5bE80?=
 =?utf-8?B?S21xL0lrOVE5SHd4R1hlQkJ2V25tTjBpT2RRc3kyTjNtVllhK3RsNllCNFIx?=
 =?utf-8?B?Tml3VFhsYlhyQk52OHQ0S2lQdERqbXhEeFAzNENqaktwMStrT0dkdjlpcHYr?=
 =?utf-8?B?SEtiR2hEaCtrOWJEOFBhVUNsWFpkckFralA3TUhDTGZvN3BaOXkzNWd0b2Zy?=
 =?utf-8?B?VFBLanpVOTVFc2JuM01sSVdKZjVCaXBrWUlpcTR1UUNQbGR3dS9HSnc4dWtR?=
 =?utf-8?B?eVhBVjZPMThHWDdSVGZVTms5MEIrckZLTGJvSXlkZTVpaG5DUUY2M0hGWjUx?=
 =?utf-8?B?blYrMUFMV29oTVUvYm52cVYvZTJWWkdHbUMyeFZkbmJhNmx2YllNRXJKTmxF?=
 =?utf-8?B?WDhpdktjQUcxd3BtSTBPK0pNNGYyak52ZHdGeXljNnFZSkthdDg0dmtQM2pO?=
 =?utf-8?B?S0U4T3Y5NWt6ZEw3Zmdsc1BkVXdneXlJRXFtNG9LZGYxOTlhdUxPZmZUc1ZL?=
 =?utf-8?B?b09rMnB3QkI2R2tGUUxQTmx6OW04UUptR1h4eWc1TWEzbHRpUWVrR3VLZ2Zm?=
 =?utf-8?B?QzI5eUp1dmRHTVU5OXVGbkdNcFkxNFozd096TE5DTVJlSG03L2hSOVJ0eFdF?=
 =?utf-8?B?Mk9iTEhTdUt4dFg3eW9KS0wwbllOWHFzeWovWmpoRFhmakp1N1lHcjBrTWlZ?=
 =?utf-8?B?NUtiSkpyb1ZyTm5rZ0JVSTc5eFFPb0REc2h2RzNzOElOQTlreUlaaVllZmUr?=
 =?utf-8?B?eTFUc0dUdWZISnhJTFY5Wm1LWTFyVFZIWWQ2RWlwS3kzSDl3SDN1Q3gyNCtF?=
 =?utf-8?B?Q3NTYSt5UzdjT2l1YVZyTjAxeS82WlI3VHVocDFRNmdhbEs0c1Nlc2ZwaFZr?=
 =?utf-8?B?Yzd3YzVrOVJnT1ZrSlU0NE0wckhzclhLUkFLVmdoWnU0REM0YkxaaVZnZ3FC?=
 =?utf-8?B?ekpMVFRKNEZnN2g0SzZqN0Z1WmFGRGZsZDh5cHBwdEIraVdzVU1HbklIWTNl?=
 =?utf-8?B?OXRVamxJcnNRZ1RFMlZ2OFM0RXVqY2lPUEtxMEgxTjFPbmZQQ0dVM3psalVM?=
 =?utf-8?B?MkhNTGErZ1BVVnFyZ25YRkZZVk9kcGR6R1N1K1BZa1NpWjhnOUlvVmgvMXRH?=
 =?utf-8?B?R2hUeWpZREZkV1F4N3ZHY0wxMHpEWUdrNVp0OGI5WU0vKzN3dXByTGV4TWhs?=
 =?utf-8?B?eGlWQzhrcTNkNDRKQkRTUS9FK3NYMEhoQ01HTVFwZFN3OEcxT0YzTG5wdjFH?=
 =?utf-8?B?YWtDcll1bzN1SUxPUjBFZU5PbkQxbkdrOEZKL2RPQUZRVjYxcVdoU1JMandS?=
 =?utf-8?B?Y1dXRnVFTkxtV1c4eTRWSzlqWUh6bERmajVaNUkzSnFnU2lxTWhENnR4TEkv?=
 =?utf-8?B?M294YmJWWnYvL3lEQ2tKYUo1Ly9BRUZMODhPRDQwdFQ2M09SODlyaXJmV2ph?=
 =?utf-8?B?QkJvcDVSUzNYS0tvNjhkR1Q0c0ZTR3ZaL1ZkZnV5QlQrU3c3MVRtNUwwaEVC?=
 =?utf-8?B?UVF4Y0VBdjA0R3pjS2dJdzBBdG5qU1VEdGJ3NDZLN05nSkxqblZLQ241dERL?=
 =?utf-8?B?bUtXRDRWTWpsNEdla1NMSXpyNXY4aEF4WnBuRFcwczNLWHlDL0c4Mk1NSThF?=
 =?utf-8?B?VnM0MVRUc3JSOGtBY0hwU2lMYWI3amdJV3c5QTFRZGFqQlNkNVdlUVhlVzha?=
 =?utf-8?B?RXgvUEdkdmcvK1pNeC9WYmN0NW5UV0FIVGpOMG04T1JLZm5QcjNhMHF4R2NI?=
 =?utf-8?B?ZEIzcGxvQWZiNUk4REdGSnFlMHk2d1hQeUVMWWZyVGp0SGJucnNlWHlJWnhD?=
 =?utf-8?B?aEVna1NZSk13RUowRXR6QXJmQ3IvcDYxQmlzRXJnT2NKVVFvZHR2OEdteWNq?=
 =?utf-8?B?SDBhWHdGMmQ3TzA2eWtsUFBNa09sVDVFbnZSSVRtRGFPM1p5d040cDdmV2pz?=
 =?utf-8?B?OUE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	Dbq0+40h8k3cgATgGVeqXXCGShnxieMXPbfWBC/gUotackplWTa95vs4PV6/yJVHvsgYw+doYo9Up5RRY2WL17eTObMMMRwJPuGaLQN7ruhT9IJNmw4Cg72rnvYbbQzMx5zMsQldXgXzXTTxI8QlcKqpbUWh1vdqyL8sW9ugCOHwrmGjnfz3JZwsPut1rTY4n7GpCUrRLJfJNjO4eCouJEySLFvK3FtquULhDDxaP653XLdoiGvx7NKAQfpWvIvnqbpSl2Wgol4wLlQkOc1CZXs8BJw1NnbNCD5akk9CwvN3JDuRGGn7Kbfo4lZz1hqsDHDnW40jiJFdUx3MZOsLbX4HRvOk/gffUw+tZdGWC+wRlNf6x8MVoziicvt/ljg9JaCG4/ZJgywHr7xntnVUWlZFEMgDFmUePbVgoXKIAIetrJdflsD5RpHF6dBD0rgGachiJZjjhLaMJPMSunIKN5z92OcPbAPjw6o6owUNroc7V91NtRugiZKJTEbvsWzrAFbRWoUcWt8eet32tLO3lTP/WBEnmHzwzV4WDiC4uEHJjfsuRLvxEf08H2X2rdsLwewjQjUTO52z4xwjexj0OKP0oTKpOeNq94zGoWrnCAI=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0a8e0dc1-ca5e-44e5-6d65-08dc4ce44d49
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2024 15:57:41.3739
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MAhKzUWpFI/556UXG2y7QcW+saA+f8ifeUxHXqNhvHWMfBXAtOPH3ZnAQFo5SYQmDdRODaPyueUwzMHv4pTCUg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR10MB4902
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-03-25_12,2024-03-21_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0
 adultscore=0 mlxscore=0 phishscore=0 malwarescore=0 spamscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2403210000 definitions=main-2403250088
X-Proofpoint-ORIG-GUID: Bp3sQeLWP5AZ8Dk0DaUnffS1RNpDBaVM
X-Proofpoint-GUID: Bp3sQeLWP5AZ8Dk0DaUnffS1RNpDBaVM


On 3/25/24 9:21 AM, Kefeng Wang wrote:
> Use group allocation/free of per-cpu counters api to accelerate
> nfsd percpu_counters init/destroy(), and also squash the
> nfsd_percpu_counters_init/reset/destroy() and nfsd_counters_init/destroy()
> into callers to simplify code.
>
> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>


Applied to nfsd-next.

> ---
> v2:
> - directly use percpu_counter_*_many helpers and drop wrappers,
>    suggested by Jeff Layton
>
>   fs/nfsd/export.c | 16 ++++++++++------
>   fs/nfsd/nfsctl.c |  5 +++--
>   fs/nfsd/stats.c  | 42 ------------------------------------------
>   fs/nfsd/stats.h  |  5 -----
>   4 files changed, 13 insertions(+), 55 deletions(-)
>
> diff --git a/fs/nfsd/export.c b/fs/nfsd/export.c
> index 7b641095a665..50b3135d07ac 100644
> --- a/fs/nfsd/export.c
> +++ b/fs/nfsd/export.c
> @@ -334,21 +334,25 @@ static void nfsd4_fslocs_free(struct nfsd4_fs_locations *fsloc)
>   static int export_stats_init(struct export_stats *stats)
>   {
>   	stats->start_time = ktime_get_seconds();
> -	return nfsd_percpu_counters_init(stats->counter, EXP_STATS_COUNTERS_NUM);
> +	return percpu_counter_init_many(stats->counter, 0, GFP_KERNEL,
> +					EXP_STATS_COUNTERS_NUM);
>   }
>   
>   static void export_stats_reset(struct export_stats *stats)
>   {
> -	if (stats)
> -		nfsd_percpu_counters_reset(stats->counter,
> -					   EXP_STATS_COUNTERS_NUM);
> +	if (stats) {
> +		int i;
> +
> +		for (i = 0; i < EXP_STATS_COUNTERS_NUM; i++)
> +			percpu_counter_set(&stats->counter[i], 0);
> +	}
>   }
>   
>   static void export_stats_destroy(struct export_stats *stats)
>   {
>   	if (stats)
> -		nfsd_percpu_counters_destroy(stats->counter,
> -					     EXP_STATS_COUNTERS_NUM);
> +		percpu_counter_destroy_many(stats->counter,
> +					    EXP_STATS_COUNTERS_NUM);
>   }
>   
>   static void svc_export_put(struct kref *ref)
> diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
> index ecd18bffeebc..93c87587e646 100644
> --- a/fs/nfsd/nfsctl.c
> +++ b/fs/nfsd/nfsctl.c
> @@ -1672,7 +1672,8 @@ static __net_init int nfsd_net_init(struct net *net)
>   	retval = nfsd_idmap_init(net);
>   	if (retval)
>   		goto out_idmap_error;
> -	retval = nfsd_stat_counters_init(nn);
> +	retval = percpu_counter_init_many(nn->counter, 0, GFP_KERNEL,
> +					  NFSD_STATS_COUNTERS_NUM);
>   	if (retval)
>   		goto out_repcache_error;
>   	memset(&nn->nfsd_svcstats, 0, sizeof(nn->nfsd_svcstats));
> @@ -1704,7 +1705,7 @@ static __net_exit void nfsd_net_exit(struct net *net)
>   	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
>   
>   	nfsd_proc_stat_shutdown(net);
> -	nfsd_stat_counters_destroy(nn);
> +	percpu_counter_destroy_many(nn->counter, NFSD_STATS_COUNTERS_NUM);
>   	nfsd_idmap_shutdown(net);
>   	nfsd_export_shutdown(net);
>   	nfsd_netns_free_versions(nn);
> diff --git a/fs/nfsd/stats.c b/fs/nfsd/stats.c
> index be52fb1e928e..bb22893f1157 100644
> --- a/fs/nfsd/stats.c
> +++ b/fs/nfsd/stats.c
> @@ -73,48 +73,6 @@ static int nfsd_show(struct seq_file *seq, void *v)
>   
>   DEFINE_PROC_SHOW_ATTRIBUTE(nfsd);
>   
> -int nfsd_percpu_counters_init(struct percpu_counter *counters, int num)
> -{
> -	int i, err = 0;
> -
> -	for (i = 0; !err && i < num; i++)
> -		err = percpu_counter_init(&counters[i], 0, GFP_KERNEL);
> -
> -	if (!err)
> -		return 0;
> -
> -	for (; i > 0; i--)
> -		percpu_counter_destroy(&counters[i-1]);
> -
> -	return err;
> -}
> -
> -void nfsd_percpu_counters_reset(struct percpu_counter counters[], int num)
> -{
> -	int i;
> -
> -	for (i = 0; i < num; i++)
> -		percpu_counter_set(&counters[i], 0);
> -}
> -
> -void nfsd_percpu_counters_destroy(struct percpu_counter counters[], int num)
> -{
> -	int i;
> -
> -	for (i = 0; i < num; i++)
> -		percpu_counter_destroy(&counters[i]);
> -}
> -
> -int nfsd_stat_counters_init(struct nfsd_net *nn)
> -{
> -	return nfsd_percpu_counters_init(nn->counter, NFSD_STATS_COUNTERS_NUM);
> -}
> -
> -void nfsd_stat_counters_destroy(struct nfsd_net *nn)
> -{
> -	nfsd_percpu_counters_destroy(nn->counter, NFSD_STATS_COUNTERS_NUM);
> -}
> -
>   void nfsd_proc_stat_init(struct net *net)
>   {
>   	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
> diff --git a/fs/nfsd/stats.h b/fs/nfsd/stats.h
> index d2753e975dfd..04aacb6c36e2 100644
> --- a/fs/nfsd/stats.h
> +++ b/fs/nfsd/stats.h
> @@ -10,11 +10,6 @@
>   #include <uapi/linux/nfsd/stats.h>
>   #include <linux/percpu_counter.h>
>   
> -int nfsd_percpu_counters_init(struct percpu_counter *counters, int num);
> -void nfsd_percpu_counters_reset(struct percpu_counter *counters, int num);
> -void nfsd_percpu_counters_destroy(struct percpu_counter *counters, int num);
> -int nfsd_stat_counters_init(struct nfsd_net *nn);
> -void nfsd_stat_counters_destroy(struct nfsd_net *nn);
>   void nfsd_proc_stat_init(struct net *net);
>   void nfsd_proc_stat_shutdown(struct net *net);
>