Received-SPF: pass (google.com: domain of linux-nfs+bounces-2010-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <9d3c4355-3a71-4127-90a9-73d3122f92a3@oracle.com>
Date: Sat, 17 Feb 2024 13:27:15 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 1/1] NFSD: send OP_CB_RECALL_ANY to clients when number of
 delegations reach a threshold
Content-Language: en-US
To: Chuck Lever <chuck.lever@oracle.com>
Cc: jlayton@kernel.org, linux-nfs@vger.kernel.org
References: <1708192859-25002-1-git-send-email-dai.ngo@oracle.com>
 <ZdEgpStNxUc94j01@klimt.1015granger.net>
From: dai.ngo@oracle.com
In-Reply-To: <ZdEgpStNxUc94j01@klimt.1015granger.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?NTlZNzBEMzN6T3lBampCd2hvZGpHNmpQZG5DUHBZdWtXNk0zMC8xaTUzVEdF?=
 =?utf-8?B?MDB5ZnZHejVUUEo3K3FLckdTVnZxdWJpdXhUdnIyQTF2MzFhdVIzZ3dPQ090?=
 =?utf-8?B?SWsxSEpJV210cW1CWEZBeEsyMzBkUjFaYXBBajY3aG8ySDVUaFBCVXVIVFFQ?=
 =?utf-8?B?ZCtFUVFlb0trNENmNkd0S3hXcCtzd3B4SzZ1SGt4UFRqWFI5dFU2Y0RlN2dN?=
 =?utf-8?B?clpPdEFrcE8ydzREandsK05LS3RnelcxYnhrTU9HdHhSNnEvTFJZNHNuV1I5?=
 =?utf-8?B?cUR0VUdEWWI2OGN5VXZOVjNMVytBMm9LbE4vV1JzUVRtRXFSV0labys4emsv?=
 =?utf-8?B?alJjeDZyb1d4c3JDdGVYZkxJNlFnWHBCckZ5Q3QxWjdZcGVJbFBNTGQ0NDRR?=
 =?utf-8?B?dERNdDc3bWVKZXlzem93NnU3ZmdQSzMzWFNJT3JuQVRjYlJuWDFLbUhXSEEv?=
 =?utf-8?B?WU43T081RlRFZis4N2JaTnZHVlpiSzA0Wnk5UkIzb0IwcGwyN29Tb044em9y?=
 =?utf-8?B?UjM1V0VuVmpKT3FmQ3h5b2RyRG43TWVCV3JSQkY1ZnY4WEs2VXhZMW0wZ0Vo?=
 =?utf-8?B?NGpMZmdGLzRtcGFhcFpQRnorMEN1dzB5VkhtR0tMeDdyT2w0VEJtTm9JWHBI?=
 =?utf-8?B?NWNDQUZJUmZqdXRMWG0vNzVzcTNERDNhL21zYWhEeStzZFcwTWVOU2U1WlYv?=
 =?utf-8?B?Y1RGRFE4VDluMTAvSkVybk9MQUlEQzMvSUhvT1o4U1pSaU5OVHpGaDAwVmsy?=
 =?utf-8?B?RVYrcFBHNjFWYmw5aWk1b05SaHhiUndqcVdxNW9ML3ZLb08vWURNa3VOZ0s0?=
 =?utf-8?B?em9hdERCRGMwMEZ1WmRjS2h4d1U4ekdxV3czM2tnRFppcWkvcEtzMlZocmxL?=
 =?utf-8?B?QWF1eFRSVERocDNnUzlsSXdtWDhXZElhSlRNeDdUNnM1ZjNwblU1cDVxOUk3?=
 =?utf-8?B?bktpUGF6eFRUa2l5dm0rRnZSL3RaK2RsL25welpJQ21hMHRaUHZkWEdtWVll?=
 =?utf-8?B?aC9vcGd2V3JKM2Q2TjcwOGduK204OEp2ZW1RWUkxNVlEY1BqTy9JKzVCOGpm?=
 =?utf-8?B?TWdnZ0RpbmVLeUVvWms0ZGFnTktucy9pSE1aNzE4VksraGZqMHgwTTNqUUFx?=
 =?utf-8?B?cDhzTnRDQnNoSlBCTkFKOXZmU1o0c1NETC9JVCs4dmV0eDBhM2Nzc1VLMCtI?=
 =?utf-8?B?dHN5Y2FLTVFwTCtyTGloQUpaNkY1VGprMzFBWW9hNmN0bXVpbUhtY3p1VFNn?=
 =?utf-8?B?RE5sUEdYVEErNEVCeW8wcm0wbWdvMldSS2QzV2xnRjU5dWIwdXhXRmhyd3M4?=
 =?utf-8?B?QlBERHNnemd4Qkt4T3Z5VUZIZHlXd0lnd2xObSt1a0tjZXpjM3NNZjNDUzRU?=
 =?utf-8?B?Rk9mU2h1ZVdhVXJDazZGSWlyTEV4N0JsNWZyaWEzejhWMFpUa0oybHdlQ3oy?=
 =?utf-8?B?NXNWeDJEZnVUa2tubVVCSWM1UDNya1ZiRFZVNUFFNmxqcGxMQW1ya3hqdk5F?=
 =?utf-8?B?RXZFdVJZdzV0TFhjbit0M2NJanJpdFNMbXNuNUtRNnRuY1NXRDFPb3luYkZC?=
 =?utf-8?B?LzJzMHExUWtBZ0V5Y25XblNKbjB0NncrcXo5OEdQWjJIVFFaZ3hDdHNBb29k?=
 =?utf-8?B?eHR3Q0dPdGszdnZVTGl6aFkwdjl4THdEcXlXWFR6YTlJMzY4K2tCZ3dnNW5E?=
 =?utf-8?B?RnhJSkM5dFVQYUVVZUREL0hmdEVxU1FJWjZ3cE85L3Q2MllwMEl6M0Z6MWJZ?=
 =?utf-8?B?Tm82aDJ2Q3duU2F1U3RKUVMxM0VxNklmZ25oVGh2RmF6TFNKTTdOa3RIM0d6?=
 =?utf-8?B?NXdnOXJaaWdVZ0JyWm9ubWxvakJBdEVCdlNsdDNhd0NkaFUvenhxUmxjb1FP?=
 =?utf-8?B?d01kS3Zna3M4MlI4WVhXcllKUHB5OG5ocDJveHppQWNXSTd2bTdzeDlNMWRQ?=
 =?utf-8?B?SlhyQmNMUkFNeUg2L3JDdkVOWGxlMFZaSDE4NCtsV3NVajZvbFFxMlJ0THlw?=
 =?utf-8?B?eC9qbmpMbXB3RzZwMDFpcE9qMm9ZSC9WZ2p1R2xGenZXVEM3VnVXUW9SRXk0?=
 =?utf-8?B?VDExeUNBaW0xSkVvVHcyL1Z1S01qTE4xV0NPUG5VdjArd3c1WS9udDJyQlYz?=
 =?utf-8?Q?RelA5kko/M8RFZbCUj01RD5uz?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	2XFv7DhAdTAzpamytD64nDbhi6QF1WFGuHy23CVPH/2aQ5seidNjzPY6PPTgooEi4JWR1FxjBDmNn2V2U1Yd3tFRac8/mIpSvHDQBfKdUn0M5GUXLi/NP3KKTkOcG7cL/W9+P20F9SZWmJrEVJGFde7hJNrLELydfb2nf2GBcZ6FaWc40rOXnD2z+PjJ80Qdn3gohJ+6Ly9WbimWAk+hiSc/leUkkX7ljWWMM2ERmjYSyoQxfl2vGJBAvLWHV+5ZX+/AvkVALT4LlNwAkFo96j6LQ2DsU8SnZvKtyE4NFgOyKEFUaRY/QUKSR2Hg+Zrf5p3x6ZIF1+o6eLgusTCDfiWTejCHdoklj2SFwrs+N0mLD4nUMqCLlAdTPRdsWeaw/sREfLsXwyuzWF8eUfKH77Ntj93tM8DQaIpyPPhJ0NSoUjd3zYJTMbMUXpg+6Llbjnh+Zsh0u6pbsv7k0fI6CQRDjxpEOpVZsX9VKZgpc28NWehQoeHGr8ArRmvQhquyqPB/rLTNlHgwbT+FNzworhprYo6T6Zlh0DWeuiIIRwhaGk95xADIhi99uYjyKXPfqE5Am8Nx388QkUCH36QCynwEVasMuDY+QAJKOaB/M5Y=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: db2bee81-ca80-4e23-d97b-08dc2fff3710
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Feb 2024 21:27:16.8382
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: K35qQDv/xwalXnCahCWPmFLN20w0csGy5cg+vd4BgwSrSrd7oupdACgFfFsDtF1jeK4pC8VbiSxg77bAcp+paQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR10MB6745
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-17_20,2024-02-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0
 phishscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 bulkscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402170177
X-Proofpoint-GUID: 1gf2h2eUqqk9kX4zxQ3fDMnBGuq9PdxK
X-Proofpoint-ORIG-GUID: 1gf2h2eUqqk9kX4zxQ3fDMnBGuq9PdxK


On 2/17/24 1:09 PM, Chuck Lever wrote:
> On Sat, Feb 17, 2024 at 10:00:59AM -0800, Dai Ngo wrote:
>> When the number of granted delegation becomes large, there are some
>> undesire effects happen on the NFS server. Besides the consumption
>> of system resources, the number of entries on the linked lists of the
>> file cache can grow significantly.
>>
>> When this condition happens, the NFS performace grounds to a halt as
>> reported here [1].
> That was a v5.15.131 kernel. The LRU problems were addressed in
> v6.2. This doesn't seem like a clean rationale for adding this
> reaper behavior in, say, v6.9.

Do you know what commits in v6.9 fix this problem?

Regardless, I think when the number of delegations is getting large,
it's good to ask the clients to release unused delegations. The
Linux client keeps unused delegations for about ~90 secs before
returning them.

>
>
>> This patch attempts to alleviate this problem by asking the clients to
>> voluntarily return any unused delegations when the number of delegation
>> reaches 3/4 of the max_delegations by sending OP_CB_RECALL_ANY to all
>> clients that hold delegations.
> I don't have a strong sense of how big max_delegations can get. Is
> there evidence that CB_RECALL_ANY has enough impact, reliably, to
> reduce the size of the filecache?

I don't have any numbers for this. But in my testing, the Linux client
returns unused delegations immediately when receiving the CB_RECALL_ANY
instead of keeping them ~90 secs.

>
> More below.
>
>
>> [1] https://lore.kernel.org/all/PH0PR14MB5493F59229B802B871407F9CAA442@PH0PR14MB5493.namprd14.prod.outlook.com
>>
>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>> ---
>>   fs/nfsd/nfs4state.c | 9 +++++++++
>>   1 file changed, 9 insertions(+)
>>
>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>> index fdc95bfbfbb6..5fb83853533f 100644
>> --- a/fs/nfsd/nfs4state.c
>> +++ b/fs/nfsd/nfs4state.c
>> @@ -130,6 +130,7 @@ static const struct nfsd4_callback_ops nfsd4_cb_notify_lock_ops;
>>   static const struct nfsd4_callback_ops nfsd4_cb_getattr_ops;
>>   
>>   static struct workqueue_struct *laundry_wq;
>> +static void deleg_reaper(struct nfsd_net *nn);
>>   
>>   int nfsd4_create_laundry_wq(void)
>>   {
>> @@ -696,6 +697,9 @@ static struct nfsd_file *find_any_file_locked(struct nfs4_file *f)
>>   static atomic_long_t num_delegations;
>>   unsigned long max_delegations;
>>   
>> +/* threshold to trigger deleg_reaper */
>> +static unsigned long delegations_soft_limit;
>> +
>>   /*
>>    * Open owner state (share locks)
>>    */
>> @@ -6466,6 +6470,7 @@ nfs4_laundromat(struct nfsd_net *nn)
>>   	struct nfs4_cpntf_state *cps;
>>   	copy_stateid_t *cps_t;
>>   	int i;
>> +	long n;
>>   
>>   	if (clients_still_reclaiming(nn)) {
>>   		lt.new_timeo = 0;
>> @@ -6550,6 +6555,9 @@ nfs4_laundromat(struct nfsd_net *nn)
>>   	/* service the server-to-server copy delayed unmount list */
>>   	nfsd4_ssc_expire_umount(nn);
>>   #endif
>> +	n = atomic_long_inc_return(&num_delegations);
> I don't think you want to modify the number of delegations here.
> atomic_long_read() instead?

Right, it's a typo, it should be a read.

>
>
>> +	if (n > delegations_soft_limit)
> This introduces a mixed-sign comparison: n is a long, but
> delegations_soft_limit is an unsigned long. I'm always suspicious
> about whether an atomic counter can underflow, and then we have
> real problems when there are mixed-sign comparisons.

Yes, n should be unsigned long.

>
> But I'm also wondering if, instead, this logic should look directly
> at the length of the filecache LRU.

Is there a quick check; counter? otherwise if we have to walk the
list then it just compounds the problem.

>
>
>> +		deleg_reaper(nn);
>>   out:
>>   	return max_t(time64_t, lt.new_timeo, NFSD_LAUNDROMAT_MINTIMEOUT);
>>   }
>> @@ -8482,6 +8490,7 @@ set_max_delegations(void)
>>   	 * giving a worst case usage of about 6% of memory.
>>   	 */
>>   	max_delegations = nr_free_buffer_pages() >> (20 - 2 - PAGE_SHIFT);
>> +	delegations_soft_limit = (max_delegations / 4) * 3;
> I don't see a strong reason to keep delegations_soft_limit as a
> separate variable.

Just to save some CPU cycles for not to recompute the soft limit every time.
However, since this is done in the laundromat then it probably not that
critical. Also recompute the soft limit will work correctly if the max_delegations
is changed dynamically.

-Dai

>
>
>>   }
>>   
>>   static int nfs4_state_create_net(struct net *net)
>> -- 
>> 2.39.3
>>