Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <52820d99-48f6-f56a-fc00-3cbb7fa07f8e@oracle.com>
Date:   Wed, 22 Jun 2022 11:28:10 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.10.0
Subject: Re: [PATCH 1/1] NFSD: Fix memory shortage problem with Courteous
 server.
Content-Language: en-US
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
References: <1655921718-28842-1-git-send-email-dai.ngo@oracle.com>
 <27586B46-7926-46A7-AFF0-4E0F322B4225@oracle.com>
From:   dai.ngo@oracle.com
In-Reply-To: <27586B46-7926-46A7-AFF0-4E0F322B4225@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?U1lSMlhuU243alQ5aTJUZjU4SjBPMW5YdC9iajc0Unc2ZngrUEtHVnVYZFJL?=
 =?utf-8?B?REtMYm1QeE1MWDNRTkUrNDc0NUlsNEoyNWxtR09Mb1FRZ29zeTYvSVNFTGJy?=
 =?utf-8?B?SVBFSm1KMUxBWExnbVd1UW95bnpOcXBlVHR1dXBlaVlsb0JLV1k1dytGTTFO?=
 =?utf-8?B?bmpMUGRxeHYvbDB1dFVPTUZ6RVNtWEp0V212NG5VNmFFRmVKZXNNUFozV3Q5?=
 =?utf-8?B?VW5xZmN5RXlBelJxZmVUQThNWVBwbE9wUzNaYzBpL1lER1ZybFdMV0YrMU1L?=
 =?utf-8?B?Um5KSVlpNDljTExtM0xreHZxOTU3MGxBNFVIVGN0aGFTbVl3RVE2TlVtM3dR?=
 =?utf-8?B?MmpoSnRKczg2c1l6ZldMS3d0Tk5xdE1RcnZudkI0RXpISElMeHBMd1ZLOFNP?=
 =?utf-8?B?dHdmRXMrYmFyVUNlbFVpNm5oOXluNk5TSVhjVyszTnVLWXZkcHVDcHNBWVdR?=
 =?utf-8?B?blhOTWhBQ2xSRmdPZHdDVGpHdllRa3NVSFJiWWV4RWljTUxKQWdLWEFKR2d5?=
 =?utf-8?B?eGVaWm16Ni9hSm5tdTZBanN2aFNHWUtOR1BLdzJlaVlPQUw5Sk5aTGpNKzhY?=
 =?utf-8?B?akRTOU8xMFVMWmJtblpmQVd1OFh2UG40YVlaSmxGMUNPcDNMRDAvZE9vZEZU?=
 =?utf-8?B?WFVvTDdxYnFMcTd5SUIwOTAyalFjRDlyd3lBbUlFdUNqb0tWcmZzNDYvN0Fn?=
 =?utf-8?B?VDRieWY3SDd1Z1pxU1JZNGtmcXgzUWhZSXBzQ0lGZlc0RmxOTlRmU0k0TzhL?=
 =?utf-8?B?S0R5RTRiMURsQ3pVQy9GTDZDWlJQTWRuQWdUY3FVZTdoL05OMndJZERORnZP?=
 =?utf-8?B?Q2ErTGtsN04rL1ljb1YvWW1EbVkrQWc5WW5UR3BmeUxWVlNFS3Q0Y1lsYXYy?=
 =?utf-8?B?ZzFHVHpuVDVHRDlXVFJNOVR6MCtwRWNyRUd4RHVTZ2UzUVdVNEhKRjJTN2lO?=
 =?utf-8?B?V0VVS3BJdzkvbTVHalRndExlV1BrTTNsS1BrdHBlNjR1cnRmUjlFMkFKVFZn?=
 =?utf-8?B?eTd5Mk9udkQ5Zy9sMjlxSUcrUTRLWm9vZXZjVEFjWUd2YzRHMGc3by9vb1Iy?=
 =?utf-8?B?L0hEdHZSQ3pQV2RXMmRibW9PQzZHempadFdXcllsMVRhVjlmeER5ZUQvYU42?=
 =?utf-8?B?Zm55NTRvbjVmM0JjSFRRMUdnSXQzU0NCcWhhOGhwYVQxZzV2aUlVTkF0RWo5?=
 =?utf-8?B?dUo5KzNtQy93S0FtNHB5cWUrRXpEYVo1blRMcEhrOWFYSHEwM1Fna1hGVi9Q?=
 =?utf-8?B?NUYvaDV5WlBFNTdJeFY2Tmg5U29TeFA5V3h3QzMzZmhocCtzZnNRTlFBeU5r?=
 =?utf-8?B?elNkQmxsNTZGQnYzcFJ5MFB0ZUs2SWxETWFBbU0zd3FLZVZ6dXJYU1Q1eE1R?=
 =?utf-8?B?YTVhZy9HenpTbTNhRFo0Rlp4dldwS2FDM21KV1hoWE1ZWEpXSy95cmE5RXU3?=
 =?utf-8?B?MC9tK3E5UVRQa1RNVE9RLzJNUi9MRjUvNWx4N0p0WmcwZXBsQW9UMkwwUG5R?=
 =?utf-8?B?QUZQZWpuWlYrQktPZEg5TjFBVUMwTWRhdWlSZ3N4NnIrOWNyZGl6b1RnUDFJ?=
 =?utf-8?B?ckQ0R0VaT2NEZ2llNW1rYnlSNHJnNVdKSjltenQ0eFpuRkcxN2ZnSnhrT3Q0?=
 =?utf-8?B?aWw0amxJa29mM2Q4cy9kcnFhUmJPb3JtTUVscmdqNEJQWXplOWV2YVo1K0cv?=
 =?utf-8?B?dVBuR2MrM1VsN2ROcEt3Y3RKQSsvRWdUVDNhSXIrQlYwV3A2UVVadW4zb1g2?=
 =?utf-8?B?ODQ2d1RwUmk3cWpJM2dZME1HYnpVaVkyZElpQS9BNHdZeXBHVVY3ZGVVYlRT?=
 =?utf-8?B?WTZCMkRxdy83WVB0TFZqTkt6Mm9mSDc1czhtUEkxY2NOd3ErREl6RE9nT0hw?=
 =?utf-8?B?Z0pXNDZIRFpoM1NmNkF3ZjNrMlo5QUZHbTNvMGQydFYvWGNrbVQzS3JkMjY4?=
 =?utf-8?B?dkZOTHI2T1BQOHhLZFJ6eDY5ZzBCUjNObTNqNHlkdFdyYWZ6VzlIeVdRNVpo?=
 =?utf-8?B?L1c4dnlwNmxESlJqWjMvOE54UHlIWXBkMjB2Z1RROGZyWG1TU2hHSTFnZDRX?=
 =?utf-8?B?Qks1VVp3YXRPV2NDc21OSlJIVE1qTE5HZktZMERUZWlwSFAvV3JUN3ljWFNs?=
 =?utf-8?B?Zmk1ZWtqdVJ6UGd4T3M4b052Vjk0OHQyV2RGRlIvOUU4emdUQkRsYUQzZ2JD?=
 =?utf-8?B?YXpJelJaR2EzTkJRT3c3aEVjQk0rMGhOV2F4ZUE4Nk9GZ2xzK1RJcjhxSUlp?=
 =?utf-8?B?Q0lvZGxjbVF0WXF4TkEzcW5ieDl5TndJaWV2bzZ6SGFndld0Z21DVndUS1ZI?=
 =?utf-8?B?ZVFmYnNDUTRHRlFqc3F6eVpzNzViNTYrcllTVUxjSXh4SzFQNkhQUT09?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d12767a6-08da-4cfb-d88c-08da547cf729
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jun 2022 18:28:12.7145
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: TGX3J4O+3vj4SI6/rmbE+JUiwXGUu0jkkzDSVUaqJm+FoMq6+bNQYZnvf/01mYbhJBhc64qwLiVZ763PfEg3Uw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3596
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883
 definitions=2022-06-22_05:2022-06-22,2022-06-22 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0
 mlxlogscore=999 suspectscore=0 adultscore=0 phishscore=0 bulkscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2204290000 definitions=main-2206220087
X-Proofpoint-ORIG-GUID: p6Upu_qUGzSpO9U9h7FI_2OCaj-yyDR7
X-Proofpoint-GUID: p6Upu_qUGzSpO9U9h7FI_2OCaj-yyDR7
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 6/22/22 11:16 AM, Chuck Lever III wrote:
>
>> On Jun 22, 2022, at 2:15 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>>
>> Currently the idle timeout for courtesy client is fixed at 1 day.
>> If there are lots of courtesy clients remain in the system it can
>> cause memory resource shortage that effects the operations of other
>> modules in the kernel. This problem can be observed by running pynfs
>> nfs4.0 CID5 test in a loop. Eventually system runs out of memory
>> and rpc.gssd fails to add new watch:
>>
>> rpc.gssd[3851]: ERROR: inotify_add_watch failed for nfsd4_cb/clnt6c2e:
>> 		No space left on device
>>
>> and also alloc_inode fails with out of memory:
>>
>> Call Trace:
>> <TASK>
>>         dump_stack_lvl+0x33/0x42
>>         dump_header+0x4a/0x1ed
>>         oom_kill_process+0x80/0x10d
>>         out_of_memory+0x237/0x25f
>>         __alloc_pages_slowpath.constprop.0+0x617/0x7b6
>>         __alloc_pages+0x132/0x1e3
>>         alloc_slab_page+0x15/0x33
>>         allocate_slab+0x78/0x1ab
>>         ? alloc_inode+0x38/0x8d
>>         ___slab_alloc+0x2af/0x373
>>         ? alloc_inode+0x38/0x8d
>>         ? slab_pre_alloc_hook.constprop.0+0x9f/0x158
>>         ? alloc_inode+0x38/0x8d
>>         __slab_alloc.constprop.0+0x1c/0x24
>>         kmem_cache_alloc_lru+0x8c/0x142
>>         alloc_inode+0x38/0x8d
>>         iget_locked+0x60/0x126
>>         kernfs_get_inode+0x18/0x105
>>         kernfs_iop_lookup+0x6d/0xbc
>>         __lookup_slow+0xb7/0xf9
>>         lookup_slow+0x3a/0x52
>>         walk_component+0x90/0x100
>>         ? inode_permission+0x87/0x128
>>         link_path_walk.part.0.constprop.0+0x266/0x2ea
>>         ? path_init+0x101/0x2f2
>>         path_lookupat+0x4c/0xfa
>>         filename_lookup+0x63/0xd7
>>         ? getname_flags+0x32/0x17a
>>         ? kmem_cache_alloc+0x11f/0x144
>>         ? getname_flags+0x16c/0x17a
>>         user_path_at_empty+0x37/0x4b
>>         do_readlinkat+0x61/0x102
>>         __x64_sys_readlinkat+0x18/0x1b
>>         do_syscall_64+0x57/0x72
>>         entry_SYSCALL_64_after_hwframe+0x46/0xb0
>>         RIP: 0033:0x7fce5410340e
>>
>> This patch adds a simple policy to dynamically adjust the idle
>> timeout based on the percentage of available memory in the system
>> as follow:
>>
>> . > 70%    : unlimited. Courtesy clients are allowed to remain valid
>>              as long as memory availability is above 70%
>> . 60% - 70%:  1 day.
>> . 50% - 60%:  1hr
>> . 40% - 50%:  30mins
>> . 30% - 40%:  15mins
>> . < 30%:      disable. Expire all existing courtesy clients and donot
>>               allow new courtesey client
> I thought our plan was to add a shrinker to do this.

I'm not familiar with kernel's memory allocation and don't want to muck
with it so I start with this simple approach but I'm open for any suggestion
on how to add a shrinker for this task. Is there any existing model that I
can use as reference?

Thanks,
-Dai

>
>
>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>> ---
>> fs/nfsd/nfs4state.c | 41 +++++++++++++++++++++++++++++++++++++++--
>> fs/nfsd/nfsd.h      |  5 ++++-
>> 2 files changed, 43 insertions(+), 3 deletions(-)
>>
>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>> index 9409a0dc1b76..a7feea9d07cf 100644
>> --- a/fs/nfsd/nfs4state.c
>> +++ b/fs/nfsd/nfs4state.c
>> @@ -5788,12 +5788,47 @@ nfs4_anylock_blockers(struct nfs4_client *clp)
>> 	return false;
>> }
>>
>> +static bool
>> +nfs4_allow_courtesy_client(struct nfsd_net *nn, unsigned int *idle_timeout)
>> +{
>> +	unsigned long avail;
>> +	bool ret = true;
>> +	unsigned int courtesy_expire = 0;
>> +	struct sysinfo si;
>> +
>> +	si_meminfo(&si);
>> +	avail = (si.freeram * 10) / (si.totalram - si.totalhigh);
>> +	switch (avail) {
>> +	case 7: case 8: case 9: case 10:
>> +		courtesy_expire = 0;		/* unlimit */
>> +		break;
>> +	case 6:
>> +		courtesy_expire = NFSD_COURTESY_CLIENT_TO_1DAY;
>> +		break;
>> +	case 5:
>> +		courtesy_expire = NFSD_COURTESY_CLIENT_TO_1HR;
>> +		break;
>> +	case 4:
>> +		courtesy_expire = NFSD_COURTESY_CLIENT_TO_30MINS;
>> +		break;
>> +	case 3:
>> +		courtesy_expire = NFSD_COURTESY_CLIENT_TO_15MINS;
>> +		break;
>> +	default:
>> +		ret = false;			/* disallow CC */
>> +	}
>> +	*idle_timeout = courtesy_expire;
>> +	return ret;
>> +}
>> +
>> static void
>> nfs4_get_client_reaplist(struct nfsd_net *nn, struct list_head *reaplist,
>> 				struct laundry_time *lt)
>> {
>> 	struct list_head *pos, *next;
>> 	struct nfs4_client *clp;
>> +	unsigned int exptime;
>> +	bool allow_cc = nfs4_allow_courtesy_client(nn, &exptime);
>>
>> 	INIT_LIST_HEAD(reaplist);
>> 	spin_lock(&nn->client_lock);
>> @@ -5803,11 +5838,13 @@ nfs4_get_client_reaplist(struct nfsd_net *nn, struct list_head *reaplist,
>> 			goto exp_client;
>> 		if (!state_expired(lt, clp->cl_time))
>> 			break;
>> +		if (!allow_cc)
>> +			goto exp_client;
>> 		if (!atomic_read(&clp->cl_rpc_users))
>> 			clp->cl_state = NFSD4_COURTESY;
>> 		if (!client_has_state(clp) ||
>> -				ktime_get_boottime_seconds() >=
>> -				(clp->cl_time + NFSD_COURTESY_CLIENT_TIMEOUT))
>> +				(exptime && ktime_get_boottime_seconds() >=
>> +				(clp->cl_time + exptime)))
>> 			goto exp_client;
>> 		if (nfs4_anylock_blockers(clp)) {
>> exp_client:
>> diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h
>> index 847b482155ae..9d4a5708f852 100644
>> --- a/fs/nfsd/nfsd.h
>> +++ b/fs/nfsd/nfsd.h
>> @@ -340,7 +340,10 @@ void		nfsd_lockd_shutdown(void);
>> #define COMPOUND_ERR_SLACK_SPACE	16     /* OP_SETATTR */
>>
>> #define NFSD_LAUNDROMAT_MINTIMEOUT      1   /* seconds */
>> -#define	NFSD_COURTESY_CLIENT_TIMEOUT	(24 * 60 * 60)	/* seconds */
>> +#define	NFSD_COURTESY_CLIENT_TO_1DAY	(24 * 60 * 60)	/* seconds */
>> +#define	NFSD_COURTESY_CLIENT_TO_1HR	(60 * 60)
>> +#define	NFSD_COURTESY_CLIENT_TO_30MINS	(30 * 60)
>> +#define	NFSD_COURTESY_CLIENT_TO_15MINS	(15 * 60)
>>
>> /*
>>   * The following attributes are currently not supported by the NFSv4 server:
>> -- 
>> 2.9.5
>>
> --
> Chuck Lever
>
>
>