Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Dai Ngo <dai.ngo@oracle.com>
CC:     Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v4 2/2] NFSD: add shrinker to reap courtesy clients on low
 memory condition
Thread-Topic: [PATCH v4 2/2] NFSD: add shrinker to reap courtesy clients on
 low memory condition
Thread-Index: AQHYvLpF3IXRGSf70UK3afu6RNBIZ63JEv+AgAJR5wCAACuHAIAAzI6AgAAUu4A=
Date:   Fri, 2 Sep 2022 17:58:19 +0000
Message-ID: <3CD64E7F-8E81-4B37-AAF3-499B47B25D19@oracle.com>
References: <1661896113-8013-1-git-send-email-dai.ngo@oracle.com>
 <1661896113-8013-3-git-send-email-dai.ngo@oracle.com>
 <FA83E721-C874-4A47-87BA-54B13E0B12A3@oracle.com>
 <2df6f1fe-c8eb-d5a1-0a11-2fd965555a33@oracle.com>
 <7041D47D-ECB3-497E-9174-96E9E36FFBDE@oracle.com>
 <eb197dde-8758-7ef4-8a7b-989273e09abc@oracle.com>
In-Reply-To: <eb197dde-8758-7ef4-8a7b-989273e09abc@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?FOengagJ4y3Zx39YLhhRzNaPnh6qQ7xh68sJr4Txy2YaC3uvbdzknGFESuVl?=
 =?us-ascii?Q?dL8WdT/N95BEsMktNdyTmHwE8hRVKhT2YjkfSBjCAsgTn/AyoSwf7PP29YMS?=
 =?us-ascii?Q?mRfC+UGC/B7hSnLdvxPicNQcSeSoExbZoFY4oZKK6SdjlcOSPtZy4Cxot0QP?=
 =?us-ascii?Q?j+FBRy3oSjNQO/Y9pRSAFuxAgDA+8Pd8TWO5pQnM5lCfOoNRmRZMSc2uVOyi?=
 =?us-ascii?Q?X5jrqWyRaYmmPqR0mAN932pFJbXH7yrDc2gH+3zUP6uIC+YJQ9n8DXZ8r37v?=
 =?us-ascii?Q?uJl+SbaidMMg3gXlPhbRGGGe3bJM0+a5QyGF0OHN3Lqp7MX57O1mi/kLwsFg?=
 =?us-ascii?Q?RICH2XkAc0GpZuTJ2EixonioMKMEUoTZZ6GeiP5VMT/y2VvtXiwdO9WlzWtk?=
 =?us-ascii?Q?6lLjO/JLKCiuzk70tE7z7qyMoo5rwuRcgFEUvJLhJkOFb2ietr9mRp3QATd4?=
 =?us-ascii?Q?R9ZB48a++rWXKg7xtxBnCC26jj3RfUNhLAfCyJJaR+s0ty7SebSqsYjw5hxK?=
 =?us-ascii?Q?eiwKiBpBs2kbVdX5fxqxYiIkiay6fvVdJQm7YYN1MnZJr/OX93pCpyzd+NG3?=
 =?us-ascii?Q?d7K0tbUYzZbSrOtU/sOAJnZLEEsTRIkqlrGL6uU9RcbMU7Xc9QD90jxoTVXH?=
 =?us-ascii?Q?9KH2fP+QJwRMI3OAZu+uy/EaONgauqjxqy/lXmNCvB2eLoDecp9JcXPsNd+Z?=
 =?us-ascii?Q?HRvYD66U99l2MuBT/lB1gKtyO/4yWcUKNsddNNRZRLjEBTqTdqxG8Gfk3jDK?=
 =?us-ascii?Q?0UJzvxrmBHA8z70uAJiP39Hu6tYsN7eVmDK8roBK+x0PBdn8nlrWIVhOpP6q?=
 =?us-ascii?Q?FKe+0UZDqMtmmoNYrFnPHn+N4Vs/egXf7s/Vge5YwFgKQPz8yt14toH0lKGu?=
 =?us-ascii?Q?anO7wVIrdT3Lod1CLFh+SeBLdn6j4NOAhH/N5xXLIiNyutoANQ6AFrOVY5y0?=
 =?us-ascii?Q?tEYM0iUEyrtf3bp7MnYtBpQrgEohDl4dEnI/+lbUWMMBpD3JWZquTtIxy2Xx?=
 =?us-ascii?Q?xGhI1KkPj/OVmd9hjkded4yCgM7i4epNmANcp1e1fxHsVN90B6aTJWSeqGVL?=
 =?us-ascii?Q?Vx5IYBlmo1pqL7ep2jhwwqithmyj/7B3BUkA7BdieCxnJdVpxN1K3iO3v8wd?=
 =?us-ascii?Q?r3QUDAcCQ2SuxKHDBenL6FqnMEstsI11jHz3aDLzhi2WYVvAT4kLdE0CH1lv?=
 =?us-ascii?Q?Anu+o55+dMBairV5fz+u9BiuzhtxveD/Rd7WK/mDWkjMzmMFSucpBgYDjt84?=
 =?us-ascii?Q?OYjcTntpfPbA+nxy2bkGY8BEuol3QvtJnGSNpNXIuoVy1FL7hjv9Jp5WSeCT?=
 =?us-ascii?Q?SOd4Ap37rSppmuq8dtX802r4KaKvHINEpwWLxNgLyt/L8dtzpG3dMlbdAI2t?=
 =?us-ascii?Q?lM3zFPxsOEcpTptANZwLl3Ajw+0xL7eVp3did4Z3l3O8jCNggD8Qh0/evA1W?=
 =?us-ascii?Q?5bYjPIjTA60Sow+Xpp/wy+h/nJbZuYq5Lf9JlnbL1A+aQrFibQHYrZ7EWVR0?=
 =?us-ascii?Q?acKE80h0ARJO3yF4DKTAxpmTzAUEpfVzxDRqKf2nQGO539xUIKyg5o9630RP?=
 =?us-ascii?Q?kPGLJTCey/jEx6MFK+sA0/r4JHTw4f43BMdotYUz/WdpgPliKqDD7C99FjTo?=
 =?us-ascii?Q?SQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <EDA399BACEB635458DAE98B16E7B76A9@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e0d73167-3195-4d0c-12aa-08da8d0cb833
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Sep 2022 17:58:19.5669
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Df1ELpcY12iDnHU5vI86GaLmG2zNyrAnpiwaFkEupSkMTVjjqN4gf7aGri+IEKpF8qlQhJ6TRxCl3rOjaMcaeA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB4960
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-09-02_04,2022-08-31_03,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999
 suspectscore=0 adultscore=0 bulkscore=0 mlxscore=0 spamscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2207270000 definitions=main-2209020082
X-Proofpoint-GUID: ZmUvEqihzo0bqZ9weO2WBpVZ_HVa9Qex
X-Proofpoint-ORIG-GUID: ZmUvEqihzo0bqZ9weO2WBpVZ_HVa9Qex
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Sep 2, 2022, at 12:44 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>=20
> On 9/1/22 9:32 PM, Chuck Lever III wrote:
>>=20
>>> On Sep 1, 2022, at 9:56 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>>>=20
>>> Hi Chuck,
>>>=20
>>> On 8/31/22 7:30 AM, Chuck Lever III wrote:
>>>>> 	struct list_head *pos, *next;
>>>>> 	struct nfs4_client *clp;
>>>>>=20
>>>>> -	maxreap =3D (atomic_read(&nn->nfs4_client_count) >=3D nn->nfs4_max_=
clients) ?
>>>>> -			NFSD_CLIENT_MAX_TRIM_PER_RUN : 0;
>>>>> +	cb_cnt =3D atomic_read(&nn->nfsd_client_shrinker_cb_count);
>>>>> +	if (atomic_read(&nn->nfs4_client_count) >=3D nn->nfs4_max_clients |=
|
>>>>> +							cb_cnt) {
>>>>> +		maxreap =3D NFSD_CLIENT_MAX_TRIM_PER_RUN;
>>>>> +		atomic_set(&nn->nfsd_client_shrinker_cb_count, 0);
>>>>> +	}
>>>> I'm not terribly happy with this, but I don't have a better suggestion
>>>> at the moment. Let me think about it.
>>> Do you have any suggestion to improve this, I want to incorporate it
>>> before sending out v5?
>> Let's consider some broad outlines...
>>=20
>> With regards to parametrizing the reaplist behavior, you want
>> a normal laundromat run to reap zero or more courtesy clients.
>> You want a shrinker laundromat run to reap more than zero. I
>> think you want a minreap variable as well as a maxreap variable
>> in there to control how the reaplist is built. Making @minreap
>> a function parameter rather than a global atomic would be a
>> plus for me, but maybe that's not practical.
>=20
> I'm not quite understand how the minreap is used, I think it
> will make the code more complex.
>=20
>>=20
>> But I would prefer a more straightforward approach overall. The
>> proposed approach seems tricky and brittle, and needs a lot of
>> explaining to understand. Other subsystems seem to get away with
>> something simpler.
>>=20
>> Can nfsd_courtesy_client_count() simply reduce
>> nn->nfs4_max_clients, kick the laundromat, and then return 0?
>> Then get rid of nfsd_courtesy_client_scan().
>=20
> I need to think more about this approach. However at first glance,
> nn->nfs4_max_clients is used to control how many clients, including
> active and courtesy clients, are allowed in the system. If we lower
> this count, it also prevent new clients from connecting to the
> system. So now the shrinker mechanism does more than just getting
> rid of unused resources, maybe that's ok?

I don't want to go down the path of kicking out active clients if
we don't need to. Maybe you can make the laundromat code more
careful to make a distinction between active and courtesy clients.

It might sense in the long run to separate the laundromat's
processing of courtesy and active clients anyway.


>> Or, nfsd_courtesy_client_count() could return
>> nfsd_couresy_client_count. nfsd_courtesy_client_scan() could
>> then look something like this:
>>=20
>> 	if ((sc->gfp_mask & GFP_KERNEL) !=3D GFP_KERNEL)
>> 		return SHRINK_STOP;
>>=20
>> 	nfsd_get_client_reaplist(nn, reaplist, lt);
>> 	list_for_each_safe(pos, next, &reaplist) {
>> 		clp =3D list_entry(pos, struct nfs4_client, cl_lru);
>> 		trace_nfsd_clid_purged(&clp->cl_clientid);
>> 		list_del_init(&clp->cl_lru);
>> 		expire_client(clp);
>> 		count++;
>> 	}
>> 	return count;
>=20
> This does not work, we cannot expire clients on the context of
> scan callback due to deadlock problem.

Correct, we don't want to start shrinker laundromat activity if
the allocation request specified that it cannot wait. But are
you sure it doesn't work if sc_gfp_flags is set to GFP_KERNEL?


> I will experiment with ways to get rid of the scan function to
> make the logic simpler.


--
Chuck Lever