Received-SPF: softfail (google.com: domain of transitioning linux-nfs-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Wang Yugui <wangyugui@e16-tech.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v2] nfsd: serialize filecache garbage collector
Thread-Topic: [PATCH v2] nfsd: serialize filecache garbage collector
Thread-Index: AQHYdNvpgilm5hKNx0yhsp1S8r+H8605BxOA
Date:   Tue, 31 May 2022 14:12:03 +0000
Message-ID: <A12757CE-17B8-4F21-9EC9-3CA0496A8B99@oracle.com>
References: <20220531103427.47769-1-wangyugui@e16-tech.com>
In-Reply-To: <20220531103427.47769-1-wangyugui@e16-tech.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?nYD27oJQf7SHjAJ/qjbB/WyQxHfGWfTjt6zpneAEuxpzbEXOEVjCmsqr5gEl?=
 =?us-ascii?Q?kA4dYX4nvDBsiEbRASJoV2h80mbj1n6cc/otwGbcyaBPKiTVl+XHuyCcL0QK?=
 =?us-ascii?Q?h2xt05QZr2j91Wlcc4LV8AcAd+yNfgD1HuBqMUYwdf9U0QU2zZRdVi9L19cE?=
 =?us-ascii?Q?8/vkwsFxgiMgegA2cTH8Ke3UDxQisfi8WFg/rO5Ob4PBULFL3i+QlAIpPzwa?=
 =?us-ascii?Q?IGctal/t0oaydMnXm3NaJvqbPUe0X43YqHicLREK/0xQxysL/jRQ2Mq/InJy?=
 =?us-ascii?Q?IhRU8fnY0AkZeDmCPfnO+2edPIvFJUUDBeE8llbJ9h3hzn8+AggMl767B7D1?=
 =?us-ascii?Q?mW0oJGw2wTQAY2IBX/mWhup5SqPtG9EpvOZanS/QoKn+ZAdlsQp0kLXjOY/z?=
 =?us-ascii?Q?7rOLhxDQ4k/OtPBk5JjF8mCdwetgdN807r3QNz3kOzAxvg25t2/ISAxiWQPE?=
 =?us-ascii?Q?8MpbNUed5rDPNqhXpfALhlTUTNb2gPoyi1uapH2UyRAbUfIfiDoes5l4IGc8?=
 =?us-ascii?Q?KG/0mJDtZN4bCmh89CI78/RgNTXyLmM6JaX0HnuDBrRN00ZUs+YriYJxsCS+?=
 =?us-ascii?Q?9mIJG1QBOoxeI/qEnnRyZWWTdR07Rxkl7ZANSF2HZJREEENEcuuVWOUg0sGv?=
 =?us-ascii?Q?dv8KvGUowVKtyguTi1DbrMM/6arWfwDcQLzyGSVjumCmM0u62VoFNFW4H+eT?=
 =?us-ascii?Q?PyIn/XW+jeL05iSB+GWgls8dSRD2QpmOzRHyC8bD7Vdp6czA2gHynkcEnDwm?=
 =?us-ascii?Q?p53DvqMoJtAYaJXdlDvuV+ZjpBtjejY3Zi4tlAl6ty7YOCC3/MKLfGMadhkF?=
 =?us-ascii?Q?48UUUNt5LL3H446BAJlmUHHynYp6jWjrfvk66SS59P49iKs3Kh6x5wZXSkBH?=
 =?us-ascii?Q?cDENGyVr2VbKkS9Pqdu7uuqnBfIOmQCcv3/LrCo95TcvRVLM11ZiXElmvH2P?=
 =?us-ascii?Q?S8XvOg3ksgDNfRIJUlUvQ8pG+rCF6csGBeafJkASKXt6mimtxGynAtj2ttv6?=
 =?us-ascii?Q?iltcdwDOMRlNlM3F5l8VB0ZBaS03zUdNa4mn3KbRLxAYwzZWJkofj83jRR5C?=
 =?us-ascii?Q?9ZhQ+0jsaDMDQaoZkAjxV2E4GjDqsNPdVFHisAwSuE8tfoBr6olKe0roaBJy?=
 =?us-ascii?Q?CzPjtacpo/6SxElHhW0jwdM0AS33TC+/xKU1jDdiVPc0pqghXUrDeeEZ/GFO?=
 =?us-ascii?Q?PTadHET7ixdOTOQmTS/X0VxsN16xJ+mMd9SiiN6psuiSuTvLjGbqCar3rCzJ?=
 =?us-ascii?Q?J2sTIPaPUogMHk06ZVvpp90ckc49i2LjPaEi6lG80wAa+sml07ZUGkQwHZwx?=
 =?us-ascii?Q?Ge3v+RUc2/Q3VCUcPLklTzGwiGliqvW7/0YBiswLEv8mBjbAIiZjOwKW20wg?=
 =?us-ascii?Q?KskYc8fRVLZlh5XUb/37hooiHn93w1FPqCWLL91AKItwkkCy3lDWLZ6rNYur?=
 =?us-ascii?Q?7f6No2F54Nm+armiWa4WEc+XwaVanI0ss/N8QtaNxwVh0TKMCMV2zxUcgDHC?=
 =?us-ascii?Q?j9nclFaD5r3LR6R6H1lkhwJYcrZMwz8ERQBFmxmeeGzMPwrwepqFRtuBdHzd?=
 =?us-ascii?Q?olZ3tyFPUQ6PGTCW6BXpmn/Gl22TBProxQ8RwYGzsE+ay9ZtnYK4LmxAC+HE?=
 =?us-ascii?Q?IXw87+1BzmlIo5aJoyZ1sxLVQYowaynY+SdB0JoQhOFY1136ozlfMXTIF+rV?=
 =?us-ascii?Q?oPz0Jy8FtdmQ7H0+4kQLmlmsX+BdE8lVDybTa0uiWf5TgXKt8fe40FJuCM9D?=
 =?us-ascii?Q?mrhyHj69r943wP9P2gK3q7n43dOKzGI=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4144C02DA9C5644885870B4C0624FBFD@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e3bc2830-c804-4efe-6fed-08da430f8925
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 May 2022 14:12:03.0692
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: oqQfG5FQDcPoTJD2T5m3ADYC5+OanOaXmEEEjaoUnuQ/jxuXUwU6//cdqdE0fMsadkIlxpHQa+2VIly4zrQXhg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB5969
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874
 definitions=2022-05-31_06:2022-05-30,2022-05-31 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999
 phishscore=0 suspectscore=0 malwarescore=0 mlxscore=0 adultscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2204290000 definitions=main-2205310073
X-Proofpoint-ORIG-GUID: -0I6sFJJrLN8PF_8KymL_l4TZ_efCMiA
X-Proofpoint-GUID: -0I6sFJJrLN8PF_8KymL_l4TZ_efCMiA
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On May 31, 2022, at 6:34 AM, Wang Yugui <wangyugui@e16-tech.com> wrote:
>=20
> When many(>NFSD_FILE_LRU_THRESHOLD) files are kept as OPEN, such as
> xfstests generic/531, nfsd proceses are in CPU high-load state,
> and nfsd_file_gc(nfsd filecache garbage collector) waste many CPU times.

Over the past few days, I've been able to reproduce a lot of bad
behavior with generic/531. My test client has 12 physical CPU
cores, and my lab network is 56Gb InfiniBand.

Unfortunately this patch doesn't really begin to address it. For
example, with this patch applied, CPU idle is in single digits
on the NFS server that exports the test's scratch device, and
that server can still get into a soft lock-up. IMO that is
because this change works around the underlying problem but
makes no attempt to root-cause or address that issue.

I agree that the NFS server's behavior needs attention, but I'm
not inclined to apply this particular patch as it is.


> concurrency nfsd_file_gc() is almost meaningless, so serialize it.
>=20
> Signed-off-by: Wang Yugui <wangyugui@e16-tech.com>
> ---
> Changes since v1:
> - add static to 'atomic_t nfsd_file_gc_running'.
>  thanks for kernel test robot <lkp@intel.com>
>=20
> fs/nfsd/filecache.c | 7 ++++++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>=20
> diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c
> index f172412447f5..28a8f8d6d235 100644
> --- a/fs/nfsd/filecache.c
> +++ b/fs/nfsd/filecache.c
> @@ -471,10 +471,15 @@ nfsd_file_lru_walk_list(struct shrink_control *sc)
> 	return ret;
> }
>=20
> +/* concurrency nfsd_file_gc() is almost meaningless, so serialize it. */
> +static atomic_t nfsd_file_gc_running =3D ATOMIC_INIT(0);
> static void
> nfsd_file_gc(void)
> {
> -	nfsd_file_lru_walk_list(NULL);
> +	if(atomic_cmpxchg(&nfsd_file_gc_running, 0, 1) =3D=3D 0) {
> +		nfsd_file_lru_walk_list(NULL);
> +		atomic_set(&nfsd_file_gc_running, 0);
> +	}
> }
>=20
> static void
> --=20
> 2.36.1
>=20

--
Chuck Lever