Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Frank van der Linden <fvdl@google.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        netdev <netdev@vger.kernel.org>,
        Dave Chinner <david@fromorbit.com>,
        "tgraf@suug.ch" <tgraf@suug.ch>, Jeff Layton <jlayton@redhat.com>
Subject: Re: [PATCH RFC 00/30] Overhaul NFSD filecache
Thread-Topic: [PATCH RFC 00/30] Overhaul NFSD filecache
Thread-Index: AQHYhkIqDsKitGhdzEmSu19Gpo96sa1dcq8AgAex3YA=
Date:   Tue, 28 Jun 2022 17:57:30 +0000
Message-ID: <A98597CB-F422-4FE4-9A08-6EABAAE8C649@oracle.com>
References: <165590626293.75778.9843437418112335153.stgit@manet.1015granger.net>
 <CAPTztWayDY7ejHaQNcCr6f3iRS8B1ytMqk0iYSUpHngp7OV-FQ@mail.gmail.com>
In-Reply-To: <CAPTztWayDY7ejHaQNcCr6f3iRS8B1ytMqk0iYSUpHngp7OV-FQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?N/HeN8EXBrMb8nd3bcOoid+pCvhyi/fkRGIKoFweatVmcdZm+TXTqVNjgL+R?=
 =?us-ascii?Q?FAtu6o6+xHeJYaQTre+5SgFt0R7EL6xngWcDuMTrxm4x9L0XBEqMH3IqtcSl?=
 =?us-ascii?Q?+Cs+O/ngDO10AAQPxvYYM44zBHQfAGBq/Pc7FMJzZgwWObFVIzy8J6iA71OC?=
 =?us-ascii?Q?GoI/10dfFry066uKkftG6lYPVheWouNce+WzPtL2uXH2mjPESEDiTUIEMjZh?=
 =?us-ascii?Q?KalGC1U26RKdRyMB6b6FpVT7OlRQTv3XEjMO+k+I8jjJiUb1JJ3HEkbayYtA?=
 =?us-ascii?Q?6b9oBkVMR54aAdQv1f+ij+dAqUvjbUW+lX2RttDFdLwvQbwmBxJicRwcZDYA?=
 =?us-ascii?Q?BHAMO1r1VfZCYbzA2elkbbVfKnFenXWo6+gC2bi722l6NXiIh/wnjUM6zb5D?=
 =?us-ascii?Q?gmLqu/z81cDBaYgPetIuGc7ug+3jK5tHA6Sb4KC+QqDb6eV3dkDiTDskzd5/?=
 =?us-ascii?Q?6UWgH0TO+NQnPZEXYQDJEYf7TqYTedpTxrLo4D7gZRd2OEF/TuREYaG3WnjA?=
 =?us-ascii?Q?Ab3Gcki2pct8fRA9J2UaHLz9w392i5ZBoryhLhAKiASFZw24SuKkTA+UnTV5?=
 =?us-ascii?Q?xJ6A5DYRG6O9Cz/6J+puLHhkZ9vM5UCX18cAFiBB2DC8zF3FcriWneS1tgWT?=
 =?us-ascii?Q?XlktuNaJFnFA5K7dX5mVtGZbf1Tczwj72ctSQRq4FNy5sPuO5uCALpx1syB0?=
 =?us-ascii?Q?dEv8XqRhFMgpr4PGGvwas3950jG9De0NUFIPB4xSApKsexyVZr8lovtbV6Bu?=
 =?us-ascii?Q?jy2l9njQy4oRa1evohiZBht5nOuhAXJsR1ZIkHzTxwbL/cgynuQo60ENXObp?=
 =?us-ascii?Q?JhOSn/EY/jGFs8WlIswwmdhfW2Rz41iGl6+cImoUXXlS6DCAtQnoRTvjcTS2?=
 =?us-ascii?Q?SZboL3Y9EoAMf4sixnczG38b91UxPl2Kxl5n+TDaMHdKoBtPHBLxSbT8OjxW?=
 =?us-ascii?Q?LPNxpS/pysd1K+EvtaHPtv9Tx6qSWJrR7ZJ9RdSR2FrnPBdixMCfnH5idpJ/?=
 =?us-ascii?Q?royun7egE62TGoIZE/cOHLgI35RmrtqTVolHErCowyEyHm+3YgkIqT7wwHkP?=
 =?us-ascii?Q?eUHfcBmmZJTV05oxRXJSXoYfVpqzMv+uDIRHhzv5VJYpMkPk8jf70pOoUY3F?=
 =?us-ascii?Q?l42rVeNX/VQhbjSGH77cJznWiOXJ6yGteU0RA7ZXKmNZM+GfDjqVcuNTAZuB?=
 =?us-ascii?Q?cM1FuqekPP79UnQ4ffsqvroIMjb8O5Vx8TIbSFK2+bJ0+dhPxRdbWLr0n36l?=
 =?us-ascii?Q?U1dTI9IowbWWRV2mqdbBdvvhf0Xus08lCioLGJvNsYSaLgxej4WBFKrioGvw?=
 =?us-ascii?Q?xOSOi8tOIEkqWQ6lSfgwa4jCwncKByueKvLDWT2/gH5IpeKD0HzaE+RDRqVI?=
 =?us-ascii?Q?W6gO/8vazo3QQyVb+eU/E6cH8jEy+7mNTg9aPsVVsDHHaVGbvBFBPghFWWU4?=
 =?us-ascii?Q?6C04TGi2bQOPif08U1EX7i/3OpR8FOY+mwTvruvkN1wi4YwI6gYTJevacLNj?=
 =?us-ascii?Q?/AjzxW7pNY3nTQE6bsaVGthxj2HgfPvZao6tabN5JkzkRFN/q+yjKkQZKSJ9?=
 =?us-ascii?Q?WhmaLHUQDcwd5wtlHCI2/sQGLcIlKVBvuwoX1oFlLNhB2/4za/vNyZ1XbDX0?=
 =?us-ascii?Q?Sg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <35B175EE833B57469E65C3A8CF112C17@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d24dcebc-6ca2-4a75-a06f-08da592fab8d
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jun 2022 17:57:30.2398
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 3KWoI/eqU94RMkRpGTcr+JSW0PQCcf1Z1yWfp10gglKx+TYcvLL0QMS6OJV7HMfdMuz9qMj5goVfscUpvVNJOw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB4971
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883
 definitions=2022-06-28_11:2022-06-28,2022-06-28 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 phishscore=0
 malwarescore=0 mlxlogscore=908 spamscore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000
 definitions=main-2206280071
X-Proofpoint-ORIG-GUID: SsjRr_UIKhhUtcUnznL-peK9rYXzGeKV
X-Proofpoint-GUID: SsjRr_UIKhhUtcUnznL-peK9rYXzGeKV
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 23, 2022, at 4:27 PM, Frank van der Linden <fvdl@google.com> wrote=
:
>=20
> On Wed, Jun 22, 2022 at 7:12 AM Chuck Lever <chuck.lever@oracle.com> wrot=
e:
>>=20
>> This series overhauls the NFSD filecache, a cache of server-side
>> "struct file" objects recently used by NFS clients. The purposes of
>> this overhaul are an immediate improvement in cache scalability in
>> the number of open files, and preparation for further improvements.
>>=20
>> There are three categories of patches in this series:
>>=20
>> 1. Add observability of cache operation so we can see what we're
>> doing as changes are made to the code.
>>=20
>> 2. Improve the scalability of filecache garbage collection,
>> addressing several bugs along the way.
>>=20
>> 3. Improve the scalability of the filecache hash table by converting
>> it to use rhashtable.
>>=20
>> The series as it stands survives typical test workloads. Running
>> stress-tests like generic/531 is the next step.
>>=20
>> These patches are also available in the linux-nfs-bugzilla-386
>> branch of
>>=20
>>  https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
>>=20
>> ---
>>=20
>> Chuck Lever (30):
>>      NFSD: Report filecache LRU size
>>      NFSD: Report count of calls to nfsd_file_acquire()
>>      NFSD: Report count of freed filecache items
>>      NFSD: Report average age of filecache items
>>      NFSD: Add nfsd_file_lru_dispose_list() helper
>>      NFSD: Refactor nfsd_file_gc()
>>      NFSD: Refactor nfsd_file_lru_scan()
>>      NFSD: Report the number of items evicted by the LRU walk
>>      NFSD: Record number of flush calls
>>      NFSD: Report filecache item construction failures
>>      NFSD: Zero counters when the filecache is re-initialized
>>      NFSD: Hook up the filecache stat file
>>      NFSD: WARN when freeing an item still linked via nf_lru
>>      NFSD: Trace filecache LRU activity
>>      NFSD: Leave open files out of the filecache LRU
>>      NFSD: Fix the filecache LRU shrinker
>>      NFSD: Never call nfsd_file_gc() in foreground paths
>>      NFSD: No longer record nf_hashval in the trace log
>>      NFSD: Remove lockdep assertion from unhash_and_release_locked()
>>      NFSD: nfsd_file_unhash can compute hashval from nf->nf_inode
>>      NFSD: Refactor __nfsd_file_close_inode()
>>      NFSD: nfsd_file_hash_remove can compute hashval
>>      NFSD: Remove nfsd_file::nf_hashval
>>      NFSD: Remove stale comment from nfsd_file_acquire()
>>      NFSD: Clean up "open file" case in nfsd_file_acquire()
>>      NFSD: Document nfsd_file_cache_purge() API contract
>>      NFSD: Replace the "init once" mechanism
>>      NFSD: Set up an rhashtable for the filecache
>>      NFSD: Convert the filecache to use rhashtable
>>      NFSD: Clean up unusued code after rhashtable conversion
>>=20
>>=20
>> fs/nfsd/filecache.c | 677 +++++++++++++++++++++++++++-----------------
>> fs/nfsd/filecache.h |   6 +-
>> fs/nfsd/nfsctl.c    |  10 +
>> fs/nfsd/trace.h     | 117 ++++++--
>> 4 files changed, 522 insertions(+), 288 deletions(-)
>>=20
>> --
>> Chuck Lever
>>=20
>=20
> Yep, looks good so far, thanks for doing this. Somewhat similar to my (bu=
ggy)
> attempt at fixing it that I sent at the time (don't put open files on
> the LRU, and
> use rhashtable), but cleaner and, presumably, less buggy :)
>=20
> Can't test it right now, but it seems like Wang already confirmed that it=
 works.

Frank, thanks to you and Wang for reporting this issue, and sorry for
taking so long to address it.


--
Chuck Lever