Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Jeff Layton <jlayton@kernel.org>
CC:     Neil Brown <neilb@suse.de>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v5 3/5] nfsd: rework refcounting in filecache
Thread-Topic: [PATCH v5 3/5] nfsd: rework refcounting in filecache
Thread-Index: AQHY7gDKf+UJMCSoA0e+WwOAy53hT64qUesAgAAKjQCAAA8pgIAABGKA
Date:   Tue, 1 Nov 2022 19:13:23 +0000
Message-ID: <0B609D3B-955F-4F15-9EDD-16B98087EF06@oracle.com>
References: <20221101144647.136696-1-jlayton@kernel.org>
 <20221101144647.136696-4-jlayton@kernel.org>
 <42F8EAC1-7383-4B98-BAD1-D676950718E0@oracle.com>
 <17bf0292cbd63b6bb13f6595b5d82c2a173b3ee4.camel@kernel.org>
 <53599736e1733bd011325170a269e9adda2f2de1.camel@kernel.org>
In-Reply-To: <53599736e1733bd011325170a269e9adda2f2de1.camel@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?1hNzXEg71tWu7h9UY52KSijGz++D/lBgCh0P3cDQc9oX6kDRQY/WJKj0vlR/?=
 =?us-ascii?Q?Q5jeGxWZC4LdAUD3ORJ77TA2VBHnR89paoKRqBPFMghcqRlsInzwSQs+VMhR?=
 =?us-ascii?Q?zaz7CysxqkXRvRNk0Iy7fVvK/kbKFfrSYyQV8fUHFDhpWgJr0aqhXVcyMgRC?=
 =?us-ascii?Q?oGZDOAo0oa45RYfIvLj+KdFVi39y/G2uyOvDXQ/AO6Gv7cY+FdyOYYjaaV7U?=
 =?us-ascii?Q?267NzZkRezBOXCLkJf4zqzpJDhh+bbyLnqNPZca4z5LON0TLH64+eSWhKhPB?=
 =?us-ascii?Q?ko+CZbBkIUY6A16kKJoEI9FcH4DODaRfMsV+ThPpNCrUHjpC01UdrliXJTIj?=
 =?us-ascii?Q?HMoYU/yFbbg/btbCUwOsDzcj3egtYGELegY60CR6Tk+SJoNghcltZbfWr5B+?=
 =?us-ascii?Q?/2ZURB7+1YJRxz7wwtGYFzmE2noF6QH/wlqKmlvJ96IxGlj6xSSFwKrUd69Y?=
 =?us-ascii?Q?EnpTs4wAQXJbcQfwjtXLwBv3VETaWqt1MEBjZzk6+vh4Lkpba9MvFMFlvTYi?=
 =?us-ascii?Q?InKPnoc+SxaRJ2mhtHno9iBCzmZWdduuiYcetQarQ4UYDF5rGcnvweCuifNl?=
 =?us-ascii?Q?LGtfZZdpZ6pR/5b0vhB1qgXzx/+FQn5Po/hDzKpjxNXIJj1OfRi4Kub5f3oc?=
 =?us-ascii?Q?xBZC3+yl97Oc0K1R3ZU/p0+X2o4aObfByzMTr7rdI7wtD+0ER2GAz4dmbKgd?=
 =?us-ascii?Q?hunBeZOeykd0RBeBX8dUc13eN8ws7JZTktJVRXkxhCwe6Uz1I0WBfdreW4RN?=
 =?us-ascii?Q?M6MNjgXnILu3BQQCaWUiyHbn/YFADKYmgzPqlY+dC48kYW48VEmq+kIg2N0t?=
 =?us-ascii?Q?6nUvwuNJ/gclCIi9TDTDzieUXtUhkk6VIPt0kfc8zzURSkPNVZAXbpeLP4C7?=
 =?us-ascii?Q?JGBToBQNbQzxpqfRdDXyOssjt304YrQbJBgtN3OXvd2o+osCKxascz3QjXTZ?=
 =?us-ascii?Q?7+AURMPT6Rdy2eSJPRRK9QE2R6MLAWT+ZvmlrtbAWKedW4lZbfHC5UsY1GWO?=
 =?us-ascii?Q?fxHX5u6wbGzrxgpdoQpWy5OOmuKLrRWkh6i3N/IbXCzMj3tglgerkxs4zFoa?=
 =?us-ascii?Q?1GrEC4brOxuyNo5SVTarNt+6yoQ2WS54g5nmbkRQqyJOCEk2WrwCZwS27QbN?=
 =?us-ascii?Q?jSriy0rIwgtP5jVe0F/uL04dOyKYci1PXQrQaurQmGmpLgSjpp7lBedgUay4?=
 =?us-ascii?Q?UNhvKvx4c83bdmVIL8HugVHgrz2fzUbFb3KQORvumg3VAJOwdSgvn8njbjw7?=
 =?us-ascii?Q?w7WD7fgeO8imsTxbDyxth9kJy2wL6riAlnoianMotejUaop1e1DHk02vsCdI?=
 =?us-ascii?Q?SpVmry75u/fDgPEHz8K6fj2VpRzdbObDQnEU/CqBa+SQZIlQJkpA//j/RbP8?=
 =?us-ascii?Q?0Qcg3Is1zKHYIYK3O6ttU4PbJ0gOSarJmFALvaVT7PwoypvW05hjHAn2tjua?=
 =?us-ascii?Q?kOM+V4UUKIli2Ll2XpUXp0kgmud0XnYBGnOZZOc3OmgQMMwcLV1z+KN3nwNp?=
 =?us-ascii?Q?PtHEGL6/+lLaNjVROAp4hgEjlQL4QTHctTezQxyh+X1sUFMQx4hzmiAKnZSN?=
 =?us-ascii?Q?lRUkMZ0JO9jcy/1JYiPhpDRb2pZBhnQwJq3zXPRVfaOX/tZYnh6HBytj8fnr?=
 =?us-ascii?Q?rA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <D921C6CBF48E714F95ADA007C424F7EA@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d1bff9b2-f5f9-4177-cf3d-08dabc3d25ae
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Nov 2022 19:13:23.6961
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: +zkl2tEcX2WNSNLreuYuZozzsfOMsTo7+2PqP/UrT66DPxZE3E13enwv3dwrGNXnYsy2zKHg4H5OGMiI9UXPoQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6706
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-11-01_09,2022-11-01_02,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 bulkscore=0
 suspectscore=0 phishscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000
 definitions=main-2211010139
X-Proofpoint-ORIG-GUID: p-G6q_SWrWfPMJvY4L9Fwh8bCPSvYj5F
X-Proofpoint-GUID: p-G6q_SWrWfPMJvY4L9Fwh8bCPSvYj5F
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Nov 1, 2022, at 2:57 PM, Jeff Layton <jlayton@kernel.org> wrote:
>=20
> On Tue, 2022-11-01 at 14:03 -0400, Jeff Layton wrote:
>> On Tue, 2022-11-01 at 17:25 +0000, Chuck Lever III wrote:
>>>=20
>>>=20
>>> Test results:
>>>=20
>>> When I run my test I "watch cat /proc/fs/nfsd/filecache". The
>>> workload is 12-thread "untar git && make git && make test" on
>>> NFSv3. It's showing worse eviction behavior than the current
>>> code.
>>>=20
>>=20
>> What do you mean by "worse" here?
>>=20
>>> Basically all cached items appear to be immediately placed on
>>> the LRU. Do you expect this behavior change? We want to keep
>>> the LRU as short as possible; but maybe the LRU callback is
>>> stopping after a few items, so it might not matter.
>>>=20
>>=20
>> Could be. I'm not sure how that works.
>>=20
>=20
> Looking more at the old LRU code, I'm not sure we can make a direct
> comparison on behavior. I think that the old code was just broken, and
> that it inappropriately took entries off the list when it shouldn't
> have. That mostly worked out in the end, but I don't think the lifetime
> of those entries was what was wanted or expected.
>=20
> With the new code, it's much more clear. The only entries on the LRU are
> GC entries with no active references. Once nfsd_file_do_acquire is
> called, the entry comes off the LRU.

The new mechanism is not working the way you might have intended. I see
the "total" and "LRU" numbers equaling each other throughout the test
run, which is a sign the LRU is not working correctly. What you said
here suggests that the "LRU" number is supposed to be less than the
total number of cached items, and that's the way the old code behaved.


> I don't see how we can make the LRU any shorter.

I'll dig into it and see what's going on.


--
Chuck Lever