Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Frank van der Linden <fllinden@amazon.com>
CC:     Wang Yugui <wangyugui@e16-tech.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Matthew Wilcox <matthew.wilcox@oracle.com>,
        Liam Howlett <liam.howlett@oracle.com>
Subject: Re: filecache LRU performance regression
Thread-Topic: filecache LRU performance regression
Thread-Index: AQHYcfvwqM9TwnG0lkutiJn4EA8Bxq0zLyiAgAAP6gCABnuxAIABBV+AgAAHvgA=
Date:   Wed, 1 Jun 2022 16:37:47 +0000
Message-ID: <4C14DB3A-A5C1-41A9-8293-DF4FC2459600@oracle.com>
References: <5C7024DA-A792-4091-BFDE-CEED59BC1B69@oracle.com>
 <20220527203721.GA10628@dev-dsk-fllinden-2c-d7720709.us-west-2.amazon.com>
 <ADD1751A-7F67-4729-BFFC-D6938CA963A0@oracle.com>
 <BED36887-054D-4DC9-A5F1-CB6DD1F0DC16@oracle.com>
 <20220601161003.GA20483@dev-dsk-fllinden-2c-d7720709.us-west-2.amazon.com>
In-Reply-To: <20220601161003.GA20483@dev-dsk-fllinden-2c-d7720709.us-west-2.amazon.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?5Cd8oidk15pYdDODvMp55+gGjL0JknXmtrUZ8+BWJ7FJ7bddOcL1lxNfBmCm?=
 =?us-ascii?Q?giw+IsrEiKkxAnxqgnmjTmVuZRDL1Kk3iGjlr341cNWutGa/Lkk9DMLS3EOi?=
 =?us-ascii?Q?HpnC1mfg6Rdv4l9K+Pc7lF4aEyFl8jDMUBCxvFf1gnuijaCpLWo8vvCDn/H+?=
 =?us-ascii?Q?ZM/1LqmJKPVReU8K3dIYw3m9bn22xDogwzuuJPBzyJ95e//qQy0R9kbaHuc+?=
 =?us-ascii?Q?skUUM+g3t9Cj/fA8NVnZts55vXENiFg2hm56JZrD3kKtJFvuA1H0k1NWCx00?=
 =?us-ascii?Q?xUkd2GRs+AExmmKwkIFou+xCzVSr51Sp8rjiOHHtD2buWtSA4WxpO45ZrCVx?=
 =?us-ascii?Q?KSKzZq1dKY1MSRCBJac+nct2EKRrRDGy0PYltH4PGT3re7IEFQOhnlDqeDAy?=
 =?us-ascii?Q?tpErs6t/an7sHvNGqluJNpmfWybRS48d+ra6stLkPl4VOQ8Hq/FuhPorpB5z?=
 =?us-ascii?Q?SwwAb+imXQnjyW0kpCwAibYaOD3QdipfUMwIZyifboJ5L5ASiwUys4aLIKg1?=
 =?us-ascii?Q?A1W1kpiSVxgc4zOP4IFug7lRvIWJprxm7CUI/sf7wArQCfdPsTPYI4jB86Zq?=
 =?us-ascii?Q?3hD4nWxIqHfFg5IUJk/migaHaEdiJ7l6IM6hluBPHksIrKnq+pYONn9wfGtE?=
 =?us-ascii?Q?vY7AXlTpBF+N5maeIzZRkcwTAP7YPLA1Ct48t5ETQS2vOfCihGyvTIj5/mva?=
 =?us-ascii?Q?v/nLctK/ftxrmk6BKOBs/N/IGLKBIIvXX7TTAmK9TKXDrcoQvLq/wKgeHsjM?=
 =?us-ascii?Q?VpNWQQk2eZJX7R5Mn3gETYU027JGMa85w5zDyhw3CXGhP6wjhH0WZjxkcOg4?=
 =?us-ascii?Q?v+SxS4dWSOy2JjfmndoKIQHo4/J9mEEhbaEP0srahF5exathz3f+0YeLe9kx?=
 =?us-ascii?Q?PSlbK29ODMl/CDRv9x7jqHLt3g09heosheAfYGacitjes4tuyCOxb82Ymacu?=
 =?us-ascii?Q?532W99xQa+9DGSHchiUQ6pWVaWFAajk9aWy5ff7n95IW7nSupN5q7iyTCW8h?=
 =?us-ascii?Q?v9aEJWAdNVpyAycexH8O0IPOurO8YG0U8mQpYHp+PkQNIobixUQufqJPRsyL?=
 =?us-ascii?Q?hYUz94lr1FJ5rLTMTq8eTVvKy9S2vmWhD9qH5nElxbPVzSgPc797jnfqEWdc?=
 =?us-ascii?Q?yLUM/JwVysAQW8NIgzybPDRuuMZTqS+TRTdrH/OQWG4j+GHcc2W/WUH8G0MH?=
 =?us-ascii?Q?b1XHyLZLbac+U5ZBSOCWN0S0GHBcox47vTyNmn9Azqqf7AbZksvHUHfV50Cm?=
 =?us-ascii?Q?v1BRqJELiqXhA2LROeqc0S5xPt+5FGSnYbkCL6QALS0BQUn5FcOrOKmPqez1?=
 =?us-ascii?Q?5kV6xQRLzWtAVSLq6/jbsu0ldF/VyFCObcEIMqr6B+obOvpviRgL6EQMOpKu?=
 =?us-ascii?Q?6UZhpD4sfb11h6lJtRE7p4vfqCkoayqVFabQJlgl5uY31/byNBixHOXEicwy?=
 =?us-ascii?Q?2av3HmogllZi76sWgEv/u0YuLjT08URy4zm6sMWxQYS4Qouc9vulk4quwPSB?=
 =?us-ascii?Q?AWLixaL6kd8Y3bRl7ybrUhGoKVswgw8SnW/dHDyiNbDmC2hQYeEe3/tMENpQ?=
 =?us-ascii?Q?vjZ/T0goiEgouIArmtlk7I45wZHNOqu1eFZ/N/Hdh1YkdYtAdbhQLxJ/t9yk?=
 =?us-ascii?Q?YHeIvi81T0EAW74Y+pRpBJ9/DQMgDs0MMRz5zRb+1Z7POvafN2sPl0NQn5bz?=
 =?us-ascii?Q?61S1WZL3XSxZjaTANMAo8R0e78VzMwEQVelyu/vJMppaYFOKrP5cALWWq8Kp?=
 =?us-ascii?Q?7qAFYxl4exWASCjjEHVxz03LZZP5AGI=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <1870259F11493346922B71BAB793BFA9@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ec54ee7d-c131-4714-ea4f-08da43ed0f5c
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jun 2022 16:37:47.0092
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: QqcJKOR787uwPIuE74NhnR2sGk2sSHZcOH675hmYeCZCX44Lod/EfNYxvVxtRDUTgVdRTGnmLUGqdKFejkmucA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3513
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874
 definitions=2022-06-01_05:2022-06-01,2022-06-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0
 suspectscore=0 spamscore=0 phishscore=0 malwarescore=0 bulkscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2204290000 definitions=main-2206010074
X-Proofpoint-GUID: zxa6WjT5yeKiioM3_wJ4nJ2dbo7kNyu-
X-Proofpoint-ORIG-GUID: zxa6WjT5yeKiioM3_wJ4nJ2dbo7kNyu-
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 1, 2022, at 12:10 PM, Frank van der Linden <fllinden@amazon.com> w=
rote:
>=20
> On Wed, Jun 01, 2022 at 12:34:34AM +0000, Chuck Lever III wrote:
>>> On May 27, 2022, at 5:34 PM, Chuck Lever III <chuck.lever@oracle.com> w=
rote:
>>>=20
>>>=20
>>>=20
>>>> On May 27, 2022, at 4:37 PM, Frank van der Linden <fllinden@amazon.com=
> wrote:
>>>>=20
>>>> On Fri, May 27, 2022 at 06:59:47PM +0000, Chuck Lever III wrote:
>>>>>=20
>>>>>=20
>>>>> Hi Frank-
>>>>>=20
>>>>> Bruce recently reminded me about this issue. Is there a bugzilla some=
where?
>>>>> Do you have a reproducer I can try?
>>>>=20
>>>> Hi Chuck,
>>>>=20
>>>> The easiest way to reproduce the issue is to run generic/531 over an
>>>> NFSv4 mount, using a system with a larger number of CPUs on the client
>>>> side (or just scaling the test up manually - it has a calculation base=
d
>>>> on the number of CPUs).
>>>>=20
>>>> The test will take a long time to finish. I initially described the
>>>> details here:
>>>>=20
>>>> https://lore.kernel.org/linux-nfs/20200608192122.GA19171@dev-dsk-fllin=
den-2c-c1893d73.us-west-2.amazon.com/
>>>>=20
>>>> Since then, it was also reported here:
>>>>=20
>>>> https://lore.kernel.org/all/20210531125948.2D37.409509F4@e16-tech.com/=
T/#m8c3e4173696e17a9d5903d2a619550f352314d20
>>>=20
>>> Thanks for the summary. So, there isn't a bugzilla tracking this
>>> issue? If not, please create one here:
>>>=20
>>> https://bugzilla.linux-nfs.org/
>>>=20
>>> Then we don't have to keep asking for a repeat summary ;-)
>>=20
>> I can easily reproduce this scenario in my lab. I've opened:
>>=20
>>  https://bugzilla.linux-nfs.org/show_bug.cgi?id=3D386
>>=20
>=20
> Thanks for taking care of that. I'm switching jobs, so I won't have much
> time to look at it or test for a few weeks.

No problem. I can reproduce the failure, and I have some ideas
of how to address the issue, so I've assigned the bug to myself.


> I think the basic problem is that the filecache is a clear win for
> v3, since that's stateless and it avoids a lookup for each operation.
>=20
> For v4, it's not clear to me that it's much of a win, and in this case
> it definitely gets in the way.
>=20
> Maybe the best thing is to not bother at all with the caching for v4,

At this point I don't think we can go that way. The NFSv4 code
uses a lot of the same infrastructural helpers as NFSv3, and
all of those now depend on the use of nfsd_file objects.

Certainly, though, the filecache plays somewhat different roles
for legacy NFS and NFSv4. I've been toying with the idea of
maintaining separate filecaches for NFSv3 and NFSv4, since
the garbage collection and shrinker rules are fundamentally
different for the two, and NFSv4 wants a file closed completely
(no lingering open) when it does a CLOSE or DELEGRETURN.

In the meantime, the obvious culprit is the LRU walk during
garbage collection is broken. I've talked with Dave Chinner,
co-author of list_lru, about a way to straighten this out so
that the LRU walk is very nicely bounded and at the same time
deals properly with NFSv4 OPEN and CLOSE. Trond also had an
idea or two here, and it seems the three of us are on nearly
the same page.

Once that is addressed, we can revisit Wang's suggestion of
serializing garbage collection, as a nice optimization.

Good luck with your new position!


> although that might hurt mixed v3/v4 clients accessing the same fs
> slightly. Not sure how common of a scenario that is, though.


--
Chuck Lever