Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Frank van der Linden <fllinden@amazon.com>
CC:     Wang Yugui <wangyugui@e16-tech.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Matthew Wilcox <matthew.wilcox@oracle.com>,
        Liam Howlett <liam.howlett@oracle.com>
Subject: Re: filecache LRU performance regression
Thread-Topic: filecache LRU performance regression
Thread-Index: AQHYcfvwqM9TwnG0lkutiJn4EA8Bxq0zLyiAgAAP6gCABnuxAIABBV+AgAAHvgCAAE5sgIAABXOA
Date:   Wed, 1 Jun 2022 21:37:58 +0000
Message-ID: <276D7504-1019-4912-822C-320E6E6180A3@oracle.com>
References: <5C7024DA-A792-4091-BFDE-CEED59BC1B69@oracle.com>
 <20220527203721.GA10628@dev-dsk-fllinden-2c-d7720709.us-west-2.amazon.com>
 <ADD1751A-7F67-4729-BFFC-D6938CA963A0@oracle.com>
 <BED36887-054D-4DC9-A5F1-CB6DD1F0DC16@oracle.com>
 <20220601161003.GA20483@dev-dsk-fllinden-2c-d7720709.us-west-2.amazon.com>
 <4C14DB3A-A5C1-41A9-8293-DF4FC2459600@oracle.com>
 <20220601211827.GA11605@dev-dsk-fllinden-2c-d7720709.us-west-2.amazon.com>
In-Reply-To: <20220601211827.GA11605@dev-dsk-fllinden-2c-d7720709.us-west-2.amazon.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Ju7ej69E5oDQ9nPIClQrti+GpViBXZFczAjFylNB9i+Rb/bovcnz6xBppSYh?=
 =?us-ascii?Q?fTfczXuS/KYHqtrmsO5zjTHVcW8TRCGRtMrU+EqiIZpVZhdsdru09zoXevV7?=
 =?us-ascii?Q?sZsI3oUjnv9iu1h8Zc0CiHMf0VtG7osvPE/oP/J3g5gKEFi2cv5ufBvMRfr8?=
 =?us-ascii?Q?DvJ64rzDD8Mas5nGvTP33QmqTFxjkHGGvQl6vVDCms5qLpsaA1yJZxKw5aD3?=
 =?us-ascii?Q?4vp/HQcjWqpu4XOF/FOIGx9IFF4dDrXP1eH84nSuDJCXDEfO8OtHmQNvqgrL?=
 =?us-ascii?Q?UFUrQaytqqf7D6PFW/n6OvSUyj2Mpaq3XLjaF6EsDJdfFIitKonIRuPKhsgO?=
 =?us-ascii?Q?8AtNpymafrU433FbKE/PEfyFUiracGG7pCMlUzjZITLgG3t/FokREVCfpJBV?=
 =?us-ascii?Q?WdMFmM5uGTA/1O9WpFA3gdCerbXCfoqY4i2bamh9YmMbylrYcCO6kUcRAB3H?=
 =?us-ascii?Q?UDCQ8W3ql86dIJWte/MtCw/beU/kYLhve2lyUqYqFyIO5z8M3wJ/LE57hnBu?=
 =?us-ascii?Q?Jjj7a46K5RvgOqY35ioDhA3DVsCzfeg20MF9m3K8owCGfxbIyWTCPtuO7Z2I?=
 =?us-ascii?Q?LhP0VKzfQTwet1x3/9FA3Hre6/DmCjvmiNXmgH+xzb72yLy8JW1VZJpG2Qoe?=
 =?us-ascii?Q?OqZnl3WJeqK+6Y9O8i5Y2R43c0qXu3kDKYBZXgB0hX7mMqCRs4yBeYhGxPUS?=
 =?us-ascii?Q?kXQrtFvwtRMMXEEOjG/VY9zFDe2LGNNsjWvzrnlhy3Rpzfm+nYHyYntbxVgN?=
 =?us-ascii?Q?BFaKrII6Ex42KwjkXa5QT/5DN31DcOYMURNlzjSnucebXTLLqeSOJ3pSUG8p?=
 =?us-ascii?Q?2uoNCxlDEzUyQhLP+/hKtCG0Nts3rhrl33HMp4mYu6wj8O6CZMeShLbgASZG?=
 =?us-ascii?Q?UIgNX/Bziva9vQGDJn7XyZLL8jp2wt6ajvFIhWD8pMMHoNLwdPo309iJ9Ub4?=
 =?us-ascii?Q?7BfRodpCum8QugsNx03ObyQMqUGDJ1ESheoQ0xaUMRNtTYB8h5cv2mYhrIrW?=
 =?us-ascii?Q?AZeZTRVWQ+f/xE/ce5RIZJvvQ24uKxvl/E9HT+7NtNHEqqO98mc+lFAeODrY?=
 =?us-ascii?Q?5YzaSigDr+7PEGv9ze9xtMo/4+OrQkpiTt89Oxx7DUqW3zKyZNbn3IljNEvC?=
 =?us-ascii?Q?bDv5G7+jE3jxsOPp7F42ZP4zykQ+U0n5sbPxuS++nzKRHnFMIBQqTH8vmgIr?=
 =?us-ascii?Q?ngBFrPvS7/Vg2wg06PUYHfZnJchUDfbYVdQeUo12lechWz6BvQwjFuNgOr4B?=
 =?us-ascii?Q?MzDnpPueZbgAtGJW+F3uoK3Twaf1e9joh3db/XkgrvlC4HYaGIPCS2usHhjX?=
 =?us-ascii?Q?8Dt9i2vehSqNnftYDv1+T7/X+AnGeEAkwyHzjhIHweuisHLIrZG4ACWIRG/r?=
 =?us-ascii?Q?dRlKRNaV/sxjCmzxlcNUe1TkOl3C02VAB77DgP9VtKo07VL65EvNrln2NNQ6?=
 =?us-ascii?Q?12HfiEYK0bzMPq0Vfg4GvV2dilWe3GWCp/gSYroA1FHCrXMS1GelkR28BFdK?=
 =?us-ascii?Q?w9+bGO0YfKgJ7IZvp5SURKGzw2XX5PFZAII1m3Qa1zhQJq8UYgRa8GBx+ly9?=
 =?us-ascii?Q?FxX/HCfIeEFV+FilNMboAoT0cWgEbSmbjfRtZp0/kcEDmfW4rrpkQ2KgHXAh?=
 =?us-ascii?Q?UUE5mnOekRPFTonv149xD2vKDxlULSGS+T2+xY6pzvCxw/wTIqCFysamlcu6?=
 =?us-ascii?Q?y0hY0KJ3ayn0GGr4V7Y+UArsPAWwSQnEe1u/64U7O+Vit/OLaBAjkJZwUSGu?=
 =?us-ascii?Q?z/tdM8xUYIWCf0NhryR27eNLGErUQMQ=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A4BB8099338F5E4783845E8893BD32A5@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f9d09ac7-0e5a-4371-2623-08da4416fef7
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jun 2022 21:37:58.3457
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: GXPGMbLf6Kj/RJFfT1/ZupMrjhuFeM5I0+RB+bjc5kjB+rrsLyom1PN6WfDpRWKAQrP+DmaNgkFi8fSyXqda9Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3947
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874
 definitions=2022-06-01_08:2022-06-01,2022-06-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 phishscore=0
 suspectscore=0 mlxlogscore=999 bulkscore=0 spamscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000
 definitions=main-2206010085
X-Proofpoint-ORIG-GUID: fJSacMrrqL3ngIoi4SNKVm4MrkME1QMH
X-Proofpoint-GUID: fJSacMrrqL3ngIoi4SNKVm4MrkME1QMH
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 1, 2022, at 5:18 PM, Frank van der Linden <fllinden@amazon.com> wr=
ote:
>=20
> On Wed, Jun 01, 2022 at 04:37:47PM +0000, Chuck Lever III wrote:
>>=20
>>> On Jun 1, 2022, at 12:10 PM, Frank van der Linden <fllinden@amazon.com>=
 wrote:
>>>=20
>>> On Wed, Jun 01, 2022 at 12:34:34AM +0000, Chuck Lever III wrote:
>>>>> On May 27, 2022, at 5:34 PM, Chuck Lever III <chuck.lever@oracle.com>=
 wrote:
>>>>>=20
>>>>>=20
>>>>>=20
>>>>>> On May 27, 2022, at 4:37 PM, Frank van der Linden <fllinden@amazon.c=
om> wrote:
>>>>>>=20
>>>>>> On Fri, May 27, 2022 at 06:59:47PM +0000, Chuck Lever III wrote:
>>>>>>>=20
>>>>>>>=20
>>>>>>> Hi Frank-
>>>>>>>=20
>>>>>>> Bruce recently reminded me about this issue. Is there a bugzilla so=
mewhere?
>>>>>>> Do you have a reproducer I can try?
>>>>>>=20
>>>>>> Hi Chuck,
>>>>>>=20
>>>>>> The easiest way to reproduce the issue is to run generic/531 over an
>>>>>> NFSv4 mount, using a system with a larger number of CPUs on the clie=
nt
>>>>>> side (or just scaling the test up manually - it has a calculation ba=
sed
>>>>>> on the number of CPUs).
>>>>>>=20
>>>>>> The test will take a long time to finish. I initially described the
>>>>>> details here:
>>>>>>=20
>>>>>> https://lore.kernel.org/linux-nfs/20200608192122.GA19171@dev-dsk-fll=
inden-2c-c1893d73.us-west-2.amazon.com/
>>>>>>=20
>>>>>> Since then, it was also reported here:
>>>>>>=20
>>>>>> https://lore.kernel.org/all/20210531125948.2D37.409509F4@e16-tech.co=
m/T/#m8c3e4173696e17a9d5903d2a619550f352314d20
>>>>>=20
>>>>> Thanks for the summary. So, there isn't a bugzilla tracking this
>>>>> issue? If not, please create one here:
>>>>>=20
>>>>> https://bugzilla.linux-nfs.org/
>>>>>=20
>>>>> Then we don't have to keep asking for a repeat summary ;-)
>>>>=20
>>>> I can easily reproduce this scenario in my lab. I've opened:
>>>>=20
>>>> https://bugzilla.linux-nfs.org/show_bug.cgi?id=3D386
>>>>=20
>>>=20
>>> Thanks for taking care of that. I'm switching jobs, so I won't have muc=
h
>>> time to look at it or test for a few weeks.
>>=20
>> No problem. I can reproduce the failure, and I have some ideas
>> of how to address the issue, so I've assigned the bug to myself.
>>=20
>>=20
>>> I think the basic problem is that the filecache is a clear win for
>>> v3, since that's stateless and it avoids a lookup for each operation.
>>>=20
>>> For v4, it's not clear to me that it's much of a win, and in this case
>>> it definitely gets in the way.
>>>=20
>>> Maybe the best thing is to not bother at all with the caching for v4,
>>=20
>> At this point I don't think we can go that way. The NFSv4 code
>> uses a lot of the same infrastructural helpers as NFSv3, and
>> all of those now depend on the use of nfsd_file objects.
>>=20
>> Certainly, though, the filecache plays somewhat different roles
>> for legacy NFS and NFSv4. I've been toying with the idea of
>> maintaining separate filecaches for NFSv3 and NFSv4, since
>> the garbage collection and shrinker rules are fundamentally
>> different for the two, and NFSv4 wants a file closed completely
>> (no lingering open) when it does a CLOSE or DELEGRETURN.
>>=20
>> In the meantime, the obvious culprit is the LRU walk during
>> garbage collection is broken. I've talked with Dave Chinner,
>> co-author of list_lru, about a way to straighten this out so
>> that the LRU walk is very nicely bounded and at the same time
>> deals properly with NFSv4 OPEN and CLOSE. Trond also had an
>> idea or two here, and it seems the three of us are on nearly
>> the same page.
>>=20
>> Once that is addressed, we can revisit Wang's suggestion of
>> serializing garbage collection, as a nice optimization.
>=20
> Sounds good, thanks!
>=20
> A related issue: there is currently no upper limit that I can see
> on the number of active OPENs for a client. So essentially, a
> client can run a server out of resources by doing a very large
> number of OPENs.
>=20
> Should there be an upper limit, above which requests are either
> denied, or old state is invalidated?

We need to explore the server's behavior in low resource
situations. I prefer graceful degradation of service rather
than adding admin knobs, plus adding maybe some mechanism that
can report unruly clients so the server admin can deal with them.

The VFS will stop giving out struct file's after a certain point.
NFSv4 OPEN operations will return an error.

Some of the server's resource pools have shrinker callbacks.
Those pools will be reaped when there is memory pressure. We
could do better there.

I don't believe there's any mechanism yet to get rid of state
that is still active. I consider that as a "heroic measure"
that adds complexity for perhaps little benefit. And doing so
is potentially catastrophic for clients that trust the server
won't rip the rug out from under them. I would rather prevent
the creation of new state at that point rather than invalidating
existing state.

There are ways that server administrators can clean out known
defunct clients. That would be perhaps a little safer, and might
go along with the idea that server admins should deal mindfully
with malfunctioning or malicious clients that are causing a
denial of service.

In the end we have to address problems like CPU soft lock-up
to be certain server administrative interfaces will remain
available when the server is under a significant workload.


--
Chuck Lever