Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Wang Yugui <wangyugui@e16-tech.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [RPC] nfsd: NFSv4 close a file completely
Thread-Topic: [RPC] nfsd: NFSv4 close a file completely
Thread-Index: AQHYfi1IwwfFdSiA302AZfT/YMJLJq1MFpUAgASGG4CAAAMIAIAAN3CA
Date:   Wed, 15 Jun 2022 18:57:28 +0000
Message-ID: <66105B5B-A0A4-4466-9BD6-0E68903BF7BA@oracle.com>
References: <20220612072253.66354-1-wangyugui@e16-tech.com>
 <0CBF71FB-7754-4992-BE16-A3CFD404DECC@oracle.com>
 <20220615232810.95CE.409509F4@e16-tech.com>
 <06C68349-C174-402B-A902-31A65BEFFB0B@oracle.com>
In-Reply-To: <06C68349-C174-402B-A902-31A65BEFFB0B@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?ETMFgDdEpcV0iY4QK/kyudEueximzGADtfV6rCO5HRw6chzGXYdMk4JhfQeV?=
 =?us-ascii?Q?Vu2+UFHAT//EzApLwASqzL8JZyx7UR5lgRg3i9Sa3y/cJgmqFQ8iFLemvvtH?=
 =?us-ascii?Q?49g847Xe0Ij0q9Hv+t6DheUb4KmUTp8m3VXdoX6jPu2ozkm0p1XJAp+Oc/Lo?=
 =?us-ascii?Q?D8cXw3Qj7vHyk2iF8SmOYZOSdm7iUnLYg3kIU3dU9hk4G5Bd9oJVrwPefmlj?=
 =?us-ascii?Q?31lwWRl25sbXZBKNArblVMfxzz/6C2uLBo1ckUMPAG9IcZ61wIwTUA8Z3mcx?=
 =?us-ascii?Q?IIkcJ/it9GXyxER/wKMClgcW8M79RRhs2WlMSV3qOCpfEKrsA4PKlwRmRw08?=
 =?us-ascii?Q?XzQz38YG9P700Qllg136cmBuN8XPMnwVgEn1gePU7qJWoKkCdVQGwplXxGPv?=
 =?us-ascii?Q?L0QCKBLNW5B3P08VU1oB+jYOWHYWUB0JuG3EyVULNqxR8UUXGZxwGC1r+ImC?=
 =?us-ascii?Q?2hruf2G/Q7DXh3tK7tij5d5aQyZB4ZaYQQcQpXO5tVwE/TaOqzqzvfkLdphN?=
 =?us-ascii?Q?9dPymIRfIjArv2dwwe5mJd6F44WMsBRMAf2Bd12LI6OjqMdkZV0haTWdR/TW?=
 =?us-ascii?Q?LILxFeehwCV0HPNDxQw4quVZPNzuJnMPa6vsmKy0A1MxC82d9Fhb4kT/7w9s?=
 =?us-ascii?Q?ATSR917uC/iVUL+s3wNxiBCU1tOhq9S1mDmhRvv6ZWm2awgAQptNI8sp0ghP?=
 =?us-ascii?Q?QZrkGFb5SLFgKIwqRtE/OWQxAyDiokFoJTZ1e+Cf3l9M/c7Ku25CkdV78XF0?=
 =?us-ascii?Q?gtaWSbQwieh7hlKhAYvNd8QUXuysaGQ9g2a27RIYIe20viD14I34EaAKCGU8?=
 =?us-ascii?Q?2dBha+FbK7WFGUytZ4IE4q9x56Ux85lkkselMWY5ejf1ufnO1+48JVcvxf2A?=
 =?us-ascii?Q?HJeO4DpoKLynndeujSPQMjVvp0mWhLhyHHcSAbvsQf5gPj0KVON1TM4FFJZr?=
 =?us-ascii?Q?4KqAqmNMTZznOOXXo/R2TwxxAWEy4xEcC14WwYDfwA8nm6JV3atQhinm6sho?=
 =?us-ascii?Q?tHCicCSUdajyyMa6DhjTgovGRXbvYIiGDS19p/IVU30xmpEQuby7qgz9bEK8?=
 =?us-ascii?Q?6qQZdBdfYpNDhhypkE03WodqeNTw3RyIjoFFPrkGYCUCioTDQuahcNBuU1Oh?=
 =?us-ascii?Q?/rMArWECs9rV32XjxG3c9RL1QFw6wYvENyO24jJati6Y/7wG0mDUYq9Cy5vh?=
 =?us-ascii?Q?lLaeeLeHmISlnbi1FCmBe/C6BFMgdbEWwPHwYRWPfZqjt/1u0z8y/isA+Zcq?=
 =?us-ascii?Q?lUZ9Ysz1uZ5hGA/u6hRAYgrcEfpsj9wigpFYnv2ov0e0Gbf7+83xlm3OAuBg?=
 =?us-ascii?Q?Q4snlUcZQpJKTbcQwAr8uweFGSWojPHRueq1kextG3qtPdJtU/W7TwS3kB/9?=
 =?us-ascii?Q?kIZWgA3DvYQCmjfNtWrD+bdSpzTRqmsVhiyr+7ZDaqtXFuu+vqJ9pk2ilLev?=
 =?us-ascii?Q?FyrQOUaen48fTU4wJVBw11yn60bZXZL2W42Unh9D1A3B3LMF+c7KarSqE6n9?=
 =?us-ascii?Q?+6GTqn3KrHkzXYb3R0kUgfOkEw176GuUJyHCnKrgS51eBPb+onhGM7XuInZd?=
 =?us-ascii?Q?cnpvlAlmmXPlx3eBdPRZi/jbhEEfJIUfpEuslGeCsMM7+xbAidIf73vBQNKU?=
 =?us-ascii?Q?MSwizfsE5NHrRw7tXHPMuXMdZq0PJ0nIhWAPN7ZaTsG2xgsgxAyEkldhIq5P?=
 =?us-ascii?Q?d1LEWf2Ow+zvo9O6Tl0/dZiWq7mkwgZfXryJd1xhrg9nz5oCQpzeXDHJuP4S?=
 =?us-ascii?Q?Y9eOERSupBX5M0Q9FWaMsd15uFfIek8=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E111E57AFF622641B427D9B64D535CFB@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b02ae5c6-db7c-4f0b-20b6-08da4f00e4e7
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Jun 2022 18:57:28.4824
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: HpQdZa8dwzjUuLe1UyQLPqzkiUs856cDMY7EK5zbQDkPRQuSjZLbb5suZ/y9jMCmT8THodD+GTt9FwwwJT82og==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB3444
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874
 definitions=2022-06-15_06:2022-06-15,2022-06-15 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 adultscore=0
 mlxlogscore=999 phishscore=0 malwarescore=0 bulkscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000
 definitions=main-2206150071
X-Proofpoint-ORIG-GUID: NvR2c2mt3WWK1o707vk2TKwqQVS1xdw1
X-Proofpoint-GUID: NvR2c2mt3WWK1o707vk2TKwqQVS1xdw1
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 15, 2022, at 11:39 AM, Chuck Lever III <chuck.lever@oracle.com> wr=
ote:
>=20
>> On Jun 15, 2022, at 11:28 AM, Wang Yugui <wangyugui@e16-tech.com> wrote:
>>=20
>> A question about the coming rhashtable.
>>=20
>> Now multiple nfsd export share a cache pool.
>>=20
>> In the coming rhashtable, a nfsd export could use a private cache pool
>> to improve scale out?
>=20
> That seems like a premature optimization. We don't know that the hashtabl=
e,
> under normal (ie, non-generic/531) workloads, is a scaling problem.
>=20
> However, I am considering (in the future) creating separate filecaches
> for each nfsd_net.

So I'm not rejecting your idea outright. To expand on this a little:

Just a single rhashtable will enable better scaling, and so will fixing
the LRU issues, and those are both in plan for my current set of fixes.
It's not clear to me that pathological workloads like generic/531 on
NFSv4 are common, so it's quite possible that just these two changes
will be enough for realistic workloads for the time being.

My near term goal for generic/531 is to prevent it from crashing NFSD.
Hopefully we can look at ways to enable that test to pass more often,
and fail gracefully when it doesn't pass. The issue is how the server
behaves when it can't open more files, which is somewhat separate from
the data structure efficiency issues you and Frank pointed out.

I'd like to get the above fixes ready for merge by the next window. So
I'm trying to narrow the changes in this set of fixes to make sure
they will be solid in a couple of months. It will be a heavier lift to
go from just one to two filecaches per server. After that, it will
likely be easier to go from two filecaches to multiple filecaches, but
I feel that's down the road.

In the medium term, supporting separate filecaches for NFSv3 and NFSv4
files is worth considering. NFSv3 nfsd_file items need to be managed
automatically and can be subject to a shrinker since there's no client
impact on releasing a cached filecache item.

NFSv4 nfsd_file items manage themselves (via OPEN/CLOSE) so an LRU isn't
really needed there (and isn't terribly effective anyway). A shrinker
can't easily release NFSv4 nfsd_file items without the server losing
state, and clients have to recover in that case.

And, it turns out that the current filecache capacity-limiting mechanism
forces NFSv3 items out of the filecache in favor of NFSv4 items when
the cache has more that NFSD_FILE_LRU_LIMIT items in it. IMO that's
obviously undesirable behavior for common mixed-version workloads.


--
Chuck Lever