Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Wang Yugui <wangyugui@e16-tech.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "david@fromorbit.com" <david@fromorbit.com>,
        "tgraf@suug.ch" <tgraf@suug.ch>, Jeff Layton <jlayton@redhat.com>
Subject: Re: [PATCH RFC 00/30] Overhaul NFSD filecache
Thread-Topic: [PATCH RFC 00/30] Overhaul NFSD filecache
Thread-Index: AQHYhkIqDsKitGhdzEmSu19Gpo96sa1bwYeAgAAHywCAAA9RgIAA2ruAgACBMACAABKgAIABauOA
Date:   Fri, 24 Jun 2022 15:30:11 +0000
Message-ID: <11579827-F7FE-49FA-9D32-D98C138796B1@oracle.com>
References: <20220623170218.7874.409509F4@e16-tech.com>
 <0292A2FC-7725-47FC-8F08-CCB8500D8E1D@oracle.com>
 <20220624015121.06F3.409509F4@e16-tech.com>
In-Reply-To: <20220624015121.06F3.409509F4@e16-tech.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?gQUvnAAOs6Yotlc3bFFw1zxB0oXZ1S+p7V2Bnt4LT9zGq2ArzJ4HGcQ71ylb?=
 =?us-ascii?Q?+F0pZ0vaoszvuxAEa7+Zzha3Vend0bxqwyaRlBN+0yLBqYxiBpO9BCQJTdBc?=
 =?us-ascii?Q?RhJEwwLbidAuLsns4AZJrVFOd8Cx3JYDP+T5JIYZXuWPLuajD08PagAhO3BN?=
 =?us-ascii?Q?4G2B2wuhETS+zXvmT+4B4r8kO/lYyRRrIyST3vqgFkOqafp5LXIOO/hN1Mfy?=
 =?us-ascii?Q?GV0Dd3Ro+s/KTvqYR5FYQX3x+TKR3TeLP8bNdLVd/OSBmsqokgojRjw59Pw5?=
 =?us-ascii?Q?kt6oRQsl1KDwvXF5xOH5l+sRDd8ZmjKE+8kASpH5RYMKfRK/rnaAqXFJ6Zvb?=
 =?us-ascii?Q?IugAQVUCiKHH4LadZdeXnViGQGlIN8iyU+lnZFJslRjtwL+ENrHGLChDZBAb?=
 =?us-ascii?Q?vqGwMuJOKHqGY2lQO5AJesnbfgFs1x9czAiiTfDXMMfJirbHKMNNHO9MpxLV?=
 =?us-ascii?Q?MPpQe3AM+QEZ6/Salf75qB5KKYfy7A9Ymt9bS06+gx0h1vDlhWiNC/vObXBb?=
 =?us-ascii?Q?Tn0/QLa+QynTVbqxqvx8nI/B34f5w4yQWyXsQ2TARngFY3jPTvDD2umGtgap?=
 =?us-ascii?Q?6hTaOlmzXcSftAkruAnwE6yFU8T9+L5jbk8ov31gO/wPaNiX/3EzRGtyPQbQ?=
 =?us-ascii?Q?Ni5RdBVDw9Ok2PN7jwyOCvNWz6B/E0zf/xhxqN70rYZmcr0HvH+kIragpx0N?=
 =?us-ascii?Q?K7m+O7bYzxAiK9l3GDVTjdTB+6JAenbRRY+Ek+N8svMsbaW96piD2CLeYPaw?=
 =?us-ascii?Q?VDr2llEoIfTqTlXKWnavcqKDYHQnPAh3L0hk7PkKqzu9EAR2MZWpOLDOual1?=
 =?us-ascii?Q?/vQ6++DSmPLoNrdF8xONUYLRoKjohNkXt5/dHMj292GqXcIgnW9QXonfLqh6?=
 =?us-ascii?Q?xnd3L/Syvoc5xeZZNzziOkeVnxBNjnZknrAz+oUvij4wkHa6+i7W5w4wrTMR?=
 =?us-ascii?Q?MTfBjf3sPMSQUHJtC8Gt0RHds0HAFyuFdn+88C+l46+jD76iI4YQ+ubErDTa?=
 =?us-ascii?Q?kNoXjel4iCZPaJYn973JHmcCYNvxOxFQa5jsEjfYIm9g9k2F5CZfyHKJS2u7?=
 =?us-ascii?Q?ISyRB/e0hQ15R/vQ6/ybl94K1zWD9hwwXNIYNYPOzVJgufJQypClcJYIuSFD?=
 =?us-ascii?Q?JO8XVxJgRuyvLLLgLDTnjJDZ8nf1DbUWBI4cCtBZXLQyAEEW+Xx4O/PViuAf?=
 =?us-ascii?Q?rqWHuWYKTjiaHZbq82kqWZ/rxKZA+SYAdNg1bx1PJpxxtMpwM7lHgtYVRpq1?=
 =?us-ascii?Q?IbyPEnL4YIrCBsTeQ39hj+uOTdn+pGydUr478jSvrMBp1zijKk81LEoaRh/3?=
 =?us-ascii?Q?tcVW4uw90Ubn/J3DdgxccQxWg/LA2MOdUD5i2ponIWizzFEvcIjlF9SxDuEe?=
 =?us-ascii?Q?qwMJd/tKBkAQo3kPZK3AIIcL1yz2uYym6lzaFEKCNkFLk63qQdu7gU0MEDnl?=
 =?us-ascii?Q?epqiSCBoO3ZSrzlm5/3kdesyOv2y/AHsiVSrc2056aVe3awdfVWS+cuIoPuV?=
 =?us-ascii?Q?9UyQW+qSv2p3Nn9isgwXKd8qSPFxPtxJA3qWHbpjN+BUeEQBqNlAyAZ3H4W7?=
 =?us-ascii?Q?DJzIhswjxSuByFZyQh9wWNkQKPClBKPs+YflTy1URlt2GwOjouqfS4dAxcRX?=
 =?us-ascii?Q?6Q=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <2014D138C26B5E41A07B3394800BBAE3@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ce2cbfcd-6742-4e02-be00-08da55f66dc7
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Jun 2022 15:30:11.7843
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: wxzDE2olwrBbeBCOQp9aFzYl1MUCufNNROY/obEg+n8DdHLLlfpkawYYVaUTOUB+F//Qf5ePW14cfzEGEZCe9g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4755
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883
 definitions=2022-06-24_07:2022-06-23,2022-06-24 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 phishscore=0
 mlxlogscore=740 mlxscore=0 suspectscore=0 spamscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000
 definitions=main-2206240059
X-Proofpoint-GUID: 3cvnVbaEmv9XvCvBtAcSChuvv9CjJc7W
X-Proofpoint-ORIG-GUID: 3cvnVbaEmv9XvCvBtAcSChuvv9CjJc7W
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 23, 2022, at 1:51 PM, Wang Yugui <wangyugui@e16-tech.com> wrote:
>=20
> Hi,
>=20
>>> On Jun 23, 2022, at 5:02 AM, Wang Yugui <wangyugui@e16-tech.com> wrote:
>>>=20
>>> Hi,
>>>=20
>>>>> On Jun 22, 2022, at 3:04 PM, Chuck Lever III <chuck.lever@oracle.com>=
 wrote:
>>>>>> On Jun 22, 2022, at 2:36 PM, Wang Yugui <wangyugui@e16-tech.com> wro=
te:
>>>>>>=20
>>>>>> Hi,
>>>>>>=20
>>>>>> fstests generic/531 triggered a panic on kernel 5.19.0-rc3 with this
>>>>>> patchset.
>>>>>=20
>>>>> As I mention in the cover letter, I haven't tried running generic/531
>>>>> yet -- no claim at all that this is finished work and that #386 has
>>>>> been fixed at this point. I'm merely interested in comments on the
>>>>> general approach.
>>>>>=20
>>>>>=20
>>>>>> [ 405.478056] BUG: kernel NULL pointer dereference, address: 0000000=
000000049
>>>>>=20
>>>>> The "RIP: " tells the location of the crash. Notice that the call
>>>>> trace here does not include that information. From your attachment:
>>>>>=20
>>>>> [ 405.518022] RIP: 0010:nfsd_do_file_acquire+0x4e1/0xb80 [nfsd]
>>>>>=20
>>>>> To match that to a line of source code:
>>>>>=20
>>>>> [cel@manet ~]$ cd src/linux/linux/
>>>>> [cel@manet linux]$ scripts/faddr2line ../obj/manet/fs/nfsd/filecache.=
o nfsd_do_file_acquire+0x4e1
>>>>> nfsd_do_file_acquire+0x4e1/0xfc0:
>>>>> rht_bucket_insert at /home/cel/src/linux/linux/include/linux/rhashtab=
le.h:303
>>>>> (inlined by) __rhashtable_insert_fast at /home/cel/src/linux/linux/in=
clude/linux/rhashtable.h:718
>>>>> (inlined by) rhashtable_lookup_get_insert_key at /home/cel/src/linux/=
linux/include/linux/rhashtable.h:982
>>>>> (inlined by) nfsd_file_insert at /home/cel/src/linux/linux/fs/nfsd/fi=
lecache.c:1031
>>>>> (inlined by) nfsd_do_file_acquire at /home/cel/src/linux/linux/fs/nfs=
d/filecache.c:1089
>>>>> [cel@manet linux]$
>>>>>=20
>>>>> This is an example, I'm sure my compiled objects don't match yours.
>>>>>=20
>>>>> And, now that I've added observability, you should be able to do:
>>>>>=20
>>>>> # watch cat /proc/fs/nfsd/filecache
>>>>>=20
>>>>> to see how many items are in the hash and LRU list while the test
>>>>> is running.
>>>>>=20
>>>>>=20
>>>>>> [ 405.608016] Call Trace:
>>>>>> [ 405.608016] <TASK>
>>>>>> [ 405.613020] nfs4_get_vfs_file+0x325/0x410 [nfsd]
>>>>>> [ 405.618018] nfsd4_process_open2+0x4ba/0x16d0 [nfsd]
>>>>>> [ 405.623016] ? inode_get_bytes+0x38/0x40
>>>>>> [ 405.623016] ? nfsd_permission+0x97/0xf0 [nfsd]
>>>>>> [ 405.628022] ? fh_verify+0x1cc/0x6f0 [nfsd]
>>>>>> [ 405.633025] nfsd4_open+0x640/0xb30 [nfsd]
>>>>>> [ 405.638025] nfsd4_proc_compound+0x3bd/0x710 [nfsd]
>>>>>> [ 405.643017] nfsd_dispatch+0x143/0x270 [nfsd]
>>>>>> [ 405.648019] svc_process_common+0x3bf/0x5b0 [sunrpc]
>>>>=20
>>>> I was able to trigger something that looks very much like this crash.
>>>> If you remove this line from fs/nfsd/filecache.c:
>>>>=20
>>>> 	.max_size		=3D 131072, /* buckets */
>>>>=20
>>>> things get a lot more stable for generic/531.
>>>>=20
>>>> I'm looking into the issue now.
>>>=20
>>> Yes. When '.max_size =3D 131072' is removed, fstests generic/531 passed=
.
>>=20
>> Great! Are you comfortable with this general approach for bug #386?
>=20
> It seems a good result for #386.
>=20
> fstests generic/531(file-max: 1M) performance result:
> base(5.19.0-rc3, 12 bits hash, serialized nfsd_file_gc): 222s
> this patchset(.min_size=3D4096): 59s
> so, a good improvement for #386.
>=20
> It seems a good(acceptable) result for #387 too.
> the period of 'text busy(exec directly from the back-end of nfs-server)'
> is about 4s.

I was surprised to learn that NFSv4 doesn't close those files outright,
that they remain in the filecache for a bit. I think we can do better
there, but I haven't looked closely at that yet.

I see that files are left open on the server by crashed NFSv4 clients too.
That will result in the server becoming unusable after significant
uptime, which I've seen on occasion (because I crash my clients a lot)
but never looked into until now.=20


--
Chuck Lever