Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Wang Yugui <wangyugui@e16-tech.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "david@fromorbit.com" <david@fromorbit.com>,
        "tgraf@suug.ch" <tgraf@suug.ch>, Jeff Layton <jlayton@redhat.com>
Subject: Re: [PATCH RFC 00/30] Overhaul NFSD filecache
Thread-Topic: [PATCH RFC 00/30] Overhaul NFSD filecache
Thread-Index: AQHYhkIqDsKitGhdzEmSu19Gpo96sa1bwYeAgAAHywCAAA9RgIAA2ruAgACBMAA=
Date:   Thu, 23 Jun 2022 16:44:42 +0000
Message-ID: <0292A2FC-7725-47FC-8F08-CCB8500D8E1D@oracle.com>
References: <FE520DC8-3C8F-4974-9F3B-84DE822CB899@oracle.com>
 <F57A580E-07D0-499A-9693-18D82D73ED3A@oracle.com>
 <20220623170218.7874.409509F4@e16-tech.com>
In-Reply-To: <20220623170218.7874.409509F4@e16-tech.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?opnT4WfC4vy8Vqg7ffzOd0f+b+lAaXZLA+ArITjWYLq1ikYQfKik0LM1skjQ?=
 =?us-ascii?Q?zFtDcAIyHYZ3lYxH/hLKX8G4SBhpsgKGfLlQbgIfGfWjnqz+YaFsOaNz8KQy?=
 =?us-ascii?Q?T7fCK/6et9bMoAfdnR8YxPeqstHpW1wiUWMvPj0bui5MmY8RPbqTfEwai5GZ?=
 =?us-ascii?Q?bIkS9JgY8nOMsWJDG71GrhuYIUs65HLeRLEzhhmqCkZcBrjUfKcJBvp9k7bu?=
 =?us-ascii?Q?koRuBQIsgYL7Q54akLMsFjqjOy2JFhuBqTnNOUb47gSEfMeHuFoTsvCrCFxd?=
 =?us-ascii?Q?6Ec5YKpSvxNi3SzG2EkakYJ+ERaqwRJz/3JijsXFpCsVrdHMv7qkCKIh02Op?=
 =?us-ascii?Q?uIv0fqC0MQ5u6ygxafm8QkQOjD65my0Ol8tUx2EiBP2K/lfYBmTXZLmpUc3z?=
 =?us-ascii?Q?Bnqe0AVi8PY0o+GTqpAVdfHGvg7XvmNz/wNy9nOB12do2VfTeXJa1aDoL0e9?=
 =?us-ascii?Q?rRZro0BbwNcVT5WxWr8k3WQIyVZ6pEkONERXKQbd9f7P7JGoS/b/o4oCG8qP?=
 =?us-ascii?Q?/hTODOUhDHodDGQl3Qonsr1RP+5JNnBPZRRkfhNynRJHic4bZo8V+cZSdYb9?=
 =?us-ascii?Q?50uGGym+4IyiPp0kvjojOTNUTEJhEwrFH1u1mbPCpE+WczO4Gtr7TRqkpVFS?=
 =?us-ascii?Q?zrN32m5M+jXTzj1PnZgmTYSKIw2QZAVRxL66aJr/RrWZ+WnePshIk/zw1Ugg?=
 =?us-ascii?Q?AYDZZdHUZnbZNS6VZ/EVzeWT+E/YbWb80KXUZKuZP1cXKSvLZrcR0zW8gQxX?=
 =?us-ascii?Q?+ZA9Qw6Ted77GRq79gat28Lrj4WtX56GDDCUNZaK25PAsOkNEj8cGNwSb7nw?=
 =?us-ascii?Q?QzL+D68r7BSfcZxxVMNUWFhO6WMFgnJzljVItxSA42Pd5HeXOHgSuSiQ+R9N?=
 =?us-ascii?Q?sw09O0xbcg4cJ/6u0q1nTO0QvKoi26NjBvOFjjBmGyTJxVEsr5guGDL0iP/i?=
 =?us-ascii?Q?GYsFZeafMba+TRU7JAxKnspHxERwgz0cS9823kD2tcXIqT/TNKic5sFxmuP/?=
 =?us-ascii?Q?+BsWrux+kJXLdWyeN6y2HEUFgsoItWh11IW0CUVrsH74ajbcyT0AAlILZGZr?=
 =?us-ascii?Q?o6PuvQ1QXpcVmsZipEP555pfzqkdDOcFMX+9CEXIh49uyQjN96ogbUurKM5G?=
 =?us-ascii?Q?aFHMjkHGbcppxX2uk+EG5fpO6Tkw/DdxoutpqmQYHHAaVETKDQ+738PBoNAw?=
 =?us-ascii?Q?oFhfbGQuL8BD86yiTMvFzqb3RL2/nco2AE0MnIqreyHoGR9E8vTTVGG2EuVN?=
 =?us-ascii?Q?Ij03NHAOdIBjgcz1u85ogJSTiBl29w7nSYnU2Xjho2Y7wssheXts+ehRlxxM?=
 =?us-ascii?Q?6EVLDvX1EBlXOs6uXes3IgEghKNvoOHm+qHVd3lRILLDPgFO6RRNXNhNdXsl?=
 =?us-ascii?Q?IvKDkqYUtf1lykOmdxe88AEXkqDlxAI923HJzPt0QRsqadWSuCLhWcM2f802?=
 =?us-ascii?Q?bfi7xBrMrj8EUveFIbhgLxkipwSWiwyqDKeegk0WwF61WapZxCFzE7+kdJAM?=
 =?us-ascii?Q?ZFY3AsT+VKgMRwckL8D6v4GeeZkgwmBkdoQSfyGL7H+QV7FkKSEhYtvTRgbR?=
 =?us-ascii?Q?7vnkhmSB8vJCckkEOmpsm4sGkHXam7cktUH+bcDIfP11Km3neKKJsvsAYa92?=
 =?us-ascii?Q?JQLl5O+NK3MAyz06/7WFD2tN2zV4MC6MWT4ZggZNkrvLei0pqEzUyXt4cNMu?=
 =?us-ascii?Q?U72kYnLYzhqfBTEFyAADWRnrSkDE+t3k2WB5sRuzh1nDD8TbJmNvBvmDnmoj?=
 =?us-ascii?Q?KA9DhwhOo24e77QTfI1C1hIevfNaSo8=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3581734C1AB870479B38B9C4C18AE776@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DS7PR10MB5134.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 79e06d2d-98e7-4949-6845-08da5537ac62
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jun 2022 16:44:42.9570
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 6qn1J/rl8t1PTsbRI2ir7IvxU00Lf+QD1Tk1vIuvrsq+BzfhXMa1siIas8UX8Sdt5iqw3SidaeSyU/tB+Rd3QQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB5782
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883
 definitions=2022-06-23_06:2022-06-23,2022-06-23 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=369 suspectscore=0
 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2204290000 definitions=main-2206230064
X-Proofpoint-ORIG-GUID: Nsi286nM2kLXAYFm7HOz2WgGe-1wBEPG
X-Proofpoint-GUID: Nsi286nM2kLXAYFm7HOz2WgGe-1wBEPG
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 23, 2022, at 5:02 AM, Wang Yugui <wangyugui@e16-tech.com> wrote:
>=20
> Hi,
>=20
>>> On Jun 22, 2022, at 3:04 PM, Chuck Lever III <chuck.lever@oracle.com> w=
rote:
>>>> On Jun 22, 2022, at 2:36 PM, Wang Yugui <wangyugui@e16-tech.com> wrote=
:
>>>>=20
>>>> Hi,
>>>>=20
>>>> fstests generic/531 triggered a panic on kernel 5.19.0-rc3 with this
>>>> patchset.
>>>=20
>>> As I mention in the cover letter, I haven't tried running generic/531
>>> yet -- no claim at all that this is finished work and that #386 has
>>> been fixed at this point. I'm merely interested in comments on the
>>> general approach.
>>>=20
>>>=20
>>>> [ 405.478056] BUG: kernel NULL pointer dereference, address: 000000000=
0000049
>>>=20
>>> The "RIP: " tells the location of the crash. Notice that the call
>>> trace here does not include that information. From your attachment:
>>>=20
>>> [ 405.518022] RIP: 0010:nfsd_do_file_acquire+0x4e1/0xb80 [nfsd]
>>>=20
>>> To match that to a line of source code:
>>>=20
>>> [cel@manet ~]$ cd src/linux/linux/
>>> [cel@manet linux]$ scripts/faddr2line ../obj/manet/fs/nfsd/filecache.o =
nfsd_do_file_acquire+0x4e1
>>> nfsd_do_file_acquire+0x4e1/0xfc0:
>>> rht_bucket_insert at /home/cel/src/linux/linux/include/linux/rhashtable=
.h:303
>>> (inlined by) __rhashtable_insert_fast at /home/cel/src/linux/linux/incl=
ude/linux/rhashtable.h:718
>>> (inlined by) rhashtable_lookup_get_insert_key at /home/cel/src/linux/li=
nux/include/linux/rhashtable.h:982
>>> (inlined by) nfsd_file_insert at /home/cel/src/linux/linux/fs/nfsd/file=
cache.c:1031
>>> (inlined by) nfsd_do_file_acquire at /home/cel/src/linux/linux/fs/nfsd/=
filecache.c:1089
>>> [cel@manet linux]$
>>>=20
>>> This is an example, I'm sure my compiled objects don't match yours.
>>>=20
>>> And, now that I've added observability, you should be able to do:
>>>=20
>>> # watch cat /proc/fs/nfsd/filecache
>>>=20
>>> to see how many items are in the hash and LRU list while the test
>>> is running.
>>>=20
>>>=20
>>>> [ 405.608016] Call Trace:
>>>> [ 405.608016] <TASK>
>>>> [ 405.613020] nfs4_get_vfs_file+0x325/0x410 [nfsd]
>>>> [ 405.618018] nfsd4_process_open2+0x4ba/0x16d0 [nfsd]
>>>> [ 405.623016] ? inode_get_bytes+0x38/0x40
>>>> [ 405.623016] ? nfsd_permission+0x97/0xf0 [nfsd]
>>>> [ 405.628022] ? fh_verify+0x1cc/0x6f0 [nfsd]
>>>> [ 405.633025] nfsd4_open+0x640/0xb30 [nfsd]
>>>> [ 405.638025] nfsd4_proc_compound+0x3bd/0x710 [nfsd]
>>>> [ 405.643017] nfsd_dispatch+0x143/0x270 [nfsd]
>>>> [ 405.648019] svc_process_common+0x3bf/0x5b0 [sunrpc]
>>=20
>> I was able to trigger something that looks very much like this crash.
>> If you remove this line from fs/nfsd/filecache.c:
>>=20
>> 	.max_size		=3D 131072, /* buckets */
>>=20
>> things get a lot more stable for generic/531.
>>=20
>> I'm looking into the issue now.
>=20
> Yes. When '.max_size =3D 131072' is removed, fstests generic/531 passed.

Great! Are you comfortable with this general approach for bug #386?


--
Chuck Lever