Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Amir Goldstein <amir73il@gmail.com>
CC:     Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v2] nfsd: more robust allocation failure handling in
 nfsd_file_cache_init
Thread-Topic: [PATCH v2] nfsd: more robust allocation failure handling in
 nfsd_file_cache_init
Thread-Index: AQHYKZoEe5El1hMKYEiEq+uh3pv5TKyjI9OAgAAW+wCAAvG6AIAAF1QA
Date:   Sat, 26 Feb 2022 20:00:43 +0000
Message-ID: <FDA50FCC-FBC9-4B5E-89E3-804C8FD32763@oracle.com>
References: <20220224161705.1041788-1-amir73il@gmail.com>
 <BB7DEB92-1E3D-4BF5-A723-650C2B95877D@oracle.com>
 <CAOQ4uxgUEn0MpBYH8YU3paeJ5r0n545FuWmzb_yLEyoa1VkVtw@mail.gmail.com>
 <CAOQ4uxiPYpVcL2_YpRFU58kfjPFvTY+Hwz6Z1FNvFGDaupv-cA@mail.gmail.com>
In-Reply-To: <CAOQ4uxiPYpVcL2_YpRFU58kfjPFvTY+Hwz6Z1FNvFGDaupv-cA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?OUcKIntCalC8BBodfUQOftoOvEY5QTfSNtea6YIQjYUpFwdAAMHy8ZnXFPSq?=
 =?us-ascii?Q?jMd7hbu9Ws5su97bx8yUf48mOq1HouIVfbF4At8pyXFH8cEmXSlhxlDJCYxj?=
 =?us-ascii?Q?6Msj1VPekH+Nod7EtCBG0LBDmLWKaqSjjfHhd0oaJZFqElJi58MZsowlLYuX?=
 =?us-ascii?Q?eIxGrq1I8Tr5aXwBNFEvpDRcw9NUzSglq6qiPRM/qJJRPrFXvuS4isT8ce8S?=
 =?us-ascii?Q?9O6eg1kGvFkgRXLu0/wxS6z2uArV+D/uu7u8PvkJGlgfGHixGYUWdCbN9sbh?=
 =?us-ascii?Q?fcICQLEJOFyhtmIzWP6sVxB05/rRukUVHrPK078OytfmW0Ha4Oul/+9sGdlv?=
 =?us-ascii?Q?UuqWDInXuGwxiR7DqJhp6r7Izcq6KfZGxW6YnigHaAXKRp4UF0NGVHsHKnbT?=
 =?us-ascii?Q?5ODULr1DMeeHZ/bwfhIpOfnSZTjICaKJWgjY10ZnWaepM5ka8AYkG5MIokc7?=
 =?us-ascii?Q?fTnnIqLejIUyJrRrLYhxKPZgNWHh+DZIdNUPW7TT0EAkm8hGmjUVezzuBoNE?=
 =?us-ascii?Q?eN7VIpU1WNiw4xPnxUgBqM3vHF/Dm+fh6MqLx3q+H3TUWmVdTMbx9eTDvCV1?=
 =?us-ascii?Q?nSY1U1bZxyImqHvOBg/+pEpsyKDNXu8+HNwiA+oICquQ5INelius5cF2eLZE?=
 =?us-ascii?Q?9fh/8qm1vOm46da3yhWy5LG52XO1GkiFYug+KFUwoLVyAwCcE5n1EONJzqR7?=
 =?us-ascii?Q?2yDn3rSrqtDgB0CU+AK7QbK+rOx5gf60GsP1n3hPusLwz8AmVHXxxXRzORUI?=
 =?us-ascii?Q?2RGssUbgbB+RQz/gPcrLLtTku87fKnoFjLDDMQ079h/UhAnWSt/JqNs/MvWP?=
 =?us-ascii?Q?IeE7rFBb66yS+rWVv6qlVDIta6/mhPkmZXz9vU1DfO8LAle7EnJvKWAa9WOg?=
 =?us-ascii?Q?BQyObOBrERt9iAroCur+PLX/UL4sGVunF+hn/x/rbrnValreoDfnO5WDuBKp?=
 =?us-ascii?Q?gkL/9MyzjWpXNXCgkriwXeyU/qs5wgeOWkFOl2s3o2sMR+PxMtmGu4M6siC5?=
 =?us-ascii?Q?FVquZZVx3QTy/NmAmw/pz5XnIsBFJV09pOZtrQnsCkMJdHyqh/w5+i0EwjdH?=
 =?us-ascii?Q?66kJhvsg3OdR+Hdy8LRyUU05DDxNgYPI7+lOlLZ7wNTss2Enir3HoQIoIyZD?=
 =?us-ascii?Q?w22IO2rH/RSUsVVke8LnfrOqvyiD8tQ8alWMfyPnkUyr+5Idlppv8ZcJuK9a?=
 =?us-ascii?Q?HihVkS2wsw2Zi/eHXY9jqsYQ5wvvTGVEEaybKB8l2phYbEEQj3/BF3vnQhNl?=
 =?us-ascii?Q?yN6W16vxy444xCwkKY75SVaKK58Gdz7Sz7QSM5CpjrlIuoNolvJMIvuUZkis?=
 =?us-ascii?Q?20znoyE/3YEm+RVgFlAXqFJWJbpK0gaLdaon8wV6NkiRxM2bX9fkJfwemrdn?=
 =?us-ascii?Q?mtNwpca35ny5iMq3YUjBUj0nA51PhCPRNKDgK+P7Bg+F+ovJp1eQjeW2f6AQ?=
 =?us-ascii?Q?0PeaDO7Kz8O/exewAra4IpP173MGZrfzQB7yT5ohkYGU5GJMF3U79HtwKvbU?=
 =?us-ascii?Q?UCRz+aeOYNp0TpZN7YyvojCfvA2ySnhEwT8/gKOJnmkbNALVbh11pAGZqGH4?=
 =?us-ascii?Q?xpvJigdMgdbkFfPglOFMmpGxwlWCTYFjll0SYv9Sd35u85NRCDKzGx0R0UzD?=
 =?us-ascii?Q?nEMzBzlJy19GCR6YEb8Eam0=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <73BEF9F6963995489A1185DB3EE7F4F2@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ff5d65c9-4a62-4780-522e-08d9f962abd9
X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Feb 2022 20:00:43.4580
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: O9Y0nPfHnzybJUh+y3GlbzXcMVfZPPTKNGLSUp1Dj+15D+NEQf5lJ3mPb3BRHV1bv48pTgfRjHkFKo20LdY6HA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2509
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10270 signatures=684655
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0
 adultscore=0 phishscore=0 spamscore=0 suspectscore=0 malwarescore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2201110000 definitions=main-2202260142
X-Proofpoint-GUID: 655mxhb_Xcm6oY-CnmbnTsGFNXv3Kag4
X-Proofpoint-ORIG-GUID: 655mxhb_Xcm6oY-CnmbnTsGFNXv3Kag4
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Feb 26, 2022, at 1:37 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>=20
> On Thu, Feb 24, 2022 at 11:39 PM Amir Goldstein <amir73il@gmail.com> wrot=
e:
>>=20
>> On Thu, Feb 24, 2022 at 10:41 PM Chuck Lever III <chuck.lever@oracle.com=
> wrote:
>>>=20
>>> Hi Amir-
>>>=20
>>>> On Feb 24, 2022, at 11:17 AM, Amir Goldstein <amir73il@gmail.com> wrot=
e:
>>>>=20
>>>> The nfsd file cache table can be pretty large and its allocation
>>>> may require as many as 80 contigious pages.
>>>>=20
>>>> Employ the same fix that was employed for similar issue that was
>>>> reported for the reply cache hash table allocation several years ago
>>>> by commit 8f97514b423a ("nfsd: more robust allocation failure handling
>>>> in nfsd_reply_cache_init").
>>>>=20
>>>> Fixes: 65294c1f2c5e ("nfsd: add a new struct file caching facility to =
nfsd")
>>>> Link: https://lore.kernel.org/linux-nfs/e3cdaeec85a6cfec980e87fc294327=
c0381c1778.camel@kernel.org/
>>>> Suggested-by: Jeff Layton <jlayton@kernel.org>
>>>> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>>>> ---
>>>>=20
>>>> Since v1:
>>>> - Use kvcalloc()
>>>> - Use kvfree()
>>>>=20
>>>> fs/nfsd/filecache.c | 6 +++---
>>>> 1 file changed, 3 insertions(+), 3 deletions(-)
>>>=20
>>> v2 passes some simple testing, so I've applied it to NFSD for-next.
>>> It should get 0-day and merge testing and is available for others
>>> to try out.
>>>=20
>>> I don't have anything that exercises low memory scenarios, though.
>>> Do you have anything like this to try?
>>=20
>> Well, it is not low memory really it's fragmented memory.
>> I would try setting:
>>=20
>> CONFIG_FAIL_PAGE_ALLOC=3Dy
>>=20
>> echo 5 > /sys/kernel/debug/fail_page_alloc/min-order
>> echo 100 > /sys/kernel/debug/fail_page_alloc/probability
>>=20
>> and starting (or restarting) nfsd.
>> hoping that other large page allocations won't get in the way.
>>=20
>> I gave it a shot, but couldn't figure out why nfsd4_files slab
>> is still there after stopping nfs-server service, meaning that
>> nfsd_file_cache_shutdown() was not called - I must be missing
>> something. I may play with this some more tomorrow.
>>=20
>=20
> Ok, I was missing some parameters.
> This configuration reproduces and failure and verified that the
> kvcalloc() fix solves the issue:
>=20
> $ systemctl stop nfs-server
> $ echo 5 > /sys/kernel/debug/fail_page_alloc/min-order
> $ echo 100 > /sys/kernel/debug/fail_page_alloc/probability
> $ echo 1 > /sys/kernel/debug/fail_page_alloc/times
> $ echo N > /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait
> $ systemctl start nfs-server
>=20
> [   24.410560] FAULT_INJECTION: forcing a failure.
> [   24.410560] name fail_page_alloc, interval 1, probability 100,
> space 0, times 1
> [   24.413887] CPU: 1 PID: 1218 Comm: rpc.nfsd Not tainted
> 5.17.0-rc2-xfstests #5927
> [   24.415625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS 1.13.0-1ubuntu1.1 04/01/2014
> [   24.417098] Call Trace:
> [   24.417098]  <TASK>
> [   24.417098]  dump_stack_lvl+0x45/0x59
> [   24.418999]  should_fail+0x11a/0x13d
> [   24.418999]  prepare_alloc_pages.isra.0+0x97/0xc5
> [   24.418999]  __alloc_pages+0x76/0x1c7
> [   24.418999]  kmalloc_order+0x35/0xa7
> [   24.418999]  kmalloc_order_trace+0x1b/0xf3
> [   24.418999]  nfsd_file_cache_init+0x5b/0x2d8
> [   24.418999]  nfsd_svc+0xcd/0x2b2
> [   24.427086]  write_threads+0x6d/0xb5
> [   24.427086]  ? get_int+0x70/0x70
> [   24.429020]  nfsctl_transaction_write+0x4f/0x67
> [   24.429020]  vfs_write+0xe3/0x14b
> [   24.429020]  ksys_write+0x7f/0xcb
> [   24.429020]  do_syscall_64+0x6d/0x80
> [   24.429020]  entry_SYSCALL_64_after_hwframe+0x44/0xae
> [   24.429020] RIP: 0033:0x7f29d80d6504
> [   24.429020] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f
> 1f 80 00 00 00 00 48 8d 05 f9 61 0d 00 8b 00 85 c0 75 13 b8 01 00 00
> 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89
> f5 53
> [   24.439028] RSP: 002b:00007ffe867a47f8 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000001
> [   24.439028] RAX: ffffffffffffffda RBX: 00007f29d8219560 RCX: 00007f29d=
80d6504
> [   24.442325] RDX: 0000000000000002 RSI: 00007f29d8219560 RDI: 000000000=
0000003
> [   24.442325] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffe8=
67a4557
> [   24.445644] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000=
0000000
> [   24.445644] R13: 0000000000000008 R14: 0000000000000000 R15: 00007f29d=
83572a0
> [   24.449026]  </TASK>
> [   24.450496] nfsd: unable to allocate nfsd_file_hashtbl
> Job for nfs-server.service canceled.
>=20
> With the fix patch, nfsd starts despite the injected failure.

Thanks. I will add your Tested-by: .


--
Chuck Lever