Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
MIME-Version: 1.0
References: <20220224161705.1041788-1-amir73il@gmail.com> <BB7DEB92-1E3D-4BF5-A723-650C2B95877D@oracle.com>
 <CAOQ4uxgUEn0MpBYH8YU3paeJ5r0n545FuWmzb_yLEyoa1VkVtw@mail.gmail.com>
In-Reply-To: <CAOQ4uxgUEn0MpBYH8YU3paeJ5r0n545FuWmzb_yLEyoa1VkVtw@mail.gmail.com>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Sat, 26 Feb 2022 20:37:12 +0200
Message-ID: <CAOQ4uxiPYpVcL2_YpRFU58kfjPFvTY+Hwz6Z1FNvFGDaupv-cA@mail.gmail.com>
Subject: Re: [PATCH v2] nfsd: more robust allocation failure handling in nfsd_file_cache_init
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Thu, Feb 24, 2022 at 11:39 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Thu, Feb 24, 2022 at 10:41 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
> >
> > Hi Amir-
> >
> > > On Feb 24, 2022, at 11:17 AM, Amir Goldstein <amir73il@gmail.com> wrote:
> > >
> > > The nfsd file cache table can be pretty large and its allocation
> > > may require as many as 80 contigious pages.
> > >
> > > Employ the same fix that was employed for similar issue that was
> > > reported for the reply cache hash table allocation several years ago
> > > by commit 8f97514b423a ("nfsd: more robust allocation failure handling
> > > in nfsd_reply_cache_init").
> > >
> > > Fixes: 65294c1f2c5e ("nfsd: add a new struct file caching facility to nfsd")
> > > Link: https://lore.kernel.org/linux-nfs/e3cdaeec85a6cfec980e87fc294327c0381c1778.camel@kernel.org/
> > > Suggested-by: Jeff Layton <jlayton@kernel.org>
> > > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> > > ---
> > >
> > > Since v1:
> > > - Use kvcalloc()
> > > - Use kvfree()
> > >
> > > fs/nfsd/filecache.c | 6 +++---
> > > 1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > v2 passes some simple testing, so I've applied it to NFSD for-next.
> > It should get 0-day and merge testing and is available for others
> > to try out.
> >
> > I don't have anything that exercises low memory scenarios, though.
> > Do you have anything like this to try?
>
> Well, it is not low memory really it's fragmented memory.
> I would try setting:
>
> CONFIG_FAIL_PAGE_ALLOC=y
>
> echo 5 > /sys/kernel/debug/fail_page_alloc/min-order
> echo 100 > /sys/kernel/debug/fail_page_alloc/probability
>
> and starting (or restarting) nfsd.
> hoping that other large page allocations won't get in the way.
>
> I gave it a shot, but couldn't figure out why nfsd4_files slab
> is still there after stopping nfs-server service, meaning that
> nfsd_file_cache_shutdown() was not called - I must be missing
> something. I may play with this some more tomorrow.
>

Ok, I was missing some parameters.
This configuration reproduces and failure and verified that the
kvcalloc() fix solves the issue:

$ systemctl stop nfs-server
$ echo 5 > /sys/kernel/debug/fail_page_alloc/min-order
$ echo 100 > /sys/kernel/debug/fail_page_alloc/probability
$ echo 1 > /sys/kernel/debug/fail_page_alloc/times
$ echo N > /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait
$ systemctl start nfs-server

[   24.410560] FAULT_INJECTION: forcing a failure.
[   24.410560] name fail_page_alloc, interval 1, probability 100,
space 0, times 1
[   24.413887] CPU: 1 PID: 1218 Comm: rpc.nfsd Not tainted
5.17.0-rc2-xfstests #5927
[   24.415625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.13.0-1ubuntu1.1 04/01/2014
[   24.417098] Call Trace:
[   24.417098]  <TASK>
[   24.417098]  dump_stack_lvl+0x45/0x59
[   24.418999]  should_fail+0x11a/0x13d
[   24.418999]  prepare_alloc_pages.isra.0+0x97/0xc5
[   24.418999]  __alloc_pages+0x76/0x1c7
[   24.418999]  kmalloc_order+0x35/0xa7
[   24.418999]  kmalloc_order_trace+0x1b/0xf3
[   24.418999]  nfsd_file_cache_init+0x5b/0x2d8
[   24.418999]  nfsd_svc+0xcd/0x2b2
[   24.427086]  write_threads+0x6d/0xb5
[   24.427086]  ? get_int+0x70/0x70
[   24.429020]  nfsctl_transaction_write+0x4f/0x67
[   24.429020]  vfs_write+0xe3/0x14b
[   24.429020]  ksys_write+0x7f/0xcb
[   24.429020]  do_syscall_64+0x6d/0x80
[   24.429020]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[   24.429020] RIP: 0033:0x7f29d80d6504
[   24.429020] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f
1f 80 00 00 00 00 48 8d 05 f9 61 0d 00 8b 00 85 c0 75 13 b8 01 00 00
00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89
f5 53
[   24.439028] RSP: 002b:00007ffe867a47f8 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[   24.439028] RAX: ffffffffffffffda RBX: 00007f29d8219560 RCX: 00007f29d80d6504
[   24.442325] RDX: 0000000000000002 RSI: 00007f29d8219560 RDI: 0000000000000003
[   24.442325] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffe867a4557
[   24.445644] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[   24.445644] R13: 0000000000000008 R14: 0000000000000000 R15: 00007f29d83572a0
[   24.449026]  </TASK>
[   24.450496] nfsd: unable to allocate nfsd_file_hashtbl
Job for nfs-server.service canceled.

With the fix patch, nfsd starts despite the injected failure.

Thanks,
Amir.