Hello again
Three weeks ago we reported on nfsd D state-induced freezes on our
bookworm-upgraded files servers [1]. The general consensus at the time seems
to have been that the real issue was deeper in our storage stack, so we headed
over to linux-block but were never able to pinpoint the issue.
We just had another instance where all our 64 nfsd processes were stuck in D
state. This time the stack traces look different and we have some more hints
in our logs, and this time we're pretty sure it's nfsd and not general block
IO.
All 64 nfds have similiar stack traces:
14 processes:
[<0>] __flush_workqueue+0x152/0x420
[<0>] nfsd4_shutdown_callback+0x49/0x130 [nfsd]
[<0>] __destroy_client+0x1f3/0x290 [nfsd]
[<0>] nfsd4_exchange_id+0x752/0x760 [nfsd]
[<0>] nfsd4_proc_compound+0x352/0x660 [nfsd]
[<0>] nfsd_dispatch+0x167/0x280 [nfsd]
[<0>] svc_process_common+0x286/0x5e0 [sunrpc]
[<0>] svc_process+0xad/0x100 [sunrpc]
[<0>] nfsd+0xd5/0x190 [nfsd]
[<0>] kthread+0xe6/0x110
[<0>] ret_from_fork+0x1f/0x30
9 processes:
[<0>] __flush_workqueue+0x152/0x420
[<0>] nfsd4_shutdown_callback+0x49/0x130 [nfsd]
[<0>] __destroy_client+0x1f3/0x290 [nfsd]
[<0>] nfsd4_exchange_id+0x358/0x760 [nfsd]
[<0>] nfsd4_proc_compound+0x352/0x660 [nfsd]
[<0>] nfsd_dispatch+0x167/0x280 [nfsd]
[<0>] svc_process_common+0x286/0x5e0 [sunrpc]
[<0>] svc_process+0xad/0x100 [sunrpc]
[<0>] nfsd+0xd5/0x190 [nfsd]
[<0>] kthread+0xe6/0x110
[<0>] ret_from_fork+0x1f/0x30
41 processes:
[<0>] __flush_workqueue+0x152/0x420
[<0>] nfsd4_destroy_session+0x1b6/0x250 [nfsd]
[<0>] nfsd4_proc_compound+0x352/0x660 [nfsd]
[<0>] nfsd_dispatch+0x167/0x280 [nfsd]
[<0>] svc_process_common+0x286/0x5e0 [sunrpc]
[<0>] svc_process+0xad/0x100 [sunrpc]
[<0>] nfsd+0xd5/0x190 [nfsd]
[<0>] kthread+0xe6/0x110
[<0>] ret_from_fork+0x1f/0x30
20 minutes prior to the first frozen nfsds, we saw messages similiar to
receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt
00000000fcdd40ac xid 182df75c
It seems these messages come from receive_cb_reply [2] and it looks like
xprt_lookup_rqst cannot find the RPC request beloning to a certain
transaction. We see these messages with different values for xpt_bc_xprt,
which, we think, correspond to the different NFS clients.
All this is on production file servers running Debian bookworm with iSCSI
block devices and XFS file systems.
Does anyone have any suggestions how to further debug this? Unfortunately we
have yet to find a way to trigger it deliberately, for the time being it
happens whenever it happens....
thanks and best regards,
-Christian
[1] https://www.spinics.net/lists/linux-nfs/msg96048.html
[2] https://elixir.bootlin.com/linux/v6.1.20/source/net/sunrpc/svcsock.c#L902
--
Dr. Christian Herzog <[email protected]> support: +41 44 633 26 68
Head, IT Services Group, HPT H 8 voice: +41 44 633 39 50
Department of Physics, ETH Zurich
8093 Zurich, Switzerland http://isg.phys.ethz.ch/