On Fri, Nov 03, 2023 at 09:40:44AM +0000, Daire Byrne wrote:
> Hi,
>
> We have large compute clusters that, amongst other things, spend their
> day mounting & unounting lots of Linux NFS servers via autofs. This
> has worked fine for many years and client kernel versions and was
> working without incident even with our current v6.3.x production
> kernels.
>
> During the v6.6-rc cycle while testing that kernel, I noticed that
> every now and then, the umount/mount would hang randomly and the
> compute host would get stuck and not complete it's work until a
> reboot. I thought I'd wait until v6.6 was released and check again -
> the issue persists.
>
> I have not had the opportunity to test the v6.4 & v6.5 kernels in
> between yet. The stack traces look something like this:
Please do bisection to find the exact commit that introduces your
regression. See Documentation/admin-guide/bug-bisect.rst in the kernel
sources for more information.
>
> [202752.264187] INFO: task umount.nfs:58118 blocked for more than 245 seconds.
> [202752.264237] Tainted: G E 6.6.0-1.dneg.x86_64 #1
Can you reproduce on untainted (vanilla) kernel?
> [202752.264267] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [202752.264296] task:umount.nfs state:D stack:0 pid:58118
> ppid:1 flags:0x00004002
> [202752.264304] Call Trace:
> [202752.264308] <TASK>
> [202752.264313] __schedule+0x30b/0xa10
> [202752.264327] schedule+0x68/0xf0
> [202752.264332] io_schedule+0x16/0x40
> [202752.264337] __folio_lock+0xfc/0x220
> [202752.264346] ? srso_alias_return_thunk+0x5/0x7f
> [202752.264353] ? __pfx_wake_page_function+0x10/0x10
> [202752.264361] truncate_inode_pages_range+0x441/0x460
> [202752.264411] truncate_inode_pages_final+0x41/0x50
> [202752.264425] nfs_evict_inode+0x1a/0x40 [nfs]
> [202752.264476] evict+0xdc/0x190
> [202752.264485] dispose_list+0x4d/0x70
> [202752.264491] evict_inodes+0x16b/0x1b0
> [202752.264499] generic_shutdown_super+0x3e/0x160
> [202752.264507] kill_anon_super+0x17/0x50
> [202752.264513] nfs_kill_super+0x27/0x50 [nfs]
> [202752.264556] deactivate_locked_super+0x35/0x90
> [202752.264562] deactivate_super+0x42/0x50
> [202752.264568] cleanup_mnt+0x109/0x170
> [202752.264574] __cleanup_mnt+0x12/0x20
> [202752.264580] task_work_run+0x61/0x90
> [202752.264588] exit_to_user_mode_prepare+0x1d8/0x200
> [202752.264596] syscall_exit_to_user_mode+0x1c/0x40
> [202752.264603] do_syscall_64+0x48/0x90
> [202752.264609] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> [202752.264617] RIP: 0033:0x7fcb9befeba7
> [202752.264622] RSP: 002b:00007ffdd63ef348 EFLAGS: 00000246 ORIG_RAX:
> 00000000000000a6
> [202752.264628] RAX: 0000000000000000 RBX: 00005561e35da010 RCX:
> 00007fcb9befeba7
> [202752.264632] RDX: 0000000000000001 RSI: 0000000000000000 RDI:
> 00005561e35da1e0
> [202752.264634] RBP: 00005561e35da1e0 R08: 00005561e35dbfa0 R09:
> 00005561e35db790
> [202752.264637] R10: 00007ffdd63eeda0 R11: 0000000000000246 R12:
> 00007fcb9c442d78
> [202752.264640] R13: 0000000000000000 R14: 00005561e35db2c0 R15:
> 00007ffdd63f0dcb
> [202752.264648] </TASK>
>
> [202752.264658] INFO: task mount.nfs:60827 blocked for more than 122 seconds.
> [202752.264686] Tainted: G E 6.6.0-1.dneg.x86_64 #1
> [202752.264713] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [202752.264743] task:mount.nfs state:D stack:0 pid:60827
> ppid:60826 flags:0x00004000
> [202752.264751] Call Trace:
> [202752.264753] <TASK>
> [202752.264757] __schedule+0x30b/0xa10
> [202752.264763] ? srso_alias_return_thunk+0x5/0x7f
> [202752.264771] schedule+0x68/0xf0
> [202752.264776] schedule_preempt_disabled+0x15/0x30
> [202752.264782] rwsem_down_write_slowpath+0x2b3/0x640
> [202752.264788] ? try_to_wake_up+0x242/0x5f0
> [202752.264797] ? __x86_indirect_jump_thunk_r15+0x20/0x20
> [202752.264803] ? wake_up_q+0x50/0x90
> [202752.264809] down_write+0x55/0x70
> [202752.264815] super_lock+0x44/0x130
> [202752.264821] ? kernfs_activate+0x54/0x60
> [202752.264828] ? srso_alias_return_thunk+0x5/0x7f
> [202752.264833] ? kernfs_add_one+0x11f/0x160
> [202752.264841] grab_super+0x2e/0x80
> [202752.264847] grab_super_dead+0x31/0xe0
> [202752.264855] ? srso_alias_return_thunk+0x5/0x7f
> [202752.264860] ? sysfs_create_link_nowarn+0x22/0x40
> [202752.264865] ? srso_alias_return_thunk+0x5/0x7f
> [202752.264871] ? __pfx_nfs_compare_super+0x10/0x10 [nfs]
> [202752.264915] sget_fc+0xd4/0x280
> [202752.264921] ? __pfx_nfs_set_super+0x10/0x10 [nfs]
> [202752.264965] nfs_get_tree_common+0x86/0x520 [nfs]
> [202752.265009] nfs_try_get_tree+0x5c/0x2e0 [nfs]
> [202752.265052] ? srso_alias_return_thunk+0x5/0x7f
> [202752.265058] ? try_module_get+0x1d/0x30
> [202752.265064] ? srso_alias_return_thunk+0x5/0x7f
> [202752.265068] ? get_nfs_version+0x29/0x90 [nfs]
> [202752.265111] ? srso_alias_return_thunk+0x5/0x7f
> [202752.265116] ? nfs_fs_context_validate+0x4fe/0x710 [nfs]
> [202752.265163] nfs_get_tree+0x38/0x60 [nfs]
> [202752.265202] vfs_get_tree+0x2a/0xe0
> [202752.265207] ? capable+0x19/0x20
> [202752.265213] path_mount+0x2fe/0xa90
> [202752.265219] ? putname+0x55/0x70
> [202752.265226] do_mount+0x80/0xa0
> [202752.265233] __x64_sys_mount+0x8b/0xe0
> [202752.265240] do_syscall_64+0x3b/0x90
> [202752.265245] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> [202752.265250] RIP: 0033:0x7fbf5d4ff26a
> [202752.265253] RSP: 002b:00007ffeb24fdd98 EFLAGS: 00000202 ORIG_RAX:
> 00000000000000a5
> [202752.265258] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
> 00007fbf5d4ff26a
> [202752.265261] RDX: 000055c814e78100 RSI: 000055c814e771e0 RDI:
> 000055c814e77320
> [202752.265264] RBP: 00007ffeb24fdfb0 R08: 000055c814e85510 R09:
> 000000000000006d
> [202752.265266] R10: 0000000000000004 R11: 0000000000000202 R12:
> 00007fbf5e2307e0
> [202752.265269] R13: 00007ffeb24fdfb0 R14: 00007ffeb24fde90 R15:
> 000055c814e855a0
> [202752.265277] </TASK>
>
> And like I said, the mount/umount against the server hangs
> indefinitely on the client. It is somewhat interesting that autofs
> still tries to trigger a subsequent mount even though the umount has
> not completed.
>
> The NFS servers are running RHEL8.5 and we are using NFSv3. I also
> reproduced it with a fairly recent nfs-utils-2.6.2 on the client
> compute hosts.
What distro on the client?
>
> Because these happen quite rarely, it takes time and many clients and
> mount/umount cycles to reproduce, so I thought I'd post here before
> working through the bisect testing. If you think this is better as a
> kernel.org bugzilla ticket, I'm happy to do that too.
For now, posting to the ML is preferred as many developers don't take
a look on bugzilla.kernel.org.
Thanks.
--
An old man doll... just what I always wanted! - Clara