Hi,
We would like to report the following bug which has been found by our
modified version of syzkaller.
======================================================
description: possible deadlock in freeze_super
affected file: fs/super.c
kernel version: 5.15.159
kernel commit: a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
git tree: upstream
kernel config: attached
crash reproducer: attached
======================================================
Crash log:
WARNING: possible circular locking dependency detected
5.15.159 #1 Not tainted
------------------------------------------------------
kworker/1:1H/156 is trying to acquire lock:
ffff8881519ec0e0 (&type->s_umount_key#54){+.+.}-{3:3}, at:
freeze_super+0x41/0x3c0 fs/super.c:1682
but task is already holding lock:
ffffc90001aefdb0
((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}, at:
process_one_work+0x8d4/0x1550 kernel/workqueue.c:2285
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 ((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}:
process_one_work+0x92b/0x1550 kernel/workqueue.c:2286
worker_thread+0x65d/0x1130 kernel/workqueue.c:2457
kthread+0x3e5/0x4d0 kernel/kthread.c:334
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300
-> #1 ((wq_completion)glock_workqueue){+.+.}-{0:0}:
flush_workqueue+0x11f/0x1300 kernel/workqueue.c:2830
gfs2_gl_hash_clear+0xad/0x270 fs/gfs2/glock.c:2178
gfs2_put_super+0x493/0x6a0 fs/gfs2/super.c:624
generic_shutdown_super+0x154/0x380 fs/super.c:475
kill_block_super+0x97/0xf0 fs/super.c:1414
gfs2_kill_sb+0x104/0x160 fs/gfs2/ops_fstype.c:1740
deactivate_locked_super+0x94/0x160 fs/super.c:335
deactivate_super+0xad/0xd0 fs/super.c:366
cleanup_mnt+0x3a2/0x540 fs/namespace.c:1143
task_work_run+0xdd/0x1a0 kernel/task_work.c:164
tracehook_notify_resume include/linux/tracehook.h:189 [inline]
exit_to_user_mode_loop kernel/entry/common.c:181 [inline]
exit_to_user_mode_prepare+0x253/0x280 kernel/entry/common.c:214
__syscall_exit_to_user_mode_work kernel/entry/common.c:296 [inline]
syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:307
do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86
entry_SYSCALL_64_after_hwframe+0x66/0xd0
-> #0 (&type->s_umount_key#54){+.+.}-{3:3}:
check_prev_add kernel/locking/lockdep.c:3053 [inline]
check_prevs_add kernel/locking/lockdep.c:3172 [inline]
validate_chain kernel/locking/lockdep.c:3788 [inline]
__lock_acquire+0x2a41/0x5340 kernel/locking/lockdep.c:5012
lock_acquire kernel/locking/lockdep.c:5623 [inline]
lock_acquire+0x1ab/0x4e0 kernel/locking/lockdep.c:5588
down_write+0x92/0x220 kernel/locking/rwsem.c:1551
freeze_super+0x41/0x3c0 fs/super.c:1682
freeze_go_sync+0x1d6/0x320 fs/gfs2/glops.c:584
do_xmote+0x2fd/0xc70 fs/gfs2/glock.c:742
run_queue+0x31f/0x680 fs/gfs2/glock.c:870
glock_work_func+0xff/0x390 fs/gfs2/glock.c:1037
process_one_work+0x9bc/0x1550 kernel/workqueue.c:2310
worker_thread+0x65d/0x1130 kernel/workqueue.c:2457
kthread+0x3e5/0x4d0 kernel/kthread.c:334
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300
other info that might help us debug this:
Chain exists of:
&type->s_umount_key#54 --> (wq_completion)glock_workqueue -->
(work_completion)(&(&gl->gl_work)->work)
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock((work_completion)(&(&gl->gl_work)->work));
lock((wq_completion)glock_workqueue);
lock((work_completion)(&(&gl->gl_work)->work));
lock(&type->s_umount_key#54);
*** DEADLOCK ***
2 locks held by kworker/1:1H/156:
#0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
#0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41
[inline]
#0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1198
[inline]
#0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: set_work_data kernel/workqueue.c:635 [inline]
#0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: set_work_pool_and_clear_pending kernel/workqueue.c:662 [inline]
#0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: process_one_work+0x8a0/0x1550 kernel/workqueue.c:2281
#1: ffffc90001aefdb0
((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}, at:
process_one_work+0x8d4/0x1550 kernel/workqueue.c:2285
stack backtrace:
CPU: 1 PID: 156 Comm: kworker/1:1H Not tainted 5.15.159 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Workqueue: glock_workqueue glock_work_func
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
check_noncircular+0x268/0x310 kernel/locking/lockdep.c:2133
check_prev_add kernel/locking/lockdep.c:3053 [inline]
check_prevs_add kernel/locking/lockdep.c:3172 [inline]
validate_chain kernel/locking/lockdep.c:3788 [inline]
__lock_acquire+0x2a41/0x5340 kernel/locking/lockdep.c:5012
lock_acquire kernel/locking/lockdep.c:5623 [inline]
lock_acquire+0x1ab/0x4e0 kernel/locking/lockdep.c:5588
down_write+0x92/0x220 kernel/locking/rwsem.c:1551
freeze_super+0x41/0x3c0 fs/super.c:1682
freeze_go_sync+0x1d6/0x320 fs/gfs2/glops.c:584
do_xmote+0x2fd/0xc70 fs/gfs2/glock.c:742
run_queue+0x31f/0x680 fs/gfs2/glock.c:870
glock_work_func+0xff/0x390 fs/gfs2/glock.c:1037
process_one_work+0x9bc/0x1550 kernel/workqueue.c:2310
worker_thread+0x65d/0x1130 kernel/workqueue.c:2457
kthread+0x3e5/0x4d0 kernel/kthread.c:334
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300
</TASK>
======================================================
Wishing you a nice day!
Best,
Marius
On Mon, Jun 10, 2024 at 02:52:36PM -0700, Marius Fleischer wrote:
> Hi,
>
> We would like to report the following bug which has been found by our
> modified version of syzkaller.
>
> ======================================================
> description: possible deadlock in freeze_super
> affected file: fs/super.c
> kernel version: 5.15.159
> kernel commit: a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
I'm sorry but this is really inconsistent information.
The kernel version points to 5.15.159. The commit referenced right after
is for v6.9 though and it goes on...
> freeze_go_sync+0x1d6/0x320 fs/gfs2/glops.c:584
That function doesn't exist on v6.9 and has been removed in
commit b77b4a4815a9 ("gfs2: Rework freeze / thaw logic") which fixes the
deadlock you supposedly detected. So, this must be from a kernel prior
to v6.5.
In general we will ignore bug reports from automated tools that are not
the official syzkaller instance because stuff like this is stealing time
spent actually writing code or fixing bugs.
Hi Christian,
Thanks for your response!
> > ======================================================
> > description: possible deadlock in freeze_super
> > affected file: fs/super.c
> > kernel version: 5.15.159
> > kernel commit: a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
>
> I'm sorry but this is really inconsistent information.
> The kernel version points to 5.15.159. The commit referenced right after
> is for v6.9 though and it goes on...
I'm very sorry for the mistake in my previous email. The kernel
version is correct but the commit hash is not. It should have
been83655231580bc07485a4ac2a6c971c3a175dd27d.
>
> > freeze_go_sync+0x1d6/0x320 fs/gfs2/glops.c:584
>
> That function doesn't exist on v6.9 and has been removed in
> commit b77b4a4815a9 ("gfs2: Rework freeze / thaw logic") which fixes the
> deadlock you supposedly detected. So, this must be from a kernel prior
> to v6.5.
>
Do you think it might be worth it to port the patch that you mentioned
back to v5.15?
Wishing you a nice day!
Best,
Marius