Hello,
syzbot found the following crash on:
HEAD commit: 6d0a5984 Merge branch 'x86-urgent-for-linus' of git://git...
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15629ab7200000
kernel config: https://syzkaller.appspot.com/x/.config?x=4fb64439e07a1ec0
dashboard link: https://syzkaller.appspot.com/bug?extid=36baa6c2180e959e19b1
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: [email protected]
=====================================
WARNING: bad unlock balance detected!
5.1.0-rc4+ #66 Not tainted
-------------------------------------
syz-executor.1/17297 is trying to release lock (rcu_callback) at:
[<ffffffff815f08d6>] __write_once_size include/linux/compiler.h:220 [inline]
[<ffffffff815f08d6>] __rcu_reclaim kernel/rcu/rcu.h:226 [inline]
[<ffffffff815f08d6>] rcu_do_batch kernel/rcu/tree.c:2475 [inline]
[<ffffffff815f08d6>] invoke_rcu_callbacks kernel/rcu/tree.c:2788 [inline]
[<ffffffff815f08d6>] rcu_core+0x906/0x13a0 kernel/rcu/tree.c:2769
but there are no more locks to release!
other info that might help us debug this:
1 lock held by syz-executor.1/17297:
#0: 00000000248f8dcd (&type->s_umount_key#59/1){+.+.}, at:
alloc_super+0x158/0x890 fs/super.c:228
stack backtrace:
CPU: 0 PID: 17297 Comm: syz-executor.1 Not tainted 5.1.0-rc4+ #66
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x172/0x1f0 lib/dump_stack.c:113
print_unlock_imbalance_bug kernel/locking/lockdep.c:3754 [inline]
print_unlock_imbalance_bug.cold+0x114/0x123 kernel/locking/lockdep.c:3731
__lock_release kernel/locking/lockdep.c:3970 [inline]
lock_release+0x67e/0xa00 kernel/locking/lockdep.c:4230
rcu_lock_release include/linux/rcupdate.h:215 [inline]
__rcu_reclaim kernel/rcu/rcu.h:228 [inline]
rcu_do_batch kernel/rcu/tree.c:2475 [inline]
invoke_rcu_callbacks kernel/rcu/tree.c:2788 [inline]
rcu_core+0x92e/0x13a0 kernel/rcu/tree.c:2769
__do_softirq+0x266/0x95a kernel/softirq.c:293
invoke_softirq kernel/softirq.c:374 [inline]
irq_exit+0x180/0x1d0 kernel/softirq.c:414
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
smp_apic_timer_interrupt+0x14a/0x570 arch/x86/kernel/apic/apic.c:1062
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807
</IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:767
[inline]
RIP: 0010:console_unlock+0xb82/0xed0 kernel/printk/printk.c:2460
Code: 92 88 48 c1 e8 03 42 80 3c 30 00 0f 85 e4 02 00 00 48 83 3d 4f fa 37
07 00 0f 84 91 01 00 00 e8 84 f0 15 00 48 8b 7d 98 57 9d <0f> 1f 44 00 00
e9 6b ff ff ff e8 6f f0 15 00 48 8b 7d 08 c7 05 11
RSP: 0018:ffff88805aa4f8d0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13
RAX: 0000000000040000 RBX: 0000000000000200 RCX: ffffc90008209000
RDX: 00000000000101de RSI: ffffffff815a9c2c RDI: 0000000000000212
RBP: ffff88805aa4f958 R08: ffff888099a66100 R09: fffffbfff11335b9
R10: fffffbfff11335b8 R11: 0000000000000001 R12: 0000000000000000
R13: ffffffff84210cf0 R14: dffffc0000000000 R15: ffffffff88f90710
vprintk_emit+0x280/0x6d0 kernel/printk/printk.c:1975
vprintk_default+0x28/0x30 kernel/printk/printk.c:2002
vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:398
printk+0xba/0xed kernel/printk/printk.c:2035
__ntfs_error.cold+0x91/0xc7 fs/ntfs/debug.c:103
ntfs_fill_super+0x2015/0x3150 fs/ntfs/super.c:2792
mount_bdev+0x307/0x3c0 fs/super.c:1346
ntfs_mount+0x35/0x40 fs/ntfs/super.c:3065
legacy_get_tree+0xf2/0x200 fs/fs_context.c:584
vfs_get_tree+0x123/0x450 fs/super.c:1481
do_new_mount fs/namespace.c:2622 [inline]
do_mount+0x1436/0x2c40 fs/namespace.c:2942
ksys_mount+0xdb/0x150 fs/namespace.c:3151
__do_sys_mount fs/namespace.c:3165 [inline]
__se_sys_mount fs/namespace.c:3162 [inline]
__x64_sys_mount+0xbe/0x150 fs/namespace.c:3162
do_syscall_64+0x103/0x610 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45b69a
Code: b8 a6 00 00 00 0f 05 48 3d 01 f0 ff ff 0f 83 2d 8e fb ff c3 66 2e 0f
1f 84 00 00 00 00 00 66 90 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff
ff 0f 83 0a 8e fb ff c3 66 0f 1f 84 00 00 00 00 00
RSP: 002b:00007f2f99d0ba88 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5
RAX: ffffffffffffffda RBX: 00007f2f99d0bb40 RCX: 000000000045b69a
RDX: 00007f2f99d0bae0 RSI: 0000000020000140 RDI: 00007f2f99d0bb00
RBP: 0000000000000000 R08: 00007f2f99d0bb40 R09: 00007f2f99d0bae0
R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000008
R13: 00000000004c7802 R14: 00000000004dd850 R15: 00000000ffffffff
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot has found a reproducer for the following crash on:
HEAD commit: 0e9d28bc Add linux-next specific files for 20191015
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=11745608e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=3d84ca04228b0bf4
dashboard link: https://syzkaller.appspot.com/bug?extid=36baa6c2180e959e19b1
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159d297f600000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16289b30e00000
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: [email protected]
=====================================
WARNING: bad unlock balance detected!
5.4.0-rc3-next-20191015 #0 Not tainted
-------------------------------------
syz-executor276/8897 is trying to release lock (rcu_callback) at:
[<ffffffff8160e7a4>] __write_once_size include/linux/compiler.h:226 [inline]
[<ffffffff8160e7a4>] __rcu_reclaim kernel/rcu/rcu.h:221 [inline]
[<ffffffff8160e7a4>] rcu_do_batch kernel/rcu/tree.c:2157 [inline]
[<ffffffff8160e7a4>] rcu_core+0x574/0x1560 kernel/rcu/tree.c:2377
but there are no more locks to release!
other info that might help us debug this:
1 lock held by syz-executor276/8897:
#0: ffff88809a3cc0d8 (&type->s_umount_key#40/1){+.+.}, at:
alloc_super+0x158/0x910 fs/super.c:229
stack backtrace:
CPU: 0 PID: 8897 Comm: syz-executor276 Not tainted 5.4.0-rc3-next-20191015
#0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x172/0x1f0 lib/dump_stack.c:113
print_unlock_imbalance_bug kernel/locking/lockdep.c:4008 [inline]
print_unlock_imbalance_bug.cold+0x114/0x123 kernel/locking/lockdep.c:3984
__lock_release kernel/locking/lockdep.c:4244 [inline]
lock_release+0x5f2/0x960 kernel/locking/lockdep.c:4505
rcu_lock_release include/linux/rcupdate.h:213 [inline]
__rcu_reclaim kernel/rcu/rcu.h:223 [inline]
rcu_do_batch kernel/rcu/tree.c:2157 [inline]
rcu_core+0x594/0x1560 kernel/rcu/tree.c:2377
rcu_core_si+0x9/0x10 kernel/rcu/tree.c:2386
__do_softirq+0x262/0x98c kernel/softirq.c:292
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x19b/0x1e0 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
smp_apic_timer_interrupt+0x1a3/0x610 arch/x86/kernel/apic/apic.c:1137
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:830
</IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:756
[inline]
RIP: 0010:console_unlock+0xbb8/0xf00 kernel/printk/printk.c:2481
Code: f3 88 48 c1 e8 03 42 80 3c 30 00 0f 85 e4 02 00 00 48 83 3d 99 9c 96
07 00 0f 84 91 01 00 00 e8 be c4 16 00 48 8b 7d 98 57 9d <0f> 1f 44 00 00
e9 6d ff ff ff e8 a9 c4 16 00 48 8b 7d 08 c7 05 eb
RSP: 0018:ffff88809fd7f8f0 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
RAX: ffff8880a8bee540 RBX: 0000000000000200 RCX: 1ffffffff138eea6
RDX: 0000000000000000 RSI: ffffffff815c8592 RDI: 0000000000000293
RBP: ffff88809fd7f978 R08: ffff8880a8bee540 R09: fffffbfff11f4119
R10: fffffbfff11f4118 R11: 0000000000000001 R12: 0000000000000000
R13: ffffffff843f8ca0 R14: dffffc0000000000 R15: ffffffff895e1130
vprintk_emit+0x2a0/0x700 kernel/printk/printk.c:1996
vprintk_default+0x28/0x30 kernel/printk/printk.c:2023
vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:386
printk+0xba/0xed kernel/printk/printk.c:2056
__ntfs_error.cold+0x91/0xc7 fs/ntfs/debug.c:89
read_ntfs_boot_sector fs/ntfs/super.c:682 [inline]
ntfs_fill_super+0x1ad3/0x3160 fs/ntfs/super.c:2784
mount_bdev+0x304/0x3c0 fs/super.c:1415
ntfs_mount+0x35/0x40 fs/ntfs/super.c:3051
legacy_get_tree+0x108/0x220 fs/fs_context.c:647
vfs_get_tree+0x8e/0x300 fs/super.c:1545
do_new_mount fs/namespace.c:2823 [inline]
do_mount+0x142e/0x1cf0 fs/namespace.c:3143
ksys_mount+0xdb/0x150 fs/namespace.c:3352
__do_sys_mount fs/namespace.c:3366 [inline]
__se_sys_mount fs/namespace.c:3363 [inline]
__x64_sys_mount+0xbe/0x150 fs/namespace.c:3363
do_syscall_64+0xfa/0x760 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4411a9
Code: e8 fc ab 02 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff
ff 0f 83 9b 09 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffffff57cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004411a9
RDX: 0000000020000140 RSI: 0000000020000280 RDI: 00000000200004c0
RBP: 00000000000114e0 R08: 0000000000000000 R09: 00000000004002c8
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000401fd0
R13: 0000000000402060 R14: 0000000000000000 R15: 0000000000000000
Hi,
On Wed, Oct 16, 2019 at 02:27:07AM -0700, syzbot wrote:
> syzbot has found a reproducer for the following crash on:
>
> HEAD commit: 0e9d28bc Add linux-next specific files for 20191015
> git tree: linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=11745608e00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=3d84ca04228b0bf4
> dashboard link: https://syzkaller.appspot.com/bug?extid=36baa6c2180e959e19b1
> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159d297f600000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16289b30e00000
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: [email protected]
>
> =====================================
> WARNING: bad unlock balance detected!
> 5.4.0-rc3-next-20191015 #0 Not tainted
> -------------------------------------
> syz-executor276/8897 is trying to release lock (rcu_callback) at:
> [<ffffffff8160e7a4>] __write_once_size include/linux/compiler.h:226 [inline]
> [<ffffffff8160e7a4>] __rcu_reclaim kernel/rcu/rcu.h:221 [inline]
> [<ffffffff8160e7a4>] rcu_do_batch kernel/rcu/tree.c:2157 [inline]
> [<ffffffff8160e7a4>] rcu_core+0x574/0x1560 kernel/rcu/tree.c:2377
> but there are no more locks to release!
>
> other info that might help us debug this:
> 1 lock held by syz-executor276/8897:
> #0: ffff88809a3cc0d8 (&type->s_umount_key#40/1){+.+.}, at:
> alloc_super+0x158/0x910 fs/super.c:229
>
> stack backtrace:
> CPU: 0 PID: 8897 Comm: syz-executor276 Not tainted 5.4.0-rc3-next-20191015
> #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> Google 01/01/2011
> Call Trace:
> <IRQ>
> __dump_stack lib/dump_stack.c:77 [inline]
> dump_stack+0x172/0x1f0 lib/dump_stack.c:113
> print_unlock_imbalance_bug kernel/locking/lockdep.c:4008 [inline]
> print_unlock_imbalance_bug.cold+0x114/0x123 kernel/locking/lockdep.c:3984
> __lock_release kernel/locking/lockdep.c:4244 [inline]
> lock_release+0x5f2/0x960 kernel/locking/lockdep.c:4505
> rcu_lock_release include/linux/rcupdate.h:213 [inline]
> __rcu_reclaim kernel/rcu/rcu.h:223 [inline]
I have little knowledge about this kind of stuff, but after seeing
the dashboard https://syzkaller.appspot.com/bug?extid=36baa6c2180e959e19b1
I guess this is highly related with ntfs, and in ntfs_fill_super, it
has lockdep_off() in ntfs_fill_super...
In detail, commit 90c1cba2b3b3 ("locking/lockdep: Zap lock classes even
with lock debugging disabled") [1], and free_zapped_rcu....
static void free_zapped_rcu(struct rcu_head *ch)
{
struct pending_free *pf;
unsigned long flags;
if (WARN_ON_ONCE(ch != &delayed_free.rcu_head))
return;
raw_local_irq_save(flags);
arch_spin_lock(&lockdep_lock);
current->lockdep_recursion = 1; <--- here
/* closed head */
pf = delayed_free.pf + (delayed_free.index ^ 1);
__free_zapped_classes(pf);
delayed_free.scheduled = false;
/*
* If there's anything on the open list, close and start a new callback.
*/
call_rcu_zapped(delayed_free.pf + delayed_free.index);
current->lockdep_recursion = 0;
arch_spin_unlock(&lockdep_lock);
raw_local_irq_restore(flags);
}
Completely guess and untest since I am not familar with that,
but in case of that, Cc related people...
If I'm wrong, ignore my comments and unintentional noise....
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=90c1cba2b3b3851c151229f61801919b2904d437
Thanks,
Gao Xiang
> rcu_do_batch kernel/rcu/tree.c:2157 [inline]
> rcu_core+0x594/0x1560 kernel/rcu/tree.c:2377
> rcu_core_si+0x9/0x10 kernel/rcu/tree.c:2386
> __do_softirq+0x262/0x98c kernel/softirq.c:292
> invoke_softirq kernel/softirq.c:373 [inline]
> irq_exit+0x19b/0x1e0 kernel/softirq.c:413
> exiting_irq arch/x86/include/asm/apic.h:536 [inline]
> smp_apic_timer_interrupt+0x1a3/0x610 arch/x86/kernel/apic/apic.c:1137
> apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:830
> </IRQ>
> RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:756
> [inline]
> RIP: 0010:console_unlock+0xbb8/0xf00 kernel/printk/printk.c:2481
> Code: f3 88 48 c1 e8 03 42 80 3c 30 00 0f 85 e4 02 00 00 48 83 3d 99 9c 96
> 07 00 0f 84 91 01 00 00 e8 be c4 16 00 48 8b 7d 98 57 9d <0f> 1f 44 00 00 e9
> 6d ff ff ff e8 a9 c4 16 00 48 8b 7d 08 c7 05 eb
> RSP: 0018:ffff88809fd7f8f0 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
> RAX: ffff8880a8bee540 RBX: 0000000000000200 RCX: 1ffffffff138eea6
> RDX: 0000000000000000 RSI: ffffffff815c8592 RDI: 0000000000000293
> RBP: ffff88809fd7f978 R08: ffff8880a8bee540 R09: fffffbfff11f4119
> R10: fffffbfff11f4118 R11: 0000000000000001 R12: 0000000000000000
> R13: ffffffff843f8ca0 R14: dffffc0000000000 R15: ffffffff895e1130
> vprintk_emit+0x2a0/0x700 kernel/printk/printk.c:1996
> vprintk_default+0x28/0x30 kernel/printk/printk.c:2023
> vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:386
> printk+0xba/0xed kernel/printk/printk.c:2056
> __ntfs_error.cold+0x91/0xc7 fs/ntfs/debug.c:89
> read_ntfs_boot_sector fs/ntfs/super.c:682 [inline]
> ntfs_fill_super+0x1ad3/0x3160 fs/ntfs/super.c:2784
> mount_bdev+0x304/0x3c0 fs/super.c:1415
> ntfs_mount+0x35/0x40 fs/ntfs/super.c:3051
> legacy_get_tree+0x108/0x220 fs/fs_context.c:647
> vfs_get_tree+0x8e/0x300 fs/super.c:1545
> do_new_mount fs/namespace.c:2823 [inline]
> do_mount+0x142e/0x1cf0 fs/namespace.c:3143
> ksys_mount+0xdb/0x150 fs/namespace.c:3352
> __do_sys_mount fs/namespace.c:3366 [inline]
> __se_sys_mount fs/namespace.c:3363 [inline]
> __x64_sys_mount+0xbe/0x150 fs/namespace.c:3363
> do_syscall_64+0xfa/0x760 arch/x86/entry/common.c:290
> entry_SYSCALL_64_after_hwframe+0x49/0xbe
> RIP: 0033:0x4411a9
> Code: e8 fc ab 02 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7
> 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
> 0f 83 9b 09 fc ff c3 66 2e 0f 1f 84 00 00 00 00
> RSP: 002b:00007ffffff57cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
> RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004411a9
> RDX: 0000000020000140 RSI: 0000000020000280 RDI: 00000000200004c0
> RBP: 00000000000114e0 R08: 0000000000000000 R09: 00000000004002c8
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000401fd0
> R13: 0000000000402060 R14: 0000000000000000 R15: 0000000000000000
>
On Wed, Oct 16, 2019 at 11:58 AM Gao Xiang <[email protected]> wrote:
>
> Hi,
>
> On Wed, Oct 16, 2019 at 02:27:07AM -0700, syzbot wrote:
> > syzbot has found a reproducer for the following crash on:
> >
> > HEAD commit: 0e9d28bc Add linux-next specific files for 20191015
> > git tree: linux-next
> > console output: https://syzkaller.appspot.com/x/log.txt?x=11745608e00000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=3d84ca04228b0bf4
> > dashboard link: https://syzkaller.appspot.com/bug?extid=36baa6c2180e959e19b1
> > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159d297f600000
> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16289b30e00000
> >
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: [email protected]
> >
> > =====================================
> > WARNING: bad unlock balance detected!
> > 5.4.0-rc3-next-20191015 #0 Not tainted
> > -------------------------------------
> > syz-executor276/8897 is trying to release lock (rcu_callback) at:
> > [<ffffffff8160e7a4>] __write_once_size include/linux/compiler.h:226 [inline]
> > [<ffffffff8160e7a4>] __rcu_reclaim kernel/rcu/rcu.h:221 [inline]
> > [<ffffffff8160e7a4>] rcu_do_batch kernel/rcu/tree.c:2157 [inline]
> > [<ffffffff8160e7a4>] rcu_core+0x574/0x1560 kernel/rcu/tree.c:2377
> > but there are no more locks to release!
> >
> > other info that might help us debug this:
> > 1 lock held by syz-executor276/8897:
> > #0: ffff88809a3cc0d8 (&type->s_umount_key#40/1){+.+.}, at:
> > alloc_super+0x158/0x910 fs/super.c:229
> >
> > stack backtrace:
> > CPU: 0 PID: 8897 Comm: syz-executor276 Not tainted 5.4.0-rc3-next-20191015
> > #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> > Google 01/01/2011
> > Call Trace:
> > <IRQ>
> > __dump_stack lib/dump_stack.c:77 [inline]
> > dump_stack+0x172/0x1f0 lib/dump_stack.c:113
> > print_unlock_imbalance_bug kernel/locking/lockdep.c:4008 [inline]
> > print_unlock_imbalance_bug.cold+0x114/0x123 kernel/locking/lockdep.c:3984
> > __lock_release kernel/locking/lockdep.c:4244 [inline]
> > lock_release+0x5f2/0x960 kernel/locking/lockdep.c:4505
> > rcu_lock_release include/linux/rcupdate.h:213 [inline]
> > __rcu_reclaim kernel/rcu/rcu.h:223 [inline]
>
> I have little knowledge about this kind of stuff, but after seeing
> the dashboard https://syzkaller.appspot.com/bug?extid=36baa6c2180e959e19b1
>
> I guess this is highly related with ntfs, and in ntfs_fill_super, it
> has lockdep_off() in ntfs_fill_super...
>
> In detail, commit 90c1cba2b3b3 ("locking/lockdep: Zap lock classes even
> with lock debugging disabled") [1], and free_zapped_rcu....
>
> static void free_zapped_rcu(struct rcu_head *ch)
> {
> struct pending_free *pf;
> unsigned long flags;
>
> if (WARN_ON_ONCE(ch != &delayed_free.rcu_head))
> return;
>
> raw_local_irq_save(flags);
> arch_spin_lock(&lockdep_lock);
> current->lockdep_recursion = 1; <--- here
>
> /* closed head */
> pf = delayed_free.pf + (delayed_free.index ^ 1);
> __free_zapped_classes(pf);
> delayed_free.scheduled = false;
>
> /*
> * If there's anything on the open list, close and start a new callback.
> */
> call_rcu_zapped(delayed_free.pf + delayed_free.index);
>
> current->lockdep_recursion = 0;
> arch_spin_unlock(&lockdep_lock);
> raw_local_irq_restore(flags);
> }
>
> Completely guess and untest since I am not familar with that,
> but in case of that, Cc related people...
> If I'm wrong, ignore my comments and unintentional noise....
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=90c1cba2b3b3851c151229f61801919b2904d437
>
> Thanks,
> Gao Xiang
Still happens a lot for the past 10 months:
https://syzkaller.appspot.com/bug?id=0d5bdaf028e4283ad7404609d17e5077f48ff26d
> > rcu_do_batch kernel/rcu/tree.c:2157 [inline]
> > rcu_core+0x594/0x1560 kernel/rcu/tree.c:2377
> > rcu_core_si+0x9/0x10 kernel/rcu/tree.c:2386
> > __do_softirq+0x262/0x98c kernel/softirq.c:292
> > invoke_softirq kernel/softirq.c:373 [inline]
> > irq_exit+0x19b/0x1e0 kernel/softirq.c:413
> > exiting_irq arch/x86/include/asm/apic.h:536 [inline]
> > smp_apic_timer_interrupt+0x1a3/0x610 arch/x86/kernel/apic/apic.c:1137
> > apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:830
> > </IRQ>
> > RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:756
> > [inline]
> > RIP: 0010:console_unlock+0xbb8/0xf00 kernel/printk/printk.c:2481
> > Code: f3 88 48 c1 e8 03 42 80 3c 30 00 0f 85 e4 02 00 00 48 83 3d 99 9c 96
> > 07 00 0f 84 91 01 00 00 e8 be c4 16 00 48 8b 7d 98 57 9d <0f> 1f 44 00 00 e9
> > 6d ff ff ff e8 a9 c4 16 00 48 8b 7d 08 c7 05 eb
> > RSP: 0018:ffff88809fd7f8f0 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
> > RAX: ffff8880a8bee540 RBX: 0000000000000200 RCX: 1ffffffff138eea6
> > RDX: 0000000000000000 RSI: ffffffff815c8592 RDI: 0000000000000293
> > RBP: ffff88809fd7f978 R08: ffff8880a8bee540 R09: fffffbfff11f4119
> > R10: fffffbfff11f4118 R11: 0000000000000001 R12: 0000000000000000
> > R13: ffffffff843f8ca0 R14: dffffc0000000000 R15: ffffffff895e1130
> > vprintk_emit+0x2a0/0x700 kernel/printk/printk.c:1996
> > vprintk_default+0x28/0x30 kernel/printk/printk.c:2023
> > vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:386
> > printk+0xba/0xed kernel/printk/printk.c:2056
> > __ntfs_error.cold+0x91/0xc7 fs/ntfs/debug.c:89
> > read_ntfs_boot_sector fs/ntfs/super.c:682 [inline]
> > ntfs_fill_super+0x1ad3/0x3160 fs/ntfs/super.c:2784
> > mount_bdev+0x304/0x3c0 fs/super.c:1415
> > ntfs_mount+0x35/0x40 fs/ntfs/super.c:3051
> > legacy_get_tree+0x108/0x220 fs/fs_context.c:647
> > vfs_get_tree+0x8e/0x300 fs/super.c:1545
> > do_new_mount fs/namespace.c:2823 [inline]
> > do_mount+0x142e/0x1cf0 fs/namespace.c:3143
> > ksys_mount+0xdb/0x150 fs/namespace.c:3352
> > __do_sys_mount fs/namespace.c:3366 [inline]
> > __se_sys_mount fs/namespace.c:3363 [inline]
> > __x64_sys_mount+0xbe/0x150 fs/namespace.c:3363
> > do_syscall_64+0xfa/0x760 arch/x86/entry/common.c:290
> > entry_SYSCALL_64_after_hwframe+0x49/0xbe
> > RIP: 0033:0x4411a9
> > Code: e8 fc ab 02 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7
> > 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
> > 0f 83 9b 09 fc ff c3 66 2e 0f 1f 84 00 00 00 00
> > RSP: 002b:00007ffffff57cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
> > RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004411a9
> > RDX: 0000000020000140 RSI: 0000000020000280 RDI: 00000000200004c0
> > RBP: 00000000000114e0 R08: 0000000000000000 R09: 00000000004002c8
> > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000401fd0
> > R13: 0000000000402060 R14: 0000000000000000 R15: 0000000000000000
> >
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/20191016100134.GA20076%40architecture4.
On 2020-02-27 07:18, Dmitry Vyukov wrote:
> On Wed, Oct 16, 2019 at 11:58 AM Gao Xiang <[email protected]> wrote:
>>
>> Hi,
>>
>> On Wed, Oct 16, 2019 at 02:27:07AM -0700, syzbot wrote:
>>> syzbot has found a reproducer for the following crash on:
>>>
>>> HEAD commit: 0e9d28bc Add linux-next specific files for 20191015
>>> git tree: linux-next
>>> console output: https://syzkaller.appspot.com/x/log.txt?x=11745608e00000
>>> kernel config: https://syzkaller.appspot.com/x/.config?x=3d84ca04228b0bf4
>>> dashboard link: https://syzkaller.appspot.com/bug?extid=36baa6c2180e959e19b1
>>> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
>>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159d297f600000
>>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16289b30e00000
>>>
>>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>>> Reported-by: [email protected]
>>>
>>> =====================================
>>> WARNING: bad unlock balance detected!
>>> 5.4.0-rc3-next-20191015 #0 Not tainted
>>> -------------------------------------
>>> syz-executor276/8897 is trying to release lock (rcu_callback) at:
>>> [<ffffffff8160e7a4>] __write_once_size include/linux/compiler.h:226 [inline]
>>> [<ffffffff8160e7a4>] __rcu_reclaim kernel/rcu/rcu.h:221 [inline]
>>> [<ffffffff8160e7a4>] rcu_do_batch kernel/rcu/tree.c:2157 [inline]
>>> [<ffffffff8160e7a4>] rcu_core+0x574/0x1560 kernel/rcu/tree.c:2377
>>> but there are no more locks to release!
>>>
>>> other info that might help us debug this:
>>> 1 lock held by syz-executor276/8897:
>>> #0: ffff88809a3cc0d8 (&type->s_umount_key#40/1){+.+.}, at:
>>> alloc_super+0x158/0x910 fs/super.c:229
>>>
>>> stack backtrace:
>>> CPU: 0 PID: 8897 Comm: syz-executor276 Not tainted 5.4.0-rc3-next-20191015
>>> #0
>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>>> Google 01/01/2011
>>> Call Trace:
>>> <IRQ>
>>> __dump_stack lib/dump_stack.c:77 [inline]
>>> dump_stack+0x172/0x1f0 lib/dump_stack.c:113
>>> print_unlock_imbalance_bug kernel/locking/lockdep.c:4008 [inline]
>>> print_unlock_imbalance_bug.cold+0x114/0x123 kernel/locking/lockdep.c:3984
>>> __lock_release kernel/locking/lockdep.c:4244 [inline]
>>> lock_release+0x5f2/0x960 kernel/locking/lockdep.c:4505
>>> rcu_lock_release include/linux/rcupdate.h:213 [inline]
>>> __rcu_reclaim kernel/rcu/rcu.h:223 [inline]
>>
>> I have little knowledge about this kind of stuff, but after seeing
>> the dashboard https://syzkaller.appspot.com/bug?extid=36baa6c2180e959e19b1
>>
>> I guess this is highly related with ntfs, and in ntfs_fill_super, it
>> has lockdep_off() in ntfs_fill_super...
>>
>> In detail, commit 90c1cba2b3b3 ("locking/lockdep: Zap lock classes even
>> with lock debugging disabled") [1], and free_zapped_rcu....
>>
>> static void free_zapped_rcu(struct rcu_head *ch)
>> {
>> struct pending_free *pf;
>> unsigned long flags;
>>
>> if (WARN_ON_ONCE(ch != &delayed_free.rcu_head))
>> return;
>>
>> raw_local_irq_save(flags);
>> arch_spin_lock(&lockdep_lock);
>> current->lockdep_recursion = 1; <--- here
>>
>> /* closed head */
>> pf = delayed_free.pf + (delayed_free.index ^ 1);
>> __free_zapped_classes(pf);
>> delayed_free.scheduled = false;
>>
>> /*
>> * If there's anything on the open list, close and start a new callback.
>> */
>> call_rcu_zapped(delayed_free.pf + delayed_free.index);
>>
>> current->lockdep_recursion = 0;
>> arch_spin_unlock(&lockdep_lock);
>> raw_local_irq_restore(flags);
>> }
>>
>> Completely guess and untest since I am not familar with that,
>> but in case of that, Cc related people...
>> If I'm wrong, ignore my comments and unintentional noise....
>>
>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=90c1cba2b3b3851c151229f61801919b2904d437
>>
>> Thanks,
>> Gao Xiang
>
>
> Still happens a lot for the past 10 months:
> https://syzkaller.appspot.com/bug?id=0d5bdaf028e4283ad7404609d17e5077f48ff26d
Unless one of the NTFS maintainers steps in, should NTFS perhaps be
excluded from testing with lockdep enabled? This is what I found in the
git log of NTFS:
commit 59345374742ee6673c2d04b0fa8c888e881b7209
Author: Ingo Molnar <[email protected]>
Date: Mon Jul 3 00:25:18 2006 -0700
[PATCH] lockdep: annotate NTFS locking rules
NTFS uses lots of type-opaque objects which acquire their true
identity runtime - so the lock validator needs to be helped in a
couple of places to figure out object types.
Many thanks to Anton Altaparmakov for giving lots of explanations
about NTFS locking rules.
Has no effect on non-lockdep kernels.
Thanks,
Bart.
syzbot suspects this bug was fixed by commit:
commit 10476e6304222ced7df9b3d5fb0a043b3c2a1ad8
Author: Peter Zijlstra <[email protected]>
Date: Fri Mar 13 08:56:38 2020 +0000
locking/lockdep: Fix bad recursion pattern
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=142baee4100000
start commit: 5a1e843c Merge tag 'mips_fixes_5.4_3' of git://git.kernel...
git tree: upstream
kernel config: https://syzkaller.appspot.com/x/.config?x=420126a10fdda0f1
dashboard link: https://syzkaller.appspot.com/bug?extid=36baa6c2180e959e19b1
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1108239ce00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13cf40a8e00000
If the result looks correct, please mark the bug fixed by replying with:
#syz fix: locking/lockdep: Fix bad recursion pattern
For information about bisection process see: https://goo.gl/tpsmEJ#bisection