2021-02-10 11:33:04

by syzbot

[permalink] [raw]
Subject: possible deadlock in dquot_commit

Hello,

syzbot found the following issue on:

HEAD commit: 1e0d27fc Merge branch 'akpm' (patches from Andrew)
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=101cf2f8d00000
kernel config: https://syzkaller.appspot.com/x/.config?x=e83e68d0a6aba5f6
dashboard link: https://syzkaller.appspot.com/bug?extid=3b6f9218b1301ddda3e2

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]

loop1: detected capacity change from 4096 to 0
EXT4-fs (loop1): mounted filesystem without journal. Opts: ,errors=continue. Quota mode: writeback.
======================================================
WARNING: possible circular locking dependency detected
5.11.0-rc6-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.1/16170 is trying to acquire lock:
ffff8880795f5b28 (&dquot->dq_lock){+.+.}-{3:3}, at: dquot_commit+0x4d/0x420 fs/quota/dquot.c:476

but task is already holding lock:
ffff88807960b438 (&ei->i_data_sem/2){++++}-{3:3}, at: ext4_map_blocks+0x5e1/0x17d0 fs/ext4/inode.c:630

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&ei->i_data_sem/2){++++}-{3:3}:
down_read+0x95/0x440 kernel/locking/rwsem.c:1353
ext4_map_blocks+0x381/0x17d0 fs/ext4/inode.c:560
ext4_getblk+0x13c/0x670 fs/ext4/inode.c:847
ext4_bread+0x29/0x210 fs/ext4/inode.c:899
ext4_quota_write+0x26b/0x670 fs/ext4/super.c:6557
write_blk+0x12e/0x220 fs/quota/quota_tree.c:73
get_free_dqblk+0xff/0x2d0 fs/quota/quota_tree.c:102
do_insert_tree+0x79c/0x1180 fs/quota/quota_tree.c:309
do_insert_tree+0xf77/0x1180 fs/quota/quota_tree.c:340
do_insert_tree+0xf77/0x1180 fs/quota/quota_tree.c:340
do_insert_tree+0xf77/0x1180 fs/quota/quota_tree.c:340
dq_insert_tree fs/quota/quota_tree.c:366 [inline]
qtree_write_dquot+0x3b7/0x580 fs/quota/quota_tree.c:385
v2_write_dquot+0x11c/0x250 fs/quota/quota_v2.c:353
dquot_acquire+0x2c5/0x590 fs/quota/dquot.c:443
ext4_acquire_dquot+0x254/0x3b0 fs/ext4/super.c:6216
dqget+0x678/0x1080 fs/quota/dquot.c:901
__dquot_initialize+0x560/0xbe0 fs/quota/dquot.c:1479
ext4_create+0x8b/0x4c0 fs/ext4/namei.c:2606
lookup_open.isra.0+0xf85/0x1350 fs/namei.c:3106
open_last_lookups fs/namei.c:3180 [inline]
path_openat+0x96d/0x2730 fs/namei.c:3368
do_filp_open+0x17e/0x3c0 fs/namei.c:3398
do_sys_openat2+0x16d/0x420 fs/open.c:1172
do_sys_open fs/open.c:1188 [inline]
__do_sys_creat fs/open.c:1262 [inline]
__se_sys_creat fs/open.c:1256 [inline]
__x64_sys_creat+0xc9/0x120 fs/open.c:1256
do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
entry_SYSCALL_64_after_hwframe+0x44/0xa9

-> #1 (&s->s_dquot.dqio_sem){++++}-{3:3}:
down_read+0x95/0x440 kernel/locking/rwsem.c:1353
v2_read_dquot+0x49/0x120 fs/quota/quota_v2.c:327
dquot_acquire+0x12e/0x590 fs/quota/dquot.c:434
ext4_acquire_dquot+0x254/0x3b0 fs/ext4/super.c:6216
dqget+0x678/0x1080 fs/quota/dquot.c:901
__dquot_initialize+0x560/0xbe0 fs/quota/dquot.c:1479
ext4_create+0x8b/0x4c0 fs/ext4/namei.c:2606
lookup_open.isra.0+0xf85/0x1350 fs/namei.c:3106
open_last_lookups fs/namei.c:3180 [inline]
path_openat+0x96d/0x2730 fs/namei.c:3368
do_filp_open+0x17e/0x3c0 fs/namei.c:3398
do_sys_openat2+0x16d/0x420 fs/open.c:1172
do_sys_open fs/open.c:1188 [inline]
__do_sys_creat fs/open.c:1262 [inline]
__se_sys_creat fs/open.c:1256 [inline]
__x64_sys_creat+0xc9/0x120 fs/open.c:1256
do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
entry_SYSCALL_64_after_hwframe+0x44/0xa9

-> #0 (&dquot->dq_lock){+.+.}-{3:3}:
check_prev_add kernel/locking/lockdep.c:2868 [inline]
check_prevs_add kernel/locking/lockdep.c:2993 [inline]
validate_chain kernel/locking/lockdep.c:3608 [inline]
__lock_acquire+0x2b26/0x54f0 kernel/locking/lockdep.c:4832
lock_acquire kernel/locking/lockdep.c:5442 [inline]
lock_acquire+0x1a8/0x720 kernel/locking/lockdep.c:5407
__mutex_lock_common kernel/locking/mutex.c:956 [inline]
__mutex_lock+0x134/0x1110 kernel/locking/mutex.c:1103
dquot_commit+0x4d/0x420 fs/quota/dquot.c:476
ext4_write_dquot+0x24e/0x310 fs/ext4/super.c:6200
ext4_mark_dquot_dirty fs/ext4/super.c:6248 [inline]
ext4_mark_dquot_dirty+0x111/0x1b0 fs/ext4/super.c:6242
mark_dquot_dirty fs/quota/dquot.c:347 [inline]
mark_all_dquot_dirty fs/quota/dquot.c:385 [inline]
__dquot_alloc_space+0x5d4/0xb60 fs/quota/dquot.c:1709
dquot_alloc_space_nodirty include/linux/quotaops.h:297 [inline]
dquot_alloc_space include/linux/quotaops.h:310 [inline]
dquot_alloc_block include/linux/quotaops.h:334 [inline]
ext4_mb_new_blocks+0x5a9/0x51a0 fs/ext4/mballoc.c:4937
ext4_ext_map_blocks+0x20da/0x5fb0 fs/ext4/extents.c:4238
ext4_map_blocks+0x653/0x17d0 fs/ext4/inode.c:637
_ext4_get_block+0x241/0x590 fs/ext4/inode.c:793
ext4_block_write_begin+0x4f8/0x1190 fs/ext4/inode.c:1077
ext4_write_begin+0x4b5/0x14b0 fs/ext4/inode.c:1202
ext4_da_write_begin+0x672/0x1150 fs/ext4/inode.c:2961
generic_perform_write+0x20a/0x4f0 mm/filemap.c:3412
ext4_buffered_write_iter+0x244/0x4d0 fs/ext4/file.c:270
ext4_file_write_iter+0x423/0x14d0 fs/ext4/file.c:664
call_write_iter include/linux/fs.h:1901 [inline]
new_sync_write+0x426/0x650 fs/read_write.c:518
vfs_write+0x791/0xa30 fs/read_write.c:605
ksys_write+0x12d/0x250 fs/read_write.c:658
do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
entry_SYSCALL_64_after_hwframe+0x44/0xa9

other info that might help us debug this:

Chain exists of:
&dquot->dq_lock --> &s->s_dquot.dqio_sem --> &ei->i_data_sem/2

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&ei->i_data_sem/2);
lock(&s->s_dquot.dqio_sem);
lock(&ei->i_data_sem/2);
lock(&dquot->dq_lock);

*** DEADLOCK ***

5 locks held by syz-executor.1/16170:
#0: ffff88802ad18b70 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xe9/0x100 fs/file.c:947
#1: ffff88802fbec460 (sb_writers#5){.+.+}-{0:0}, at: ksys_write+0x12d/0x250 fs/read_write.c:658
#2: ffff88807960b648 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: inode_lock include/linux/fs.h:773 [inline]
#2: ffff88807960b648 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: ext4_buffered_write_iter+0xb6/0x4d0 fs/ext4/file.c:264
#3: ffff88807960b438 (&ei->i_data_sem/2){++++}-{3:3}, at: ext4_map_blocks+0x5e1/0x17d0 fs/ext4/inode.c:630
#4: ffffffff8bf1be58 (dquot_srcu){....}-{0:0}, at: i_dquot fs/quota/dquot.c:926 [inline]
#4: ffffffff8bf1be58 (dquot_srcu){....}-{0:0}, at: __dquot_alloc_space+0x1b4/0xb60 fs/quota/dquot.c:1671

stack backtrace:
CPU: 0 PID: 16170 Comm: syz-executor.1 Not tainted 5.11.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:79 [inline]
dump_stack+0x107/0x163 lib/dump_stack.c:120
check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2117
check_prev_add kernel/locking/lockdep.c:2868 [inline]
check_prevs_add kernel/locking/lockdep.c:2993 [inline]
validate_chain kernel/locking/lockdep.c:3608 [inline]
__lock_acquire+0x2b26/0x54f0 kernel/locking/lockdep.c:4832
lock_acquire kernel/locking/lockdep.c:5442 [inline]
lock_acquire+0x1a8/0x720 kernel/locking/lockdep.c:5407
__mutex_lock_common kernel/locking/mutex.c:956 [inline]
__mutex_lock+0x134/0x1110 kernel/locking/mutex.c:1103
dquot_commit+0x4d/0x420 fs/quota/dquot.c:476
ext4_write_dquot+0x24e/0x310 fs/ext4/super.c:6200
ext4_mark_dquot_dirty fs/ext4/super.c:6248 [inline]
ext4_mark_dquot_dirty+0x111/0x1b0 fs/ext4/super.c:6242
mark_dquot_dirty fs/quota/dquot.c:347 [inline]
mark_all_dquot_dirty fs/quota/dquot.c:385 [inline]
__dquot_alloc_space+0x5d4/0xb60 fs/quota/dquot.c:1709
dquot_alloc_space_nodirty include/linux/quotaops.h:297 [inline]
dquot_alloc_space include/linux/quotaops.h:310 [inline]
dquot_alloc_block include/linux/quotaops.h:334 [inline]
ext4_mb_new_blocks+0x5a9/0x51a0 fs/ext4/mballoc.c:4937
ext4_ext_map_blocks+0x20da/0x5fb0 fs/ext4/extents.c:4238
ext4_map_blocks+0x653/0x17d0 fs/ext4/inode.c:637
_ext4_get_block+0x241/0x590 fs/ext4/inode.c:793
ext4_block_write_begin+0x4f8/0x1190 fs/ext4/inode.c:1077
ext4_write_begin+0x4b5/0x14b0 fs/ext4/inode.c:1202
ext4_da_write_begin+0x672/0x1150 fs/ext4/inode.c:2961
generic_perform_write+0x20a/0x4f0 mm/filemap.c:3412
ext4_buffered_write_iter+0x244/0x4d0 fs/ext4/file.c:270
ext4_file_write_iter+0x423/0x14d0 fs/ext4/file.c:664
call_write_iter include/linux/fs.h:1901 [inline]
new_sync_write+0x426/0x650 fs/read_write.c:518
vfs_write+0x791/0xa30 fs/read_write.c:605
ksys_write+0x12d/0x250 fs/read_write.c:658
do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x465b09
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f8097ffc188 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000000000056bf60 RCX: 0000000000465b09
RDX: 000000000d4ba0ff RSI: 00000000200009c0 RDI: 0000000000000003
RBP: 00000000004b069f R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf60
R13: 00007ffefc77f01f R14: 00007f8097ffc300 R15: 0000000000022000


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.


2021-02-11 12:00:37

by Jan Kara

[permalink] [raw]
Subject: Re: possible deadlock in dquot_commit

On Wed 10-02-21 03:25:22, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 1e0d27fc Merge branch 'akpm' (patches from Andrew)
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=101cf2f8d00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=e83e68d0a6aba5f6
> dashboard link: https://syzkaller.appspot.com/bug?extid=3b6f9218b1301ddda3e2
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: [email protected]
>
> loop1: detected capacity change from 4096 to 0
> EXT4-fs (loop1): mounted filesystem without journal. Opts: ,errors=continue. Quota mode: writeback.
> ======================================================
> WARNING: possible circular locking dependency detected
> 5.11.0-rc6-syzkaller #0 Not tainted
> ------------------------------------------------------
> syz-executor.1/16170 is trying to acquire lock:
> ffff8880795f5b28 (&dquot->dq_lock){+.+.}-{3:3}, at: dquot_commit+0x4d/0x420 fs/quota/dquot.c:476
>
> but task is already holding lock:
> ffff88807960b438 (&ei->i_data_sem/2){++++}-{3:3}, at: ext4_map_blocks+0x5e1/0x17d0 fs/ext4/inode.c:630
>
> which lock already depends on the new lock.

<snip>

All snipped stacktraces look perfectly fine and the lock dependencies are as
expected.

> 5 locks held by syz-executor.1/16170:
> #0: ffff88802ad18b70 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xe9/0x100 fs/file.c:947
> #1: ffff88802fbec460 (sb_writers#5){.+.+}-{0:0}, at: ksys_write+0x12d/0x250 fs/read_write.c:658
> #2: ffff88807960b648 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: inode_lock include/linux/fs.h:773 [inline]
> #2: ffff88807960b648 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: ext4_buffered_write_iter+0xb6/0x4d0 fs/ext4/file.c:264
> #3: ffff88807960b438 (&ei->i_data_sem/2){++++}-{3:3}, at: ext4_map_blocks+0x5e1/0x17d0 fs/ext4/inode.c:630
> #4: ffffffff8bf1be58 (dquot_srcu){....}-{0:0}, at: i_dquot fs/quota/dquot.c:926 [inline]
> #4: ffffffff8bf1be58 (dquot_srcu){....}-{0:0}, at: __dquot_alloc_space+0x1b4/0xb60 fs/quota/dquot.c:1671

This actually looks problematic: We acquired &ei->i_data_sem/2 (i.e.,
I_DATA_SEM_QUOTA subclass) in ext4_map_blocks() called from
ext4_block_write_begin(). This suggests that the write has been happening
directly to the quota file (or that lockdep annotation of the inode went
wrong somewhere). Now we normally protect quota files with IMMUTABLE flag
so writing it should not be possible. We also don't allow clearing this
flag on used quota file. Finally I'd checked lockdep annotation and
everything looks correct. So at this point the best theory I have is that a
filesystem has been suitably corrupted and quota file supposed to be
inaccessible from userspace got exposed but I'd expect other problems to
hit first in that case. Anyway without a reproducer I have no more ideas...

Honza

>
> stack backtrace:
> CPU: 0 PID: 16170 Comm: syz-executor.1 Not tainted 5.11.0-rc6-syzkaller #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:79 [inline]
> dump_stack+0x107/0x163 lib/dump_stack.c:120
> check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2117
> check_prev_add kernel/locking/lockdep.c:2868 [inline]
> check_prevs_add kernel/locking/lockdep.c:2993 [inline]
> validate_chain kernel/locking/lockdep.c:3608 [inline]
> __lock_acquire+0x2b26/0x54f0 kernel/locking/lockdep.c:4832
> lock_acquire kernel/locking/lockdep.c:5442 [inline]
> lock_acquire+0x1a8/0x720 kernel/locking/lockdep.c:5407
> __mutex_lock_common kernel/locking/mutex.c:956 [inline]
> __mutex_lock+0x134/0x1110 kernel/locking/mutex.c:1103
> dquot_commit+0x4d/0x420 fs/quota/dquot.c:476
> ext4_write_dquot+0x24e/0x310 fs/ext4/super.c:6200
> ext4_mark_dquot_dirty fs/ext4/super.c:6248 [inline]
> ext4_mark_dquot_dirty+0x111/0x1b0 fs/ext4/super.c:6242
> mark_dquot_dirty fs/quota/dquot.c:347 [inline]
> mark_all_dquot_dirty fs/quota/dquot.c:385 [inline]
> __dquot_alloc_space+0x5d4/0xb60 fs/quota/dquot.c:1709
> dquot_alloc_space_nodirty include/linux/quotaops.h:297 [inline]
> dquot_alloc_space include/linux/quotaops.h:310 [inline]
> dquot_alloc_block include/linux/quotaops.h:334 [inline]
> ext4_mb_new_blocks+0x5a9/0x51a0 fs/ext4/mballoc.c:4937
> ext4_ext_map_blocks+0x20da/0x5fb0 fs/ext4/extents.c:4238
> ext4_map_blocks+0x653/0x17d0 fs/ext4/inode.c:637
> _ext4_get_block+0x241/0x590 fs/ext4/inode.c:793
> ext4_block_write_begin+0x4f8/0x1190 fs/ext4/inode.c:1077
> ext4_write_begin+0x4b5/0x14b0 fs/ext4/inode.c:1202
> ext4_da_write_begin+0x672/0x1150 fs/ext4/inode.c:2961
> generic_perform_write+0x20a/0x4f0 mm/filemap.c:3412
> ext4_buffered_write_iter+0x244/0x4d0 fs/ext4/file.c:270
> ext4_file_write_iter+0x423/0x14d0 fs/ext4/file.c:664
> call_write_iter include/linux/fs.h:1901 [inline]
> new_sync_write+0x426/0x650 fs/read_write.c:518
> vfs_write+0x791/0xa30 fs/read_write.c:605
> ksys_write+0x12d/0x250 fs/read_write.c:658
> do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x465b09
> Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007f8097ffc188 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> RAX: ffffffffffffffda RBX: 000000000056bf60 RCX: 0000000000465b09
> RDX: 000000000d4ba0ff RSI: 00000000200009c0 RDI: 0000000000000003
> RBP: 00000000004b069f R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf60
> R13: 00007ffefc77f01f R14: 00007f8097ffc300 R15: 0000000000022000
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at [email protected].
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
--
Jan Kara <[email protected]>
SUSE Labs, CR

2021-02-11 12:01:50

by Dmitry Vyukov

[permalink] [raw]
Subject: Re: possible deadlock in dquot_commit

On Thu, Feb 11, 2021 at 12:37 PM Jan Kara <[email protected]> wrote:
>
> On Wed 10-02-21 03:25:22, syzbot wrote:
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit: 1e0d27fc Merge branch 'akpm' (patches from Andrew)
> > git tree: upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=101cf2f8d00000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=e83e68d0a6aba5f6
> > dashboard link: https://syzkaller.appspot.com/bug?extid=3b6f9218b1301ddda3e2
> >
> > Unfortunately, I don't have any reproducer for this issue yet.
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: [email protected]
> >
> > loop1: detected capacity change from 4096 to 0
> > EXT4-fs (loop1): mounted filesystem without journal. Opts: ,errors=continue. Quota mode: writeback.
> > ======================================================
> > WARNING: possible circular locking dependency detected
> > 5.11.0-rc6-syzkaller #0 Not tainted
> > ------------------------------------------------------
> > syz-executor.1/16170 is trying to acquire lock:
> > ffff8880795f5b28 (&dquot->dq_lock){+.+.}-{3:3}, at: dquot_commit+0x4d/0x420 fs/quota/dquot.c:476
> >
> > but task is already holding lock:
> > ffff88807960b438 (&ei->i_data_sem/2){++++}-{3:3}, at: ext4_map_blocks+0x5e1/0x17d0 fs/ext4/inode.c:630
> >
> > which lock already depends on the new lock.
>
> <snip>
>
> All snipped stacktraces look perfectly fine and the lock dependencies are as
> expected.
>
> > 5 locks held by syz-executor.1/16170:
> > #0: ffff88802ad18b70 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xe9/0x100 fs/file.c:947
> > #1: ffff88802fbec460 (sb_writers#5){.+.+}-{0:0}, at: ksys_write+0x12d/0x250 fs/read_write.c:658
> > #2: ffff88807960b648 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: inode_lock include/linux/fs.h:773 [inline]
> > #2: ffff88807960b648 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: ext4_buffered_write_iter+0xb6/0x4d0 fs/ext4/file.c:264
> > #3: ffff88807960b438 (&ei->i_data_sem/2){++++}-{3:3}, at: ext4_map_blocks+0x5e1/0x17d0 fs/ext4/inode.c:630
> > #4: ffffffff8bf1be58 (dquot_srcu){....}-{0:0}, at: i_dquot fs/quota/dquot.c:926 [inline]
> > #4: ffffffff8bf1be58 (dquot_srcu){....}-{0:0}, at: __dquot_alloc_space+0x1b4/0xb60 fs/quota/dquot.c:1671
>
> This actually looks problematic: We acquired &ei->i_data_sem/2 (i.e.,
> I_DATA_SEM_QUOTA subclass) in ext4_map_blocks() called from
> ext4_block_write_begin(). This suggests that the write has been happening
> directly to the quota file (or that lockdep annotation of the inode went
> wrong somewhere). Now we normally protect quota files with IMMUTABLE flag
> so writing it should not be possible. We also don't allow clearing this
> flag on used quota file. Finally I'd checked lockdep annotation and
> everything looks correct. So at this point the best theory I have is that a
> filesystem has been suitably corrupted and quota file supposed to be
> inaccessible from userspace got exposed but I'd expect other problems to
> hit first in that case. Anyway without a reproducer I have no more ideas...

There is a reproducer for 4.19 available on the dashboard. Maybe it will help.
I don't why it did not pop up on upstream yet, there lots of potential
reasons for this.

> Honza
>
> >
> > stack backtrace:
> > CPU: 0 PID: 16170 Comm: syz-executor.1 Not tainted 5.11.0-rc6-syzkaller #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> > Call Trace:
> > __dump_stack lib/dump_stack.c:79 [inline]
> > dump_stack+0x107/0x163 lib/dump_stack.c:120
> > check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2117
> > check_prev_add kernel/locking/lockdep.c:2868 [inline]
> > check_prevs_add kernel/locking/lockdep.c:2993 [inline]
> > validate_chain kernel/locking/lockdep.c:3608 [inline]
> > __lock_acquire+0x2b26/0x54f0 kernel/locking/lockdep.c:4832
> > lock_acquire kernel/locking/lockdep.c:5442 [inline]
> > lock_acquire+0x1a8/0x720 kernel/locking/lockdep.c:5407
> > __mutex_lock_common kernel/locking/mutex.c:956 [inline]
> > __mutex_lock+0x134/0x1110 kernel/locking/mutex.c:1103
> > dquot_commit+0x4d/0x420 fs/quota/dquot.c:476
> > ext4_write_dquot+0x24e/0x310 fs/ext4/super.c:6200
> > ext4_mark_dquot_dirty fs/ext4/super.c:6248 [inline]
> > ext4_mark_dquot_dirty+0x111/0x1b0 fs/ext4/super.c:6242
> > mark_dquot_dirty fs/quota/dquot.c:347 [inline]
> > mark_all_dquot_dirty fs/quota/dquot.c:385 [inline]
> > __dquot_alloc_space+0x5d4/0xb60 fs/quota/dquot.c:1709
> > dquot_alloc_space_nodirty include/linux/quotaops.h:297 [inline]
> > dquot_alloc_space include/linux/quotaops.h:310 [inline]
> > dquot_alloc_block include/linux/quotaops.h:334 [inline]
> > ext4_mb_new_blocks+0x5a9/0x51a0 fs/ext4/mballoc.c:4937
> > ext4_ext_map_blocks+0x20da/0x5fb0 fs/ext4/extents.c:4238
> > ext4_map_blocks+0x653/0x17d0 fs/ext4/inode.c:637
> > _ext4_get_block+0x241/0x590 fs/ext4/inode.c:793
> > ext4_block_write_begin+0x4f8/0x1190 fs/ext4/inode.c:1077
> > ext4_write_begin+0x4b5/0x14b0 fs/ext4/inode.c:1202
> > ext4_da_write_begin+0x672/0x1150 fs/ext4/inode.c:2961
> > generic_perform_write+0x20a/0x4f0 mm/filemap.c:3412
> > ext4_buffered_write_iter+0x244/0x4d0 fs/ext4/file.c:270
> > ext4_file_write_iter+0x423/0x14d0 fs/ext4/file.c:664
> > call_write_iter include/linux/fs.h:1901 [inline]
> > new_sync_write+0x426/0x650 fs/read_write.c:518
> > vfs_write+0x791/0xa30 fs/read_write.c:605
> > ksys_write+0x12d/0x250 fs/read_write.c:658
> > do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
> > entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > RIP: 0033:0x465b09
> > Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
> > RSP: 002b:00007f8097ffc188 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> > RAX: ffffffffffffffda RBX: 000000000056bf60 RCX: 0000000000465b09
> > RDX: 000000000d4ba0ff RSI: 00000000200009c0 RDI: 0000000000000003
> > RBP: 00000000004b069f R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf60
> > R13: 00007ffefc77f01f R14: 00007f8097ffc300 R15: 0000000000022000
> >
> >
> > ---
> > This report is generated by a bot. It may contain errors.
> > See https://goo.gl/tpsmEJ for more information about syzbot.
> > syzbot engineers can be reached at [email protected].
> >
> > syzbot will keep track of this issue. See:
> > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> >
> --
> Jan Kara <[email protected]>
> SUSE Labs, CR
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/20210211113718.GM19070%40quack2.suse.cz.

2021-02-11 16:59:26

by Jan Kara

[permalink] [raw]
Subject: Re: possible deadlock in dquot_commit

On Thu 11-02-21 12:47:18, Dmitry Vyukov wrote:
> On Thu, Feb 11, 2021 at 12:37 PM Jan Kara <[email protected]> wrote:
> >
> > On Wed 10-02-21 03:25:22, syzbot wrote:
> > > Hello,
> > >
> > > syzbot found the following issue on:
> > >
> > > HEAD commit: 1e0d27fc Merge branch 'akpm' (patches from Andrew)
> > > git tree: upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=101cf2f8d00000
> > > kernel config: https://syzkaller.appspot.com/x/.config?x=e83e68d0a6aba5f6
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=3b6f9218b1301ddda3e2
> > >
> > > Unfortunately, I don't have any reproducer for this issue yet.
> > >
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: [email protected]
> > >
> > > loop1: detected capacity change from 4096 to 0
> > > EXT4-fs (loop1): mounted filesystem without journal. Opts: ,errors=continue. Quota mode: writeback.
> > > ======================================================
> > > WARNING: possible circular locking dependency detected
> > > 5.11.0-rc6-syzkaller #0 Not tainted
> > > ------------------------------------------------------
> > > syz-executor.1/16170 is trying to acquire lock:
> > > ffff8880795f5b28 (&dquot->dq_lock){+.+.}-{3:3}, at: dquot_commit+0x4d/0x420 fs/quota/dquot.c:476
> > >
> > > but task is already holding lock:
> > > ffff88807960b438 (&ei->i_data_sem/2){++++}-{3:3}, at: ext4_map_blocks+0x5e1/0x17d0 fs/ext4/inode.c:630
> > >
> > > which lock already depends on the new lock.
> >
> > <snip>
> >
> > All snipped stacktraces look perfectly fine and the lock dependencies are as
> > expected.
> >
> > > 5 locks held by syz-executor.1/16170:
> > > #0: ffff88802ad18b70 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xe9/0x100 fs/file.c:947
> > > #1: ffff88802fbec460 (sb_writers#5){.+.+}-{0:0}, at: ksys_write+0x12d/0x250 fs/read_write.c:658
> > > #2: ffff88807960b648 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: inode_lock include/linux/fs.h:773 [inline]
> > > #2: ffff88807960b648 (&sb->s_type->i_mutex_key#9){++++}-{3:3}, at: ext4_buffered_write_iter+0xb6/0x4d0 fs/ext4/file.c:264
> > > #3: ffff88807960b438 (&ei->i_data_sem/2){++++}-{3:3}, at: ext4_map_blocks+0x5e1/0x17d0 fs/ext4/inode.c:630
> > > #4: ffffffff8bf1be58 (dquot_srcu){....}-{0:0}, at: i_dquot fs/quota/dquot.c:926 [inline]
> > > #4: ffffffff8bf1be58 (dquot_srcu){....}-{0:0}, at: __dquot_alloc_space+0x1b4/0xb60 fs/quota/dquot.c:1671
> >
> > This actually looks problematic: We acquired &ei->i_data_sem/2 (i.e.,
> > I_DATA_SEM_QUOTA subclass) in ext4_map_blocks() called from
> > ext4_block_write_begin(). This suggests that the write has been happening
> > directly to the quota file (or that lockdep annotation of the inode went
> > wrong somewhere). Now we normally protect quota files with IMMUTABLE flag
> > so writing it should not be possible. We also don't allow clearing this
> > flag on used quota file. Finally I'd checked lockdep annotation and
> > everything looks correct. So at this point the best theory I have is that a
> > filesystem has been suitably corrupted and quota file supposed to be
> > inaccessible from userspace got exposed but I'd expect other problems to
> > hit first in that case. Anyway without a reproducer I have no more ideas...
>
> There is a reproducer for 4.19 available on the dashboard. Maybe it will help.
> I don't why it did not pop up on upstream yet, there lots of potential
> reasons for this.

OK, so I've checked the fs images generated by the syzkaller reproducer and
they indeed have QUOTA feature enabled. Also inodes used by quota files are
not marked as allocated so there is some potential for surprises. But all
the possible paths I could think of seem to be covered and return
EFSCORRUPTED. Also note that the reproducer didn't trigger the
lockdep splat for me so the problem still isn't clear to me.

Honza

> > >
> > > stack backtrace:
> > > CPU: 0 PID: 16170 Comm: syz-executor.1 Not tainted 5.11.0-rc6-syzkaller #0
> > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> > > Call Trace:
> > > __dump_stack lib/dump_stack.c:79 [inline]
> > > dump_stack+0x107/0x163 lib/dump_stack.c:120
> > > check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2117
> > > check_prev_add kernel/locking/lockdep.c:2868 [inline]
> > > check_prevs_add kernel/locking/lockdep.c:2993 [inline]
> > > validate_chain kernel/locking/lockdep.c:3608 [inline]
> > > __lock_acquire+0x2b26/0x54f0 kernel/locking/lockdep.c:4832
> > > lock_acquire kernel/locking/lockdep.c:5442 [inline]
> > > lock_acquire+0x1a8/0x720 kernel/locking/lockdep.c:5407
> > > __mutex_lock_common kernel/locking/mutex.c:956 [inline]
> > > __mutex_lock+0x134/0x1110 kernel/locking/mutex.c:1103
> > > dquot_commit+0x4d/0x420 fs/quota/dquot.c:476
> > > ext4_write_dquot+0x24e/0x310 fs/ext4/super.c:6200
> > > ext4_mark_dquot_dirty fs/ext4/super.c:6248 [inline]
> > > ext4_mark_dquot_dirty+0x111/0x1b0 fs/ext4/super.c:6242
> > > mark_dquot_dirty fs/quota/dquot.c:347 [inline]
> > > mark_all_dquot_dirty fs/quota/dquot.c:385 [inline]
> > > __dquot_alloc_space+0x5d4/0xb60 fs/quota/dquot.c:1709
> > > dquot_alloc_space_nodirty include/linux/quotaops.h:297 [inline]
> > > dquot_alloc_space include/linux/quotaops.h:310 [inline]
> > > dquot_alloc_block include/linux/quotaops.h:334 [inline]
> > > ext4_mb_new_blocks+0x5a9/0x51a0 fs/ext4/mballoc.c:4937
> > > ext4_ext_map_blocks+0x20da/0x5fb0 fs/ext4/extents.c:4238
> > > ext4_map_blocks+0x653/0x17d0 fs/ext4/inode.c:637
> > > _ext4_get_block+0x241/0x590 fs/ext4/inode.c:793
> > > ext4_block_write_begin+0x4f8/0x1190 fs/ext4/inode.c:1077
> > > ext4_write_begin+0x4b5/0x14b0 fs/ext4/inode.c:1202
> > > ext4_da_write_begin+0x672/0x1150 fs/ext4/inode.c:2961
> > > generic_perform_write+0x20a/0x4f0 mm/filemap.c:3412
> > > ext4_buffered_write_iter+0x244/0x4d0 fs/ext4/file.c:270
> > > ext4_file_write_iter+0x423/0x14d0 fs/ext4/file.c:664
> > > call_write_iter include/linux/fs.h:1901 [inline]
> > > new_sync_write+0x426/0x650 fs/read_write.c:518
> > > vfs_write+0x791/0xa30 fs/read_write.c:605
> > > ksys_write+0x12d/0x250 fs/read_write.c:658
> > > do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
> > > entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > RIP: 0033:0x465b09
> > > Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
> > > RSP: 002b:00007f8097ffc188 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> > > RAX: ffffffffffffffda RBX: 000000000056bf60 RCX: 0000000000465b09
> > > RDX: 000000000d4ba0ff RSI: 00000000200009c0 RDI: 0000000000000003
> > > RBP: 00000000004b069f R08: 0000000000000000 R09: 0000000000000000
> > > R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf60
> > > R13: 00007ffefc77f01f R14: 00007f8097ffc300 R15: 0000000000022000
> > >
> > >
> > > ---
> > > This report is generated by a bot. It may contain errors.
> > > See https://goo.gl/tpsmEJ for more information about syzbot.
> > > syzbot engineers can be reached at [email protected].
> > >
> > > syzbot will keep track of this issue. See:
> > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > >
> > --
> > Jan Kara <[email protected]>
> > SUSE Labs, CR
> >
> > --
> > You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> > To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/20210211113718.GM19070%40quack2.suse.cz.
--
Jan Kara <[email protected]>
SUSE Labs, CR

2021-02-11 21:51:31

by Theodore Ts'o

[permalink] [raw]
Subject: Re: possible deadlock in dquot_commit

On Thu, Feb 11, 2021 at 12:47:18PM +0100, Dmitry Vyukov wrote:
> > This actually looks problematic: We acquired &ei->i_data_sem/2 (i.e.,
> > I_DATA_SEM_QUOTA subclass) in ext4_map_blocks() called from
> > ext4_block_write_begin(). This suggests that the write has been happening
> > directly to the quota file (or that lockdep annotation of the inode went
> > wrong somewhere). Now we normally protect quota files with IMMUTABLE flag
> > so writing it should not be possible. We also don't allow clearing this
> > flag on used quota file. Finally I'd checked lockdep annotation and
> > everything looks correct. So at this point the best theory I have is that a
> > filesystem has been suitably corrupted and quota file supposed to be
> > inaccessible from userspace got exposed but I'd expect other problems to
> > hit first in that case. Anyway without a reproducer I have no more ideas...
>
> There is a reproducer for 4.19 available on the dashboard. Maybe it will help.
> I don't why it did not pop up on upstream yet, there lots of potential
> reasons for this.

The 4.19 version of the syzbot report has a very different stack
trace. Instead of it being related to an apparent write to the quota
file, it is apparently caused by a call to rmdir:

dump_stack+0x22c/0x33e lib/dump_stack.c:118
print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1221
...
__mutex_lock+0xd7/0x13f0 kernel/locking/mutex.c:1072
dquot_commit+0x4d/0x400 fs/quota/dquot.c:469
ext4_write_dquot+0x1f2/0x2a0 fs/ext4/super.c:5644
...
ext4_evict_inode+0x933/0x1830 fs/ext4/inode.c:298
evict+0x2ed/0x780 fs/inode.c:559
iput_final fs/inode.c:1555 [inline]
...
vfs_rmdir fs/namei.c:3865 [inline]
do_rmdir+0x3af/0x420 fs/namei.c:3943
__do_sys_unlinkat fs/namei.c:4105 [inline]
__se_sys_unlinkat fs/namei.c:4099 [inline]
__x64_sys_unlinkat+0xdf/0x120 fs/namei.c:4099
do_syscall_64+0xf9/0x670 arch/x86/entry/common.c:293
entry_SYSCALL_64_after_hwframe+0x49/0xbe

Which leads me to another apparent contradiction. Looking at the C
reproducer source code, and running the C reproducer under "strace
-ff", there is never any attempt to run rmdir() on the corrupted file
system that is mounted. Neither as observed by my running the C
reproducer, or by looking at the C reproducer source code.

Looking at the code, I did see a number of things which seemed to be
bugs; procid never gets incremented, so all of the threads only
operate on /dev/loop0, and each call to the execute() function tries
to setup two file systems on /dev/loop0. So the each thread to run
creates a temp file, binds it to /dev/loop0, and then creates another
temp file, tries to bind it to /dev/loop0 (which will fail), tries to
mount /dev/loop0 (again) on the samee mount point (which will
succeed).

I'm not sure if this is just some insanity that was consed up by the
fuzzer... or I'm wondering if this was an unfaithful translation of
the syzbot repro to C. Am I correct in understanding that when syzbot
is running, it uses the syzbot repro, and not the C repro?

- Ted

2021-02-12 11:06:49

by Dmitry Vyukov

[permalink] [raw]
Subject: Re: possible deadlock in dquot_commit

On Thu, Feb 11, 2021 at 10:46 PM Theodore Ts'o <[email protected]> wrote:
>
> On Thu, Feb 11, 2021 at 12:47:18PM +0100, Dmitry Vyukov wrote:
> > > This actually looks problematic: We acquired &ei->i_data_sem/2 (i.e.,
> > > I_DATA_SEM_QUOTA subclass) in ext4_map_blocks() called from
> > > ext4_block_write_begin(). This suggests that the write has been happening
> > > directly to the quota file (or that lockdep annotation of the inode went
> > > wrong somewhere). Now we normally protect quota files with IMMUTABLE flag
> > > so writing it should not be possible. We also don't allow clearing this
> > > flag on used quota file. Finally I'd checked lockdep annotation and
> > > everything looks correct. So at this point the best theory I have is that a
> > > filesystem has been suitably corrupted and quota file supposed to be
> > > inaccessible from userspace got exposed but I'd expect other problems to
> > > hit first in that case. Anyway without a reproducer I have no more ideas...
> >
> > There is a reproducer for 4.19 available on the dashboard. Maybe it will help.
> > I don't why it did not pop up on upstream yet, there lots of potential
> > reasons for this.
>
> The 4.19 version of the syzbot report has a very different stack
> trace. Instead of it being related to an apparent write to the quota
> file, it is apparently caused by a call to rmdir:
>
> dump_stack+0x22c/0x33e lib/dump_stack.c:118
> print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1221
> ...
> __mutex_lock+0xd7/0x13f0 kernel/locking/mutex.c:1072
> dquot_commit+0x4d/0x400 fs/quota/dquot.c:469
> ext4_write_dquot+0x1f2/0x2a0 fs/ext4/super.c:5644
> ...
> ext4_evict_inode+0x933/0x1830 fs/ext4/inode.c:298
> evict+0x2ed/0x780 fs/inode.c:559
> iput_final fs/inode.c:1555 [inline]
> ...
> vfs_rmdir fs/namei.c:3865 [inline]
> do_rmdir+0x3af/0x420 fs/namei.c:3943
> __do_sys_unlinkat fs/namei.c:4105 [inline]
> __se_sys_unlinkat fs/namei.c:4099 [inline]
> __x64_sys_unlinkat+0xdf/0x120 fs/namei.c:4099
> do_syscall_64+0xf9/0x670 arch/x86/entry/common.c:293
> entry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> Which leads me to another apparent contradiction. Looking at the C
> reproducer source code, and running the C reproducer under "strace
> -ff", there is never any attempt to run rmdir() on the corrupted file
> system that is mounted. Neither as observed by my running the C
> reproducer, or by looking at the C reproducer source code.
>
> Looking at the code, I did see a number of things which seemed to be
> bugs; procid never gets incremented, so all of the threads only
> operate on /dev/loop0, and each call to the execute() function tries
> to setup two file systems on /dev/loop0. So the each thread to run
> creates a temp file, binds it to /dev/loop0, and then creates another
> temp file, tries to bind it to /dev/loop0 (which will fail), tries to
> mount /dev/loop0 (again) on the samee mount point (which will
> succeed).
>
> I'm not sure if this is just some insanity that was consed up by the
> fuzzer... or I'm wondering if this was an unfaithful translation of
> the syzbot repro to C. Am I correct in understanding that when syzbot
> is running, it uses the syzbot repro, and not the C repro?

Hi Ted,

The 4.19 reproducer may reproducer something else, you know better. I
just want to answer points re syzkaller reproducers. FTR the 4.19
reproducer/reproducer is here:
https://syzkaller.appspot.com/bug?id=b6cacc9fa48fea07154b8797236727de981c1e02

> there is never any attempt to run rmdir() on the corrupted file system that is mounted.

Recursive rmdir happens as part of test cleanup implicitly, you can
see rmdir call in remove_dir function in the C reproducer:
https://syzkaller.appspot.com/text?tag=ReproC&x=12caea37900000

> procid never gets incremented, so all of the threads only operate on /dev/loop0

This is intentional. procid is supposed to "isolate" parallel test
processes (if any). This reproducer does not use parallel test
processes, thus procid has constant value.

> Am I correct in understanding that when syzbot is running, it uses the syzbot repro, and not the C repro?

It tries both. If first tries to interpret "syzkaller program" as it
was done when the bug was triggered during fuzzing. But then it tries
to convert it to a corresponding stand-alone C program and confirms
that it still triggers the bug. If it provides a C reproducer, it
means that it did trigger the bug using this exact C program on a
freshly booted kernel (and the provided kernel oops is the
corresponding oops obtained on this exact program).
If it fails to reproduce the bug with a C reproducer, then it provides
only the "syzkaller program" to not mislead developers.

2021-02-12 16:14:49

by Theodore Ts'o

[permalink] [raw]
Subject: Re: possible deadlock in dquot_commit

>From: Theodore Ts'o <[email protected]>

On Fri, Feb 12, 2021 at 12:01:51PM +0100, Dmitry Vyukov wrote:
> > >
> > > There is a reproducer for 4.19 available on the dashboard. Maybe it will help.
> > > I don't why it did not pop up on upstream yet, there lots of potential
> > > reasons for this.
> >
> > The 4.19 version of the syzbot report has a very different stack
> > trace. Instead of it being related to an apparent write to the quota
> > file, it is apparently caused by a call to rmdir:
> >
>
> The 4.19 reproducer may reproducer something else, you know better. I
> just want to answer points re syzkaller reproducers. FTR the 4.19
> reproducer/reproducer is here:
> https://syzkaller.appspot.com/bug?id=b6cacc9fa48fea07154b8797236727de981c1e02

Yes, I know. That was my point. I don't think it's useful for
debugging the upstream dquot_commit syzbot report (for which we don't
have a reproducer yet).

> > there is never any attempt to run rmdir() on the corrupted file system that is mounted.
>
> Recursive rmdir happens as part of test cleanup implicitly, you can
> see rmdir call in remove_dir function in the C reproducer:
> https://syzkaller.appspot.com/text?tag=ReproC&x=12caea37900000

That rmdir() removes the mountpoint, which is *not* the fuzzed file
system which has the quota feature enabled.

> > procid never gets incremented, so all of the threads only operate on /dev/loop0
>
> This is intentional. procid is supposed to "isolate" parallel test
> processes (if any). This reproducer does not use parallel test
> processes, thus procid has constant value.

Um... yes it does:

int main(void)
{
syscall(__NR_mmap, 0x1ffff000ul, 0x1000ul, 0ul, 0x32ul, -1, 0ul);
syscall(__NR_mmap, 0x20000000ul, 0x1000000ul, 7ul, 0x32ul, -1, 0ul);
syscall(__NR_mmap, 0x21000000ul, 0x1000ul, 0ul, 0x32ul, -1, 0ul);
use_temporary_dir();
loop();
return 0;
}

and what is loop?

static void loop(void)
{
int iter = 0;
for (;; iter++) {
...
reset_loop();
int pid = fork();
if (pid < 0)
exit(1);
if (pid == 0) {
if (chdir(cwdbuf))
exit(1);
setup_test();
execute_one();
exit(0);
}
...
remove_dir(cwdbuf);
}
}

> > Am I correct in understanding that when syzbot is running, it uses the syzbot repro, and not the C repro?
>
> It tries both. If first tries to interpret "syzkaller program" as it
> was done when the bug was triggered during fuzzing. But then it tries
> to convert it to a corresponding stand-alone C program and confirms
> that it still triggers the bug. If it provides a C reproducer, it
> means that it did trigger the bug using this exact C program on a
> freshly booted kernel (and the provided kernel oops is the
> corresponding oops obtained on this exact program).
> If it fails to reproduce the bug with a C reproducer, then it provides
> only the "syzkaller program" to not mislead developers.

Well, looking at the C reproducer, it doesn't reproduce on upstream,
and the stack trace makes no sense to me. The rmdir() executes at the
end of the test, as part of the cleanup, and looking at the syzkaller
console, the stack trace involving rmdir happens *early* while test
threads are still trying to mount the file system.

- Ted

2021-02-15 12:56:33

by Dmitry Vyukov

[permalink] [raw]
Subject: Re: possible deadlock in dquot_commit

On Fri, Feb 12, 2021 at 5:10 PM Theodore Ts'o <[email protected]> wrote:
>
> >From: Theodore Ts'o <[email protected]>
>
> On Fri, Feb 12, 2021 at 12:01:51PM +0100, Dmitry Vyukov wrote:
> > > >
> > > > There is a reproducer for 4.19 available on the dashboard. Maybe it will help.
> > > > I don't why it did not pop up on upstream yet, there lots of potential
> > > > reasons for this.
> > >
> > > The 4.19 version of the syzbot report has a very different stack
> > > trace. Instead of it being related to an apparent write to the quota
> > > file, it is apparently caused by a call to rmdir:
> > >
> >
> > The 4.19 reproducer may reproducer something else, you know better. I
> > just want to answer points re syzkaller reproducers. FTR the 4.19
> > reproducer/reproducer is here:
> > https://syzkaller.appspot.com/bug?id=b6cacc9fa48fea07154b8797236727de981c1e02
>
> Yes, I know. That was my point. I don't think it's useful for
> debugging the upstream dquot_commit syzbot report (for which we don't
> have a reproducer yet).
>
> > > there is never any attempt to run rmdir() on the corrupted file system that is mounted.
> >
> > Recursive rmdir happens as part of test cleanup implicitly, you can
> > see rmdir call in remove_dir function in the C reproducer:
> > https://syzkaller.appspot.com/text?tag=ReproC&x=12caea37900000
>
> That rmdir() removes the mountpoint, which is *not* the fuzzed file
> system which has the quota feature enabled.

remove_dir function is recursive, so rmdir should be called for all
subdirectories starting from the deepest ones. At least that was the
intention. Do you see it's not working this way? That would be
something to fix.

> > > procid never gets incremented, so all of the threads only operate on /dev/loop0
> >
> > This is intentional. procid is supposed to "isolate" parallel test
> > processes (if any). This reproducer does not use parallel test
> > processes, thus procid has constant value.
>
> Um... yes it does:

There is waitpid before remove_dir. So these are sequential test
processes, not parallel.

> int main(void)
> {
> syscall(__NR_mmap, 0x1ffff000ul, 0x1000ul, 0ul, 0x32ul, -1, 0ul);
> syscall(__NR_mmap, 0x20000000ul, 0x1000000ul, 7ul, 0x32ul, -1, 0ul);
> syscall(__NR_mmap, 0x21000000ul, 0x1000ul, 0ul, 0x32ul, -1, 0ul);
> use_temporary_dir();
> loop();
> return 0;
> }
>
> and what is loop?
>
> static void loop(void)
> {
> int iter = 0;
> for (;; iter++) {
> ...
> reset_loop();
> int pid = fork();
> if (pid < 0)
> exit(1);
> if (pid == 0) {
> if (chdir(cwdbuf))
> exit(1);
> setup_test();
> execute_one();
> exit(0);
> }
> ...
> remove_dir(cwdbuf);
> }
> }
>
> > > Am I correct in understanding that when syzbot is running, it uses the syzbot repro, and not the C repro?
> >
> > It tries both. If first tries to interpret "syzkaller program" as it
> > was done when the bug was triggered during fuzzing. But then it tries
> > to convert it to a corresponding stand-alone C program and confirms
> > that it still triggers the bug. If it provides a C reproducer, it
> > means that it did trigger the bug using this exact C program on a
> > freshly booted kernel (and the provided kernel oops is the
> > corresponding oops obtained on this exact program).
> > If it fails to reproduce the bug with a C reproducer, then it provides
> > only the "syzkaller program" to not mislead developers.
>
> Well, looking at the C reproducer, it doesn't reproduce on upstream,
> and the stack trace makes no sense to me. The rmdir() executes at the
> end of the test, as part of the cleanup, and looking at the syzkaller
> console, the stack trace involving rmdir happens *early* while test
> threads are still trying to mount the file system.

My assumption that the 4.19 reproducer for a somewhat similarly
looking bug may also reproduce this upstream bug is false then.