2024-02-19 10:54:49

by syzbot

[permalink] [raw]
Subject: [syzbot] [nilfs?] INFO: task hung in nilfs_segctor_thread (2)

Hello,

syzbot found the following issue on:

HEAD commit: f735966ee23c Merge branches 'for-next/reorg-va-space' and ..
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
console output: https://syzkaller.appspot.com/x/log.txt?x=12dbb3dc180000
kernel config: https://syzkaller.appspot.com/x/.config?x=d47605a39da2cf06
dashboard link: https://syzkaller.appspot.com/bug?extid=c8166c541d3971bf6c87
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/bdea2316c4db/disk-f735966e.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/75ba7806a91c/vmlinux-f735966e.xz
kernel image: https://storage.googleapis.com/syzbot-assets/208f119d45ed/Image-f735966e.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]

INFO: task segctord:26558 blocked for more than 143 seconds.
Not tainted 6.8.0-rc3-syzkaller-gf735966ee23c #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:segctord state:D stack:0 pid:26558 tgid:26558 ppid:2 flags:0x00000008
Call trace:
__switch_to+0x314/0x560 arch/arm64/kernel/process.c:556
context_switch kernel/sched/core.c:5400 [inline]
__schedule+0x1498/0x24b4 kernel/sched/core.c:6727
__schedule_loop kernel/sched/core.c:6802 [inline]
schedule+0xb8/0x19c kernel/sched/core.c:6817
schedule_preempt_disabled+0x18/0x2c kernel/sched/core.c:6874
rwsem_down_write_slowpath+0xcfc/0x1aa0 kernel/locking/rwsem.c:1178
__down_write_common kernel/locking/rwsem.c:1306 [inline]
__down_write kernel/locking/rwsem.c:1315 [inline]
down_write+0xb4/0xc0 kernel/locking/rwsem.c:1580
nilfs_transaction_lock+0x178/0x33c fs/nilfs2/segment.c:357
nilfs_segctor_thread_construct fs/nilfs2/segment.c:2523 [inline]
nilfs_segctor_thread+0x3cc/0xd78 fs/nilfs2/segment.c:2608
kthread+0x288/0x310 kernel/kthread.c:388
ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860

Showing all locks held in the system:
1 lock held by khungtaskd/29:
#0: ffff80008ee43fc0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0xc/0x44 include/linux/rcupdate.h:297
2 locks held by getty/5931:
#0: ffff0000d82710a0 (&tty->ldisc_sem){++++}-{0:0}, at: ldsem_down_read+0x3c/0x4c drivers/tty/tty_ldsem.c:340
#1: ffff800093fe72f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x41c/0x1228 drivers/tty/n_tty.c:2201
1 lock held by syz-executor.0/6205:
#0: ffff0000d6d12c68 (&pipe->mutex/1){+.+.}-{3:3}, at: rcu_lock_acquire+0xc/0x44 include/linux/rcupdate.h:297
2 locks held by kworker/u4:26/13298:
6 locks held by syz-executor.2/26553:
1 lock held by segctord/26558:
#0: ffff00011fc2d2a0
(&nilfs->ns_segctor_sem){++++}-{3:3}, at: nilfs_transaction_lock+0x178/0x33c fs/nilfs2/segment.c:357
1 lock held by syz-executor.3/11586:
#0: ffff0000c346f8b8 (&nft_net->commit_mutex){+.+.}-{3:3}, at: nf_tables_valid_genid+0x3c/0xd4 net/netfilter/nf_tables_api.c:10624
1 lock held by syz-executor.1/11588:
1 lock held by syz-executor.2/11593:
#0: ffff0001485282b8 (&nft_net->commit_mutex){+.+.}-{3:3}, at: nf_tables_valid_genid+0x3c/0xd4 net/netfilter/nf_tables_api.c:10624
1 lock held by syz-executor.4/11594:
#0: ffff0000d343fcb8 (&nft_net->commit_mutex){+.+.}-{3:3}, at: nf_tables_valid_genid+0x3c/0xd4 net/netfilter/nf_tables_api.c:10624
4 locks held by syz-executor.0/11595:
#0: ffff0001b400ef58 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested kernel/sched/core.c:559 [inline]
#0: ffff0001b400ef58 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock kernel/sched/sched.h:1385 [inline]
#0: ffff0001b400ef58 (&rq->__lock){-.-.}-{2:2}, at: rq_lock kernel/sched/sched.h:1699 [inline]
#0: ffff0001b400ef58 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x2e0/0x24b4 kernel/sched/core.c:6643
#1: ffff0001b3ffac88 (&per_cpu_ptr(group->pcpu, cpu)->seq){-.-.}-{0:0}, at: psi_task_switch+0x3c0/0x618 kernel/sched/psi.c:988
#2: ffff0001b401cc88 (&per_cpu_ptr(group->pcpu, cpu)->seq){-.-.}-{0:0}, at: psi_task_change+0x100/0x234 kernel/sched/psi.c:912
#3: ffff0001b401cc88 (&per_cpu_ptr(group->pcpu, cpu)->seq){-.-.}-{0:0}, at: psi_task_change+0x100/0x234 kernel/sched/psi.c:912

=============================================



---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup


2024-06-02 12:31:45

by syzbot

[permalink] [raw]
Subject: Re: [syzbot] [nilfs?] INFO: task hung in nilfs_segctor_thread (2)

syzbot has found a reproducer for the following issue on:

HEAD commit: 89be4025b0db Merge tag '6.10-rc1-smb3-client-fixes' of git..
git tree: upstream
console+strace: https://syzkaller.appspot.com/x/log.txt?x=17667026980000
kernel config: https://syzkaller.appspot.com/x/.config?x=47d282ddffae809f
dashboard link: https://syzkaller.appspot.com/bug?extid=c8166c541d3971bf6c87
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1164d8bc980000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=120ae206980000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/1b4c4cbe2fc3/disk-89be4025.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/992efea7573e/vmlinux-89be4025.xz
kernel image: https://storage.googleapis.com/syzbot-assets/40ebdc35acdd/bzImage-89be4025.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/2d66f1d4455b/mount_0.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]

INFO: task segctord:5081 blocked for more than 143 seconds.
Not tainted 6.10.0-rc1-syzkaller-00296-g89be4025b0db #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:segctord state:D stack:28088 pid:5081 tgid:5081 ppid:2 flags:0x00004000
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5408 [inline]
__schedule+0x1796/0x49d0 kernel/sched/core.c:6745
__schedule_loop kernel/sched/core.c:6822 [inline]
schedule+0x14b/0x320 kernel/sched/core.c:6837
schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6894
rwsem_down_write_slowpath+0xeeb/0x13b0 kernel/locking/rwsem.c:1178
__down_write_common+0x1af/0x200 kernel/locking/rwsem.c:1306
nilfs_transaction_lock+0x25d/0x4f0 fs/nilfs2/segment.c:357
nilfs_segctor_thread_construct fs/nilfs2/segment.c:2512 [inline]
nilfs_segctor_thread+0x551/0x11b0 fs/nilfs2/segment.c:2598
kthread+0x2f0/0x390 kernel/kthread.c:389
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
</TASK>

Showing all locks held in the system:
6 locks held by kworker/u8:0/11:
1 lock held by khungtaskd/30:
#0: ffffffff8e333f60 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline]
#0: ffffffff8e333f60 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline]
#0: ffffffff8e333f60 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x55/0x2a0 kernel/locking/lockdep.c:6614
2 locks held by getty/4833:
#0: ffff88802a5840a0 (
&tty->ldisc_sem){++++}-{0:0}
, at: tty_ldisc_ref_wait+0x25/0x70 drivers/tty/tty_ldisc.c:243
#1:
ffffc900031432f0
(&ldata->atomic_read_lock){+.+.}-{3:3}
, at: n_tty_read+0x6b5/0x1e10 drivers/tty/n_tty.c:2201
9 locks held by syz-executor240/5078:
1 lock held by segctord/5081:
#0:
ffff888020ff52a0 (&nilfs->ns_segctor_sem){++++}-{3:3}, at: nilfs_transaction_lock+0x25d/0x4f0 fs/nilfs2/segment.c:357

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 30 Comm: khungtaskd Not tainted 6.10.0-rc1-syzkaller-00296-g89be4025b0db #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
nmi_cpu_backtrace+0x49c/0x4d0 lib/nmi_backtrace.c:113
nmi_trigger_cpumask_backtrace+0x198/0x320 lib/nmi_backtrace.c:62
trigger_all_cpu_backtrace include/linux/nmi.h:162 [inline]
check_hung_uninterruptible_tasks kernel/hung_task.c:223 [inline]
watchdog+0xfde/0x1020 kernel/hung_task.c:379
kthread+0x2f0/0x390 kernel/kthread.c:389
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
</TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 5078 Comm: syz-executor240 Not tainted 6.10.0-rc1-syzkaller-00296-g89be4025b0db #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
RIP: 0010:_raw_spin_unlock_irqrestore+0x5/0x140 kernel/locking/spinlock.c:193
Code: 58 78 f5 5b c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 <48> 89 e5 41 57 41 56 41 55 41 54 53 48 83 e4 e0 48 83 ec 60 49 89
RSP: 0018:ffffc90000007ea8 EFLAGS: 00000006
RAX: 0000000000000000 RBX: ffff8880b942c8e0 RCX: ffff888023b41e00
RDX: 0000000000010002 RSI: 0000000000000046 RDI: ffff8880b942c880
RBP: ffff8880b942ca68 R08: ffffffff81836780 R09: 0000000000000000
R10: ffff8880b942d1a8 R11: ffffed1017285a37 R12: 000000430d8aa980
R13: ffff8880b942c880 R14: dffffc0000000000 R15: ffff8880b942cc68
FS: 0000555585626380(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007facfac43580 CR3: 000000007fff0000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<NMI>
</NMI>
<IRQ>
hrtimer_interrupt+0x540/0x990 kernel/time/hrtimer.c:1823
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1032 [inline]
__sysvec_apic_timer_interrupt+0x110/0x3f0 arch/x86/kernel/apic/apic.c:1049
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
sysvec_apic_timer_interrupt+0xa1/0xc0 arch/x86/kernel/apic/apic.c:1043
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:console_trylock_spinning kernel/printk/printk.c:2007 [inline]
RIP: 0010:vprintk_emit+0x576/0x770 kernel/printk/printk.c:2344
Code: 0a 20 00 4c 21 e3 0f 85 3a 01 00 00 e8 13 05 20 00 4d 89 ec 4d 85 ff 75 07 e8 06 05 20 00 eb 06 e8 ff 04 20 00 fb 44 8b 3c 24 <48> c7 c7 20 fa 20 8e 31 f6 ba 01 00 00 00 31 c9 41 b8 01 00 00 00
RSP: 0018:ffffc9000342f740 EFLAGS: 00000293
RAX: ffffffff81761091 RBX: 0000000000000000 RCX: ffff888023b41e00
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc9000342f830 R08: ffffffff8176106f R09: 1ffffffff25ee4c9
R10: dffffc0000000000 R11: fffffbfff25ee4ca R12: dffffc0000000000
R13: dffffc0000000000 R14: ffffffff81760eef R15: 000000000000009c
_printk+0xd5/0x120 kernel/printk/printk.c:2370
__nilfs_error+0x193/0x730 fs/nilfs2/super.c:131
nilfs_check_folio+0x423/0x660 fs/nilfs2/dir.c:164
nilfs_get_folio+0x13f/0x240 fs/nilfs2/dir.c:192
nilfs_empty_dir+0x127/0x660 fs/nilfs2/dir.c:608
nilfs_rmdir+0x10e/0x250 fs/nilfs2/namei.c:326
vfs_rmdir+0x3a3/0x510 fs/namei.c:4214
do_rmdir+0x3b5/0x580 fs/namei.c:4273
__do_sys_rmdir fs/namei.c:4292 [inline]
__se_sys_rmdir fs/namei.c:4290 [inline]
__x64_sys_rmdir+0x49/0x60 fs/namei.c:4290
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7faa1dacbdc7
Code: 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 54 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffd03552de8 EFLAGS: 00000246 ORIG_RAX: 0000000000000054
RAX: ffffffffffffffda RBX: 0000555585626338 RCX: 00007faa1dacbdc7
RDX: 0000555585657fff RSI: 0000000000000009 RDI: 00007ffd03553f90
RBP: 0000000000000064 R08: 000055558563f7db R09: 0000000000000000
R10: 0000000000001000 R11: 0000000000000246 R12: 00007ffd03553f90
R13: 0000555585637740 R14: 431bde82d7b634db R15: 00007ffd03556110
</TASK>
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.567 msecs


---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

2024-06-04 14:22:15

by Ryusuke Konishi

[permalink] [raw]
Subject: [PATCH] nilfs2: fix nilfs_empty_dir() misjudgment and long loop on I/O errors

The error handling in nilfs_empty_dir() when a directory folio/page
read fails is incorrect, as in the old ext2 implementation, and if the
folio/page cannot be read or nilfs_check_folio() fails, it will falsely
determine the directory as empty and corrupt the file system.

In addition, since nilfs_empty_dir() does not immediately return on
a failed folio/page read, but continues to loop, this can cause a long
loop with I/O if i_size of the directory's inode is also corrupted,
causing the log writer thread to wait and hang, as reported by syzbot.

Fix these issues by making nilfs_empty_dir() immediately return a false
value (0) if it fails to get a directory folio/page.

Signed-off-by: Ryusuke Konishi <[email protected]>
Reported-by: [email protected]
Closes: https://syzkaller.appspot.com/bug?extid=c8166c541d3971bf6c87
Fixes: 2ba466d74ed7 ("nilfs2: directory entry operations")
Tested-by: Ryusuke Konishi <[email protected]>
Cc: [email protected]
---
Hi Andrew, please apply this as a bug fix.

This fixes a bug in the empty directory function and the resulting
hang issue reported by syzbot.

Thanks,
Ryusuke Konishi

fs/nilfs2/dir.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nilfs2/dir.c b/fs/nilfs2/dir.c
index a002a44ff161..52e50b1b7f22 100644
--- a/fs/nilfs2/dir.c
+++ b/fs/nilfs2/dir.c
@@ -607,7 +607,7 @@ int nilfs_empty_dir(struct inode *inode)

kaddr = nilfs_get_folio(inode, i, &folio);
if (IS_ERR(kaddr))
- continue;
+ return 0;

de = (struct nilfs_dir_entry *)kaddr;
kaddr += nilfs_last_byte(inode, i) - NILFS_DIR_REC_LEN(1);
--
2.34.1