2024-03-26 19:01:22

by syzbot

Subject: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

Hello,

syzbot found the following issue on:

HEAD commit: fe46a7dd189e Merge tag 'sound-6.9-rc1' of git://git.kernel..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11547a65180000
kernel config: https://syzkaller.appspot.com/x/.config?x=4d90a36f0cab495a
dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/f6c04726a2ae/disk-fe46a7dd.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/09c26ce901ea/vmlinux-fe46a7dd.xz
kernel image: https://storage.googleapis.com/syzbot-assets/134acf7f5322/bzImage-fe46a7dd.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]

======================================================
WARNING: possible circular locking dependency detected
6.8.0-syzkaller-08951-gfe46a7dd189e #0 Not tainted
------------------------------------------------------
syz-executor.3/6590 is trying to acquire lock:
ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444

but task is already holding lock:
ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&trie->lock){..-.}-{2:2}:
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
trie_delete_elem+0x96/0x6a0 kernel/bpf/lpm_trie.c:451
bpf_prog_510c7248c5f60c92+0x2e/0x46
bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
__bpf_prog_run include/linux/filter.h:657 [inline]
bpf_prog_run include/linux/filter.h:664 [inline]
__bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
trace_timer_start include/trace/events/timer.h:52 [inline]
enqueue_timer+0x396/0x550 kernel/time/timer.c:663
internal_add_timer kernel/time/timer.c:688 [inline]
__mod_timer+0xa0e/0xeb0 kernel/time/timer.c:1183
call_timer_fn+0x17e/0x600 kernel/time/timer.c:1792
expire_timers kernel/time/timer.c:1843 [inline]
__run_timers kernel/time/timer.c:2408 [inline]
__run_timer_base+0x66a/0x8e0 kernel/time/timer.c:2419
run_timer_base kernel/time/timer.c:2428 [inline]
run_timer_softirq+0xb7/0x170 kernel/time/timer.c:2438
__do_softirq+0x2bc/0x943 kernel/softirq.c:554
invoke_softirq kernel/softirq.c:428 [inline]
__irq_exit_rcu+0xf2/0x1c0 kernel/softirq.c:633
irq_exit_rcu+0x9/0x30 kernel/softirq.c:645
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1043
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
memory_is_poisoned_n mm/kasan/generic.c:130 [inline]
memory_is_poisoned mm/kasan/generic.c:161 [inline]
check_region_inline mm/kasan/generic.c:180 [inline]
kasan_check_range+0x4f/0x290 mm/kasan/generic.c:189
instrument_atomic_read_write include/linux/instrumented.h:96 [inline]
atomic_try_cmpxchg_acquire include/linux/atomic/atomic-instrumented.h:1301 [inline]
queued_spin_lock include/asm-generic/qspinlock.h:111 [inline]
do_raw_spin_lock+0x14f/0x370 kernel/locking/spinlock_debug.c:116
spin_lock include/linux/spinlock.h:351 [inline]
lockref_get+0x15/0x60 lib/lockref.c:50
dget include/linux/dcache.h:333 [inline]
__traverse_mounts+0x3b4/0x580 fs/namei.c:1401
traverse_mounts fs/namei.c:1442 [inline]
handle_mounts fs/namei.c:1545 [inline]
step_into+0x5e5/0x1080 fs/namei.c:1842
walk_component fs/namei.c:2010 [inline]
link_path_walk+0x748/0xea0 fs/namei.c:2331
path_lookupat+0xa9/0x450 fs/namei.c:2484
filename_lookup+0x256/0x610 fs/namei.c:2514
user_path_at_empty+0x42/0x60 fs/namei.c:2921
do_readlinkat+0x118/0x3b0 fs/stat.c:499
__do_sys_readlink fs/stat.c:532 [inline]
__se_sys_readlink fs/stat.c:529 [inline]
__x64_sys_readlink+0x7f/0x90 fs/stat.c:529
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75

-> #1 (&base->lock){-.-.}-{2:2}:
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
lock_timer_base+0x112/0x240 kernel/time/timer.c:1051
__mod_timer+0x1ca/0xeb0 kernel/time/timer.c:1132
queue_delayed_work_on+0x15a/0x260 kernel/workqueue.c:2595
kvfree_call_rcu+0x47f/0x790 kernel/rcu/tree.c:3472
rtnl_register_internal+0x482/0x590 net/core/rtnetlink.c:265
rtnl_register+0x36/0x80 net/core/rtnetlink.c:315
ip_rt_init+0x2f5/0x3a0 net/ipv4/route.c:3719
ip_init+0xe/0x20 net/ipv4/ip_output.c:1664
inet_init+0x3d8/0x580 net/ipv4/af_inet.c:2022
do_one_initcall+0x238/0x830 init/main.c:1241
do_initcall_level+0x157/0x210 init/main.c:1303
do_initcalls+0x3f/0x80 init/main.c:1319
kernel_init_freeable+0x435/0x5d0 init/main.c:1550
kernel_init+0x1d/0x2a0 init/main.c:1439
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243

-> #0 (krc.lock){..-.}-{2:2}:
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
__sys_bpf+0x377/0x810
__do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
__x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75

other info that might help us debug this:

Chain exists of:
krc.lock --> &base->lock --> &trie->lock

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&trie->lock);
lock(&base->lock);
lock(&trie->lock);
lock(krc.lock);

*** DEADLOCK ***

2 locks held by syz-executor.3/6590:
#0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
#0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
#0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: bpf_map_update_value+0x3c4/0x540 kernel/bpf/syscall.c:202
#1: ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324

stack backtrace:
CPU: 1 PID: 6590 Comm: syz-executor.3 Not tainted 6.8.0-syzkaller-08951-gfe46a7dd189e #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
__sys_bpf+0x377/0x810
__do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
__x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75
RIP: 0033:0x7fe5f987dda9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fe5fa6000c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007fe5f99abf80 RCX: 00007fe5f987dda9
RDX: 0000000000000038 RSI: 0000000020000240 RDI: 000000000000001a
RBP: 00007fe5f98ca47a R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007fe5f99abf80 R15: 00007ffe908076c8
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

2024-03-26 19:54:04

by Alexei Starovoitov

Subject: false positive deadlock? Was: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

Hi Paul,

syzbot found an interesting false positive deadlock.
See below.
My understanding is the following:

cpu 2:
grabs timer_base lock
spins on bpf_lpm lock

cpu 1:
grab rcu krcp lock
spins on timer_base lock

cpu 0:
grab bpf_lpm lock
spins on rcu krcp lock

bpf_lpm lock can be the same.
timer_base lock can also be the same due to timer migration.

but rcu krcp lock is always per-cpu, so it cannot be the same lock.
Hence it's a false positive, but still interesting.

I don't think rcu can tell lockdep that these are different locks.

Few ideas/questions on how to address this:

1. in kernel/rcu/tree.c:
if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
schedule_delayed_monitor_work(krcp);

unlock_return:
krc_this_cpu_unlock(krcp, flags);

moving schedule_delayed_monitor_work() after unlock will not work, right?

2. if not, we can move 4 kfree_rcu-s in kernel/bpf/lpm_trie.c
to a place after unlock of lpm_trie

3. move bpf_lpm_trie to bpf_mem_alloc.

The 2 or 3 will address this false positive.

Other ideas?

On Tue, Mar 26, 2024 at 12:00 PM syzbot
<[email protected]> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: fe46a7dd189e Merge tag 'sound-6.9-rc1' of git://git.kernel..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=11547a65180000
> kernel config: https://syzkaller.appspot.com/x/.config?x=4d90a36f0cab495a
> dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/f6c04726a2ae/disk-fe46a7dd.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/09c26ce901ea/vmlinux-fe46a7dd.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/134acf7f5322/bzImage-fe46a7dd.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: [email protected]
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 6.8.0-syzkaller-08951-gfe46a7dd189e #0 Not tainted
> ------------------------------------------------------
> syz-executor.3/6590 is trying to acquire lock:
> ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
>
> but task is already holding lock:
> ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #2 (&trie->lock){..-.}-{2:2}:
> lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
> trie_delete_elem+0x96/0x6a0 kernel/bpf/lpm_trie.c:451
> bpf_prog_510c7248c5f60c92+0x2e/0x46
> bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
> __bpf_prog_run include/linux/filter.h:657 [inline]
> bpf_prog_run include/linux/filter.h:664 [inline]
> __bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
> bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
> trace_timer_start include/trace/events/timer.h:52 [inline]
> enqueue_timer+0x396/0x550 kernel/time/timer.c:663
> internal_add_timer kernel/time/timer.c:688 [inline]
> __mod_timer+0xa0e/0xeb0 kernel/time/timer.c:1183
> call_timer_fn+0x17e/0x600 kernel/time/timer.c:1792
> expire_timers kernel/time/timer.c:1843 [inline]
> __run_timers kernel/time/timer.c:2408 [inline]
> __run_timer_base+0x66a/0x8e0 kernel/time/timer.c:2419
> run_timer_base kernel/time/timer.c:2428 [inline]
> run_timer_softirq+0xb7/0x170 kernel/time/timer.c:2438
> __do_softirq+0x2bc/0x943 kernel/softirq.c:554
> invoke_softirq kernel/softirq.c:428 [inline]
> __irq_exit_rcu+0xf2/0x1c0 kernel/softirq.c:633
> irq_exit_rcu+0x9/0x30 kernel/softirq.c:645
> instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
> sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1043
> asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
> memory_is_poisoned_n mm/kasan/generic.c:130 [inline]
> memory_is_poisoned mm/kasan/generic.c:161 [inline]
> check_region_inline mm/kasan/generic.c:180 [inline]
> kasan_check_range+0x4f/0x290 mm/kasan/generic.c:189
> instrument_atomic_read_write include/linux/instrumented.h:96 [inline]
> atomic_try_cmpxchg_acquire include/linux/atomic/atomic-instrumented.h:1301 [inline]
> queued_spin_lock include/asm-generic/qspinlock.h:111 [inline]
> do_raw_spin_lock+0x14f/0x370 kernel/locking/spinlock_debug.c:116
> spin_lock include/linux/spinlock.h:351 [inline]
> lockref_get+0x15/0x60 lib/lockref.c:50
> dget include/linux/dcache.h:333 [inline]
> __traverse_mounts+0x3b4/0x580 fs/namei.c:1401
> traverse_mounts fs/namei.c:1442 [inline]
> handle_mounts fs/namei.c:1545 [inline]
> step_into+0x5e5/0x1080 fs/namei.c:1842
> walk_component fs/namei.c:2010 [inline]
> link_path_walk+0x748/0xea0 fs/namei.c:2331
> path_lookupat+0xa9/0x450 fs/namei.c:2484
> filename_lookup+0x256/0x610 fs/namei.c:2514
> user_path_at_empty+0x42/0x60 fs/namei.c:2921
> do_readlinkat+0x118/0x3b0 fs/stat.c:499
> __do_sys_readlink fs/stat.c:532 [inline]
> __se_sys_readlink fs/stat.c:529 [inline]
> __x64_sys_readlink+0x7f/0x90 fs/stat.c:529
> do_syscall_64+0xfb/0x240
> entry_SYSCALL_64_after_hwframe+0x6d/0x75
>
> -> #1 (&base->lock){-.-.}-{2:2}:
> lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
> lock_timer_base+0x112/0x240 kernel/time/timer.c:1051
> __mod_timer+0x1ca/0xeb0 kernel/time/timer.c:1132
> queue_delayed_work_on+0x15a/0x260 kernel/workqueue.c:2595
> kvfree_call_rcu+0x47f/0x790 kernel/rcu/tree.c:3472
> rtnl_register_internal+0x482/0x590 net/core/rtnetlink.c:265
> rtnl_register+0x36/0x80 net/core/rtnetlink.c:315
> ip_rt_init+0x2f5/0x3a0 net/ipv4/route.c:3719
> ip_init+0xe/0x20 net/ipv4/ip_output.c:1664
> inet_init+0x3d8/0x580 net/ipv4/af_inet.c:2022
> do_one_initcall+0x238/0x830 init/main.c:1241
> do_initcall_level+0x157/0x210 init/main.c:1303
> do_initcalls+0x3f/0x80 init/main.c:1319
> kernel_init_freeable+0x435/0x5d0 init/main.c:1550
> kernel_init+0x1d/0x2a0 init/main.c:1439
> ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243
>
> -> #0 (krc.lock){..-.}-{2:2}:
> check_prev_add kernel/locking/lockdep.c:3134 [inline]
> check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
> _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
> krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
> bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
> generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
> bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
> __sys_bpf+0x377/0x810
> __do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
> __se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
> __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
> do_syscall_64+0xfb/0x240
> entry_SYSCALL_64_after_hwframe+0x6d/0x75
>
> other info that might help us debug this:
>
> Chain exists of:
> krc.lock --> &base->lock --> &trie->lock
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(&trie->lock);
> lock(&base->lock);
> lock(&trie->lock);
> lock(krc.lock);
>
> *** DEADLOCK ***
>
> 2 locks held by syz-executor.3/6590:
> #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
> #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
> #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: bpf_map_update_value+0x3c4/0x540 kernel/bpf/syscall.c:202
> #1: ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324
>
> stack backtrace:
> CPU: 1 PID: 6590 Comm: syz-executor.3 Not tainted 6.8.0-syzkaller-08951-gfe46a7dd189e #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
> Call Trace:
> <TASK>
> __dump_stack lib/dump_stack.c:88 [inline]
> dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
> check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
> check_prev_add kernel/locking/lockdep.c:3134 [inline]
> check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
> _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
> krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
> bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
> generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
> bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
> __sys_bpf+0x377/0x810
> __do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
> __se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
> __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
> do_syscall_64+0xfb/0x240
> entry_SYSCALL_64_after_hwframe+0x6d/0x75
> RIP: 0033:0x7fe5f987dda9
> Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007fe5fa6000c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
> RAX: ffffffffffffffda RBX: 00007fe5f99abf80 RCX: 00007fe5f987dda9
> RDX: 0000000000000038 RSI: 0000000020000240 RDI: 000000000000001a
> RBP: 00007fe5f98ca47a R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> R13: 000000000000000b R14: 00007fe5f99abf80 R15: 00007ffe908076c8
> </TASK>
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at [email protected].
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> If the report is already addressed, let syzbot know by replying with:
> #syz fix: exact-commit-title
>
> If you want to overwrite report's subsystems, reply with:
> #syz set subsystems: new-subsystem
> (See the list of subsystem names on the web dashboard)
>
> If the report is a duplicate of another one, reply with:
> #syz dup: exact-subject-of-another-report
>
> If you want to undo deduplication, reply with:
> #syz undup
>

2024-03-27 04:38:04

by Paul E. McKenney

Subject: Re: false positive deadlock? Was: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Tue, Mar 26, 2024 at 12:53:35PM -0700, Alexei Starovoitov wrote:
> Hi Paul,
>
> syzbot found an interesting false positive deadlock.
> See below.
> My understanding is the following:
>
> cpu 2:
> grabs timer_base lock
> spins on bpf_lpm lock
>
> cpu 1:
> grab rcu krcp lock
> spins on timer_base lock
>
> cpu 0:
> grab bpf_lpm lock
> spins on rcu krcp lock
>
> bpf_lpm lock can be the same.
> timer_base lock can also be the same due to timer migration.
>
> but rcu krcp lock is always per-cpu, so it cannot be the same lock.
> Hence it's a false positive, but still interesting.
>
> I don't think rcu can tell lockdep that these are different locks.

It might be possible. I will play with this tomorrow, modeling after
the use of lockdep_set_class_and_name() in rcu_init_one(). I am a bit
concerned about systems with thousands of CPUs, but it just might be OK.

> Few ideas/questions on how to address this:
>
> 1. in kernel/rcu/tree.c:
> if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
> schedule_delayed_monitor_work(krcp);
>
> unlock_return:
> krc_this_cpu_unlock(krcp, flags);
>
> moving schedule_delayed_monitor_work() after unlock will not work, right?

If telling lockdep that these are different locks works, that should
be easier. Though maybe Uladzislau can assure me that moving this
schedule_delayed_monitor_work() is OK.

Thanx, Paul

> 2. if not, we can move 4 kfree_rcu-s in kernel/bpf/lpm_trie.c
> to a place after unlock of lpm_trie
>
> 3. move bpf_lpm_trie to bpf_mem_alloc.
>
> The 2 or 3 will address this false positive.
>
> Other ideas?
>
> On Tue, Mar 26, 2024 at 12:00 PM syzbot
> <[email protected]> wrote:
> >
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit: fe46a7dd189e Merge tag 'sound-6.9-rc1' of git://git.kernel..
> > git tree: upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=11547a65180000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=4d90a36f0cab495a
> > dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
> > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> >
> > Unfortunately, I don't have any reproducer for this issue yet.
> >
> > Downloadable assets:
> > disk image: https://storage.googleapis.com/syzbot-assets/f6c04726a2ae/disk-fe46a7dd.raw.xz
> > vmlinux: https://storage.googleapis.com/syzbot-assets/09c26ce901ea/vmlinux-fe46a7dd.xz
> > kernel image: https://storage.googleapis.com/syzbot-assets/134acf7f5322/bzImage-fe46a7dd.xz
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: [email protected]
> >
> > ======================================================
> > WARNING: possible circular locking dependency detected
> > 6.8.0-syzkaller-08951-gfe46a7dd189e #0 Not tainted
> > ------------------------------------------------------
> > syz-executor.3/6590 is trying to acquire lock:
> > ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> > ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> > ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> >
> > but task is already holding lock:
> > ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324
> >
> > which lock already depends on the new lock.
> >
> >
> > the existing dependency chain (in reverse order) is:
> >
> > -> #2 (&trie->lock){..-.}-{2:2}:
> > lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> > __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> > _raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
> > trie_delete_elem+0x96/0x6a0 kernel/bpf/lpm_trie.c:451
> > bpf_prog_510c7248c5f60c92+0x2e/0x46
> > bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
> > __bpf_prog_run include/linux/filter.h:657 [inline]
> > bpf_prog_run include/linux/filter.h:664 [inline]
> > __bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
> > bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
> > trace_timer_start include/trace/events/timer.h:52 [inline]
> > enqueue_timer+0x396/0x550 kernel/time/timer.c:663
> > internal_add_timer kernel/time/timer.c:688 [inline]
> > __mod_timer+0xa0e/0xeb0 kernel/time/timer.c:1183
> > call_timer_fn+0x17e/0x600 kernel/time/timer.c:1792
> > expire_timers kernel/time/timer.c:1843 [inline]
> > __run_timers kernel/time/timer.c:2408 [inline]
> > __run_timer_base+0x66a/0x8e0 kernel/time/timer.c:2419
> > run_timer_base kernel/time/timer.c:2428 [inline]
> > run_timer_softirq+0xb7/0x170 kernel/time/timer.c:2438
> > __do_softirq+0x2bc/0x943 kernel/softirq.c:554
> > invoke_softirq kernel/softirq.c:428 [inline]
> > __irq_exit_rcu+0xf2/0x1c0 kernel/softirq.c:633
> > irq_exit_rcu+0x9/0x30 kernel/softirq.c:645
> > instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
> > sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1043
> > asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
> > memory_is_poisoned_n mm/kasan/generic.c:130 [inline]
> > memory_is_poisoned mm/kasan/generic.c:161 [inline]
> > check_region_inline mm/kasan/generic.c:180 [inline]
> > kasan_check_range+0x4f/0x290 mm/kasan/generic.c:189
> > instrument_atomic_read_write include/linux/instrumented.h:96 [inline]
> > atomic_try_cmpxchg_acquire include/linux/atomic/atomic-instrumented.h:1301 [inline]
> > queued_spin_lock include/asm-generic/qspinlock.h:111 [inline]
> > do_raw_spin_lock+0x14f/0x370 kernel/locking/spinlock_debug.c:116
> > spin_lock include/linux/spinlock.h:351 [inline]
> > lockref_get+0x15/0x60 lib/lockref.c:50
> > dget include/linux/dcache.h:333 [inline]
> > __traverse_mounts+0x3b4/0x580 fs/namei.c:1401
> > traverse_mounts fs/namei.c:1442 [inline]
> > handle_mounts fs/namei.c:1545 [inline]
> > step_into+0x5e5/0x1080 fs/namei.c:1842
> > walk_component fs/namei.c:2010 [inline]
> > link_path_walk+0x748/0xea0 fs/namei.c:2331
> > path_lookupat+0xa9/0x450 fs/namei.c:2484
> > filename_lookup+0x256/0x610 fs/namei.c:2514
> > user_path_at_empty+0x42/0x60 fs/namei.c:2921
> > do_readlinkat+0x118/0x3b0 fs/stat.c:499
> > __do_sys_readlink fs/stat.c:532 [inline]
> > __se_sys_readlink fs/stat.c:529 [inline]
> > __x64_sys_readlink+0x7f/0x90 fs/stat.c:529
> > do_syscall_64+0xfb/0x240
> > entry_SYSCALL_64_after_hwframe+0x6d/0x75
> >
> > -> #1 (&base->lock){-.-.}-{2:2}:
> > lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> > __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> > _raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
> > lock_timer_base+0x112/0x240 kernel/time/timer.c:1051
> > __mod_timer+0x1ca/0xeb0 kernel/time/timer.c:1132
> > queue_delayed_work_on+0x15a/0x260 kernel/workqueue.c:2595
> > kvfree_call_rcu+0x47f/0x790 kernel/rcu/tree.c:3472
> > rtnl_register_internal+0x482/0x590 net/core/rtnetlink.c:265
> > rtnl_register+0x36/0x80 net/core/rtnetlink.c:315
> > ip_rt_init+0x2f5/0x3a0 net/ipv4/route.c:3719
> > ip_init+0xe/0x20 net/ipv4/ip_output.c:1664
> > inet_init+0x3d8/0x580 net/ipv4/af_inet.c:2022
> > do_one_initcall+0x238/0x830 init/main.c:1241
> > do_initcall_level+0x157/0x210 init/main.c:1303
> > do_initcalls+0x3f/0x80 init/main.c:1319
> > kernel_init_freeable+0x435/0x5d0 init/main.c:1550
> > kernel_init+0x1d/0x2a0 init/main.c:1439
> > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243
> >
> > -> #0 (krc.lock){..-.}-{2:2}:
> > check_prev_add kernel/locking/lockdep.c:3134 [inline]
> > check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> > validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> > __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> > lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> > __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
> > _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
> > krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> > add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> > kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> > trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
> > bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
> > generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
> > bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
> > __sys_bpf+0x377/0x810
> > __do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
> > __se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
> > __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
> > do_syscall_64+0xfb/0x240
> > entry_SYSCALL_64_after_hwframe+0x6d/0x75
> >
> > other info that might help us debug this:
> >
> > Chain exists of:
> > krc.lock --> &base->lock --> &trie->lock
> >
> > Possible unsafe locking scenario:
> >
> > CPU0 CPU1
> > ---- ----
> > lock(&trie->lock);
> > lock(&base->lock);
> > lock(&trie->lock);
> > lock(krc.lock);
> >
> > *** DEADLOCK ***
> >
> > 2 locks held by syz-executor.3/6590:
> > #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
> > #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
> > #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: bpf_map_update_value+0x3c4/0x540 kernel/bpf/syscall.c:202
> > #1: ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324
> >
> > stack backtrace:
> > CPU: 1 PID: 6590 Comm: syz-executor.3 Not tainted 6.8.0-syzkaller-08951-gfe46a7dd189e #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
> > Call Trace:
> > <TASK>
> > __dump_stack lib/dump_stack.c:88 [inline]
> > dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
> > check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
> > check_prev_add kernel/locking/lockdep.c:3134 [inline]
> > check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> > validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> > __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> > lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> > __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
> > _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
> > krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> > add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> > kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> > trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
> > bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
> > generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
> > bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
> > __sys_bpf+0x377/0x810
> > __do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
> > __se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
> > __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
> > do_syscall_64+0xfb/0x240
> > entry_SYSCALL_64_after_hwframe+0x6d/0x75
> > RIP: 0033:0x7fe5f987dda9
> > Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
> > RSP: 002b:00007fe5fa6000c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
> > RAX: ffffffffffffffda RBX: 00007fe5f99abf80 RCX: 00007fe5f987dda9
> > RDX: 0000000000000038 RSI: 0000000020000240 RDI: 000000000000001a
> > RBP: 00007fe5f98ca47a R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> > R13: 000000000000000b R14: 00007fe5f99abf80 R15: 00007ffe908076c8
> > </TASK>
> >
> >
> > ---
> > This report is generated by a bot. It may contain errors.
> > See https://goo.gl/tpsmEJ for more information about syzbot.
> > syzbot engineers can be reached at [email protected].
> >
> > syzbot will keep track of this issue. See:
> > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> >
> > If the report is already addressed, let syzbot know by replying with:
> > #syz fix: exact-commit-title
> >
> > If you want to overwrite report's subsystems, reply with:
> > #syz set subsystems: new-subsystem
> > (See the list of subsystem names on the web dashboard)
> >
> > If the report is a duplicate of another one, reply with:
> > #syz dup: exact-subject-of-another-report
> >
> > If you want to undo deduplication, reply with:
> > #syz undup
> >

2024-03-27 05:05:00

by Paul E. McKenney

Subject: Re: false positive deadlock? Was: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Tue, Mar 26, 2024 at 09:37:43PM -0700, Paul E. McKenney wrote:
> On Tue, Mar 26, 2024 at 12:53:35PM -0700, Alexei Starovoitov wrote:
> > Hi Paul,
> >
> > syzbot found an interesting false positive deadlock.
> > See below.
> > My understanding is the following:
> >
> > cpu 2:
> > grabs timer_base lock
> > spins on bpf_lpm lock
> >
> > cpu 1:
> > grab rcu krcp lock
> > spins on timer_base lock
> >
> > cpu 0:
> > grab bpf_lpm lock
> > spins on rcu krcp lock
> >
> > bpf_lpm lock can be the same.
> > timer_base lock can also be the same due to timer migration.
> >
> > but rcu krcp lock is always per-cpu, so it cannot be the same lock.
> > Hence it's a false positive, but still interesting.
> >
> > I don't think rcu can tell lockdep that these are different locks.
>
> It might be possible. I will play with this tomorrow, modeling after
> the use of lockdep_set_class_and_name() in rcu_init_one(). I am a bit
> concerned about systems with thousands of CPUs, but it just might be OK.

Except that each of the resulting separate locks would eventually be
classified as participating in the same type of potential deadlock cycle. :-(

> > Few ideas/questions on how to address this:
> >
> > 1. in kernel/rcu/tree.c:
> > if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
> > schedule_delayed_monitor_work(krcp);
> >
> > unlock_return:
> > krc_this_cpu_unlock(krcp, flags);
> >
> > moving schedule_delayed_monitor_work() after unlock will not work, right?
>
> If telling lockdep that these are different locks works, that should
> be easier. Though maybe Uladzislau can assure me that moving this
> schedule_delayed_monitor_work() is OK.

So it looks like this one needs to be to focus of RCU-side investigation.

Thanx, Paul

> > 2. if not, we can move 4 kfree_rcu-s in kernel/bpf/lpm_trie.c
> > to a place after unlock of lpm_trie
> >
> > 3. move bpf_lpm_trie to bpf_mem_alloc.
> >
> > The 2 or 3 will address this false positive.
> >
> > Other ideas?
> >
> > On Tue, Mar 26, 2024 at 12:00 PM syzbot
> > <[email protected]> wrote:
> > >
> > > Hello,
> > >
> > > syzbot found the following issue on:
> > >
> > > HEAD commit: fe46a7dd189e Merge tag 'sound-6.9-rc1' of git://git.kernel..
> > > git tree: upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=11547a65180000
> > > kernel config: https://syzkaller.appspot.com/x/.config?x=4d90a36f0cab495a
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
> > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> > >
> > > Unfortunately, I don't have any reproducer for this issue yet.
> > >
> > > Downloadable assets:
> > > disk image: https://storage.googleapis.com/syzbot-assets/f6c04726a2ae/disk-fe46a7dd.raw.xz
> > > vmlinux: https://storage.googleapis.com/syzbot-assets/09c26ce901ea/vmlinux-fe46a7dd.xz
> > > kernel image: https://storage.googleapis.com/syzbot-assets/134acf7f5322/bzImage-fe46a7dd.xz
> > >
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: [email protected]
> > >
> > > ======================================================
> > > WARNING: possible circular locking dependency detected
> > > 6.8.0-syzkaller-08951-gfe46a7dd189e #0 Not tainted
> > > ------------------------------------------------------
> > > syz-executor.3/6590 is trying to acquire lock:
> > > ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> > > ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> > > ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> > >
> > > but task is already holding lock:
> > > ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324
> > >
> > > which lock already depends on the new lock.
> > >
> > >
> > > the existing dependency chain (in reverse order) is:
> > >
> > > -> #2 (&trie->lock){..-.}-{2:2}:
> > > lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> > > __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> > > _raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
> > > trie_delete_elem+0x96/0x6a0 kernel/bpf/lpm_trie.c:451
> > > bpf_prog_510c7248c5f60c92+0x2e/0x46
> > > bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
> > > __bpf_prog_run include/linux/filter.h:657 [inline]
> > > bpf_prog_run include/linux/filter.h:664 [inline]
> > > __bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
> > > bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
> > > trace_timer_start include/trace/events/timer.h:52 [inline]
> > > enqueue_timer+0x396/0x550 kernel/time/timer.c:663
> > > internal_add_timer kernel/time/timer.c:688 [inline]
> > > __mod_timer+0xa0e/0xeb0 kernel/time/timer.c:1183
> > > call_timer_fn+0x17e/0x600 kernel/time/timer.c:1792
> > > expire_timers kernel/time/timer.c:1843 [inline]
> > > __run_timers kernel/time/timer.c:2408 [inline]
> > > __run_timer_base+0x66a/0x8e0 kernel/time/timer.c:2419
> > > run_timer_base kernel/time/timer.c:2428 [inline]
> > > run_timer_softirq+0xb7/0x170 kernel/time/timer.c:2438
> > > __do_softirq+0x2bc/0x943 kernel/softirq.c:554
> > > invoke_softirq kernel/softirq.c:428 [inline]
> > > __irq_exit_rcu+0xf2/0x1c0 kernel/softirq.c:633
> > > irq_exit_rcu+0x9/0x30 kernel/softirq.c:645
> > > instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
> > > sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1043
> > > asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
> > > memory_is_poisoned_n mm/kasan/generic.c:130 [inline]
> > > memory_is_poisoned mm/kasan/generic.c:161 [inline]
> > > check_region_inline mm/kasan/generic.c:180 [inline]
> > > kasan_check_range+0x4f/0x290 mm/kasan/generic.c:189
> > > instrument_atomic_read_write include/linux/instrumented.h:96 [inline]
> > > atomic_try_cmpxchg_acquire include/linux/atomic/atomic-instrumented.h:1301 [inline]
> > > queued_spin_lock include/asm-generic/qspinlock.h:111 [inline]
> > > do_raw_spin_lock+0x14f/0x370 kernel/locking/spinlock_debug.c:116
> > > spin_lock include/linux/spinlock.h:351 [inline]
> > > lockref_get+0x15/0x60 lib/lockref.c:50
> > > dget include/linux/dcache.h:333 [inline]
> > > __traverse_mounts+0x3b4/0x580 fs/namei.c:1401
> > > traverse_mounts fs/namei.c:1442 [inline]
> > > handle_mounts fs/namei.c:1545 [inline]
> > > step_into+0x5e5/0x1080 fs/namei.c:1842
> > > walk_component fs/namei.c:2010 [inline]
> > > link_path_walk+0x748/0xea0 fs/namei.c:2331
> > > path_lookupat+0xa9/0x450 fs/namei.c:2484
> > > filename_lookup+0x256/0x610 fs/namei.c:2514
> > > user_path_at_empty+0x42/0x60 fs/namei.c:2921
> > > do_readlinkat+0x118/0x3b0 fs/stat.c:499
> > > __do_sys_readlink fs/stat.c:532 [inline]
> > > __se_sys_readlink fs/stat.c:529 [inline]
> > > __x64_sys_readlink+0x7f/0x90 fs/stat.c:529
> > > do_syscall_64+0xfb/0x240
> > > entry_SYSCALL_64_after_hwframe+0x6d/0x75
> > >
> > > -> #1 (&base->lock){-.-.}-{2:2}:
> > > lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> > > __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> > > _raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
> > > lock_timer_base+0x112/0x240 kernel/time/timer.c:1051
> > > __mod_timer+0x1ca/0xeb0 kernel/time/timer.c:1132
> > > queue_delayed_work_on+0x15a/0x260 kernel/workqueue.c:2595
> > > kvfree_call_rcu+0x47f/0x790 kernel/rcu/tree.c:3472
> > > rtnl_register_internal+0x482/0x590 net/core/rtnetlink.c:265
> > > rtnl_register+0x36/0x80 net/core/rtnetlink.c:315
> > > ip_rt_init+0x2f5/0x3a0 net/ipv4/route.c:3719
> > > ip_init+0xe/0x20 net/ipv4/ip_output.c:1664
> > > inet_init+0x3d8/0x580 net/ipv4/af_inet.c:2022
> > > do_one_initcall+0x238/0x830 init/main.c:1241
> > > do_initcall_level+0x157/0x210 init/main.c:1303
> > > do_initcalls+0x3f/0x80 init/main.c:1319
> > > kernel_init_freeable+0x435/0x5d0 init/main.c:1550
> > > kernel_init+0x1d/0x2a0 init/main.c:1439
> > > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> > > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243
> > >
> > > -> #0 (krc.lock){..-.}-{2:2}:
> > > check_prev_add kernel/locking/lockdep.c:3134 [inline]
> > > check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> > > validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> > > __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> > > lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> > > __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
> > > _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
> > > krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> > > add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> > > kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> > > trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
> > > bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
> > > generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
> > > bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
> > > __sys_bpf+0x377/0x810
> > > __do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
> > > __se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
> > > __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
> > > do_syscall_64+0xfb/0x240
> > > entry_SYSCALL_64_after_hwframe+0x6d/0x75
> > >
> > > other info that might help us debug this:
> > >
> > > Chain exists of:
> > > krc.lock --> &base->lock --> &trie->lock
> > >
> > > Possible unsafe locking scenario:
> > >
> > > CPU0 CPU1
> > > ---- ----
> > > lock(&trie->lock);
> > > lock(&base->lock);
> > > lock(&trie->lock);
> > > lock(krc.lock);
> > >
> > > *** DEADLOCK ***
> > >
> > > 2 locks held by syz-executor.3/6590:
> > > #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
> > > #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
> > > #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: bpf_map_update_value+0x3c4/0x540 kernel/bpf/syscall.c:202
> > > #1: ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324
> > >
> > > stack backtrace:
> > > CPU: 1 PID: 6590 Comm: syz-executor.3 Not tainted 6.8.0-syzkaller-08951-gfe46a7dd189e #0
> > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
> > > Call Trace:
> > > <TASK>
> > > __dump_stack lib/dump_stack.c:88 [inline]
> > > dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
> > > check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
> > > check_prev_add kernel/locking/lockdep.c:3134 [inline]
> > > check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> > > validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> > > __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> > > lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> > > __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
> > > _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
> > > krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> > > add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> > > kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> > > trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
> > > bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
> > > generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
> > > bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
> > > __sys_bpf+0x377/0x810
> > > __do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
> > > __se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
> > > __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
> > > do_syscall_64+0xfb/0x240
> > > entry_SYSCALL_64_after_hwframe+0x6d/0x75
> > > RIP: 0033:0x7fe5f987dda9
> > > Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
> > > RSP: 002b:00007fe5fa6000c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
> > > RAX: ffffffffffffffda RBX: 00007fe5f99abf80 RCX: 00007fe5f987dda9
> > > RDX: 0000000000000038 RSI: 0000000020000240 RDI: 000000000000001a
> > > RBP: 00007fe5f98ca47a R08: 0000000000000000 R09: 0000000000000000
> > > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> > > R13: 000000000000000b R14: 00007fe5f99abf80 R15: 00007ffe908076c8
> > > </TASK>
> > >
> > >
> > > ---
> > > This report is generated by a bot. It may contain errors.
> > > See https://goo.gl/tpsmEJ for more information about syzbot.
> > > syzbot engineers can be reached at [email protected].
> > >
> > > syzbot will keep track of this issue. See:
> > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > >
> > > If the report is already addressed, let syzbot know by replying with:
> > > #syz fix: exact-commit-title
> > >
> > > If you want to overwrite report's subsystems, reply with:
> > > #syz set subsystems: new-subsystem
> > > (See the list of subsystem names on the web dashboard)
> > >
> > > If the report is a duplicate of another one, reply with:
> > > #syz dup: exact-subject-of-another-report
> > >
> > > If you want to undo deduplication, reply with:
> > > #syz undup
> > >

2024-03-27 23:29:43

by syzbot

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

syzbot has found a reproducer for the following issue on:

HEAD commit: 443574b03387 riscv, bpf: Fix kfunc parameters incompatibil..
git tree: bpf
console+strace: https://syzkaller.appspot.com/x/log.txt?x=146398f9180000
kernel config: https://syzkaller.appspot.com/x/.config?x=6fb1be60a193d440
dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12055cc6180000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ca53c9180000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/3f355021a085/disk-443574b0.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/44cf4de7472a/vmlinux-443574b0.xz
kernel image: https://storage.googleapis.com/syzbot-assets/a99a36c7ad65/bzImage-443574b0.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]

======================================================
WARNING: possible circular locking dependency detected
6.8.0-syzkaller-05236-g443574b03387 #0 Not tainted
------------------------------------------------------
syz-executor271/5074 is trying to acquire lock:
ffff8880b9529470 (krc.lock){....}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
ffff8880b9529470 (krc.lock){....}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
ffff8880b9529470 (krc.lock){....}-{2:2}, at: kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444

but task is already holding lock:
ffff888029e171f8 (&trie->lock){-...}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&trie->lock){-...}-{2:2}:
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
trie_delete_elem+0x96/0x6a0 kernel/bpf/lpm_trie.c:451
bpf_prog_2c29ac5cdc6b1842+0x42/0x46
bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
__bpf_prog_run include/linux/filter.h:657 [inline]
bpf_prog_run include/linux/filter.h:664 [inline]
__bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
trace_hrtimer_start include/trace/events/timer.h:222 [inline]
debug_activate kernel/time/hrtimer.c:479 [inline]
enqueue_hrtimer+0x335/0x3a0 kernel/time/hrtimer.c:1090
__hrtimer_start_range_ns kernel/time/hrtimer.c:1265 [inline]
hrtimer_start_range_ns+0xaa0/0xc60 kernel/time/hrtimer.c:1305
hrtimer_start_expires include/linux/hrtimer.h:289 [inline]
hrtimer_sleeper_start_expires kernel/time/hrtimer.c:1972 [inline]
schedule_hrtimeout_range_clock+0x277/0x480 kernel/time/hrtimer.c:2309
poll_schedule_timeout fs/select.c:244 [inline]
do_poll fs/select.c:965 [inline]
do_sys_poll+0xe40/0x1330 fs/select.c:1016
__do_sys_ppoll fs/select.c:1122 [inline]
__se_sys_ppoll+0x2a0/0x330 fs/select.c:1102
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75

-> #1 (hrtimer_bases.lock){-.-.}-{2:2}:
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
lock_hrtimer_base kernel/time/hrtimer.c:175 [inline]
hrtimer_start_range_ns+0xdf/0xc60 kernel/time/hrtimer.c:1303
hrtimer_start include/linux/hrtimer.h:275 [inline]
run_page_cache_worker kernel/rcu/tree.c:3341 [inline]
kvfree_call_rcu+0x5e6/0x790 kernel/rcu/tree.c:3446
rtnl_register_internal+0x482/0x590 net/core/rtnetlink.c:265
rtnl_register+0x36/0x80 net/core/rtnetlink.c:315
ip_rt_init+0x2f5/0x3a0 net/ipv4/route.c:3719
ip_init+0xe/0x20 net/ipv4/ip_output.c:1664
inet_init+0x3d8/0x580 net/ipv4/af_inet.c:2022
do_one_initcall+0x238/0x830 init/main.c:1241
do_initcall_level+0x157/0x210 init/main.c:1303
do_initcalls+0x3f/0x80 init/main.c:1319
kernel_init_freeable+0x435/0x5d0 init/main.c:1557
kernel_init+0x1d/0x2a0 init/main.c:1446
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243

-> #0 (krc.lock){....}-{2:2}:
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
__sys_bpf+0x377/0x810
__do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
__x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75

other info that might help us debug this:

Chain exists of:
krc.lock --> hrtimer_bases.lock --> &trie->lock

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&trie->lock);
lock(hrtimer_bases.lock);
lock(&trie->lock);
lock(krc.lock);

*** DEADLOCK ***

2 locks held by syz-executor271/5074:
#0: ffffffff8e131920 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
#0: ffffffff8e131920 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
#0: ffffffff8e131920 (rcu_read_lock){....}-{1:2}, at: bpf_map_update_value+0x3c4/0x540 kernel/bpf/syscall.c:202
#1: ffff888029e171f8 (&trie->lock){-...}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324

stack backtrace:
CPU: 1 PID: 5074 Comm: syz-executor271 Not tainted 6.8.0-syzkaller-05236-g443574b03387 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2e0 lib/dump_stack.c:106
check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
__sys_bpf+0x377/0x810
__do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
__x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75
RIP: 0033:0x7f485af3c8e9
Code: 48 83 c4 28 c3 e8 37 17 00 00 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffc5f419378 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007ffc5f419548 RCX: 00007f485af3c8e9
RDX: 0000000000000038 RSI: 0000000020000000 RDI: 000000000000001a
RBP: 00007f485afaf610 R08: 00007ffc5f419548 R09: 00007ffc5f419548
R10: 00007ffc5f419548 R11: 0000000000000246 R12: 0000000000000001
R13: 00007ffc5f419538 R14: 0000000000000001 R15: 0000000000000001
</TASK>


---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

2024-03-28 00:14:32

by Alexei Starovoitov

Subject: Re: false positive deadlock? Was: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Tue, Mar 26, 2024 at 10:04 PM Paul E. McKenney <[email protected]> wrote:
>
> On Tue, Mar 26, 2024 at 09:37:43PM -0700, Paul E. McKenney wrote:
> > On Tue, Mar 26, 2024 at 12:53:35PM -0700, Alexei Starovoitov wrote:
> > > Hi Paul,
> > >
> > > syzbot found an interesting false positive deadlock.
> > > See below.
> > > My understanding is the following:
> > >
> > > cpu 2:
> > > grabs timer_base lock
> > > spins on bpf_lpm lock
> > >
> > > cpu 1:
> > > grab rcu krcp lock
> > > spins on timer_base lock
> > >
> > > cpu 0:
> > > grab bpf_lpm lock
> > > spins on rcu krcp lock
> > >
> > > bpf_lpm lock can be the same.
> > > timer_base lock can also be the same due to timer migration.
> > >
> > > but rcu krcp lock is always per-cpu, so it cannot be the same lock.
> > > Hence it's a false positive, but still interesting.
> > >
> > > I don't think rcu can tell lockdep that these are different locks.
> >
> > It might be possible. I will play with this tomorrow, modeling after
> > the use of lockdep_set_class_and_name() in rcu_init_one(). I am a bit
> > concerned about systems with thousands of CPUs, but it just might be OK.
>
> Except that each of the resulting separate locks would eventually be
> classified as participating in the same type of potential deadlock cycle. :-(

That sounds like we have to address it on bpf side,
since we're being spammed with syzbot reports of various
forms and all of them are about this false positive.

2024-03-28 23:13:28

by Hillf Danton

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Wed, 27 Mar 2024 16:27:19 -0700
> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: 443574b03387 riscv, bpf: Fix kfunc parameters incompatibil..
> git tree: bpf
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ca53c9180000

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

--- x/kernel/rcu/tree.c
+++ y/kernel/rcu/tree.c
@@ -5149,6 +5149,8 @@ static void __init rcu_dump_rcu_node_tre

struct workqueue_struct *rcu_gp_wq;

+static DEFINE_PER_CPU(struct lock_class_key, krc_lock_key);
+
static void __init kfree_rcu_batch_init(void)
{
int cpu;
@@ -5169,6 +5171,11 @@ static void __init kfree_rcu_batch_init(

for_each_possible_cpu(cpu) {
struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu);
+ struct lock_class_key *key = per_cpu_ptr(&krc_lock_key, cpu);
+ char name[32] = {0};
+
+ sprintf(name, "krc%d", cpu);
+ lockdep_set_class_and_name(&krcp->lock, key, name);

for (i = 0; i < KFREE_N_BATCHES; i++) {
INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work);
--

2024-03-29 16:17:12

by syzbot

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

Hello,

syzbot tried to test the proposed patch but the build/boot failed:

rd
[ 8.126582][ T1] usbcore: registered new interface driver dln2
[ 8.128975][ T1] usbcore: registered new interface driver pn533_usb
[ 8.136035][ T1] nfcsim 0.2 initialized
[ 8.137231][ T1] usbcore: registered new interface driver port100
[ 8.138748][ T1] usbcore: registered new interface driver nfcmrvl
[ 8.145859][ T1] Loading iSCSI transport class v2.0-870.
[ 8.163499][ T1] virtio_scsi virtio0: 1/0/0 default/read/poll queues
[ 8.175560][ T1] ------------[ cut here ]------------
[ 8.176754][ T1] refcount_t: decrement hit 0; leaking memory.
[ 8.178277][ T1] WARNING: CPU: 0 PID: 1 at lib/refcount.c:31 refcount_warn_saturate+0xfa/0x1d0
[ 8.179920][ T1] Modules linked in:
[ 8.180674][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc1-syzkaller-g317c7bc0ef03-dirty #0
[ 8.183676][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
[ 8.188610][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
[ 8.190118][ T1] Code: b2 00 00 00 e8 b7 d1 e9 fc 5b 5d c3 cc cc cc cc e8 ab d1 e9 fc c6 05 6e 76 e8 0a 01 90 48 c7 c7 e0 33 1f 8c e8 e7 6d ac fc 90 <0f> 0b 90 90 eb d9 e8 8b d1 e9 fc c6 05 4b 76 e8 0a 01 90 48 c7 c7
[ 8.193646][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
[ 8.195167][ T1] RAX: f6afc450a77aa400 RBX: ffff8880207ba75c RCX: ffff8880166d0000
[ 8.197227][ T1] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
[ 8.198422][ T1] RBP: 0000000000000004 R08: ffffffff815800c2 R09: fffffbfff1c396e0
[ 8.200207][ T1] R10: dffffc0000000000 R11: fffffbfff1c396e0 R12: ffffea000501edc0
[ 8.201595][ T1] R13: ffffea000501edc8 R14: 1ffffd4000a03db9 R15: 0000000000000000
[ 8.204095][ T1] FS: 0000000000000000(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
[ 8.206467][ T1] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8.207845][ T1] CR2: ffff88823ffff000 CR3: 000000000e132000 CR4: 00000000003506f0
[ 8.209685][ T1] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8.211033][ T1] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 8.212686][ T1] Call Trace:
[ 8.213195][ T1] <TASK>
[ 8.213986][ T1] ? __warn+0x163/0x4e0
[ 8.215506][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 8.216913][ T1] ? report_bug+0x2b3/0x500
[ 8.218714][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 8.220709][ T1] ? handle_bug+0x3e/0x70
[ 8.222091][ T1] ? exc_invalid_op+0x1a/0x50
[ 8.223846][ T1] ? asm_exc_invalid_op+0x1a/0x20
[ 8.225196][ T1] ? __warn_printk+0x292/0x360
[ 8.226085][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 8.227407][ T1] ? refcount_warn_saturate+0xf9/0x1d0
[ 8.228175][ T1] __free_pages_ok+0xc60/0xd90
[ 8.229175][ T1] make_alloc_exact+0xa3/0xf0
[ 8.230474][ T1] vring_alloc_queue_split+0x20a/0x600
[ 8.231715][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
[ 8.233617][ T1] ? vp_find_vqs+0x4c/0x4e0
[ 8.234631][ T1] ? virtscsi_probe+0x3ea/0xf60
[ 8.236180][ T1] ? virtio_dev_probe+0x991/0xaf0
[ 8.236983][ T1] ? really_probe+0x2b8/0xad0
[ 8.237853][ T1] ? driver_probe_device+0x50/0x430
[ 8.238614][ T1] vring_create_virtqueue_split+0xc6/0x310
[ 8.239928][ T1] ? ret_from_fork+0x4b/0x80
[ 8.242045][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
[ 8.245917][ T1] vring_create_virtqueue+0xca/0x110
[ 8.247389][ T1] ? __pfx_vp_notify+0x10/0x10
[ 8.248236][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.250654][ T1] setup_vq+0xe9/0x2d0
[ 8.252034][ T1] ? __pfx_vp_notify+0x10/0x10
[ 8.254018][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.256278][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.258795][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.260398][ T1] vp_setup_vq+0xbf/0x330
[ 8.261387][ T1] ? __pfx_vp_config_changed+0x10/0x10
[ 8.262783][ T1] ? ioread16+0x2f/0x90
[ 8.265102][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.267083][ T1] vp_find_vqs_msix+0x8b2/0xc80
[ 8.268386][ T1] vp_find_vqs+0x4c/0x4e0
[ 8.270033][ T1] virtscsi_init+0x8db/0xd00
[ 8.272136][ T1] ? __pfx_virtscsi_init+0x10/0x10
[ 8.273670][ T1] ? __pfx_default_calc_sets+0x10/0x10
[ 8.275138][ T1] ? scsi_host_alloc+0xa57/0xea0
[ 8.276322][ T1] ? vp_get+0xfd/0x140
[ 8.277792][ T1] virtscsi_probe+0x3ea/0xf60
[ 8.278863][ T1] ? __pfx_virtscsi_probe+0x10/0x10
[ 8.280218][ T1] ? kernfs_add_one+0x156/0x8b0
[ 8.282404][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
[ 8.284631][ T1] ? virtio_features_ok+0x10c/0x270
[ 8.286348][ T1] virtio_dev_probe+0x991/0xaf0
[ 8.287773][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
[ 8.288905][ T1] really_probe+0x2b8/0xad0
[ 8.290078][ T1] __driver_probe_device+0x1a2/0x390
[ 8.292149][ T1] driver_probe_device+0x50/0x430
[ 8.293916][ T1] __driver_attach+0x45f/0x710
[ 8.295546][ T1] ? __pfx___driver_attach+0x10/0x10
[ 8.296706][ T1] bus_for_each_dev+0x239/0x2b0
[ 8.297510][ T1] ? __pfx___driver_attach+0x10/0x10
[ 8.298632][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
[ 8.299941][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
[ 8.301483][ T1] bus_add_driver+0x347/0x620
[ 8.302606][ T1] driver_register+0x23a/0x320
[ 8.304274][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 8.305978][ T1] virtio_scsi_init+0x65/0xe0
[ 8.306774][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 8.307813][ T1] do_one_initcall+0x248/0x880
[ 8.309150][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 8.310617][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
[ 8.313354][ T1] ? __pfx_do_one_initcall+0x10/0x10
[ 8.315007][ T1] ? __pfx_parse_args+0x10/0x10
[ 8.317331][ T1] ? do_initcalls+0x1c/0x80
[ 8.318887][ T1] ? rcu_is_watching+0x15/0xb0
[ 8.320094][ T1] do_initcall_level+0x157/0x210
[ 8.322194][ T1] do_initcalls+0x3f/0x80
[ 8.323354][ T1] kernel_init_freeable+0x435/0x5d0
[ 8.324941][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
[ 8.326006][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
[ 8.327488][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.329100][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.330857][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.332703][ T1] kernel_init+0x1d/0x2b0
[ 8.334765][ T1] ret_from_fork+0x4b/0x80
[ 8.335714][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.336933][ T1] ret_from_fork_asm+0x1a/0x30
[ 8.337933][ T1] </TASK>
[ 8.338682][ T1] Kernel panic - not syncing: kernel: panic_on_warn set ...
[ 8.341330][ T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.0-rc1-syzkaller-g317c7bc0ef03-dirty #0
[ 8.344143][ T1] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
[ 8.344784][ T1] Call Trace:
[ 8.344784][ T1] <TASK>
[ 8.344784][ T1] dump_stack_lvl+0x241/0x360
[ 8.344784][ T1] ? __pfx_dump_stack_lvl+0x10/0x10
[ 8.344784][ T1] ? __pfx__printk+0x10/0x10
[ 8.344784][ T1] ? _printk+0xd5/0x120
[ 8.344784][ T1] ? vscnprintf+0x5d/0x90
[ 8.354884][ T1] panic+0x349/0x860
[ 8.354884][ T1] ? __warn+0x172/0x4e0
[ 8.354884][ T1] ? __pfx_panic+0x10/0x10
[ 8.354884][ T1] ? show_trace_log_lvl+0x4e6/0x520
[ 8.354884][ T1] ? ret_from_fork_asm+0x1a/0x30
[ 8.354884][ T1] __warn+0x346/0x4e0
[ 8.354884][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 8.354884][ T1] report_bug+0x2b3/0x500
[ 8.364918][ T1] ? refcount_warn_saturate+0xfa/0x1d0
[ 8.364918][ T1] handle_bug+0x3e/0x70
[ 8.364918][ T1] exc_invalid_op+0x1a/0x50
[ 8.364918][ T1] asm_exc_invalid_op+0x1a/0x20
[ 8.364918][ T1] RIP: 0010:refcount_warn_saturate+0xfa/0x1d0
[ 8.364918][ T1] Code: b2 00 00 00 e8 b7 d1 e9 fc 5b 5d c3 cc cc cc cc e8 ab d1 e9 fc c6 05 6e 76 e8 0a 01 90 48 c7 c7 e0 33 1f 8c e8 e7 6d ac fc 90 <0f> 0b 90 90 eb d9 e8 8b d1 e9 fc c6 05 4b 76 e8 0a 01 90 48 c7 c7
[ 8.374911][ T1] RSP: 0000:ffffc90000066e18 EFLAGS: 00010246
[ 8.374911][ T1] RAX: f6afc450a77aa400 RBX: ffff8880207ba75c RCX: ffff8880166d0000
[ 8.374911][ T1] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
[ 8.374911][ T1] RBP: 0000000000000004 R08: ffffffff815800c2 R09: fffffbfff1c396e0
[ 8.384883][ T1] R10: dffffc0000000000 R11: fffffbfff1c396e0 R12: ffffea000501edc0
[ 8.384883][ T1] R13: ffffea000501edc8 R14: 1ffffd4000a03db9 R15: 0000000000000000
[ 8.384883][ T1] ? __warn_printk+0x292/0x360
[ 8.384883][ T1] ? refcount_warn_saturate+0xf9/0x1d0
[ 8.384883][ T1] __free_pages_ok+0xc60/0xd90
[ 8.394940][ T1] make_alloc_exact+0xa3/0xf0
[ 8.394940][ T1] vring_alloc_queue_split+0x20a/0x600
[ 8.394940][ T1] ? __pfx_vring_alloc_queue_split+0x10/0x10
[ 8.394940][ T1] ? vp_find_vqs+0x4c/0x4e0
[ 8.394940][ T1] ? virtscsi_probe+0x3ea/0xf60
[ 8.404787][ T1] ? virtio_dev_probe+0x991/0xaf0
[ 8.404787][ T1] ? really_probe+0x2b8/0xad0
[ 8.404787][ T1] ? driver_probe_device+0x50/0x430
[ 8.404787][ T1] vring_create_virtqueue_split+0xc6/0x310
[ 8.404787][ T1] ? ret_from_fork+0x4b/0x80
[ 8.404787][ T1] ? __pfx_vring_create_virtqueue_split+0x10/0x10
[ 8.404787][ T1] vring_create_virtqueue+0xca/0x110
[ 8.414898][ T1] ? __pfx_vp_notify+0x10/0x10
[ 8.414898][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.414898][ T1] setup_vq+0xe9/0x2d0
[ 8.414898][ T1] ? __pfx_vp_notify+0x10/0x10
[ 8.414898][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.414898][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.414898][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.424822][ T1] vp_setup_vq+0xbf/0x330
[ 8.424822][ T1] ? __pfx_vp_config_changed+0x10/0x10
[ 8.424822][ T1] ? ioread16+0x2f/0x90
[ 8.424822][ T1] ? __pfx_virtscsi_ctrl_done+0x10/0x10
[ 8.424822][ T1] vp_find_vqs_msix+0x8b2/0xc80
[ 8.424822][ T1] vp_find_vqs+0x4c/0x4e0
[ 8.424822][ T1] virtscsi_init+0x8db/0xd00
[ 8.434924][ T1] ? __pfx_virtscsi_init+0x10/0x10
[ 8.434924][ T1] ? __pfx_default_calc_sets+0x10/0x10
[ 8.434924][ T1] ? scsi_host_alloc+0xa57/0xea0
[ 8.434924][ T1] ? vp_get+0xfd/0x140
[ 8.434924][ T1] virtscsi_probe+0x3ea/0xf60
[ 8.434924][ T1] ? __pfx_virtscsi_probe+0x10/0x10
[ 8.434924][ T1] ? kernfs_add_one+0x156/0x8b0
[ 8.444785][ T1] ? virtio_no_restricted_mem_acc+0x9/0x10
[ 8.444785][ T1] ? virtio_features_ok+0x10c/0x270
[ 8.444785][ T1] virtio_dev_probe+0x991/0xaf0
[ 8.444785][ T1] ? __pfx_virtio_dev_probe+0x10/0x10
[ 8.454872][ T1] really_probe+0x2b8/0xad0
[ 8.454872][ T1] __driver_probe_device+0x1a2/0x390
[ 8.454872][ T1] driver_probe_device+0x50/0x430
[ 8.454872][ T1] __driver_attach+0x45f/0x710
[ 8.454872][ T1] ? __pfx___driver_attach+0x10/0x10
[ 8.454872][ T1] bus_for_each_dev+0x239/0x2b0
[ 8.454872][ T1] ? __pfx___driver_attach+0x10/0x10
[ 8.464890][ T1] ? __pfx_bus_for_each_dev+0x10/0x10
[ 8.464890][ T1] ? do_raw_spin_unlock+0x13c/0x8b0
[ 8.464890][ T1] bus_add_driver+0x347/0x620
[ 8.464890][ T1] driver_register+0x23a/0x320
[ 8.464890][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 8.464890][ T1] virtio_scsi_init+0x65/0xe0
[ 8.464890][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 8.464890][ T1] do_one_initcall+0x248/0x880
[ 8.474920][ T1] ? __pfx_virtio_scsi_init+0x10/0x10
[ 8.474920][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
[ 8.474920][ T1] ? __pfx_do_one_initcall+0x10/0x10
[ 8.474920][ T1] ? __pfx_parse_args+0x10/0x10
[ 8.474920][ T1] ? do_initcalls+0x1c/0x80
[ 8.474920][ T1] ? rcu_is_watching+0x15/0xb0
[ 8.474920][ T1] do_initcall_level+0x157/0x210
[ 8.474920][ T1] do_initcalls+0x3f/0x80
[ 8.474920][ T1] kernel_init_freeable+0x435/0x5d0
[ 8.474920][ T1] ? __pfx_kernel_init_freeable+0x10/0x10
[ 8.484767][ T1] ? __pfx_lockdep_hardirqs_on_prepare+0x10/0x10
[ 8.484767][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.484767][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.484767][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.484767][ T1] kernel_init+0x1d/0x2b0
[ 8.484767][ T1] ret_from_fork+0x4b/0x80
[ 8.484767][ T1] ? __pfx_kernel_init+0x10/0x10
[ 8.484767][ T1] ret_from_fork_asm+0x1a/0x30
[ 8.484767][ T1] </TASK>
[ 8.484767][ T1] Kernel Offset: disabled
[ 8.484767][ T1] Rebooting in 86400 seconds..


syzkaller build log:
go env (err=<nil>)
GO111MODULE='auto'
GOARCH='amd64'
GOBIN=''
GOCACHE='/syzkaller/.cache/go-build'
GOENV='/syzkaller/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/syzkaller/jobs-2/linux/gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/syzkaller/jobs-2/linux/gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/syzkaller/jobs-2/linux/gopath/src/github.com/google/syzkaller/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3987916201=/tmp/go-build -gno-record-gcc-switches'

git status (err=<nil>)
HEAD detached at 454571b6a
nothing to commit, working tree clean


tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Makefile:31: run command via tools/syz-env for best compatibility, see:
Makefile:32: https://github.com/google/syzkaller/blob/master/docs/contributing.md#using-syz-env
go list -f '{{.Stale}}' ./sys/syz-sysgen | grep -q false || go install ./sys/syz-sysgen
make .descriptions
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Makefile:31: run command via tools/syz-env for best compatibility, see:
Makefile:32: https://github.com/google/syzkaller/blob/master/docs/contributing.md#using-syz-env
bin/syz-sysgen
touch .descriptions
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=454571b6a16598f5a6e015b9fb1a04932bce7ab9 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240326-163935'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-fuzzer github.com/google/syzkaller/syz-fuzzer
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=454571b6a16598f5a6e015b9fb1a04932bce7ab9 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240326-163935'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-execprog github.com/google/syzkaller/tools/syz-execprog
GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w -X github.com/google/syzkaller/prog.GitRevision=454571b6a16598f5a6e015b9fb1a04932bce7ab9 -X 'github.com/google/syzkaller/prog.gitRevisionDate=20240326-163935'" "-tags=syz_target syz_os_linux syz_arch_amd64 " -o ./bin/linux_amd64/syz-stress github.com/google/syzkaller/tools/syz-stress
mkdir -p ./bin/linux_amd64
gcc -o ./bin/linux_amd64/syz-executor executor/executor.cc \
-m64 -O2 -pthread -Wall -Werror -Wparentheses -Wunused-const-variable -Wframe-larger-than=16384 -Wno-stringop-overflow -Wno-array-bounds -Wno-format-overflow -Wno-unused-but-set-variable -Wno-unused-command-line-argument -static-pie -fpermissive -w -DGOOS_linux=1 -DGOARCH_amd64=1 \
-DHOSTGOOS_linux=1 -DGIT_REVISION=\"454571b6a16598f5a6e015b9fb1a04932bce7ab9\"


Error text is too large and was truncated, full error text is at:
https://syzkaller.appspot.com/x/error.txt?x=17e67d0d180000


Tested on:

commit: 317c7bc0 Merge tag 'mmc-v6.9-rc1' of git://git.kernel...
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
kernel config: https://syzkaller.appspot.com/x/.config?x=7b667bc37450fdcd
dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=144ea745180000

2024-03-30 00:28:33

by Hillf Danton

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Wed, 27 Mar 2024 16:27:19 -0700
> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: 443574b03387 riscv, bpf: Fix kfunc parameters incompatibil..
> git tree: bpf
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ca53c9180000

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git fe46a7dd189e

--- x/kernel/rcu/tree.c
+++ y/kernel/rcu/tree.c
@@ -5149,6 +5149,8 @@ static void __init rcu_dump_rcu_node_tre

struct workqueue_struct *rcu_gp_wq;

+static DEFINE_PER_CPU(struct lock_class_key, krc_lock_key);
+
static void __init kfree_rcu_batch_init(void)
{
int cpu;
@@ -5169,6 +5171,11 @@ static void __init kfree_rcu_batch_init(

for_each_possible_cpu(cpu) {
struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu);
+ struct lock_class_key *key = per_cpu_ptr(&krc_lock_key, cpu);
+ char name[32] = {0};
+
+ sprintf(name, "krc%d", cpu);
+ lockdep_set_class_and_name(&krcp->lock, key, name);

for (i = 0; i < KFREE_N_BATCHES; i++) {
INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work);
--

2024-03-30 15:34:12

by syzbot

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
possible deadlock in kvfree_call_rcu

======================================================
WARNING: possible circular locking dependency detected
6.8.0-syzkaller-08951-gfe46a7dd189e-dirty #0 Not tainted
------------------------------------------------------
syz-executor.0/5491 is trying to acquire lock:
ffff8880b9529470 (#2){..-.}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
ffff8880b9529470 (#2){..-.}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
ffff8880b9529470 (#2){..-.}-{2:2}, at: kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444

but task is already holding lock:
ffff88806aa8d9f8 (&trie->lock){....}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&trie->lock){....}-{2:2}:
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
trie_delete_elem+0x96/0x6a0 kernel/bpf/lpm_trie.c:451
bpf_prog_2c29ac5cdc6b1842+0x42/0x46
bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
__bpf_prog_run include/linux/filter.h:657 [inline]
bpf_prog_run include/linux/filter.h:664 [inline]
__bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
trace_hrtimer_start include/trace/events/timer.h:222 [inline]
debug_activate kernel/time/hrtimer.c:479 [inline]
enqueue_hrtimer+0x335/0x3a0 kernel/time/hrtimer.c:1090
__hrtimer_start_range_ns kernel/time/hrtimer.c:1265 [inline]
hrtimer_start_range_ns+0xaa0/0xc60 kernel/time/hrtimer.c:1305
futex_wait_queue+0xb0/0x1d0 kernel/futex/waitwake.c:357
__futex_wait+0x17f/0x320 kernel/futex/waitwake.c:669
futex_wait+0x101/0x360 kernel/futex/waitwake.c:697
do_futex+0x33b/0x560 kernel/futex/syscalls.c:102
__do_sys_futex kernel/futex/syscalls.c:179 [inline]
__se_sys_futex+0x3f9/0x480 kernel/futex/syscalls.c:160
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75

-> #1 (hrtimer_bases.lock){-.-.}-{2:2}:
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
lock_hrtimer_base kernel/time/hrtimer.c:175 [inline]
hrtimer_start_range_ns+0xdf/0xc60 kernel/time/hrtimer.c:1303
hrtimer_start include/linux/hrtimer.h:275 [inline]
run_page_cache_worker kernel/rcu/tree.c:3341 [inline]
kvfree_call_rcu+0x5e6/0x790 kernel/rcu/tree.c:3446
net_assign_generic net/core/net_namespace.c:115 [inline]
ops_init+0x2c6/0x610 net/core/net_namespace.c:130
__register_pernet_operations net/core/net_namespace.c:1243 [inline]
register_pernet_operations+0x2cb/0x660 net/core/net_namespace.c:1312
register_pernet_device+0x33/0x80 net/core/net_namespace.c:1399
init_mac80211_hwsim+0x12f/0xa90 drivers/net/wireless/virtual/mac80211_hwsim.c:6705
do_one_initcall+0x238/0x830 init/main.c:1241
do_initcall_level+0x157/0x210 init/main.c:1303
do_initcalls+0x3f/0x80 init/main.c:1319
kernel_init_freeable+0x435/0x5d0 init/main.c:1550
kernel_init+0x1d/0x2a0 init/main.c:1439
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243

-> #0 (#2){..-.}-{2:2}:
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
__sys_bpf+0x377/0x810
__do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
__x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75

other info that might help us debug this:

Chain exists of:
#2 --> hrtimer_bases.lock --> &trie->lock

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&trie->lock);
lock(hrtimer_bases.lock);
lock(&trie->lock);
lock(#2);

*** DEADLOCK ***

2 locks held by syz-executor.0/5491:
#0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
#0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
#0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: bpf_map_update_value+0x3c4/0x540 kernel/bpf/syscall.c:202
#1: ffff88806aa8d9f8 (&trie->lock){....}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324

stack backtrace:
CPU: 1 PID: 5491 Comm: syz-executor.0 Not tainted 6.8.0-syzkaller-08951-gfe46a7dd189e-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
__sys_bpf+0x377/0x810
__do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
__x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75
RIP: 0033:0x7f4922e7dda9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f4923b110c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007f4922fabf80 RCX: 00007f4922e7dda9
RDX: 0000000000000038 RSI: 0000000020000000 RDI: 000000000000001a
RBP: 00007f4922eca47a R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f4922fabf80 R15: 00007ffdff5b6398
</TASK>


Tested on:

commit: fe46a7dd Merge tag 'sound-6.9-rc1' of git://git.kernel..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=106b0795180000
kernel config: https://syzkaller.appspot.com/x/.config?x=fe78468a74fdc3b7
dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=15a8bd41180000

2024-03-30 17:56:01

by Uladzislau Rezki

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Tue, Mar 26, 2024 at 12:00:22PM -0700, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: fe46a7dd189e Merge tag 'sound-6.9-rc1' of git://git.kernel..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=11547a65180000
> kernel config: https://syzkaller.appspot.com/x/.config?x=4d90a36f0cab495a
> dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/f6c04726a2ae/disk-fe46a7dd.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/09c26ce901ea/vmlinux-fe46a7dd.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/134acf7f5322/bzImage-fe46a7dd.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: [email protected]
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 6.8.0-syzkaller-08951-gfe46a7dd189e #0 Not tainted
> ------------------------------------------------------
> syz-executor.3/6590 is trying to acquire lock:
> ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> ffff8880b9529470 (krc.lock){..-.}-{2:2}, at: kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
>
> but task is already holding lock:
> ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #2 (&trie->lock){..-.}-{2:2}:
> lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
> trie_delete_elem+0x96/0x6a0 kernel/bpf/lpm_trie.c:451
> bpf_prog_510c7248c5f60c92+0x2e/0x46
> bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
> __bpf_prog_run include/linux/filter.h:657 [inline]
> bpf_prog_run include/linux/filter.h:664 [inline]
> __bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
> bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
> trace_timer_start include/trace/events/timer.h:52 [inline]
> enqueue_timer+0x396/0x550 kernel/time/timer.c:663
> internal_add_timer kernel/time/timer.c:688 [inline]
> __mod_timer+0xa0e/0xeb0 kernel/time/timer.c:1183
> call_timer_fn+0x17e/0x600 kernel/time/timer.c:1792
> expire_timers kernel/time/timer.c:1843 [inline]
> __run_timers kernel/time/timer.c:2408 [inline]
> __run_timer_base+0x66a/0x8e0 kernel/time/timer.c:2419
> run_timer_base kernel/time/timer.c:2428 [inline]
> run_timer_softirq+0xb7/0x170 kernel/time/timer.c:2438
> __do_softirq+0x2bc/0x943 kernel/softirq.c:554
> invoke_softirq kernel/softirq.c:428 [inline]
> __irq_exit_rcu+0xf2/0x1c0 kernel/softirq.c:633
> irq_exit_rcu+0x9/0x30 kernel/softirq.c:645
> instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
> sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1043
> asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
> memory_is_poisoned_n mm/kasan/generic.c:130 [inline]
> memory_is_poisoned mm/kasan/generic.c:161 [inline]
> check_region_inline mm/kasan/generic.c:180 [inline]
> kasan_check_range+0x4f/0x290 mm/kasan/generic.c:189
> instrument_atomic_read_write include/linux/instrumented.h:96 [inline]
> atomic_try_cmpxchg_acquire include/linux/atomic/atomic-instrumented.h:1301 [inline]
> queued_spin_lock include/asm-generic/qspinlock.h:111 [inline]
> do_raw_spin_lock+0x14f/0x370 kernel/locking/spinlock_debug.c:116
> spin_lock include/linux/spinlock.h:351 [inline]
> lockref_get+0x15/0x60 lib/lockref.c:50
> dget include/linux/dcache.h:333 [inline]
> __traverse_mounts+0x3b4/0x580 fs/namei.c:1401
> traverse_mounts fs/namei.c:1442 [inline]
> handle_mounts fs/namei.c:1545 [inline]
> step_into+0x5e5/0x1080 fs/namei.c:1842
> walk_component fs/namei.c:2010 [inline]
> link_path_walk+0x748/0xea0 fs/namei.c:2331
> path_lookupat+0xa9/0x450 fs/namei.c:2484
> filename_lookup+0x256/0x610 fs/namei.c:2514
> user_path_at_empty+0x42/0x60 fs/namei.c:2921
> do_readlinkat+0x118/0x3b0 fs/stat.c:499
> __do_sys_readlink fs/stat.c:532 [inline]
> __se_sys_readlink fs/stat.c:529 [inline]
> __x64_sys_readlink+0x7f/0x90 fs/stat.c:529
> do_syscall_64+0xfb/0x240
> entry_SYSCALL_64_after_hwframe+0x6d/0x75
>
> -> #1 (&base->lock){-.-.}-{2:2}:
> lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
> lock_timer_base+0x112/0x240 kernel/time/timer.c:1051
> __mod_timer+0x1ca/0xeb0 kernel/time/timer.c:1132
> queue_delayed_work_on+0x15a/0x260 kernel/workqueue.c:2595
> kvfree_call_rcu+0x47f/0x790 kernel/rcu/tree.c:3472
> rtnl_register_internal+0x482/0x590 net/core/rtnetlink.c:265
> rtnl_register+0x36/0x80 net/core/rtnetlink.c:315
> ip_rt_init+0x2f5/0x3a0 net/ipv4/route.c:3719
> ip_init+0xe/0x20 net/ipv4/ip_output.c:1664
> inet_init+0x3d8/0x580 net/ipv4/af_inet.c:2022
> do_one_initcall+0x238/0x830 init/main.c:1241
> do_initcall_level+0x157/0x210 init/main.c:1303
> do_initcalls+0x3f/0x80 init/main.c:1319
> kernel_init_freeable+0x435/0x5d0 init/main.c:1550
> kernel_init+0x1d/0x2a0 init/main.c:1439
> ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
> ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:243
>
> -> #0 (krc.lock){..-.}-{2:2}:
> check_prev_add kernel/locking/lockdep.c:3134 [inline]
> check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
> _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
> krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
> bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
> generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
> bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
> __sys_bpf+0x377/0x810
> __do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
> __se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
> __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
> do_syscall_64+0xfb/0x240
> entry_SYSCALL_64_after_hwframe+0x6d/0x75
>
> other info that might help us debug this:
>
> Chain exists of:
> krc.lock --> &base->lock --> &trie->lock
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(&trie->lock);
> lock(&base->lock);
> lock(&trie->lock);
> lock(krc.lock);
>
> *** DEADLOCK ***
>
> 2 locks held by syz-executor.3/6590:
> #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
> #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
> #0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: bpf_map_update_value+0x3c4/0x540 kernel/bpf/syscall.c:202
> #1: ffff888021a271f8 (&trie->lock){..-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324
>
> stack backtrace:
> CPU: 1 PID: 6590 Comm: syz-executor.3 Not tainted 6.8.0-syzkaller-08951-gfe46a7dd189e #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
> Call Trace:
> <TASK>
> __dump_stack lib/dump_stack.c:88 [inline]
> dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
> check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
> check_prev_add kernel/locking/lockdep.c:3134 [inline]
> check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
> _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
> krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
> add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3359 [inline]
> kvfree_call_rcu+0x18a/0x790 kernel/rcu/tree.c:3444
> trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
> bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
> generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
> bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
> __sys_bpf+0x377/0x810
> __do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
> __se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
> __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
> do_syscall_64+0xfb/0x240
> entry_SYSCALL_64_after_hwframe+0x6d/0x75
> RIP: 0033:0x7fe5f987dda9
> Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007fe5fa6000c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
> RAX: ffffffffffffffda RBX: 00007fe5f99abf80 RCX: 00007fe5f987dda9
> RDX: 0000000000000038 RSI: 0000000020000240 RDI: 000000000000001a
> RBP: 00007fe5f98ca47a R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> R13: 000000000000000b R14: 00007fe5f99abf80 R15: 00007ffe908076c8
> </TASK>
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at [email protected].
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> If the report is already addressed, let syzbot know by replying with:
> #syz fix: exact-commit-title
>
> If you want to overwrite report's subsystems, reply with:
> #syz set subsystems: new-subsystem
> (See the list of subsystem names on the web dashboard)
>
> If the report is a duplicate of another one, reply with:
> #syz dup: exact-subject-of-another-report
>
> If you want to undo deduplication, reply with:
> #syz undup
>
Could you please test below:

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index d9642dd06c25..8867aac3668c 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3467,19 +3467,19 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
*/
kmemleak_ignore(ptr);

- // Set timer to drain after KFREE_DRAIN_JIFFIES.
- if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
- schedule_delayed_monitor_work(krcp);
-
unlock_return:
krc_this_cpu_unlock(krcp, flags);

- /*
- * Inline kvfree() after synchronize_rcu(). We can do
- * it from might_sleep() context only, so the current
- * CPU can pass the QS state.
- */
- if (!success) {
+ if (success) {
+ // Set timer to drain after KFREE_DRAIN_JIFFIES.
+ if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
+ schedule_delayed_monitor_work(krcp);
+ } else {
+ /*
+ * Inline kvfree() after synchronize_rcu(). We can do
+ * it from might_sleep() context only, so the current
+ * CPU can pass the QS state.
+ */
debug_rcu_head_unqueue((struct rcu_head *) ptr);
synchronize_rcu();
kvfree(ptr);


--
Uladzislau Rezki

2024-03-30 23:42:52

by Hillf Danton

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Wed, 27 Mar 2024 16:27:19 -0700
> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: 443574b03387 riscv, bpf: Fix kfunc parameters incompatibil..
> git tree: bpf
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ca53c9180000

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git fe46a7dd189e

--- x/kernel/rcu/tree.c
+++ y/kernel/rcu/tree.c
@@ -2957,7 +2957,8 @@ krc_this_cpu_lock(unsigned long *flags)

local_irq_save(*flags); // For safely calling this_cpu_ptr().
krcp = this_cpu_ptr(&krc);
- raw_spin_lock(&krcp->lock);
+ while (!raw_spin_trylock(&krcp->lock))
+ ;

return krcp;
}
--

2024-03-31 05:43:14

by syzbot

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
possible deadlock in hrtimer_start_range_ns

======================================================
WARNING: possible circular locking dependency detected
6.8.0-syzkaller-08951-gfe46a7dd189e-dirty #0 Not tainted
------------------------------------------------------
syz-executor.0/5489 is trying to acquire lock:
ffff8880b952c8d8 (hrtimer_bases.lock){-.-.}-{2:2}, at: lock_hrtimer_base kernel/time/hrtimer.c:175 [inline]
ffff8880b952c8d8 (hrtimer_bases.lock){-.-.}-{2:2}, at: hrtimer_start_range_ns+0xdf/0xc60 kernel/time/hrtimer.c:1303

but task is already holding lock:
ffff88802195e9f8 (&trie->lock){-.-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&trie->lock){-.-.}-{2:2}:
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
trie_delete_elem+0x96/0x6a0 kernel/bpf/lpm_trie.c:451
0xffffffffa00060ce
bpf_dispatcher_nop_func include/linux/bpf.h:1234 [inline]
__bpf_prog_run include/linux/filter.h:657 [inline]
bpf_prog_run include/linux/filter.h:664 [inline]
__bpf_trace_run kernel/trace/bpf_trace.c:2381 [inline]
bpf_trace_run2+0x204/0x420 kernel/trace/bpf_trace.c:2420
trace_hrtimer_start include/trace/events/timer.h:222 [inline]
debug_activate kernel/time/hrtimer.c:479 [inline]
enqueue_hrtimer+0x335/0x3a0 kernel/time/hrtimer.c:1090
__hrtimer_start_range_ns kernel/time/hrtimer.c:1265 [inline]
hrtimer_start_range_ns+0xaa0/0xc60 kernel/time/hrtimer.c:1305
futex_wait_queue+0xb0/0x1d0 kernel/futex/waitwake.c:357
__futex_wait+0x17f/0x320 kernel/futex/waitwake.c:669
futex_wait+0x101/0x360 kernel/futex/waitwake.c:697
do_futex+0x33b/0x560 kernel/futex/syscalls.c:102
__do_sys_futex kernel/futex/syscalls.c:179 [inline]
__se_sys_futex+0x3f9/0x480 kernel/futex/syscalls.c:160
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75

-> #0 (hrtimer_bases.lock){-.-.}-{2:2}:
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
lock_hrtimer_base kernel/time/hrtimer.c:175 [inline]
hrtimer_start_range_ns+0xdf/0xc60 kernel/time/hrtimer.c:1303
hrtimer_start include/linux/hrtimer.h:275 [inline]
run_page_cache_worker kernel/rcu/tree.c:3342 [inline]
kvfree_call_rcu+0x60e/0x7c0 kernel/rcu/tree.c:3447
trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
__sys_bpf+0x377/0x810
__do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
__x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75

other info that might help us debug this:

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&trie->lock);
lock(hrtimer_bases.lock);
lock(&trie->lock);
lock(hrtimer_bases.lock);

*** DEADLOCK ***

3 locks held by syz-executor.0/5489:
#0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:298 [inline]
#0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:750 [inline]
#0: ffffffff8e132020 (rcu_read_lock){....}-{1:2}, at: bpf_map_update_value+0x3c4/0x540 kernel/bpf/syscall.c:202
#1: ffff88802195e9f8 (&trie->lock){-.-.}-{2:2}, at: trie_update_elem+0xcb/0xc10 kernel/bpf/lpm_trie.c:324
#2: ffff8880b9529470 (krc.lock){....}-{2:2}, at: krc_this_cpu_lock kernel/rcu/tree.c:2960 [inline]
#2: ffff8880b9529470 (krc.lock){....}-{2:2}, at: add_ptr_to_bulk_krc_lock kernel/rcu/tree.c:3360 [inline]
#2: ffff8880b9529470 (krc.lock){....}-{2:2}, at: kvfree_call_rcu+0x192/0x7c0 kernel/rcu/tree.c:3445

stack backtrace:
CPU: 1 PID: 5489 Comm: syz-executor.0 Not tainted 6.8.0-syzkaller-08951-gfe46a7dd189e-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0xd5/0x120 kernel/locking/spinlock.c:162
lock_hrtimer_base kernel/time/hrtimer.c:175 [inline]
hrtimer_start_range_ns+0xdf/0xc60 kernel/time/hrtimer.c:1303
hrtimer_start include/linux/hrtimer.h:275 [inline]
run_page_cache_worker kernel/rcu/tree.c:3342 [inline]
kvfree_call_rcu+0x60e/0x7c0 kernel/rcu/tree.c:3447
trie_update_elem+0x819/0xc10 kernel/bpf/lpm_trie.c:385
bpf_map_update_value+0x4d3/0x540 kernel/bpf/syscall.c:203
generic_map_update_batch+0x60d/0x900 kernel/bpf/syscall.c:1876
bpf_map_do_batch+0x3e0/0x690 kernel/bpf/syscall.c:5145
__sys_bpf+0x377/0x810
__do_sys_bpf kernel/bpf/syscall.c:5738 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5736 [inline]
__x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:5736
do_syscall_64+0xfb/0x240
entry_SYSCALL_64_after_hwframe+0x6d/0x75
RIP: 0033:0x7f44a5a7dda9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f44a686a0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007f44a5babf80 RCX: 00007f44a5a7dda9
RDX: 0000000000000038 RSI: 0000000020000000 RDI: 000000000000001a
RBP: 00007f44a5aca47a R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f44a5babf80 R15: 00007ffc1ffeb058
</TASK>


Tested on:

commit: fe46a7dd Merge tag 'sound-6.9-rc1' of git://git.kernel..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=148dffe6180000
kernel config: https://syzkaller.appspot.com/x/.config?x=fe78468a74fdc3b7
dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=15a5ffe6180000

2024-03-31 06:21:03

by Hillf Danton

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Wed, 27 Mar 2024 16:27:19 -0700
> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: 443574b03387 riscv, bpf: Fix kfunc parameters incompatibil..
> git tree: bpf
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ca53c9180000

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git fe46a7dd189e

--- x/kernel/rcu/tree.c
+++ y/kernel/rcu/tree.c
@@ -2957,7 +2957,8 @@ krc_this_cpu_lock(unsigned long *flags)

local_irq_save(*flags); // For safely calling this_cpu_ptr().
krcp = this_cpu_ptr(&krc);
- raw_spin_lock(&krcp->lock);
+ while (!raw_spin_trylock(&krcp->lock))
+ ;

return krcp;
}
--- x/kernel/bpf/lpm_trie.c
+++ y/kernel/bpf/lpm_trie.c
@@ -382,9 +382,10 @@ static long trie_update_elem(struct bpf_
trie->n_entries--;

rcu_assign_pointer(*slot, new_node);
- kfree_rcu(node, rcu);

- goto out;
+ spin_unlock_irqrestore(&trie->lock, irq_flags);
+ kfree_rcu(node, rcu);
+ return 0;
}

/* If the new node matches the prefix completely, it must be inserted
--

2024-03-31 06:26:30

by Hillf Danton

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Sat, 30 Mar 2024 18:55:41 +0100 Uladzislau Rezki <[email protected]>
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index d9642dd06c25..8867aac3668c 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -3467,19 +3467,19 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
> */
> kmemleak_ignore(ptr);
>
> - // Set timer to drain after KFREE_DRAIN_JIFFIES.
> - if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
> - schedule_delayed_monitor_work(krcp);
> -

This is not enough at least WRT run_page_cache_worker() [1]

[1] https://lore.kernel.org/lkml/[email protected]/

while the reason why syzbot failed to catch the zone->per_cpu_pageset in
setup_zone_pageset() in mm/page_alloc.c is trylock [2]

[2] https://lore.kernel.org/lkml/[email protected]/

> unlock_return:
> krc_this_cpu_unlock(krcp, flags);
>
> - /*
> - * Inline kvfree() after synchronize_rcu(). We can do
> - * it from might_sleep() context only, so the current
> - * CPU can pass the QS state.
> - */
> - if (!success) {
> + if (success) {
> + // Set timer to drain after KFREE_DRAIN_JIFFIES.
> + if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
> + schedule_delayed_monitor_work(krcp);
> + } else {
> + /*
> + * Inline kvfree() after synchronize_rcu(). We can do
> + * it from might_sleep() context only, so the current
> + * CPU can pass the QS state.
> + */
> debug_rcu_head_unqueue((struct rcu_head *) ptr);
> synchronize_rcu();
> kvfree(ptr);
>
>
> --

2024-03-31 06:28:34

by Uladzislau Rezki

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Tue, Mar 26, 2024 at 12:00:22PM -0700, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: fe46a7dd189e Merge tag 'sound-6.9-rc1' of git://git.kernel..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=11547a65180000
> kernel config: https://syzkaller.appspot.com/x/.config?x=4d90a36f0cab495a
> dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 7033999ecd7b

--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3467,19 +3467,19 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
*/
kmemleak_ignore(ptr);

- // Set timer to drain after KFREE_DRAIN_JIFFIES.
- if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
- schedule_delayed_monitor_work(krcp);
-
unlock_return:
krc_this_cpu_unlock(krcp, flags);

- /*
- * Inline kvfree() after synchronize_rcu(). We can do
- * it from might_sleep() context only, so the current
- * CPU can pass the QS state.
- */
- if (!success) {
+ if (success) {
+ // Set timer to drain after KFREE_DRAIN_JIFFIES.
+ if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
+ schedule_delayed_monitor_work(krcp);
+ } else {
+ /*
+ * Inline kvfree() after synchronize_rcu(). We can do
+ * it from might_sleep() context only, so the current
+ * CPU can pass the QS state.
+ */
debug_rcu_head_unqueue((struct rcu_head *) ptr);
synchronize_rcu();
kvfree(ptr);

2024-03-31 06:54:12

by syzbot

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

Hello,

syzbot tried to test the proposed patch but the build/boot failed:

failed to apply patch:
checking file kernel/rcu/tree.c
patch: **** unexpected end of file in patch



Tested on:

commit: 7033999e Merge tag 'printk-for-6.9-rc2' of git://git.k..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel config: https://syzkaller.appspot.com/x/.config?x=6fb1be60a193d440
dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
compiler:
patch: https://syzkaller.appspot.com/x/patch.diff?x=1406ae29180000

2024-03-31 07:03:22

by syzbot

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: [email protected]

Tested on:

commit: fe46a7dd Merge tag 'sound-6.9-rc1' of git://git.kernel..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=1424fe29180000
kernel config: https://syzkaller.appspot.com/x/.config?x=fe78468a74fdc3b7
dashboard link: https://syzkaller.appspot.com/bug?extid=1fa663a2100308ab6eab
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=11a3b92d180000

Note: testing is done by a robot and is best-effort only.

2024-03-31 07:26:09

by Uladzislau Rezki

Subject: Re: [syzbot] [bpf?] possible deadlock in kvfree_call_rcu

On Sun, Mar 31, 2024 at 02:23:14PM +0800, Hillf Danton wrote:
> On Sat, 30 Mar 2024 18:55:41 +0100 Uladzislau Rezki <[email protected]>
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index d9642dd06c25..8867aac3668c 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -3467,19 +3467,19 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
> > */
> > kmemleak_ignore(ptr);
> >
> > - // Set timer to drain after KFREE_DRAIN_JIFFIES.
> > - if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
> > - schedule_delayed_monitor_work(krcp);
> > -
>
> This is not enough at least WRT run_page_cache_worker() [1]
>
> [1] https://lore.kernel.org/lkml/[email protected]/
>
page-cache-worker should be move out of the krcp-lock also. I will
update the patch.

--
Uladzislau Rezki