Hello,
syzbot found the following crash on:
HEAD commit: b0be0eff Merge tag 'x86-pti-2020-01-28' of git://git.kerne..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15232735e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=9784e57c96a92f20
dashboard link: https://syzkaller.appspot.com/bug?extid=61cba5033e2072d61806
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: [email protected]
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.5.0-syzkaller #0 Not tainted
-----------------------------------------------------
syz-executor.3/20867 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
ffff888022b68098 (&(&s->cache_lock)->rlock){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
ffff888022b68098 (&(&s->cache_lock)->rlock){+.+.}, at: sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
and this task is already holding:
ffffffff898605d8 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
ffffffff898605d8 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
which would create a new lock dependency:
(&(&nf_conntrack_locks[i])->rlock){+.-.} -> (&(&s->cache_lock)->rlock){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&nf_conntrack_locks[i])->rlock){+.-.}
... which became SOFTIRQ-irq-safe at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
nf_conntrack_double_lock.isra.0+0xac/0x100 net/netfilter/nf_conntrack_core.c:134
__nf_conntrack_confirm+0x252/0x1710 net/netfilter/nf_conntrack_core.c:963
nf_conntrack_confirm include/net/netfilter/nf_conntrack_core.h:63 [inline]
nf_confirm+0x3e7/0x4d0 net/netfilter/nf_conntrack_proto.c:154
ipv4_confirm+0x14c/0x240 net/netfilter/nf_conntrack_proto.c:169
nf_hook_entry_hookfn include/linux/netfilter.h:135 [inline]
nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:512
nf_hook include/linux/netfilter.h:262 [inline]
NF_HOOK include/linux/netfilter.h:305 [inline]
ip_local_deliver+0x352/0x520 net/ipv4/ip_input.c:252
dst_input include/net/dst.h:442 [inline]
ip_sublist_rcv_finish+0x9b/0x2d0 net/ipv4/ip_input.c:549
ip_list_rcv_finish net/ipv4/ip_input.c:599 [inline]
ip_sublist_rcv+0x5a6/0xa20 net/ipv4/ip_input.c:607
ip_list_rcv+0x37e/0x4bf net/ipv4/ip_input.c:642
__netif_receive_skb_list_ptype net/core/dev.c:5193 [inline]
__netif_receive_skb_list_core+0x5fc/0x9d0 net/core/dev.c:5241
__netif_receive_skb_list net/core/dev.c:5293 [inline]
netif_receive_skb_list_internal+0x7eb/0xe50 net/core/dev.c:5388
gro_normal_list.part.0+0x1e/0xb0 net/core/dev.c:5499
gro_normal_list net/core/dev.c:6232 [inline]
napi_complete_done+0x208/0x510 net/core/dev.c:6220
virtqueue_napi_complete+0x31/0xa0 drivers/net/virtio_net.c:329
virtnet_poll+0xbad/0xe90 drivers/net/virtio_net.c:1432
napi_poll net/core/dev.c:6534 [inline]
net_rx_action+0x508/0x1120 net/core/dev.c:6602
__do_softirq+0x262/0x98c kernel/softirq.c:292
run_ksoftirqd kernel/softirq.c:603 [inline]
run_ksoftirqd+0x8e/0x110 kernel/softirq.c:595
smpboot_thread_fn+0x6a3/0xa40 kernel/smpboot.c:165
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
to a SOFTIRQ-irq-unsafe lock:
(&(&s->cache_lock)->rlock){+.+.}
... which became SOFTIRQ-irq-unsafe at:
...
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
avc_audit security/selinux/include/avc.h:140 [inline]
avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
security_mmap_file+0xa4/0x1e0 security/security.c:1482
vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
vm_mmap+0x94/0xc0 mm/util.c:525
elf_map+0x10a/0x2b0 fs/binfmt_elf.c:377
load_elf_binary+0xd4b/0x5310 fs/binfmt_elf.c:982
search_binary_handler fs/exec.c:1658 [inline]
search_binary_handler+0x16d/0x570 fs/exec.c:1635
exec_binprm fs/exec.c:1701 [inline]
__do_execve_file.isra.0+0x1329/0x22b0 fs/exec.c:1821
do_execveat_common fs/exec.c:1867 [inline]
do_execve fs/exec.c:1884 [inline]
__do_sys_execve fs/exec.c:1960 [inline]
__se_sys_execve fs/exec.c:1955 [inline]
__x64_sys_execve+0x8f/0xc0 fs/exec.c:1955
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
other info that might help us debug this:
Possible interrupt unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&(&s->cache_lock)->rlock);
local_irq_disable();
lock(&(&nf_conntrack_locks[i])->rlock);
lock(&(&s->cache_lock)->rlock);
<Interrupt>
lock(&(&nf_conntrack_locks[i])->rlock);
*** DEADLOCK ***
4 locks held by syz-executor.3/20867:
#0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnl_lock net/netfilter/nfnetlink.c:62 [inline]
#0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnetlink_rcv_msg+0x9ee/0xfb0 net/netfilter/nfnetlink.c:224
#1: ffff88800ce3c5d8 (nlk_cb_mutex-NETFILTER){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
#2: ffffffff898605d8 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
#2: ffffffff898605d8 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
#3: ffffffff8b7df008 (&selinux_ss.policy_rwlock){.+.?}, at: security_sid_to_context_core+0x1ca/0x3c0 security/selinux/ss/services.c:1344
the dependencies between SOFTIRQ-irq-safe lock and the holding lock:
-> (&(&nf_conntrack_locks[i])->rlock){+.-.} {
HARDIRQ-ON-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
get_next_corpse net/netfilter/nf_conntrack_core.c:2001 [inline]
nf_ct_iterate_cleanup+0x1d2/0x4e0 net/netfilter/nf_conntrack_core.c:2035
nf_ct_iterate_destroy+0x12e/0x160 net/netfilter/nf_conntrack_core.c:2162
nf_conntrack_helper_unregister+0x111/0x160 net/netfilter/nf_conntrack_helper.c:469
nf_conntrack_helpers_unregister net/netfilter/nf_conntrack_helper.c:532 [inline]
nf_conntrack_helpers_register+0xaa/0xe0 net/netfilter/nf_conntrack_helper.c:523
nf_conntrack_sip_init+0x266/0x2a4 net/netfilter/nf_conntrack_sip.c:1693
do_one_initcall+0x120/0x820 init/main.c:939
do_initcall_level init/main.c:1007 [inline]
do_initcalls init/main.c:1015 [inline]
do_basic_setup init/main.c:1032 [inline]
kernel_init_freeable+0x4ca/0x570 init/main.c:1203
kernel_init+0x12/0x1bf init/main.c:1110
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
IN-SOFTIRQ-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
nf_conntrack_double_lock.isra.0+0xac/0x100 net/netfilter/nf_conntrack_core.c:134
__nf_conntrack_confirm+0x252/0x1710 net/netfilter/nf_conntrack_core.c:963
nf_conntrack_confirm include/net/netfilter/nf_conntrack_core.h:63 [inline]
nf_confirm+0x3e7/0x4d0 net/netfilter/nf_conntrack_proto.c:154
ipv4_confirm+0x14c/0x240 net/netfilter/nf_conntrack_proto.c:169
nf_hook_entry_hookfn include/linux/netfilter.h:135 [inline]
nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:512
nf_hook include/linux/netfilter.h:262 [inline]
NF_HOOK include/linux/netfilter.h:305 [inline]
ip_local_deliver+0x352/0x520 net/ipv4/ip_input.c:252
dst_input include/net/dst.h:442 [inline]
ip_sublist_rcv_finish+0x9b/0x2d0 net/ipv4/ip_input.c:549
ip_list_rcv_finish net/ipv4/ip_input.c:599 [inline]
ip_sublist_rcv+0x5a6/0xa20 net/ipv4/ip_input.c:607
ip_list_rcv+0x37e/0x4bf net/ipv4/ip_input.c:642
__netif_receive_skb_list_ptype net/core/dev.c:5193 [inline]
__netif_receive_skb_list_core+0x5fc/0x9d0 net/core/dev.c:5241
__netif_receive_skb_list net/core/dev.c:5293 [inline]
netif_receive_skb_list_internal+0x7eb/0xe50 net/core/dev.c:5388
gro_normal_list.part.0+0x1e/0xb0 net/core/dev.c:5499
gro_normal_list net/core/dev.c:6232 [inline]
napi_complete_done+0x208/0x510 net/core/dev.c:6220
virtqueue_napi_complete+0x31/0xa0 drivers/net/virtio_net.c:329
virtnet_poll+0xbad/0xe90 drivers/net/virtio_net.c:1432
napi_poll net/core/dev.c:6534 [inline]
net_rx_action+0x508/0x1120 net/core/dev.c:6602
__do_softirq+0x262/0x98c kernel/softirq.c:292
run_ksoftirqd kernel/softirq.c:603 [inline]
run_ksoftirqd+0x8e/0x110 kernel/softirq.c:595
smpboot_thread_fn+0x6a3/0xa40 kernel/smpboot.c:165
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
INITIAL USE at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
get_next_corpse net/netfilter/nf_conntrack_core.c:2001 [inline]
nf_ct_iterate_cleanup+0x1d2/0x4e0 net/netfilter/nf_conntrack_core.c:2035
nf_ct_iterate_destroy+0x12e/0x160 net/netfilter/nf_conntrack_core.c:2162
nf_conntrack_helper_unregister+0x111/0x160 net/netfilter/nf_conntrack_helper.c:469
nf_conntrack_helpers_unregister net/netfilter/nf_conntrack_helper.c:532 [inline]
nf_conntrack_helpers_register+0xaa/0xe0 net/netfilter/nf_conntrack_helper.c:523
nf_conntrack_sip_init+0x266/0x2a4 net/netfilter/nf_conntrack_sip.c:1693
do_one_initcall+0x120/0x820 init/main.c:939
do_initcall_level init/main.c:1007 [inline]
do_initcalls init/main.c:1015 [inline]
do_basic_setup init/main.c:1032 [inline]
kernel_init_freeable+0x4ca/0x570 init/main.c:1203
kernel_init+0x12/0x1bf init/main.c:1110
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
}
... key at: [<ffffffff8c1ad440>] __key.79496+0x0/0x40
... acquired at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put security/selinux/ss/sidtab.c:594 [inline]
sidtab_sid2str_get+0x3b5/0x66e security/selinux/ss/sidtab.c:594
sidtab_entry_to_string+0x37/0x110 security/selinux/ss/services.c:1271
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
selinux_secid_to_secctx+0x2d/0x40 security/selinux/hooks.c:6451
security_secid_to_secctx+0x75/0xc0 security/security.c:1950
ctnetlink_dump_secctx.isra.0+0x8c/0x330 net/netfilter/nf_conntrack_netlink.c:335
ctnetlink_dump_info net/netfilter/nf_conntrack_netlink.c:531 [inline]
ctnetlink_fill_info+0x750/0xd40 net/netfilter/nf_conntrack_netlink.c:591
ctnetlink_dump_table+0x68c/0x1130 net/netfilter/nf_conntrack_netlink.c:979
netlink_dump+0x558/0xfb0 net/netlink/af_netlink.c:2244
__netlink_dump_start+0x66a/0x930 net/netlink/af_netlink.c:2352
netlink_dump_start include/linux/netlink.h:233 [inline]
ctnetlink_get_conntrack+0x55e/0x6a0 net/netfilter/nf_conntrack_netlink.c:1355
nfnetlink_rcv_msg+0xcf2/0xfb0 net/netfilter/nfnetlink.c:229
netlink_rcv_skb+0x177/0x450 net/netlink/af_netlink.c:2477
nfnetlink_rcv+0x1ba/0x460 net/netfilter/nfnetlink.c:563
netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline]
netlink_unicast+0x58c/0x7d0 net/netlink/af_netlink.c:1328
netlink_sendmsg+0x91c/0xea0 net/netlink/af_netlink.c:1917
sock_sendmsg_nosec net/socket.c:639 [inline]
sock_sendmsg+0xd7/0x130 net/socket.c:659
____sys_sendmsg+0x753/0x880 net/socket.c:2330
___sys_sendmsg+0x100/0x170 net/socket.c:2384
__sys_sendmsg+0x105/0x1d0 net/socket.c:2417
__do_sys_sendmsg net/socket.c:2426 [inline]
__se_sys_sendmsg net/socket.c:2424 [inline]
__x64_sys_sendmsg+0x78/0xb0 net/socket.c:2424
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
the dependencies between the lock to be acquired
and SOFTIRQ-irq-unsafe lock:
-> (&(&s->cache_lock)->rlock){+.+.} {
HARDIRQ-ON-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
avc_audit security/selinux/include/avc.h:140 [inline]
avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
security_mmap_file+0xa4/0x1e0 security/security.c:1482
vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
vm_mmap+0x94/0xc0 mm/util.c:525
elf_map+0x10a/0x2b0 fs/binfmt_elf.c:377
load_elf_binary+0xd4b/0x5310 fs/binfmt_elf.c:982
search_binary_handler fs/exec.c:1658 [inline]
search_binary_handler+0x16d/0x570 fs/exec.c:1635
exec_binprm fs/exec.c:1701 [inline]
__do_execve_file.isra.0+0x1329/0x22b0 fs/exec.c:1821
do_execveat_common fs/exec.c:1867 [inline]
do_execve fs/exec.c:1884 [inline]
__do_sys_execve fs/exec.c:1960 [inline]
__se_sys_execve fs/exec.c:1955 [inline]
__x64_sys_execve+0x8f/0xc0 fs/exec.c:1955
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
SOFTIRQ-ON-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
avc_audit security/selinux/include/avc.h:140 [inline]
avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
security_mmap_file+0xa4/0x1e0 security/security.c:1482
vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
vm_mmap+0x94/0xc0 mm/util.c:525
elf_map+0x10a/0x2b0 fs/binfmt_elf.c:377
load_elf_binary+0xd4b/0x5310 fs/binfmt_elf.c:982
search_binary_handler fs/exec.c:1658 [inline]
search_binary_handler+0x16d/0x570 fs/exec.c:1635
exec_binprm fs/exec.c:1701 [inline]
__do_execve_file.isra.0+0x1329/0x22b0 fs/exec.c:1821
do_execveat_common fs/exec.c:1867 [inline]
do_execve fs/exec.c:1884 [inline]
__do_sys_execve fs/exec.c:1960 [inline]
__se_sys_execve fs/exec.c:1955 [inline]
__x64_sys_execve+0x8f/0xc0 fs/exec.c:1955
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
INITIAL USE at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
avc_audit security/selinux/include/avc.h:140 [inline]
avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
security_mmap_file+0xa4/0x1e0 security/security.c:1482
vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
vm_mmap+0x94/0xc0 mm/util.c:525
elf_map+0x10a/0x2b0 fs/binfmt_elf.c:377
load_elf_binary+0xd4b/0x5310 fs/binfmt_elf.c:982
search_binary_handler fs/exec.c:1658 [inline]
search_binary_handler+0x16d/0x570 fs/exec.c:1635
exec_binprm fs/exec.c:1701 [inline]
__do_execve_file.isra.0+0x1329/0x22b0 fs/exec.c:1821
do_execveat_common fs/exec.c:1867 [inline]
do_execve fs/exec.c:1884 [inline]
__do_sys_execve fs/exec.c:1960 [inline]
__se_sys_execve fs/exec.c:1955 [inline]
__x64_sys_execve+0x8f/0xc0 fs/exec.c:1955
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
}
... key at: [<ffffffff8b7dec80>] __key.69173+0x0/0x40
... acquired at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put security/selinux/ss/sidtab.c:594 [inline]
sidtab_sid2str_get+0x3b5/0x66e security/selinux/ss/sidtab.c:594
sidtab_entry_to_string+0x37/0x110 security/selinux/ss/services.c:1271
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
selinux_secid_to_secctx+0x2d/0x40 security/selinux/hooks.c:6451
security_secid_to_secctx+0x75/0xc0 security/security.c:1950
ctnetlink_dump_secctx.isra.0+0x8c/0x330 net/netfilter/nf_conntrack_netlink.c:335
ctnetlink_dump_info net/netfilter/nf_conntrack_netlink.c:531 [inline]
ctnetlink_fill_info+0x750/0xd40 net/netfilter/nf_conntrack_netlink.c:591
ctnetlink_dump_table+0x68c/0x1130 net/netfilter/nf_conntrack_netlink.c:979
netlink_dump+0x558/0xfb0 net/netlink/af_netlink.c:2244
__netlink_dump_start+0x66a/0x930 net/netlink/af_netlink.c:2352
netlink_dump_start include/linux/netlink.h:233 [inline]
ctnetlink_get_conntrack+0x55e/0x6a0 net/netfilter/nf_conntrack_netlink.c:1355
nfnetlink_rcv_msg+0xcf2/0xfb0 net/netfilter/nfnetlink.c:229
netlink_rcv_skb+0x177/0x450 net/netlink/af_netlink.c:2477
nfnetlink_rcv+0x1ba/0x460 net/netfilter/nfnetlink.c:563
netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline]
netlink_unicast+0x58c/0x7d0 net/netlink/af_netlink.c:1328
netlink_sendmsg+0x91c/0xea0 net/netlink/af_netlink.c:1917
sock_sendmsg_nosec net/socket.c:639 [inline]
sock_sendmsg+0xd7/0x130 net/socket.c:659
____sys_sendmsg+0x753/0x880 net/socket.c:2330
___sys_sendmsg+0x100/0x170 net/socket.c:2384
__sys_sendmsg+0x105/0x1d0 net/socket.c:2417
__do_sys_sendmsg net/socket.c:2426 [inline]
__se_sys_sendmsg net/socket.c:2424 [inline]
__x64_sys_sendmsg+0x78/0xb0 net/socket.c:2424
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
stack backtrace:
CPU: 0 PID: 20867 Comm: syz-executor.3 Not tainted 5.5.0-syzkaller #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x197/0x210 lib/dump_stack.c:118
print_bad_irq_dependency kernel/locking/lockdep.c:2094 [inline]
check_irq_usage.cold+0x586/0x6fe kernel/locking/lockdep.c:2292
check_prev_add kernel/locking/lockdep.c:2479 [inline]
check_prevs_add kernel/locking/lockdep.c:2580 [inline]
validate_chain kernel/locking/lockdep.c:2970 [inline]
__lock_acquire+0x25b4/0x4a00 kernel/locking/lockdep.c:3954
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put security/selinux/ss/sidtab.c:594 [inline]
sidtab_sid2str_get+0x3b5/0x66e security/selinux/ss/sidtab.c:594
sidtab_entry_to_string+0x37/0x110 security/selinux/ss/services.c:1271
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
selinux_secid_to_secctx+0x2d/0x40 security/selinux/hooks.c:6451
security_secid_to_secctx+0x75/0xc0 security/security.c:1950
ctnetlink_dump_secctx.isra.0+0x8c/0x330 net/netfilter/nf_conntrack_netlink.c:335
ctnetlink_dump_info net/netfilter/nf_conntrack_netlink.c:531 [inline]
ctnetlink_fill_info+0x750/0xd40 net/netfilter/nf_conntrack_netlink.c:591
ctnetlink_dump_table+0x68c/0x1130 net/netfilter/nf_conntrack_netlink.c:979
netlink_dump+0x558/0xfb0 net/netlink/af_netlink.c:2244
__netlink_dump_start+0x66a/0x930 net/netlink/af_netlink.c:2352
netlink_dump_start include/linux/netlink.h:233 [inline]
ctnetlink_get_conntrack+0x55e/0x6a0 net/netfilter/nf_conntrack_netlink.c:1355
nfnetlink_rcv_msg+0xcf2/0xfb0 net/netfilter/nfnetlink.c:229
netlink_rcv_skb+0x177/0x450 net/netlink/af_netlink.c:2477
nfnetlink_rcv+0x1ba/0x460 net/netfilter/nfnetlink.c:563
netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline]
netlink_unicast+0x58c/0x7d0 net/netlink/af_netlink.c:1328
netlink_sendmsg+0x91c/0xea0 net/netlink/af_netlink.c:1917
sock_sendmsg_nosec net/socket.c:639 [inline]
sock_sendmsg+0xd7/0x130 net/socket.c:659
____sys_sendmsg+0x753/0x880 net/socket.c:2330
___sys_sendmsg+0x100/0x170 net/socket.c:2384
__sys_sendmsg+0x105/0x1d0 net/socket.c:2417
__do_sys_sendmsg net/socket.c:2426 [inline]
__se_sys_sendmsg net/socket.c:2424 [inline]
__x64_sys_sendmsg+0x78/0xb0 net/socket.c:2424
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45b3f9
Code: bd b1 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 8b b1 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fbe326f8c88 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 000000000072bf00 RCX: 000000000045b3f9
RDX: 0000000000000000 RSI: 0000000020000500 RDI: 0000000000000003
RBP: 00007fbe326f96d4 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000000008f1 R14: 00000000004ba9a8 R15: 00000000006eaa80
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot has found a reproducer for the following crash on:
HEAD commit: b0be0eff Merge tag 'x86-pti-2020-01-28' of git://git.kerne..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1432aebee00000
kernel config: https://syzkaller.appspot.com/x/.config?x=9784e57c96a92f20
dashboard link: https://syzkaller.appspot.com/bug?extid=61cba5033e2072d61806
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10088e95e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fa605ee00000
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: [email protected]
=====================================================
WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
5.5.0-syzkaller #0 Not tainted
-----------------------------------------------------
syz-executor305/10624 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
and this task is already holding:
ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
which would create a new lock dependency:
(&(&nf_conntrack_locks[i])->rlock){+.-.} -> (&(&s->cache_lock)->rlock){+.+.}
but this new dependency connects a SOFTIRQ-irq-safe lock:
(&(&nf_conntrack_locks[i])->rlock){+.-.}
... which became SOFTIRQ-irq-safe at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
nf_conntrack_double_lock.isra.0+0xac/0x100 net/netfilter/nf_conntrack_core.c:134
__nf_conntrack_confirm+0x252/0x1710 net/netfilter/nf_conntrack_core.c:963
nf_conntrack_confirm include/net/netfilter/nf_conntrack_core.h:63 [inline]
nf_confirm+0x3e7/0x4d0 net/netfilter/nf_conntrack_proto.c:154
ipv4_confirm+0x14c/0x240 net/netfilter/nf_conntrack_proto.c:169
nf_hook_entry_hookfn include/linux/netfilter.h:135 [inline]
nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:512
nf_hook include/linux/netfilter.h:262 [inline]
NF_HOOK include/linux/netfilter.h:305 [inline]
ip_local_deliver+0x352/0x520 net/ipv4/ip_input.c:252
dst_input include/net/dst.h:442 [inline]
ip_sublist_rcv_finish+0x9b/0x2d0 net/ipv4/ip_input.c:549
ip_list_rcv_finish net/ipv4/ip_input.c:599 [inline]
ip_sublist_rcv+0x5a6/0xa20 net/ipv4/ip_input.c:607
ip_list_rcv+0x37e/0x4bf net/ipv4/ip_input.c:642
__netif_receive_skb_list_ptype net/core/dev.c:5193 [inline]
__netif_receive_skb_list_core+0x5fc/0x9d0 net/core/dev.c:5241
__netif_receive_skb_list net/core/dev.c:5293 [inline]
netif_receive_skb_list_internal+0x7eb/0xe50 net/core/dev.c:5388
gro_normal_list.part.0+0x1e/0xb0 net/core/dev.c:5499
gro_normal_list net/core/dev.c:6232 [inline]
napi_complete_done+0x208/0x510 net/core/dev.c:6220
virtqueue_napi_complete+0x31/0xa0 drivers/net/virtio_net.c:329
virtnet_poll+0xbad/0xe90 drivers/net/virtio_net.c:1432
napi_poll net/core/dev.c:6534 [inline]
net_rx_action+0x508/0x1120 net/core/dev.c:6602
__do_softirq+0x262/0x98c kernel/softirq.c:292
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x19b/0x1e0 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
do_IRQ+0xe3/0x280 arch/x86/kernel/irq.c:263
ret_from_intr+0x0/0x36
native_safe_halt+0xe/0x10 arch/x86/include/asm/irqflags.h:60
arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:690
default_idle_call+0x84/0xb0 kernel/sched/idle.c:94
cpuidle_idle_call kernel/sched/idle.c:154 [inline]
do_idle+0x3c8/0x6e0 kernel/sched/idle.c:269
cpu_startup_entry+0x1b/0x20 kernel/sched/idle.c:361
rest_init+0x23b/0x371 init/main.c:451
arch_call_rest_init+0xe/0x1b
start_kernel+0x8a3/0x8e2 init/main.c:785
x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:490
x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:471
secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:242
to a SOFTIRQ-irq-unsafe lock:
(&(&s->cache_lock)->rlock){+.+.}
... which became SOFTIRQ-irq-unsafe at:
...
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
avc_audit security/selinux/include/avc.h:140 [inline]
avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
security_mmap_file+0xa4/0x1e0 security/security.c:1482
vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
vm_mmap+0x94/0xc0 mm/util.c:525
elf_map+0x10a/0x2b0 fs/binfmt_elf.c:377
load_elf_binary+0xd4b/0x5310 fs/binfmt_elf.c:982
search_binary_handler fs/exec.c:1658 [inline]
search_binary_handler+0x16d/0x570 fs/exec.c:1635
exec_binprm fs/exec.c:1701 [inline]
__do_execve_file.isra.0+0x1329/0x22b0 fs/exec.c:1821
do_execveat_common fs/exec.c:1867 [inline]
do_execve fs/exec.c:1884 [inline]
__do_sys_execve fs/exec.c:1960 [inline]
__se_sys_execve fs/exec.c:1955 [inline]
__x64_sys_execve+0x8f/0xc0 fs/exec.c:1955
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
other info that might help us debug this:
Possible interrupt unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&(&s->cache_lock)->rlock);
local_irq_disable();
lock(&(&nf_conntrack_locks[i])->rlock);
lock(&(&s->cache_lock)->rlock);
<Interrupt>
lock(&(&nf_conntrack_locks[i])->rlock);
*** DEADLOCK ***
4 locks held by syz-executor305/10624:
#0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnl_lock net/netfilter/nfnetlink.c:62 [inline]
#0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnetlink_rcv_msg+0x9ee/0xfb0 net/netfilter/nfnetlink.c:224
#1: ffff8880836415d8 (nlk_cb_mutex-NETFILTER){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
#2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
#2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
#3: ffffffff8b7df008 (&selinux_ss.policy_rwlock){.+.?}, at: security_sid_to_context_core+0x1ca/0x3c0 security/selinux/ss/services.c:1344
the dependencies between SOFTIRQ-irq-safe lock and the holding lock:
-> (&(&nf_conntrack_locks[i])->rlock){+.-.} {
HARDIRQ-ON-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
get_next_corpse net/netfilter/nf_conntrack_core.c:2001 [inline]
nf_ct_iterate_cleanup+0x1d2/0x4e0 net/netfilter/nf_conntrack_core.c:2035
nf_ct_iterate_destroy+0x12e/0x160 net/netfilter/nf_conntrack_core.c:2162
nf_conntrack_helper_unregister+0x111/0x160 net/netfilter/nf_conntrack_helper.c:469
nf_conntrack_helpers_unregister net/netfilter/nf_conntrack_helper.c:532 [inline]
nf_conntrack_helpers_register+0xaa/0xe0 net/netfilter/nf_conntrack_helper.c:523
nf_conntrack_sip_init+0x266/0x2a4 net/netfilter/nf_conntrack_sip.c:1693
do_one_initcall+0x120/0x820 init/main.c:939
do_initcall_level init/main.c:1007 [inline]
do_initcalls init/main.c:1015 [inline]
do_basic_setup init/main.c:1032 [inline]
kernel_init_freeable+0x4ca/0x570 init/main.c:1203
kernel_init+0x12/0x1bf init/main.c:1110
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
IN-SOFTIRQ-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
nf_conntrack_double_lock.isra.0+0xac/0x100 net/netfilter/nf_conntrack_core.c:134
__nf_conntrack_confirm+0x252/0x1710 net/netfilter/nf_conntrack_core.c:963
nf_conntrack_confirm include/net/netfilter/nf_conntrack_core.h:63 [inline]
nf_confirm+0x3e7/0x4d0 net/netfilter/nf_conntrack_proto.c:154
ipv4_confirm+0x14c/0x240 net/netfilter/nf_conntrack_proto.c:169
nf_hook_entry_hookfn include/linux/netfilter.h:135 [inline]
nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:512
nf_hook include/linux/netfilter.h:262 [inline]
NF_HOOK include/linux/netfilter.h:305 [inline]
ip_local_deliver+0x352/0x520 net/ipv4/ip_input.c:252
dst_input include/net/dst.h:442 [inline]
ip_sublist_rcv_finish+0x9b/0x2d0 net/ipv4/ip_input.c:549
ip_list_rcv_finish net/ipv4/ip_input.c:599 [inline]
ip_sublist_rcv+0x5a6/0xa20 net/ipv4/ip_input.c:607
ip_list_rcv+0x37e/0x4bf net/ipv4/ip_input.c:642
__netif_receive_skb_list_ptype net/core/dev.c:5193 [inline]
__netif_receive_skb_list_core+0x5fc/0x9d0 net/core/dev.c:5241
__netif_receive_skb_list net/core/dev.c:5293 [inline]
netif_receive_skb_list_internal+0x7eb/0xe50 net/core/dev.c:5388
gro_normal_list.part.0+0x1e/0xb0 net/core/dev.c:5499
gro_normal_list net/core/dev.c:6232 [inline]
napi_complete_done+0x208/0x510 net/core/dev.c:6220
virtqueue_napi_complete+0x31/0xa0 drivers/net/virtio_net.c:329
virtnet_poll+0xbad/0xe90 drivers/net/virtio_net.c:1432
napi_poll net/core/dev.c:6534 [inline]
net_rx_action+0x508/0x1120 net/core/dev.c:6602
__do_softirq+0x262/0x98c kernel/softirq.c:292
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x19b/0x1e0 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
do_IRQ+0xe3/0x280 arch/x86/kernel/irq.c:263
ret_from_intr+0x0/0x36
native_safe_halt+0xe/0x10 arch/x86/include/asm/irqflags.h:60
arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:690
default_idle_call+0x84/0xb0 kernel/sched/idle.c:94
cpuidle_idle_call kernel/sched/idle.c:154 [inline]
do_idle+0x3c8/0x6e0 kernel/sched/idle.c:269
cpu_startup_entry+0x1b/0x20 kernel/sched/idle.c:361
rest_init+0x23b/0x371 init/main.c:451
arch_call_rest_init+0xe/0x1b
start_kernel+0x8a3/0x8e2 init/main.c:785
x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:490
x86_64_start_kernel+0x77/0x7b arch/x86/kernel/head64.c:471
secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:242
INITIAL USE at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
get_next_corpse net/netfilter/nf_conntrack_core.c:2001 [inline]
nf_ct_iterate_cleanup+0x1d2/0x4e0 net/netfilter/nf_conntrack_core.c:2035
nf_ct_iterate_destroy+0x12e/0x160 net/netfilter/nf_conntrack_core.c:2162
nf_conntrack_helper_unregister+0x111/0x160 net/netfilter/nf_conntrack_helper.c:469
nf_conntrack_helpers_unregister net/netfilter/nf_conntrack_helper.c:532 [inline]
nf_conntrack_helpers_register+0xaa/0xe0 net/netfilter/nf_conntrack_helper.c:523
nf_conntrack_sip_init+0x266/0x2a4 net/netfilter/nf_conntrack_sip.c:1693
do_one_initcall+0x120/0x820 init/main.c:939
do_initcall_level init/main.c:1007 [inline]
do_initcalls init/main.c:1015 [inline]
do_basic_setup init/main.c:1032 [inline]
kernel_init_freeable+0x4ca/0x570 init/main.c:1203
kernel_init+0x12/0x1bf init/main.c:1110
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
}
... key at: [<ffffffff8c1ad440>] __key.79496+0x0/0x40
... acquired at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
selinux_secid_to_secctx+0x2d/0x40 security/selinux/hooks.c:6451
security_secid_to_secctx+0x75/0xc0 security/security.c:1950
ctnetlink_dump_secctx.isra.0+0x8c/0x330 net/netfilter/nf_conntrack_netlink.c:335
ctnetlink_dump_info net/netfilter/nf_conntrack_netlink.c:531 [inline]
ctnetlink_fill_info+0x750/0xd40 net/netfilter/nf_conntrack_netlink.c:591
ctnetlink_dump_table+0x68c/0x1130 net/netfilter/nf_conntrack_netlink.c:979
netlink_dump+0x558/0xfb0 net/netlink/af_netlink.c:2244
__netlink_dump_start+0x66a/0x930 net/netlink/af_netlink.c:2352
netlink_dump_start include/linux/netlink.h:233 [inline]
ctnetlink_get_conntrack+0x55e/0x6a0 net/netfilter/nf_conntrack_netlink.c:1355
nfnetlink_rcv_msg+0xcf2/0xfb0 net/netfilter/nfnetlink.c:229
netlink_rcv_skb+0x177/0x450 net/netlink/af_netlink.c:2477
nfnetlink_rcv+0x1ba/0x460 net/netfilter/nfnetlink.c:563
netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline]
netlink_unicast+0x58c/0x7d0 net/netlink/af_netlink.c:1328
netlink_sendmsg+0x91c/0xea0 net/netlink/af_netlink.c:1917
sock_sendmsg_nosec net/socket.c:639 [inline]
sock_sendmsg+0xd7/0x130 net/socket.c:659
____sys_sendmsg+0x753/0x880 net/socket.c:2330
___sys_sendmsg+0x100/0x170 net/socket.c:2384
__sys_sendmsg+0x105/0x1d0 net/socket.c:2417
__do_sys_sendmsg net/socket.c:2426 [inline]
__se_sys_sendmsg net/socket.c:2424 [inline]
__x64_sys_sendmsg+0x78/0xb0 net/socket.c:2424
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
the dependencies between the lock to be acquired
and SOFTIRQ-irq-unsafe lock:
-> (&(&s->cache_lock)->rlock){+.+.} {
HARDIRQ-ON-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
avc_audit security/selinux/include/avc.h:140 [inline]
avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
security_mmap_file+0xa4/0x1e0 security/security.c:1482
vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
vm_mmap+0x94/0xc0 mm/util.c:525
elf_map+0x10a/0x2b0 fs/binfmt_elf.c:377
load_elf_binary+0xd4b/0x5310 fs/binfmt_elf.c:982
search_binary_handler fs/exec.c:1658 [inline]
search_binary_handler+0x16d/0x570 fs/exec.c:1635
exec_binprm fs/exec.c:1701 [inline]
__do_execve_file.isra.0+0x1329/0x22b0 fs/exec.c:1821
do_execveat_common fs/exec.c:1867 [inline]
do_execve fs/exec.c:1884 [inline]
__do_sys_execve fs/exec.c:1960 [inline]
__se_sys_execve fs/exec.c:1955 [inline]
__x64_sys_execve+0x8f/0xc0 fs/exec.c:1955
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
SOFTIRQ-ON-W at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
avc_audit security/selinux/include/avc.h:140 [inline]
avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
security_mmap_file+0xa4/0x1e0 security/security.c:1482
vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
vm_mmap+0x94/0xc0 mm/util.c:525
elf_map+0x10a/0x2b0 fs/binfmt_elf.c:377
load_elf_binary+0xd4b/0x5310 fs/binfmt_elf.c:982
search_binary_handler fs/exec.c:1658 [inline]
search_binary_handler+0x16d/0x570 fs/exec.c:1635
exec_binprm fs/exec.c:1701 [inline]
__do_execve_file.isra.0+0x1329/0x22b0 fs/exec.c:1821
do_execveat_common fs/exec.c:1867 [inline]
do_execve fs/exec.c:1884 [inline]
__do_sys_execve fs/exec.c:1960 [inline]
__se_sys_execve fs/exec.c:1955 [inline]
__x64_sys_execve+0x8f/0xc0 fs/exec.c:1955
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
INITIAL USE at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
avc_audit security/selinux/include/avc.h:140 [inline]
avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
security_mmap_file+0xa4/0x1e0 security/security.c:1482
vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
vm_mmap+0x94/0xc0 mm/util.c:525
elf_map+0x10a/0x2b0 fs/binfmt_elf.c:377
load_elf_binary+0xd4b/0x5310 fs/binfmt_elf.c:982
search_binary_handler fs/exec.c:1658 [inline]
search_binary_handler+0x16d/0x570 fs/exec.c:1635
exec_binprm fs/exec.c:1701 [inline]
__do_execve_file.isra.0+0x1329/0x22b0 fs/exec.c:1821
do_execveat_common fs/exec.c:1867 [inline]
do_execve fs/exec.c:1884 [inline]
__do_sys_execve fs/exec.c:1960 [inline]
__se_sys_execve fs/exec.c:1955 [inline]
__x64_sys_execve+0x8f/0xc0 fs/exec.c:1955
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
}
... key at: [<ffffffff8b7dec80>] __key.69173+0x0/0x40
... acquired at:
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
selinux_secid_to_secctx+0x2d/0x40 security/selinux/hooks.c:6451
security_secid_to_secctx+0x75/0xc0 security/security.c:1950
ctnetlink_dump_secctx.isra.0+0x8c/0x330 net/netfilter/nf_conntrack_netlink.c:335
ctnetlink_dump_info net/netfilter/nf_conntrack_netlink.c:531 [inline]
ctnetlink_fill_info+0x750/0xd40 net/netfilter/nf_conntrack_netlink.c:591
ctnetlink_dump_table+0x68c/0x1130 net/netfilter/nf_conntrack_netlink.c:979
netlink_dump+0x558/0xfb0 net/netlink/af_netlink.c:2244
__netlink_dump_start+0x66a/0x930 net/netlink/af_netlink.c:2352
netlink_dump_start include/linux/netlink.h:233 [inline]
ctnetlink_get_conntrack+0x55e/0x6a0 net/netfilter/nf_conntrack_netlink.c:1355
nfnetlink_rcv_msg+0xcf2/0xfb0 net/netfilter/nfnetlink.c:229
netlink_rcv_skb+0x177/0x450 net/netlink/af_netlink.c:2477
nfnetlink_rcv+0x1ba/0x460 net/netfilter/nfnetlink.c:563
netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline]
netlink_unicast+0x58c/0x7d0 net/netlink/af_netlink.c:1328
netlink_sendmsg+0x91c/0xea0 net/netlink/af_netlink.c:1917
sock_sendmsg_nosec net/socket.c:639 [inline]
sock_sendmsg+0xd7/0x130 net/socket.c:659
____sys_sendmsg+0x753/0x880 net/socket.c:2330
___sys_sendmsg+0x100/0x170 net/socket.c:2384
__sys_sendmsg+0x105/0x1d0 net/socket.c:2417
__do_sys_sendmsg net/socket.c:2426 [inline]
__se_sys_sendmsg net/socket.c:2424 [inline]
__x64_sys_sendmsg+0x78/0xb0 net/socket.c:2424
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
stack backtrace:
CPU: 1 PID: 10624 Comm: syz-executor305 Not tainted 5.5.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x197/0x210 lib/dump_stack.c:118
print_bad_irq_dependency kernel/locking/lockdep.c:2094 [inline]
check_irq_usage.cold+0x586/0x6fe kernel/locking/lockdep.c:2292
check_prev_add kernel/locking/lockdep.c:2479 [inline]
check_prevs_add kernel/locking/lockdep.c:2580 [inline]
validate_chain kernel/locking/lockdep.c:2970 [inline]
__lock_acquire+0x25b4/0x4a00 kernel/locking/lockdep.c:3954
lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
spin_lock include/linux/spinlock.h:338 [inline]
sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
selinux_secid_to_secctx+0x2d/0x40 security/selinux/hooks.c:6451
security_secid_to_secctx+0x75/0xc0 security/security.c:1950
ctnetlink_dump_secctx.isra.0+0x8c/0x330 net/netfilter/nf_conntrack_netlink.c:335
ctnetlink_dump_info net/netfilter/nf_conntrack_netlink.c:531 [inline]
ctnetlink_fill_info+0x750/0xd40 net/netfilter/nf_conntrack_netlink.c:591
ctnetlink_dump_table+0x68c/0x1130 net/netfilter/nf_conntrack_netlink.c:979
netlink_dump+0x558/0xfb0 net/netlink/af_netlink.c:2244
__netlink_dump_start+0x66a/0x930 net/netlink/af_netlink.c:2352
netlink_dump_start include/linux/netlink.h:233 [inline]
ctnetlink_get_conntrack+0x55e/0x6a0 net/netfilter/nf_conntrack_netlink.c:1355
nfnetlink_rcv_msg+0xcf2/0xfb0 net/netfilter/nfnetlink.c:229
netlink_rcv_skb+0x177/0x450 net/netlink/af_netlink.c:2477
nfnetlink_rcv+0x1ba/0x460 net/netfilter/nfnetlink.c:563
netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline]
netlink_unicast+0x58c/0x7d0 net/netlink/af_netlink.c:1328
netlink_sendmsg+0x91c/0xea0 net/netlink/af_netlink.c:1917
sock_sendmsg_nosec net/socket.c:639 [inline]
sock_sendmsg+0xd7/0x130 net/socket.c:659
____sys_sendmsg+0x753/0x880 net/socket.c:2330
___sys_sendmsg+0x100/0x170 net/socket.c:2384
__sys_sendmsg+0x105/0x1d0 net/socket.c:2417
__do_sys_sendmsg net/socket.c:2426 [inline]
__se_sys_sendmsg net/socket.c:2424 [inline]
__x64_sys_sendmsg+0x78/0xb0 net/socket.c:2424
do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4401e9
Code: 18 89 d0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 fb 13 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffe6ca42208 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 00000000004002c8 RCX: 00000000004401e9
RDX: 0000000000000000 RSI: 0000000020d65000 RDI: 0000000000000003
RBP: 00000000006ca018 R08: 0000000000000000 R09: 0000000
On Tue, Jan 28, 2020 at 7:50 AM syzbot
<[email protected]> wrote:
>
> syzbot has found a reproducer for the following crash on:
>
> HEAD commit: b0be0eff Merge tag 'x86-pti-2020-01-28' of git://git.kerne..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=1432aebee00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=9784e57c96a92f20
> dashboard link: https://syzkaller.appspot.com/bug?extid=61cba5033e2072d61806
> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10088e95e00000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fa605ee00000
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: [email protected]
>
> =====================================================
> WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
> 5.5.0-syzkaller #0 Not tainted
> -----------------------------------------------------
> syz-executor305/10624 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
>
> and this task is already holding:
> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> which would create a new lock dependency:
> (&(&nf_conntrack_locks[i])->rlock){+.-.} -> (&(&s->cache_lock)->rlock){+.+.}
>
> but this new dependency connects a SOFTIRQ-irq-safe lock:
> (&(&nf_conntrack_locks[i])->rlock){+.-.}
>
> ... which became SOFTIRQ-irq-safe at:
> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:338 [inline]
> nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
...
> to a SOFTIRQ-irq-unsafe lock:
> (&(&s->cache_lock)->rlock){+.+.}
>
> ... which became SOFTIRQ-irq-unsafe at:
> ...
> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:338 [inline]
> sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
> sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
> sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
> sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
> security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
> security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
> avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
> common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
> slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
> avc_audit security/selinux/include/avc.h:140 [inline]
> avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
> inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
> selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
> security_mmap_file+0xa4/0x1e0 security/security.c:1482
> vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
...
> other info that might help us debug this:
>
> Possible interrupt unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(&(&s->cache_lock)->rlock);
> local_irq_disable();
> lock(&(&nf_conntrack_locks[i])->rlock);
> lock(&(&s->cache_lock)->rlock);
> <Interrupt>
> lock(&(&nf_conntrack_locks[i])->rlock);
>
> *** DEADLOCK ***
>
> 4 locks held by syz-executor305/10624:
> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnl_lock net/netfilter/nfnetlink.c:62 [inline]
> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnetlink_rcv_msg+0x9ee/0xfb0 net/netfilter/nfnetlink.c:224
> #1: ffff8880836415d8 (nlk_cb_mutex-NETFILTER){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> #3: ffffffff8b7df008 (&selinux_ss.policy_rwlock){.+.?}, at: security_sid_to_context_core+0x1ca/0x3c0 security/selinux/ss/services.c:1344
I think this is going to be tricky to fix due to the differing
contexts from which sidtab_sid2str_put() may be called. We already
have a check for !in_task() in sidtab_sid2str_put(), do we want to add
a check for !in_serving_softirq() too?
--
paul moore
http://www.paul-moore.com
On 1/28/20 8:39 AM, Paul Moore wrote:
> On Tue, Jan 28, 2020 at 7:50 AM syzbot
> <[email protected]> wrote:
>>
>> syzbot has found a reproducer for the following crash on:
>>
>> HEAD commit: b0be0eff Merge tag 'x86-pti-2020-01-28' of git://git.kerne..
>> git tree: upstream
>> console output: https://syzkaller.appspot.com/x/log.txt?x=1432aebee00000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=9784e57c96a92f20
>> dashboard link: https://syzkaller.appspot.com/bug?extid=61cba5033e2072d61806
>> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10088e95e00000
>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fa605ee00000
>>
>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> Reported-by: [email protected]
>>
>> =====================================================
>> WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
>> 5.5.0-syzkaller #0 Not tainted
>> -----------------------------------------------------
>> syz-executor305/10624 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
>> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
>> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
>>
>> and this task is already holding:
>> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
>> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
>> which would create a new lock dependency:
>> (&(&nf_conntrack_locks[i])->rlock){+.-.} -> (&(&s->cache_lock)->rlock){+.+.}
>>
>> but this new dependency connects a SOFTIRQ-irq-safe lock:
>> (&(&nf_conntrack_locks[i])->rlock){+.-.}
>>
>> ... which became SOFTIRQ-irq-safe at:
>> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
>> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
>> spin_lock include/linux/spinlock.h:338 [inline]
>> nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
>
> ...
>
>> to a SOFTIRQ-irq-unsafe lock:
>> (&(&s->cache_lock)->rlock){+.+.}
>>
>> ... which became SOFTIRQ-irq-unsafe at:
>> ...
>> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
>> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
>> spin_lock include/linux/spinlock.h:338 [inline]
>> sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
>> sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
>> sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
>> sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
>> security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
>> security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
>> avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
>> common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
>> slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
>> avc_audit security/selinux/include/avc.h:140 [inline]
>> avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
>> inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
>> selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
>> security_mmap_file+0xa4/0x1e0 security/security.c:1482
>> vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
>
> ...
>
>> other info that might help us debug this:
>>
>> Possible interrupt unsafe locking scenario:
>>
>> CPU0 CPU1
>> ---- ----
>> lock(&(&s->cache_lock)->rlock);
>> local_irq_disable();
>> lock(&(&nf_conntrack_locks[i])->rlock);
>> lock(&(&s->cache_lock)->rlock);
>> <Interrupt>
>> lock(&(&nf_conntrack_locks[i])->rlock);
>>
>> *** DEADLOCK ***
>>
>> 4 locks held by syz-executor305/10624:
>> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnl_lock net/netfilter/nfnetlink.c:62 [inline]
>> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnetlink_rcv_msg+0x9ee/0xfb0 net/netfilter/nfnetlink.c:224
>> #1: ffff8880836415d8 (nlk_cb_mutex-NETFILTER){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
>> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
>> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
>> #3: ffffffff8b7df008 (&selinux_ss.policy_rwlock){.+.?}, at: security_sid_to_context_core+0x1ca/0x3c0 security/selinux/ss/services.c:1344
>
> I think this is going to be tricky to fix due to the differing
> contexts from which sidtab_sid2str_put() may be called. We already
> have a check for !in_task() in sidtab_sid2str_put(), do we want to add
> a check for !in_serving_softirq() too?
No, we should just use spin_lock_irqsave/unlock_irqrestore() IMHO, but
that then means we need to re-evaluate the performance gain of this change.
On Tue, Jan 28, 2020 at 2:39 PM Paul Moore <[email protected]> wrote:
> On Tue, Jan 28, 2020 at 7:50 AM syzbot
> <[email protected]> wrote:
> >
> > syzbot has found a reproducer for the following crash on:
> >
> > HEAD commit: b0be0eff Merge tag 'x86-pti-2020-01-28' of git://git.kerne..
> > git tree: upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=1432aebee00000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=9784e57c96a92f20
> > dashboard link: https://syzkaller.appspot.com/bug?extid=61cba5033e2072d61806
> > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10088e95e00000
> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fa605ee00000
> >
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: [email protected]
> >
> > =====================================================
> > WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
> > 5.5.0-syzkaller #0 Not tainted
> > -----------------------------------------------------
> > syz-executor305/10624 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
> > ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> > ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
> >
> > and this task is already holding:
> > ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> > ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> > which would create a new lock dependency:
> > (&(&nf_conntrack_locks[i])->rlock){+.-.} -> (&(&s->cache_lock)->rlock){+.+.}
> >
> > but this new dependency connects a SOFTIRQ-irq-safe lock:
> > (&(&nf_conntrack_locks[i])->rlock){+.-.}
> >
> > ... which became SOFTIRQ-irq-safe at:
> > lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> > __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> > _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> > spin_lock include/linux/spinlock.h:338 [inline]
> > nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
>
> ...
>
> > to a SOFTIRQ-irq-unsafe lock:
> > (&(&s->cache_lock)->rlock){+.+.}
> >
> > ... which became SOFTIRQ-irq-unsafe at:
> > ...
> > lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> > __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> > _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> > spin_lock include/linux/spinlock.h:338 [inline]
> > sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
> > sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
> > sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
> > sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
> > security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
> > security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
> > avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
> > common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
> > slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
> > avc_audit security/selinux/include/avc.h:140 [inline]
> > avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
> > inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
> > selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
> > security_mmap_file+0xa4/0x1e0 security/security.c:1482
> > vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
>
> ...
>
> > other info that might help us debug this:
> >
> > Possible interrupt unsafe locking scenario:
> >
> > CPU0 CPU1
> > ---- ----
> > lock(&(&s->cache_lock)->rlock);
> > local_irq_disable();
> > lock(&(&nf_conntrack_locks[i])->rlock);
> > lock(&(&s->cache_lock)->rlock);
> > <Interrupt>
> > lock(&(&nf_conntrack_locks[i])->rlock);
> >
> > *** DEADLOCK ***
> >
> > 4 locks held by syz-executor305/10624:
> > #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnl_lock net/netfilter/nfnetlink.c:62 [inline]
> > #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnetlink_rcv_msg+0x9ee/0xfb0 net/netfilter/nfnetlink.c:224
> > #1: ffff8880836415d8 (nlk_cb_mutex-NETFILTER){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
> > #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> > #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> > #3: ffffffff8b7df008 (&selinux_ss.policy_rwlock){.+.?}, at: security_sid_to_context_core+0x1ca/0x3c0 security/selinux/ss/services.c:1344
>
> I think this is going to be tricky to fix due to the differing
> contexts from which sidtab_sid2str_put() may be called. We already
> have a check for !in_task() in sidtab_sid2str_put(), do we want to add
> a check for !in_serving_softirq() too?
I don't think that is going to fix it. The issue here is that an
interrupt may happen while we are holding the lock and in that
interrupt we try to take some other lock, which another thread has
taken before the cache lock... I'm afraid using the irqsave/irqrestore
locking is the only possible fix here... (Either that or giving up on
the proper least-recently-used logic.)
--
Ondrej Mosnacek <omosnace at redhat dot com>
Software Engineer, Security Technologies
Red Hat, Inc.
On Tue, Jan 28, 2020 at 2:44 PM Stephen Smalley <[email protected]> wrote:
> On 1/28/20 8:39 AM, Paul Moore wrote:
> > On Tue, Jan 28, 2020 at 7:50 AM syzbot
> > <[email protected]> wrote:
> >>
> >> syzbot has found a reproducer for the following crash on:
> >>
> >> HEAD commit: b0be0eff Merge tag 'x86-pti-2020-01-28' of git://git.kerne..
> >> git tree: upstream
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=1432aebee00000
> >> kernel config: https://syzkaller.appspot.com/x/.config?x=9784e57c96a92f20
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=61cba5033e2072d61806
> >> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> >> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10088e95e00000
> >> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fa605ee00000
> >>
> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> >> Reported-by: [email protected]
> >>
> >> =====================================================
> >> WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
> >> 5.5.0-syzkaller #0 Not tainted
> >> -----------------------------------------------------
> >> syz-executor305/10624 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
> >> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> >> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
> >>
> >> and this task is already holding:
> >> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> >> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> >> which would create a new lock dependency:
> >> (&(&nf_conntrack_locks[i])->rlock){+.-.} -> (&(&s->cache_lock)->rlock){+.+.}
> >>
> >> but this new dependency connects a SOFTIRQ-irq-safe lock:
> >> (&(&nf_conntrack_locks[i])->rlock){+.-.}
> >>
> >> ... which became SOFTIRQ-irq-safe at:
> >> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> >> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> >> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> >> spin_lock include/linux/spinlock.h:338 [inline]
> >> nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> >
> > ...
> >
> >> to a SOFTIRQ-irq-unsafe lock:
> >> (&(&s->cache_lock)->rlock){+.+.}
> >>
> >> ... which became SOFTIRQ-irq-unsafe at:
> >> ...
> >> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> >> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> >> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> >> spin_lock include/linux/spinlock.h:338 [inline]
> >> sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
> >> sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
> >> sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
> >> sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
> >> security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
> >> security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
> >> avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
> >> common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
> >> slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
> >> avc_audit security/selinux/include/avc.h:140 [inline]
> >> avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
> >> inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
> >> selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
> >> security_mmap_file+0xa4/0x1e0 security/security.c:1482
> >> vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
> >
> > ...
> >
> >> other info that might help us debug this:
> >>
> >> Possible interrupt unsafe locking scenario:
> >>
> >> CPU0 CPU1
> >> ---- ----
> >> lock(&(&s->cache_lock)->rlock);
> >> local_irq_disable();
> >> lock(&(&nf_conntrack_locks[i])->rlock);
> >> lock(&(&s->cache_lock)->rlock);
> >> <Interrupt>
> >> lock(&(&nf_conntrack_locks[i])->rlock);
> >>
> >> *** DEADLOCK ***
> >>
> >> 4 locks held by syz-executor305/10624:
> >> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnl_lock net/netfilter/nfnetlink.c:62 [inline]
> >> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnetlink_rcv_msg+0x9ee/0xfb0 net/netfilter/nfnetlink.c:224
> >> #1: ffff8880836415d8 (nlk_cb_mutex-NETFILTER){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
> >> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> >> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> >> #3: ffffffff8b7df008 (&selinux_ss.policy_rwlock){.+.?}, at: security_sid_to_context_core+0x1ca/0x3c0 security/selinux/ss/services.c:1344
> >
> > I think this is going to be tricky to fix due to the differing
> > contexts from which sidtab_sid2str_put() may be called. We already
> > have a check for !in_task() in sidtab_sid2str_put(), do we want to add
> > a check for !in_serving_softirq() too?
>
> No, we should just use spin_lock_irqsave/unlock_irqrestore() IMHO, but
> that then means we need to re-evaluate the performance gain of this change.
I just tested a patch that switches to the IRQ-disabling locking under
the systemd-journald scenario from [1] and the impact seems not that
bad - the security_secid_to_secctx() function only takes up 3.18% of
total run time vs. ~2% I reported in the original patch
(d97bd23c2d7d). I'll post the patch shortly.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1733259, i.e.:
cat /dev/urandom | base64 | logger &
timeout 30s perf record -p $(pidof systemd-journald) -a -g
--
Ondrej Mosnacek <omosnace at redhat dot com>
Software Engineer, Security Technologies
Red Hat, Inc.
syzbot has bisected this bug to:
commit d97bd23c2d7d866e99eb3a927c742715c85a90ef
Author: Ondrej Mosnacek <[email protected]>
Date: Tue Nov 26 13:57:00 2019 +0000
selinux: cache the SID -> context string translation
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=13fe5f69e00000
start commit: b0be0eff Merge tag 'x86-pti-2020-01-28' of git://git.kerne..
git tree: upstream
final crash: https://syzkaller.appspot.com/x/report.txt?x=10015f69e00000
console output: https://syzkaller.appspot.com/x/log.txt?x=17fe5f69e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=9784e57c96a92f20
dashboard link: https://syzkaller.appspot.com/bug?extid=61cba5033e2072d61806
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10088e95e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fa605ee00000
Reported-by: [email protected]
Fixes: d97bd23c2d7d ("selinux: cache the SID -> context string translation")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
On Tue, Jan 28, 2020 at 9:27 AM Ondrej Mosnacek <[email protected]> wrote:
> On Tue, Jan 28, 2020 at 2:44 PM Stephen Smalley <[email protected]> wrote:
> > On 1/28/20 8:39 AM, Paul Moore wrote:
> > > On Tue, Jan 28, 2020 at 7:50 AM syzbot
> > > <[email protected]> wrote:
> > >>
> > >> syzbot has found a reproducer for the following crash on:
> > >>
> > >> HEAD commit: b0be0eff Merge tag 'x86-pti-2020-01-28' of git://git.kerne..
> > >> git tree: upstream
> > >> console output: https://syzkaller.appspot.com/x/log.txt?x=1432aebee00000
> > >> kernel config: https://syzkaller.appspot.com/x/.config?x=9784e57c96a92f20
> > >> dashboard link: https://syzkaller.appspot.com/bug?extid=61cba5033e2072d61806
> > >> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > >> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10088e95e00000
> > >> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fa605ee00000
> > >>
> > >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > >> Reported-by: [email protected]
> > >>
> > >> =====================================================
> > >> WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
> > >> 5.5.0-syzkaller #0 Not tainted
> > >> -----------------------------------------------------
> > >> syz-executor305/10624 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
> > >> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> > >> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
> > >>
> > >> and this task is already holding:
> > >> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> > >> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> > >> which would create a new lock dependency:
> > >> (&(&nf_conntrack_locks[i])->rlock){+.-.} -> (&(&s->cache_lock)->rlock){+.+.}
> > >>
> > >> but this new dependency connects a SOFTIRQ-irq-safe lock:
> > >> (&(&nf_conntrack_locks[i])->rlock){+.-.}
> > >>
> > >> ... which became SOFTIRQ-irq-safe at:
> > >> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> > >> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> > >> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> > >> spin_lock include/linux/spinlock.h:338 [inline]
> > >> nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> > >
> > > ...
> > >
> > >> to a SOFTIRQ-irq-unsafe lock:
> > >> (&(&s->cache_lock)->rlock){+.+.}
> > >>
> > >> ... which became SOFTIRQ-irq-unsafe at:
> > >> ...
> > >> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> > >> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> > >> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> > >> spin_lock include/linux/spinlock.h:338 [inline]
> > >> sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
> > >> sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
> > >> sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
> > >> sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
> > >> security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
> > >> security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
> > >> avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
> > >> common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
> > >> slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
> > >> avc_audit security/selinux/include/avc.h:140 [inline]
> > >> avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
> > >> inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
> > >> selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
> > >> security_mmap_file+0xa4/0x1e0 security/security.c:1482
> > >> vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
> > >
> > > ...
> > >
> > >> other info that might help us debug this:
> > >>
> > >> Possible interrupt unsafe locking scenario:
> > >>
> > >> CPU0 CPU1
> > >> ---- ----
> > >> lock(&(&s->cache_lock)->rlock);
> > >> local_irq_disable();
> > >> lock(&(&nf_conntrack_locks[i])->rlock);
> > >> lock(&(&s->cache_lock)->rlock);
> > >> <Interrupt>
> > >> lock(&(&nf_conntrack_locks[i])->rlock);
> > >>
> > >> *** DEADLOCK ***
> > >>
> > >> 4 locks held by syz-executor305/10624:
> > >> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnl_lock net/netfilter/nfnetlink.c:62 [inline]
> > >> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnetlink_rcv_msg+0x9ee/0xfb0 net/netfilter/nfnetlink.c:224
> > >> #1: ffff8880836415d8 (nlk_cb_mutex-NETFILTER){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
> > >> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> > >> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> > >> #3: ffffffff8b7df008 (&selinux_ss.policy_rwlock){.+.?}, at: security_sid_to_context_core+0x1ca/0x3c0 security/selinux/ss/services.c:1344
> > >
> > > I think this is going to be tricky to fix due to the differing
> > > contexts from which sidtab_sid2str_put() may be called. We already
> > > have a check for !in_task() in sidtab_sid2str_put(), do we want to add
> > > a check for !in_serving_softirq() too?
> >
> > No, we should just use spin_lock_irqsave/unlock_irqrestore() IMHO, but
> > that then means we need to re-evaluate the performance gain of this change.
>
> I just tested a patch that switches to the IRQ-disabling locking under
> the systemd-journald scenario from [1] and the impact seems not that
> bad - the security_secid_to_secctx() function only takes up 3.18% of
> total run time vs. ~2% I reported in the original patch
> (d97bd23c2d7d). I'll post the patch shortly.
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1733259, i.e.:
> cat /dev/urandom | base64 | logger &
> timeout 30s perf record -p $(pidof systemd-journald) -a -g
I'm skeptical of the real world impact of disabling IRQs here,
although I'm not sure off the top of my head how we could manage a
cache (I'm thinking more about adding to, and evicting entries, from
the cache) without some form of locking. Perhaps we look into per-cpu
caching?
--
paul moore
http://www.paul-moore.com
On Tue, Jan 28, 2020 at 4:45 PM Paul Moore <[email protected]> wrote:
> On Tue, Jan 28, 2020 at 9:27 AM Ondrej Mosnacek <[email protected]> wrote:
> > On Tue, Jan 28, 2020 at 2:44 PM Stephen Smalley <[email protected]> wrote:
> > > On 1/28/20 8:39 AM, Paul Moore wrote:
> > > > On Tue, Jan 28, 2020 at 7:50 AM syzbot
> > > > <[email protected]> wrote:
> > > >>
> > > >> syzbot has found a reproducer for the following crash on:
> > > >>
> > > >> HEAD commit: b0be0eff Merge tag 'x86-pti-2020-01-28' of git://git.kerne..
> > > >> git tree: upstream
> > > >> console output: https://syzkaller.appspot.com/x/log.txt?x=1432aebee00000
> > > >> kernel config: https://syzkaller.appspot.com/x/.config?x=9784e57c96a92f20
> > > >> dashboard link: https://syzkaller.appspot.com/bug?extid=61cba5033e2072d61806
> > > >> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > > >> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10088e95e00000
> > > >> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fa605ee00000
> > > >>
> > > >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > >> Reported-by: [email protected]
> > > >>
> > > >> =====================================================
> > > >> WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
> > > >> 5.5.0-syzkaller #0 Not tainted
> > > >> -----------------------------------------------------
> > > >> syz-executor305/10624 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
> > > >> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> > > >> ffff888098c14098 (&(&s->cache_lock)->rlock){+.+.}, at: sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
> > > >>
> > > >> and this task is already holding:
> > > >> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> > > >> ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> > > >> which would create a new lock dependency:
> > > >> (&(&nf_conntrack_locks[i])->rlock){+.-.} -> (&(&s->cache_lock)->rlock){+.+.}
> > > >>
> > > >> but this new dependency connects a SOFTIRQ-irq-safe lock:
> > > >> (&(&nf_conntrack_locks[i])->rlock){+.-.}
> > > >>
> > > >> ... which became SOFTIRQ-irq-safe at:
> > > >> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> > > >> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> > > >> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> > > >> spin_lock include/linux/spinlock.h:338 [inline]
> > > >> nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> > > >
> > > > ...
> > > >
> > > >> to a SOFTIRQ-irq-unsafe lock:
> > > >> (&(&s->cache_lock)->rlock){+.+.}
> > > >>
> > > >> ... which became SOFTIRQ-irq-unsafe at:
> > > >> ...
> > > >> lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4484
> > > >> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> > > >> _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
> > > >> spin_lock include/linux/spinlock.h:338 [inline]
> > > >> sidtab_sid2str_put.part.0+0x36/0x880 security/selinux/ss/sidtab.c:533
> > > >> sidtab_sid2str_put+0xa0/0xc0 security/selinux/ss/sidtab.c:566
> > > >> sidtab_entry_to_string security/selinux/ss/services.c:1279 [inline]
> > > >> sidtab_entry_to_string+0xf2/0x110 security/selinux/ss/services.c:1266
> > > >> security_sid_to_context_core+0x2c6/0x3c0 security/selinux/ss/services.c:1361
> > > >> security_sid_to_context+0x34/0x40 security/selinux/ss/services.c:1384
> > > >> avc_audit_post_callback+0x102/0x790 security/selinux/avc.c:709
> > > >> common_lsm_audit+0x5ac/0x1e00 security/lsm_audit.c:466
> > > >> slow_avc_audit+0x16a/0x1f0 security/selinux/avc.c:782
> > > >> avc_audit security/selinux/include/avc.h:140 [inline]
> > > >> avc_has_perm+0x543/0x610 security/selinux/avc.c:1185
> > > >> inode_has_perm+0x1a8/0x230 security/selinux/hooks.c:1631
> > > >> selinux_mmap_file+0x10a/0x1d0 security/selinux/hooks.c:3701
> > > >> security_mmap_file+0xa4/0x1e0 security/security.c:1482
> > > >> vm_mmap_pgoff+0xf0/0x230 mm/util.c:502
> > > >
> > > > ...
> > > >
> > > >> other info that might help us debug this:
> > > >>
> > > >> Possible interrupt unsafe locking scenario:
> > > >>
> > > >> CPU0 CPU1
> > > >> ---- ----
> > > >> lock(&(&s->cache_lock)->rlock);
> > > >> local_irq_disable();
> > > >> lock(&(&nf_conntrack_locks[i])->rlock);
> > > >> lock(&(&s->cache_lock)->rlock);
> > > >> <Interrupt>
> > > >> lock(&(&nf_conntrack_locks[i])->rlock);
> > > >>
> > > >> *** DEADLOCK ***
> > > >>
> > > >> 4 locks held by syz-executor305/10624:
> > > >> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnl_lock net/netfilter/nfnetlink.c:62 [inline]
> > > >> #0: ffffffff8c1acc68 (&table[i].mutex){+.+.}, at: nfnetlink_rcv_msg+0x9ee/0xfb0 net/netfilter/nfnetlink.c:224
> > > >> #1: ffff8880836415d8 (nlk_cb_mutex-NETFILTER){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
> > > >> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
> > > >> #2: ffffffff89865770 (&(&nf_conntrack_locks[i])->rlock){+.-.}, at: nf_conntrack_lock+0x17/0x70 net/netfilter/nf_conntrack_core.c:91
> > > >> #3: ffffffff8b7df008 (&selinux_ss.policy_rwlock){.+.?}, at: security_sid_to_context_core+0x1ca/0x3c0 security/selinux/ss/services.c:1344
> > > >
> > > > I think this is going to be tricky to fix due to the differing
> > > > contexts from which sidtab_sid2str_put() may be called. We already
> > > > have a check for !in_task() in sidtab_sid2str_put(), do we want to add
> > > > a check for !in_serving_softirq() too?
> > >
> > > No, we should just use spin_lock_irqsave/unlock_irqrestore() IMHO, but
> > > that then means we need to re-evaluate the performance gain of this change.
> >
> > I just tested a patch that switches to the IRQ-disabling locking under
> > the systemd-journald scenario from [1] and the impact seems not that
> > bad - the security_secid_to_secctx() function only takes up 3.18% of
> > total run time vs. ~2% I reported in the original patch
> > (d97bd23c2d7d). I'll post the patch shortly.
> >
> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1733259, i.e.:
> > cat /dev/urandom | base64 | logger &
> > timeout 30s perf record -p $(pidof systemd-journald) -a -g
>
> I'm skeptical of the real world impact of disabling IRQs here,
> although I'm not sure off the top of my head how we could manage a
> cache (I'm thinking more about adding to, and evicting entries, from
> the cache) without some form of locking. Perhaps we look into per-cpu
> caching?
(I hope the ML won't mind an SVG attachment...) This is a flamegraph
of the scenario above, with the sipn_[un]lock_irq{save|restore}()
patch applied. You can see that the overhead of locking is not so bad
- the kmemdup (which is unavoidable here) still takes up most of the
computation time. We can try to find a more clever cache
implementation later, but this solution should be still much better
than (in case of dense MCS contexts) to comparable to (in case of no
MCS categories) no caching.
BTW I'm trying to do some final testing of the patch on a multi-CPU
system, but the current rawhide 5.5.0-1 kernel seems to have some bug
that prevents it from booting on anything with more than one core.
I'll see if I can work around it somehow...
--
Ondrej Mosnacek <omosnace at redhat dot com>
Software Engineer, Security Technologies
Red Hat, Inc.
On Tue, Jan 28, 2020 at 11:31 AM Ondrej Mosnacek <[email protected]> wrote:
> ... the current rawhide 5.5.0-1 kernel seems to have some bug
> that prevents it from booting on anything with more than one core.
> I'll see if I can work around it somehow...
I'm not sure how you are building your kernels, but gcc v10 is causing
a lot of problems, I would suggest compiling with an earlier gcc for
the near future until things get sorted (I'm doing the kernel-secnext
builds on stable Fedora, not Rawhide, for now).
--
paul moore
http://www.paul-moore.com
On Tue, Jan 28, 2020 at 6:29 PM Paul Moore <[email protected]> wrote:
> On Tue, Jan 28, 2020 at 11:31 AM Ondrej Mosnacek <[email protected]> wrote:
> > ... the current rawhide 5.5.0-1 kernel seems to have some bug
> > that prevents it from booting on anything with more than one core.
> > I'll see if I can work around it somehow...
>
> I'm not sure how you are building your kernels, but gcc v10 is causing
> a lot of problems, I would suggest compiling with an earlier gcc for
> the near future until things get sorted (I'm doing the kernel-secnext
> builds on stable Fedora, not Rawhide, for now).
Right, thanks, I was using the Rawhide buildroot to build the test
kernel (derived from Rawhide dist-git source + selinux-next + the
patch). Fortunately the Rawhide kernel can be also built against the
f31 target without any additional hacks, so I managed to build a an
upstream-based kernel with GCC 9 and it didn't have the
crash-on-multi-core issue. Regardless, I wasn't able to reproduce the
syzbot crash locally, so I had to ask syzbot to test the patch from my
git tree [1] and it passed.
Nonetheless, I checked to see how the sidtab string cache + IRQ-safe
locking (assuming mostly cache hits) compares to the non-cache
situation with a category-free label
(unconfined_u:unconfined_r:unconfined_t:s0) and the cache (3.2% impact
when mostly hits) is still faster than the non-cache version (5.5%
impact best case, 65% impact worst case).
I intend to incorporate all this information into the log message and
then post the patch.
[1] https://groups.google.com/d/msg/syzkaller-bugs/1UwATFnIiW8/kOpRrjyNAAAJ
--
Ondrej Mosnacek <omosnace at redhat dot com>
Software Engineer, Security Technologies
Red Hat, Inc.