在 2022/5/27 17:57, David Howells 写道:
> Hi Zhu, Bob, Steve,
>
> There seems to be a locking bug in the softRoCE driver when mounting a cifs
> share. See attached trace. I'm guessing the problem is that a softirq
> handler is accessing the xarray, but other accesses to the xarray aren't
> guarded by _bh or _irq markers on the lock primitives.
>
> I wonder if rxe_pool_get_index() should just rely on the RCU read lock and not
> take the spinlock.
>
> Alternatively, __rxe_add_to_pool() should be using xa_alloc_cyclic_bh() or
> xa_alloc_cyclic_irq().
>
> I used the following commands:
>
> rdma link add rxe0 type rxe netdev enp6s0 # andromeda, softRoCE
> mount //192.168.6.1/scratch /xfstest.scratch -o user=shares,rdma,pass=...
>
> talking to ksmbd on the other side.
It seems a known bug. Please make tests with the patches in the
attachment. If not work, please let me know.
Thanks a lot.
Zhu Yanjun
>
> Kernel is v5.18-rc6.
>
> David
> ---
> infiniband rxe0: set active
> infiniband rxe0: added enp6s0
> RDS/IB: rxe0: added
> CIFS: No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3.1.1), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3.1.1 (or even SMB3 or SMB2.1) specify vers=1.0 on mount.
> CIFS: Attempting to mount \\192.168.6.1\scratch
>
> ================================
> WARNING: inconsistent lock state
> 5.18.0-rc6-build2+ #465 Not tainted
> --------------------------------
> inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage.
> ksoftirqd/1/20 [HC0[0]:SC1[1]:HE0:SE0] takes:
> ffff888134d11310 (&xa->xa_lock#12){+.?.}-{2:2}, at: rxe_pool_get_index+0x19/0x69
> {SOFTIRQ-ON-W} state was registered at:
> mark_usage+0x169/0x17b
> __lock_acquire+0x50c/0x96a
> lock_acquire+0x2f4/0x37b
> _raw_spin_lock+0x2f/0x39
> xa_alloc_cyclic.constprop.0+0x20/0x55
> __rxe_add_to_pool+0xe3/0xf2
> __ib_alloc_pd+0xa2/0x26b
> ib_mad_port_open+0x1ac/0x4a1
> ib_mad_init_device+0x9b/0x1b9
> add_client_context+0x133/0x1b3
> enable_device_and_get+0x129/0x248
> ib_register_device+0x256/0x2fd
> rxe_register_device+0x18e/0x1b7
> rxe_net_add+0x57/0x71
> rxe_newlink+0x71/0x8e
> nldev_newlink+0x200/0x26a
> rdma_nl_rcv_msg+0x260/0x2ab
> rdma_nl_rcv+0x108/0x1a7
> netlink_unicast+0x1fc/0x2b3
> netlink_sendmsg+0x4ce/0x51b
> sock_sendmsg_nosec+0x41/0x4f
> __sys_sendto+0x157/0x1cc
> __x64_sys_sendto+0x76/0x82
> do_syscall_64+0x39/0x46
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> irq event stamp: 194111
> hardirqs last enabled at (194110): [<ffffffff81094eb2>] __local_bh_enable_ip+0xb8/0xcc
> hardirqs last disabled at (194111): [<ffffffff82040077>] _raw_spin_lock_irqsave+0x1b/0x51
> softirqs last enabled at (194100): [<ffffffff8240043a>] __do_softirq+0x43a/0x489
> softirqs last disabled at (194105): [<ffffffff81094d30>] run_ksoftirqd+0x31/0x56
>
> other info that might help us debug this:
> Possible unsafe locking scenario:
>
> CPU0
> ----
> lock(&xa->xa_lock#12);
> <Interrupt>
> lock(&xa->xa_lock#12);
>
> *** DEADLOCK ***
>
> no locks held by ksoftirqd/1/20.
>
> stack backtrace:
> CPU: 1 PID: 20 Comm: ksoftirqd/1 Not tainted 5.18.0-rc6-build2+ #465
> Hardware name: ASUS All Series/H97-PLUS, BIOS 2306 10/09/2014
> Call Trace:
> <TASK>
> dump_stack_lvl+0x45/0x59
> valid_state+0x56/0x61
> mark_lock_irq+0x9b/0x2ec
> ? ret_from_fork+0x1f/0x30
> ? valid_state+0x61/0x61
> ? stack_trace_save+0x8f/0xbe
> ? filter_irq_stacks+0x58/0x58
> ? jhash.constprop.0+0x1ad/0x202
> ? save_trace+0x17c/0x196
> mark_lock.part.0+0x10c/0x164
> mark_usage+0xe6/0x17b
> __lock_acquire+0x50c/0x96a
> lock_acquire+0x2f4/0x37b
> ? rxe_pool_get_index+0x19/0x69
> ? rcu_read_unlock+0x52/0x52
> ? jhash.constprop.0+0x1ad/0x202
> ? lockdep_unlock+0xde/0xe6
> ? validate_chain+0x44a/0x4a8
> ? req_next_wqe+0x312/0x363
> _raw_spin_lock_irqsave+0x41/0x51
> ? rxe_pool_get_index+0x19/0x69
> rxe_pool_get_index+0x19/0x69
> rxe_get_av+0xbe/0x14b
> rxe_requester+0x6b5/0xbb0
> ? rnr_nak_timer+0x16/0x16
> ? lock_downgrade+0xad/0xad
> ? rcu_read_lock_bh_held+0xab/0xab
> ? __wake_up+0xf/0xf
> ? mark_held_locks+0x1f/0x78
> ? __local_bh_enable_ip+0xb8/0xcc
> ? rnr_nak_timer+0x16/0x16
> rxe_do_task+0xb5/0x13d
> ? rxe_detach_mcast+0x1d6/0x1d6
> tasklet_action_common.constprop.0+0xda/0x145
> __do_softirq+0x202/0x489
> ? __irq_exit_rcu+0x108/0x108
> ? _local_bh_enable+0x1c/0x1c
> run_ksoftirqd+0x31/0x56
> smpboot_thread_fn+0x35c/0x376
> ? sort_range+0x1c/0x1c
> kthread+0x164/0x173
> ? kthread_complete_and_exit+0x20/0x20
> ret_from_fork+0x1f/0x30
> </TASK>
> CIFS: VFS: RDMA transport established
>