LinuxLists.cc - excuse-lockdep help

2021-04-06 11:35:27

Subject: excuse-lockdep help

Hi maintainer,and stranger v5.4-rt kernel, there some BUGs error:

[ 29.286213] BUG: workqueue leaked lock or atomic: kworker/u8:0/0x00000000/7
last function: rpc_async_schedule
[ 29.286215] 9 locks held by kworker/u8:0/7:
[ 29.286217] #0: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
[ 29.286222] #1: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
[ 29.286226] #2: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
[ 29.286230] #3: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
[ 29.286241] #4: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
[ 29.286245] #5: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
[ 29.286249] #6: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
[ 29.286253] #7: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
[ 29.286258] #8: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
[ 29.286262] CPU: 2 PID: 7 Comm: kworker/u8:0 Tainted: G W 5.4.107-rt44-yocto-preempt-rt #1
[ 29.286263] Hardware name: Intel Corporation NUC7i5DNKE/NUC7i5DNB, BIOS DNKBLi5v.86A.0026.2017.0915.2016 09/15/2017
[ 29.286266] Workqueue: rpciod rpc_async_schedule
[ 29.286266] Call Trace:
[ 29.286268] show_stack+0x4e/0x52
[ 29.286272] dump_stack+0x7a/0xa4
[ 29.286276] process_one_work.cold+0x43/0x48
[ 29.286292] worker_thread+0x217/0x330
[ 29.286300] kthread+0x177/0x190
[ 29.286302] ? process_one_work+0x410/0x410
[ 29.286303] ? __kthread_parkme+0xc0/0xc0
[ 29.286310] ret_from_fork+0x3a/0x50

On 4/6/21 1:57 AM, Chuck Lever III wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
>> On Mar 28, 2021, at 10:31 PM, jun.miao <[email protected]> wrote:
>>
>> Hi maintainers
>>
>> I find a BUG calltrace in v5.4-rt:
>>
>> [ 29.286213] BUG: workqueue leaked lock or atomic: kworker/u8:0/0x00000000/7
>> last function: rpc_async_schedule
>> [ 29.286215] 9 locks held by kworker/u8:0/7:
>> [ 29.286217] #0: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 29.286222] #1: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 29.286226] #2: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 29.286230] #3: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 29.286241] #4: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 29.286245] #5: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 29.286249] #6: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 29.286253] #7: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 29.286258] #8: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 29.286262] CPU: 2 PID: 7 Comm: kworker/u8:0 Tainted: G W 5.4.107-rt44-yocto-preempt-rt #1
>> [ 29.286263] Hardware name: Intel Corporation NUC7i5DNKE/NUC7i5DNB, BIOS DNKBLi5v.86A.0026.2017.0915.2016 09/15/2017
>> [ 29.286266] Workqueue: rpciod rpc_async_schedule
>> [ 29.286266] Call Trace:
>> [ 29.286268] show_stack+0x4e/0x52
>> [ 29.286272] dump_stack+0x7a/0xa4
>> [ 29.286276] process_one_work.cold+0x43/0x48
>> [ 29.286292] worker_thread+0x217/0x330
>> [ 29.286300] kthread+0x177/0x190
>> [ 29.286302] ? process_one_work+0x410/0x410
>> [ 29.286303] ? __kthread_parkme+0xc0/0xc0
>> [ 29.286310] ret_from_fork+0x3a/0x50
>>
>>
>> When i git am this patch:
>>
>> commit 1a33e10e4a95cb109ff1145098175df3113313ef
>> Author: Cong Wang <[email protected]>
>> Date: Sat May 2 22:22:19 2020 -0700
>>
>> net: partially revert dynamic lockdep key changes
>>
>> The BUG also print:
>>
>> BUG: workqueue leaked lock or atomic: kworker/u8:0/0x00000000/7
>> [ 28.869159] 000: last function: rpc_async_release
>> [ 28.869161] 000: 13 locks held by kworker/u8:0/7:
>> [ 28.869162] 000: #0: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869167] 000: #1: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869172] 000: #2: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869177] 000: #3: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869189] 000: #4: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869193] 000: #5: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869198] 000: #6: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869203] 000: #7: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869208] 000: #8: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869213] 000: #9: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869218] 000: #10: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869223] 000: #11: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869228] 000: #12: ffff90b817de42f0 (&(&sch->running)->seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
>> [ 28.869233] 000: CPU: 0 PID: 7 Comm: kworker/u8:0 Tainted: G W 5.4.107-rt44-yocto-preempt-rt #1
>> [ 28.869234] 000: Hardware name: Intel Corporation NUC7i5DNKE/NUC7i5DNB, BIOS DNKBLi5v.86A.0026.2017.0915.2016 09/15/2017
>> [ 28.869236] 000: Workqueue: nfsiod rpc_async_release
>> [ 28.869236] 000: Call Trace:
>> [ 28.869238] 000: show_stack+0x4e/0x52
>> [ 28.869242] 000: dump_stack+0x7a/0xa4
>> [ 28.869247] 000: process_one_work.cold+0x43/0x48
>> [ 28.869266] 000: worker_thread+0x53/0x330
>> [ 28.869275] 000: kthread+0x177/0x190
>> [ 28.869278] 000: ? process_one_work+0x410/0x410
>> [ 28.869279] 000: ? __kthread_parkme+0xc0/0xc0
>> [ 28.869286] 000: ret_from_fork+0x3a/0x50
>>
>>
>>
>> Could you give me some advice about rpc_async_schedule() in sunrpc File ?
>>
>> Thanks very much
> Hello-
>
> rpc_async_schedule() is part of the RPC client, whose maintainers
> are Trond Myklebust and Anna Schumaker. Please post this WQ splat
> to [email protected] and Cc: Trond and Anna. Thanks!
>
>
> --
> Chuck Lever
>
>
>

2021-04-06 13:02:07

by Trond Myklebust

[permalink] [raw]

Subject: Re: excuse-lockdep help

On Tue, 2021-04-06 at 09:31 +0800, jun.miao wrote:
> Hi maintainer,and stranger v5.4-rt kernel, there some BUGs error:
>
> [   29.286213] BUG: workqueue leaked lock or atomic:
> kworker/u8:0/0x00000000/7
>                      last function: rpc_async_schedule
> [   29.286215] 9 locks held by kworker/u8:0/7:
> [   29.286217] #0: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.},
> at: __dev_queue_xmit+0x32e/0x610
> [   29.286222] #1: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.},
> at: __dev_queue_xmit+0x32e/0x610
> [   29.286226] #2: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.},
> at: __dev_queue_xmit+0x32e/0x610
> [   29.286230] #3: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.},
> at: __dev_queue_xmit+0x32e/0x610
> [   29.286241] #4: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.},
> at: __dev_queue_xmit+0x32e/0x610
> [   29.286245] #5: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.},
> at: __dev_queue_xmit+0x32e/0x610
> [   29.286249] #6: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.},
> at: __dev_queue_xmit+0x32e/0x610
> [   29.286253] #7: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.},
> at: __dev_queue_xmit+0x32e/0x610
> [   29.286258] #8: ffff953b5745c2f0 (&dev->qdisc_running_key){+.+.},
> at: __dev_queue_xmit+0x32e/0x610
> [   29.286262] CPU: 2 PID: 7 Comm: kworker/u8:0 Tainted: G W
> 5.4.107-rt44-yocto-preempt-rt #1
> [   29.286263] Hardware name: Intel Corporation NUC7i5DNKE/NUC7i5DNB,
> BIOS DNKBLi5v.86A.0026.2017.0915.2016 09/15/2017
> [   29.286266] Workqueue: rpciod rpc_async_schedule
> [   29.286266] Call Trace:
> [   29.286268] show_stack+0x4e/0x52
> [   29.286272] dump_stack+0x7a/0xa4
> [   29.286276] process_one_work.cold+0x43/0x48
> [   29.286292] worker_thread+0x217/0x330
> [   29.286300] kthread+0x177/0x190
> [   29.286302] ? process_one_work+0x410/0x410
> [   29.286303] ? __kthread_parkme+0xc0/0xc0
> [   29.286310] ret_from_fork+0x3a/0x50
>

I can't see how this could be a RPC or NFS bug. The lock being leaked
is deep in the networking code, and is never taken by the higher
levels.

Furthermore, this is not an upstream kernel, but some Yocto project
franken-kernel with proprietary drivers having been loaded. I can't see
how we can support it.

>
>
> On 4/6/21 1:57 AM, Chuck Lever III wrote:
> > [Please note: This e-mail is from an EXTERNAL e-mail address]
> >
> > > On Mar 28, 2021, at 10:31 PM, jun.miao <[email protected]>
> > > wrote:
> > >
> > > Hi maintainers
> > >
> > > I find a BUG calltrace in v5.4-rt:
> > >
> > > [   29.286213] BUG: workqueue leaked lock or atomic:
> > > kworker/u8:0/0x00000000/7
> > >                      last function: rpc_async_schedule
> > > [   29.286215] 9 locks held by kworker/u8:0/7:
> > > [   29.286217] #0: ffff953b5745c2f0 (&dev-
> > > >qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   29.286222] #1: ffff953b5745c2f0 (&dev-
> > > >qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   29.286226] #2: ffff953b5745c2f0 (&dev-
> > > >qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   29.286230] #3: ffff953b5745c2f0 (&dev-
> > > >qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   29.286241] #4: ffff953b5745c2f0 (&dev-
> > > >qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   29.286245] #5: ffff953b5745c2f0 (&dev-
> > > >qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   29.286249] #6: ffff953b5745c2f0 (&dev-
> > > >qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   29.286253] #7: ffff953b5745c2f0 (&dev-
> > > >qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   29.286258] #8: ffff953b5745c2f0 (&dev-
> > > >qdisc_running_key){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   29.286262] CPU: 2 PID: 7 Comm: kworker/u8:0 Tainted: G
> > > W         5.4.107-rt44-yocto-preempt-rt #1
> > > [   29.286263] Hardware name: Intel Corporation
> > > NUC7i5DNKE/NUC7i5DNB, BIOS DNKBLi5v.86A.0026.2017.0915.2016
> > > 09/15/2017
> > > [   29.286266] Workqueue: rpciod rpc_async_schedule
> > > [   29.286266] Call Trace:
> > > [   29.286268] show_stack+0x4e/0x52
> > > [   29.286272] dump_stack+0x7a/0xa4
> > > [   29.286276] process_one_work.cold+0x43/0x48
> > > [   29.286292] worker_thread+0x217/0x330
> > > [   29.286300] kthread+0x177/0x190
> > > [   29.286302] ? process_one_work+0x410/0x410
> > > [   29.286303] ? __kthread_parkme+0xc0/0xc0
> > > [   29.286310] ret_from_fork+0x3a/0x50
> > >
> > >
> > > When i git am this patch:
> > >
> > >      commit 1a33e10e4a95cb109ff1145098175df3113313ef
> > > Author: Cong Wang <[email protected]>
> > > Date:   Sat May 2 22:22:19 2020 -0700
> > >
> > >      net: partially revert dynamic lockdep key changes
> > >
> > > The BUG also print:
> > >
> > > BUG: workqueue leaked lock or atomic: kworker/u8:0/0x00000000/7
> > > [   28.869159] 000:      last function: rpc_async_release
> > > [   28.869161] 000: 13 locks held by kworker/u8:0/7:
> > > [   28.869162] 000: #0: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869167] 000: #1: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869172] 000: #2: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869177] 000: #3: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869189] 000: #4: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869193] 000: #5: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869198] 000: #6: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869203] 000: #7: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869208] 000: #8: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869213] 000: #9: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869218] 000: #10: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869223] 000: #11: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869228] 000: #12: ffff90b817de42f0 (&(&sch->running)-
> > > >seqcount){+.+.}, at: __dev_queue_xmit+0x32e/0x610
> > > [   28.869233] 000: CPU: 0 PID: 7 Comm: kworker/u8:0 Tainted:
> > > G        W         5.4.107-rt44-yocto-preempt-rt #1
> > > [   28.869234] 000: Hardware name: Intel Corporation
> > > NUC7i5DNKE/NUC7i5DNB, BIOS DNKBLi5v.86A.0026.2017.0915.2016
> > > 09/15/2017
> > > [   28.869236] 000: Workqueue: nfsiod rpc_async_release
> > > [   28.869236] 000: Call Trace:
> > > [   28.869238] 000: show_stack+0x4e/0x52
> > > [   28.869242] 000: dump_stack+0x7a/0xa4
> > > [   28.869247] 000: process_one_work.cold+0x43/0x48
> > > [   28.869266] 000: worker_thread+0x53/0x330
> > > [   28.869275] 000: kthread+0x177/0x190
> > > [   28.869278] 000: ? process_one_work+0x410/0x410
> > > [   28.869279] 000: ? __kthread_parkme+0xc0/0xc0
> > > [   28.869286] 000: ret_from_fork+0x3a/0x50
> > >
> > >
> > >
> > > Could you give me some advice about rpc_async_schedule() in
> > > sunrpc File ?
> > >
> > > Thanks very much
> > Hello-
> >
> > rpc_async_schedule() is part of the RPC client, whose maintainers
> > are Trond Myklebust and Anna Schumaker. Please post this WQ splat
> > to [email protected] and Cc: Trond and Anna. Thanks!
> >
> >
> > --
> > Chuck Lever
> >
> >
> >

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]