Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
QL41000 ethernet controller:
BUG: scheduling while atomic: kworker/0:4/531/0x00000200
[qed_probe:488()]hw prepare failed
kernel BUG at mm/vmalloc.c:2355!
Internal error: Oops - BUG: 0 [#1] SMP
CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
pstate: 00400009 (nzcv daif +PAN -UAO)
Call trace:
vunmap+0x4c/0x50
iounmap+0x48/0x58
qed_free_pci+0x60/0x80 [qed]
qed_probe+0x35c/0x688 [qed]
__qede_probe+0x88/0x5c8 [qede]
qede_probe+0x60/0xe0 [qede]
local_pci_probe+0x48/0xa0
work_for_cpu_fn+0x24/0x38
process_one_work+0x1d0/0x468
worker_thread+0x238/0x4e0
kthread+0xf0/0x118
ret_from_fork+0x10/0x18
In this case, qed_hw_prepare() returns error due to hw/fw error, but in
theory work queue should be in process context instead of interrupt.
The root cause might be the unpaired spin_{un}lock_bh() in
_qed_mcp_cmd_and_union(), which causes botton half is disabled incorrectly.
Reported-by: Lijian Zhang <[email protected]>
Signed-off-by: Jia He <[email protected]>
---
drivers/net/ethernet/qlogic/qed/qed_mcp.c | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_mcp.c b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
index 4387292c37e2..79d879a5d663 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_mcp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
@@ -474,14 +474,18 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
- if (!qed_mcp_has_pending_cmd(p_hwfn))
+ if (!qed_mcp_has_pending_cmd(p_hwfn)) {
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
break;
+ }
rc = qed_mcp_update_pending_cmd(p_hwfn, p_ptt);
- if (!rc)
+ if (!rc) {
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
break;
- else if (rc != -EAGAIN)
+ } else if (rc != -EAGAIN) {
goto err;
+ }
spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
@@ -498,6 +502,8 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
return -EAGAIN;
}
+ spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
+
/* Send the mailbox command */
qed_mcp_reread_offsets(p_hwfn, p_ptt);
seq_num = ++p_hwfn->mcp_info->drv_mb_seq;
@@ -524,14 +530,18 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
- if (p_cmd_elem->b_is_completed)
+ if (p_cmd_elem->b_is_completed) {
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
break;
+ }
rc = qed_mcp_update_pending_cmd(p_hwfn, p_ptt);
- if (!rc)
+ if (!rc) {
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
break;
- else if (rc != -EAGAIN)
+ } else if (rc != -EAGAIN) {
goto err;
+ }
spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
} while (++cnt < max_retries);
@@ -554,6 +564,7 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
return -EAGAIN;
}
+ spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
qed_mcp_cmd_del_elem(p_hwfn, p_cmd_elem);
spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
--
2.17.1
Hello:
This patch was applied to netdev/net.git (refs/heads/master):
On Thu, 15 Jul 2021 16:08:21 +0800 you wrote:
> Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> QL41000 ethernet controller:
> BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> [qed_probe:488()]hw prepare failed
> kernel BUG at mm/vmalloc.c:2355!
> Internal error: Oops - BUG: 0 [#1] SMP
> CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
> pstate: 00400009 (nzcv daif +PAN -UAO)
> Call trace:
> vunmap+0x4c/0x50
> iounmap+0x48/0x58
> qed_free_pci+0x60/0x80 [qed]
> qed_probe+0x35c/0x688 [qed]
> __qede_probe+0x88/0x5c8 [qede]
> qede_probe+0x60/0xe0 [qede]
> local_pci_probe+0x48/0xa0
> work_for_cpu_fn+0x24/0x38
> process_one_work+0x1d0/0x468
> worker_thread+0x238/0x4e0
> kthread+0xf0/0x118
> ret_from_fork+0x10/0x18
>
> [...]
Here is the summary with links:
- qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
https://git.kernel.org/netdev/net/c/6206b7981a36
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
> -----Original Message-----
> From: [email protected] <patchwork-
> [email protected]>
> Sent: Friday, July 16, 2021 3:50 AM
> To: Justin He <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected]; [email protected]; linux-
> [email protected]; nd <[email protected]>
> Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> _qed_mcp_cmd_and_union()
>
> Hello:
>
> This patch was applied to netdev/net.git (refs/heads/master):
>
> On Thu, 15 Jul 2021 16:08:21 +0800 you wrote:
> > Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> > QL41000 ethernet controller:
> > BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> > [qed_probe:488()]hw prepare failed
> > kernel BUG at mm/vmalloc.c:2355!
> > Internal error: Oops - BUG: 0 [#1] SMP
> > CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
> Ubuntu
> > pstate: 00400009 (nzcv daif +PAN -UAO)
> > Call trace:
> > vunmap+0x4c/0x50
> > iounmap+0x48/0x58
> > qed_free_pci+0x60/0x80 [qed]
> > qed_probe+0x35c/0x688 [qed]
> > __qede_probe+0x88/0x5c8 [qede]
> > qede_probe+0x60/0xe0 [qede]
> > local_pci_probe+0x48/0xa0
> > work_for_cpu_fn+0x24/0x38
> > process_one_work+0x1d0/0x468
> > worker_thread+0x238/0x4e0
> > kthread+0xf0/0x118
> > ret_from_fork+0x10/0x18
> >
> > [...]
>
> Here is the summary with links:
> - qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
> https://git.kernel.org/netdev/net/c/6206b7981a36
>
> You are awesome, thank you!
Thanks.
If possible, please also Cc: [email protected] because the bug seemed to
be there for a long time.
--
Cheers,
Justin (Jia He)
Hi Jia,
On Thu, Jul 15, 2021 at 2:28 PM Jia He <[email protected]> wrote:
>
> Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> QL41000 ethernet controller:
> BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> [qed_probe:488()]hw prepare failed
> kernel BUG at mm/vmalloc.c:2355!
> Internal error: Oops - BUG: 0 [#1] SMP
> CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
> pstate: 00400009 (nzcv daif +PAN -UAO)
> Call trace:
> vunmap+0x4c/0x50
> iounmap+0x48/0x58
> qed_free_pci+0x60/0x80 [qed]
> qed_probe+0x35c/0x688 [qed]
> __qede_probe+0x88/0x5c8 [qede]
> qede_probe+0x60/0xe0 [qede]
> local_pci_probe+0x48/0xa0
> work_for_cpu_fn+0x24/0x38
> process_one_work+0x1d0/0x468
> worker_thread+0x238/0x4e0
> kthread+0xf0/0x118
> ret_from_fork+0x10/0x18
>
> In this case, qed_hw_prepare() returns error due to hw/fw error, but in
> theory work queue should be in process context instead of interrupt.
>
> The root cause might be the unpaired spin_{un}lock_bh() in
> _qed_mcp_cmd_and_union(), which causes botton half is disabled incorrectly.
>
> Reported-by: Lijian Zhang <[email protected]>
> Signed-off-by: Jia He <[email protected]>
> ---
This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired
spin_{un}lock_bh.
Also,
as per description, looks like you are not sure actual the root-cause.
does this patch really solved the problem?
--pk
Hi Prabhakar
> -----Original Message-----
> From: Prabhakar Kushwaha <[email protected]>
> Sent: Monday, July 19, 2021 6:36 PM
> To: Justin He <[email protected]>
> Cc: Ariel Elior <[email protected]>; [email protected];
> David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>;
> [email protected]; Linux Kernel Mailing List <linux-
> [email protected]>; nd <[email protected]>; Shai Malin <[email protected]>;
> Shai Malin <[email protected]>; Prabhakar Kushwaha <[email protected]>
> Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> _qed_mcp_cmd_and_union()
>
> Hi Jia,
>
> On Thu, Jul 15, 2021 at 2:28 PM Jia He <[email protected]> wrote:
> >
> > Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> > QL41000 ethernet controller:
> > BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> > [qed_probe:488()]hw prepare failed
> > kernel BUG at mm/vmalloc.c:2355!
> > Internal error: Oops - BUG: 0 [#1] SMP
> > CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
> Ubuntu
> > pstate: 00400009 (nzcv daif +PAN -UAO)
> > Call trace:
> > vunmap+0x4c/0x50
> > iounmap+0x48/0x58
> > qed_free_pci+0x60/0x80 [qed]
> > qed_probe+0x35c/0x688 [qed]
> > __qede_probe+0x88/0x5c8 [qede]
> > qede_probe+0x60/0xe0 [qede]
> > local_pci_probe+0x48/0xa0
> > work_for_cpu_fn+0x24/0x38
> > process_one_work+0x1d0/0x468
> > worker_thread+0x238/0x4e0
> > kthread+0xf0/0x118
> > ret_from_fork+0x10/0x18
> >
> > In this case, qed_hw_prepare() returns error due to hw/fw error, but in
> > theory work queue should be in process context instead of interrupt.
> >
> > The root cause might be the unpaired spin_{un}lock_bh() in
> > _qed_mcp_cmd_and_union(), which causes botton half is disabled
> incorrectly.
> >
> > Reported-by: Lijian Zhang <[email protected]>
> > Signed-off-by: Jia He <[email protected]>
> > ---
>
> This patch is adding additional spin_{un}lock_bh().
> Can you please enlighten about the exact flow causing this unpaired
> spin_{un}lock_bh.
>
For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop
if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
}
> Also,
> as per description, looks like you are not sure actual the root-cause.
> does this patch really solved the problem?
I don't have that ThunderX2 to verify the patch.
But I searched all the spin_lock/unlock_bh and spin_lock_irqsave/irqrestore
under driver/.../qlogic, this is the only problematic point I could figure
out. And this might be possible code path of qed_probe().
--
Cheers,
Justin (Jia He)
Hi Justin,
On Mon, Jul 19, 2021 at 6:47 PM Justin He <[email protected]> wrote:
>
> Hi Prabhakar
>
> > -----Original Message-----
> > From: Prabhakar Kushwaha <[email protected]>
> > Sent: Monday, July 19, 2021 6:36 PM
> > To: Justin He <[email protected]>
> > Cc: Ariel Elior <[email protected]>; [email protected];
> > David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>;
> > [email protected]; Linux Kernel Mailing List <linux-
> > [email protected]>; nd <[email protected]>; Shai Malin <[email protected]>;
> > Shai Malin <[email protected]>; Prabhakar Kushwaha <[email protected]>
> > Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> > _qed_mcp_cmd_and_union()
> >
> > Hi Jia,
> >
> > On Thu, Jul 15, 2021 at 2:28 PM Jia He <[email protected]> wrote:
> > >
> > > Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
> > > QL41000 ethernet controller:
> > > BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> > > [qed_probe:488()]hw prepare failed
> > > kernel BUG at mm/vmalloc.c:2355!
> > > Internal error: Oops - BUG: 0 [#1] SMP
> > > CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
> > Ubuntu
> > > pstate: 00400009 (nzcv daif +PAN -UAO)
> > > Call trace:
> > > vunmap+0x4c/0x50
> > > iounmap+0x48/0x58
> > > qed_free_pci+0x60/0x80 [qed]
> > > qed_probe+0x35c/0x688 [qed]
> > > __qede_probe+0x88/0x5c8 [qede]
> > > qede_probe+0x60/0xe0 [qede]
> > > local_pci_probe+0x48/0xa0
> > > work_for_cpu_fn+0x24/0x38
> > > process_one_work+0x1d0/0x468
> > > worker_thread+0x238/0x4e0
> > > kthread+0xf0/0x118
> > > ret_from_fork+0x10/0x18
> > >
> > > In this case, qed_hw_prepare() returns error due to hw/fw error, but in
> > > theory work queue should be in process context instead of interrupt.
> > >
> > > The root cause might be the unpaired spin_{un}lock_bh() in
> > > _qed_mcp_cmd_and_union(), which causes botton half is disabled
> > incorrectly.
> > >
> > > Reported-by: Lijian Zhang <[email protected]>
> > > Signed-off-by: Jia He <[email protected]>
> > > ---
> >
> > This patch is adding additional spin_{un}lock_bh().
> > Can you please enlighten about the exact flow causing this unpaired
> > spin_{un}lock_bh.
> >
> For instance:
> _qed_mcp_cmd_and_union()
> In while loop
> spin_lock_bh()
> qed_mcp_has_pending_cmd() (assume false), will break the loop
I agree till here.
> if (cnt >= max_retries) {
> ...
> return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
> }
>
Because of break, cnt has not been increased.
- cnt is still less than max_retries.
- if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
Hence pairing completed.
I am not seeing any issue here.
> > Also,
> > as per description, looks like you are not sure actual the root-cause.
> > does this patch really solved the problem?
>
> I don't have that ThunderX2 to verify the patch.
> But I searched all the spin_lock/unlock_bh and spin_lock_irqsave/irqrestore
> under driver/.../qlogic, this is the only problematic point I could figure
> out. And this might be possible code path of qed_probe().
>
Without testing and proper root-cause, it is tough to accept the suggested fix.
--pk
> -----Original Message-----
> From: Prabhakar Kushwaha <[email protected]>
> Sent: Monday, July 19, 2021 10:51 PM
> To: Justin He <[email protected]>
> Cc: Ariel Elior <[email protected]>; [email protected];
> David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>;
> [email protected]; Linux Kernel Mailing List <linux-
> [email protected]>; nd <[email protected]>; Shai Malin <[email protected]>;
> Shai Malin <[email protected]>; Prabhakar Kushwaha <[email protected]>
> Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> _qed_mcp_cmd_and_union()
>
> Hi Justin,
>
> On Mon, Jul 19, 2021 at 6:47 PM Justin He <[email protected]> wrote:
> >
> > Hi Prabhakar
> >
> > > -----Original Message-----
> > > From: Prabhakar Kushwaha <[email protected]>
> > > Sent: Monday, July 19, 2021 6:36 PM
> > > To: Justin He <[email protected]>
> > > Cc: Ariel Elior <[email protected]>; [email protected];
> > > David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>;
> > > [email protected]; Linux Kernel Mailing List <linux-
> > > [email protected]>; nd <[email protected]>; Shai Malin
> <[email protected]>;
> > > Shai Malin <[email protected]>; Prabhakar Kushwaha
> <[email protected]>
> > > Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> > > _qed_mcp_cmd_and_union()
> > >
> > > Hi Jia,
> > >
> > > On Thu, Jul 15, 2021 at 2:28 PM Jia He <[email protected]> wrote:
> > > >
> > > > Liajian reported a bug_on hit on a ThunderX2 arm64 server with
> FastLinQ
> > > > QL41000 ethernet controller:
> > > > BUG: scheduling while atomic: kworker/0:4/531/0x00000200
> > > > [qed_probe:488()]hw prepare failed
> > > > kernel BUG at mm/vmalloc.c:2355!
> > > > Internal error: Oops - BUG: 0 [#1] SMP
> > > > CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic
> #86-
> > > Ubuntu
> > > > pstate: 00400009 (nzcv daif +PAN -UAO)
> > > > Call trace:
> > > > vunmap+0x4c/0x50
> > > > iounmap+0x48/0x58
> > > > qed_free_pci+0x60/0x80 [qed]
> > > > qed_probe+0x35c/0x688 [qed]
> > > > __qede_probe+0x88/0x5c8 [qede]
> > > > qede_probe+0x60/0xe0 [qede]
> > > > local_pci_probe+0x48/0xa0
> > > > work_for_cpu_fn+0x24/0x38
> > > > process_one_work+0x1d0/0x468
> > > > worker_thread+0x238/0x4e0
> > > > kthread+0xf0/0x118
> > > > ret_from_fork+0x10/0x18
> > > >
> > > > In this case, qed_hw_prepare() returns error due to hw/fw error, but
> in
> > > > theory work queue should be in process context instead of interrupt.
> > > >
> > > > The root cause might be the unpaired spin_{un}lock_bh() in
> > > > _qed_mcp_cmd_and_union(), which causes botton half is disabled
> > > incorrectly.
> > > >
> > > > Reported-by: Lijian Zhang <[email protected]>
> > > > Signed-off-by: Jia He <[email protected]>
> > > > ---
> > >
> > > This patch is adding additional spin_{un}lock_bh().
> > > Can you please enlighten about the exact flow causing this unpaired
> > > spin_{un}lock_bh.
> > >
> > For instance:
> > _qed_mcp_cmd_and_union()
> > In while loop
> > spin_lock_bh()
> > qed_mcp_has_pending_cmd() (assume false), will break the loop
>
> I agree till here.
>
> > if (cnt >= max_retries) {
> > ...
> > return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
> > }
> >
>
> Because of break, cnt has not been increased.
> - cnt is still less than max_retries.
> - if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
> Hence pairing completed.
Sorry, indeed. Let me check other possibilities.
@David S. Miller Sorry for the inconvenience, could you please revert it
in netdev tree?
Apologies again.
--
Cheers,
Justin (Jia He)
On Tue, 20 Jul 2021 02:02:26 +0000, Justin He wrote:
> > > For instance:
> > > _qed_mcp_cmd_and_union()
> > > In while loop
> > > spin_lock_bh()
> > > qed_mcp_has_pending_cmd() (assume false), will break the loop
> >
> > I agree till here.
> >
> > > if (cnt >= max_retries) {
> > > ...
> > > return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
> > > }
> > >
> >
> > Because of break, cnt has not been increased.
> > - cnt is still less than max_retries.
> > - if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
> > Hence pairing completed.
>
> Sorry, indeed. Let me check other possibilities.
> @David S. Miller Sorry for the inconvenience, could you please revert it
> in netdev tree?
Please submit a revert patch with the conclusions from the discussion
included in the commit message.
> -----Original Message-----
> From: Jakub Kicinski <[email protected]>
> Sent: Tuesday, July 20, 2021 5:06 PM
> To: Justin He <[email protected]>
> Cc: Prabhakar Kushwaha <[email protected]>; David S. Miller
> <[email protected]>; Ariel Elior <[email protected]>; GR-everest-linux-
> [email protected]; [email protected]; Linux Kernel Mailing List <linux-
> [email protected]>; nd <[email protected]>; Shai Malin <[email protected]>;
> Shai Malin <[email protected]>; Prabhakar Kushwaha <[email protected]>
> Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
> _qed_mcp_cmd_and_union()
>
> On Tue, 20 Jul 2021 02:02:26 +0000, Justin He wrote:
> > > > For instance:
> > > > _qed_mcp_cmd_and_union()
> > > > In while loop
> > > > spin_lock_bh()
> > > > qed_mcp_has_pending_cmd() (assume false), will break the loop
> > >
> > > I agree till here.
> > >
> > > > if (cnt >= max_retries) {
> > > > ...
> > > > return -EAGAIN; <-- here returns -EAGAIN without invoking bh
> unlock
> > > > }
> > > >
> > >
> > > Because of break, cnt has not been increased.
> > > - cnt is still less than max_retries.
> > > - if (cnt >= max_retries) will not be *true*, leading to
> spin_unlock_bh().
> > > Hence pairing completed.
> >
> > Sorry, indeed. Let me check other possibilities.
> > @David S. Miller Sorry for the inconvenience, could you please revert it
> > in netdev tree?
>
> Please submit a revert patch with the conclusions from the discussion
> included in the commit message.
Okay??will do that
Thanks for the reminder
--
Cheers,
Justin (Jia He)