PCIe AER error_detected caused a race issue with igb_resume.
Protect error_detected when igb is in down state.
Error logs:
kernel: igb 0000:02:00.0: disabling already-disabled device
kernel: WARNING: CPU: 0 PID: 277 at drivers/pci/pci.c:2248 pci_disable_device+0xc4/0xf0
kernel: RIP: 0010:pci_disable_device+0xc4/0xf0
kernel: Call Trace:
kernel: <TASK>
kernel: igb_io_error_detected+0x3e/0x60
kernel: report_error_detected+0xd6/0x1c0
kernel: ? __pfx_report_normal_detected+0x10/0x10
kernel: report_normal_detected+0x16/0x30
kernel: pci_walk_bus+0x74/0xa0
kernel: pcie_do_recovery+0xb9/0x340
kernel: ? __pfx_aer_root_reset+0x10/0x10
kernel: aer_process_err_devices+0x168/0x220
kernel: aer_isr+0x1b5/0x1e0
kernel: ? __pfx_irq_thread_fn+0x10/0x10
kernel: irq_thread_fn+0x21/0x70
kernel: irq_thread+0xf8/0x1c0
kernel: ? __pfx_irq_thread_dtor+0x10/0x10
kernel: ? __pfx_irq_thread+0x10/0x10
kernel: kthread+0xef/0x120
kernel: ? __pfx_kthread+0x10/0x10
kernel: ret_from_fork+0x29/0x50
kernel: </TASK>
kernel: ---[ end trace 0000000000000000 ]---
Link: https://bugzilla.kernel.org/show_bug.cgi?id=217446
Signed-off-by: Aaron Ma <[email protected]>
---
drivers/net/ethernet/intel/igb/igb_main.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index 58872a4c2540..8333d4ac8169 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c
@@ -9581,14 +9581,21 @@ static pci_ers_result_t igb_io_error_detected(struct pci_dev *pdev,
struct net_device *netdev = pci_get_drvdata(pdev);
struct igb_adapter *adapter = netdev_priv(netdev);
+ if (test_bit(__IGB_DOWN, &adapter->state))
+ return PCI_ERS_RESULT_DISCONNECT;
+
+ rtnl_lock();
netif_device_detach(netdev);
- if (state == pci_channel_io_perm_failure)
+ if (state == pci_channel_io_perm_failure) {
+ rtnl_unlock();
return PCI_ERS_RESULT_DISCONNECT;
+ }
if (netif_running(netdev))
igb_down(adapter);
pci_disable_device(pdev);
+ rtnl_unlock();
/* Request a slot reset. */
return PCI_ERS_RESULT_NEED_RESET;
--
2.34.1
On 5/26/2023 9:30 AM, Aaron Ma wrote:
Could you designate the target tree for the patch (net or net-next). I
presume it's net...
> PCIe AER error_detected caused a race issue with igb_resume.
> Protect error_detected when igb is in down state.
>
> Error logs:
> kernel: igb 0000:02:00.0: disabling already-disabled device
> kernel: WARNING: CPU: 0 PID: 277 at drivers/pci/pci.c:2248 pci_disable_device+0xc4/0xf0
> kernel: RIP: 0010:pci_disable_device+0xc4/0xf0
> kernel: Call Trace:
> kernel: <TASK>
> kernel: igb_io_error_detected+0x3e/0x60
> kernel: report_error_detected+0xd6/0x1c0
> kernel: ? __pfx_report_normal_detected+0x10/0x10
> kernel: report_normal_detected+0x16/0x30
> kernel: pci_walk_bus+0x74/0xa0
> kernel: pcie_do_recovery+0xb9/0x340
> kernel: ? __pfx_aer_root_reset+0x10/0x10
> kernel: aer_process_err_devices+0x168/0x220
> kernel: aer_isr+0x1b5/0x1e0
> kernel: ? __pfx_irq_thread_fn+0x10/0x10
> kernel: irq_thread_fn+0x21/0x70
> kernel: irq_thread+0xf8/0x1c0
> kernel: ? __pfx_irq_thread_dtor+0x10/0x10
> kernel: ? __pfx_irq_thread+0x10/0x10
> kernel: kthread+0xef/0x120
> kernel: ? __pfx_kthread+0x10/0x10
> kernel: ret_from_fork+0x29/0x50
> kernel: </TASK>
> kernel: ---[ end trace 0000000000000000 ]---
>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=217446
> Signed-off-by: Aaron Ma <[email protected]>
... which should have a Fixes: tag since this is bug fix.
https://www.kernel.org/doc/html/latest/process/maintainer-netdev.html#netdev-faq
Thanks,
Tony
PCIe AER error_detected caused a race issue with igb_resume.
Protect error_detected when igb is in down state.
Error logs:
kernel: igb 0000:02:00.0: disabling already-disabled device
kernel: WARNING: CPU: 0 PID: 277 at drivers/pci/pci.c:2248 pci_disable_device+0xc4/0xf0
kernel: RIP: 0010:pci_disable_device+0xc4/0xf0
kernel: Call Trace:
kernel: <TASK>
kernel: igb_io_error_detected+0x3e/0x60
kernel: report_error_detected+0xd6/0x1c0
kernel: ? __pfx_report_normal_detected+0x10/0x10
kernel: report_normal_detected+0x16/0x30
kernel: pci_walk_bus+0x74/0xa0
kernel: pcie_do_recovery+0xb9/0x340
kernel: ? __pfx_aer_root_reset+0x10/0x10
kernel: aer_process_err_devices+0x168/0x220
kernel: aer_isr+0x1b5/0x1e0
kernel: ? __pfx_irq_thread_fn+0x10/0x10
kernel: irq_thread_fn+0x21/0x70
kernel: irq_thread+0xf8/0x1c0
kernel: ? __pfx_irq_thread_dtor+0x10/0x10
kernel: ? __pfx_irq_thread+0x10/0x10
kernel: kthread+0xef/0x120
kernel: ? __pfx_kthread+0x10/0x10
kernel: ret_from_fork+0x29/0x50
kernel: </TASK>
kernel: ---[ end trace 0000000000000000 ]---
Link: https://bugzilla.kernel.org/show_bug.cgi?id=217446
Fixes: 9d5c824399de ("igb: PCI-Express 82575 Gigabit Ethernet driver")
Signed-off-by: Aaron Ma <[email protected]>
Reviewed-by: Mateusz Palczewski <[email protected]>
---
V1->V2: Add target tree tag net and Fixes tag.
drivers/net/ethernet/intel/igb/igb_main.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index 58872a4c2540..8333d4ac8169 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c
@@ -9581,14 +9581,21 @@ static pci_ers_result_t igb_io_error_detected(struct pci_dev *pdev,
struct net_device *netdev = pci_get_drvdata(pdev);
struct igb_adapter *adapter = netdev_priv(netdev);
+ if (test_bit(__IGB_DOWN, &adapter->state))
+ return PCI_ERS_RESULT_DISCONNECT;
+
+ rtnl_lock();
netif_device_detach(netdev);
- if (state == pci_channel_io_perm_failure)
+ if (state == pci_channel_io_perm_failure) {
+ rtnl_unlock();
return PCI_ERS_RESULT_DISCONNECT;
+ }
if (netif_running(netdev))
igb_down(adapter);
pci_disable_device(pdev);
+ rtnl_unlock();
/* Request a slot reset. */
return PCI_ERS_RESULT_NEED_RESET;
--
2.34.1
> -----Original Message-----
> From: Intel-wired-lan <[email protected]> On Behalf Of Aaron Ma
> Sent: Tuesday, June 6, 2023 6:57 PM
> To: Brandeburg, Jesse <[email protected]>; Nguyen, Anthony L <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Jeff Garzik <[email protected]>; Auke Kok <[email protected]>; [email protected]; [email protected]; [email protected]
> Subject: [Intel-wired-lan] [PATCH net v2] igb: fix hang issue of AER error during resume
>
> PCIe AER error_detected caused a race issue with igb_resume.
> Protect error_detected when igb is in down state.
>
> Error logs:
> kernel: igb 0000:02:00.0: disabling already-disabled device
> kernel: WARNING: CPU: 0 PID: 277 at drivers/pci/pci.c:2248 pci_disable_device+0xc4/0xf0
> kernel: RIP: 0010:pci_disable_device+0xc4/0xf0
> kernel: Call Trace:
> kernel: <TASK>
> kernel: igb_io_error_detected+0x3e/0x60
> kernel: report_error_detected+0xd6/0x1c0
> kernel: ? __pfx_report_normal_detected+0x10/0x10
> kernel: report_normal_detected+0x16/0x30
> kernel: pci_walk_bus+0x74/0xa0
> kernel: pcie_do_recovery+0xb9/0x340
> kernel: ? __pfx_aer_root_reset+0x10/0x10
> kernel: aer_process_err_devices+0x168/0x220
> kernel: aer_isr+0x1b5/0x1e0
> kernel: ? __pfx_irq_thread_fn+0x10/0x10
> kernel: irq_thread_fn+0x21/0x70
> kernel: irq_thread+0xf8/0x1c0
> kernel: ? __pfx_irq_thread_dtor+0x10/0x10
> kernel: ? __pfx_irq_thread+0x10/0x10
> kernel: kthread+0xef/0x120
> kernel: ? __pfx_kthread+0x10/0x10
> kernel: ret_from_fork+0x29/0x50
> kernel: </TASK>
> kernel: ---[ end trace 0000000000000000 ]---
>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=217446
> Fixes: 9d5c824399de ("igb: PCI-Express 82575 Gigabit Ethernet driver")
> Signed-off-by: Aaron Ma <[email protected]>
> Reviewed-by: Mateusz Palczewski <[email protected]>
> ---
> V1->V2: Add target tree tag net and Fixes tag.
>
> drivers/net/ethernet/intel/igb/igb_main.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
Tested-by: Sunitha Mekala <[email protected]> (A Contingent worker at Intel)
Hi Aaron,
On Wed, Jun 7, 2023 at 9:56 AM Aaron Ma <[email protected]> wrote:
>
> PCIe AER error_detected caused a race issue with igb_resume.
> Protect error_detected when igb is in down state.
>
> Error logs:
> kernel: igb 0000:02:00.0: disabling already-disabled device
> kernel: WARNING: CPU: 0 PID: 277 at drivers/pci/pci.c:2248 pci_disable_device+0xc4/0xf0
> kernel: RIP: 0010:pci_disable_device+0xc4/0xf0
> kernel: Call Trace:
> kernel: <TASK>
> kernel: igb_io_error_detected+0x3e/0x60
> kernel: report_error_detected+0xd6/0x1c0
> kernel: ? __pfx_report_normal_detected+0x10/0x10
> kernel: report_normal_detected+0x16/0x30
> kernel: pci_walk_bus+0x74/0xa0
> kernel: pcie_do_recovery+0xb9/0x340
> kernel: ? __pfx_aer_root_reset+0x10/0x10
> kernel: aer_process_err_devices+0x168/0x220
> kernel: aer_isr+0x1b5/0x1e0
> kernel: ? __pfx_irq_thread_fn+0x10/0x10
> kernel: irq_thread_fn+0x21/0x70
> kernel: irq_thread+0xf8/0x1c0
> kernel: ? __pfx_irq_thread_dtor+0x10/0x10
> kernel: ? __pfx_irq_thread+0x10/0x10
> kernel: kthread+0xef/0x120
> kernel: ? __pfx_kthread+0x10/0x10
> kernel: ret_from_fork+0x29/0x50
> kernel: </TASK>
> kernel: ---[ end trace 0000000000000000 ]---
>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=217446
> Fixes: 9d5c824399de ("igb: PCI-Express 82575 Gigabit Ethernet driver")
> Signed-off-by: Aaron Ma <[email protected]>
> Reviewed-by: Mateusz Palczewski <[email protected]>
Can you please give [1] a try?
I think avoid using rtnl_lock() is more ideal.
[1] https://lore.kernel.org/intel-wired-lan/[email protected]/
Kai-Heng
> ---
> V1->V2: Add target tree tag net and Fixes tag.
>
> drivers/net/ethernet/intel/igb/igb_main.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
> index 58872a4c2540..8333d4ac8169 100644
> --- a/drivers/net/ethernet/intel/igb/igb_main.c
> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> @@ -9581,14 +9581,21 @@ static pci_ers_result_t igb_io_error_detected(struct pci_dev *pdev,
> struct net_device *netdev = pci_get_drvdata(pdev);
> struct igb_adapter *adapter = netdev_priv(netdev);
>
> + if (test_bit(__IGB_DOWN, &adapter->state))
> + return PCI_ERS_RESULT_DISCONNECT;
> +
> + rtnl_lock();
> netif_device_detach(netdev);
>
> - if (state == pci_channel_io_perm_failure)
> + if (state == pci_channel_io_perm_failure) {
> + rtnl_unlock();
> return PCI_ERS_RESULT_DISCONNECT;
> + }
>
> if (netif_running(netdev))
> igb_down(adapter);
> pci_disable_device(pdev);
> + rtnl_unlock();
>
> /* Request a slot reset. */
> return PCI_ERS_RESULT_NEED_RESET;
> --
> 2.34.1
>
> _______________________________________________
> Intel-wired-lan mailing list
> [email protected]
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan