In a setup where a Thunderbolt hub connects to Ethernet and a display
through USB Type-C, users may experience a hung task timeout when they
remove the cable between the PC and the Thunderbolt hub.
This is because the igb_down function is called multiple times when
the Thunderbolt hub is unplugged. For example, the igb_io_error_detected
triggers the first call, and the igb_remove triggers the second call.
The second call to igb_down will block at napi_synchronize.
Here's the call trace:
__schedule+0x3b0/0xddb
? __mod_timer+0x164/0x5d3
schedule+0x44/0xa8
schedule_timeout+0xb2/0x2a4
? run_local_timers+0x4e/0x4e
msleep+0x31/0x38
igb_down+0x12c/0x22a [igb 6615058754948bfde0bf01429257eb59f13030d4]
__igb_close+0x6f/0x9c [igb 6615058754948bfde0bf01429257eb59f13030d4]
igb_close+0x23/0x2b [igb 6615058754948bfde0bf01429257eb59f13030d4]
__dev_close_many+0x95/0xec
dev_close_many+0x6e/0x103
unregister_netdevice_many+0x105/0x5b1
unregister_netdevice_queue+0xc2/0x10d
unregister_netdev+0x1c/0x23
igb_remove+0xa7/0x11c [igb 6615058754948bfde0bf01429257eb59f13030d4]
pci_device_remove+0x3f/0x9c
device_release_driver_internal+0xfe/0x1b4
pci_stop_bus_device+0x5b/0x7f
pci_stop_bus_device+0x30/0x7f
pci_stop_bus_device+0x30/0x7f
pci_stop_and_remove_bus_device+0x12/0x19
pciehp_unconfigure_device+0x76/0xe9
pciehp_disable_slot+0x6e/0x131
pciehp_handle_presence_or_link_change+0x7a/0x3f7
pciehp_ist+0xbe/0x194
irq_thread_fn+0x22/0x4d
? irq_thread+0x1fd/0x1fd
irq_thread+0x17b/0x1fd
? irq_forced_thread_fn+0x5f/0x5f
kthread+0x142/0x153
? __irq_get_irqchip_state+0x46/0x46
? kthread_associate_blkcg+0x71/0x71
ret_from_fork+0x1f/0x30
In this case, igb_io_error_detected detaches the network interface
and requests a PCIE slot reset, however, the PCIE reset callback is
not being invoked and thus the Ethernet connection breaks down.
As the PCIE error in this case is a non-fatal one, requesting a
slot reset can be avoided.
This patch fixes the task hung issue and preserves Ethernet
connection by ignoring non-fatal PCIE errors.
Signed-off-by: Ying Hsu <[email protected]>
---
This commit has been tested on a HP Elite Dragonfly Chromebook and
a Caldigit TS3+ Thunderbolt hub. The Ethernet driver for the hub
is igb. Non-fatal PCIE errors happen when users hot-plug the cables
connected to the chromebook or to the external display.
drivers/net/ethernet/intel/igb/igb_main.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index 58872a4c2540..a8b217368ca1 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c
@@ -9581,6 +9581,11 @@ static pci_ers_result_t igb_io_error_detected(struct pci_dev *pdev,
struct net_device *netdev = pci_get_drvdata(pdev);
struct igb_adapter *adapter = netdev_priv(netdev);
+ if (state == pci_channel_io_normal) {
+ dev_warn(&pdev->dev, "Non-correctable non-fatal error reported.\n");
+ return PCI_ERS_RESULT_CAN_RECOVER;
+ }
+
netif_device_detach(netdev);
if (state == pci_channel_io_perm_failure)
--
2.40.1.606.ga4b1b128d6-goog
On Thu, May 18, 2023 at 12:58 PM Ying Hsu <[email protected]> wrote:
>
> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
> index 58872a4c2540..a8b217368ca1 100644
> --- a/drivers/net/ethernet/intel/igb/igb_main.c
> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> @@ -9581,6 +9581,11 @@ static pci_ers_result_t igb_io_error_detected(struct pci_dev *pdev,
> struct net_device *netdev = pci_get_drvdata(pdev);
> struct igb_adapter *adapter = netdev_priv(netdev);
>
> + if (state == pci_channel_io_normal) {
> + dev_warn(&pdev->dev, "Non-correctable non-fatal error reported.\n");
> + return PCI_ERS_RESULT_CAN_RECOVER;
> + }
> +
This code may be good to have. But not sure if this should be the fix
for igb_down() synchronization.
Intel guys may comment.
> netif_device_detach(netdev);
>
> if (state == pci_channel_io_perm_failure)
> --
> 2.40.1.606.ga4b1b128d6-goog
>
>
On Thu, May 18, 2023 at 3:36 AM Pavan Chebbi <[email protected]> wrote:
>
> On Thu, May 18, 2023 at 12:58 PM Ying Hsu <[email protected]> wrote:
> >
> > diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
> > index 58872a4c2540..a8b217368ca1 100644
> > --- a/drivers/net/ethernet/intel/igb/igb_main.c
> > +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> > @@ -9581,6 +9581,11 @@ static pci_ers_result_t igb_io_error_detected(struct pci_dev *pdev,
> > struct net_device *netdev = pci_get_drvdata(pdev);
> > struct igb_adapter *adapter = netdev_priv(netdev);
> >
> > + if (state == pci_channel_io_normal) {
> > + dev_warn(&pdev->dev, "Non-correctable non-fatal error reported.\n");
> > + return PCI_ERS_RESULT_CAN_RECOVER;
> > + }
> > +
>
> This code may be good to have. But not sure if this should be the fix
> for igb_down() synchronization.
I have the same opinion. This appears to solve the problem - but I
don't know if there is a better way to solve this problem.
> Intel guys may comment.
Ping? Can we please get feedback from IGB/IGC maintainers this week?
(I hope igc maintainers can confirm this isn't an issue for igc.)
cheers,
grant
>
> > netif_device_detach(netdev);
> >
> > if (state == pci_channel_io_perm_failure)
> > --
> > 2.40.1.606.ga4b1b128d6-goog
> >
> >
On 5/22/2023 1:16 PM, Grant Grundler wrote:
> On Thu, May 18, 2023 at 3:36 AM Pavan Chebbi <[email protected]> wrote:
>>
>> On Thu, May 18, 2023 at 12:58 PM Ying Hsu <[email protected]> wrote:
>>>
>>> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
>>> index 58872a4c2540..a8b217368ca1 100644
>>> --- a/drivers/net/ethernet/intel/igb/igb_main.c
>>> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
>>> @@ -9581,6 +9581,11 @@ static pci_ers_result_t igb_io_error_detected(struct pci_dev *pdev,
>>> struct net_device *netdev = pci_get_drvdata(pdev);
>>> struct igb_adapter *adapter = netdev_priv(netdev);
>>>
>>> + if (state == pci_channel_io_normal) {
>>> + dev_warn(&pdev->dev, "Non-correctable non-fatal error reported.\n");
>>> + return PCI_ERS_RESULT_CAN_RECOVER;
>>> + }
>>> +
>>
>> This code may be good to have. But not sure if this should be the fix
>> for igb_down() synchronization.
>
> I have the same opinion. This appears to solve the problem - but I
> don't know if there is a better way to solve this problem.
>
>> Intel guys may comment.
>
> Ping? Can we please get feedback from IGB/IGC maintainers this week?
>
> (I hope igc maintainers can confirm this isn't an issue for igc.)
Adding some of the igb and igc developers.
> cheers,
> grant
>
>>
>>> netif_device_detach(netdev);
>>>
>>> if (state == pci_channel_io_perm_failure)
>>> --
>>> 2.40.1.606.ga4b1b128d6-goog
>>>
>>>
Good day Tony
We reviewed the patch and have nothing against.
With the best regards
Alex
ND ITP Linux 40G base driver TL
> -----Original Message-----
> From: Nguyen, Anthony L <[email protected]>
> Sent: Tuesday, May 23, 2023 8:04 PM
> To: Grant Grundler <[email protected]>; Pavan Chebbi
> <[email protected]>; Loktionov, Aleksandr
> <[email protected]>; Neftin, Sasha <[email protected]>;
> Ruinskiy, Dima <[email protected]>
> Cc: Ying Hsu <[email protected]>; [email protected]; David S.
> Miller <[email protected]>; Eric Dumazet <[email protected]>;
> Jakub Kicinski <[email protected]>; Brandeburg, Jesse
> <[email protected]>; Paolo Abeni <[email protected]>; intel-
> [email protected]; [email protected]
> Subject: Re: [PATCH] igb: Fix igb_down hung on surprise removal
>
> On 5/22/2023 1:16 PM, Grant Grundler wrote:
> > On Thu, May 18, 2023 at 3:36 AM Pavan Chebbi
> <[email protected]> wrote:
> >>
> >> On Thu, May 18, 2023 at 12:58 PM Ying Hsu <[email protected]>
> wrote:
> >>>
> >>> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c
> >>> b/drivers/net/ethernet/intel/igb/igb_main.c
> >>> index 58872a4c2540..a8b217368ca1 100644
> >>> --- a/drivers/net/ethernet/intel/igb/igb_main.c
> >>> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> >>> @@ -9581,6 +9581,11 @@ static pci_ers_result_t
> igb_io_error_detected(struct pci_dev *pdev,
> >>> struct net_device *netdev = pci_get_drvdata(pdev);
> >>> struct igb_adapter *adapter = netdev_priv(netdev);
> >>>
> >>> + if (state == pci_channel_io_normal) {
> >>> + dev_warn(&pdev->dev, "Non-correctable non-fatal error
> reported.\n");
> >>> + return PCI_ERS_RESULT_CAN_RECOVER;
> >>> + }
> >>> +
> >>
> >> This code may be good to have. But not sure if this should be the fix
> >> for igb_down() synchronization.
> >
> > I have the same opinion. This appears to solve the problem - but I
> > don't know if there is a better way to solve this problem.
> >
> >> Intel guys may comment.
> >
> > Ping? Can we please get feedback from IGB/IGC maintainers this week?
> >
> > (I hope igc maintainers can confirm this isn't an issue for igc.)
>
> Adding some of the igb and igc developers.
>
> > cheers,
> > grant
> >
> >>
> >>> netif_device_detach(netdev);
> >>>
> >>> if (state == pci_channel_io_perm_failure)
> >>> --
> >>> 2.40.1.606.ga4b1b128d6-goog
> >>>
> >>>
On Wed, May 24, 2023 at 5:34 AM Loktionov, Aleksandr
<[email protected]> wrote:
>
> Good day Tony
>
> We reviewed the patch and have nothing against.
Thank you for reviewing!
Can I take this as the equivalent of "Signed-off-by: Loktionov,
Aleksandr <[email protected]>"?
Or since Tony is listed in MAINTAINERS for drivers/net/ethernet/intel,
is he supposed to provide that?
cheers,
grant
>
> With the best regards
> Alex
> ND ITP Linux 40G base driver TL
>
>
>
> > -----Original Message-----
> > From: Nguyen, Anthony L <[email protected]>
> > Sent: Tuesday, May 23, 2023 8:04 PM
> > To: Grant Grundler <[email protected]>; Pavan Chebbi
> > <[email protected]>; Loktionov, Aleksandr
> > <[email protected]>; Neftin, Sasha <[email protected]>;
> > Ruinskiy, Dima <[email protected]>
> > Cc: Ying Hsu <[email protected]>; [email protected]; David S.
> > Miller <[email protected]>; Eric Dumazet <[email protected]>;
> > Jakub Kicinski <[email protected]>; Brandeburg, Jesse
> > <[email protected]>; Paolo Abeni <[email protected]>; intel-
> > [email protected]; [email protected]
> > Subject: Re: [PATCH] igb: Fix igb_down hung on surprise removal
> >
> > On 5/22/2023 1:16 PM, Grant Grundler wrote:
> > > On Thu, May 18, 2023 at 3:36 AM Pavan Chebbi
> > <[email protected]> wrote:
> > >>
> > >> On Thu, May 18, 2023 at 12:58 PM Ying Hsu <[email protected]>
> > wrote:
> > >>>
> > >>> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c
> > >>> b/drivers/net/ethernet/intel/igb/igb_main.c
> > >>> index 58872a4c2540..a8b217368ca1 100644
> > >>> --- a/drivers/net/ethernet/intel/igb/igb_main.c
> > >>> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> > >>> @@ -9581,6 +9581,11 @@ static pci_ers_result_t
> > igb_io_error_detected(struct pci_dev *pdev,
> > >>> struct net_device *netdev = pci_get_drvdata(pdev);
> > >>> struct igb_adapter *adapter = netdev_priv(netdev);
> > >>>
> > >>> + if (state == pci_channel_io_normal) {
> > >>> + dev_warn(&pdev->dev, "Non-correctable non-fatal error
> > reported.\n");
> > >>> + return PCI_ERS_RESULT_CAN_RECOVER;
> > >>> + }
> > >>> +
> > >>
> > >> This code may be good to have. But not sure if this should be the fix
> > >> for igb_down() synchronization.
> > >
> > > I have the same opinion. This appears to solve the problem - but I
> > > don't know if there is a better way to solve this problem.
> > >
> > >> Intel guys may comment.
> > >
> > > Ping? Can we please get feedback from IGB/IGC maintainers this week?
> > >
> > > (I hope igc maintainers can confirm this isn't an issue for igc.)
> >
> > Adding some of the igb and igc developers.
> >
> > > cheers,
> > > grant
> > >
> > >>
> > >>> netif_device_detach(netdev);
> > >>>
> > >>> if (state == pci_channel_io_perm_failure)
> > >>> --
> > >>> 2.40.1.606.ga4b1b128d6-goog
> > >>>
> > >>>
On Wed, May 24, 2023 at 3:22 PM Tony Nguyen <[email protected]> wrote:
>
> Hi Grant,
>
> On 5/24/2023 2:01 PM, Grant Grundler wrote:
> > On Wed, May 24, 2023 at 5:34 AM Loktionov, Aleksandr
> > <[email protected]> wrote:
> >>
> >> Good day Tony
> >>
> >> We reviewed the patch and have nothing against.
> >
> > Thank you for reviewing!
> >
> > Can I take this as the equivalent of "Signed-off-by: Loktionov,
> > Aleksandr <[email protected]>"?
>
> Unless a tag is explicitly given, I don't believe one can be inferred.
Yes - that's what I thought and was asking in case that's what
Aleksandr meant (and could easily confirm)
>
> > Or since Tony is listed in MAINTAINERS for drivers/net/ethernet/intel,
> > is he supposed to provide that?
>
> Assuming there's no comments/issues brought up, I'll apply it to the
> respective Intel Wired LAN tree for our validation to have a pass at it.
> Upon successful completion, I'll send the patch on to netdev for them to
> pull. Hope that helps.
Yes - that's what I needed to know. Thank you Tony! :)
Ying Hsu will apply this patch to Chrome OS kernels:
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/4548800
cheers,
grant
>
> Thanks,
> Tony
Hi Grant,
On 5/24/2023 2:01 PM, Grant Grundler wrote:
> On Wed, May 24, 2023 at 5:34 AM Loktionov, Aleksandr
> <[email protected]> wrote:
>>
>> Good day Tony
>>
>> We reviewed the patch and have nothing against.
>
> Thank you for reviewing!
>
> Can I take this as the equivalent of "Signed-off-by: Loktionov,
> Aleksandr <[email protected]>"?
Unless a tag is explicitly given, I don't believe one can be inferred.
> Or since Tony is listed in MAINTAINERS for drivers/net/ethernet/intel,
> is he supposed to provide that?
Assuming there's no comments/issues brought up, I'll apply it to the
respective Intel Wired LAN tree for our validation to have a pass at it.
Upon successful completion, I'll send the patch on to netdev for them to
pull. Hope that helps.
Thanks,
Tony
> -----Original Message-----
> From: Intel-wired-lan <[email protected]> On Behalf Of Ying Hsu
> Sent: Thursday, May 18, 2023 12:57 PM
> To: [email protected]
> Cc: [email protected]; [email protected]; Ying Hsu <[email protected]>; Brandeburg, Jesse <[email protected]>; [email protected]; Eric Dumazet <[email protected]>; Nguyen, Anthony L <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; David S. Miller <[email protected]>
> Subject: [Intel-wired-lan] [PATCH] igb: Fix igb_down hung on surprise removal
>
> In a setup where a Thunderbolt hub connects to Ethernet and a display through USB Type-C, users may experience a hung task timeout when they remove the cable between the PC and the Thunderbolt hub.
> This is because the igb_down function is called multiple times when the Thunderbolt hub is unplugged. For example, the igb_io_error_detected triggers the first call, and the igb_remove triggers the second call.
> The second call to igb_down will block at napi_synchronize.
> Here's the call trace:
> __schedule+0x3b0/0xddb
> ? __mod_timer+0x164/0x5d3
> schedule+0x44/0xa8
> schedule_timeout+0xb2/0x2a4
> ? run_local_timers+0x4e/0x4e
> msleep+0x31/0x38
> igb_down+0x12c/0x22a [igb 6615058754948bfde0bf01429257eb59f13030d4]
> __igb_close+0x6f/0x9c [igb 6615058754948bfde0bf01429257eb59f13030d4]
> igb_close+0x23/0x2b [igb 6615058754948bfde0bf01429257eb59f13030d4]
> __dev_close_many+0x95/0xec
> dev_close_many+0x6e/0x103
> unregister_netdevice_many+0x105/0x5b1
> unregister_netdevice_queue+0xc2/0x10d
> unregister_netdev+0x1c/0x23
> igb_remove+0xa7/0x11c [igb 6615058754948bfde0bf01429257eb59f13030d4]
> pci_device_remove+0x3f/0x9c
> device_release_driver_internal+0xfe/0x1b4
> pci_stop_bus_device+0x5b/0x7f
> pci_stop_bus_device+0x30/0x7f
> pci_stop_bus_device+0x30/0x7f
> pci_stop_and_remove_bus_device+0x12/0x19
> pciehp_unconfigure_device+0x76/0xe9
> pciehp_disable_slot+0x6e/0x131
> pciehp_handle_presence_or_link_change+0x7a/0x3f7
> pciehp_ist+0xbe/0x194
> irq_thread_fn+0x22/0x4d
> ? irq_thread+0x1fd/0x1fd
> irq_thread+0x17b/0x1fd
> ? irq_forced_thread_fn+0x5f/0x5f
> kthread+0x142/0x153
> ? __irq_get_irqchip_state+0x46/0x46
> ? kthread_associate_blkcg+0x71/0x71
> ret_from_fork+0x1f/0x30
>
> In this case, igb_io_error_detected detaches the network interface and requests a PCIE slot reset, however, the PCIE reset callback is not being invoked and thus the Ethernet connection breaks down.
> As the PCIE error in this case is a non-fatal one, requesting a slot reset can be avoided.
> This patch fixes the task hung issue and preserves Ethernet connection by ignoring non-fatal PCIE errors.
>
> Signed-off-by: Ying Hsu <[email protected]>
> ---
> This commit has been tested on a HP Elite Dragonfly Chromebook and a Caldigit TS3+ Thunderbolt hub. The Ethernet driver for the hub is igb. Non-fatal PCIE errors happen when users hot-plug the cables connected to the chromebook or to the external display.
>
> drivers/net/ethernet/intel/igb/igb_main.c | 5 +++++
> 1 file changed, 5 insertions(+)
>
Tested-by: Pucha Himasekhar Reddy <[email protected]> (A Contingent worker at Intel)