HI all,
I encountered an issue while doing S3, it shows below message and then
failed to enter S3
[ 106.731140] CPU 31 has 116 vectors, 85 available. Cannot disable CPU
[ 106.731551] ACPI: \_PR_.C01F: Found 2 idle states
[ 106.732610] Error taking CPU31 down: -28
[ 106.732612] Non-boot CPUs are not disabled
CPU: AMD Ryzen Threadripper PRO 3955WX 16-Cores
Kernel: v5.19-rc7
There are 5 PCI to 4 type-c ports USB cards on the machine, and It
wouldn't lead to the issue if only 4 cards are plugged. So, it looks
like it can't handle 5 cards, and failed on the IRQ migration.
The workaround provided by kaiheng is to release the irq while
suspending and request irq while resuming.
I'm wondering do we have a better solution for this kind of issue?
Thanks.
diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
index edc6881c8a1b..91c79b21cb57 100644
--- a/drivers/usb/host/xhci.c
+++ b/drivers/usb/host/xhci.c
@@ -17,6 +17,7 @@
#include <linux/slab.h>
#include <linux/dmi.h>
#include <linux/dma-mapping.h>
+#include <linux/suspend.h>
#include "xhci.h"
#include "xhci-trace.h"
@@ -1079,6 +1080,9 @@ int xhci_suspend(struct xhci_hcd *xhci, bool do_wakeup)
__func__);
}
+ if (pm_suspend_via_firmware())
+ xhci_cleanup_msix(xhci);
+
/* step 5: remove core well power */
/* synchronize irq when using MSI-X */
xhci_msix_sync_irqs(xhci);
@@ -1114,6 +1118,9 @@ int xhci_resume(struct xhci_hcd *xhci, bool hibernated)
time_before(jiffies, xhci->usb3_rhub.bus_state.next_statechange))
msleep(100);
+ if (pm_resume_via_firmware())
+ xhci_setup_msix(xhci);
+
set_bit(HCD_FLAG_HW_ACCESSIBLE, &hcd->flags);
set_bit(HCD_FLAG_HW_ACCESSIBLE, &xhci->shared_hcd->flags);
$ lspci -vt
-+-[0000:60]-+-00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse Root Complex
| +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
| +-01.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-01.1-[61-66]----00.0-[62-66]--+-08.0-[63]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | +-09.0-[64]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | +-10.0-[65]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | \-11.0-[66]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| +-02.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-03.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-03.1-[67-6a]----00.0-[68-6a]--+-08.0-[69]--+-00.0
Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
| | | +-00.1
Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| | | \-00.3
Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
| | \-0a.0-[6a]----00.0
Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
| +-03.2-[6b]----00.0 ASMedia Technology Inc. ASM3242 USB
3.2 Host Controller
| +-03.3-[6c]----00.0 Intel Corporation I210 Gigabit
Network Connection
| +-03.4-[6d-6e]----00.0-[6e]----00.0 ASPEED Technology,
Inc. ASPEED Graphics Family
| +-03.5-[6f]----00.0 Aquantia Corp. Device 14c0
| +-04.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-05.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-07.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-07.1-[70]----00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Function
| +-08.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| \-08.1-[71]----00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse Reserved SPP
+-[0000:40]-+-00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse Root Complex
| +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
| +-01.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-01.1-[41-42]--+-00.0 Advanced Micro Devices, Inc.
[AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
| | \-00.1 Advanced Micro Devices, Inc.
[AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series]
| +-02.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-03.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-03.1-[43-48]----00.0-[44-48]--+-08.0-[45]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | +-09.0-[46]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | +-10.0-[47]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | \-11.0-[48]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| +-04.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-05.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-07.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-07.1-[49]----00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Function
| +-08.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| \-08.1-[4a]----00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse Reserved SPP
+-[0000:20]-+-00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse Root Complex
| +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
| +-01.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-01.1-[21-26]----00.0-[22-26]--+-08.0-[23]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | +-09.0-[24]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | +-10.0-[25]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | \-11.0-[26]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| +-02.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-03.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-03.1-[27-2c]----00.0-[28-2c]--+-08.0-[29]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | +-09.0-[2a]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | +-10.0-[2b]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| | \-11.0-[2c]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| +-04.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-05.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-07.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| +-07.1-[2d]----00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Function
| +-08.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
| \-08.1-[2e]--+-00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse Reserved SPP
| +-00.1 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse Cryptographic Coprocessor PSPCPP
| +-00.3 Advanced Micro Devices, Inc. [AMD]
Starship USB 3.0 Host Controller
| \-00.4 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse HD Audio Controller
\-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse Root Complex
+-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
+-01.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
+-01.1-[01-06]----00.0-[02-06]--+-08.0-[03]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| +-09.0-[04]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| +-10.0-[05]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
| \-11.0-[06]----00.0
ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller
+-02.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
+-03.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
+-03.4-[07]----00.0 Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD
+-04.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
+-05.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
+-07.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
+-07.1-[08]----00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Function
+-08.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse PCIe Dummy Host Bridge
+-08.1-[09]--+-00.0 Advanced Micro Devices, Inc. [AMD]
Starship/Matisse Reserved SPP
| \-00.3 Advanced Micro Devices, Inc. [AMD]
Starship USB 3.0 Host Controller
+-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
+-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
+-18.0 Advanced Micro Devices, Inc. [AMD] Starship Device
24; Function 0
+-18.1 Advanced Micro Devices, Inc. [AMD] Starship Device
24; Function 1
+-18.2 Advanced Micro Devices, Inc. [AMD] Starship Device
24; Function 2
+-18.3 Advanced Micro Devices, Inc. [AMD] Starship Device
24; Function 3
+-18.4 Advanced Micro Devices, Inc. [AMD] Starship Device
24; Function 4
+-18.5 Advanced Micro Devices, Inc. [AMD] Starship Device
24; Function 5
+-18.6 Advanced Micro Devices, Inc. [AMD] Starship Device
24; Function 6
\-18.7 Advanced Micro Devices, Inc. [AMD] Starship Device
24; Function 7
[- Jason]
On Tue, 19 Jul 2022 06:55:21 +0100,
AceLan Kao <[email protected]> wrote:
>
> HI all,
>
> I encountered an issue while doing S3, it shows below message and then
> failed to enter S3
> [ 106.731140] CPU 31 has 116 vectors, 85 available. Cannot disable CPU
> [ 106.731551] ACPI: \_PR_.C01F: Found 2 idle states
> [ 106.732610] Error taking CPU31 down: -28
> [ 106.732612] Non-boot CPUs are not disabled
>
> CPU: AMD Ryzen Threadripper PRO 3955WX 16-Cores
> Kernel: v5.19-rc7
> There are 5 PCI to 4 type-c ports USB cards on the machine, and It
> wouldn't lead to the issue if only 4 cards are plugged. So, it looks
> like it can't handle 5 cards, and failed on the IRQ migration.
>
> The workaround provided by kaiheng is to release the irq while
> suspending and request irq while resuming.
> I'm wondering do we have a better solution for this kind of issue?
> Thanks.
>
> diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> index edc6881c8a1b..91c79b21cb57 100644
> --- a/drivers/usb/host/xhci.c
> +++ b/drivers/usb/host/xhci.c
> @@ -17,6 +17,7 @@
> #include <linux/slab.h>
> #include <linux/dmi.h>
> #include <linux/dma-mapping.h>
> +#include <linux/suspend.h>
>
> #include "xhci.h"
> #include "xhci-trace.h"
> @@ -1079,6 +1080,9 @@ int xhci_suspend(struct xhci_hcd *xhci, bool do_wakeup)
> __func__);
> }
>
> + if (pm_suspend_via_firmware())
> + xhci_cleanup_msix(xhci);
I'm a bit clueless when it comes to the combination of x86 and xhci,
but doesn't this prevent resuming on a xhci interrupt?
M.
--
Without deviation from the norm, progress is not possible.
Marc Zyngier <[email protected]> 於 2022年7月19日 週二 下午6:48寫道：
>
> [- Jason]
>
> On Tue, 19 Jul 2022 06:55:21 +0100,
> AceLan Kao <[email protected]> wrote:
> >
> > HI all,
> >
> > I encountered an issue while doing S3, it shows below message and then
> > failed to enter S3
> > [ 106.731140] CPU 31 has 116 vectors, 85 available. Cannot disable CPU
> > [ 106.731551] ACPI: \_PR_.C01F: Found 2 idle states
> > [ 106.732610] Error taking CPU31 down: -28
> > [ 106.732612] Non-boot CPUs are not disabled
> >
> > CPU: AMD Ryzen Threadripper PRO 3955WX 16-Cores
> > Kernel: v5.19-rc7
> > There are 5 PCI to 4 type-c ports USB cards on the machine, and It
> > wouldn't lead to the issue if only 4 cards are plugged. So, it looks
> > like it can't handle 5 cards, and failed on the IRQ migration.
> >
> > The workaround provided by kaiheng is to release the irq while
> > suspending and request irq while resuming.
> > I'm wondering do we have a better solution for this kind of issue?
> > Thanks.
> >
> > diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> > index edc6881c8a1b..91c79b21cb57 100644
> > --- a/drivers/usb/host/xhci.c
> > +++ b/drivers/usb/host/xhci.c
> > @@ -17,6 +17,7 @@
> > #include <linux/slab.h>
> > #include <linux/dmi.h>
> > #include <linux/dma-mapping.h>
> > +#include <linux/suspend.h>
> >
> > #include "xhci.h"
> > #include "xhci-trace.h"
> > @@ -1079,6 +1080,9 @@ int xhci_suspend(struct xhci_hcd *xhci, bool do_wakeup)
> > __func__);
> > }
> >
> > + if (pm_suspend_via_firmware())
> > + xhci_cleanup_msix(xhci);
>
> I'm a bit clueless when it comes to the combination of x86 and xhci,
> but doesn't this prevent resuming on a xhci interrupt?
The PCI cards provide 4 type-c USB ports, and in the beginning we
found that removing one PCI card fixed the issue, so we were trying to
fix the issue in xhci driver.
The USB ports on the PCI cards can't resume the system from S3 even
without the workaround,
but the USB ports on the rear panel of the motherboard still work with
the workaround.
[+Cc Rafael, linux-pm]
On Wed, Jul 20, 2022 at 10:53 AM AceLan Kao <[email protected]> wrote:
>
> Marc Zyngier <[email protected]> 於 2022年7月19日 週二 下午6:48寫道：
> >
> > [- Jason]
> >
> > On Tue, 19 Jul 2022 06:55:21 +0100,
> > AceLan Kao <[email protected]> wrote:
> > >
> > > HI all,
> > >
> > > I encountered an issue while doing S3, it shows below message and then
> > > failed to enter S3
> > > [ 106.731140] CPU 31 has 116 vectors, 85 available. Cannot disable CPU
> > > [ 106.731551] ACPI: \_PR_.C01F: Found 2 idle states
> > > [ 106.732610] Error taking CPU31 down: -28
> > > [ 106.732612] Non-boot CPUs are not disabled
> > >
> > > CPU: AMD Ryzen Threadripper PRO 3955WX 16-Cores
> > > Kernel: v5.19-rc7
> > > There are 5 PCI to 4 type-c ports USB cards on the machine, and It
> > > wouldn't lead to the issue if only 4 cards are plugged. So, it looks
> > > like it can't handle 5 cards, and failed on the IRQ migration.
> > >
> > > The workaround provided by kaiheng is to release the irq while
> > > suspending and request irq while resuming.
> > > I'm wondering do we have a better solution for this kind of issue?
> > > Thanks.
> > >
> > > diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> > > index edc6881c8a1b..91c79b21cb57 100644
> > > --- a/drivers/usb/host/xhci.c
> > > +++ b/drivers/usb/host/xhci.c
> > > @@ -17,6 +17,7 @@
> > > #include <linux/slab.h>
> > > #include <linux/dmi.h>
> > > #include <linux/dma-mapping.h>
> > > +#include <linux/suspend.h>
> > >
> > > #include "xhci.h"
> > > #include "xhci-trace.h"
> > > @@ -1079,6 +1080,9 @@ int xhci_suspend(struct xhci_hcd *xhci, bool do_wakeup)
> > > __func__);
> > > }
> > >
> > > + if (pm_suspend_via_firmware())
> > > + xhci_cleanup_msix(xhci);
> >
> > I'm a bit clueless when it comes to the combination of x86 and xhci,
> > but doesn't this prevent resuming on a xhci interrupt?
> The PCI cards provide 4 type-c USB ports, and in the beginning we
> found that removing one PCI card fixed the issue, so we were trying to
> fix the issue in xhci driver.
> The USB ports on the PCI cards can't resume the system from S3 even
> without the workaround,
> but the USB ports on the rear panel of the motherboard still work with
> the workaround.
The isn't xHCI specific. The issue here is that CPU0 APIC doesn't have
enough IRQ vector for ACPI S3 suspend.
Ideally we don't want to tear down IRQs during suspend, but for this
case minimizing IRQ numbers means successful S3.
So maybe we can have a suspend flow like this:
- At the beginning of suspend, check if there's enough free IRQ for
CPU0 migration.
- If there isn't enough free slots, hint drivers to tear down non-wake
IRQs. Maybe use a global variable if we don't want to add a new
parameter to current PM ops.
- If it's still not enough, abort suspend.
For suspend that doesn't unplug CPU like suspend-to-idle, no
modification is needed.
I wonder if that makes sense?
Kai-Heng
On Wed, Jul 20, 2022 at 5:16 AM Kai-Heng Feng
<[email protected]> wrote:
>
> [+Cc Rafael, linux-pm]
>
> On Wed, Jul 20, 2022 at 10:53 AM AceLan Kao <[email protected]> wrote:
> >
> > Marc Zyngier <[email protected]> 於 2022年7月19日 週二 下午6:48寫道：
> > >
> > > [- Jason]
> > >
> > > On Tue, 19 Jul 2022 06:55:21 +0100,
> > > AceLan Kao <[email protected]> wrote:
> > > >
> > > > HI all,
> > > >
> > > > I encountered an issue while doing S3, it shows below message and then
> > > > failed to enter S3
> > > > [ 106.731140] CPU 31 has 116 vectors, 85 available. Cannot disable CPU
> > > > [ 106.731551] ACPI: \_PR_.C01F: Found 2 idle states
> > > > [ 106.732610] Error taking CPU31 down: -28
> > > > [ 106.732612] Non-boot CPUs are not disabled
> > > >
> > > > CPU: AMD Ryzen Threadripper PRO 3955WX 16-Cores
> > > > Kernel: v5.19-rc7
> > > > There are 5 PCI to 4 type-c ports USB cards on the machine, and It
> > > > wouldn't lead to the issue if only 4 cards are plugged. So, it looks
> > > > like it can't handle 5 cards, and failed on the IRQ migration.
> > > >
> > > > The workaround provided by kaiheng is to release the irq while
> > > > suspending and request irq while resuming.
> > > > I'm wondering do we have a better solution for this kind of issue?
> > > > Thanks.
> > > >
> > > > diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> > > > index edc6881c8a1b..91c79b21cb57 100644
> > > > --- a/drivers/usb/host/xhci.c
> > > > +++ b/drivers/usb/host/xhci.c
> > > > @@ -17,6 +17,7 @@
> > > > #include <linux/slab.h>
> > > > #include <linux/dmi.h>
> > > > #include <linux/dma-mapping.h>
> > > > +#include <linux/suspend.h>
> > > >
> > > > #include "xhci.h"
> > > > #include "xhci-trace.h"
> > > > @@ -1079,6 +1080,9 @@ int xhci_suspend(struct xhci_hcd *xhci, bool do_wakeup)
> > > > __func__);
> > > > }
> > > >
> > > > + if (pm_suspend_via_firmware())
> > > > + xhci_cleanup_msix(xhci);
> > >
> > > I'm a bit clueless when it comes to the combination of x86 and xhci,
> > > but doesn't this prevent resuming on a xhci interrupt?
> > The PCI cards provide 4 type-c USB ports, and in the beginning we
> > found that removing one PCI card fixed the issue, so we were trying to
> > fix the issue in xhci driver.
> > The USB ports on the PCI cards can't resume the system from S3 even
> > without the workaround,
> > but the USB ports on the rear panel of the motherboard still work with
> > the workaround.
>
> The isn't xHCI specific. The issue here is that CPU0 APIC doesn't have
> enough IRQ vector for ACPI S3 suspend.
> Ideally we don't want to tear down IRQs during suspend, but for this
> case minimizing IRQ numbers means successful S3.
>
> So maybe we can have a suspend flow like this:
> - At the beginning of suspend, check if there's enough free IRQ for
> CPU0 migration.
> - If there isn't enough free slots, hint drivers to tear down non-wake
> IRQs. Maybe use a global variable if we don't want to add a new
> parameter to current PM ops.
> - If it's still not enough, abort suspend.
>
> For suspend that doesn't unplug CPU like suspend-to-idle, no
> modification is needed.
> I wonder if that makes sense?
Quite probably, IRQs need not be migrated during system suspend, so it
should be possible to avoid doing that entirely on "hot remove" if it
is part of the suspend flow.
On Wed, Jul 20, 2022 at 6:11 PM Rafael J. Wysocki <[email protected]> wrote:
>
> On Wed, Jul 20, 2022 at 5:16 AM Kai-Heng Feng
> <[email protected]> wrote:
> >
> > [+Cc Rafael, linux-pm]
> >
> > On Wed, Jul 20, 2022 at 10:53 AM AceLan Kao <[email protected]> wrote:
> > >
> > > Marc Zyngier <[email protected]> 於 2022年7月19日 週二 下午6:48寫道：
> > > >
> > > > [- Jason]
> > > >
> > > > On Tue, 19 Jul 2022 06:55:21 +0100,
> > > > AceLan Kao <[email protected]> wrote:
> > > > >
> > > > > HI all,
> > > > >
> > > > > I encountered an issue while doing S3, it shows below message and then
> > > > > failed to enter S3
> > > > > [ 106.731140] CPU 31 has 116 vectors, 85 available. Cannot disable CPU
> > > > > [ 106.731551] ACPI: \_PR_.C01F: Found 2 idle states
> > > > > [ 106.732610] Error taking CPU31 down: -28
> > > > > [ 106.732612] Non-boot CPUs are not disabled
> > > > >
> > > > > CPU: AMD Ryzen Threadripper PRO 3955WX 16-Cores
> > > > > Kernel: v5.19-rc7
> > > > > There are 5 PCI to 4 type-c ports USB cards on the machine, and It
> > > > > wouldn't lead to the issue if only 4 cards are plugged. So, it looks
> > > > > like it can't handle 5 cards, and failed on the IRQ migration.
> > > > >
> > > > > The workaround provided by kaiheng is to release the irq while
> > > > > suspending and request irq while resuming.
> > > > > I'm wondering do we have a better solution for this kind of issue?
> > > > > Thanks.
> > > > >
> > > > > diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> > > > > index edc6881c8a1b..91c79b21cb57 100644
> > > > > --- a/drivers/usb/host/xhci.c
> > > > > +++ b/drivers/usb/host/xhci.c
> > > > > @@ -17,6 +17,7 @@
> > > > > #include <linux/slab.h>
> > > > > #include <linux/dmi.h>
> > > > > #include <linux/dma-mapping.h>
> > > > > +#include <linux/suspend.h>
> > > > >
> > > > > #include "xhci.h"
> > > > > #include "xhci-trace.h"
> > > > > @@ -1079,6 +1080,9 @@ int xhci_suspend(struct xhci_hcd *xhci, bool do_wakeup)
> > > > > __func__);
> > > > > }
> > > > >
> > > > > + if (pm_suspend_via_firmware())
> > > > > + xhci_cleanup_msix(xhci);
> > > >
> > > > I'm a bit clueless when it comes to the combination of x86 and xhci,
> > > > but doesn't this prevent resuming on a xhci interrupt?
> > > The PCI cards provide 4 type-c USB ports, and in the beginning we
> > > found that removing one PCI card fixed the issue, so we were trying to
> > > fix the issue in xhci driver.
> > > The USB ports on the PCI cards can't resume the system from S3 even
> > > without the workaround,
> > > but the USB ports on the rear panel of the motherboard still work with
> > > the workaround.
> >
> > The isn't xHCI specific. The issue here is that CPU0 APIC doesn't have
> > enough IRQ vector for ACPI S3 suspend.
> > Ideally we don't want to tear down IRQs during suspend, but for this
> > case minimizing IRQ numbers means successful S3.
> >
> > So maybe we can have a suspend flow like this:
> > - At the beginning of suspend, check if there's enough free IRQ for
> > CPU0 migration.
> > - If there isn't enough free slots, hint drivers to tear down non-wake
> > IRQs. Maybe use a global variable if we don't want to add a new
> > parameter to current PM ops.
> > - If it's still not enough, abort suspend.
> >
> > For suspend that doesn't unplug CPU like suspend-to-idle, no
> > modification is needed.
> > I wonder if that makes sense?
>
> Quite probably, IRQs need not be migrated during system suspend, so it
> should be possible to avoid doing that entirely on "hot remove" if it
> is part of the suspend flow.
So is it required to disable all CPUs except CPU0 for ACPI S3?
ACPI spec rev 6.3, "7.4.2.4 System \_S3 State" says "The processors
are not executing instructions. The processor-complex context is not
maintained." it doesn't say the CPUs have to be disabled.
So as long as CPUs are quiesced, is pm_sleep_disable_secondary_cpus()
still needed for S3?
Kai-Heng
Kai-Heng Feng <[email protected]> 於 2022年7月21日 週四 上午11:02寫道：
>
> On Wed, Jul 20, 2022 at 6:11 PM Rafael J. Wysocki <[email protected]> wrote:
> >
> > On Wed, Jul 20, 2022 at 5:16 AM Kai-Heng Feng
> > <[email protected]> wrote:
> > >
> > > [+Cc Rafael, linux-pm]
> > >
> > > On Wed, Jul 20, 2022 at 10:53 AM AceLan Kao <[email protected]> wrote:
> > > >
> > > > Marc Zyngier <[email protected]> 於 2022年7月19日 週二 下午6:48寫道：
> > > > >
> > > > > [- Jason]
> > > > >
> > > > > On Tue, 19 Jul 2022 06:55:21 +0100,
> > > > > AceLan Kao <[email protected]> wrote:
> > > > > >
> > > > > > HI all,
> > > > > >
> > > > > > I encountered an issue while doing S3, it shows below message and then
> > > > > > failed to enter S3
> > > > > > [ 106.731140] CPU 31 has 116 vectors, 85 available. Cannot disable CPU
> > > > > > [ 106.731551] ACPI: \_PR_.C01F: Found 2 idle states
> > > > > > [ 106.732610] Error taking CPU31 down: -28
> > > > > > [ 106.732612] Non-boot CPUs are not disabled
> > > > > >
> > > > > > CPU: AMD Ryzen Threadripper PRO 3955WX 16-Cores
> > > > > > Kernel: v5.19-rc7
> > > > > > There are 5 PCI to 4 type-c ports USB cards on the machine, and It
> > > > > > wouldn't lead to the issue if only 4 cards are plugged. So, it looks
> > > > > > like it can't handle 5 cards, and failed on the IRQ migration.
> > > > > >
> > > > > > The workaround provided by kaiheng is to release the irq while
> > > > > > suspending and request irq while resuming.
> > > > > > I'm wondering do we have a better solution for this kind of issue?
> > > > > > Thanks.
> > > > > >
> > > > > > diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
> > > > > > index edc6881c8a1b..91c79b21cb57 100644
> > > > > > --- a/drivers/usb/host/xhci.c
> > > > > > +++ b/drivers/usb/host/xhci.c
> > > > > > @@ -17,6 +17,7 @@
> > > > > > #include <linux/slab.h>
> > > > > > #include <linux/dmi.h>
> > > > > > #include <linux/dma-mapping.h>
> > > > > > +#include <linux/suspend.h>
> > > > > >
> > > > > > #include "xhci.h"
> > > > > > #include "xhci-trace.h"
> > > > > > @@ -1079,6 +1080,9 @@ int xhci_suspend(struct xhci_hcd *xhci, bool do_wakeup)
> > > > > > __func__);
> > > > > > }
> > > > > >
> > > > > > + if (pm_suspend_via_firmware())
> > > > > > + xhci_cleanup_msix(xhci);
> > > > >
> > > > > I'm a bit clueless when it comes to the combination of x86 and xhci,
> > > > > but doesn't this prevent resuming on a xhci interrupt?
> > > > The PCI cards provide 4 type-c USB ports, and in the beginning we
> > > > found that removing one PCI card fixed the issue, so we were trying to
> > > > fix the issue in xhci driver.
> > > > The USB ports on the PCI cards can't resume the system from S3 even
> > > > without the workaround,
> > > > but the USB ports on the rear panel of the motherboard still work with
> > > > the workaround.
> > >
> > > The isn't xHCI specific. The issue here is that CPU0 APIC doesn't have
> > > enough IRQ vector for ACPI S3 suspend.
> > > Ideally we don't want to tear down IRQs during suspend, but for this
> > > case minimizing IRQ numbers means successful S3.
> > >
> > > So maybe we can have a suspend flow like this:
> > > - At the beginning of suspend, check if there's enough free IRQ for
> > > CPU0 migration.
> > > - If there isn't enough free slots, hint drivers to tear down non-wake
> > > IRQs. Maybe use a global variable if we don't want to add a new
> > > parameter to current PM ops.
> > > - If it's still not enough, abort suspend.
> > >
> > > For suspend that doesn't unplug CPU like suspend-to-idle, no
> > > modification is needed.
> > > I wonder if that makes sense?
> >
> > Quite probably, IRQs need not be migrated during system suspend, so it
> > should be possible to avoid doing that entirely on "hot remove" if it
> > is part of the suspend flow.
>
> So is it required to disable all CPUs except CPU0 for ACPI S3?
> ACPI spec rev 6.3, "7.4.2.4 System \_S3 State" says "The processors
> are not executing instructions. The processor-complex context is not
> maintained." it doesn't say the CPUs have to be disabled.
> So as long as CPUs are quiesced, is pm_sleep_disable_secondary_cpus()
> still needed for S3?
>
I'm still doing some tests for this issue and tried to not migrate
IRQs or do not offline CPUs while suspending,
but I can't find a good path to do it well, and the system may fail to
enter suspended or fail to be waken up.
I also checked the available vectors when the CPU is going offline,
and found the available vector reduces by around 200 when one CPU is
offline.
So that makes the available vectors are not sufficient at the end,
I'm wondering if the calculation for the deducted vectors is reasonable?
Or could we allocate more vectors in the beginning?
u@u-M12SWA-TF:~$ sudo cat /sys/kernel/debug/irq/domains/VECTOR
name: VECTOR
size: 0
mapped: 371
flags: 0x00000003
Online bitmaps: 32
Global available: 6150
Global reserved: 57
Total allocated: 282
System: 39: 0-19,29,32,50,128,236,240-242,244,246-255
| CPU | avl | man | mac | act | vectors
0 192 1 1 9 33-40,48
1 191 1 1 10 33-42
2 192 1 1 9 33-41
3 192 1 1 9 33-36,38-42
4 192 1 1 9 33-36,38-42
5 192 1 1 9 33-40,43
6 193 1 1 8 33-36,38-39,41-42
7 191 1 1 10 33-42
8 192 1 1 9 33-38,40-42
9 192 1 1 9 33-41
10 192 1 1 9 33-41
11 192 1 1 9 33-41
12 192 1 1 9 33-41
13 194 1 1 7 33-36,38,40-41
14 193 1 1 8 33-36,38-41
15 193 1 1 8 33-38,40-41
16 192 1 1 9 33-41
17 194 1 1 7 33-39
18 192 1 1 9 33-41
19 191 1 1 10 33-42
20 193 1 1 8 33-38,40,42
21 192 1 1 9 33-38,40-42
22 192 1 1 9 33-40,42
23 192 1 1 9 33-41
24 191 1 1 10 33-37,39-43
25 192 1 1 9 33-41
26 192 1 1 9 33-40,43
27 193 1 1 8 33-40
28 193 1 1 8 33-39,42
29 192 1 1 9 33-41
30 192 1 1 9 33-41
31 192 1 1 9 33-41
u@u-M12SWA-TF:~$ sudo chcpu -d 1
CPU 1 disabled
u@u-M12SWA-TF:~$ sudo cat /sys/kernel/debug/irq/domains/VECTOR
name: VECTOR
size: 0
mapped: 371
flags: 0x00000003
Online bitmaps: 31
Global available: 5950
Global reserved: 57
Total allocated: 281
System: 39: 0-19,29,32,50,128,236,240-242,244,246-255
| CPU | avl | man | mac | act | vectors
0 192 1 1 9 33-40,48
2 192 1 1 9 33-41
3 192 1 1 9 33-36,38-42
4 192 1 1 9 33-36,38-42
5 192 1 1 9 33-40,43
6 193 1 1 8 33-36,38-39,41-42
7 191 1 1 10 33-42
8 192 1 1 9 33-38,40-42
9 192 1 1 9 33-41
10 192 1 1 9 33-41
11 192 1 1 9 33-41
12 192 1 1 9 33-41
13 193 1 1 8 33-38,40-41
14 193 1 1 8 33-36,38-41
15 193 1 1 8 33-38,40-41
16 192 1 1 9 33-41
17 186 1 1 15 33-47
18 192 1 1 9 33-41
19 191 1 1 10 33-42
20 193 1 1 8 33-38,40,42
21 192 1 1 9 33-38,40-42
22 192 1 1 9 33-40,42
23 192 1 1 9 33-41
24 191 1 1 10 33-37,39-43
25 192 1 1 9 33-41
26 192 1 1 9 33-40,43
27 193 1 1 8 33-40
28 193 1 1 8 33-39,42
29 192 1 1 9 33-41
30 192 1 1 9 33-41
31 192 1 1 9 33-41
u@u-M12SWA-TF:~$ sudo chcpu -d 2
CPU 2 disabled
u@u-M12SWA-TF:~$ sudo cat /sys/kernel/debug/irq/domains/VECTOR
name: VECTOR
size: 0
mapped: 371
flags: 0x00000003
Online bitmaps: 30
Global available: 5750
Global reserved: 57
Total allocated: 280
System: 39: 0-19,29,32,50,128,236,240-242,244,246-255
| CPU | avl | man | mac | act | vectors
0 192 1 1 9 33-40,48
3 192 1 1 9 33-36,38-42
4 192 1 1 9 33-36,38-42
5 192 1 1 9 33-40,43
6 193 1 1 8 33-36,38-39,41-42
7 191 1 1 10 33-42
8 192 1 1 9 33-38,40-42
9 192 1 1 9 33-41
10 192 1 1 9 33-41
11 192 1 1 9 33-41
12 192 1 1 9 33-41
13 193 1 1 8 33-38,40-41
14 193 1 1 8 33-36,38-41
15 193 1 1 8 33-38,40-41
16 192 1 1 9 33-41
17 186 1 1 15 33-47
18 184 1 1 17 33-49
19 191 1 1 10 33-42
20 193 1 1 8 33-38,40,42
21 192 1 1 9 33-38,40-42
22 192 1 1 9 33-40,42
23 192 1 1 9 33-41
24 191 1 1 10 33-37,39-43
25 192 1 1 9 33-41
26 192 1 1 9 33-40,43
27 193 1 1 8 33-40
28 193 1 1 8 33-39,42
29 192 1 1 9 33-41
30 192 1 1 9 33-41
31 192 1 1 9 33-41