The documentation for the freeze() method says that it "should quiesce
the device so that it doesn't generate IRQs or DMA". The unspoken
consequence of not doing this is that MSIs aimed at non-boot CPUs may
get fully lost if they're sent during the period where the target CPU is
offline.
The current behavior of the USB subsystem still allows interrupts to
come in after freeze, both in terms of remote wakeups and HC events
related to things like root plug port activity. This can get controllers
like XHCI, which is very sensitive to lost interrupts, in a wedged
state. This series attempts to fully quiesce interrupts coming from USB
across in a freeze or quiescent state.
These patches are grouped together because they serve a united purpose,
but are actually independent. They could be merged or reverted
individually.
You may be able to reproduce this issue on your own machine via the
following:
1. Disable runtime PM on your XHCI controller
2. Aim your XHCI IRQ at a non-boot CPU (replace 174): echo 2 >
/proc/irq/174/smp_affinity
3. Attempt to hibernate (no need to actually go all the way down).
I run 2 and 3 in a loop, and can usually hit a hang or dead XHCI
controller within 1-2 iterations. I happened to notice this on an
Alderlake system where runtime PM is accidentally disabled for one of
the XHCI controllers. Some more discussion and debugging can be found at
[1].
[1] https://lore.kernel.org/linux-pci/CAE=gft4a-QL82iFJE_xRQ3JrMmz-KZKWREtz=MghhjFbJeK=8A@mail.gmail.com/T/#u
Changes in v2:
- Added the patch modifying the remote wakeup state
- Removed the change to freeze_noirq/thaw_noirq
Evan Green (2):
USB: core: Disable remote wakeup for freeze/quiesce
USB: hcd-pci: Fully suspend across freeze/thaw cycle
drivers/usb/core/driver.c | 20 +++++++++-----------
drivers/usb/core/hcd-pci.c | 4 ++--
2 files changed, 11 insertions(+), 13 deletions(-)
--
2.31.0
The documentation for the freeze() method says that it "should quiesce
the device so that it doesn't generate IRQs or DMA". The unspoken
consequence of not doing this is that MSIs aimed at non-boot CPUs may
get fully lost if they're sent during the period where the target CPU is
offline.
The current callbacks for USB HCD do not fully quiesce interrupts,
specifically on XHCI. Change to use the full suspend/resume flow for
freeze/thaw to ensure interrupts are fully quiesced. This fixes issues
where USB devices fail to thaw during hibernation because XHCI misses
its interrupt and cannot recover.
Signed-off-by: Evan Green <[email protected]>
---
Changes in v2:
- Added the patch modifying the remote wakeup state
- Removed the change to freeze_noirq/thaw_noirq
drivers/usb/core/hcd-pci.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/usb/core/hcd-pci.c b/drivers/usb/core/hcd-pci.c
index 8176bc81a635d6..ae5e6d572376be 100644
--- a/drivers/usb/core/hcd-pci.c
+++ b/drivers/usb/core/hcd-pci.c
@@ -616,10 +616,10 @@ const struct dev_pm_ops usb_hcd_pci_pm_ops = {
.suspend_noirq = hcd_pci_suspend_noirq,
.resume_noirq = hcd_pci_resume_noirq,
.resume = hcd_pci_resume,
- .freeze = check_root_hub_suspended,
+ .freeze = hcd_pci_suspend,
.freeze_noirq = check_root_hub_suspended,
.thaw_noirq = NULL,
- .thaw = NULL,
+ .thaw = hcd_pci_resume,
.poweroff = hcd_pci_suspend,
.poweroff_noirq = hcd_pci_suspend_noirq,
.restore_noirq = hcd_pci_resume_noirq,
--
2.31.0
The PM_EVENT_FREEZE and PM_EVENT_QUIESCE messages should cause the
device to stop generating interrupts. USB core was previously allowing
devices that were already runtime suspended to keep remote wakeup
enabled if they had gone down that way. This violates the contract with
pm, and can potentially cause MSI interrupts to be lost.
Change that so that if a device is runtime suspended with remote wakeups
enabled, it will be resumed to ensure remote wakeup is always disabled
across a freeze.
Signed-off-by: Evan Green <[email protected]>
---
(no changes since v1)
drivers/usb/core/driver.c | 20 +++++++++-----------
1 file changed, 9 insertions(+), 11 deletions(-)
diff --git a/drivers/usb/core/driver.c b/drivers/usb/core/driver.c
index 355ed33a21792b..93c8cf66adccec 100644
--- a/drivers/usb/core/driver.c
+++ b/drivers/usb/core/driver.c
@@ -1533,20 +1533,18 @@ static void choose_wakeup(struct usb_device *udev, pm_message_t msg)
{
int w;
- /* Remote wakeup is needed only when we actually go to sleep.
- * For things like FREEZE and QUIESCE, if the device is already
- * autosuspended then its current wakeup setting is okay.
+ /* For FREEZE/QUIESCE, disable remote wakeups so no interrupts get generated
+ * by the device.
*/
if (msg.event == PM_EVENT_FREEZE || msg.event == PM_EVENT_QUIESCE) {
- if (udev->state != USB_STATE_SUSPENDED)
- udev->do_remote_wakeup = 0;
- return;
- }
+ w = 0;
- /* Enable remote wakeup if it is allowed, even if no interface drivers
- * actually want it.
- */
- w = device_may_wakeup(&udev->dev);
+ } else {
+ /* Enable remote wakeup if it is allowed, even if no interface drivers
+ * actually want it.
+ */
+ w = device_may_wakeup(&udev->dev);
+ }
/* If the device is autosuspended with the wrong wakeup setting,
* autoresume now so the setting can be changed.
--
2.31.0
On 18.04.22 23:00, Evan Green wrote:
> The documentation for the freeze() method says that it "should quiesce
> the device so that it doesn't generate IRQs or DMA". The unspoken
> consequence of not doing this is that MSIs aimed at non-boot CPUs may
> get fully lost if they're sent during the period where the target CPU is
> offline.
>
> The current behavior of the USB subsystem still allows interrupts to
> come in after freeze, both in terms of remote wakeups and HC events
> related to things like root plug port activity. This can get controllers
> like XHCI, which is very sensitive to lost interrupts, in a wedged
> state. This series attempts to fully quiesce interrupts coming from USB
> across in a freeze or quiescent state.
>
> These patches are grouped together because they serve a united purpose,
> but are actually independent. They could be merged or reverted
> individually.
Hi,
sorry for being a bit late in this discussion. There was something that
I didn't remember immediately.
We have a set of quirky devices that need HID_QUIRK_ALWAYS_POLL.
They have the nasty firmware bug that, if you suspend them without
remote wakeup, they will crash or reset themselves.
I am afraid that has an obvious relevance to your cool patches.
I am not completely sure how to deal with this. It seems to me that the
quirk will need to be shifted from HID to core USB and thaw() needs to
be translated into usb_device_reset() + reset_resume() for them,
but I am not really sure about the optimal mechanism.
Regards
Oliver
On Tue, Apr 19, 2022 at 7:41 AM Alan Stern <[email protected]> wrote:
>
> On Mon, Apr 18, 2022 at 02:00:45PM -0700, Evan Green wrote:
> > The PM_EVENT_FREEZE and PM_EVENT_QUIESCE messages should cause the
> > device to stop generating interrupts. USB core was previously allowing
> > devices that were already runtime suspended to keep remote wakeup
> > enabled if they had gone down that way. This violates the contract with
> > pm, and can potentially cause MSI interrupts to be lost.
> >
> > Change that so that if a device is runtime suspended with remote wakeups
> > enabled, it will be resumed to ensure remote wakeup is always disabled
> > across a freeze.
> >
> > Signed-off-by: Evan Green <[email protected]>
> > ---
> >
> > (no changes since v1)
> >
> > drivers/usb/core/driver.c | 20 +++++++++-----------
> > 1 file changed, 9 insertions(+), 11 deletions(-)
> >
> > diff --git a/drivers/usb/core/driver.c b/drivers/usb/core/driver.c
> > index 355ed33a21792b..93c8cf66adccec 100644
> > --- a/drivers/usb/core/driver.c
> > +++ b/drivers/usb/core/driver.c
> > @@ -1533,20 +1533,18 @@ static void choose_wakeup(struct usb_device *udev, pm_message_t msg)
> > {
> > int w;
> >
> > - /* Remote wakeup is needed only when we actually go to sleep.
> > - * For things like FREEZE and QUIESCE, if the device is already
> > - * autosuspended then its current wakeup setting is okay.
> > + /* For FREEZE/QUIESCE, disable remote wakeups so no interrupts get generated
> > + * by the device.
>
> You mean "by the host controller". USB devices don't generate
> interrupts; they generate wakeup requests (which can cause a host
> controller to generate an interrupt).
Right, I guess I mean "at the behest of the device". I could probably
just delete "by the device", since the goal of the comment is simply
to highlight that we're trying to kill interrupts, and describing
their provenance isn't as relevant.
>
> > */
> > if (msg.event == PM_EVENT_FREEZE || msg.event == PM_EVENT_QUIESCE) {
> > - if (udev->state != USB_STATE_SUSPENDED)
> > - udev->do_remote_wakeup = 0;
> > - return;
> > - }
> > + w = 0;
> >
> > - /* Enable remote wakeup if it is allowed, even if no interface drivers
> > - * actually want it.
> > - */
> > - w = device_may_wakeup(&udev->dev);
> > + } else {
> > + /* Enable remote wakeup if it is allowed, even if no interface drivers
> > + * actually want it.
> > + */
> > + w = device_may_wakeup(&udev->dev);
> > + }
> >
> > /* If the device is autosuspended with the wrong wakeup setting,
> > * autoresume now so the setting can be changed.
> > --
>
> I would prefer it if you reformatted the comments to agree with the
> current style:
>
> /*
> * Blah blah blah
> */
>
> and to avoid line wrap beyond 80 columns. Apart from that:
Sure, I can fix these up, add your tags, and repost.
>
> Acked-by: Alan Stern <[email protected]>
Thanks!
>
> Alan Stern
On Mon, Apr 18, 2022 at 02:00:45PM -0700, Evan Green wrote:
> The PM_EVENT_FREEZE and PM_EVENT_QUIESCE messages should cause the
> device to stop generating interrupts. USB core was previously allowing
> devices that were already runtime suspended to keep remote wakeup
> enabled if they had gone down that way. This violates the contract with
> pm, and can potentially cause MSI interrupts to be lost.
>
> Change that so that if a device is runtime suspended with remote wakeups
> enabled, it will be resumed to ensure remote wakeup is always disabled
> across a freeze.
>
> Signed-off-by: Evan Green <[email protected]>
> ---
>
> (no changes since v1)
>
> drivers/usb/core/driver.c | 20 +++++++++-----------
> 1 file changed, 9 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/usb/core/driver.c b/drivers/usb/core/driver.c
> index 355ed33a21792b..93c8cf66adccec 100644
> --- a/drivers/usb/core/driver.c
> +++ b/drivers/usb/core/driver.c
> @@ -1533,20 +1533,18 @@ static void choose_wakeup(struct usb_device *udev, pm_message_t msg)
> {
> int w;
>
> - /* Remote wakeup is needed only when we actually go to sleep.
> - * For things like FREEZE and QUIESCE, if the device is already
> - * autosuspended then its current wakeup setting is okay.
> + /* For FREEZE/QUIESCE, disable remote wakeups so no interrupts get generated
> + * by the device.
You mean "by the host controller". USB devices don't generate
interrupts; they generate wakeup requests (which can cause a host
controller to generate an interrupt).
> */
> if (msg.event == PM_EVENT_FREEZE || msg.event == PM_EVENT_QUIESCE) {
> - if (udev->state != USB_STATE_SUSPENDED)
> - udev->do_remote_wakeup = 0;
> - return;
> - }
> + w = 0;
>
> - /* Enable remote wakeup if it is allowed, even if no interface drivers
> - * actually want it.
> - */
> - w = device_may_wakeup(&udev->dev);
> + } else {
> + /* Enable remote wakeup if it is allowed, even if no interface drivers
> + * actually want it.
> + */
> + w = device_may_wakeup(&udev->dev);
> + }
>
> /* If the device is autosuspended with the wrong wakeup setting,
> * autoresume now so the setting can be changed.
> --
I would prefer it if you reformatted the comments to agree with the
current style:
/*
* Blah blah blah
*/
and to avoid line wrap beyond 80 columns. Apart from that:
Acked-by: Alan Stern <[email protected]>
Alan Stern
On Tue, Apr 19, 2022 at 09:05:53AM +0200, Oliver Neukum wrote:
>
>
> On 18.04.22 23:00, Evan Green wrote:
> > The documentation for the freeze() method says that it "should quiesce
> > the device so that it doesn't generate IRQs or DMA". The unspoken
> > consequence of not doing this is that MSIs aimed at non-boot CPUs may
> > get fully lost if they're sent during the period where the target CPU is
> > offline.
> >
> > The current behavior of the USB subsystem still allows interrupts to
> > come in after freeze, both in terms of remote wakeups and HC events
> > related to things like root plug port activity. This can get controllers
> > like XHCI, which is very sensitive to lost interrupts, in a wedged
> > state. This series attempts to fully quiesce interrupts coming from USB
> > across in a freeze or quiescent state.
> >
> > These patches are grouped together because they serve a united purpose,
> > but are actually independent. They could be merged or reverted
> > individually.
> Hi,
>
> sorry for being a bit late in this discussion. There was something that
> I didn't remember immediately.
>
> We have a set of quirky devices that need HID_QUIRK_ALWAYS_POLL.
> They have the nasty firmware bug that, if you suspend them without
> remote wakeup, they will crash or reset themselves.
> I am afraid that has an obvious relevance to your cool patches.
> I am not completely sure how to deal with this. It seems to me that the
> quirk will need to be shifted from HID to core USB and thaw() needs to
> be translated into usb_device_reset() + reset_resume() for them,
> but I am not really sure about the optimal mechanism.
We may not need to do anything. This patch specifically addresses
hibernation, not system suspend or runtime suspend. A device crashing
or resetting during hibernation is not at all unusual; we should be able
to handle such cases properly.
The THAW part of suspend-to-hibernation is used only for writing the
memory image to permanent storage. I doubt that a malfunctioning HID
device would interfere with this process.
Alan Stern
On 19.04.22 16:35, Alan Stern wrote:
> On Tue, Apr 19, 2022 at 09:05:53AM +0200, Oliver Neukum wrote:
>
>
> We have a set of quirky devices that need HID_QUIRK_ALWAYS_POLL.
> They have the nasty firmware bug that, if you suspend them without
> remote wakeup, they will crash or reset themselves.
> I am afraid that has an obvious relevance to your cool patches.
> I am not completely sure how to deal with this. It seems to me that the
> quirk will need to be shifted from HID to core USB and thaw() needs to
> be translated into usb_device_reset() + reset_resume() for them,
> but I am not really sure about the optimal mechanism.
> We may not need to do anything. This patch specifically addresses
> hibernation, not system suspend or runtime suspend. A device crashing
> or resetting during hibernation is not at all unusual; we should be able
> to handle such cases properly.
>
> The THAW part of suspend-to-hibernation is used only for writing the
> memory image to permanent storage. I doubt that a malfunctioning HID
> device would interfere with this process.
>
True, if and only if all goes well. At the time thaw() has run writing
the image to disk can still fail. In that case the devices will still
be needed.
Regards
Oliver
On Tue, Apr 19, 2022 at 05:51:38PM +0200, Oliver Neukum wrote:
>
>
> On 19.04.22 16:35, Alan Stern wrote:
> > On Tue, Apr 19, 2022 at 09:05:53AM +0200, Oliver Neukum wrote:
> >
> >
> > We have a set of quirky devices that need HID_QUIRK_ALWAYS_POLL.
> > They have the nasty firmware bug that, if you suspend them without
> > remote wakeup, they will crash or reset themselves.
> > I am afraid that has an obvious relevance to your cool patches.
> > I am not completely sure how to deal with this. It seems to me that the
> > quirk will need to be shifted from HID to core USB and thaw() needs to
> > be translated into usb_device_reset() + reset_resume() for them,
> > but I am not really sure about the optimal mechanism.
> > We may not need to do anything. This patch specifically addresses
> > hibernation, not system suspend or runtime suspend. A device crashing
> > or resetting during hibernation is not at all unusual; we should be able
> > to handle such cases properly.
> >
> > The THAW part of suspend-to-hibernation is used only for writing the
> > memory image to permanent storage. I doubt that a malfunctioning HID
> > device would interfere with this process.
> >
> True, if and only if all goes well. At the time thaw() has run writing
> the image to disk can still fail. In that case the devices will still
> be needed.
Consider adding a mechanism to usbcore which would allow an interface
driver to request that the next time its device is resumed, the core
should perform a reset-resume. Would that help?
Alan Stern
On Wed, Apr 20, 2022 at 10:47:27AM +0200, Oliver Neukum wrote:
>
>
> On 19.04.22 19:49, Alan Stern wrote:
> > On Tue, Apr 19, 2022 at 05:51:38PM +0200, Oliver Neukum wrote:
> >>
> >> On 19.04.22 16:35, Alan Stern wrote:
> >>> On Tue, Apr 19, 2022 at 09:05:53AM +0200, Oliver Neukum wrote:
> >>>
> >>>
> >>> The THAW part of suspend-to-hibernation is used only for writing the
> >>> memory image to permanent storage. I doubt that a malfunctioning HID
> >>> device would interfere with this process.
> >>>
> >> True, if and only if all goes well. At the time thaw() has run writing
> >> the image to disk can still fail. In that case the devices will still
> >> be needed.
> > Consider adding a mechanism to usbcore which would allow an interface
> > driver to request that the next time its device is resumed, the core
> > should perform a reset-resume. Would that help?
> >
> >
>
> Strictly speaking no. We already have that in form of the RESET_RESUME
> quirk. The broken devices we are talking about here can do runtime PM
> perfectly fine, if and only if remote wakeup is requested.
> So we need that flag to translate only in freeze()/thaw() resulting in that
> behavior, as opposed to every pair of suspend()/resume()
That was my point. The HID driver can check at suspend time whether or
not remote wakeup will be enabled. If yes, well and good, no changes
are needed. If not, the driver can then request that the following
resume be a runtime-resume.
Another possibility is, as you mentioned before, adding a USB quirk for
devices which require reset-resume whenever they are resumed with wakeup
disabled.
Alan Stern
On Mon, Apr 18, 2022 at 02:00:46PM -0700, Evan Green wrote:
> The documentation for the freeze() method says that it "should quiesce
> the device so that it doesn't generate IRQs or DMA". The unspoken
> consequence of not doing this is that MSIs aimed at non-boot CPUs may
> get fully lost if they're sent during the period where the target CPU is
> offline.
>
> The current callbacks for USB HCD do not fully quiesce interrupts,
> specifically on XHCI. Change to use the full suspend/resume flow for
> freeze/thaw to ensure interrupts are fully quiesced. This fixes issues
> where USB devices fail to thaw during hibernation because XHCI misses
> its interrupt and cannot recover.
>
> Signed-off-by: Evan Green <[email protected]>
>
> ---
Acked-by: Alan Stern <[email protected]>
> Changes in v2:
> - Added the patch modifying the remote wakeup state
That wasn't a change to this patch. No matter.
> - Removed the change to freeze_noirq/thaw_noirq
>
> drivers/usb/core/hcd-pci.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/usb/core/hcd-pci.c b/drivers/usb/core/hcd-pci.c
> index 8176bc81a635d6..ae5e6d572376be 100644
> --- a/drivers/usb/core/hcd-pci.c
> +++ b/drivers/usb/core/hcd-pci.c
> @@ -616,10 +616,10 @@ const struct dev_pm_ops usb_hcd_pci_pm_ops = {
> .suspend_noirq = hcd_pci_suspend_noirq,
> .resume_noirq = hcd_pci_resume_noirq,
> .resume = hcd_pci_resume,
> - .freeze = check_root_hub_suspended,
> + .freeze = hcd_pci_suspend,
> .freeze_noirq = check_root_hub_suspended,
> .thaw_noirq = NULL,
> - .thaw = NULL,
> + .thaw = hcd_pci_resume,
> .poweroff = hcd_pci_suspend,
> .poweroff_noirq = hcd_pci_suspend_noirq,
> .restore_noirq = hcd_pci_resume_noirq,
On 19.04.22 19:49, Alan Stern wrote:
> On Tue, Apr 19, 2022 at 05:51:38PM +0200, Oliver Neukum wrote:
>>
>> On 19.04.22 16:35, Alan Stern wrote:
>>> On Tue, Apr 19, 2022 at 09:05:53AM +0200, Oliver Neukum wrote:
>>>
>>>
>>> The THAW part of suspend-to-hibernation is used only for writing the
>>> memory image to permanent storage. I doubt that a malfunctioning HID
>>> device would interfere with this process.
>>>
>> True, if and only if all goes well. At the time thaw() has run writing
>> the image to disk can still fail. In that case the devices will still
>> be needed.
> Consider adding a mechanism to usbcore which would allow an interface
> driver to request that the next time its device is resumed, the core
> should perform a reset-resume. Would that help?
>
>
Strictly speaking no. We already have that in form of the RESET_RESUME
quirk. The broken devices we are talking about here can do runtime PM
perfectly fine, if and only if remote wakeup is requested.
So we need that flag to translate only in freeze()/thaw() resulting in that
behavior, as opposed to every pair of suspend()/resume()
Regards
Oliver