From:   Maxim Levitsky <mlevitsk@redhat.com>
To:     linux-kernel@vger.kernel.org
Cc:     David Woodhouse <dwmw@amazon.co.uk>,
        Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        Joerg Roedel <joro@8bytes.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        iommu@lists.linux-foundation.org (open list:AMD IOMMU (AMD-VI)),
        Will Deacon <will@kernel.org>,
        Maxim Levitsky <mlevitsk@redhat.com>
Subject: [PATCH 0/5] iommu/amd: fixes for suspend/resume
Date:   Tue, 23 Nov 2021 18:10:33 +0200
Message-Id: <20211123161038.48009-1-mlevitsk@redhat.com>
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

As I sadly found out, a s3 cycle makes the AMD's iommu stop sending interru=
pts=0D
until the system is rebooted.=0D
=0D
I only noticed it now because otherwise the IOMMU works, and these interrup=
ts=0D
are only used for errors and for GA log which I tend not to use by=0D
making my VMs do mwait/pause/etc in guest (cpu-pm=3Don).=0D
=0D
There are two issues here that prevent interrupts from being generated afte=
r=0D
s3 cycle:=0D
=0D
1. GA log base address was not restored after resume, and was all zeroed=0D
after resume (by BIOS or such).=0D
=0D
In theory if BIOS writes some junk to it, that can even cause a memory corr=
uption.=0D
Patch 2 fixes that.=0D
=0D
2. INTX (aka x2apic mode) settings were not restored after resume.=0D
That mode is used regardless if the host uses/supports x2apic, but rather w=
hen=0D
the IOMMU supports it, and mine does.=0D
Patches 3-4 fix that.=0D
=0D
Note that there is still one slight (userspace) bug remaining:=0D
During suspend all but the boot CPU are offlined and then after resume=0D
are onlined again.=0D
=0D
The offlining moves all non-affinity managed interrupts to CPU0, and=0D
later when all other CPUs are onlined, there is nothing in the kernel=0D
to spread back the interrupts over the cores.=0D
=0D
The userspace 'irqbalance' daemon does fix this but it seems to ignore=0D
the IOMMU interrupts in INTX mode since they are not attached to any=0D
PCI device, and thus they remain on CPU0 after a s3 cycle,=0D
which is suboptimal when the system has multiple IOMMUs=0D
(mine has 4 of them).=0D
=0D
Setting the IRQ affinity manually via /proc/irq/ does work.=0D
=0D
This was tested on my 3970X with both INTX and regular MSI mode (later was =
enabled=0D
by patching out INTX detection), by running a guest with AVIC enabled and w=
ith=0D
a PCI assigned device (network card), and observing interrupts from=0D
IOMMU while guest is mostly idle.=0D
=0D
This was also tested on my AMD laptop with 4650U (which has the same issue)=
=0D
(I tested only INTX mode)=0D
=0D
Patch 1 is a small refactoring to remove an unused struct field.=0D
=0D
Best regards,=0D
   Maxim Levitsky=0D
=0D
Maxim Levitsky (5):=0D
  iommu/amd: restore GA log/tail pointer on host resume=0D
  iommu/amd: x2apic mode: re-enable after resume=0D
  iommu/amd: x2apic mode: setup the INTX registers on mask/unmask=0D
  iommu/amd: x2apic mode: mask/unmask interrupts on suspend/resume=0D
  iommu/amd: remove useless irq affinity notifier=0D
=0D
 drivers/iommu/amd/amd_iommu_types.h |   2 -=0D
 drivers/iommu/amd/init.c            | 107 +++++++++++++++-------------=0D
 2 files changed, 58 insertions(+), 51 deletions(-)=0D
=0D
-- =0D
2.26.3=0D
=0D