Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20221226153048.1208359-1-kai.heng.feng@canonical.com> <20221226154643.kidhqooedmijj3iw@pali>
In-Reply-To: <20221226154643.kidhqooedmijj3iw@pali>
From:   Kai-Heng Feng <kai.heng.feng@canonical.com>
Date:   Thu, 29 Dec 2022 11:45:51 +0800
Message-ID: <CAAd53p4CU+K5sOumJYxENRE-Ci7zPKxk0ROszvBUPWV=1xYZyw@mail.gmail.com>
Subject: Re: [PATCH] PCI/portdrv: Avoid enabling AER on Thunderbolt devices
To:     =?UTF-8?Q?Pali_Roh=C3=A1r?= <pali@kernel.org>
Cc:     bhelgaas@google.com, Mario Limonciello <mario.limonciello@amd.com>,
        Mika Westerberg <mika.westerberg@linux.intel.com>,
        Keith Busch <kbusch@kernel.org>,
        Kuppuswamy Sathyanarayanan 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        Stefan Roese <sr@denx.de>, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Hi Pali,


On Mon, Dec 26, 2022 at 11:46 PM Pali Roh=C3=A1r <pali@kernel.org> wrote:
>
> On Monday 26 December 2022 23:30:31 Kai-Heng Feng wrote:
> > We are seeing igc ethernet device on Thunderbolt dock stops working
> > after S3 resume because of AER error, or even make S3 resume freeze:
>
> Hello! Is igc ethernet the only device which does not work after resume?

Seems so.
A Thunderbolt NVMe enclosure plugged to the dock doesn't exhibit this
issue. I don't have an eGPU to try.

> Or do you have also more devices to test and check for this issue?
>
> I'm asking it just because to know if we are dealing with one device or
> there are lot of more. Because if it is just one device then it could be
> better to disable AER only for one targeted device instead of all. Error
> reporting is a feature which may help to detect broken HW unit and be use=
ful.
>
> > pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:=
1d.0
> > pcieport 0000:00:1d.0: PCIe Bus Error: severity=3DCorrected, type=3DTra=
nsaction Layer, (Receiver ID)
> > pcieport 0000:00:1d.0:   device [8086:7ab0] error status/mask=3D0000800=
0/00002000
> > pcieport 0000:00:1d.0:    [15] HeaderOF
> > pcieport 0000:00:1d.0: AER: Multiple Uncorrected (Non-Fatal) error rece=
ived: 0000:00:1d.0
> > pcieport 0000:00:1d.0: PCIe Bus Error: severity=3DUncorrected (Non-Fata=
l), type=3DTransaction Layer, (Requester ID)
> > pcieport 0000:00:1d.0:   device [8086:7ab0] error status/mask=3D0010000=
0/00004000
> > pcieport 0000:00:1d.0:    [20] UnsupReq               (First)
> > pcieport 0000:00:1d.0: AER:   TLP Header: 34000000 0a000052 00000000 00=
000000
> > pcieport 0000:00:1d.0: AER:   Error of this Agent is reported first
> > pcieport 0000:04:01.0: PCIe Bus Error: severity=3DUncorrected (Non-Fata=
l), type=3DTransaction Layer, (Requester ID)
> > pcieport 0000:04:01.0:   device [8086:1136] error status/mask=3D0030000=
0/00000000
> > pcieport 0000:04:01.0:    [20] UnsupReq               (First)
> > pcieport 0000:04:01.0:    [21] ACSViol
> > pcieport 0000:04:01.0: AER:   TLP Header: 34000000 04000052 00000000 00=
000000
> > thunderbolt 0000:05:00.0: AER: can't recover (no error_detected callbac=
k)
> >
> > This supposedly should be fixed by commit c01163dbd1b8 ("PCI/PM: Always=
 disable
> > PTM for all devices during suspend"), but somehow it doesn't work for
> > this case.
> >
> > By dumping the PCI_PTM_CTRL register on resume, it turns out PTM is
> > already flipped on by either the Thunderbolt dock firmware or the host
> > BIOS. Writing 0 to PCI_PTM_CTRL yields the same result.
> >
> > Windows is however not affected by this issue, by using WinDbg's !pci
> > command, it shows that AER is not enabled for devices connected via
> > Thunderbolt port, and that's the reason why Windows doesn't exhibit the
> > issue.
>
> Could you try to manually enable AER on Windows (via touching PCIe
> config registers) if Windows can trigger this issue too, or not?

Actually I misread the output of WinDbg !pci command, the AER is also
enabled under Windows.
!pci command also shows the same PTM error in Header Log. I can also
find the AER warnings in Windows' Event Viewer.

I am asking hardware vendor to see if it's possible to fix it at firmware s=
ide.

However, on Windows the ACS is disabled for all downstream ports in
the dock, so unlike Linux there's no ACS violation.
That can be the reason why the igc device continues to work on Windows
despite AER errors.

So yes, this patch is wrong. Let me dig this issue a bit more.

Kai-Heng

>
> > So turn a blind eye on external Thunderbolt devices like Windows does b=
y
> > disabling AER.
> >
> > Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=3D216850
> > Cc: Mario Limonciello <mario.limonciello@amd.com>
> > Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
> > Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
> > ---
> >  drivers/pci/pcie/portdrv.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/pci/pcie/portdrv.c b/drivers/pci/pcie/portdrv.c
> > index 2cc2e60bcb396..59d00e20e57bf 100644
> > --- a/drivers/pci/pcie/portdrv.c
> > +++ b/drivers/pci/pcie/portdrv.c
> > @@ -237,7 +237,8 @@ static int get_port_device_capability(struct pci_de=
v *dev)
> >       if ((pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_ROOT_PORT ||
> >               pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_RC_EC) &&
> >           dev->aer_cap && pci_aer_available() &&
> > -         (pcie_ports_native || host->native_aer))
> > +         (pcie_ports_native || host->native_aer) &&
> > +         !dev_is_removable(&dev->dev))
> >               services |=3D PCIE_PORT_SERVICE_AER;
> >  #endif
> >
> > --
> > 2.34.1
> >