Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Kai-Heng Feng <kai.heng.feng@canonical.com>
To:     bhelgaas@google.com
Cc:     Kai-Heng Feng <kai.heng.feng@canonical.com>,
        Mario Limonciello <mario.limonciello@amd.com>,
        Mika Westerberg <mika.westerberg@linux.intel.com>,
        Keith Busch <kbusch@kernel.org>,
        Kuppuswamy Sathyanarayanan 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        =?UTF-8?q?Pali=20Roh=C3=A1r?= <pali@kernel.org>,
        Stefan Roese <sr@denx.de>, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: [PATCH] PCI/portdrv: Avoid enabling AER on Thunderbolt devices
Date:   Mon, 26 Dec 2022 23:30:31 +0800
Message-Id: <20221226153048.1208359-1-kai.heng.feng@canonical.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

We are seeing igc ethernet device on Thunderbolt dock stops working
after S3 resume because of AER error, or even make S3 resume freeze:
pcieport 0000:00:1d.0: AER: Multiple Corrected error received: 0000:00:1d.0
pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
pcieport 0000:00:1d.0:   device [8086:7ab0] error status/mask=00008000/00002000
pcieport 0000:00:1d.0:    [15] HeaderOF
pcieport 0000:00:1d.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:00:1d.0
pcieport 0000:00:1d.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
pcieport 0000:00:1d.0:   device [8086:7ab0] error status/mask=00100000/00004000
pcieport 0000:00:1d.0:    [20] UnsupReq               (First)
pcieport 0000:00:1d.0: AER:   TLP Header: 34000000 0a000052 00000000 00000000
pcieport 0000:00:1d.0: AER:   Error of this Agent is reported first
pcieport 0000:04:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
pcieport 0000:04:01.0:   device [8086:1136] error status/mask=00300000/00000000
pcieport 0000:04:01.0:    [20] UnsupReq               (First)
pcieport 0000:04:01.0:    [21] ACSViol
pcieport 0000:04:01.0: AER:   TLP Header: 34000000 04000052 00000000 00000000
thunderbolt 0000:05:00.0: AER: can't recover (no error_detected callback)

This supposedly should be fixed by commit c01163dbd1b8 ("PCI/PM: Always disable
PTM for all devices during suspend"), but somehow it doesn't work for
this case.

By dumping the PCI_PTM_CTRL register on resume, it turns out PTM is
already flipped on by either the Thunderbolt dock firmware or the host
BIOS. Writing 0 to PCI_PTM_CTRL yields the same result.

Windows is however not affected by this issue, by using WinDbg's !pci
command, it shows that AER is not enabled for devices connected via
Thunderbolt port, and that's the reason why Windows doesn't exhibit the
issue.

So turn a blind eye on external Thunderbolt devices like Windows does by
disabling AER.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=216850
Cc: Mario Limonciello <mario.limonciello@amd.com>
Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
---
 drivers/pci/pcie/portdrv.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/pcie/portdrv.c b/drivers/pci/pcie/portdrv.c
index 2cc2e60bcb396..59d00e20e57bf 100644
--- a/drivers/pci/pcie/portdrv.c
+++ b/drivers/pci/pcie/portdrv.c
@@ -237,7 +237,8 @@ static int get_port_device_capability(struct pci_dev *dev)
 	if ((pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
              pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC) &&
 	    dev->aer_cap && pci_aer_available() &&
-	    (pcie_ports_native || host->native_aer))
+	    (pcie_ports_native || host->native_aer) &&
+	    !dev_is_removable(&dev->dev))
 		services |= PCIE_PORT_SERVICE_AER;
 #endif
 
-- 
2.34.1