Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 12 May 2023 20:24:38 -0500
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Cc:     Peter Geis <pgwipeout@gmail.com>, kw@linux.com, heiko@sntech.de,
        robh@kernel.org, linux-pci@vger.kernel.org,
        shawn.lin@rock-chips.com, linux-kernel@vger.kernel.org,
        lgirdwood@gmail.com, linux-rockchip@lists.infradead.org,
        broonie@kernel.org, bhelgaas@google.com,
        linux-kernel-mentees@lists.linuxfoundation.org,
        lpieralisi@kernel.org, linux-arm-kernel@lists.infradead.org,
        Dan Johansen <strit@manjaro.org>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>
Subject: Re: [PATCH v1] drivers: pci: introduce configurable delay for
 Rockchip PCIe bus scan
Message-ID: <ZF7m1npzLZmawT8Y@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CSK8M39MQL2C.3S7JO031H0BA2@vincent-arch>
Precedence: bulk

[+cc ARM64 folks, in case you have abort handling tips; thread at:
https://lore.kernel.org/r/20230509153912.515218-1-vincenzopalazzodev@gmail.com]

Pine64 RockPro64 panics while enumerating some PCIe devices.  Adding a
delay avoids the panic.  My theory is a PCIe Request Retry Status to a
Vendor ID config read causes an abort that we don't handle.

> On Tue, May 09, 2023 at 05:39:12PM +0200, Vincenzo Palazzo wrote:
>> ...
>> [    1.229856] SError Interrupt on CPU4, code 0xbf000002 -- SError
>> [    1.229860] CPU: 4 PID: 1 Comm: swapper/0 Not tainted 5.9.9-2.0-MANJARO-ARM
>> #1
>> [    1.229862] Hardware name: Pine64 RockPro64 v2.1 (DT)
>> [    1.229864] pstate: 60000085 (nZCv daIf -PAN -UAO BTYPE=--)
>> [    1.229866] pc : rockchip_pcie_rd_conf+0xb4/0x270
>> [    1.229868] lr : rockchip_pcie_rd_conf+0x1b4/0x270
>> ...
>> [    1.229939] Kernel panic - not syncing: Asynchronous SError Interrupt
>> ...
>> [    1.229955]  nmi_panic+0x8c/0x90
>> [    1.229956]  arm64_serror_panic+0x78/0x84
>> [    1.229958]  do_serror+0x15c/0x160
>> [    1.229960]  el1_error+0x84/0x100
>> [    1.229962]  rockchip_pcie_rd_conf+0xb4/0x270
>> [    1.229964]  pci_bus_read_config_dword+0x6c/0xd0
>> [    1.229966]  pci_bus_generic_read_dev_vendor_id+0x34/0x1b0
>> [    1.229968]  pci_scan_single_device+0xa4/0x144

On Fri, May 12, 2023 at 12:46:21PM +0200, Vincenzo Palazzo wrote:
> ... Is there any way to tell the kernel "hey we need some more time
> here"?

We enumerate PCI devices by trying to read the Vendor ID of every
possible device address (see pci_scan_slot()).  On PCIe, if a device
doesn't exist at that address, the Vendor ID config read will be
terminated with Unsupported Request (UR) status.  This is normal
and happens every time we enumerate devices.

The crash doesn't happen every time we enumerate, so I don't think
this UR is the problem.  Also, if it *were* the problem, adding a
delay would not make any difference.

There *is* a way for a PCIe device to say "I need more time".  It does
this by responding to that Vendor ID config read with Request Retry
Status (RRS, aka CRS in older specs), which means "I'm not ready yet,
but I will be ready in the future."  Adding a delay would definitely
make a difference here, so my guess is this is what's happening.

Most root complexes return ~0 data to the CPU when a config read
terminates with UR or RRS.  It sounds like rockchip does this for UR
but possibly not for RRS.

There is a "RRS Software Visibility" feature, which is supposed to
turn the RRS into a special value (Vendor ID == 0x0001), but per [1],
rockchip doesn't support it (lspci calls it "CRSVisible").

But the CPU load instruction corresponding to the config read has to
complete by reading *something* or else be aborted.  It sounds like
it's aborted in this case.  I don't know the arm64 details, but if we
could catch that abort and determine that it was an RRS and not a UR,
maybe we could fabricate the magic RRS 0x0001 value.

imx6q_pcie_abort_handler() does something like that, although I think
it's for arm32, not arm64.  But obviously we already catch the abort
enough to dump the register state and panic, so maybe there's a way to
extend that?

Bjorn

[1] https://lore.kernel.org/linux-pci/CAMdYzYpOFAVq30N+O2gOxXiRtpoHpakFg3LKq3TEZq4S6Y0y0g@mail.gmail.com/