Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 6 Apr 2023 17:21:35 -0500
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     Sathyanarayanan Kuppuswamy 
        <sathyanarayanan.kuppuswamy@linux.intel.com>
Cc:     "Natu, Mahesh" <mahesh.natu@intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] PCI/EDR: Clear PCIe Device Status errors after EDR
 error recovery
Message-ID: <20230406222135.GA3743643@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <a4803e7e-bbfe-8925-88e9-935775960e63@linux.intel.com>
Precedence: bulk

On Thu, Apr 06, 2023 at 02:52:02PM -0700, Sathyanarayanan Kuppuswamy wrote:
> On 4/6/23 2:07 PM, Bjorn Helgaas wrote:
> > On Wed, Mar 15, 2023 at 04:54:49PM -0700, Kuppuswamy Sathyanarayanan wrote:
> >> Commit 068c29a248b6 ("PCI/ERR: Clear PCIe Device Status errors only if
> >> OS owns AER") adds support to clear error status in the Device Status
> >> Register(DEVSTA) only if OS owns the AER support. But this change
> >> breaks the requirement of the EDR feature which requires OS to cleanup
> >> the error registers even if firmware owns the control of AER support.
> >>
> >> More details about this requirement can be found in PCIe Firmware
> >> specification v3.3, Table 4-6 Interpretation of the _OSC Control Field.
> >> If the OS supports the Error Disconnect Recover (EDR) feature and
> >> firmware sends the EDR event, then during the EDR recovery window, OS
> >> is responsible for the device error recovery and holds the ownership of
> >> the following error registers.
> >>
> >> • Device Status Register
> >> • Uncorrectable Error Status Register
> >> • Correctable Error Status Register
> >> • Root Error Status Register
> >> • RP PIO Status Register
> >>
> >> So call pcie_clear_device_status() in edr_handle_event() if the error
> >> recovery is successful.
> >>
> >> Reported-by: Tsaur Erwin <erwin.tsaur@intel.com>
> >> Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
> >> ---
> >>
> >> Changes since v1:
> >>  * Rebased on top of v6.3-rc1.
> >>  * Fixed a typo in pcie_clear_device_status().
> >>
> >>  drivers/pci/pcie/edr.c | 1 +
> >>  1 file changed, 1 insertion(+)
> >>
> >> diff --git a/drivers/pci/pcie/edr.c b/drivers/pci/pcie/edr.c
> >> index a6b9b479b97a..87734e4c3c20 100644
> >> --- a/drivers/pci/pcie/edr.c
> >> +++ b/drivers/pci/pcie/edr.c
> >> @@ -193,6 +193,7 @@ static void edr_handle_event(acpi_handle handle, u32 event, void *data)
> >>  	 */
> >>  	if (estate == PCI_ERS_RESULT_RECOVERED) {
> >>  		pci_dbg(edev, "DPC port successfully recovered\n");
> >> +		pcie_clear_device_status(edev);
> >>  		acpi_send_edr_status(pdev, edev, EDR_OST_SUCCESS);
> > 
> > The implementation note in PCI Firmware r3.3, sec 4.6.12, shows the OS
> > clearing error status *after* _OST is evaluated.
> > 
> > On the other hand, the _OSC DPC control bit in table 4-6 says that if
> > the OS does not have DPC control, it can only write the Device Status
> > error bits between the EDR Notify and invoking _OST.
> > 
> > Is one of those wrong, or am I missing something?
> 
> Agree. It is conflicting info. IMO, the argument that the OS is allowed to
> clear the error registers during the EDR windows makes more sense. If OS
> is allowed to touch error registers owned by firmware after that window,
> it would lead to race conditions.
> 
> Mahesh, let us know your comments. Maybe we need to fix this in the firmware
> specification.

My assumption was this sequence is something like this, where firmware
*can't* collect error status from devices below the Downstream Port
because DPC has been triggered and they are not accessible:

  - Hardware triggers DPC in a Downstream Port
  - Firmware fields error interrupt
  - Firmware captures Downstream Port error info (devices below are
    not accessible because of DPC)
  - Firmware sends EDR Notify to OS
  - OS brings Downstream Port out of DPC
  - OS collects error status from devices below Downstream Port
  - OS evaluates _OST
  - Firmware captures error status from devices below Downstream Port

But that doesn't explain why *firmware* could not clear the error
status of those devices after it captures it.

I guess the flowchart *does* show firmware clearing the error status
in the "do not continue recovery" path.