Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <18ccb5e6-0bc3-82ab-0c54-653ccd994c7b@linux.intel.com>
Date:   Thu, 6 Apr 2023 22:31:20 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.9.0
Subject: Re: [PATCH v2] PCI/EDR: Clear PCIe Device Status errors after EDR
 error recovery
Content-Language: en-US
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     "Natu, Mahesh" <mahesh.natu@intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <20230406222135.GA3743643@bhelgaas>
From:   Sathyanarayanan Kuppuswamy 
        <sathyanarayanan.kuppuswamy@linux.intel.com>
In-Reply-To: <20230406222135.GA3743643@bhelgaas>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk

Hi Bjorn,

On 4/6/23 3:21 PM, Bjorn Helgaas wrote:
> On Thu, Apr 06, 2023 at 02:52:02PM -0700, Sathyanarayanan Kuppuswamy wrote:
>> On 4/6/23 2:07 PM, Bjorn Helgaas wrote:
>>> On Wed, Mar 15, 2023 at 04:54:49PM -0700, Kuppuswamy Sathyanarayanan wrote:
>>>> Commit 068c29a248b6 ("PCI/ERR: Clear PCIe Device Status errors only if
>>>> OS owns AER") adds support to clear error status in the Device Status
>>>> Register(DEVSTA) only if OS owns the AER support. But this change
>>>> breaks the requirement of the EDR feature which requires OS to cleanup
>>>> the error registers even if firmware owns the control of AER support.
>>>>
>>>> More details about this requirement can be found in PCIe Firmware
>>>> specification v3.3, Table 4-6 Interpretation of the _OSC Control Field.
>>>> If the OS supports the Error Disconnect Recover (EDR) feature and
>>>> firmware sends the EDR event, then during the EDR recovery window, OS
>>>> is responsible for the device error recovery and holds the ownership of
>>>> the following error registers.
>>>>
>>>> • Device Status Register
>>>> • Uncorrectable Error Status Register
>>>> • Correctable Error Status Register
>>>> • Root Error Status Register
>>>> • RP PIO Status Register
>>>>
>>>> So call pcie_clear_device_status() in edr_handle_event() if the error
>>>> recovery is successful.
>>>>
>>>> Reported-by: Tsaur Erwin <erwin.tsaur@intel.com>
>>>> Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
>>>> ---
>>>>
>>>> Changes since v1:
>>>>  * Rebased on top of v6.3-rc1.
>>>>  * Fixed a typo in pcie_clear_device_status().
>>>>
>>>>  drivers/pci/pcie/edr.c | 1 +
>>>>  1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/drivers/pci/pcie/edr.c b/drivers/pci/pcie/edr.c
>>>> index a6b9b479b97a..87734e4c3c20 100644
>>>> --- a/drivers/pci/pcie/edr.c
>>>> +++ b/drivers/pci/pcie/edr.c
>>>> @@ -193,6 +193,7 @@ static void edr_handle_event(acpi_handle handle, u32 event, void *data)
>>>>  	 */
>>>>  	if (estate == PCI_ERS_RESULT_RECOVERED) {
>>>>  		pci_dbg(edev, "DPC port successfully recovered\n");
>>>> +		pcie_clear_device_status(edev);
>>>>  		acpi_send_edr_status(pdev, edev, EDR_OST_SUCCESS);
>>>
>>> The implementation note in PCI Firmware r3.3, sec 4.6.12, shows the OS
>>> clearing error status *after* _OST is evaluated.
>>>
>>> On the other hand, the _OSC DPC control bit in table 4-6 says that if
>>> the OS does not have DPC control, it can only write the Device Status
>>> error bits between the EDR Notify and invoking _OST.
>>>
>>> Is one of those wrong, or am I missing something?
>>
>> Agree. It is conflicting info. IMO, the argument that the OS is allowed to
>> clear the error registers during the EDR windows makes more sense. If OS
>> is allowed to touch error registers owned by firmware after that window,
>> it would lead to race conditions.
>>
>> Mahesh, let us know your comments. Maybe we need to fix this in the firmware
>> specification.
> 
> My assumption was this sequence is something like this, where firmware
> *can't* collect error status from devices below the Downstream Port
> because DPC has been triggered and they are not accessible:
> 
>   - Hardware triggers DPC in a Downstream Port
>   - Firmware fields error interrupt
>   - Firmware captures Downstream Port error info (devices below are
>     not accessible because of DPC)
>   - Firmware sends EDR Notify to OS
>   - OS brings Downstream Port out of DPC
>   - OS collects error status from devices below Downstream Port
>   - OS evaluates _OST
>   - Firmware captures error status from devices below Downstream Port
> 
> But that doesn't explain why *firmware* could not clear the error
> status of those devices after it captures it.
> 
> I guess the flowchart *does* show firmware clearing the error status
> in the "do not continue recovery" path.

In this patch, we are clearing the port error status. So I think it is
fine to do it in EDR window. Agree?

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer