Message-ID: <fba22d6b-c225-4b44-674b-2c62306135ed@amd.com>
Date:   Tue, 14 Feb 2023 01:33:54 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.11.0
Subject: Re: [PATCH 1/2] PCI: pciehp: Add support for OS-First Hotplug and
 AER/DPC
Content-Language: en-US
To:     Lukas Wunner <lukas@wunner.de>
Cc:     linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org,
        Bjorn Helgaas <bhelgaas@google.com>, oohall@gmail.com,
        Mahesh J Salgaonkar <mahesh@linux.ibm.com>,
        Kuppuswamy Sathyanarayanan 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        Yazen Ghannam <yazen.ghannam@amd.com>,
        Fontenot Nathan <Nathan.Fontenot@amd.com>
References: <20221101000719.36828-1-Smita.KoralahalliChannabasappa@amd.com>
 <20221101000719.36828-2-Smita.KoralahalliChannabasappa@amd.com>
 <20221104101536.GA11363@wunner.de>
From:   Smita Koralahalli <Smita.KoralahalliChannabasappa@amd.com>
In-Reply-To: <20221104101536.GA11363@wunner.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cGoya0lnRFVnaTV2bmVzbEhoREFPWDdhb2JvTXJGcjgrMzVkdFBTaThUSE1z?=
 =?utf-8?B?OWZPajdQYk5LMHhLM2dpWDNSa29jNFdRc2J2cjFIajBKS2pFWVV0U1ZzcDR4?=
 =?utf-8?B?dFV2eFQzSUkxQi9jSGtoZ0pKMTYyQTNJd0VwWWtxbyt6azlXNzFGYXBYRUcr?=
 =?utf-8?B?by94ZjlTdXlOVVRlZXJmM1p5QzdaWm03TVlBUjZpZlIyWEZzeTZwUFBKcldB?=
 =?utf-8?B?K1ZzVFdHT2ExS25ZUERzWEhhWTJ2UzVDYUlvNFZSRTNOMW9uZm5MR2ZNMGhz?=
 =?utf-8?B?aTBWK04xL1Y2cUE2d3J1ZUpGSE1sWFdNTll2TEwzcmQ5V2lTalJOUUZrNmtv?=
 =?utf-8?B?Z3hoaWxIYnNVbThPOXJjWGh2bXFJZEJNSmU2YmFydm1CbWNIL25xN09SSGpx?=
 =?utf-8?B?citEZzRzQmQ1YmhxWWtRUzNNNUtFYTZxT3Vac0NRUmdKa3h1SzBhVGI0YUZs?=
 =?utf-8?B?Umd4RzlhY1QvUGlaK0J6bytEQWtnTVg3NTVKbFpqUnB0aGV4LzFvcG9HMEpq?=
 =?utf-8?B?eCtydEhGUzQ1ZVREM3orTmNDaG1HRDV4dkRvK25PWUd2TjRhbnFzRG05cFFP?=
 =?utf-8?B?Q1BqTnBTSnJ0ZUtyUjVtTnJhUlVDdlcvdndsZjFCSU4xVzlwemR6OTFpWUZz?=
 =?utf-8?B?QnJEZnh2bE5tbWRDcE9XVnMrb0VsbFp1M3ZQbHZDcElQMFlHR2FHWHRTb1M5?=
 =?utf-8?B?eld1TnZyUzgraCtTcFpldFRXejNieFU5QkVUeUxuSWVhK1pkS1NaY3pENjI3?=
 =?utf-8?B?TFVuZWZTZ2F3TG5mR0lEZEtNT0tEektkSnVlSU9pYWkvUTdNRWsramVCd3By?=
 =?utf-8?B?azExZzI5dGZaQ2VybG1iTEJaZW5NQVB0dGZ1YjVuTEg1T3gzSDlBdUUrbUEw?=
 =?utf-8?B?eGFMRjVENnorUWpoN0paaGF1dDMxSzlDMVZjWHVlcTEvay9Nb1AwRXlyaVp3?=
 =?utf-8?B?cGlrczlrS3NyeVBGOUtDbUtzSlZzYlR3bEFiSmJCMUlvTVFYK3plbUZubjJF?=
 =?utf-8?B?MmladlBISnNldGpWMUlwMExSMU9zWFFCTHZVVXZkNHcxd3g0UStuVXJ0bUQx?=
 =?utf-8?B?dXljUG9HRXkwVUg5eHhYdWNud0YzdVFYSTBGNEJ4WCtESUt1OU5zWmJ2cW42?=
 =?utf-8?B?N21YVU4zZWFkeE1vNHlvcDU3MEoyY29peDkwd2N3YnczS0FQRFFzdGttZ3Ez?=
 =?utf-8?B?QzFRNGVBbk9JbU9pTXpIN0h2M1prSDY0SjdEaGlZY29OdUp4UWlmTU9WS0JP?=
 =?utf-8?B?eWxxaGVsVWVSZUk0eTR3YkRkUDRvWVl3KzFnVm1vYjJBNE4wVkZtaDVVZHNj?=
 =?utf-8?B?cktFWFJDaE5FdW82c25LdVBySE5uMzFqOHlNRVMxY1RyV2NoRnFOQ2twMmps?=
 =?utf-8?B?QStFajdOWVUxVE16Y05xUTlNT0YwWEZWMlcxWE1LVUFISkREYzNEL0o5NjJR?=
 =?utf-8?B?cVVlTTZBMDIwbXIvQkNGTnA3TGZRbkJZdVlDVjhNZllsU2tzVVlzM0tOWEZZ?=
 =?utf-8?B?NFVjdjRLUnF5b2E5QTRneXlncmFRVXBXUDdTZjU5SXRtU0k3KzJaUndJczdi?=
 =?utf-8?B?akpTL0FyZkZzMGRWSGRBVUVNdXpGenVpWFZ0bnhNRFluS2FhOHJrSUxWeWNN?=
 =?utf-8?B?cVFVWXlHR0pVNE9wY2dYNWd0dUtZcU5jc3lUSStwSnFEMTNzL0FobkxFaUNm?=
 =?utf-8?B?VDBuU05XWDg0RU9OTFVrdVJzRjFYbXVVRTRaRjdHdEtpMGJ5NXNqemg1UFlo?=
 =?utf-8?B?WDJmL0RYL0s5aUlLVXRZbXhaczFRdGQvQjBQZzhVYmtnR0NPTEdIUmdUKzVF?=
 =?utf-8?B?azZyMm85UzZiMnk4bURXN0ovQ1B5Rk45b0dkZHdHbktOd3ZKVUY4c2FrQXJB?=
 =?utf-8?B?aktRa2RKMHE5elkvNFFpWDZZVUlXcTZ3UzFXUkJnZlZBNkxIblRuc04yRkxZ?=
 =?utf-8?B?QzVnTm92L1hQQnBsYld4YnpOTE5ReFlNcXVlT2ZsQTVtZkdpV0RlT3hORVdG?=
 =?utf-8?B?ZWlsVVZsYmt6dlY1Y05udS9BcEtRWE5PNjZxL283NVNVUkwxVWgxSjY3K3hl?=
 =?utf-8?B?OGowZkdaRUxNUVg0bVgzS1cxYTBOVUd2a1oxb2RvRitxSzQ0dkZCcnNHR1RW?=
 =?utf-8?Q?hX8af+/eiynJNJytL2TV9lmuq?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d8f8cb5c-59f7-4a74-849b-08db0e6e97f9
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB2869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2023 09:33:56.3523
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FqweDdDAReAHaCHN+j0fcNF413yH50ZcWyhm1DOSfbO1SbiJQm1ha1BSNr3VpjW0csE92ZfqCtBWo3TDzoYfhg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4914
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 11/4/2022 3:15 AM, Lukas Wunner wrote:
> On Tue, Nov 01, 2022 at 12:07:18AM +0000, Smita Koralahalli wrote:
>> The implementation is as follows: On an async remove a DPC is triggered as
>> a side-effect along with an MSI to the OS. Determine it's an async remove
>> by checking for DPC Trigger Status in DPC Status Register and Surprise
>> Down Error Status in AER Uncorrected Error Status to be non-zero. If true,
>> treat the DPC event as a side-effect of async remove, clear the error
>> status registers and continue with hot-plug tear down routines. If not,
>> follow the existing routine to handle AER/DPC errors.
> Instead of having the OS recognize and filter Surprise Down events,
> it would also be possible to simply set the Surprise Down bit in the
> Uncorrectable Error Mask Register.  This could be constrained to
> Downstream Ports capable of surprise removal, i.e. those where the
> is_hotplug_bridge in struct pci_dev is set.  And that check and the
> register change could be performed in pci_dpc_init().
>
> Have you considered such an alternative approach?  If you have, what
> was the reason to prefer the more complex solution you're proposing?

By setting the Surprise down bit in Uncorrectable Error Mask register we 
will
not get a DPC event. What I know so far is, we cannot set this bit at 
run-time
after we determine its a surprise down event or probably I don't know
enough how to do it! (once an pciehp interrupt is triggered..).

And setting this bit at initialization might not trigger true DPC events..

Second thing, is masking Surprise Down bit has no impact on logging 
errors in
AER registers.

So, I think that approach probably will not resolve the issue of 
clearing the logs
in AER registers and complicate things while differentiating true errors 
vs surprise
down events. Please correct me if I'm wrong!!

I did few testing after I read your comments.  What I realized is that, 
these DPC
events (side effects of hot remove) are actually benign on AMD systems. 
On a hot
remove I see a Presence Detect change and a DPC event. This PD state change
will trigger a pciehp isr and calls pciehp_handle_presence_or_link_change()
and disables the slot normally. So essentially, this patch will boil 
down to the point
to clearing the logs in AER registers and also handling those error 
messages ("device
recovery failed"....) in dmesg which might confuse users on a hot remove.
What do you think?

Now, I'm not sure whether there will be a PD state change across other 
systems on a
hot remove when AER/DPC is native (OS First) in which case we should 
call the
pciehp_disable_slot() from dpc handler as well.. Any inputs would be
appreciated here..
>
>
>> +static void pci_clear_surpdn_errors(struct pci_dev *pdev)
>> +{
>> +	u16 reg16;
>> +	u32 reg32;
>> +
>> +	pci_read_config_dword(pdev, pdev->dpc_cap + PCI_EXP_DPC_RP_PIO_STATUS, &reg32);
>> +	pci_write_config_dword(pdev, pdev->dpc_cap + PCI_EXP_DPC_RP_PIO_STATUS, reg32);
>> +
>> +	pci_read_config_word(pdev, PCI_STATUS, &reg16);
>> +	pci_write_config_word(pdev, PCI_STATUS, reg16);
>> +
>> +	pcie_capability_read_word(pdev, PCI_EXP_DEVSTA, &reg16);
>> +	pcie_capability_write_word(pdev, PCI_EXP_DEVSTA, reg16);
>> +}
> I don't understand why PCI_STATUS and PCI_EXP_DEVSTA need to be
> touched here?

This is just to mask any kind of appearance that there was an error since
the errors would have been induced by the hot plug event (just duplicating
our BIOS functionality here..).  But, please let me know if OS is already
handling the things here and if it is not required.
>
>
>> +static void pciehp_handle_surprise_removal(struct pci_dev *pdev)
> Since this function is located in dpc.c and is strictly called from
> other functions in the same file, it should be prefixed dpc_, not
> pciehp_.

Sure, will fix in v2.
>
>
>> +	/*
>> +	 * According to Section 6.13 and 6.15 of the PCIe Base Spec 6.0,
>> +	 * following a hot-plug event, clear the ARI Forwarding Enable bit
>> +	 * and AtomicOp Requester Enable as its not determined whether the
>> +	 * next device inserted will support these capabilities. AtomicOp
>> +	 * capabilities are not supported on PCI Express to PCI/PCI-X Bridges
>> +	 * and any newly added component may not be an ARI device.
>> +	 */
>> +	pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL2,
>> +				   (PCI_EXP_DEVCTL2_ARI | PCI_EXP_DEVCTL2_ATOMIC_REQ));
> That looks like a reasonable change, but it belongs in a separate
> patch.  And I think it should be performed as part of (de-)enumeration,
> not as part of DPC error handling.  What about Downstream Ports which
> are not DPC-capable, I guess the bits should be cleared as well, no?

AFAIK, DPC will work on all AMD root ports. I'm not sure how could we handle
on a per port basis if the bridges/ports downstream to root ports don't 
support
DPC..
>
> How about clearing the bits in pciehp_unconfigure_device()?

Okay.

Thanks,
Smita
>
> Thanks,
>
> Lukas