Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Message-ID: <0d217128-b08c-4c56-ba7c-ed7b7ddfb75a@nvidia.com>
Date:   Tue, 7 Nov 2023 08:44:53 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: Question: Clearing error bits in the root port post enumeration
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     Lorenzo Pieralisi <lpieralisi@kernel.org>,
        Vikram Sethi <vsethi@nvidia.com>,
        Thierry Reding <treding@nvidia.com>,
        Jonathan Hunter <jonathanh@nvidia.com>,
        Krishna Thota <kthota@nvidia.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <20231103182053.GA160440@bhelgaas>
Content-Language: en-US
From:   Vidya Sagar <vidyas@nvidia.com>
In-Reply-To: <20231103182053.GA160440@bhelgaas>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bzFPVlI0VnlsRGUyRnQ4TGdFNHkwRU9XdVp1MWk4WkF4L1R5VWdza295NDVi?=
 =?utf-8?B?ZjVCR1dUZjhnZ2hHTGJ3a1o5UzhGc0R1S1ExN3RLUHZ5YXE1L3hHRlF5T3Yv?=
 =?utf-8?B?MjE5MENqbTlPbFozQzRWQXIvaE45clkzMnlObUlXQm1CRGZhdHI1Ky8rSy9v?=
 =?utf-8?B?UCtlbEozQXVTcjJ6eHpvUzEyS2VodDZaa2FGaUw5UzVVTTdRMUtlaHBteUpX?=
 =?utf-8?B?RWc4YjRDV09LYzdyc3VvNlV6b2laL2RDeWc5MmFoWWxlTWtsYStRL2dmKzhu?=
 =?utf-8?B?V2I4US9wRDV4RXVmaUc4cVAxOXZVWVpJRmxhcUpJZDN0NXhqY1lzL0xvNm5q?=
 =?utf-8?B?NE9HR0p0WFg4TWV1K1BxdzErYytVVUZUeXNwTUh4c0FrdlVyK1NiUFUvWGVl?=
 =?utf-8?B?UFJPOCtsanQ1azBXYXVJbi8yNDlRUFpOSzRKbFlMa05DZmxvUXV5SWVSb3pl?=
 =?utf-8?B?RFdodHBGZDJJbTBhVFk5bUtUMW9pOGE1MERGOXVWWjdnSW5vUWc1L0R0eklw?=
 =?utf-8?B?dUNlMmczTTBLOElScnRDRkFxeVRJOExpWWk3UVpsN3k3YUo2NXRlUHRaZDMv?=
 =?utf-8?B?OFo5aGFNejU1d1l3bUh0NVAxTE5WZ1ZUdGF2bEtGWmdSb0VkbHJiakZScWpq?=
 =?utf-8?B?S2cvYzNSelR5ZXNxM2hJR0ZBeGQ5eFBDUkp4R29pMW5ORy9icWFVaDdUblpt?=
 =?utf-8?B?NnhET1kxaDMyc1V3eW92OGpqMnE1dDM3VnR1SHYvMFRITmx3bURMOGZXNEkr?=
 =?utf-8?B?MVhNazc1aXlkK1JJVmdtYUxsa1prMkxveVduN3pXaTNGcmMvNm1HS1M5cy9p?=
 =?utf-8?B?V1d5NUhBbjA4SXRvU2V5V2xwNkQ4MFdHM3doZURXRDFUeHE4NlpKU2FWQ2V0?=
 =?utf-8?B?Tk1zNituVXB4NDRxN3lEajVpNUljNkp1RzF0b3ROTDA5UUtWbWhCcmNQZnhy?=
 =?utf-8?B?aklPekdUV2N2OXR1TXcxRklIQ3ZaUi95Snp6QWxYYStPWGovbUYwaUtVN1Uw?=
 =?utf-8?B?RFllUUgwY1kxcmRWckFaclZCY0ZqRlB4ZSt0bUFJSCtpU0FQcjl1blZSdlpo?=
 =?utf-8?B?N1B6WVJOdW1aaldteXQ4TTAzV2N0RWhFdHYrYUpSY1lwU2NpYTRTdE9kQ0RW?=
 =?utf-8?B?a2JpRHNROTM5d2pvUUU2MGtIeG4yYlplNE1Zc0V6bVhYQ0V0NmlXWGx4SUlr?=
 =?utf-8?B?QUFXZlpaVmlGNjlSS3RraC9lTkowekMrOS9Eb0xlUm85M3dra2h4VkZzazl4?=
 =?utf-8?B?U3ArUjNRMGtITWYrS2YwTjRjdkowc05Zb1BEd2d4SGczbUFLWjRubGRWNTR0?=
 =?utf-8?B?bEtLWjVwOUdxTEEyNnFtbjU5Szl4WC8zdlZUNVg4R2xmSlovVVNnVER5eHQ3?=
 =?utf-8?B?NjgvNFZBbC9TVWxTS2ZpWWQ0Z3JPTGc4UlBueTlZeXpFWDVrNjNiNDNaajlP?=
 =?utf-8?B?VnNrQmw1bnBja1htS0Z4amxyRUlhd0xlWWZxa0psYkhmRldMcWs2dlE1eDV1?=
 =?utf-8?B?NFlqTkVRRkJkb1RQeThEQkZ5UitSR2c5TnV4bktNd1NVQ241Z25TSzR0THNG?=
 =?utf-8?B?VitVY0svaG52b0tSVWZNeGlGOUF1UGYzV0ZqbkhlMjRiRlpXNTJTcGM0WlIv?=
 =?utf-8?B?WTFTaWxNQ2dXM0VWYjhqbXRUSjk0Y1gzSVZFK2RKOVBKYTcxZHdXSENJbUFu?=
 =?utf-8?B?ekVZZXFsYXY5M1BnTHIwWWpGU0FCZ3dKWmkwclFaUmZSeVVJaTFYYUtiSG0v?=
 =?utf-8?B?SUdhcFBzZ0tIQ1N3K01SVkloK0xnU1BvbkF2dzRuUHJaczZWQVAremxidjB5?=
 =?utf-8?B?RlBXdk5ZQzdHNnFUajRkTGpoUk9rbll0RzNyRE16Y3NaL1Bmd2RTWUI5am1R?=
 =?utf-8?B?MUtnNnQ2d0ZzbXg4cGtEcVh1OFZLUk9EUTFjNXZudTVvUDhtbHp0Y2pLbWZs?=
 =?utf-8?B?ZXlCdkg2VjNSNlFlSkFhSDdhTHVtSmZQdWhwUHdHbkhabEpHSGVEVTdMQ0ly?=
 =?utf-8?B?SWN3MDlaRHg5NFdHL0E1RW9ReDNrZXdlZlF6SmFlOWFHbkFTNmRURzc3MzA5?=
 =?utf-8?B?MGFvZ0lCMjBuNXY0Z2cyYlpIb3lrb3dQa2JZcE9QRytUMkVsQTRyem1JT2o2?=
 =?utf-8?Q?8re/ye8ERe4Qw/Ml2Nm/Px6Wp?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f7682541-8b3c-4bad-6281-08dbdf3fbb8b
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB2900.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2023 03:15:02.9572
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kv8NwqSK8LWcMkgIAwn6NVva+o+45GHF4/yi3sohZjbRR/yrXyS9qFhyauGkoV6ZqMjcN6QReuZ6pxJfWeoqag==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4159
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 06 Nov 2023 19:15:22 -0800 (PST)


On 11/3/2023 11:50 PM, Bjorn Helgaas wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Tue, Oct 31, 2023 at 12:26:31PM +0000, Vidya Sagar wrote:
>> Hi folks,
>>
>> I would like to know your comments on the following scenario where
>> we are observing the root port logging errors because of the
>> enumeration flow being followed.
>>
>> DUT information:
>> - Has a root port and an endpoint connected to it
>> - Uses ECAM mechanism to access the configuration space
>> - Booted through ACPI flow
>> - Has a Firmware-First approach for handling the errors
>> - System is configured to treat Unsupported Requests as
>>    AdvisoryNon-Fatal errors
>>
>> As we all know, when a configuration read request comes in for a
>> device number that is not implemented, a UR would be returned as per
>> the PCIe spec.
>>
>> As part of the enumeration flow on DUT, when the kernel reads offset
>> 0x0 of B:D:F=0:0:0, the root port responds with its valid Vendor-ID
>> and Device-ID values.  But, when B:D:F=0:1:0 is probed, since there
>> is no device present there, the root port responds with an
>> Unsupported Request and simultaneously logs the same in the Device
>> Status register (i.e. bit-3).  Because of it, there is a UR logged
>> in the Device Status register of the RP by the time enumeration is
>> complete.
>>
>> In the case of AER capability natively owned by the kernel, the AER
>> driver's init call would clear all such pending bits.
>>
>> Since we are going with the Firmware-First approach, and the system
>> is configured to treat Unsupported Requests as AdvisoryNon-Fatal
>> errors, only a correctable error interrupt can be raised to the
>> Firmware which takes care of clearing the corresponding status
>> registers.  The firmware can't know about the UnsupReq bit being set
>> as the interrupt it received is for a correctable error hence it
>> clears only bits related to correctable error.
>>
>> All these events leave a freshly booted system with the following
>> bits set.
>>
>> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-          (MAbort)
>> DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-                                                              (UnsupReq)
>> UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-   (UnsupReq)
>>
>> Since the reason for UR is well understood at this point, I would
>> like to weigh in on the idea of clearing the aforementioned bits in
>> the root port once the enumeration is done particularly to cater to
>> the configurations where Firmware-First approach is in place.
>> Please let me know your comments on this approach.
> 
> I think Secondary status (PCI_SEC_STATUS) is always owned by the OS
> and is not affected by _OSC negotiation, right?  Linux does basically
> nothing with that today, but I think it *could* clear the "Received
> Master Abort" bit.
Yes. PCI_SEC_STATUS is always owned by the OS and _OSC negotiation 
doesn't really affect that.

> 
> I'm not very familiar with Advisory Non-Fatal errors.  I'm curious
> about the UESta situation: why can't firmware know about UnsupReq
> being set?  I assume PCI_ERR_COR_ADV_NFAT is the Correctable Error
> Status bit the firmware *does* see and clear.
Yes, PCI_ERR_COR_ADV_NFAT is indeed cleared by the firmware.
> 
> But isn't the whole point of Advisory Non-Fatal errors that an error
> that is logged as an Uncorrectable Error and that normally would be
> signaled with ERR_NONFATAL is signaled with ERR_COR instead?  So
> doesn't PCI_ERR_COR_ADV_NFAT being set imply that some
> PCI_ERR_UNCOR_STATUS must be set as well?  If so, I would think
> firmware *could* figure that out and clear the PCI_ERR_UNCOR_STATUS
> bit.
So, are you suggesting that let the firmware only clear the 
PCI_ERR_UNCOR_STATUS also? if so, then, I can even make the firmware 
clear the PCI_SEC_STATUS also thereby leaving the firmware responsible 
for clearing all the error bits. Does that sound ok?
> 
> Bjorn