Received-SPF: pass (google.com: domain of linux-kernel+bounces-102678-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <d93a3f29-b260-4910-aaf5-d734e6242223@nvidia.com>
Date: Thu, 14 Mar 2024 06:09:57 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH V2] PCI: Clear errors logged in Secondary Status Register
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: bhelgaas@google.com, linux-pci@vger.kernel.org,
 linux-kernel@vger.kernel.org, treding@nvidia.com, jonathanh@nvidia.com,
 kthota@nvidia.com, mmaddireddy@nvidia.com, sagar.tv@gmail.com
References: <20240122230026.GA290856@bhelgaas>
Content-Language: en-US
From: Vidya Sagar <vidyas@nvidia.com>
In-Reply-To: <20240122230026.GA290856@bhelgaas>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?dVhHTXY5em00U2wxVWVERHIyalFxdFlXY1RnbUVhekVUTUhiVlI0LzFGbU9I?=
 =?utf-8?B?SUR4VTlPYnIrVlJLTzhNZ2loOCtNQWhvMzlyNHF5Z1JDRVJ2bDVrelA1ZFpa?=
 =?utf-8?B?S0FtV3k4QTg3eUtrNWR2RlY0dlE2YnlCVVhpK2ZoU0xQL0RHUVZ6L25qc0Rq?=
 =?utf-8?B?Ym8wbGpza0o3eElhalQ2QWVRMEtmb1FraC9nWXlTdWV2cnFjL1J1R0JnaFds?=
 =?utf-8?B?UWZHZ0RnTkYvUE1PTXZ0WVhNU3Q2cU1sNFpZVWJXUTZIL2dJdVRzZ0QwL1dD?=
 =?utf-8?B?M0hyZldNQVV5RVRmODFwSlYvLzNOcVlTVjRZTGZlV0lpZU9waER1NVpFa0hl?=
 =?utf-8?B?RHFId3haS2ZteG9pQWYvL0FhNlgwWUYrNUVNQ2hQTDkwaVpUVWZEb3B3VDJB?=
 =?utf-8?B?aDlXQW9WMUFWMERkT2IwdE95dUtvemEycHBvT05WRE1LamwrTnNCODlCcURh?=
 =?utf-8?B?Q0pOSElNL1k3M0J4ZGZ4Z3FlMzJ3OVZ5eVFCUFdKK3YrTFA4ZzJtaVN4UzBW?=
 =?utf-8?B?WnRwNmtVNVVvK080MWtFUml3c3ZRc0NIMG9tV0RablNzZUx6NFNkTlBwVXNL?=
 =?utf-8?B?QlhmalVVd2huUmR1ZGJodnNVM0RGVkRYalFSVHVOVWVkQmNya3dOUlNJWS8x?=
 =?utf-8?B?U3dwNGQza3BITlliMHVMUkFPUzE4enRWd3dVMElrU2l3UHV6VzVXNU40WkZu?=
 =?utf-8?B?b3NMeDlCK2NKN1o0VklOemJ6amdxS2k2MHArUWl0SEM1QzJzeURCTCtHbmJB?=
 =?utf-8?B?VnRydmdySGhCeDk3S0IxOTFBaGNoMzNxNUxEODRJSWM5QUNwR1BhbUR2YzMr?=
 =?utf-8?B?OEhzaFR2VDh1T0pPcnFEa0F6MWZwMDluZFhvWlhYUTdramRKY0laYmQreVB2?=
 =?utf-8?B?SEJYQm42YkxUSW5OSmUwZkRYbkF3a3VadE8wSC9ueVpuNkIvSjVLYW1YUThk?=
 =?utf-8?B?a09TNmRqUmNnbG9RalRGM1dydlJBQ0kxdlpWQ29LV2RVU3pLNTZoc3lERUxn?=
 =?utf-8?B?ZGpJN2hlRndBQkpvMjQvbEZyNkhUdU4xWXo0S2cyaUhtY0NRWWZJQ2c1b2dH?=
 =?utf-8?B?RzFnME9pd093T29Yb0VlK09MK1h4WFFZUHZhQ25ETGFZWHFyTi95eUZQcDJw?=
 =?utf-8?B?SXR1bVhoS0x0aUkxU2xNM0RVYUNZMzFJNi9vTExyc1NBeVEvYko2QXRweXlX?=
 =?utf-8?B?T1lvTDlhT3pqNUZiOEU1TDJ2NURiUkltUnZJQUllKzJMZTNFenpLaDcwdk5n?=
 =?utf-8?B?eEgrczhnbWRZdmZIOWNkOG01RjVHVmVXNXFWYncrSVA1ZWNWSS93UWZNY0Fm?=
 =?utf-8?B?d2ZlUWs3RFB2R2JKMVRBODNHRGhKUmw0S0RWbldOcUVkWk05SDhjT2lTeGh4?=
 =?utf-8?B?OGxrV2tBa3V1Y0hLeG9vUlJOUXZIYVp6SURtMG00WVg2MzdPZjZpOXltS3VW?=
 =?utf-8?B?ZExlZllrT1YydU4vaGNDYUh4NDRBaXRIQzJMVHVGSnYwaE1iQk84WU5LVXRv?=
 =?utf-8?B?d2hOVWhBdWRacE5xRmtWU2ZoaXpLZ0FhNG1ESHEzTkZBTDdrUFkxUGs3VUx4?=
 =?utf-8?B?QkhjSWRYNjMyZmpYZ2Q4ZFhyTkMwa2UzVnl2VjhMU1N5Q3EwUW15Z1RLRnhK?=
 =?utf-8?B?TkRKVEEzNm1iQ3RTZU5SZFJGelhLWktoOXBWSUg2NGtYdTVibWhMbmpZWVR1?=
 =?utf-8?B?cHZCYnpab3hwQnJHdmx5U1RURDdyUzhuZ3NsZEhLcmdtVzd3cEswRG5jL2NQ?=
 =?utf-8?B?ei9xanZhb0szMjJ5K2wxR2dMSzVGSkNSV2VJU3pneDNCQnlZT00rVi83VTNB?=
 =?utf-8?B?VWV0azA2YVZKY1dFOFNRM1NGeVIzNXM3TXFkU3Vjais5QjdPNldaR3BReVlv?=
 =?utf-8?B?V0FpWE93TDU3TWQ2NFVYV2pxMDB6WVh1TDhJcWkwb2pTeDNUUEVzL3lRaGVu?=
 =?utf-8?B?R0dWb3ZuM0RYSTFLWWxZUjhIV3haeHpselhYUjIraGgvZ1BGQzY1SnZIUEtr?=
 =?utf-8?B?NGZCUy85c09lOUtoSm5BbUJ0SkhJbDAxVml5c21OYlNPVGd0cWs4MWdPSVd6?=
 =?utf-8?B?UTN6bXo0YnVvNExYZmlJOXY5R3dDTEl1Z2Q5TnRRYUVtREVqTlVtUFhxZE9o?=
 =?utf-8?Q?6NHhOXGak57oqsNs3BFtARsfw?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: afeec0b5-7619-4ab4-7804-08dc43bf4ab1
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR12MB6686.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Mar 2024 00:40:05.2985
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: emgudeJNm9yuR6ykX9tqf7x6ifD7Uo3hV82QTD4trhjMMeP8ErkY9bfLdJ9XsxcS6p2zUVaiF4e4ip5Bkk0rUg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB5883


On 23-01-2024 04:30, Bjorn Helgaas wrote:
> External email: Use caution opening links or attachments
>
>
> On Tue, Jan 16, 2024 at 08:02:58PM +0530, Vidya Sagar wrote:
>> The enumeration process leaves the 'Received Master Abort' bit set in
>> the Secondary Status Register of the downstream port in the following
>> scenarios.
>>
>> (1) The device connected to the downstream port has ARI capability
>>      and that makes the kernel set the 'ARI Forwarding Enable' bit in
>>      the Device Control 2 Register of the downstream port. This
>>      effectively makes the downstream port forward the configuration
>>      requests targeting the devices downstream of it, even though they
>>      don't exist in reality. It causes the downstream devices return
>>      completions with UR set in the status in turn causing 'Received
>>      Master Abort' bit set.
>>
>>      In contrast, if the downstream device doesn't have ARI capability,
>>      the 'ARI Forwarding Enable' bit in the downstream port is not set
>>      and any configuration requests targeting the downstream devices
>>      that don't exist are terminated (section 6.13 of PCI Express Base
>>      6.0 spec) in the downstream port itself resulting in no change of
>>      the 'Received Master Abort' bit.
>>
>> (2) A PCIe switch is connected to the downstream port and when the
>>      enumeration flow tries to explore the presence of devices that
>>      don't really exist downstream of the switch, the downstream
>>      port receives the completions with UR set causing the 'Received
>>      Master Abort' bit set.
> Are these the only possible ways this error is logged?  I expected
> them to be logged when we enumerate below a Root Port that has nothing
> attached, for example.
In this case, there won't be any TLP sent downstream. I talked about 
this scenario in the
second paragraph of point (1) above.
> Does clearing them in pci_scan_bridge_extend() cover all ways this
> error might be logged during enumeration?  I can't remember whether
> all enumeration goes through this path.
So far in my testing, clearing it in pci_scan_bridge_extend() covers all 
the cases.

>> Clear 'Received Master Abort' bit to keep the bridge device in a clean
>> state post enumeration.
>>
>> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
>> ---
>> V2:
>> * Changed commit message based on Bjorn's feedback
>>
>>   drivers/pci/probe.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> index 795534589b98..640d2871b061 100644
>> --- a/drivers/pci/probe.c
>> +++ b/drivers/pci/probe.c
>> @@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,
>>        }
>>
>>   out:
>> +     /* Clear errors in the Secondary Status Register */
>> +     pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
>> +
>>        pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
>>
>>        pm_runtime_put(&dev->dev);
>> --
>> 2.25.1
>>