Received-SPF: pass (google.com: domain of linux-kernel+bounces-149922-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <1427a905-1d01-4b90-8af5-acd1a7f5b1d7@nvidia.com>
Date: Thu, 18 Apr 2024 16:22:43 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH V2] PCI: Clear errors logged in Secondary Status Register
From: Vidya Sagar <vidyas@nvidia.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: bhelgaas@google.com, linux-pci@vger.kernel.org,
 linux-kernel@vger.kernel.org, treding@nvidia.com, jonathanh@nvidia.com,
 kthota@nvidia.com, mmaddireddy@nvidia.com, sagar.tv@gmail.com
References: <20240122230026.GA290856@bhelgaas>
 <d93a3f29-b260-4910-aaf5-d734e6242223@nvidia.com>
 <0c948351-9715-4c5c-ad0c-3727cd2ba8a8@nvidia.com>
Content-Language: en-US
In-Reply-To: <0c948351-9715-4c5c-ad0c-3727cd2ba8a8@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?RUx2eE53cjJoVU9QL0toUzRKck5ydTFtNUhGcUhLNlhhR2tGeWFpT1hOUmpj?=
 =?utf-8?B?MTZnVnpiZG9XY0xOY21PcVk0TUJRSnJ3R0tYd0xrMEptK1cxWDkxU2pqZ1Bu?=
 =?utf-8?B?SU42dXB5WDdkWmNtckF5QXVmVFV5VUQ3OXp1Mk1obGU4KytRYkNZSU1LSHYr?=
 =?utf-8?B?VnF3a3NLTWxtTC9GazVpYy9QclpOVE5pR2ZtUHRLNzhoSXBVZVV5Ti9NUkNU?=
 =?utf-8?B?NWE0cGNFOXdsS3lrR3pWakY0LzVjYVNnZzRzV1dEM2RjV3VCNnJDMDYxdnhn?=
 =?utf-8?B?Wnc0ZkVBbVBTWGM0UlVETitmM2VveklLT2VLOGpiZWVxY1lNQmhGTU1xdlRs?=
 =?utf-8?B?YkxwcURvTUVvRm9mTWhmS3lIL3ZZNzdCS0U2VTJxS0Nwc3h3Vy9LeGhhZDR2?=
 =?utf-8?B?M1VpY0srREk2NlFxekNqM3ZFVjZVUDhmbVJ2QVBOdldUS1dMRDdGbkRzMHEw?=
 =?utf-8?B?dkl5cVY2NlBpQ1FDN0RkRmJCYzcwa2dpVG1hYXNYeHRianMrc0FqQ2N1ZTQ0?=
 =?utf-8?B?VWVMb1NnOXB4SmkwUFhrSUhvN2Vmc2dBc2ZPZENEc0JQc1JVQVZSSTJ2MTdJ?=
 =?utf-8?B?T1dWVkx4NmJYeUc0YnUxMUhDY20vYWtuNmNOd0xNa1hQaVRiRTYreFZqUU9I?=
 =?utf-8?B?dFB2QWpoMEdKWFBIa0h2ODVVL0U2M2pXM2s1dkZ4OFJpSTBOWEhYbGhhdnV3?=
 =?utf-8?B?Vk1RRVJ3NlhRUnlPZzdldFY2aWs4MHc4QmlFUFNzdUxMS0FOSWIxUFNVbEdQ?=
 =?utf-8?B?bE9qdzNOM3Jnek1aYXV6RFdVM1d0VG5obHlzS05FeVZsVGFpUTErV24vY0Ny?=
 =?utf-8?B?OFBsYmFCc1BvN0lJdHZwVHd0dHFlK20yS2VsRlM1ekpMT2VweXdJeHFPNmxv?=
 =?utf-8?B?dlNUK2svb1crSjE5NjFiRWM0bVljUmZOcnFvZDJGMnBLeitBMzhMMFdkek0r?=
 =?utf-8?B?eHFkZExvaEJSSWZNOHZuczZZcTFjMjYrV2M1ejR2WDgzNnZzYkNuem9yMzY4?=
 =?utf-8?B?b0NUYldwS0pWT25vamVWZUtBbmc4dGhPRGtEZ0VBZldrUHBpT2NwTGNVQy85?=
 =?utf-8?B?UzgwVWY5NkxXRHVUNlB4SytNSjg1Rk9pKy9Qb1ZoamlQT3dQWFB1a24zUmNT?=
 =?utf-8?B?VXZLZjh3b3Z0cTYzV0d0L1JLblBtZE5qMk83bzhLNHBvblhFUG1MU0xkNDZ4?=
 =?utf-8?B?b3Q2NGY0anBYcFZDNUZ3Qy9aN3UyZkY3cjhUYmRwNUJaZjY1NnoyZzNWMnpw?=
 =?utf-8?B?QVc0aFNqLzJaT2JqOUNwd2FQS2l2ZFNtWExCVEE0RmIzbHpWSVNaZmJHd3BC?=
 =?utf-8?B?Z3UxRlYrRTM5dDd5YVZ5dEhPTEVIbDZOUGp0aDdoYVY0dDNEc1JoaUUzOUlQ?=
 =?utf-8?B?ZlBCU3VkWDJzMXc1QlRTeG40SDIvTlAxT21kV05uRWxsbzRZdk5vclJwUFhD?=
 =?utf-8?B?RkhjQ2hweVdodzh6VFpPQUpwOUZ6YUVWOHNDaEhrQWpCTEIxQmhxNFc0b2s3?=
 =?utf-8?B?MGJ2d05iQlllNFY5KzQrSm40S3c1OStaL3JMRU8xYktJNk1tbnl2cXBzYkJr?=
 =?utf-8?B?YzRWOGs0TTVOUG81WUV4OHUwNXg0NDBKcU84WkhCYjd5MnBtSlZSU1RxVUNz?=
 =?utf-8?B?aExXTGgxWHpmeGNuN25LbkF4eTNLZDcyZDJ2ZXJDazZmaENwMHhaRnIvNzh0?=
 =?utf-8?B?Qlo1VzhoTG5kd2NqOTd2SG1mcEhwTmVKLzRpT2JsM1VoTUppcm4rVGNYbEtE?=
 =?utf-8?B?QzZXTE1Ya1NKNUFXKzZPRndHMk1SUDJpVkxSbFR3Q2pvaUxsNm9BM2h5MWtl?=
 =?utf-8?B?bnNkN2hWSzZESmdpSlZEWEhoeFZHckhjTitwRTYrMGNUYjA5U3BZa2tmdHJI?=
 =?utf-8?B?NUdrUzJSNHhJSjA3VzdTSm01NWdOb1F0dGd1M2J0SVEzZ2M1RVkyeDFMakN4?=
 =?utf-8?B?U3dESWpOVldSM1phMDM3dDdxNjJvTkllSmdTNXhTVkNMVEhkenBrOE0rRjJs?=
 =?utf-8?B?VU9vWGV1ckpmK1Y5YklwSEVEVjRMS1RnSi9oUDd4eW0veVFzRjBMckJiWHhR?=
 =?utf-8?B?bnhEL2hIcktGWGJXQVl6MnRySVV4QVRSZ1pEam9ZODNlR3pwU3g0RjJ4clNj?=
 =?utf-8?Q?qu2+25vPddTmIeqSf8g3vD2vj?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 73cd9b77-eb02-4737-abc8-08dc5f95b1e1
X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB6674.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Apr 2024 10:52:52.2346
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Vmvzy1yR+ADUmOqb7ccaQ04JZxADCoyPzncnoa8+ReGrFe3uJcOP92xW+2lT+IoP/XPr+j8+Qp3lAHnmUL5BkQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7520

Hi Bjorn,
Sorry to bug you.
Is this change good to be accepted?

Thanks,
Vidya Sagar

On 01-04-2024 13:29, Vidya Sagar wrote:
> Hi Bjorn,
> Just checking on this thread.
> Is there anything else you want me to clarify on?
>
> Thanks,
> Vidya Sagar
>
> On 14-03-2024 06:09, Vidya Sagar wrote:
>>
>>
>> On 23-01-2024 04:30, Bjorn Helgaas wrote:
>>> External email: Use caution opening links or attachments
>>>
>>>
>>> On Tue, Jan 16, 2024 at 08:02:58PM +0530, Vidya Sagar wrote:
>>>> The enumeration process leaves the 'Received Master Abort' bit set in
>>>> the Secondary Status Register of the downstream port in the following
>>>> scenarios.
>>>>
>>>> (1) The device connected to the downstream port has ARI capability
>>>>      and that makes the kernel set the 'ARI Forwarding Enable' bit in
>>>>      the Device Control 2 Register of the downstream port. This
>>>>      effectively makes the downstream port forward the configuration
>>>>      requests targeting the devices downstream of it, even though they
>>>>      don't exist in reality. It causes the downstream devices return
>>>>      completions with UR set in the status in turn causing 'Received
>>>>      Master Abort' bit set.
>>>>
>>>>      In contrast, if the downstream device doesn't have ARI capability,
>>>>      the 'ARI Forwarding Enable' bit in the downstream port is not set
>>>>      and any configuration requests targeting the downstream devices
>>>>      that don't exist are terminated (section 6.13 of PCI Express Base
>>>>      6.0 spec) in the downstream port itself resulting in no change of
>>>>      the 'Received Master Abort' bit.
>>>>
>>>> (2) A PCIe switch is connected to the downstream port and when the
>>>>      enumeration flow tries to explore the presence of devices that
>>>>      don't really exist downstream of the switch, the downstream
>>>>      port receives the completions with UR set causing the 'Received
>>>>      Master Abort' bit set.
>>> Are these the only possible ways this error is logged?  I expected
>>> them to be logged when we enumerate below a Root Port that has nothing
>>> attached, for example.
>> In this case, there won't be any TLP sent downstream. I talked about this 
>> scenario in the
>> second paragraph of point (1) above.
>>> Does clearing them in pci_scan_bridge_extend() cover all ways this
>>> error might be logged during enumeration?  I can't remember whether
>>> all enumeration goes through this path.
>> So far in my testing, clearing it in pci_scan_bridge_extend() covers all the 
>> cases.
>>
>>>> Clear 'Received Master Abort' bit to keep the bridge device in a clean
>>>> state post enumeration.
>>>>
>>>> Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
>>>> ---
>>>> V2:
>>>> * Changed commit message based on Bjorn's feedback
>>>>
>>>>   drivers/pci/probe.c | 3 +++
>>>>   1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>>>> index 795534589b98..640d2871b061 100644
>>>> --- a/drivers/pci/probe.c
>>>> +++ b/drivers/pci/probe.c
>>>> @@ -1470,6 +1470,9 @@ static int pci_scan_bridge_extend(struct pci_bus 
>>>> *bus, struct pci_dev *dev,
>>>>        }
>>>>
>>>>   out:
>>>> +     /* Clear errors in the Secondary Status Register */
>>>> +     pci_write_config_word(dev, PCI_SEC_STATUS, 0xffff);
>>>> +
>>>>        pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bctl);
>>>>
>>>>        pm_runtime_put(&dev->dev);
>>>> -- 
>>>> 2.25.1
>>>>
>>
>