Received-SPF: pass (google.com: domain of linux-kernel+bounces-23675-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <7345c2d2-5446-49a6-9ceb-0f1b9ee4ec18@nvidia.com>
Date: Thu, 11 Jan 2024 19:14:54 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH V3] PCI: pciehp: Disable ACS Source Validation during
 hot-remove
Content-Language: en-US
To: Lukas Wunner <lukas@wunner.de>
Cc: bhelgaas@google.com, alex.williamson@redhat.com, treding@nvidia.com,
 jonathanh@nvidia.com, linux-pci@vger.kernel.org,
 linux-kernel@vger.kernel.org, vsethi@nvidia.com, kthota@nvidia.com,
 mmaddireddy@nvidia.com, sagar.tv@gmail.com
References: <20230111190533.29979-1-vidyas@nvidia.com>
 <20230730191519.3124390-1-vidyas@nvidia.com>
 <20230730194026.GA19962@wunner.de>
 <7880aa6e-9fc5-c026-138f-42bef3c48b69@nvidia.com>
 <20230731195930.GA13000@wunner.de>
 <f7324ca4-2c57-459c-a9e8-aac09ba65e87@nvidia.com>
 <20240108141901.GA17779@wunner.de>
From: Vidya Sagar <vidyas@nvidia.com>
In-Reply-To: <20240108141901.GA17779@wunner.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?cjdMeDJMZkhJSXN5cjdscTFDNEM3UUdZanBBSmRFYjkvQ3NTREc2VW83dk5t?=
 =?utf-8?B?RW9OWjBjODd6ZzY4b1NoN1h5WFdzNitUM25naEsrSkJxTndmVUNyQVBNakpG?=
 =?utf-8?B?TnlVdWsyOGZqbURlczZDVEhXalpxaXNRZnVZY0tUNjhWbGpBV2REa2I2dWVO?=
 =?utf-8?B?VzFXTXh3OUxMbjdMV1dQaDcwaG12YkUzT3RPWjUxZFVVUG9FbEtWREJwOWxQ?=
 =?utf-8?B?VHUxL0R6c1UxNC9OZTIvMU42TklZeUlXcTdSQ2lGa3J1MzNvUjFwRDdGR3dl?=
 =?utf-8?B?aGIrUFl6WkZTektUQ0RkaHJJcndCMS9EUW8ydnlKTXFTNDArNzc2bElHblJp?=
 =?utf-8?B?R3lzYzRwb0ptaU1MNFJObWlSMHR0SWhya2Vaa0N2eUV5V3ZRRmRnUzR6YmV3?=
 =?utf-8?B?aHA5MEo4ZEd1MHBpalhvTnpNTlBJL0RWVmJmaVVPZlRRMnlrY3gvUmdNUUpY?=
 =?utf-8?B?WUdiTGZ4VXljZ05JUmoxVS9PNWhvaUczTUVSbGsvKzVoYjJVcjNwL2k4UXky?=
 =?utf-8?B?aGFBQlB5Z1ZXWjU3TlNhWkMrR250TkxJSlMyaGdkaGF4ZHZ0TVNod25LTGZD?=
 =?utf-8?B?MVI2Y1RkT3pFZ3VISG9QdG0wcW1sZWVzcklSd28wQjBVait2dUtnQkk4c3Rz?=
 =?utf-8?B?bXBENzg4OXNKNVdiRkpkdEd2aGlWUlV5L1c5YWFDazBmclAvZExNUE1xckp6?=
 =?utf-8?B?VXlxNjQza2RtcVkwcFlCbmZ0RUZOWk1Pb0ZydGxpVjFvQ3hqUmJBVHdaTHhZ?=
 =?utf-8?B?WjBtMzQ3QW9jLzQ2ZGRuTk93VWZyMHZBOFFlMmdhMUVmY3FYdENvWERUMDdD?=
 =?utf-8?B?dXdoT01sOHo2LzdLQmV2T3Z0Z3VaZkdjYkFQTk8vWklhc3dhUmZTakhpSGlX?=
 =?utf-8?B?OVFoRnVLRWFMYUpVekR2dVRYckhZMnJWWVVaTCtCdzFtV3MrYUp0REVES2R0?=
 =?utf-8?B?aENEMHQxOE1xSk5zRXo2Tk5ycU8wTWYvZ3pvRFJHSmw3NElYQk84R0h6cEQ0?=
 =?utf-8?B?bG85QnFNSnhlMjVWcTNpYkRoaG90d3JJVXF1N1J0a0d0Mk9TMmRLWExIT0Rp?=
 =?utf-8?B?c2RWdTRmWU5UWUNlSFVMYWY3K1ljZXAvZ01MTWsxSmIrTGZ2a3VjeVJqVTRa?=
 =?utf-8?B?N0x5em1tRXkwdlJuY2hFYWU0SXZKYnpQSHZWcHZYOUdnMGNrVWg4TW5ROTMx?=
 =?utf-8?B?TUJac1d6SndEWHBSK3dGd3NQYno5UUNxeHZTcFlPcjVpSVBuamNXVS95bEpI?=
 =?utf-8?B?MGFyc0lKZjZETEI5dUZHSC9XYmd0ZCt3bEc1U0tpa01mdlFYek1YeWFkcU9v?=
 =?utf-8?B?NTd6clUxaTBiaFhRbklzczY2MStFQi9sSllRSHJzdjQ2WXJvN0xGUlNFdlJO?=
 =?utf-8?B?QjZucmNKQi9obTVlSHZ0K3B4QVEvWm9DZFlGdUs1WkcrWnIzV2hnWTdqWDMw?=
 =?utf-8?B?Ynowb2RoUFFZN3ZUQTlzWnJCUXVTSW1qZU5VKzU2dERtZFBxcmJ0Lys4ZkRL?=
 =?utf-8?B?Z0pTRmE3SkljRllWZmZFbjhpWEY0c21wUjRaZldYVU5hbDVxZlRleFdhREJh?=
 =?utf-8?B?ajBza1cwck9oVGVONDZDQ2x4NGU1V2thVzlBWFMyR2dsckVJZHNLWnVzTERX?=
 =?utf-8?B?V3cvWmRpTzRkSDNnajNIQ2lFN0RvenVpR0wrMU1kdkE2WWY0azE0RUtaY0lV?=
 =?utf-8?B?Y0NMU2tKNWQ4UER3Zyt1SjlMdERudlBRUlNkVVYvSDQ2VHp5YUR4ZG9rVjBO?=
 =?utf-8?B?WjFXNXRwUVhmU0xUY2k3RlJkUTlaNjQxTlZIYVRTV1RkN3hYT04yc0ppTkdQ?=
 =?utf-8?B?YnFWcmtzaEZ6NXNTdmUyczFoQTFrb1RrUXpiOGxJMC8xMkUzOVYyTHVNYmxT?=
 =?utf-8?B?V0VkcCs1SU1QYWdWUlFjdzc5c3dVOEttZE45R3hMendNaEFGOTkvZjhVYXR6?=
 =?utf-8?B?WmU5Y2xHQXR5cm9zNEZRSldtYUpZZlJRbWVFZmVEN0srcXhZM3lZOW5IZ3hi?=
 =?utf-8?B?RFNvSE8ySTJzN0JGdmgyc01GNUNHWUZkRDhVRFA4eTMvdk5yVUVMaytkcm1s?=
 =?utf-8?B?SnlQZUhJS2hFOUVsZDlXUmVpdU03WlVxQU1jdk5TdzFMRStGNGVPdkQ3NHJN?=
 =?utf-8?Q?XuryX1vdTlA1cnZuBcf6Ek2Ye?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9e5bfb75-1626-4268-46ee-08dc12ab8392
X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB6674.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2024 13:45:03.9840
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RR+xawGAqBtiWpdQYKAzauZJgJ77TOWZdMqD6e905EzDG3RqKbruJ2TQ7Kvn73HfOsd8oB967H5UGLaicmiAjw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4591


On 1/8/2024 7:49 PM, Lukas Wunner wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Thu, Jan 04, 2024 at 08:01:06PM +0530, Vidya Sagar wrote:
>> On 8/1/2023 1:29 AM, Lukas Wunner wrote:
>>> As an alternative to disabling ACS, have you explored masking ACS
>>> Violations (PCI_ERR_UNC_ACSV) upon de-enumeration of a device and
>>> unmasking them after assignment of a bus number?
>>
>> I explored this option and it seemed to work as expected. But, the issue
>> is that this works only if the AER registers are owned by the OS. If the
>> AER registers are owned by the firmware (i.e. Firmware-First approach of
>> handling the errors), OS is not supposed to access the AER registers and
>> there is no indication from the OS to the firmware as to when the
>> enumeration is completed and time is apt to unmask the ACSViolation
>> errors in the AER's Uncorrectable Error Mask register.
>> Any thoughts on accommodating the Firmware-First approach also?
> 
> Are you actually using firmware-controlled AER or is it a theoretical
> question?
Yes. We indeed have a system with Firmware-Controlled AER.

> 
> PCI Firmware Spec r3.3 sec 4.6.12 talks about a _DSM to disable DPC
> on surprise-hotplug-capable ports.  Maybe that would be an option?
It looks like this _DSM is totally dependent on the port having SFI
capability implemented and unfortunately our system doesn't have
SFI implemented.

> 
> BTW what happens if the system resumes from sleep and a device in
> a hotplug-capable port doesn't have a bus number configured yet
> (because it's been powered off and is now in D0uninitialized state)?
Theoretically the answer seems to be yes, but, since the platform we
have is a server platform, there is no support for sleep and resume on
this platform and hence can't really confirm this behavior though.

> Could the ACS Violations then occur as well?  Do we have to mask
> ACS Violations *generally* on Root Ports and Downstream Ports when
> going to system sleep and unmask them after setting a bus number
> in the attached device on resume?  And I suppose that would not
> only be necessary for hotplug ports?
Again, how to do that in a system where AER is not handled natively in
the OS? AFAIU, there is no mechanism for the OS to inform about the time
it updates the bus number.

> 
> Thanks,
> 
> Lukas