Received-SPF: pass (google.com: domain of linux-kernel+bounces-29659-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <b746288b-32f0-4f08-8c9a-6adbe195d429@nvidia.com>
Date: Thu, 18 Jan 2024 07:57:10 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH V3] PCI: pciehp: Disable ACS Source Validation during
 hot-remove
Content-Language: en-US
From: Vidya Sagar <vidyas@nvidia.com>
To: Lukas Wunner <lukas@wunner.de>, bhelgaas@google.com
Cc: alex.williamson@redhat.com, treding@nvidia.com, jonathanh@nvidia.com,
 linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, vsethi@nvidia.com,
 kthota@nvidia.com, mmaddireddy@nvidia.com, sagar.tv@gmail.com
References: <20230111190533.29979-1-vidyas@nvidia.com>
 <20230730191519.3124390-1-vidyas@nvidia.com>
 <20230730194026.GA19962@wunner.de>
 <7880aa6e-9fc5-c026-138f-42bef3c48b69@nvidia.com>
 <20230731195930.GA13000@wunner.de>
 <f7324ca4-2c57-459c-a9e8-aac09ba65e87@nvidia.com>
 <20240108141901.GA17779@wunner.de>
 <7345c2d2-5446-49a6-9ceb-0f1b9ee4ec18@nvidia.com>
In-Reply-To: <7345c2d2-5446-49a6-9ceb-0f1b9ee4ec18@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?ejdYRHBUWjNLSWhTN3lEbDZRcTNQVmhZN0xFT01zUTI0Y05EWWY2K2oreis5?=
 =?utf-8?B?eHBmMTg3QXh2cDdGaDBub0tGZkdObGNSYVhvTlZGUGIzWk9iMjc3R1hkRXBL?=
 =?utf-8?B?dS9taDd1QWRlOXNWam9wd3pxRFltdUowQjRsNlhmZjE4NXFMV3FFUTdUOUNR?=
 =?utf-8?B?YzBMQTBIbjBpd1RhRks1QmNMMmhlakpZMDZ1TXdXZ2RkcVF4dWwxQzYxT1JB?=
 =?utf-8?B?ek9OaHl6MHhPRUZjMFVPL0tDejQ3NlpXTTNOYzFLRFRoR3RFU1hSQzNDVWxM?=
 =?utf-8?B?NVVQbCtSaCs0UGkwTDJWUi9haElKOXNiVUVGQWs3azhWNkdkV29DTnFQUWFp?=
 =?utf-8?B?VWdwSDAyQkFEUmZra29ZVDdCUVViVVk4RWZtVGNBSWM4OGlBcHN0N0JaczIy?=
 =?utf-8?B?SkZISjFYb3B2eHRhdDJqZHRSc1BzQTU4LytkL3lQZW5JSE1JbkIvcWNXdHpX?=
 =?utf-8?B?SDRMVWVHRmlxMmlJNGZseStZZkZDZkZsSWtIc2lFcjhwN0RmdW80bkczcVQ0?=
 =?utf-8?B?QUU1ckRSbEhXU2hzSzByUnhaRkVXOHhGclNKUitNby8zSmN3eTZ2K0Fqb0R6?=
 =?utf-8?B?bFhhVmRHaXNCY1EzakdaQ0hjY2srOFIzZTltdnRDd2NRWCtKK1RvVk0ycWFY?=
 =?utf-8?B?c1dhMVhDMlh6aE5VOG1kcTZVL2NLbXpta2pqWUF1Q1pHanFxR3U0RitoRCs2?=
 =?utf-8?B?dVMrMUFXREZtSGFWYTZBb2RUVFJZayt5ZTUydHJWbXdGU3lsRjNrbzM2WkNr?=
 =?utf-8?B?MUdjWE1Bck5lcEFEUG9iUTM2OWxmditTZ0t0UHlvdTVsWndLYnFsUkZxTHJL?=
 =?utf-8?B?WVd3anFhdXA4RXc2azVNdzJ0VWJLZHVQeUZlcWVSeUxaVGNIa0t4Z0ZIWkh3?=
 =?utf-8?B?cDBob0hFa3hMZmtWNDYzb3dGZGVoVWhIMGo1UzRuRExvL21SK3J0U0xmSjBl?=
 =?utf-8?B?VW9UZDg3VzNsR0NwMzZ6Rjd4YlN2MFJCOHh2VEFack5rdEZyek94QzREOXNO?=
 =?utf-8?B?bzliWkxXeHpkem40eUFYT0JFVTRFK2ZBQUZ2OUhxaStvR2cwWWJweUNxRXAy?=
 =?utf-8?B?Wi9ZalNzTGxram5zNWlhSHVBWENINTg1QU5QOG1sczRtelpqWDdsTU1xc2pu?=
 =?utf-8?B?dkM4YW1od3hWcTdmekFuVm1LNUpvRGU0OWZCaUVTQW1TbjZaV1JuSVZURmdP?=
 =?utf-8?B?cERMNEV6MG1mVi9xUUVqU09oTnFDUlhycGJCNmhTU0UwSEt4cWtpYkIxbGR2?=
 =?utf-8?B?TncySEJXSGY1SDU1VzhCcU1FNUFMUGc1dDNFaGVleWlvVEFJZnd6WHhTVDNH?=
 =?utf-8?B?cDJEYU5LKzZnWGZCZ1F4dzIrcVQvL3dDc2ZIUmpwU2RKM0R6NVZGZlk1OFlo?=
 =?utf-8?B?ZDNhUXZlTWVXb29VRkdtYzdnQVltT2JLTVEzaVppbkUvTW04WGxhNitLaTlp?=
 =?utf-8?B?cUw5bU5YUDYyd2dWTFhxKzZnek5mRit4Q2p1a1dHTkFzczNBYy84c09FalJB?=
 =?utf-8?B?aHpuWlg5dnViQk04MFVOY3o2WGJwY2Y0MjlhMGZ3RTZxUk9TaE9IMitsbEpS?=
 =?utf-8?B?WithZkE1aHR3TWtWMUtrQ1ljN3V3QTIreFhjOG5KaWsvdkYvUXZjVW9uZ0Jv?=
 =?utf-8?B?SWNTaHdjT1FHczVEbGZLSk9iZllhZU5XRnErS0ZBeVAybnlyZTBaenJkeTMr?=
 =?utf-8?B?SEdPQ0pEOFRIQVlWcnRVbmV1QmtKSjY2UFJTQWxWM3Q2Y3dOYVdsMDZYdFUy?=
 =?utf-8?B?UzFWQnVNNDBCdXd6SUVNZ1ZCWXh3MHdreUJkNytORDhiRmgzRmk1dG9lbFMz?=
 =?utf-8?B?NzdYeW9nVVloRWlyUGlRUzBhT0p4ME1FbVdtZGI4ek5MV3FYa3FydHp4TVB3?=
 =?utf-8?B?L29TcHU5c1Y5L2NNVVBLNFFYdzdmNktCR05IMnRSRE1UYVlKWG9EdWhmZDdC?=
 =?utf-8?B?QWx0R2NCUzR4TWs3R0M0QnJnV01hWlJ3VVhnaTlrWGhmZTJZL213SnNGZW5h?=
 =?utf-8?B?MldmVnFpMnlOblFiN3VJREpzVTRBR0xNWDFaUk9BTFdEMm0yeGdwY0hSY1Jl?=
 =?utf-8?B?SGxkbDErU2J1U3RzWElya25PRytoM0xOVXoya2NUdXl1WWI5VTBiQmN2K0gy?=
 =?utf-8?Q?gDmji3Czp8v8UKJERBXuprcgE?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2e99be52-8b2a-41cd-12c7-08dc17cd019f
X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB6674.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jan 2024 02:27:24.8041
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: VqI5FTBJQ8C4brX3kEkKNwQNf8yGUpPQYXxULUDBSAMM+13FUQJrFK2JmDnysAjKJevL2uNrEHXL03tjAluMxA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6267

Hi Lucas/Bjorn, any thoughts on this?

On 1/11/2024 7:14 PM, Vidya Sagar wrote:
> 
> 
> On 1/8/2024 7:49 PM, Lukas Wunner wrote:
>> External email: Use caution opening links or attachments
>>
>>
>> On Thu, Jan 04, 2024 at 08:01:06PM +0530, Vidya Sagar wrote:
>>> On 8/1/2023 1:29 AM, Lukas Wunner wrote:
>>>> As an alternative to disabling ACS, have you explored masking ACS
>>>> Violations (PCI_ERR_UNC_ACSV) upon de-enumeration of a device and
>>>> unmasking them after assignment of a bus number?
>>>
>>> I explored this option and it seemed to work as expected. But, the issue
>>> is that this works only if the AER registers are owned by the OS. If the
>>> AER registers are owned by the firmware (i.e. Firmware-First approach of
>>> handling the errors), OS is not supposed to access the AER registers and
>>> there is no indication from the OS to the firmware as to when the
>>> enumeration is completed and time is apt to unmask the ACSViolation
>>> errors in the AER's Uncorrectable Error Mask register.
>>> Any thoughts on accommodating the Firmware-First approach also?
>>
>> Are you actually using firmware-controlled AER or is it a theoretical
>> question?
> Yes. We indeed have a system with Firmware-Controlled AER.
> 
>>
>> PCI Firmware Spec r3.3 sec 4.6.12 talks about a _DSM to disable DPC
>> on surprise-hotplug-capable ports.  Maybe that would be an option?
> It looks like this _DSM is totally dependent on the port having SFI
> capability implemented and unfortunately our system doesn't have
> SFI implemented.
> 
>>
>> BTW what happens if the system resumes from sleep and a device in
>> a hotplug-capable port doesn't have a bus number configured yet
>> (because it's been powered off and is now in D0uninitialized state)?
> Theoretically the answer seems to be yes, but, since the platform we
> have is a server platform, there is no support for sleep and resume on
> this platform and hence can't really confirm this behavior though.
> 
>> Could the ACS Violations then occur as well?  Do we have to mask
>> ACS Violations *generally* on Root Ports and Downstream Ports when
>> going to system sleep and unmask them after setting a bus number
>> in the attached device on resume?  And I suppose that would not
>> only be necessary for hotplug ports?
> Again, how to do that in a system where AER is not handled natively in
> the OS? AFAIU, there is no mechanism for the OS to inform about the time
> it updates the bus number.
> 
>>
>> Thanks,
>>
>> Lukas