Received-SPF: pass (google.com: domain of linux-kernel+bounces-153552-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <84eb39a1-60dd-40cb-8c7c-dfd9a988732d@amd.com>
Date: Mon, 22 Apr 2024 09:34:05 -0500
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH 1/6] PCI/RCEC: Introduce pcie_walk_rcec_all()
Content-Language: en-US
To: Dan Williams <dan.j.williams@intel.com>, Li Ming <ming4.li@intel.com>,
 rrichter@amd.com
Cc: linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org
References: <20240313083602.239201-1-ming4.li@intel.com>
 <20240313083602.239201-2-ming4.li@intel.com>
 <d69c2157-a0da-4d8c-8684-d42afd285191@amd.com>
 <661e00eb808e4_4d56129429@dwillia2-mobl3.amr.corp.intel.com.notmuch>
From: Terry Bowman <Terry.Bowman@amd.com>
In-Reply-To: <661e00eb808e4_4d56129429@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?YWZNakpyamxEOTZwc2gxRTMydEhmTHVJaUJDWk9XdzhvK0YvZUw0TVFIK0Fu?=
 =?utf-8?B?ekc1S1Y2Z2haWHlUb2VXM3QzUlBMYUlCVkpDR3FHaXVoYnN6akdZZzRVdFAy?=
 =?utf-8?B?Sys2WmtkT0RySWhmSEQ3ckgzMTkzb1hlNWdZSzRXSEhiUkoreGhwN0xpNHUv?=
 =?utf-8?B?dFFydXNPdjkxZi9mTGlKT0tLT1pSYVdMMUYzV0VPRVI4T25wSDB1TnYzYVMw?=
 =?utf-8?B?STZFWE0rbzJyUUhVUGdpaUdVR0U2RUdscG1MU3gwby9VSGFuTi9hdnVOLy9Y?=
 =?utf-8?B?TlRCOTVZK0I5T1gwMkxjT1gzcjdXeHBpbUk0bUFXcGxFbHFTcFdTekJXYXQy?=
 =?utf-8?B?QWQ3ZGRCOGNBR1BhY2ZTMldiekdpV3pQcTZJTGdNRFd2WHYxOWpUY0grRnVv?=
 =?utf-8?B?d2JSVWVmY01JRnRkMytZTVdJaVVTckQrQU02bGhhTWE0MjdiL0lHZnNDSDRa?=
 =?utf-8?B?R00yUWNZbkR0V2EyY3JYUWF5Yk1kNnNQcTdlU3dRY1pDUWdaVGZXRG16RlMx?=
 =?utf-8?B?UzhOb0lsOXRXVU1YZkpuQUtvWCtPUFQybW0vWllLWmVZUDI4bXNRbTloSHhI?=
 =?utf-8?B?TER1bXNpVmlnTktDQWJuSCtGU3VENXA3cWpLanV4aG9uMWdmeEdFaURoYklM?=
 =?utf-8?B?ZEtEdFdhOUQzZHdlZUNNaDl4T2p1MUF5SVlkcGdMMWtlWHJjL2xjUWcvVm5G?=
 =?utf-8?B?TW9uYnJ3V2lKc1E0akNpUXZaaTVxZCtWWk9kdjl5b0dTTDBQem9KaVk3S0hk?=
 =?utf-8?B?dFZtVDkrcEhsOGo4eElwOUpINndBOFA2bDUrTVlId2ZuNHZvSjBaSVlCKzJu?=
 =?utf-8?B?Tkk1cytVNXBPRm9nT21DNitMR0dYRUFuZ2JWdFhpOE1uRHUxOXBnR3d3WktE?=
 =?utf-8?B?T1VzQzFDYVd5MURsQks1czhhQnRMSXk3VnRLUFVTVHJMaGtyWHAyODJjQ0Yz?=
 =?utf-8?B?Ty96czBUMm5xcUdZUHJqL2tVUUt1aW5UMEF0OXZPd1VqczJna0pxSnZwMnEz?=
 =?utf-8?B?WnJPYitaZnJyY3VzZEtLWWw1aGhad1Y5NmNOTXdmSFR5WWZGTkJqUDdoWVZp?=
 =?utf-8?B?ZHpPWHVPdUFvSkRPM2NON1BGL1FIWWtEbXJ3YTVjUC9RbzZnbDNpWUYvaXkr?=
 =?utf-8?B?eVZQQXNpaCtHQW1HQ1Jxc3BsMUtEUExVQlJ6THVHMXQrWEZ4d1VkaU9OS3Mr?=
 =?utf-8?B?bjRZSjdkTEhYL3RxTXovN0w0dktmMFZTKzlSeW9LV21JUE9PTjJOTjZ3Q01H?=
 =?utf-8?B?VXM0aTBqd2dNZkNvQ1VGaGFwaGVMYjA0K3kreHpiR1crV01tRFpFam1yMVJT?=
 =?utf-8?B?Mm9HbloydlZXSkhyRWZwODIwb09CSWloZ3hXa3Y5NG9KTm5OWTJiQ2JYVUNY?=
 =?utf-8?B?SXBjTDBEb3E4MFpkS3p3MVQ4eXBIRllqcXE2SU5mcnFvWkRSWm5HTE9HOU9P?=
 =?utf-8?B?eUNzaExMYXdrS2tTSFR2bldGTElvOEs5MmY5ZGsra0k5Z1BwM0Z3cUZZUGcv?=
 =?utf-8?B?N1JyRVR6VFRTV0xoUVVNWmhCNTRWZGtiQWkzN2hQdHV4Y0VmTFd2bllGZG9P?=
 =?utf-8?B?WVAweldTQjZNSFZmU3BQaHh0eTNpVFZINlZoVnl3cUhIcjFoVUlvTFYwMnJS?=
 =?utf-8?B?aUc5cVF3OEVMWWVRSDVIRzBsZ0pDZlY5aHdWTUVvY0ZSb2UzaGZxZExEV3lp?=
 =?utf-8?B?cW9sTFJoa3JmbExxQ0VYRnpySi8wVzdBQ0dFUW8wZHAyVW1SaFZFMTNUenIw?=
 =?utf-8?B?d3JnQXhMTDR0UnRyV2EvdlViY0RwdDRmcVZtZ1ZBcmwycWt2UlIrS1BESktp?=
 =?utf-8?B?MEFxZGtqa1lqa0NEb2NwVmxSOWZnemJiMVFZbGVCb0F5bVppcVc0SUJJb3JE?=
 =?utf-8?B?NTBYcHhxNEF0N3hHVE5yanhsRTloSTRneFcwU1UwT3d5TFkraGtmTWNUK3Jl?=
 =?utf-8?B?N0huTm1IdkpUbXpPdytHKzh4NVgyYU5wNHhGdzlVbjF1MFJ5ait5ZWZtdjcx?=
 =?utf-8?B?WXN0Y2dVSWQ4UE9hUGtnUlF6alJTb201Mm1GV09HalRUTXpNYVZsd3NXOEt1?=
 =?utf-8?B?ZGNCOUllckJyd1h3aVJqYzI1YjhBbjIxMUxncWFhaEc2aytKODhxWEdweTU3?=
 =?utf-8?Q?4LNYHPMETr5TjFC62K8QORqP5?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: df033453-a221-4fe2-3f18-08dc62d94433
X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB6390.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Apr 2024 14:34:07.2630
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: eAmIQ4WtgXjV+JS04h+Km4Ottsz6qEz/FedPQu4D8bubNwERv3WptgUF4d3+eIOGCxih8iTyTxmZ8LDeF26xUQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB7743

Hi Dan,

On 4/15/24 23:39, Dan Williams wrote:
> Terry Bowman wrote:
>> Hi Li,
>>
>> I added comments below.
>>
>> On 3/13/24 03:35, Li Ming wrote:
>>> PCIe RCEC core only provides pcie_walk_rcec() to walk all RCiEP devices
>>> associating with RCEC, but CXL subsystem needs a helper function which
>>> can walk all devices in RCEC associated bus range other than RCiEPs for
>>> below RAS error case.
>>>
>>> CXL r3.1 section 12.2.2 mentions that the CXL.cachemem protocol errors
>>> detected by a CXL root port could be logged in RCEC AER Extended
>>> Capability. The recommendation solution from CXL r3.1 section 9.18.1.5
>>> is:
>>>
>>> 	"Probe all CXL Downstream Ports and determine whether they have
>>> 	logged an error in the CXL.io or CXL.cachemem status registers."
>>>
>>> The new helper function called pcie_walk_rcec_all(), CXL RAS error
>>> handler can use it to locate all CXL root ports or CXL devices in RCEC
>>> associated bus range.
>>
>> The RCEC-root port relation you mention is new to me. Typically, not in 
>> all cases, RCH-RCD has a RCEC. And a VH mode system has a root port 
>> instead. The RCH RCEC and VH root port are both bound to the PCIeport 
>> bus driver that supports handling and logging AER. This allows the PCIe 
>> port bus driver to handle AER in a RCEC and root port AER using the same 
>> procedure and accesses to the AER capability registers. 
>>
>> This is oversimplified but are you looking to handle root port AER error 
>> in the RCEC from the below diagram? 
>>
>> RCEC <--> CXL root port (bridge) <--> Endpoint
>>
>>>
>>> Signed-off-by: Li Ming <ming4.li@intel.com>
>>> ---
>>>  drivers/pci/pci.h       |  6 ++++++
>>>  drivers/pci/pcie/rcec.c | 44 +++++++++++++++++++++++++++++++++++++++--
>>>  2 files changed, 48 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
>>> index 5ecbcf041179..a068f2d7dd28 100644
>>> --- a/drivers/pci/pci.h
>>> +++ b/drivers/pci/pci.h
>>> @@ -444,6 +444,9 @@ void pcie_link_rcec(struct pci_dev *rcec);
>>>  void pcie_walk_rcec(struct pci_dev *rcec,
>>>  		    int (*cb)(struct pci_dev *, void *),
>>>  		    void *userdata);
>>> +void pcie_walk_rcec_all(struct pci_dev *rcec,
>>> +			int (*cb)(struct pci_dev *, void *),
>>> +			void *userdata);
>>>  #else
>>>  static inline void pci_rcec_init(struct pci_dev *dev) { }
>>>  static inline void pci_rcec_exit(struct pci_dev *dev) { }
>>> @@ -451,6 +454,9 @@ static inline void pcie_link_rcec(struct pci_dev *rcec) { }
>>>  static inline void pcie_walk_rcec(struct pci_dev *rcec,
>>>  				  int (*cb)(struct pci_dev *, void *),
>>>  				  void *userdata) { }
>>> +static inline void pcie_walk_rcec_all(struct pci_dev *rcec,
>>> +				      int (*cb)(struct pci_dev *, void *),
>>> +				      void *userdata) { }
>>>  #endif
>>>  
>>>  #ifdef CONFIG_PCI_ATS
>>> diff --git a/drivers/pci/pcie/rcec.c b/drivers/pci/pcie/rcec.c
>>> index d0bcd141ac9c..189de280660c 100644
>>> --- a/drivers/pci/pcie/rcec.c
>>> +++ b/drivers/pci/pcie/rcec.c
>>> @@ -65,6 +65,15 @@ static int walk_rcec_helper(struct pci_dev *dev, void *data)
>>>  	return 0;
>>>  }
>>>  
>>> +static int walk_rcec_all_helper(struct pci_dev *dev, void *data)
>>> +{
>>> +	struct walk_rcec_data *rcec_data = data;
>>> +
>>> +	rcec_data->user_callback(dev, rcec_data->user_data);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>>  static void walk_rcec(int (*cb)(struct pci_dev *dev, void *data),
>>>  		      void *userdata)
>>>  {
>>> @@ -83,7 +92,7 @@ static void walk_rcec(int (*cb)(struct pci_dev *dev, void *data),
>>>  	nextbusn = rcec->rcec_ea->nextbusn;
>>>  	lastbusn = rcec->rcec_ea->lastbusn;
>>>  
>>> -	/* All RCiEP devices are on the same bus as the RCEC */
>>> +	/* All devices are on the same bus as the RCEC */
>>
>> RCiEPs are not guaranteed to be on same bus as RCEC. Details for associated 
>> next and last busses:
>>
>> "This register does not indicate association between an Event Collector and 
>> any Function on the same Bus Number as the Event Collector itself, however 
>> it is permitted for the Association Bus Range to include the Bus Number of 
>> the Root Complex Event Collector."[1]
>>
>> [1] PCI Spec 6.0 - RCEC Associated Bus Numbers Register 9Ofset 08h)
> 
> Hi Terry,
> 
> This patchset is responding to the implications of the implementation
> note in 9.18.1.5 RCEC Downstream Port Association Structure (RDPAS).
> That says that CXL.io and CXL.cachemem errors in Root Ports may indeed
> be signaled to an RCEC. Do you expect that implementation note to cause
> any issues on platforms that do not follow that CXL spec behavior?
> 
> My expectation is that it may just cause extra polling for errors, but
> not cause any harm.

AMD platforms in RCH/RCD mode consume protocol errors in the RCEC's AER driver. AMD 
platforms in VH mode consume protocol errors (including root port errors) in the 
root port's AER driver. The exception is the VH mode host with CXL1.1 endpoint and 
RCH downstream errors. CXL1.1 endpoint and RCH downstream errors in a VH host are 
consumed in the RCEC.

I don't believe these patchset changes would affect this behavior. But, I will need 
to test to confirm.

Regards,
Terry