Received-SPF: pass (google.com: domain of linux-kernel+bounces-104170-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <16ab732d-a009-45ee-a438-3faf048c7acd@intel.com>
Date: Fri, 15 Mar 2024 16:40:47 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH 0/6] Add support for root port RAS error handling
To: Dan Williams <dan.j.williams@intel.com>, <rrichter@amd.com>,
	<terry.bowman@amd.com>
CC: <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>
References: <20240313083602.239201-1-ming4.li@intel.com>
 <65f3a842988d6_a9b4294f7@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Content-Language: en-US
From: "Li, Ming" <ming4.li@intel.com>
Organization: Intel
In-Reply-To: <65f3a842988d6_a9b4294f7@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UXVNZVZrR0hQRFRzYlhKbENldm93R2R3R1BnSkdjVUxqaDNISldtclFtL3lM?=
 =?utf-8?B?aDRUMWkzSnQrbS96dDVSczErZkJXSktoeGFHYklhYTlOTFZlSEQ2anNZNkxq?=
 =?utf-8?B?VWpyT2tqTVNEdlVNSFZPUjVNWjRKcFR4U3Y5Mkt4Y2tSSWovS0QrTVlBQXpp?=
 =?utf-8?B?Q3BRaXhub25YbkVKTHVPeDc3WUVBeHNZMjVkSDRTZzZ0WWJtbjNMditxMHIr?=
 =?utf-8?B?clBPOTlJd1k2Y0VmdXlEYk5DV21RbFplNTFBWkVrRWZIZEV4eFFHTXRoVWRt?=
 =?utf-8?B?OHNJVnBMTGxsb0p2UHM3RitwcHZiN2h3cFJTZVBINnllOXNlVGhMYkxHYzgr?=
 =?utf-8?B?bEo5UTRML2Z5L00wbVgwU3MwT1ZYOElNbnI4Nm5Sbkoyc1lQSmE1MnpYNzMx?=
 =?utf-8?B?T2EzYW5FNUwzR0VlcEl2RGVsK0xlbWZNMjNsRC9vVjdlV0JSMlNZbGYyMFJi?=
 =?utf-8?B?bitUaUcydlpYODVPbU44N043TVBzRXRpZEZndlg5aTRYNHZNdjJXZDQzSC9n?=
 =?utf-8?B?dU52c1RaM04vMDkxeXFibzUvR0EyN3pxZyt4c2JEY0RVUGZ1YlBsZ0x2Y2oy?=
 =?utf-8?B?M2pudDZaSmpxYUpyOHV4WjVTdDNOYkJNa05jZm9mWS9JTzdaL3dPVnlBZndn?=
 =?utf-8?B?SVR4Rms4dUE2emlXN2RGUnJaZzJZRUF3eWFrYnhod051YnIraHV5L0VlcTh2?=
 =?utf-8?B?OTdpWkpaa0xkenM5MFVBQUtVRDJ6cGtZcmRXYTUvVWtMUDRyeW94MHpzOHdp?=
 =?utf-8?B?VHRycnVhNFljWTJVUEQrK2FZbG55SWFMMXlDQzJlWFc4dTZaNHdjeENSKy92?=
 =?utf-8?B?UHMrdVU4Q1FzTm80MkFkcWR5bkVOVldwdnF4Q3pkRE50K3kyN3BQbHcrYzl2?=
 =?utf-8?B?d2dtNmRocSt4ZGd0N1Q5OWgydVd1RjgxaFZpY3JFOVpCSmFUb2hhTGZzK3dE?=
 =?utf-8?B?a3ZiNWFMUE9iSlRyNnVCWmdMZ2NrMVJ6WmtwVXl4Yi8yekdXOGt4THhSZFVl?=
 =?utf-8?B?Rm1GRTBBdVdTWlZ6RGxWQ1dxSWFDME80bzViNGQ4OGdtdjRpMzRMWFhjciti?=
 =?utf-8?B?UlNCQjVmVWN2c3d6WjkyVFUxTjRiZVpkL2xodmVpeExyK2dua3R4c1V0UitI?=
 =?utf-8?B?VGhBNXFzdkd4VE41c2FvRW9sZTRzSHFONGE3VTloR0JoUmxibjRheDBTbG4r?=
 =?utf-8?B?VXh5VVdYK2J4aUQwR2hLR0hxZWNBdXRBbmg3RzJKQXJWdUVBTy9XV3FlUHZs?=
 =?utf-8?B?UEFxL3Z5WkxoVG50SFBEVmR3U0t2NHk3YlFLam5SQk1VYjdId0x4bEkrRDh0?=
 =?utf-8?B?ak5oRXJ6UUdJU0VPT2lTbTJnSTFFcFBDQ2lyYVRRdng4MFQ4YW1aODNYbnVF?=
 =?utf-8?B?RkxuWHZnR0wwYUZIdkx4dFVPYTNxRUg0dzFMN2FHWUNXTGd0WG5YdXZsajNE?=
 =?utf-8?B?QkJIWVo0STdxU3JFbGF2OWhYV2JJL0J4NlhzQllTTVp0NDE3elpVb3k3dlhu?=
 =?utf-8?B?dzhFeUpjQmQ4V1ZTOEpUTWpnSjR3SjhOczJ6aFo1ZjlCTUw2aWgwUVY1SThq?=
 =?utf-8?B?c3dndzZIYXpSL1BtMUpmdUZKd1NyNy9uVmRhK0RvZnZoRS9Wd0d6ZnFoRmxn?=
 =?utf-8?B?UkFRT2RDZmdocXVGaEdCV0NiRDQ4MWI0YnpWekQ0bitWbXBycEJ4dWx6NHBT?=
 =?utf-8?B?aEsxN3cxaWJtbFM5a2VQTzkwYzBibkRJL2JrbjZlTlNSUFh2N0x6d2pFN3Rs?=
 =?utf-8?B?dzh5THgxRG8zVER2S2Q2NEYzVS9MeWdoeWplUDVQQjA3UjkwMWpWRjRpbzZP?=
 =?utf-8?B?bEEvNGFiNk5CWUFpTUFwODN3MFdmMDRmNkNFWmN0aEZGblg3Y3k2K1YzYURy?=
 =?utf-8?B?c2pWd1ZtSytLTDNPUmFacjhZVnZkMUxQSnBMNVhHeng3ZUc3Y1MzYXFwWS9B?=
 =?utf-8?B?anZPVXJwSmpVRVUyb2VTUStBUzdHVFI5aVByWDVFTis3MU9NSWtRcmd3eCsr?=
 =?utf-8?B?dldiWlc5T2tLYUNYQlh6ZFc1WEliTVc3ek5VOHZFZ1JGWUZ3VVNCSkM4V1U0?=
 =?utf-8?B?TkVuYmlRUWJxeFdIdFUwR05vdTMvaFJlajg0clBTRUxXaHExWW1jay9uSUxy?=
 =?utf-8?Q?+v6hNkTOACEbWjo555/kMlg0Y?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0cd8a890-17b4-4f01-1a7a-08dc44cba120
X-MS-Exchange-CrossTenant-AuthSource: IA1PR11MB7200.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Mar 2024 08:40:55.5674
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4+R4PTyx+ipmbuDEhgNJFrbVG3ZAGuOUfcfHNdVnNqFsUJYzXdCWWvIMEFLsyQ9n0G4YlTZllcQt0tYZo2SiGw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5299
X-OriginatorOrg: intel.com

On 3/15/2024 9:45 AM, Dan Williams wrote:
> Li Ming wrote:
>> Protocol errors signaled to a CXL root port may be captured by a Root
>> Complex Event Collector(RCEC). If those errors are not cleared and
>> reported the system owner loses forensic information for system failure
>> analysis.
>>
>> Per CXL r3.1 section 9.18.1.5, the recommendation for this case from CXL
>> specification is the 'Else' statement in 'IMPLEMENTATION NODE' under
>> 'Table 9-24 RDPAS Structure':
>>
>> 	"Probe all CXL Downstream Ports and determine whether they have logged an
>> 	error in the CXL.io or CXL.cachemem status registers."
>>
>> The CXL subsystem already supports RCH RAS Error handling that has a
>> dependency on the RCEC. Reuse and extend that RCH topoogy support to
>> handle reported errors in the VH topology case. The implementation is
>> composed of:
>> * Provide a new interface from RCEC side to support walk all devices
>>   under RCEC and RCEC associated bus range. PCIe AER core uses this
>>   interface to walk all CXL endpoints and all CXL root ports under the
>>   bus ranges.
>> * Update the PCIe AER core to enable Uncorrectable Internal Errors and
>>   Correctable Internal Errors report for root ports.
> 
> Thanks for the above background.
> 
>> * Invoke the cxl_pci error handler for RCEC reported errors.
> 
> So what do you expect happens when a switch is involved? In the RCH case
> it knows that the only thing that can fire RCEC is a root complex
> integrated endpoint implementation driven by cxl_pci. In the VH case it
> could be a switch.
> 
>> * Handle root-port errors in the cxl_pci handler when the device is
>>   direct attached.
> 
> I do expect direct-attach to be a predominant use case, but I want to
> make sure that the implementation at least does not make the switch port
> error handling case more difficult to implement.

Hi Dan,

Currently, A rough idea I have is that:
If a CXL switch connected to the CXL RP, there should be two cases,
1. no CXL memory device connected to the switch, in this case, I'm not sure whether CXL.cachemem protocol errors is still possibly happened between RP and switch without CXL memory device. If not, maybe we don't need to consider such case?

2. a CXL memory device connected to the switch. I think cxl_pci error handler could also help to handle CXL.cachemem protocol errors happened in switch USP/DSP.