Received-SPF: pass (google.com: domain of linux-kernel+bounces-108915-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <9c4b275e-45cf-42ee-b356-d858a2fd3755@intel.com>
Date: Wed, 20 Mar 2024 20:48:09 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH 0/6] Add support for root port RAS error handling
Content-Language: en-US
To: Dan Williams <dan.j.williams@intel.com>, <rrichter@amd.com>,
	<terry.bowman@amd.com>
CC: <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>
References: <20240313083602.239201-1-ming4.li@intel.com>
 <65f3a842988d6_a9b4294f7@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <16ab732d-a009-45ee-a438-3faf048c7acd@intel.com>
 <65f491b64f15a_aa2229497@dwillia2-mobl3.amr.corp.intel.com.notmuch>
From: "Li, Ming" <ming4.li@intel.com>
Organization: Intel
In-Reply-To: <65f491b64f15a_aa2229497@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QTFLTWx2UjYybmRBS3F5MnFKOGFKd0F4bllJWTBSd1R6Z2lQWHhsUnZES0VS?=
 =?utf-8?B?TlhIeEZsU0pxeGxmMkwxa1QvVVY5RzdVbTdmU1lZUGlHdWxob2xEaVRETk1Q?=
 =?utf-8?B?SUVSMmJHRHM1bCtDd3BDSnYzSmlWQUxKU0ovUjhoWWk4ajFkUHFaTVJuVWQv?=
 =?utf-8?B?cUFWL3oxSjhSampZNmViS3M0ZElwc3Z1VUxBSWYwK2orc3lZNWZKVzZMTTFN?=
 =?utf-8?B?KzlPREJqSnlkOFhvcHpyaEpDMDFWMC9qdDRRY0Z6bit4KzhyeFhBd1hNMkt0?=
 =?utf-8?B?MDBrMTlLR29POVBQOUNycTFkOHY4elBnMjFEQXU1ZC8yRHNsQklteU5CeWFH?=
 =?utf-8?B?UnMxaEw1K0NTam5jMUZ0LzN4MFZRUGF5T2ZQcGFVdjBVc1duL0o4VXU5N3Zm?=
 =?utf-8?B?dXRpeTMrem52YzV3QlZjNEk3bjh4T3Y4Y3E5dk9QcmJRZWxZb2FPYkREWE0x?=
 =?utf-8?B?MWI1aC9BZlhQT2NTZXgxZzI2VFJQOHRkbWVkUEkzaEZoTGN0T2ZONk1Wc0Nr?=
 =?utf-8?B?dnhobWRWQXZCVkRLT1ZwVkxTTW5YaWtVdkZOM3hza2J3SkppOG1YQlVmNmFj?=
 =?utf-8?B?U1EvYUxtRzRCd0ZuVTNBS2NoZ0NrcG1TK01pd0xkZks0MWZuQUcrRk83cW55?=
 =?utf-8?B?MFhaYmZ6SmsxV3FOdytmMlBwdzVrdTlDdmlpMnIwbCtRU1g1RjByWVlqYWNP?=
 =?utf-8?B?clVVYmlETVJncGRDc0VyUlhZVkVqUVpjUjhiY3FaWTJrT2xMTm1YOHdlVDAy?=
 =?utf-8?B?VFhXMFVEYUhpWEp4d3RaM0hxaUV1TUZDUndaaW1ENWNNdE5Pdm5VaUxmWGFl?=
 =?utf-8?B?cXhaNCtmaTR3L0svTjVnekJCaGsxcG5VYVRtRUpUMGdHUjB5SWtIVUp2V2ta?=
 =?utf-8?B?bzUrVENqOGJRL004RzRyUE4vRXJxd3ozcEZUOEozWlY5RHdSZWhJVFpTaUlX?=
 =?utf-8?B?QU9scUJyWjZ6aWEvbjhqK0UrZ1I4L1RBUUJvM1dlUndVL0trYk9KUmNuQjgy?=
 =?utf-8?B?K1VmSWZ3RGtuZCtJM0l5UjA4U3N1cG4ydDNyL2wyTHE4TEhNV3EzNDVRdFRV?=
 =?utf-8?B?dGhVREttMWYxSTlleGI5QTRaUWZsZUJ3NnBWelY5WEpxK3RtVjU4c1VLdFNI?=
 =?utf-8?B?b0lzYkh0WDJyUUVnR2FYOFlPSE15R3VjcHlhbHlZNzlBQ2dNbUtneHhuVXU4?=
 =?utf-8?B?NzNQZytVQXlTNXJIV2RzOERMWkJ4alFkcytvMm1RSU1hc3MrYTZYa2pDV3dX?=
 =?utf-8?B?ZHZ1QzJ0VWNsZ2I1OTJBdUp4QUpwY1BsT20rMkt3c3o2S1A5ZGxmbGRLSWZH?=
 =?utf-8?B?d3I2bFpQVDhSSTZCOEV0VWJDdEw5LzhwOHVIV0grQVpjZW9rT0VjOThIUllH?=
 =?utf-8?B?N0lTSnRMVVhSSGRhQkk0V3B4WitSVUI5Yi8rNWNtTGQ1MzFSOUNNc05JbkVI?=
 =?utf-8?B?QmNtRWNJMzI5M0pOa1RoTzFFZmE3clhYeUF4azZmRHNsM1g0eDhSTWZJU0p0?=
 =?utf-8?B?UTIvT1Y3ZjhqOHNnWDdqOXp5SFJtb29RZ2Y5bVpsK2N6RDhJU1J5N3d1SEFR?=
 =?utf-8?B?MUFET1lTdEJTYXQ3K3dySkhnRVNtTkh3eWMvL1A3eTd5ZzBCZzZ6OHRmWVhq?=
 =?utf-8?B?M1VEZk45L1RMdE9FSjJlMkw0b2tlNjRkUHl1RUsyKy8yYkI0d0FmbnpTVCtT?=
 =?utf-8?B?OFcyMm0wUUUrR2NNa01nMWxGZUVhRDUrcUJIM0VZQXlDZlZ3K2pCN3BQSExn?=
 =?utf-8?B?UDJSUFlYcGpaSjFMUi9HTTJyN3VuVkVVVXBidmlWVWlTbVZVOTZTZWtUQkZ2?=
 =?utf-8?B?VTVtL2RRczczdEdDZW51a2JKUU05amxkRy81cFhjZnkzMDlNdG8wQmZUc0M2?=
 =?utf-8?B?L3hPdXZVTVBoN3IwZzNUVlVrRDVsNysxTnVnalpMbTdSRlViNjkwT2JZYjhr?=
 =?utf-8?B?d1NVeFZSYXY0OVg4MVBSOXZvTEw3NlpnajllaXc0YmlkdVl1Wm9IOUFkcVBl?=
 =?utf-8?B?eCtER1FUdE9WVnlpV0Z4ODJVUnpwc0hYRkM1SjJMd0tXWnBVNWE1a1UvcjBZ?=
 =?utf-8?B?U0RZdE44Z01ZcWxGLzRneHBjYmZnT1NXeTVsdWZjYndlVjFaSGZDMUN3cnlq?=
 =?utf-8?Q?v1wirq6FnqbjJ+LNx6UA4m2zv?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b8bbaac2-5589-42e7-e53b-08dc48dc03a3
X-MS-Exchange-CrossTenant-AuthSource: IA1PR11MB7200.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Mar 2024 12:48:17.4172
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ak4Wksgvl27vQYFGJIT1TP7xQEX5O1By05YsBlhCd2umfcFD89NRD5uXwd+wHQhhnsVG4Txr9Yu8Nh2gBIzztw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6351
X-OriginatorOrg: intel.com

On 3/16/2024 2:21 AM, Dan Williams wrote:
> Li, Ming wrote:
> [..]
>>> I do expect direct-attach to be a predominant use case, but I want to
>>> make sure that the implementation at least does not make the switch port
>>> error handling case more difficult to implement.
>>
>> Hi Dan,
>>
>> Currently, A rough idea I have is that:
>> If a CXL switch connected to the CXL RP, there should be two cases,
>> 1. no CXL memory device connected to the switch, in this case, I'm not
>> sure whether CXL.cachemem protocol errors is still possibly happened
>> between RP and switch without CXL memory device. If not, maybe we
>> don't need to consider such case?
> 
> Protocol errors can happen between any 2 ports, just like PCI protocol
> errors.

Seems like if we want to handle CXL protocol error for such case on CXL driver side, we need some changes for CXL switch port/dport enumeration, I remembered that switch USP and DSP are enumerated only when a CXL memory device attached.

> 
>> 2. a CXL memory device connected to the switch. I think cxl_pci error
>> handler could also help to handle CXL.cachemem protocol errors
>> happened in switch USP/DSP.
> 
> No, for 2 reasons:
> 
> * The cxl_pci driver is only for general CXL type-3 memory
>   expanders. Even though no CXL.cache devices have upstream drivers they
>   do exist and they would experience protocol errors that the PCI core
>   needs to consider.
> 
> * When a switch is present it is possible to have a protocol error
>   between the switch upstream port and the root port, and not between
>   the switch downstream port and the endpoint.
> 
> The more I think about it I do not think it is appropriate for cxl_pci
> to be involved in clearing root port errors in the VH case, it only
> works the RCH case because of the way the device and the root-port get
> combined.

Thank you for your explanation. 

I think I need some time to consider a appropriate proposal. maybe we can register a CXL error handler into struct pci_driver during CXL port/dport enumeration for root port and switch cases, PCIe AER driver will invoke this registered CXL error handler to handle errors on CXL subsystem side. But I think this solution also meets above issue(current CXL subsystem won't enumerate CXL switch port/dport without CXL mem device).