Received-SPF: pass (google.com: domain of linux-kernel+bounces-104039-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Thu, 14 Mar 2024 21:05:15 -0700
From: Dan Williams <dan.j.williams@intel.com>
To: "Li, Ming" <ming4.li@intel.com>, Dan Williams <dan.j.williams@intel.com>,
	<rrichter@amd.com>, <terry.bowman@amd.com>
CC: <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH 4/6] PCI/AER: Extend RCH RAS error handling to
 support VH topology case
Message-ID: <65f3c8fb458d0_aa2229435@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20240313083602.239201-1-ming4.li@intel.com>
 <20240313083602.239201-5-ming4.li@intel.com>
 <65f3b2b29cfe9_aa22294e8@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <f2dd9f36-5e69-4c6e-9640-7140a694086c@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <f2dd9f36-5e69-4c6e-9640-7140a694086c@intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?AY0TubINO3P+EW/LpTXIgfC/SXiewzaJpfjB8QQRZJLtmniT06pEtLiVsxXP?=
 =?us-ascii?Q?gPIycFjjWlfj+b7EjPSDhgMLyYWh7tTVPgOF7xExgKjsjbiPfyuN6w0fanTj?=
 =?us-ascii?Q?4E0lEwiLSgWYGOrL87TQWWS+bQ4Lk+7q3XnCSSyeKtjhCtkZk0FS2geO/xKq?=
 =?us-ascii?Q?0LRLreAUqMrj3n1pqNuVNrJsNg+GoYbcddHaaYS6eCuW0DBGuP6EhoOe3qt4?=
 =?us-ascii?Q?Zz4dW5XXs2OHhBgHoDytYNSUNOy9bLRMfsjXVjszsxmL9sd1YmrEn/3bsMpy?=
 =?us-ascii?Q?U+wIxm93FqH3kAWXf7q0VrqnStucZGu1JZ2JyENk8gE8RKchXDZiLqb0ynu6?=
 =?us-ascii?Q?VSuod84gkdquSo9Xt34gk9I0B/FU5+Ef+zPSWgnHYzMpI2PVXM5yiTBjZruJ?=
 =?us-ascii?Q?NBDq39CmbVIvXcINYbRJVe17LNrkuN4aSo2k6gzC68aBPUbmZTl3xaXObCjs?=
 =?us-ascii?Q?OTav29bOfF/w84qVsZdf0pLXQdsRVOujm/LM5Auwo0zSawh0/FDszkCKfoxe?=
 =?us-ascii?Q?wDCWHACAPgQlY54MJrWR8MICbZJDEGBoJB65cEBYG/LgPNjhyUfE5BHXpCbc?=
 =?us-ascii?Q?jgPSSp1yAPiuJ4CWAvwyIzNsuopTwy3mkFLvySTourTgJqzMexYbt0/9D6Uo?=
 =?us-ascii?Q?6H4IPUXfn7rNXVtsZDzFE8i0dAnhmtmucKnzcvq4pIAD8AyzELh/l7RZm8DR?=
 =?us-ascii?Q?1RGdqhdTe0NiEYdBdykOc7b+eebTKsdxLpM3qps5duPitGiQYdgK3y8x/3ef?=
 =?us-ascii?Q?8EkOvyCiSUrMBGlaurY+04iVAA78m+cTeIxluRwApSUZgeHCgcRhy1AwaCVY?=
 =?us-ascii?Q?l2BZ2bCbN7zp7Q37PBxUxwIgJ+xP3wsaX8AVWBNAveKl9lnnzrZOKKdXLuyc?=
 =?us-ascii?Q?2ZGrXFCIRa4Hp3ys2uSvFJlZtwR8GA4JMPt5EOaVSSQInHY97YwaNOJ5plcE?=
 =?us-ascii?Q?dir1CHv2z9ieBAu4AVLIxc4wA0j6stiTYG2b24u1QiIIySYOtbdDcliBgV2v?=
 =?us-ascii?Q?ypGhWHnp8tz/vPs4+Tx7bTQtgWXGG4e/O9fnCmqq4pxvW9HLfzxMV/w90FN/?=
 =?us-ascii?Q?tmzEI5tX3L4/JEQb5mOyjGxKu/jXwV4vivxKDV3j3VqHGhe23qOKlDIGpNiz?=
 =?us-ascii?Q?dF72faFCUQSSRKgbG1qawNPkzyeB+cIy5/FE3c9Q6trc4STKnnGdBygMBVCP?=
 =?us-ascii?Q?RshpyQYoq3edVHVwjNqUzgfVKuZxFKi+qrJSLiHlTpy5p6RVP9H5qICr8sIZ?=
 =?us-ascii?Q?oPjZrCB3X9ql7bLvrSYDbjgRwuZa0vWegP7szHXbaWJWblwiomCSKrdxqyTF?=
 =?us-ascii?Q?2ma/QkuU9pysB+ewpNhYSY22CT4y7BkNEDOf+iAJNolcrOHYijlDyCJgSqFc?=
 =?us-ascii?Q?MnyT0GLAoz3+8yN8CqSUlEDmIoYoQajybuYMPlBfgOwXDl1iY/biokZr2uNy?=
 =?us-ascii?Q?jgCsbSDtUjoonLsY0Ymz9w0BAd9p76lztvzxWZi8GNzV74Z57YipOjqkczxf?=
 =?us-ascii?Q?+UJzKiFrPmx3QtyhwX2Uo+2YxzEtadoeO2C4vCG7bVmI/ovNLv0RVPawlSCs?=
 =?us-ascii?Q?MivkiJzoYvm/iJULnIrNdMShajixamGX6fyw1E9Fbcc8rer6TZ234ZG1A+/P?=
 =?us-ascii?Q?gw=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: bc4dd232-7bef-44d9-e7f5-08dc44a521ee
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Mar 2024 04:05:21.1278
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FaD9M4cJ27HvCusBjskuJKLjrYSl2I1FzyFmgyEE2Z5UlttegLlcj62hjcPxUZ5RYUsjjxsCmY0D8sGSiuaDTNEjcfht4y4VPBtyAgvP4Es=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR11MB8076
X-OriginatorOrg: intel.com

Li, Ming wrote:
> On 3/15/2024 10:30 AM, Dan Williams wrote:
> > Li Ming wrote:
> >> When RCEC captures CXL.cachemem protocol errors detected by CXL root
> >> port, the recommendation from CXL r3.1 9.18.1.5 is :
> >>
> >> 	"Probe all CXL Downstream Ports and determine whether they have logged an
> >> 	error in the CXL.io or CXL.cachemem status registers."
> >>
> >> The flow is similar with RCH RAS error handling, so reuse it to support
> >> above case.
> >>
> >> Signed-off-by: Li Ming <ming4.li@intel.com>
> >> ---
> >>  drivers/pci/pcie/aer.c | 20 ++++++++++++--------
> >>  1 file changed, 12 insertions(+), 8 deletions(-)
> >>
> >> diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c
> >> index 364c74e47273..79bfa5fb78f4 100644
> >> --- a/drivers/pci/pcie/aer.c
> >> +++ b/drivers/pci/pcie/aer.c
> >> @@ -996,11 +996,15 @@ static bool is_internal_error(struct aer_err_info *info)
> >>  	return info->status & PCI_ERR_UNC_INTN;
> >>  }
> >>  
> >> -static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)
> >> +static int cxl_handle_error_iter(struct pci_dev *dev, void *data)
> >>  {
> >>  	struct aer_err_info *info = (struct aer_err_info *)data;
> >>  	const struct pci_error_handlers *err_handler;
> >>  
> >> +	/* Skip the RCiEP devices not associating with RCEC */
> >> +	if ((pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) &&
> >> +	    !dev->rcec)
> >> +		return 0;
> >>  	if (!is_cxl_mem_dev(dev) || !cxl_error_is_native(dev))
> >>  		return 0;
> > 
> > is_cxl_mem_dev(dev) will always be false in the VH case, so how does
> > this change help the VH case?
> 
> Hi Dan,
> 
> I think it won't be false if the CXL memory device is an endpoint.
> pcie_walk_rcec_all() will walk all pci_dev in RCEC assocaited bus ranges. So these two checkings can help us to filter:
> 1. CXL memory device is an RCiEP associated with RCEC in the RCH case
> 2. CXL memory device is not an RCiEP, so it should be an endpoint in the VH case.

It will be an endpoint, but I though cxl_handle_error_iter() is only
called for RCIEPs and RPs that are share a bus range with the RCEC. The
endpoint in the VH case is downstream of the RP.

I had been assuming that pci_walk_bus() limits itself to buses within
the Root Complex however it descends the entire bus hierarchy so this
implementation will walk the entire topology on all root ports
associated with the RCEC looking for any CXL device. That feels wrong.

I would expect that this limits it self to only finding root ports and
then only proceeding if that root port has a directly attached CXL
device.

Note, when you send a v2 of this RFC be sure to copy linux-pci for these
core changes to PCI error handling.