Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 31 Aug 2023 13:35:36 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
        <vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <dave.jiang@intel.com>, <Jonathan.Cameron@huawei.com>,
        <linux-cxl@vger.kernel.org>
CC:     <terry.bowman@amd.com>, <rrichter@amd.com>,
        <linux-kernel@vger.kernel.org>, <bhelgaas@google.com>,
        Oliver O'Halloran <oohall@gmail.com>,
        <linuxppc-dev@lists.ozlabs.org>, <linux-pci@vger.kernel.org>
Subject: RE: [PATCH v10 13/15] PCI/AER: Forward RCH downstream port-detected
 errors to the CXL.mem dev handler
Message-ID: <64f0f9984932b_31c2db29461@dwillia2-xfh.jf.intel.com.notmuch>
References: <20230831170248.185078-1-terry.bowman@amd.com>
 <20230831170248.185078-6-terry.bowman@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230831170248.185078-6-terry.bowman@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?rR/xxPTTU5OCQvuJuignUMFsac/rnN9Jg8d7pPxnW7ioStPy6A1tQeDzNZeS?=
 =?us-ascii?Q?fVSH9z1Ep4SnCQBtXvb0qViJtv08wVdBZxYD1ORmuZHOWSRFaN8TNMylbdKq?=
 =?us-ascii?Q?OSX02f7IelwdPa3RS4I4xOoYdKUaqPVnzEDiGwvrDB9i1bmUGgWt6evz6K+p?=
 =?us-ascii?Q?xBz+vwTlTrQXXoZLKyDMo61eFPf01O8mJhmnVpjOkdZdwL1rX1uzBs0HdFlr?=
 =?us-ascii?Q?w4UYolrNgrF2CBAmmSYQPEcs1f6gI3M0iXf9ipnfv1J3sB0iOZa+AMDZ9MBW?=
 =?us-ascii?Q?Vgh2Fx+vBEGwcQQ1a9XoaI8BtQd1QKPcsE2FSfOEUyP6Mq7q4JBbYOm4bNgR?=
 =?us-ascii?Q?XggnU4KEEfLq1Lu1DLm648WxJcCsbFLAhyc/vnoMn8jHn+9Y6uN1+aBizNgG?=
 =?us-ascii?Q?de/9S63M+wio0vdJ7UWWf/2El/t3skyXl/tWAtFeI3V9251cWlf8zsQYo9uh?=
 =?us-ascii?Q?TSOm2vaPLREDsM2nUjIM28tlHq1wdnLmwYQlAdxdIkraxWmw8vEcPOYTalBz?=
 =?us-ascii?Q?5ffyvXj5gCzTMf6fjQvyU8eRO+mXneUyjgE0nGlwdA4qASMpqOntvXg6h0gJ?=
 =?us-ascii?Q?fkbAdfNErQcD9lEGzRBY6nXSXob9LGIrmcKO0uNXvR2ULP9y0e5JuhiWIcHp?=
 =?us-ascii?Q?V5T3tylf18DVqBKnRg/cPKu2ifDRnLByhQBUaY1LgV2eDr74QBwFbc9ueEGZ?=
 =?us-ascii?Q?AyG9PvwTO6uqOTGfXrQXIB63ljwPdm46oeakUUGIlVGImHXPPOtRAl+MYiUZ?=
 =?us-ascii?Q?ybNuNso88m0NDyROflY8uh6YNvAkBdVh7kE+9HdqdJnYKes8hytUuN0o6ir8?=
 =?us-ascii?Q?ZkvLd/hPv3wu70SM162GkTVu3P9aG8XvLz+adiWFD9QtqObSr3irRwN16iqb?=
 =?us-ascii?Q?U8IIddl7GMlS0r+C+RFP46ZI1tifTl24EvNPdgIqToHpiHhpekb+LQR2dt8L?=
 =?us-ascii?Q?234gUOrj1lZRUysQi9tSNQMR7wjZR+b5D56CCM+M2eCSlfej131HAt12yicP?=
 =?us-ascii?Q?pGvw1IaeMGEn7eOGrcWAmbN5OYR4H26Rrey7KxvWWnK3FfNBwHWlNHSXrL2V?=
 =?us-ascii?Q?wu5s8PTKH4pHtxh+rL+Z9jx+dw8XmyeJAmpyWAHsWIpXb36USmGZ7H3Nbkyz?=
 =?us-ascii?Q?jFaTr59lXm8UcbsjLcMcQ0KRl4cxQxoHCNgU47QzrYASSxLCAmLB/BAlKOf4?=
 =?us-ascii?Q?zyBH4kWZ6dBHiDVJB2ZEBOAy0PSE6q5CZLDPtio9cd2Cp+NTREeRqCDfpTuj?=
 =?us-ascii?Q?ZuQzlNDEWOak5Y+FJ7RnZahzsZQ5Z7God9MgNt0WuVmLZ07bw3L4+is+5sR9?=
 =?us-ascii?Q?mrE/afAdnwveqTBYtqmtrm2+4ebpm4O3jmjZMJWWoBFkfd2c9D85H01U/4QE?=
 =?us-ascii?Q?nZCYs+Ywkdacyx++bCIVJUzRyqcXFl4JE33VGsIRzv/Ufrxaqc8eB1nfJ5Dc?=
 =?us-ascii?Q?EG9Y4eMp5c27nl/zeixOArVBK3K0kiICzY/ZwkpGFT9hXSF0JgK45p2Qm2Si?=
 =?us-ascii?Q?S3IsxX0mP/yPeMk6SmLhoIglKP3qvJory7AN09soZ6jGCGCb7Musx8S/wDrl?=
 =?us-ascii?Q?J/RyjOe4iwYT8VPZVfBX8nv5PS8805V9E4wu6/l6uFYRv11sLbel5SVFz2Vk?=
 =?us-ascii?Q?nA=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a50dd91d-7023-4aad-7b8d-08dbaa61d6d2
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Aug 2023 20:35:39.7645
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: x0jXd+WK/NL0bJH6FVG/EoO8HLRmLoCHcx0bnfXI5rP/YzQ85luMCJZtzAeYTBrSovyRBWtYcvMJvM4wX4p4b0aRAV6KWHRbTA9ErVE8mDM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB8451
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Terry Bowman wrote:
> From: Robert Richter <rrichter@amd.com>
> 
> In Restricted CXL Device (RCD) mode a CXL device is exposed as an
> RCiEP, but CXL downstream and upstream ports are not enumerated and
> not visible in the PCIe hierarchy. [1] Protocol and link errors from
> these non-enumerated ports are signaled as internal AER errors, either
> Uncorrectable Internal Error (UIE) or Corrected Internal Errors (CIE)
> via an RCEC.
> 
> Restricted CXL host (RCH) downstream port-detected errors have the
> Requester ID of the RCEC set in the RCEC's AER Error Source ID
> register. A CXL handler must then inspect the error status in various
> CXL registers residing in the dport's component register space (CXL
> RAS capability) or the dport's RCRB (PCIe AER extended
> capability). [2]
> 
> Errors showing up in the RCEC's error handler must be handled and
> connected to the CXL subsystem. Implement this by forwarding the error
> to all CXL devices below the RCEC. Since the entire CXL device is
> controlled only using PCIe Configuration Space of device 0, function
> 0, only pass it there [3]. The error handling is limited to currently
> supported devices with the Memory Device class code set (CXL Type 3
> Device, PCI_CLASS_MEMORY_CXL, 502h), handle downstream port errors in
> the device's cxl_pci driver. Support for other CXL Device Types
> (e.g. a CXL.cache Device) can be added later.
> 
> To handle downstream port errors in addition to errors directed to the
> CXL endpoint device, a handler must also inspect the CXL RAS and PCIe
> AER capabilities of the CXL downstream port the device is connected
> to.
> 
> Since CXL downstream port errors are signaled using internal errors,
> the handler requires those errors to be unmasked. This is subject of a
> follow-on patch.
> 
> The reason for choosing this implementation is that the AER service
> driver claims the RCEC device, but does not allow it to register a
> custom specific handler to support CXL. Connecting the RCEC hard-wired
> with a CXL handler does not work, as the CXL subsystem might not be
> present all the time. The alternative to add an implementation to the
> portdrv to allow the registration of a custom RCEC error handler isn't
> worth doing it as CXL would be its only user. Instead, just check for
> an CXL RCEC and pass it down to the connected CXL device's error
> handler. With this approach the code can entirely be implemented in
> the PCIe AER driver and is independent of the CXL subsystem. The CXL
> driver only provides the handler.
> 
> [1] CXL 3.0 spec: 9.11.8 CXL Devices Attached to an RCH
> [2] CXL 3.0 spec, 12.2.1.1 RCH Downstream Port-detected Errors
> [3] CXL 3.0 spec, 8.1.3 PCIe DVSEC for CXL Devices
> 
> Co-developed-by: Terry Bowman <terry.bowman@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> Cc: "Oliver O'Halloran" <oohall@gmail.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> Cc: linuxppc-dev@lists.ozlabs.org
> Cc: linux-pci@vger.kernel.org
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/pci/pcie/Kconfig | 12 +++++
>  drivers/pci/pcie/aer.c   | 96 +++++++++++++++++++++++++++++++++++++++-
>  2 files changed, 106 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/pcie/Kconfig b/drivers/pci/pcie/Kconfig
> index 228652a59f27..4f0e70fafe2d 100644
> --- a/drivers/pci/pcie/Kconfig
> +++ b/drivers/pci/pcie/Kconfig
> @@ -49,6 +49,18 @@ config PCIEAER_INJECT
>  	  gotten from:
>  	     https://git.kernel.org/cgit/linux/kernel/git/gong.chen/aer-inject.git/
>  
> +config PCIEAER_CXL
> +	bool "PCI Express CXL RAS support for Restricted Hosts (RCH)"

Why the "for Restricted Hosts (RCH)" clarification? I am seeing nothing
that prevents this from working with RCECs on VH topologies.


> +	default y

Minor, but I think "default PCIEAER" makes it slightly clearer that CXL
error handling comes along for the ride with PCIE AER.

> +	depends on PCIEAER && CXL_PCI
> +	help
> +	  Enables error handling of downstream ports of a CXL host
> +	  that is operating in RCD mode (Restricted CXL Host, RCH).
> +	  The downstream port reports AER errors to a given RCEC.
> +	  Errors are handled by the CXL memory device driver.
> +
> +	  If unsure, say Y.
> +
>  #
>  # PCI Express ECRC
>  #
> diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c
> index d3344fcf1f79..c354ca5e8f2b 100644
> --- a/drivers/pci/pcie/aer.c
> +++ b/drivers/pci/pcie/aer.c
> @@ -946,14 +946,100 @@ static bool find_source_device(struct pci_dev *parent,
>  	return true;
>  }
>  
> +#ifdef CONFIG_PCIEAER_CXL
> +
> +static bool is_cxl_mem_dev(struct pci_dev *dev)
> +{
> +	/*
> +	 * The capability, status, and control fields in Device 0,
> +	 * Function 0 DVSEC control the CXL functionality of the
> +	 * entire device (CXL 3.0, 8.1.3).
> +	 */
> +	if (dev->devfn != PCI_DEVFN(0, 0))
> +		return false;
> +
> +	/*
> +	 * CXL Memory Devices must have the 502h class code set (CXL
> +	 * 3.0, 8.1.12.1).
> +	 */
> +	if ((dev->class >> 8) != PCI_CLASS_MEMORY_CXL)
> +		return false;

Type-2 devices are going to support the same error flows but without
advertising the CXL class code. Should this perhaps be something that
CXL drivers can opt into by setting a flag in the pci_dev? It is already
the case that the driver needs to be attached for the error handler to
be found, so might as well allow the CXL AER handling to be opted-in by
the driver as well.

> +
> +	return true;
> +}
> +
> +static bool cxl_error_is_native(struct pci_dev *dev)
> +{
> +	struct pci_host_bridge *host = pci_find_host_bridge(dev->bus);
> +
> +	if (pcie_ports_native)
> +		return true;
> +
> +	return host->native_aer && host->native_cxl_error;
> +}
> +
> +static bool is_internal_error(struct aer_err_info *info)
> +{
> +	if (info->severity == AER_CORRECTABLE)
> +		return info->status & PCI_ERR_COR_INTERNAL;
> +
> +	return info->status & PCI_ERR_UNC_INTN;
> +}
> +
> +static int cxl_rch_handle_error_iter(struct pci_dev *dev, void *data)
> +{
> +	struct aer_err_info *info = (struct aer_err_info *)data;
> +	const struct pci_error_handlers *err_handler;
> +
> +	if (!is_cxl_mem_dev(dev) || !cxl_error_is_native(dev))
> +		return 0;
> +
> +	/* protect dev->driver */
> +	device_lock(&dev->dev);
> +
> +	err_handler = dev->driver ? dev->driver->err_handler : NULL;
> +	if (!err_handler)
> +		goto out;
> +
> +	if (info->severity == AER_CORRECTABLE) {
> +		if (err_handler->cor_error_detected)
> +			err_handler->cor_error_detected(dev);
> +	} else if (err_handler->error_detected) {
> +		if (info->severity == AER_NONFATAL)
> +			err_handler->error_detected(dev, pci_channel_io_normal);
> +		else if (info->severity == AER_FATAL)
> +			err_handler->error_detected(dev, pci_channel_io_frozen);
> +	}
> +out:
> +	device_unlock(&dev->dev);
> +	return 0;
> +}
> +
> +static void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
> +{
> +	/*
> +	 * Internal errors of an RCEC indicate an AER error in an
> +	 * RCH's downstream port. Check and handle them in the CXL.mem
> +	 * device driver.
> +	 */
> +	if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC &&
> +	    is_internal_error(info))
> +		pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);

This would seem to work generically for RCEC reported errors in a VH
topology, so I think the "_rch" distinction can be dropped.

> +}
> +
> +#else
> +static inline void cxl_rch_handle_error(struct pci_dev *dev,
> +					struct aer_err_info *info) { }
> +#endif
> +
>  /**
> - * handle_error_source - handle logging error into an event log
> + * pci_aer_handle_error - handle logging error into an event log
>   * @dev: pointer to pci_dev data structure of error source device
>   * @info: comprehensive error information
>   *
>   * Invoked when an error being detected by Root Port.
>   */
> -static void handle_error_source(struct pci_dev *dev, struct aer_err_info *info)
> +static void pci_aer_handle_error(struct pci_dev *dev, struct aer_err_info *info)
>  {
>  	int aer = dev->aer_cap;
>  
> @@ -977,6 +1063,12 @@ static void handle_error_source(struct pci_dev *dev, struct aer_err_info *info)
>  		pcie_do_recovery(dev, pci_channel_io_normal, aer_root_reset);
>  	else if (info->severity == AER_FATAL)
>  		pcie_do_recovery(dev, pci_channel_io_frozen, aer_root_reset);
> +}
> +
> +static void handle_error_source(struct pci_dev *dev, struct aer_err_info *info)
> +{
> +	cxl_rch_handle_error(dev, info);
> +	pci_aer_handle_error(dev, info);
>  	pci_dev_put(dev);
>  }
>  
> -- 
> 2.34.1
>