Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Date:   Fri, 27 Oct 2023 15:12:47 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Robert Richter <rrichter@amd.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Jonathan Cameron <jonathan.cameron@huawei.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        Ira Weiny <ira.weiny@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        Dan Williams <dan.j.williams@intel.com>
CC:     <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Terry Bowman <terry.bowman@amd.com>,
        Robert Richter <rrichter@amd.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>
Subject: RE: [PATCH v12 11/20] cxl/pci: Add RCH downstream port AER register
 discovery
Message-ID: <653c35df632a5_244c7829490@dwillia2-xfh.jf.intel.com.notmuch>
References: <20231018171713.1883517-1-rrichter@amd.com>
 <20231018171713.1883517-12-rrichter@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20231018171713.1883517-12-rrichter@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?0zf3JaacFa2efEQ+GJAoyImSsDWI62AfWCk2hH7ovXA0QISJDUz6GXpKvAv2?=
 =?us-ascii?Q?J+6IkwzVRcA7jNWB5MQbTUp5/o8MpJ/ews8LOseO9DoFkQY+w697J/r5E63J?=
 =?us-ascii?Q?TWk6oic1QfrpRF3Z7ptX5l5gusfVECwHZOhw3pldQPdihSa4WvcjsLwPpO4v?=
 =?us-ascii?Q?PLXmyfFBnJ3AOU05klriNOoZbgJy/3DjiGM6/FYAYC8aKpBEp4AhUmDWZTB/?=
 =?us-ascii?Q?lzvThiBNPjvRWxhHpV6zt4EVktJHQSlBmASULEnpanq/o2KBXFPphYyrfNnk?=
 =?us-ascii?Q?MsVJXJS20dgy4jTQgx3d5b2iR8b7If+TmW1fmDA0GcCtkKnPpaRNK+QTYRze?=
 =?us-ascii?Q?d8FFvfs396BA7ikb2pPp1oGuOJMjgPpO2xgBWq31JsSDg0ykwfSt8l9AcF9k?=
 =?us-ascii?Q?pT19+n4yin5rGd38zAu/PIDtkTHS+CZ9V2uTX9Ecans6b9eJ1F6JEj773pFi?=
 =?us-ascii?Q?VYOw9UTccQUgW5/9hlK6nOIW2fTQLujMn5+gLEwPOfjmcEdAohtScJ+y35lS?=
 =?us-ascii?Q?SyQRJyrf53N7FZ567HUUz2UtVgfV3S89HYucONTXW2ytG2DJMn3fyt9qOqG3?=
 =?us-ascii?Q?j2x+alcy9Z2tBZvZEMfIDsHc+YfhPJC1wr8b8LyunMN7vdkZr+9+/Enurhej?=
 =?us-ascii?Q?vVWF25lPjQnc1vdjpoixuyVYBJnPXUevOsLxM5c/tIvu5WdHdYfZ3143wcsK?=
 =?us-ascii?Q?v6SjmFwWxFxtwkPE/hfncoQe3V5df1ozAU+fojRG8BwxdwxDw1e6QQAJjgmL?=
 =?us-ascii?Q?QhO8h2cQOX3E4UaNmP2WoGQb405J1S6F8u/9Ue7iGXNfaeDA8PtNPnATGTKQ?=
 =?us-ascii?Q?EpHtXvKNBZ3jePx574gnk5uRPA9FlASJy8vU7yPMAzSx4sVL99fd4ss+Ukq/?=
 =?us-ascii?Q?Mc7Z3lyZIsQ44qzcLhX7Ut8VfZQ9P/K/eyBLRDxpP08y+Kh9Xw4ygoKN6/aq?=
 =?us-ascii?Q?+l5TY0mYtqMIKaU4tf9MI5giED1nPnqL2BE9GbAneSIoqozdAipV2y/6Mk1w?=
 =?us-ascii?Q?ytStVLEQL3mfA53RpL9DF54KXDiIJp2e6ER81jA1hxyLZRM8b17V68yxXxbh?=
 =?us-ascii?Q?tQkFAeNtBDiy1Atfpp2Cnjd1W48/azqsaaOeaLDH0Pb4d39+kqeGjBT8iAk3?=
 =?us-ascii?Q?K4CfFVHm/lIzBUecTnm4KSuPIzWFk4E7fiykUkLtYpBX3miRY433r8udv2az?=
 =?us-ascii?Q?9yQo6suLvLu1x5nCzZzMdgfyONtUI+ukmioSP5KftP2JWBVsBRNmmg4T9W0o?=
 =?us-ascii?Q?zUWOQONy/aHOcITnTuE9iqFrAcNztKK/z045xNX9KkEcFzspybB3z4bbfqiu?=
 =?us-ascii?Q?wOjX9wZDylOpKPBX2XP7Yo8nRX2Bk6yQBvO3S9KLS60hUxTEUfSuF/P1koG6?=
 =?us-ascii?Q?nh92X1o+Y67TbJGyprzBW0BlQz3/khcDJd20L39rbkAFH72OIWrpgXTMHi+s?=
 =?us-ascii?Q?7RbNCF3iZWhrGDHWzjdYmD3SBSEC5BEYhCMBK2t2fvcVO0+gai/4zYzEQdeA?=
 =?us-ascii?Q?/1TpQx1ugJutxTU9lx0ll8r65Yf2aieYupbWYyKKQLHAo7jhx3Izl0s6tAcD?=
 =?us-ascii?Q?1O7RdUWhPv/75T9Hd3g0fj+W4h4rlau1z0r/+zNoNgw/YxihrO4kRSzPZ5FF?=
 =?us-ascii?Q?Ow=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f166a252-0804-4adb-0513-08dbd739db7e
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2023 22:12:49.9727
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: yRABnNkVszS8sRTWwTgd0FfMejTrNc9E4HdFx4U0NBmFDDm/JINL3fz2qAIFsDx6afJM3GvQ60FX2ZFPccXeNQy8YTNSsfqW2jG/jN1IZMI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB4778
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 27 Oct 2023 15:13:30 -0700 (PDT)

Robert Richter wrote:
> From: Terry Bowman <terry.bowman@amd.com>
> 
> Restricted CXL host (RCH) downstream port AER information is not currently
> logged while in the error state. One problem preventing the error logging
> is the AER and RAS registers are not accessible. The CXL driver requires
> changes to find RCH downstream port AER and RAS registers for purpose of
> error logging.
> 
> RCH downstream ports are not enumerated during a PCI bus scan and are
> instead discovered using system firmware, ACPI in this case.[1] The
> downstream port is implemented as a Root Complex Register Block (RCRB).
> The RCRB is a 4k memory block containing PCIe registers based on the PCIe
> root port.[2] The RCRB includes AER extended capability registers used for
> reporting errors. Note, the RCH's AER Capability is located in the RCRB
> memory space instead of PCI configuration space, thus its register access
> is different. Existing kernel PCIe AER functions can not be used to manage
> the downstream port AER capabilities and RAS registers because the port was
> not enumerated during PCI scan and the registers are not PCI config
> accessible.
> 
> Discover RCH downstream port AER extended capability registers. Use MMIO
> accesses to search for extended AER capability in RCRB register space.
> 
> [1] CXL 3.0 Spec, 9.11.2 - System Firmware View of CXL 1.1 Hierarchy
> [2] CXL 3.0 Spec, 8.2.1.1 - RCH Downstream Port RCRB
> 
> Co-developed-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/cxl/core/core.h |  1 +
>  drivers/cxl/core/pci.c  |  6 ++++++
>  drivers/cxl/core/regs.c | 36 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 43 insertions(+)
> 
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index 45e7e044cf4a..f470ef5c0a6a 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -73,6 +73,7 @@ struct cxl_rcrb_info;
>  resource_size_t __rcrb_to_component(struct device *dev,
>  				    struct cxl_rcrb_info *ri,
>  				    enum cxl_rcrb which);
> +u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb);
>  
>  extern struct rw_semaphore cxl_dpa_rwsem;
>  
> diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> index 7c3fbf9815e9..cbccc222bb91 100644
> --- a/drivers/cxl/core/pci.c
> +++ b/drivers/cxl/core/pci.c
> @@ -722,6 +722,12 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)
>  
>  void cxl_setup_parent_dport(struct device *host, struct cxl_dport *dport)
>  {
> +	struct device *dport_dev = dport->dport_dev;
> +	struct pci_host_bridge *host_bridge;
> +
> +	host_bridge = to_pci_host_bridge(dport_dev);
> +	if (host_bridge->native_cxl_error)
> +		dport->rcrb.aer_cap = cxl_rcrb_to_aer(dport_dev, dport->rcrb.base);

Nothing in this function has a compile dependency on CONFIG_PCIEAER_CXL
the enumeration can be done unconditionally, it's the usage that needs
to be gated.

I believe I had commented on this before, given the late hour I will
need to fix this up post -rc1.