Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7f46c837-fe05-bb61-6247-584eed27aad2@intel.com>
Date:   Thu, 22 Jun 2023 17:10:30 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Betterbird/102.11.1
Subject: Re: [PATCH v6 20/27] cxl/pci: Add RCH downstream port AER register
 discovery
To:     Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
        <vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <Jonathan.Cameron@huawei.com>, <linux-cxl@vger.kernel.org>
CC:     <rrichter@amd.com>, <linux-kernel@vger.kernel.org>,
        <bhelgaas@google.com>
References: <20230622035126.4130151-1-terry.bowman@amd.com>
 <20230622035126.4130151-21-terry.bowman@amd.com>
Content-Language: en-US
From:   Dave Jiang <dave.jiang@intel.com>
In-Reply-To: <20230622035126.4130151-21-terry.bowman@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OWRPeFpma2pwb056YklTYUU0VGFiaHdHc2d2amJIVVYyaDQ4bDJEb1dDd3dj?=
 =?utf-8?B?WlNsNXdLRDNBaEszZDc3dmRyeE04bzFHUWlibE5aZVJPanJCV2xhNHZlbnha?=
 =?utf-8?B?TzUyampLTjlYUk5JcWRZUU1NRDhTc1ZMbUhsVzhKa0NZbGFWZjBVU1VPTGlu?=
 =?utf-8?B?MDU1MlpsZVUydTAzNCtyZEhIcFIyNndxaDVRa0tnYmpkUklobERLMWRhVGln?=
 =?utf-8?B?MkFBNkEwUCtiekpUNXprRXozOU1vcG9zcE5naHJyTGVnZlRUL2NQK2EvUlpU?=
 =?utf-8?B?VHlIKzg1Q0VDMnFqc1BjenFpTUFSaitOWkZhU2R6QzdiNDJURTMyWHc0b2Iw?=
 =?utf-8?B?WTg2cXdJSWo3Q3UvSVkvWGZSSDFuK1Z2bkxHNEpVVEZqMFp3NFZwV0g4ZkE1?=
 =?utf-8?B?eTVkWk1TSHBBaHQxb2dCRFVhbUs3UlRrZGw0OUl4cVJKbU03N3ExcWkySUxD?=
 =?utf-8?B?ZkwxOTdHWkxLdVlXVjh1bktQeWpDQ1pIUEZXUGtQemViWmFUT2dKTGx6VkpU?=
 =?utf-8?B?dnVPaWRJM0IxVnFtWFdZNi9RM0t3V0UxcFZOWk1mdG5FclJidTBxZ1hnOWJR?=
 =?utf-8?B?aWpDZ1hPYktvM044ZjFNRjVLSGFpUk1pVHVHcldmU084TFpDS3N4L215TG9V?=
 =?utf-8?B?NFdYSnRhOUJ3UG5uYzJxQlZhaE1KQ0liTHVIS0FnMmxtNjJVOUY5RFI1YkEw?=
 =?utf-8?B?WlorbVltTTFXbXlIL2FibjAxOTN1U1l5Ynh1WHV5VEVzQUZXQVlmMlJmcDVR?=
 =?utf-8?B?Q3B1eG1OcDN2aWhOZ0FxMXV0OS9UNldTd0Jla0xqWkRLU2djeTlDcHNOdmcw?=
 =?utf-8?B?Z1pMcUJuaWdxVEwvcWVtRHl5MjhMNi9PeklxcHUvQy9mOFA5K2dzUjNyME8r?=
 =?utf-8?B?SStzckVHd1dPY3NUajlZS1EvVjZVS0Y3RUtNR1FqM2lRYndZYWhULzhoaHRG?=
 =?utf-8?B?V2dnQzNVVWVwRWtEVDFNejdZWklLYlp3T1IwUlBLZ2taQktpVDlLOHdNZjYy?=
 =?utf-8?B?SGFQVDBGV1lBeEVlV1NkRmRJcmtkQVJqZitXQ0RYU3p1NHNYaTVGbTZRWGh6?=
 =?utf-8?B?ZnZSdXo1dS91eW5qZ1pzdUhqRVhVYUpoNC9rRHpQdEREV1BIbzdZTGFFSTRE?=
 =?utf-8?B?bjZSRDJQUzBzYUdnQk9QeEhFQ24ramQwUWxRUTZaV2lGbDhJaEFFM29zNmtu?=
 =?utf-8?B?N3dPMmFVWlZFSi9TRC9LZHhpMlMza0FOdVg0TkpyK3hrRXdwWHF5Q0R0KzJT?=
 =?utf-8?B?M1VlR0NOQmh6QUZUZWc4bGlIWGpseVp3K3dMTms4V0UrU0w3bzFUbktUd2tV?=
 =?utf-8?B?bXUrS2xWTTlrdytYTGlqT3ZjWFEyd25vZ09pdFBKdU11Nkc5d0M3ZmtBdjMx?=
 =?utf-8?B?S3A5QXdZTGxxQXNYTFRybWZ5aGp5ODZWZjB5eThjakQxbWZORUZSUHhJZnV4?=
 =?utf-8?B?emhHekJBRFFuN1ZRR1hsbmJXZThiMlh3NWQxUGt5d1lFQU85VWF3eldraXFN?=
 =?utf-8?B?N1psNW92UU02RkIxdnZDN1BxTFNZNmdqNXRKTjd2b3Y4OUxvcmgrY2p5dnZk?=
 =?utf-8?B?LzNSNEc1eW0xdkJHUzNXMWVFc1QxNDdVcDk5N3N1cnhzY1ZSSEFnMzk1cTRV?=
 =?utf-8?B?Y2VLWXZqNTRzWDVIVnRjS2lUMkROeXlUNXRpR1hNMmhybW1NWkFibTBkQk1R?=
 =?utf-8?B?TFV5REl2aTVXUlg2S2FIMC9tcHdTcVcwYXVZZ3V2MlRqY3g3MW9pOEJmemdX?=
 =?utf-8?B?TExRbWN3dkVVeUNrbStQT3daT2NpZzRJQy9PSmVaeHJ4UXdhL0R2eG5HUHRq?=
 =?utf-8?B?a2dGbi8vZ2NJZk8yL1dTcWo5ZnZFRGI2S0VqVzdzbXBDM0ljRDREUWZUWTBn?=
 =?utf-8?B?dG1EM3RtWHRHamxlaGxGcC9JVFdTUzlMNW9wNXdYYTNKSWRtYnpsMEw3K3Bp?=
 =?utf-8?B?R2ZkYjlocnJMTlF5THBsMy9iLzEvZDd2NXBjWW0reDR1NFR2YlhVcEVhQ0Fv?=
 =?utf-8?B?QkxIZGFQaGlrOVdXSE13cFV5a1lpSHpnNzRTUUNUZStiS25GeVhybHZURkhh?=
 =?utf-8?B?VkVaT1o2TkhGaTFmd2RQYmdPVXZuWHVpTGIraStaNmg0amlEam9PbmJhUHMz?=
 =?utf-8?Q?Se95Ui4IdI18G+E3n9xOfaiDU?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9d836d38-dba1-4e60-70dc-08db737e4384
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5984.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jun 2023 00:10:34.0885
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: H6ATNSBIsrXT3cMiHyCS91fG7Lu2TXSWOUwiOcor5a1ZMyq3iG8i5R1uNIWncZu6UdA4TviMzsKOeXQn7Fe+nQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR11MB7259
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/21/23 20:51, Terry Bowman wrote:
> Restricted CXL host (RCH) downstream port AER information is not currently
> logged while in the error state. One problem preventing the error logging
> is the AER and RAS registers are not accessible. The CXL driver requires
> changes to find RCH downstream port AER and RAS registers for purpose of
> error logging.
> 
> RCH downstream ports are not enumerated during a PCI bus scan and are
> instead discovered using system firmware, ACPI in this case.[1] The
> downstream port is implemented as a Root Complex Register Block (RCRB).
> The RCRB is a 4k memory block containing PCIe registers based on the PCIe
> root port.[2] The RCRB includes AER extended capability registers used for
> reporting errors. Note, the RCH's AER Capability is located in the RCRB
> memory space instead of PCI configuration space, thus its register access
> is different. Existing kernel PCIe AER functions can not be used to manage
> the downstream port AER capabilities and RAS registers because the port was
> not enumerated during PCI scan and the registers are not PCI config
> accessible.
> 
> Discover RCH downstream port AER extended capability registers. Use MMIO
> accesses to search for extended AER capability in RCRB register space.
> 
> [1] CXL 3.0 Spec, 9.11.2 - System Firmware View of CXL 1.1 Hierarchy
> [2] CXL 3.0 Spec, 8.2.1.1 - RCH Downstream Port RCRB
> 
> Co-developed-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Reviewed-by: Dave Jiang <dave.jiang@intel.com>

> ---
>   drivers/cxl/core/core.h |  1 +
>   drivers/cxl/core/port.c |  7 ++++++-
>   drivers/cxl/core/regs.c | 35 +++++++++++++++++++++++++++++++++++
>   3 files changed, 42 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index b001669a5133..87467c633123 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -72,6 +72,7 @@ struct cxl_rcrb_info;
>   resource_size_t __rcrb_to_component(struct device *dev,
>   				    struct cxl_rcrb_info *ri,
>   				    enum cxl_rcrb which);
> +u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb);
>   
>   extern struct rw_semaphore cxl_dpa_rwsem;
>   
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index cfae8307de90..37ecd756b224 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -678,7 +678,6 @@ static struct cxl_port *cxl_port_alloc(struct device *uport_dev,
>   	device_set_pm_not_required(dev);
>   	dev->bus = &cxl_bus_type;
>   	dev->type = &cxl_port_type;
> -

As Jonathan pointed out, stray blank line

DJ

>   	return port;
>   
>   err:
> @@ -974,6 +973,8 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
>   		return ERR_PTR(-ENOMEM);
>   
>   	if (rcrb != CXL_RESOURCE_NONE) {
> +		struct pci_host_bridge *host_bridge;
> +
>   		dport->rcrb.base = rcrb;
>   		component_reg_phys = __rcrb_to_component(dport_dev, &dport->rcrb,
>   							 CXL_RCRB_DOWNSTREAM);
> @@ -982,6 +983,10 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
>   			return ERR_PTR(-ENXIO);
>   		}
>   
> +		host_bridge = to_pci_host_bridge(dport_dev);
> +		if (host_bridge->native_cxl_error)
> +			dport->rcrb.aer_cap = cxl_rcrb_to_aer(dport_dev, dport->rcrb.base);
> +
>   		dport->rch = true;
>   	}
>   
> diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
> index e68848075bb6..60b7ba7f4030 100644
> --- a/drivers/cxl/core/regs.c
> +++ b/drivers/cxl/core/regs.c
> @@ -407,6 +407,41 @@ int cxl_setup_regs(struct cxl_register_map *map)
>   }
>   EXPORT_SYMBOL_NS_GPL(cxl_setup_regs, CXL);
>   
> +u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb)
> +{
> +	void __iomem *addr;
> +	u16 offset = 0;
> +	u32 cap_hdr;
> +
> +	if (WARN_ON_ONCE(rcrb == CXL_RESOURCE_NONE))
> +		return 0;
> +
> +	if (!request_mem_region(rcrb, SZ_4K, dev_name(dev)))
> +		return 0;
> +
> +	addr = ioremap(rcrb, SZ_4K);
> +	if (!addr) {
> +		release_mem_region(rcrb, SZ_4K);
> +		return 0;
> +	}
> +
> +	cap_hdr = readl(addr + offset);
> +	while (PCI_EXT_CAP_ID(cap_hdr) != PCI_EXT_CAP_ID_ERR) {
> +		offset = PCI_EXT_CAP_NEXT(cap_hdr);
> +		if (!offset)
> +			break;
> +		cap_hdr = readl(addr + offset);
> +	}
> +
> +	if (offset)
> +		dev_dbg(dev, "found AER extended capability (0x%x)\n", offset);
> +
> +	iounmap(addr);
> +	release_mem_region(rcrb, SZ_4K);
> +
> +	return offset;
> +}
> +
>   resource_size_t __rcrb_to_component(struct device *dev, struct cxl_rcrb_info *ri,
>   				    enum cxl_rcrb which)
>   {