Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <b8baeeea-6b2e-5047-8ba6-dc575a0a4949@intel.com>
Date:   Fri, 23 Jun 2023 14:58:34 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Betterbird/102.11.1
Subject: Re: [PATCH v7 22/27] cxl/pci: Map RCH downstream AER registers for
 logging protocol errors
To:     Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
        <vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <Jonathan.Cameron@huawei.com>, <linux-cxl@vger.kernel.org>
CC:     <rrichter@amd.com>, <linux-kernel@vger.kernel.org>,
        <bhelgaas@google.com>
References: <20230622205523.85375-1-terry.bowman@amd.com>
 <20230622205523.85375-23-terry.bowman@amd.com>
Content-Language: en-US
From:   Dave Jiang <dave.jiang@intel.com>
In-Reply-To: <20230622205523.85375-23-terry.bowman@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aXZSWnJQckxWQW1NbmpXdkpSakx3TjRDcTNkVWpGaThSMzdjYVFlTjQyS2l6?=
 =?utf-8?B?Z09pdDVzYTBxbWRtNUN0ZW15Q3drdU4xY3IvVFlNNmhNemVmSWp5RzdLVUdS?=
 =?utf-8?B?VWl0SG9KWC9GMjVnRDlld1lzV2JpYzhNNGVIejg5UlpIVDc1aEhSS2JQZTdl?=
 =?utf-8?B?NHh0TDJKMHlFNzhqdUwxSDNVMVo4Rncrb1p5ZVJZdE8ybEFlZEtiOWdUdHpL?=
 =?utf-8?B?eXpiL2VpV3JjemU2WFhaemdmZ0VNWUhDcFFPdDhZMkN5djFBd01YRnI4aVBD?=
 =?utf-8?B?WFVWTGd2SUhxTUYyUFZtQ2VxRHVqSDJBM2FuWjlwOVRSUXRPdzFHT2QrNnRH?=
 =?utf-8?B?SzZoM3BwaS9ZdTkvanNzQ1NvTDkzengrK2o1dWNXQ1lnOTJFQUIyOUZpTk1y?=
 =?utf-8?B?QjBDK3hDZGhZVWIzRmRvSVVoN2tVZ3dEL3pVVzdKTjA4Umk2cmp4bmJPd2ds?=
 =?utf-8?B?enZEc1lmRGs2Yk02cDdKaWxlWEFOMURqSlRpSHlhcU5QZTNmNGI2YUgwREpK?=
 =?utf-8?B?UjU4RzltVnBKSWZHK0tZMEJ2UUVNUjl0NlAxcm1jSldoaFgzOURXcFpSRnNT?=
 =?utf-8?B?NlF6UkY0T2taNHgvZ2tTQkdZUkUrdk50MFY1MlNjT0wycFhWRzFLWkFRM05L?=
 =?utf-8?B?Y0RvTEpOMDVLbVZMc2JqbXpoZGlUWm5lK0JwKzB0U2xZdmN3VXVTV2I3UCtD?=
 =?utf-8?B?aC9TZnhQWDJIaGgzYXhSbHV0L0xoKy94M3I0dGl2T2FyMU0ydURFNkZjOFNt?=
 =?utf-8?B?aDJEalo4alRGd21xa2w1MHYwME9Eck1Xa3V0TFQyRTFINVhqK3FKUmpoUUpu?=
 =?utf-8?B?bEU2bWQ5TUltTml2TmkvZFRsMy91ZytNbUVtcktNcTV6bHJJRk1Rek1QTTdr?=
 =?utf-8?B?NU9yc0F4eCtYZ3J2VlhZN2F6V012NWdTNGJlQnJVdkNtUHk4c04xenJaSlZV?=
 =?utf-8?B?TVZQL08rZDFqWjMvSzZ5Z1Uvb29LZkNoYWxneDBBNmQ0YkVEYjZHaURqWkNk?=
 =?utf-8?B?RFBFY1BkZlBYRS9icGNZUU0wb3BqSlJWNTRsU1VoM04wTkZob1ZKS1gra2Vs?=
 =?utf-8?B?aGJsOEZIMUIyTHpWcCtieXVJSzF0K3pEV1RJcVBHY0VIb2orRGRTY1ZScnBx?=
 =?utf-8?B?YVBnZGV4SlhYWjFmTExFTXR3N0hMM2tCekVGWWdGQ0NiVXYrUXRvM1kvYWdr?=
 =?utf-8?B?N2MvZ2lEYUxMYXViYjI1WEkyd3NpSTBWY3B0S2QrenZiZHZGVFFJVGVxQ1pk?=
 =?utf-8?B?aU1BQ0dkUTA1dTFtMEZXZC9OeVJ5clYxc1dWUVRBcWk3NW41dnQvOTlvQzNC?=
 =?utf-8?B?YW1OdCtqMW5Kck5ObnN0NjRQY1IwNFZUZnBiZkJFS2I2Q2Z0MU9BQUdwYktn?=
 =?utf-8?B?V2xXOGN5MllVT3hjYnpzZUorbVREWkZrRGhhVytUY3FXWUlJK2p4ZzJDbzNu?=
 =?utf-8?B?VVRpYVZvLzdMOUgvY0xHUlJGMzNnUjBBM3VjU0FVbTJJM1R0SWJTeTN1SlZF?=
 =?utf-8?B?NFZ1UGNld2VGYXMzdkVJbXl0NXNmWnBlRElPRWplV1o4V0NZdGhEaHdUV3dH?=
 =?utf-8?B?bmJZdDdqeWJUK2h5bE1OR0FmY0xiTWUwdW1FRDE1OWdMSEJlaWRweGlRZTc2?=
 =?utf-8?B?ODVJalhra3hzdzcxMTcvb09jTmJ3NUtINlpuc3lsaEtJYjZUK296K0k2SDZq?=
 =?utf-8?B?cTBlbVNMRnB0d3FjamsraSsyVXY0bnpkcS90WUNVT2NsRjlSRkhQRTkreGVS?=
 =?utf-8?B?Y1NoUGMzdEZVdFBMNnRVd2RXVlVOVXBLZzlpdXpRcHNiMzdtdys4aHdoRk5p?=
 =?utf-8?B?cUNwWlh4Mjh3QWpCVEJxbkpmbHhjQlRNdStuZ1d5L0JxWS9EcmRDMVZySDho?=
 =?utf-8?B?STJCVU8zays0YUs2anNBYU1FVnY0UUhxVmZoaWxYRndjckJ0clRSQmxpOEQ1?=
 =?utf-8?B?UkJ1T2hvZUl6WjQ4SXlRQ0VjRVltZkNKYnhsc2d1N3pNZEEzUVYrZmxRR053?=
 =?utf-8?B?YzQ0UmJ1VTZpNzBIQ1FOR0pIVCtQSUxSa01MckdBTGY3bktXU1pVbEkrSHVs?=
 =?utf-8?B?VkdSak1WYTU2dTgzekZUcVdlZzltZmVJYkRRWXR3RUVMcUtqak10Uzd5VGdl?=
 =?utf-8?Q?lkkm97LMDythTHxCTiMTLcalU?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 661e67a3-5b6d-4f23-144d-08db7434ffaa
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5984.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jun 2023 21:58:38.2105
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Vldf/vLRZLsfdS5ONrArQEqBXYZjucWfE29E2412XMQi0H2WFCcnu/PqftubKbewS+E9Jp27mPVYpXgRDeW0ZQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB8065
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/22/23 13:55, Terry Bowman wrote:
> The restricted CXL host (RCH) error handler will log protocol errors
> using AER and RAS status registers. The AER and RAS registers need
> to be virtually memory mapped before enabling interrupts. Update
> __devm_cxl_add_dport() to include RCH RAS and AER mapping.
> 
> Add 'struct cxl_regs' to 'struct cxl_dport' for saving a pointer to
> the RCH downstream port's AER and RAS registers.
> 
> Co-developed-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Reviewed-by: Dave Jiang <dave.jiang@intel.com>

> ---
>   drivers/cxl/core/port.c | 33 +++++++++++++++++++++++++++++++++
>   drivers/cxl/core/regs.c |  1 +
>   drivers/cxl/cxl.h       | 11 +++++++++++
>   3 files changed, 45 insertions(+)
> 
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index 683f40e98119..a92794e6b7b4 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -8,6 +8,7 @@
>   #include <linux/pci.h>
>   #include <linux/slab.h>
>   #include <linux/idr.h>
> +#include <linux/aer.h>
>   #include <cxlmem.h>
>   #include <cxlpci.h>
>   #include <cxl.h>
> @@ -945,6 +946,36 @@ static void cxl_dport_unlink(void *data)
>   	sysfs_remove_link(&port->dev.kobj, link_name);
>   }
>   
> +static void cxl_dport_map_rch_aer(struct cxl_dport *dport)
> +{
> +	struct cxl_rcrb_info *ri = &dport->rcrb;
> +	struct cxl_port *port = dport->port;
> +	void __iomem *dport_aer = NULL;
> +	resource_size_t aer_phys;
> +
> +	if (dport->rch && ri->aer_cap) {
> +		aer_phys = ri->aer_cap + ri->base;
> +		dport_aer = devm_cxl_iomap_block(&port->dev, aer_phys,
> +				sizeof(struct aer_capability_regs));
> +	}
> +
> +	dport->regs.dport_aer = dport_aer;
> +}
> +
> +static void cxl_dport_map_regs(struct cxl_dport *dport)
> +{
> +	struct cxl_register_map *map = &dport->comp_map;
> +
> +	if (!map->component_map.ras.valid)
> +		dev_dbg(map->dev, "RAS registers not found\n");
> +	else if (cxl_map_component_regs(map, &dport->regs.component,
> +					BIT(CXL_CM_CAP_CAP_ID_RAS)))
> +		dev_dbg(dport->dport_dev, "Failed to map RAS capability.\n");
> +
> +	if (dport->rch)
> +		cxl_dport_map_rch_aer(dport);
> +}
> +
>   static struct cxl_dport *
>   __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
>   		     int port_id, resource_size_t component_reg_phys,
> @@ -1004,6 +1035,8 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
>   	if (rc)
>   		return ERR_PTR(rc);
>   
> +	cxl_dport_map_regs(dport);
> +
>   	cond_cxl_root_lock(port);
>   	rc = add_dport(port, dport);
>   	cond_cxl_root_unlock(port);
> diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
> index 60b7ba7f4030..982ef79670c7 100644
> --- a/drivers/cxl/core/regs.c
> +++ b/drivers/cxl/core/regs.c
> @@ -198,6 +198,7 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
>   
>   	return ret_val;
>   }
> +EXPORT_SYMBOL_NS_GPL(devm_cxl_iomap_block, CXL);
>   
>   int cxl_map_component_regs(struct cxl_register_map *map,
>   			   struct cxl_component_regs *regs,
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 9f46a4e1fbec..12f11cf8e280 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -209,6 +209,13 @@ struct cxl_regs {
>   	struct_group_tagged(cxl_device_regs, device_regs,
>   		void __iomem *status, *mbox, *memdev;
>   	);
> +	/*
> +	 * RCH downstream port specific RAS register
> +	 * @aer: CXL 3.0 8.2.1.1 RCH Downstream Port RCRB
> +	 */
> +	struct_group_tagged(cxl_rch_regs, rch_regs,
> +		void __iomem *dport_aer;
> +	);
>   };
>   
>   struct cxl_reg_map {
> @@ -255,6 +262,8 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
>   			      struct cxl_component_reg_map *map);
>   void cxl_probe_device_regs(struct device *dev, void __iomem *base,
>   			   struct cxl_device_reg_map *map);
> +void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
> +				   resource_size_t length);
>   int cxl_map_component_regs(struct cxl_register_map *map,
>   			   struct cxl_component_regs *regs,
>   			   unsigned long map_mask);
> @@ -598,6 +607,7 @@ struct cxl_rcrb_info {
>    * @rcrb: Data about the Root Complex Register Block layout
>    * @rch: Indicate whether this dport was enumerated in RCH or VH mode
>    * @port: reference to cxl_port that contains this downstream port
> + * @regs: Dport parsed register blocks
>    */
>   struct cxl_dport {
>   	struct device *dport_dev;
> @@ -606,6 +616,7 @@ struct cxl_dport {
>   	struct cxl_rcrb_info rcrb;
>   	bool rch;
>   	struct cxl_port *port;
> +	struct cxl_regs regs;
>   };
>   
>   /**