Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <cf141508-797f-8ae0-4a1a-ce5b046fb364@intel.com>
Date:   Thu, 22 Jun 2023 15:51:29 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Betterbird/102.11.1
Subject: Re: [PATCH v6 04/27] cxl/rch: Prepare for caching the MMIO mapped
 PCIe AER capability
To:     Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
        <vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <Jonathan.Cameron@huawei.com>, <linux-cxl@vger.kernel.org>
CC:     <rrichter@amd.com>, <linux-kernel@vger.kernel.org>,
        <bhelgaas@google.com>
References: <20230622035126.4130151-1-terry.bowman@amd.com>
 <20230622035126.4130151-5-terry.bowman@amd.com>
Content-Language: en-US
From:   Dave Jiang <dave.jiang@intel.com>
In-Reply-To: <20230622035126.4130151-5-terry.bowman@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dnFEVGpnb2xsaGdxc2V3SjVZRDUyc2FmdENueDIxS0dCYVBpT3NEOUR0OTcx?=
 =?utf-8?B?MkhRNWJFTGVFNkg3QXQ1ZWhNZlU5ejlTbE9XbHFQSDdoMm85SmQ5SjJoNFpR?=
 =?utf-8?B?YzVpQytNMEhBUndQVTlOMk5rU3JSNDJyMHZGcEdaT0Zoc3FqMWRpQTJCRjFy?=
 =?utf-8?B?alNMaG9WOGR1NFdmZHlEMlhCbFdLRGM1OUY3dkNaZXEvZTZlUjlMYnBKNEla?=
 =?utf-8?B?SE5XcFVBMUpJcXM0aWNOaFFtN3h6S2tUNzZIK0cydjhGSGVRTVlKSWMzNGJi?=
 =?utf-8?B?TU1jZjdOOGRGMW5OUlU2Q1kxQS9WME1pL0U5VjhtRW1xb0Q4QU9yL3Jzc2dN?=
 =?utf-8?B?QysrOUNRNjRVaWRjaUhLK25HYVBPQjUrVFZIb09nV0Y1QUl1N1huOGVTNHZR?=
 =?utf-8?B?UUtMbWNwWHgxVTAyckRWanNGZ21XVi8vTU0zTjdMb1FyTFZGZ0RsOXdmZzdO?=
 =?utf-8?B?K0JGdlAxMVd1TEh6TmtRamlVcEFRUVp6bU9SdEZIQUl4bFNWWFJHSGczbW1M?=
 =?utf-8?B?SkJuOVQ1S3M5RlU4Q0ppekZzZWZLbjlRa1VkM3dmcVFtdGszTk80ZDMxOGtY?=
 =?utf-8?B?OGpjdU9rWjJEZ3B1MDhkOUNVV0x1MElva3oxZGNGZnc2TldzZkVjRnVSK042?=
 =?utf-8?B?MFVEK3F0cE1VMjJVWmg5dnQ3aHJWOUlSTFRmR05jVE1KL2d3YWF6cFpCU0dK?=
 =?utf-8?B?elozcHErbmZQTElZNSttQ2M5UnpCekh6QXdQUlJoQVlweHdWWlJFZDE0THZt?=
 =?utf-8?B?eXNxdXUrUkJDVmJFdlI0bVRrblF2b205NmZoeEk1bFRLbXdiT0xaWUUxRDdx?=
 =?utf-8?B?aUlpeTRUeXh3aVN1Ulp6b3Z0OXFqbjFicDRCVlh6NHZ4M1Y2a2JuektWdk1o?=
 =?utf-8?B?UHp4cDBVdnJQOUFSL04wS1FRbURwSDUveFVTdnVnMDduUnBxMjBhZWhVVUxp?=
 =?utf-8?B?c2dZMEMrd1dGUHc2OEo5UGVhckY5Z0xMbExvaGkvTWhybDdGODQxakR0VVJz?=
 =?utf-8?B?emU1ZVN1MHZxV0J2M29weVYwaUkxNE9XR2NnbDBadjZvMkxBYWNDTjN6VGdy?=
 =?utf-8?B?VnBzWmhPQVFRaWg5cWpiWnBvbTJqMGRlTGJ3dDBGK05lSFhtL2xVVm5xT002?=
 =?utf-8?B?a0tDY1BQZGlYSWw5eUxRZlpQeTNKb2Y1cFBNbEtWTmsrYmNuNVlYWUhKemdp?=
 =?utf-8?B?a1pySVNKTm85MkNERHAvTEIrMmoyemJ0cDNEUXp0WHBjekNKUXZKd0M4bnYx?=
 =?utf-8?B?VW14dmxyaTdMdXQ0cEZ3a3FmY3Nydm5jK2x3aG91dGg1dWY0VEU1Z2Q1bllU?=
 =?utf-8?B?WllTSmVvV1pFUUZWMkhVZ0I1RUFsc3RkTW12b2xXVTNsRjRrM1ErL29UcWpG?=
 =?utf-8?B?YnpRZ1hLTGZFbXRSa0hPRU0zNDFMTUFHNWZjTUFLbmFWVE8zM0FBSW9sVFVa?=
 =?utf-8?B?ckpYR1JnYlQ1dUF6ZllUdXR1aVZZc3A2TXRsNkx2b3hDTWFPRGp6N2dhUjJm?=
 =?utf-8?B?RkQvcTFSZGovWGd4Q0NsTm44L3lSanlGT1M4THg3YjU5NmthOFY3VXY5Uk13?=
 =?utf-8?B?eHpCYU9lOVhNTkVKNzhKMzdrWFVIRzdnWkIrRG4xOGxZbno2cG0wY01NY1l1?=
 =?utf-8?B?alU1UWRreGxycWlSZ0hEWDVib0x6TU1MY1d1OEgxQjFBL2FHT2cyYm9aU1g3?=
 =?utf-8?B?Uk83N1Q0WVpldTh5dEVPL1hBSmtvYmptTnIyYUNMcjBkREhnWGZGSmd2OWlp?=
 =?utf-8?B?VmY3WVM5aGtZOFBzMlpwb1dsYUFKMDA0VTZQcm16N3pQcEYrSG8veVB3WTB3?=
 =?utf-8?B?NmlmakNhYmJVTHQweVRXRTBNZnNGYU1pc1cwMkRYc2c1MnhPYTRheXNialNQ?=
 =?utf-8?B?MUNZTTBqYmdRbk9KeU4rUitRQXg3eFVaa2FadEpLZ1ZaNXN4U0RsenhFNERn?=
 =?utf-8?B?aHZLR2JXbHJsVjJUYW5ldC9LdjlKbXpYYWZMNGptdDFrRGJ5OWpCcVNTWG0v?=
 =?utf-8?B?V2k3dzd0UlY5eTd5bnA5MGFLWDlCclRSNzkyVmJlYmJMcUZwQ1h3NkpwVEs2?=
 =?utf-8?B?SGErcGFOQXZMTDBJMExQTGYyVkhENzVzSTZ0emM5VHRKc25GQ3ZRMDBvYnZO?=
 =?utf-8?Q?ABgRsYrVjDUY0SNGqQDm3OCVV?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e621827b-10d6-4120-8c54-08db737339e0
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5984.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jun 2023 22:51:33.4049
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3v8JaFRGzpkbpsyEnS1vSFt9Oodimv+iRZUikax7EZMcG2XxhYg5Z2YFsmfRqIZv9tLLhbeqovFCEQB3sg7MkA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB6408
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/21/23 20:51, Terry Bowman wrote:
> From: Dan Williams <dan.j.williams@intel.com>
> 
> Prepare cxl_probe_rcrb() for retrieving more than just the component
> register block. The RCH AER handling code wants to get back to the AER
> capability that happens to be MMIO mapped rather then configuration
> cycles.
> 
> Move RCRB specific downstream port data, like the RCRB base and the
> AER capability offset, into its own data structure ('struct
> cxl_rcrb_info') for cxl_probe_rcrb() to fill. Extend 'struct
> cxl_dport' to include a 'struct cxl_rcrb_info' attribute.
> 
> This centralizes all RCRB scanning in one routine.
> 
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> ---
>   drivers/cxl/core/core.h       | 4 +++-
>   drivers/cxl/core/port.c       | 4 ++--
>   drivers/cxl/core/regs.c       | 5 +++--
>   drivers/cxl/cxl.h             | 9 +++++++--
>   tools/testing/cxl/test/mock.c | 4 +++-
>   5 files changed, 18 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index bd0a5788c696..b001669a5133 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -68,7 +68,9 @@ enum cxl_rcrb {
>   	CXL_RCRB_DOWNSTREAM,
>   	CXL_RCRB_UPSTREAM,
>   };
> -resource_size_t __rcrb_to_component(struct device *dev, resource_size_t rcrb,
> +struct cxl_rcrb_info;
> +resource_size_t __rcrb_to_component(struct device *dev,
> +				    struct cxl_rcrb_info *ri,
>   				    enum cxl_rcrb which);
>   
>   extern struct rw_semaphore cxl_dpa_rwsem;
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index 45f5299af7a6..76888c75dae4 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -939,7 +939,8 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
>   		return ERR_PTR(-ENOMEM);
>   
>   	if (rcrb != CXL_RESOURCE_NONE) {
> -		component_reg_phys = __rcrb_to_component(dport_dev, rcrb,
> +		dport->rcrb.base = rcrb;
> +		component_reg_phys = __rcrb_to_component(dport_dev, &dport->rcrb,
>   							 CXL_RCRB_DOWNSTREAM);
>   		if (component_reg_phys == CXL_RESOURCE_NONE) {
>   			dev_warn(dport_dev, "Invalid Component Registers in RCRB");
> @@ -957,7 +958,6 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
>   	dport->port_id = port_id;
>   	dport->component_reg_phys = component_reg_phys;
>   	dport->port = port;
> -	dport->rcrb = rcrb;
>   
>   	cond_cxl_root_lock(port);
>   	rc = add_dport(port, dport);
> diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
> index 564dd430258a..6c4b33133918 100644
> --- a/drivers/cxl/core/regs.c
> +++ b/drivers/cxl/core/regs.c
> @@ -332,10 +332,11 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
>   }
>   EXPORT_SYMBOL_NS_GPL(cxl_find_regblock, CXL);
>   
> -resource_size_t __rcrb_to_component(struct device *dev, resource_size_t rcrb,
> +resource_size_t __rcrb_to_component(struct device *dev, struct cxl_rcrb_info *ri,
>   				    enum cxl_rcrb which)
>   {
>   	resource_size_t component_reg_phys;
> +	resource_size_t rcrb = ri->base;
>   	void __iomem *addr;
>   	u32 bar0, bar1;
>   	u16 cmd;
> @@ -400,6 +401,6 @@ resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
>   {
>   	if (!dport->rch)
>   		return CXL_RESOURCE_NONE;
> -	return __rcrb_to_component(dev, dport->rcrb, CXL_RCRB_UPSTREAM);
> +	return __rcrb_to_component(dev, &dport->rcrb, CXL_RCRB_UPSTREAM);
>   }
>   EXPORT_SYMBOL_NS_GPL(cxl_rcd_component_reg_phys, CXL);
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 28888bb0c088..7c8674079f1a 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -582,12 +582,17 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev)
>   	return xa_load(&port->dports, (unsigned long)dport_dev);
>   }
>   
> +struct cxl_rcrb_info {
> +	resource_size_t base;
> +	u16 aer_cap;
> +};
> +
>   /**
>    * struct cxl_dport - CXL downstream port
>    * @dport: PCI bridge or firmware device representing the downstream link
>    * @port_id: unique hardware identifier for dport in decoder target list
>    * @component_reg_phys: downstream port component registers
> - * @rcrb: base address for the Root Complex Register Block
> + * @rcrb: Data about the Root Complex Register Block layout
>    * @rch: Indicate whether this dport was enumerated in RCH or VH mode
>    * @port: reference to cxl_port that contains this downstream port
>    */
> @@ -595,7 +600,7 @@ struct cxl_dport {
>   	struct device *dport;
>   	int port_id;
>   	resource_size_t component_reg_phys;
> -	resource_size_t rcrb;
> +	struct cxl_rcrb_info rcrb;
>   	bool rch;
>   	struct cxl_port *port;
>   };
> diff --git a/tools/testing/cxl/test/mock.c b/tools/testing/cxl/test/mock.c
> index 30119a16ae85..dbeef5c6f606 100644
> --- a/tools/testing/cxl/test/mock.c
> +++ b/tools/testing/cxl/test/mock.c
> @@ -271,8 +271,10 @@ struct cxl_dport *__wrap_devm_cxl_add_rch_dport(struct cxl_port *port,
>   	if (ops && ops->is_mock_port(dport_dev)) {
>   		dport = devm_cxl_add_dport(port, dport_dev, port_id,
>   					   CXL_RESOURCE_NONE);
> -		if (!IS_ERR(dport))
> +		if (!IS_ERR(dport)) {
> +			dport->rcrb.base = rcrb;
>   			dport->rch = true;
> +		}
>   	} else
>   		dport = devm_cxl_add_rch_dport(port, dport_dev, port_id, rcrb);
>   	put_cxl_mock_ops(index);