Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <43eee2aa-8a3d-fa8c-c311-937fe2b02928@intel.com>
Date:   Thu, 22 Jun 2023 16:58:08 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Betterbird/102.11.1
Subject: Re: [PATCH v6 14/27] cxl/pci: Early setup RCH dport component
 registers from RCRB
To:     Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
        <vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <Jonathan.Cameron@huawei.com>, <linux-cxl@vger.kernel.org>
CC:     <rrichter@amd.com>, <linux-kernel@vger.kernel.org>,
        <bhelgaas@google.com>
References: <20230622035126.4130151-1-terry.bowman@amd.com>
 <20230622035126.4130151-15-terry.bowman@amd.com>
Content-Language: en-US
From:   Dave Jiang <dave.jiang@intel.com>
In-Reply-To: <20230622035126.4130151-15-terry.bowman@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NXNxUE9OcG9hM01qSFRZdjczaFV5dTlwTndxNEZyTXlkRmVFR002U1V2WUpY?=
 =?utf-8?B?cUhObXVkc3VsTXBwM3VsYmYvWW9lNzB0RmtaQ3JRRlhqbUZyVFVOWTFhQUVp?=
 =?utf-8?B?aXNrQzhkU1BXaEt2VXBmQzRRL0hNWitIVnNlV1BSWG1vckpuT01FamRmRW1y?=
 =?utf-8?B?Y0FEaDFES1gxRjZ1cDB6Yjhnajc4SEhtU2ZKcmpLY0cwOG9QUUZ4SEtYQjg4?=
 =?utf-8?B?SThmQ3ZKU3FjUmxGUUpsZm1NbGFaSWdEaHZkY3FlYUwwZlh4dHVrTVNLVzVU?=
 =?utf-8?B?WFFYUWoxc2M2M2I2NFlFd2ZKK2ZSQlc5M21JcDNpb0prQnJLUS8yRThsT282?=
 =?utf-8?B?N3ZzaWdrYnNnbmJtZXo0MWpuZjNFdXg4K0YrUTRQT0N0aHhjRGhmRVZaUWEv?=
 =?utf-8?B?cEtXZmFRMDBzZE85S2hMaDU5ME52c3JVTFJDOUZrNGlFL2h1NXYxUklPMGRs?=
 =?utf-8?B?VnJtYjRWQlRJMDdMaDUyY3Qyay92aVJ6WEZ6azdCd0Z0UVpJemhzdHpjeDNI?=
 =?utf-8?B?cWhhRnJBSW02V2N5UXdSSUNmOG42eWFjRkZBNTFHTzZuZm81WGRHZmJCdk1J?=
 =?utf-8?B?b0hLN3ZvSGxrSjNlSmZzbUc1cmM5WmR0MWwvMmhVNHNQT1JzZUlYc1g4SXJ6?=
 =?utf-8?B?U2pFekFRem1USTdBL3ZJU3I0TmNEdkpmY01kVEY5a2o1ZHlyNTFWQ0tCcDlt?=
 =?utf-8?B?RnBuOENvZTVpVHBSN0dQaWswaFliczJWUzFJVVN3d3NwUGpEVi9qblZxL2RX?=
 =?utf-8?B?VWpHckJZNk9NT1J0ZUNuOVhYQ1NFRlJkSitlS2lReDEyb2JIVFVDb2FhZDJk?=
 =?utf-8?B?ZlZXSzBqOFp5MmZnZGl1Q213RzJDdG55MEw0S2svb3BlOVpCdGtYbGZmL2hS?=
 =?utf-8?B?VWN1TXFrMGs2dE1jTnI5K1FkOEtjbFNYN3V2M2s1cnlwVEVwei9IUVFubVJV?=
 =?utf-8?B?UWM1bkU5ZWFqc2xmeDdHeGhOZHpKVnEyNit5ME8xY2E1S1V3ZG1mYk5WZ2RT?=
 =?utf-8?B?YVVReGlzNGtVV3Uwd1Y3ZXpqQjVnb1hZQ3VwUlAvQ2Y4YUxNeUxOdDBBYTRU?=
 =?utf-8?B?amN2WHFWZnIxM3E3ZVJPak5POWJieGpWM2kvZ3lqZ0xvRDR4ZllQdUNpaVl4?=
 =?utf-8?B?TC96ZGVJWjZ4ZVc5NGZVbUVxRUhlbjlpTUZYYUR2WGVZTytiQTBlYnYzU2Zu?=
 =?utf-8?B?R21DakJ5NkVTeW5iSytONjNwR0VPL1V6YUxaZHJwZHJubmViVHVEVnhNdUlE?=
 =?utf-8?B?aitpQlVkN0JNSVM1alJlR002SU40TDloLzVpcmN5Q2ExWU5acnBsQkdXS3ox?=
 =?utf-8?B?TXkreGVqNmZidUhRbUVZUVo1Z0RoZ0wrdTBpMzFCa2ZmQmc1NmdhVnJvbDZR?=
 =?utf-8?B?WkIxYmNxMnRoK0VIUEFaQkhmMUF5S2xjM2pxeUNkdHQzZFlNSVg4aldtcEYy?=
 =?utf-8?B?UEd3VjVHQW9qZXJjU3hWcGpud0VCaEFqeXpsd21LS2pab0dTUnZrb2Q2VURt?=
 =?utf-8?B?N0UyVksvSVdNWDdLdklUQml0dEtydDJGM2d1S0hnRnhMQlcrTmVaQ0VQVndL?=
 =?utf-8?B?bkRGWUhITGV0YUI1YmgzdGdTM2dFUE1lLzREWEFBUzRFWG92dGdzUEp2Rm9J?=
 =?utf-8?B?NEtJeFRzdHlNUzlRQTVGNVpnbG1PTVUwMVJqZFgvakJ4aFlTS2EzVVRhZ2NF?=
 =?utf-8?B?Zm9JbkJVUkE2cmFrRmdOQ29najlnTGRIR3lLV0paQmMyVy9oRkI1YnhNYXpx?=
 =?utf-8?B?ZUNHeTdYQXA1d2VtT21Ka0p5bGVLOVJZYjlsQWlpZG0wczRkaEM1ODJKOUFV?=
 =?utf-8?B?QXdBb1poWWVnNzNndXJIR1N2RFFxMm5STDNJZ3l4VTNvWWVIaXdaSXMvS1dH?=
 =?utf-8?B?dHhWSVMvQnN2RnpyNzFCY0VZZHd4ZGFEUHhtTFpJUE8yRW50Q3ZOcVp1dXBS?=
 =?utf-8?B?U3J3N2lad1Rzem85MGgwaERyNnYrR3loQ21SM24wRmluMWtXM3RjdEZ2bkMy?=
 =?utf-8?B?MDAwckIzUzJ1Um9FUTBrNEZzVlNNa0VyMGpvNW9YTVR2YlEwRUZ0WnBOUVhp?=
 =?utf-8?B?aHVZR1puN3pLQ0hjQXdXTUJXUXJVUlIrYmZacjYrcHNCdGpNU1lkKzl0WElN?=
 =?utf-8?Q?8FrC4cUDRXs3wsBdIqAlkCuN/?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0093dd63-eb23-4dc7-c5e8-08db737c89df
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5984.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jun 2023 23:58:13.0868
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0AOyY3051YKFx04Pb1mVLuiVsc9s9QmptUA9m5Geqez5eqJ3H6eDJvlLiG00SxLi+OvDWCL4ghnVVShDv5dEEw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR11MB5649
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/21/23 20:51, Terry Bowman wrote:
> From: Robert Richter <rrichter@amd.com>
> 
> CXL RAS capabilities must be enabled and accessible as soon as the CXL
> endpoint is detected in the PCI hierarchy and bound to the cxl_pci
> driver. This needs to be independent of other modules such as cxl_port
> or cxl_mem.
> 
> CXL RAS capabilities reside in the Component Registers. For an RCH
> this is determined by probing RCRB which is implemented very late once
> the CXL Memory Device is created.
> 
> Change this by moving the RCRB probe to the cxl_pci driver. Do this by
> using a new introduced function cxl_pci_find_port() similar to
> cxl_mem_find_port() to determine the involved dport by the endpoint's
> PCI handle. Plug this into the existing cxl_pci_setup_regs() function
> to setup Component Registers. Probe the RCRB in case the Component
> Registers cannot be located through the CXL Register Locator
> capability.
> 
> This unifies code and early sets up the Component Registers at the
> same time for both, VH and RCH mode. Only the cxl_pci driver is
> involved for this. This allows an early mapping of the CXL RAS
> capability registers.
> 
> Signed-off-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> ---
>   drivers/cxl/core/port.c |  7 +++++
>   drivers/cxl/cxl.h       |  2 ++
>   drivers/cxl/mem.c       |  9 -------
>   drivers/cxl/pci.c       | 57 ++++++++++++++++++++++++++++++++++-------
>   4 files changed, 57 insertions(+), 18 deletions(-)
> 
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index e0d2e7596440..679226023f0c 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -1480,6 +1480,13 @@ int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd)
>   }
>   EXPORT_SYMBOL_NS_GPL(devm_cxl_enumerate_ports, CXL);
>   
> +struct cxl_port *cxl_pci_find_port(struct pci_dev *pdev,
> +				   struct cxl_dport **dport)
> +{
> +	return find_cxl_port(pdev->dev.parent, dport);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_pci_find_port, CXL);
> +
>   struct cxl_port *cxl_mem_find_port(struct cxl_memdev *cxlmd,
>   				   struct cxl_dport **dport)
>   {
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 7fbc52b81554..fe95f08acb69 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -664,6 +664,8 @@ struct cxl_port *find_cxl_root(struct cxl_port *port);
>   int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd);
>   void cxl_bus_rescan(void);
>   void cxl_bus_drain(void);
> +struct cxl_port *cxl_pci_find_port(struct pci_dev *pdev,
> +				   struct cxl_dport **dport);
>   struct cxl_port *cxl_mem_find_port(struct cxl_memdev *cxlmd,
>   				   struct cxl_dport **dport);
>   bool schedule_cxl_memdev_detach(struct cxl_memdev *cxlmd);
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index 7638a7f8f333..205e2e280aed 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -65,15 +65,6 @@ static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd,
>   		ep->next = down;
>   	}
>   
> -	/*
> -	 * The component registers for an RCD might come from the
> -	 * host-bridge RCRB if they are not already mapped via the
> -	 * typical register locator mechanism.
> -	 */
> -	if (parent_dport->rch && cxlds->component_reg_phys == CXL_RESOURCE_NONE)
> -		cxlds->component_reg_phys =
> -			cxl_rcd_component_reg_phys(&cxlmd->dev, parent_dport);
> -
>   	endpoint = devm_cxl_add_port(host, &cxlmd->dev,
>   				     cxlds->component_reg_phys,
>   				     parent_dport);
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 945ca0304d68..99a75c54ee39 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -274,27 +274,66 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)
>   	return 0;
>   }
>   
> +/*
> + * Assume that any RCIEP that emits the CXL memory expander class code
> + * is an RCD
> + */
> +static bool is_cxl_restricted(struct pci_dev *pdev)
> +{
> +	return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END;
> +}
> +
> +static int cxl_rcrb_get_comp_regs(struct pci_dev *pdev,
> +				  struct cxl_register_map *map)
> +{
> +	struct cxl_port *port;
> +	struct cxl_dport *dport;
> +	resource_size_t component_reg_phys;
> +
> +	*map = (struct cxl_register_map) {
> +		.dev = &pdev->dev,
> +		.resource = CXL_RESOURCE_NONE,
> +	};
> +
> +	port = cxl_pci_find_port(pdev, &dport);
> +	if (!port)
> +		return -EPROBE_DEFER;
> +
> +	component_reg_phys = cxl_rcd_component_reg_phys(&pdev->dev, dport);
> +
> +	put_device(&port->dev);
> +
> +	if (component_reg_phys == CXL_RESOURCE_NONE)
> +		return -ENXIO;
> +
> +	map->resource = component_reg_phys;
> +	map->reg_type = CXL_REGLOC_RBI_COMPONENT;
> +	map->max_size = CXL_COMPONENT_REG_BLOCK_SIZE;
> +
> +	return 0;
> +}
> +
>   static int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
>   			      struct cxl_register_map *map)
>   {
>   	int rc;
>   
>   	rc = cxl_find_regblock(pdev, type, map);
> +
> +	/*
> +	 * If the Register Locator DVSEC does not exist, check if it
> +	 * is an RCH and try to extract the Component Registers from
> +	 * an RCRB.
> +	 */
> +	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev))
> +		rc = cxl_rcrb_get_comp_regs(pdev, map);
> +
>   	if (rc)
>   		return rc;
>   
>   	return cxl_setup_regs(map);
>   }
>   
> -/*
> - * Assume that any RCIEP that emits the CXL memory expander class code
> - * is an RCD
> - */
> -static bool is_cxl_restricted(struct pci_dev *pdev)
> -{
> -	return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END;
> -}
> -
>   /*
>    * CXL v3.0 6.2.3 Table 6-4
>    * The table indicates that if PCIe Flit Mode is set, then CXL is in 256B flits