Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <c6e238ed-e60b-86f7-0c41-26318d4d63c3@intel.com>
Date:   Fri, 23 Jun 2023 14:43:30 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Betterbird/102.11.1
Subject: Re: [PATCH v7 21/27] cxl/pci: Update CXL error logging to use RAS
 register address
To:     Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
        <vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <Jonathan.Cameron@huawei.com>, <linux-cxl@vger.kernel.org>
CC:     <rrichter@amd.com>, <linux-kernel@vger.kernel.org>,
        <bhelgaas@google.com>
References: <20230622205523.85375-1-terry.bowman@amd.com>
 <20230622205523.85375-22-terry.bowman@amd.com>
Content-Language: en-US
From:   Dave Jiang <dave.jiang@intel.com>
In-Reply-To: <20230622205523.85375-22-terry.bowman@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?enpoNDQwSzlONVovY1Y1R3VaV2tDaEpSOFRtYnc5WHNMVmpia0UxY0FubXNW?=
 =?utf-8?B?NUtibXlpZGlvMXdna2c3azVnQXlXVDNINEl3UUxjUHdhLzZBT2tIb2htMWFm?=
 =?utf-8?B?MmYxdDJFOTF5ckcxN21EVGRCNDVjNzcwR2xiVkJOdWkrc1JjRzk4bmhEbTRT?=
 =?utf-8?B?ZjJzbmllWk9ZU3Bkc2lRdlhXQ04wWnVKU0pBZW0wbWJlWjhxK3p6cU0vVSt1?=
 =?utf-8?B?SHJ0L3ZNbFlRT016UUFBZmMyU1p3TnF4bEhxMmtaMkppMWpkNmhoU3NRdE1l?=
 =?utf-8?B?d2VIektGNll6b2NQbWZqbEhiL3A2S015dUFrNjIrajBqZlppMVNHZmJxM0FY?=
 =?utf-8?B?MUY4cXRMYjdrMFdzUDdmNGFKZHZpUlRFakVoa0d2VnZsdW5TTytESjBxQVpO?=
 =?utf-8?B?VHZFMU1MdjJpMXUveHk1cU0vdDNGZURacUoyUnFsN3NtUHBDdXBGbXo0VExi?=
 =?utf-8?B?Yk9IYml3c3BBTDd6WllPN3ZES0dzS1lRMEtQaDNGRHVnQWJhOFJpcW1qMDV3?=
 =?utf-8?B?aGIyQ1cyYmFmeGJ5QTRkZVZMNnZKVnNWVWVkQWF6VGFoL0NyT205OUtPQXBn?=
 =?utf-8?B?Q3Z4YWJjc3lEdWVaVEVCOFFDV3F6NzVEbHVCK1QwK1p6ZG0zK1djb2ZSWC8r?=
 =?utf-8?B?ZEgyczJFemhmTnd4TFF4YkdiVnpWeDJnR2lPSjgrSjBJVUVGcWZtQUVWSjQ5?=
 =?utf-8?B?UW1WVjBmaWNRdmZuUW5pUDJhQlA3TVRneXVWanpOcnVqSEZZYnZLM3NDSzg0?=
 =?utf-8?B?bUh0YThwelZRT3Y0R1ZjR2tLTlBWTlplQkFMNkxWZFh3NlhPcEZZdmcydTRK?=
 =?utf-8?B?MDE0RFgwNDd6c0hyYW5yZzJ5YW56cEU4ZHFQa2F0V2RHUEVXVWZPTitadXZv?=
 =?utf-8?B?amRVU2VXcGdIaVhhL3dzMExpbkxPTVh1dlZWRkxzdkJ1U3NxQUs5dXhWd2JD?=
 =?utf-8?B?RmI5YjBSN2wyN1p3WHoybVZwdGN1ZUQzZW9Ed1o2bU8vUGNhdFpVMS9OVWVX?=
 =?utf-8?B?ektpTmFLQzRFcUo3YXU2RWs0eW43ZjNlVTVTMkhMUmU4ZVF0eFJsV09UVFBx?=
 =?utf-8?B?dWlERkc1YXJBb0RLUlZvN3RmOWxDb013NFN1Mm1rcG5HbnRjTlNacFE2ZXpY?=
 =?utf-8?B?VEVKZ2pURm1lb0VvVzg5VzVrRC9hZ2l6bW1YcndsWU5YVktKZE9wSHJ2aEVs?=
 =?utf-8?B?QTgrZFNWSFVLTERQSENNbzdUSGVDUE81M0N6REhvWG9yeUVPcFdCejJLbUZ3?=
 =?utf-8?B?K2w4eDdhV2ZYMEpOQ29Za2tER2RPS1B1ZEFicEdNNm5BNjJPY2lnRFNQNWcy?=
 =?utf-8?B?VGpYRmk3UnRPQnV6OXRzcXgxejVWSUQ3bTd2anAxVHQzREZVd28wcVY2NUpn?=
 =?utf-8?B?SmEwNXFYNWlrRkUxUndKN1BoUlkxSGppSTZGYzlUVFM1OHNJQWNKbUdhd2pH?=
 =?utf-8?B?T1dVU2tFOUdVMENLaFV0bGVEc0dWSHNRS2JjeDc3NndJdzlDL0FxRXhsN2NH?=
 =?utf-8?B?L2F6a3BVazdRVGJ0cFV4ZUVURU5MeHlBdUQzWlR1Z3B4dlNON3I0TWNQZHhB?=
 =?utf-8?B?K2xnNk9LcUNGcCtCTDl0OENMTXJ0N1pYWDFvZUY2NWdWWVlCYU5FTnNSN3p1?=
 =?utf-8?B?ai9iYWE1NTVzUWRmLzhiYTU1M1llTXlBdDJRZWZlOTVDL2JKU293cXNMM1Vi?=
 =?utf-8?B?YXJWZ2V3dFFjV2FyYUpDT0ZHRXJiVjdhL01RZERTbXptdG91cXM2a1NUVVVE?=
 =?utf-8?B?QUZXenFuT2lGUXBMYk9zeGFPSDZRa2JTWnRadElZekJiWTB3c0FpOEtLYzM1?=
 =?utf-8?B?SEFOSlYzcHM0MTlVbCtwd3FwMkxHOW5Gd3FjRVZ6OGZyRGlFeVRIMHRGTG56?=
 =?utf-8?B?YlZLZ0FhREd3bVNER0M3alV6VGtsU1NMcVFubEFUVEo2VEhJN1BoODhkRUJU?=
 =?utf-8?B?NkdJRFkrVmhKVXl1QkZnZjNQS05ac3VvVlZhaUhFZlRCSzR5T0ZkUitUR0p4?=
 =?utf-8?B?NjZKUW40Wmx1WFptemlNY01wSVgrT0ZXZUZyWEttTks0V3JjYW9EVUxvUjgy?=
 =?utf-8?B?N1hWMW90U0FwaFpBVllKNk5iQk5WaXpTdzRXcWVyUDdzd2lUSktVUFkvalRQ?=
 =?utf-8?B?ZmVXNWgrSURyL21oU0VVQy9kMHZvd3JGdkRFaTFSYTdNY0U3Qnp6dXBRT2xo?=
 =?utf-8?B?alE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 14644eb4-0ab0-41aa-3aa6-08db7432e531
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5984.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jun 2023 21:43:34.8838
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: bAhb/vWjXYAxSa2/qO4s33EfMlhS1OO9gavhxPHK02sS25k4w8iQm08WCjBIqii2Adta8F+KtUSTH0fxprhhPQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR11MB4980
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/22/23 13:55, Terry Bowman wrote:
> The CXL error handler currently only logs endpoint RAS status. The CXL
> topology includes several components providing RAS details to be logged
> during error handling.[1] Update the current handler's RAS logging to use a
> RAS register address. Also, update the error handler function names to be
> consistent with correctable and uncorrecable RAS. This will allow for

s/uncorrecable/uncorrectable/

> adding support to log other CXL component's RAS details in the future.
> 
> [1] CXL3.0 Table 8-22 CXL_Capability_ID Assignment
> 
> Co-developed-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

Reviewed-by: Dave Jiang <dave.jiang@intel.com>

> ---
>   drivers/cxl/core/pci.c | 44 +++++++++++++++++++++++++++++-------------
>   1 file changed, 31 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> index 375f01c6cad6..9cb39835e154 100644
> --- a/drivers/cxl/core/pci.c
> +++ b/drivers/cxl/core/pci.c
> @@ -665,32 +665,36 @@ void read_cdat_data(struct cxl_port *port)
>   }
>   EXPORT_SYMBOL_NS_GPL(read_cdat_data, CXL);
>   
> -void cxl_cor_error_detected(struct pci_dev *pdev)
> +static void __cxl_handle_cor_ras(struct cxl_dev_state *cxlds,
> +				 void __iomem *ras_base)
>   {
> -	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
>   	void __iomem *addr;
>   	u32 status;
>   
> -	if (!cxlds->regs.ras)
> +	if (!ras_base)
>   		return;
>   
> -	addr = cxlds->regs.ras + CXL_RAS_CORRECTABLE_STATUS_OFFSET;
> +	addr = ras_base + CXL_RAS_CORRECTABLE_STATUS_OFFSET;
>   	status = readl(addr);
>   	if (status & CXL_RAS_CORRECTABLE_STATUS_MASK) {
>   		writel(status & CXL_RAS_CORRECTABLE_STATUS_MASK, addr);
>   		trace_cxl_aer_correctable_error(cxlds->cxlmd, status);
>   	}
>   }
> -EXPORT_SYMBOL_NS_GPL(cxl_cor_error_detected, CXL);
> +
> +static void cxl_handle_endpoint_cor_ras(struct cxl_dev_state *cxlds)
> +{
> +	return __cxl_handle_cor_ras(cxlds, cxlds->regs.ras);
> +}
>   
>   /* CXL spec rev3.0 8.2.4.16.1 */
> -static void header_log_copy(struct cxl_dev_state *cxlds, u32 *log)
> +static void header_log_copy(void __iomem *ras_base, u32 *log)
>   {
>   	void __iomem *addr;
>   	u32 *log_addr;
>   	int i, log_u32_size = CXL_HEADERLOG_SIZE / sizeof(u32);
>   
> -	addr = cxlds->regs.ras + CXL_RAS_HEADER_LOG_OFFSET;
> +	addr = ras_base + CXL_RAS_HEADER_LOG_OFFSET;
>   	log_addr = log;
>   
>   	for (i = 0; i < log_u32_size; i++) {
> @@ -704,17 +708,18 @@ static void header_log_copy(struct cxl_dev_state *cxlds, u32 *log)
>    * Log the state of the RAS status registers and prepare them to log the
>    * next error status. Return 1 if reset needed.
>    */
> -static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)
> +static bool __cxl_handle_ras(struct cxl_dev_state *cxlds,
> +				  void __iomem *ras_base)
>   {
>   	u32 hl[CXL_HEADERLOG_SIZE_U32];
>   	void __iomem *addr;
>   	u32 status;
>   	u32 fe;
>   
> -	if (!cxlds->regs.ras)
> +	if (!ras_base)
>   		return false;
>   
> -	addr = cxlds->regs.ras + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET;
> +	addr = ras_base + CXL_RAS_UNCORRECTABLE_STATUS_OFFSET;
>   	status = readl(addr);
>   	if (!(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK))
>   		return false;
> @@ -722,7 +727,7 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)
>   	/* If multiple errors, log header points to first error from ctrl reg */
>   	if (hweight32(status) > 1) {
>   		void __iomem *rcc_addr =
> -			cxlds->regs.ras + CXL_RAS_CAP_CONTROL_OFFSET;
> +			ras_base + CXL_RAS_CAP_CONTROL_OFFSET;
>   
>   		fe = BIT(FIELD_GET(CXL_RAS_CAP_CONTROL_FE_MASK,
>   				   readl(rcc_addr)));
> @@ -730,13 +735,26 @@ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds)
>   		fe = status;
>   	}
>   
> -	header_log_copy(cxlds, hl);
> +	header_log_copy(ras_base, hl);
>   	trace_cxl_aer_uncorrectable_error(cxlds->cxlmd, status, fe, hl);
>   	writel(status & CXL_RAS_UNCORRECTABLE_STATUS_MASK, addr);
>   
>   	return true;
>   }
>   
> +static bool cxl_handle_endpoint_ras(struct cxl_dev_state *cxlds)
> +{
> +	return __cxl_handle_ras(cxlds, cxlds->regs.ras);
> +}
> +
> +void cxl_cor_error_detected(struct pci_dev *pdev)
> +{
> +	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
> +
> +	cxl_handle_endpoint_cor_ras(cxlds);
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_cor_error_detected, CXL);
> +
>   pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
>   				    pci_channel_state_t state)
>   {
> @@ -751,7 +769,7 @@ pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
>   	 * chance the situation is recoverable dump the status of the RAS
>   	 * capability registers and bounce the active state of the memdev.
>   	 */
> -	ue = cxl_report_and_clear(cxlds);
> +	ue = cxl_handle_endpoint_ras(cxlds);
>   
>   	switch (state) {
>   	case pci_channel_io_normal: