Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <60626151-d838-c335-47e9-246bdbe02dec@intel.com>
Date:   Fri, 23 Jun 2023 15:18:51 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Betterbird/102.11.1
Subject: Re: [PATCH v7 26/27] PCI/AER: Unmask RCEC internal errors to enable
 RCH downstream port error handling
To:     Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
        <vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <Jonathan.Cameron@huawei.com>, <linux-cxl@vger.kernel.org>
CC:     <rrichter@amd.com>, <linux-kernel@vger.kernel.org>,
        <bhelgaas@google.com>
References: <20230622205523.85375-1-terry.bowman@amd.com>
 <20230622205523.85375-27-terry.bowman@amd.com>
Content-Language: en-US
From:   Dave Jiang <dave.jiang@intel.com>
In-Reply-To: <20230622205523.85375-27-terry.bowman@amd.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YWdEczJzODg5QVk0dmRpRTB5aVN0bExMaU9xb0IzY24zb1RrM1NES2ppVGZP?=
 =?utf-8?B?QTZ3MjRiUGlGRWNpOForeE1ySFAyRFB5TGFNSi83bGdKcHdLZTh5UXdOTi9G?=
 =?utf-8?B?MGxCWmJYdy9qZXNLOEx1N1pybVFjUGp5MWExZFdkVkNqaFU0STBxaGZvTFh5?=
 =?utf-8?B?a0M1VWJMTWNjZFg1bzJDRnZ5V0hyTS93cDNNQVFOejZnd1E3UmdmcmpZZWRE?=
 =?utf-8?B?YWpBS3NFNmpVN1FkaWdLMExWTU9kYTlMcHdNbStDQXJnZ2NzdklkM1VqZWJF?=
 =?utf-8?B?VVJPRG9SNWFBblNxT053Q05RS0pndnVPa1ZxbWVlcDFtRkErUUNiSlhncGg0?=
 =?utf-8?B?MUpJVGFrc2dybkYveEhoK3pnd3MxR2FGZVNMK2s2VHp1SVNuYUdtWkc4Z1pK?=
 =?utf-8?B?QmlDUy9ycGFUVmRBSWxiWHRJQ3J6SmFxRTZETHkvRlg2ak5hQm8rMFM0eCt2?=
 =?utf-8?B?dkVNeGlnZTU0eFFGZTlwb25YcFFITUk1c2pTNkhWZGdZT250a01zbDlSellH?=
 =?utf-8?B?YytObkVQSkRXSmFtdCtleDFOWDdGMHZiTVJrVzdaQlpJWjZSeFBVYllpZ1No?=
 =?utf-8?B?RkFLZ3BGYURMaCt5MER6MGNnRWxDRTBxa2Z2dWh5N3VObnJpWE1NRjM5NTJp?=
 =?utf-8?B?UXF0WHFtS3EyNlNIejllbmJZLytKNU5TekFWajBwL09iRTlrYTBQd2pnWWhS?=
 =?utf-8?B?MUZrNnRuL1V5YTdkMFdvNWVEV3RHOWpMVDQ5NktRVzVRU25kZ0RGTUM0K2tS?=
 =?utf-8?B?VjJFNkpZcnhPWmF3eWpNdm9wZEtBeTFDSkUxNnhicGtDelRkYzgvSU0ya3Ba?=
 =?utf-8?B?U01wa2t3aXkzYnlXTUM1dnhPQzF0SklMQ1d0Y2dTQUdXTDIzQlhNTXVYK08z?=
 =?utf-8?B?V1ZucWFSVTR0OERVSFlhYjd4RHA4ZDA2VkhubFcxSlZFc3JjTmZRbyt3cTJH?=
 =?utf-8?B?Y0xrZEF3dGJoQmI4eGxUNEZETVBmV243M1h2R2Z1TFo0YkJuY09YcEh4UnhP?=
 =?utf-8?B?Q1EzNlNTS3Z6K04vR2JaQzRYekZTTmlSMkhta0FXSVYyQlh1M2Fhd0VtL2JK?=
 =?utf-8?B?NWZFa1JiM3JydmZwbzJkMnJ3cFNxcFZSbTQvR2Q2TEo5U1BCa1kyc01FWSt1?=
 =?utf-8?B?M3htakx5WEkxY2NzajdGMTVDNGE3QzFJekxYMUJvL21UK3hidW5NRE9rUVdm?=
 =?utf-8?B?dzg1SXJyckpvdTJWTkx2eDU3dEx1aThMUjIyTUJlZnNJVjVGaG1ISGdCaEFG?=
 =?utf-8?B?dmV5bDZxYlJsdllHMFI3blh4OGxrRjJhWHpoYzY0Z3V6RUZvcWxtL3kwVitW?=
 =?utf-8?B?NFUyaVNpMDIrY2xMa0VDdms3NUVqTlg0TTRzWVBnUEJUYzNtV2Y5NzJqZ094?=
 =?utf-8?B?YW9GcWlUWmJidWZHSjZhN051WTE3Y3I3dGR1V2E4ek55T3hWZFAvVlNwUlVs?=
 =?utf-8?B?WTkxTlZaM1JKeEhMMmJnVFI4K1FBcU5hdnhlT3d3ZXJxaXVyZ2ErYk0yT0ZS?=
 =?utf-8?B?TFJuMlFwZWNtaGhmd1N3cjg5U1VyUnBVSEczNUdmbXoxVEFxcGF4L2o4YVFy?=
 =?utf-8?B?RWZOZDNLV3lLdTZyYWNwdTBzNFE2Wk93MlAvWE12RWFHdzdTS1hKQ0xaNWEr?=
 =?utf-8?B?djVGcHJZZnpqbGE3WlZ4Q242WUgxSnlQSmpYeTJzWW9TZktEcTY4dnJnaDZ2?=
 =?utf-8?B?WVpHN0NmclhheGkwWWtVclpUanlGZWROcTZlQTN6T3o0M2k3dU11RFZ6VlFO?=
 =?utf-8?B?MzRkSEo2V3ZUSW0xR2ZHV3BGRXoxalpTMlkzblZKdjhQL1RuV09aWENpUkZy?=
 =?utf-8?B?aDlOL0dFUk4wamNEZStERERPSXo1Si9Uall0ckd0RGVORm5ZVUtna3hDZzJN?=
 =?utf-8?B?dEtMRkN1L2ZsQ2lGeHFkQk9heVZtRms3MEhvcG5TVzB5ZXVmenZIbUdBYUUw?=
 =?utf-8?B?OHArYktwVzJJVUhOMWZ3NWFpMUlKRHpYVjJ1elpEWncrT2Q4bnJkSmxEM1p6?=
 =?utf-8?B?aEZFK3dod1Y2b3BzbTNyMkRlczlGcUI2M0VEWlpyWGVBS2JmOVR4aUN1d0Zr?=
 =?utf-8?B?bHpOY0dqbUJ5eVJPNldwSGVlTHhhV2NCVlRHbzRlNWFGWWswMFN4NjBjT3di?=
 =?utf-8?Q?1PJ8is0ao3cnNtAC8FreCXvwJ?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5be109e0-88f3-48c3-793b-08db7437d580
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5984.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jun 2023 22:18:55.8456
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: w4C5CTUNk1NLDgNtjfhottgEJ418PVEP3BdNZRPvnc+8GRvr6V/9pXCYONbmWX7hHHREKZU5m6Lcjwx0og4LKQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR11MB5430
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/22/23 13:55, Terry Bowman wrote:
> From: Robert Richter <rrichter@amd.com>
> 
> AER corrected and uncorrectable internal errors (CIE/UIE) are masked
> in their corresponding mask registers per default once in power-up
> state. [1][2] Enable internal errors for RCECs to receive CXL
> downstream port errors of Restricted CXL Hosts (RCHs).
> 
> [1] CXL 3.0 Spec, 12.2.1.1 - RCH Downstream Port Detected Errors
> [2] PCIe Base Spec r6.0, 7.8.4.3 Uncorrectable Error Mask Register,
>      7.8.4.6 Correctable Error Mask Register
> 
> Co-developed-by: Terry Bowman <terry.bowman@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>

> ---
>   drivers/pci/pcie/aer.c | 57 ++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 57 insertions(+)
> 
> diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c
> index c354ca5e8f2b..916fbca95e53 100644
> --- a/drivers/pci/pcie/aer.c
> +++ b/drivers/pci/pcie/aer.c
> @@ -948,6 +948,30 @@ static bool find_source_device(struct pci_dev *parent,
>   
>   #ifdef CONFIG_PCIEAER_CXL
>   
> +/**
> + * pci_aer_unmask_internal_errors - unmask internal errors
> + * @dev: pointer to the pcie_dev data structure
> + *
> + * Unmasks internal errors in the Uncorrectable and Correctable Error
> + * Mask registers.
> + *
> + * Note: AER must be enabled and supported by the device which must be
> + * checked in advance, e.g. with pcie_aer_is_native().
> + */
> +static void pci_aer_unmask_internal_errors(struct pci_dev *dev)
> +{
> +	int aer = dev->aer_cap;
> +	u32 mask;
> +
> +	pci_read_config_dword(dev, aer + PCI_ERR_UNCOR_MASK, &mask);
> +	mask &= ~PCI_ERR_UNC_INTN;
> +	pci_write_config_dword(dev, aer + PCI_ERR_UNCOR_MASK, mask);
> +
> +	pci_read_config_dword(dev, aer + PCI_ERR_COR_MASK, &mask);
> +	mask &= ~PCI_ERR_COR_INTERNAL;
> +	pci_write_config_dword(dev, aer + PCI_ERR_COR_MASK, mask);
> +}
> +
>   static bool is_cxl_mem_dev(struct pci_dev *dev)
>   {
>   	/*
> @@ -1027,7 +1051,39 @@ static void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
>   		pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
>   }
>   
> +static int handles_cxl_error_iter(struct pci_dev *dev, void *data)
> +{
> +	bool *handles_cxl = data;
> +
> +	if (!*handles_cxl)
> +		*handles_cxl = is_cxl_mem_dev(dev) && cxl_error_is_native(dev);
> +
> +	/* Non-zero terminates iteration */
> +	return *handles_cxl;
> +}
> +
> +static bool handles_cxl_errors(struct pci_dev *rcec)
> +{
> +	bool handles_cxl = false;
> +
> +	if (pci_pcie_type(rcec) == PCI_EXP_TYPE_RC_EC &&
> +	    pcie_aer_is_native(rcec))
> +		pcie_walk_rcec(rcec, handles_cxl_error_iter, &handles_cxl);
> +
> +	return handles_cxl;
> +}
> +
> +static void cxl_rch_enable_rcec(struct pci_dev *rcec)
> +{
> +	if (!handles_cxl_errors(rcec))
> +		return;
> +
> +	pci_aer_unmask_internal_errors(rcec);
> +	pci_info(rcec, "CXL: Internal errors unmasked");
> +}
> +
>   #else
> +static inline void cxl_rch_enable_rcec(struct pci_dev *dev) { }
>   static inline void cxl_rch_handle_error(struct pci_dev *dev,
>   					struct aer_err_info *info) { }
>   #endif
> @@ -1428,6 +1484,7 @@ static int aer_probe(struct pcie_device *dev)
>   		return status;
>   	}
>   
> +	cxl_rch_enable_rcec(port);
>   	aer_enable_rootport(rpc);
>   	pci_info(port, "enabled with IRQ %d\n", dev->irq);
>   	return 0;