Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 12 Jun 2023 15:57:27 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
        <vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <dave.jiang@intel.com>, <Jonathan.Cameron@huawei.com>,
        <linux-cxl@vger.kernel.org>
CC:     <terry.bowman@amd.com>, <rrichter@amd.com>,
        <linux-kernel@vger.kernel.org>, <bhelgaas@google.com>
Subject: RE: [PATCH v5 26/26] PCI/AER: Unmask RCEC internal errors to enable
 RCH downstream port error handling
Message-ID: <6487a2d7e89de_1433ac294e6@dwillia2-xfh.jf.intel.com.notmuch>
References: <20230607221651.2454764-1-terry.bowman@amd.com>
 <20230607221651.2454764-27-terry.bowman@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230607221651.2454764-27-terry.bowman@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?viuhRkZMZPo7xfPVHSMqfSIwvkosp7Zz5m7svgjQYxTZumo3e3C4U+H4OoX9?=
 =?us-ascii?Q?Ioh/RLAioqPbYoK1UotO8QPNO4PDa1um2Zz6REU9QwecYvSop0y8LmD65Ay7?=
 =?us-ascii?Q?oI+fIL3IcLAcJYepumASwlCxPebxRWrmuuOnTbErnGlekp6p7ucIllOA7UFk?=
 =?us-ascii?Q?gue4IDgwSdX3upYIYsGeoxLQukCodPDlM4sjQcI1njRZNlzW5QFwRqHR+zJ3?=
 =?us-ascii?Q?Lz/9HWT+1wz5nOLopEMtdHPAzVMOwv61PyS9aLHDCkAyPNSvxfP2NVMHAKew?=
 =?us-ascii?Q?G7xLRdpHA2AudSPMvHc/bfIPEXuTW7wDbGVS81Fnd0WfnukGrzWu3IbPqJOH?=
 =?us-ascii?Q?zhqaL2M/HrGSUaCBDuHOPGB+gTA786Ka6xtSNd7ENDnyLfcKOOxlRcRnTkQU?=
 =?us-ascii?Q?gFh2bIagslr4Hfec0IrdFCHy2Q5GaVFYtQu47W/O+Yqfi7sUM03+Vm9t5xEG?=
 =?us-ascii?Q?V71BI6sIWYVQ4kiq4kh2EXRC8+zUGQP4MQrN9myEcvpLRXk4yY30MN5/eSD3?=
 =?us-ascii?Q?8UnuL+T8Oyf4qLiUbqndY5VHCJBtGtDt3T8XfN0rA6zW0eAL0bwFi51w37Nu?=
 =?us-ascii?Q?fB2Nu604DgRo/nfYaLHLtpFGyN2MX4M1aAHPbkl9D0taOY7jFpRhtiyvx6NK?=
 =?us-ascii?Q?+OA3t/vm/w0mfgLfeKysCECmnqHpEQmmTYeims56UlsgxpXXxEOu1YUEPsiu?=
 =?us-ascii?Q?j6CS1kLp7X541hQ4mAijnF/MEz+1ys66uvJ+TWPX0jZrsDLXrovToeqRAzcq?=
 =?us-ascii?Q?31lY6zCoeyLZtUN85B55ao9hc9fcsFDdCOslrtUIv2TZV8tFwi9Qj9MIbajk?=
 =?us-ascii?Q?gW3l7U3pnGdyheD23glZv05B/fWWtQnnL/ltr2IBfGRIOv6TTO7LDR65EtKp?=
 =?us-ascii?Q?nOhlAzwYoT368BFXyp3KK1LlYOvXK/fPOD8PIgEFCuqqLOsDWmH0bvn/9Yih?=
 =?us-ascii?Q?UR/LkBs+LSG2iEpWswlPWaC4vF5z4iqT8mJknBOc8dboXm5PN+5WoZB+hJyw?=
 =?us-ascii?Q?jB369eIWV0E+jyZ6CwRPeIJL/QXjrGwbT6sRxYCC+myNV5BbC9++duhVXYea?=
 =?us-ascii?Q?FcA3cp7hzP0DkLj79Z4ZGoBqhja5Neex4mZVERzUFlReEJ9n6ttO7aO6wguy?=
 =?us-ascii?Q?WZP8U2kXveTT/t1ZvTwnybcB9rPueanRgNb3mQS85sjgfgTEzGogjhiX2H7i?=
 =?us-ascii?Q?rK26K872MkSDe1tby5jopcUS6jaZsTTusRY/JzBakwvNIiqLpBqFdYTCpxqT?=
 =?us-ascii?Q?kBvVKbL1PXnW2Mn2OBcnR2ROJeaqXDHo2fWvSHwtqjg+yiPeDF/DUgOgdAqo?=
 =?us-ascii?Q?I96vEwxP+1FiSpUnK8HwzfdKNHMAocL9Wie4+RYLkjqx6vpV/K/osY4asJ21?=
 =?us-ascii?Q?yk8TT6KkrF2TcIB88yqFbcnVd6vUuXM3EuCnoOUuycxgPz+adrdDufouX9JW?=
 =?us-ascii?Q?OOPi4I6jRKnLO5mfHWtCoEf4wfA08pTC6g+qpix504eGthN1kQ3wQHOuLFeo?=
 =?us-ascii?Q?ozYQG9NDHxENQSE7UEkxGN7cOYjtBEWTVLxAhq8qtl/aAwzsx733bvWtv4GL?=
 =?us-ascii?Q?IQ/WUvLW7sog9xnZMzZtazqemIo1gdmz78hmk94RqeHDkdlpIw9btvU3GFAv?=
 =?us-ascii?Q?JQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e5a51b4c-ff2f-4101-f3ae-08db6b986695
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jun 2023 22:57:30.5136
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: m8Pj4vs5pNHGyxbzx4Y35LQi68ufFF8/zggtcGSsq1XjeaIGemzL1fiwsZn8RpB6VBLLxeuMooIJc5x3LCf8Qm1kO4NP9GVhJrc73Qx85H4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR11MB7487
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Terry Bowman wrote:
> From: Robert Richter <rrichter@amd.com>
> 
> AER corrected and uncorrectable internal errors (CIE/UIE) are masked
> in their corresponding mask registers per default once in power-up
> state. [1][2] Enable internal errors for RCECs to receive CXL
> downstream port errors of Restricted CXL Hosts (RCHs).
> 
> [1] CXL 3.0 Spec, 12.2.1.1 - RCH Downstream Port Detected Errors
> [2] PCIe Base Spec r6.0, 7.8.4.3 Uncorrectable Error Mask Register,
>     7.8.4.6 Correctable Error Mask Register
> 
> Co-developed-by: Terry Bowman <terry.bowman@amd.com>
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> ---
>  drivers/pci/pcie/aer.c | 57 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 57 insertions(+)
> 
> diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c
> index c354ca5e8f2b..4f9203e27c62 100644
> --- a/drivers/pci/pcie/aer.c
> +++ b/drivers/pci/pcie/aer.c
> @@ -948,6 +948,30 @@ static bool find_source_device(struct pci_dev *parent,
>  
>  #ifdef CONFIG_PCIEAER_CXL
>  
> +/**
> + * pci_aer_unmask_internal_errors - unmask internal errors
> + * @dev: pointer to the pcie_dev data structure
> + *
> + * Unmasks internal errors in the Uncorrectable and Correctable Error
> + * Mask registers.
> + *
> + * Note: AER must be enabled and supported by the device which must be
> + * checked in advance, e.g. with pcie_aer_is_native().
> + */
> +static void pci_aer_unmask_internal_errors(struct pci_dev *dev)
> +{
> +	int aer = dev->aer_cap;
> +	u32 mask;
> +
> +	pci_read_config_dword(dev, aer + PCI_ERR_UNCOR_MASK, &mask);
> +	mask &= ~PCI_ERR_UNC_INTN;
> +	pci_write_config_dword(dev, aer + PCI_ERR_UNCOR_MASK, mask);
> +
> +	pci_read_config_dword(dev, aer + PCI_ERR_COR_MASK, &mask);
> +	mask &= ~PCI_ERR_COR_INTERNAL;
> +	pci_write_config_dword(dev, aer + PCI_ERR_COR_MASK, mask);
> +}
> +
>  static bool is_cxl_mem_dev(struct pci_dev *dev)
>  {
>  	/*
> @@ -1027,7 +1051,39 @@ static void cxl_rch_handle_error(struct pci_dev *dev, struct aer_err_info *info)
>  		pcie_walk_rcec(dev, cxl_rch_handle_error_iter, info);
>  }
>  
> +static int handles_cxl_error_iter(struct pci_dev *dev, void *data)
> +{
> +	int *handles_cxl = data;
> +
> +	if (!*handles_cxl)
> +		*handles_cxl = is_cxl_mem_dev(dev) && cxl_error_is_native(dev);
> +
> +	/* Non-zero terminates iteration */
> +	return *handles_cxl;
> +}
> +
> +static bool handles_cxl_errors(struct pci_dev *rcec)
> +{
> +	int handles_cxl = 0;
> +
> +	if (pci_pcie_type(rcec) == PCI_EXP_TYPE_RC_EC &&
> +	    pcie_aer_is_native(rcec))
> +		pcie_walk_rcec(rcec, handles_cxl_error_iter, &handles_cxl);
> +
> +	return !!handles_cxl;
> +}
> +
> +static void cxl_rch_enable_rcec(struct pci_dev *rcec)
> +{
> +	if (!handles_cxl_errors(rcec))
> +		return;
> +
> +	pci_aer_unmask_internal_errors(rcec);
> +	pci_info(rcec, "CXL: Internal errors unmasked");
> +}
> +
>  #else
> +static inline void cxl_rch_enable_rcec(struct pci_dev *dev) { }
>  static inline void cxl_rch_handle_error(struct pci_dev *dev,
>  					struct aer_err_info *info) { }
>  #endif
> @@ -1428,6 +1484,7 @@ static int aer_probe(struct pcie_device *dev)
>  		return status;
>  	}
>  
> +	cxl_rch_enable_rcec(port);

Similar to the last patch, I wonder if it is sufficient to call this
cxl_enable_rcec() in anticipation of VH support and the fact that this
only depends on RCiEPs not necessarily anything RCH specific like RCRB
shenanigans.

Shouldn't there also be a corresponding cxl_disable_rcec() in
aer_remove()?