Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 17 Apr 2023 17:06:32 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Terry Bowman <terry.bowman@amd.com>, <alison.schofield@intel.com>,
        <vishal.l.verma@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <dave.jiang@intel.com>, <Jonathan.Cameron@huawei.com>,
        <linux-cxl@vger.kernel.org>
CC:     <terry.bowman@amd.com>, <rrichter@amd.com>,
        <linux-kernel@vger.kernel.org>, <bhelgaas@google.com>
Subject: RE: [PATCH v3 4/6] cxl/pci: Add RCH downstream port error logging
Message-ID: <643ddf0869622_1b662946@dwillia2-xfh.jf.intel.com.notmuch>
References: <20230411180302.2678736-1-terry.bowman@amd.com>
 <20230411180302.2678736-5-terry.bowman@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230411180302.2678736-5-terry.bowman@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?qOsIp6OWUAEifit7kk2MqtVT05Mw3ZmrDotZantwTy/Ol/D3o0/0s74vro2k?=
 =?us-ascii?Q?ujnAW702aLkSErprpGXaBeOSW0keoIULgo02xgKF77uwJoNtKUs1L7pyV4oT?=
 =?us-ascii?Q?e9PFjLxAgDfmeQds0bys4wDAZIrbnQG+GTVLqCyBnaoa71bJRUqtAWj9YoS1?=
 =?us-ascii?Q?BBYMMorKpHSHYFUbezJgLSfVVocfITL0wxhNnKkkps0hDII3fpfz5xiBteqX?=
 =?us-ascii?Q?o7BJtnr8yNGhgNyUIHoloHq0SGVmeyeVeCVU2lZelD5mSgkrbWQvU7sPXNwK?=
 =?us-ascii?Q?4jFZjfp3Ak5Ij0blC+f9kIlq0EDFdcIJuXvpN0y8tRkLjCKLh61VURpK40W9?=
 =?us-ascii?Q?Gub1uL+1UCDAuKKQSDZTixgKbwYSepXSnm3dYFd9VlPS3YTymQnBmn2ighkM?=
 =?us-ascii?Q?zyRa7BAQ2teArD2Kqrk+6eDu6CE/62kKUg7z9Y4iIUpyLk6eae6j3WcPBafh?=
 =?us-ascii?Q?r9XTklkgWv4dm4kpIAnzl/bmSqWlRBZLbNfuQ03kSxt0sBRlrZBzEEmZKFQQ?=
 =?us-ascii?Q?kgD/OxOJHm13ODvre9G7dldTqK1cDfZ47iurRQn9G3u0idm1mj1jB1GD/DPb?=
 =?us-ascii?Q?wsMERJwqRGekDqYwMgakahXy2MIO3dwu3sbFPSvCDcziG9BEifWRndUY1j7Q?=
 =?us-ascii?Q?V1LC0lPmz4Oddkdcibb64fmq0w/qJb/lA0pnahHnS5WjuaUR0nD4oMkatq8W?=
 =?us-ascii?Q?LSFcOY4+509tv5VGcoau124dm334IrrA4P0e+3G494fKDH5DKZQbEzZpMhJ+?=
 =?us-ascii?Q?wRPZEdE+zFdboNZByg/ttai2mF5z8uD9C78cASFduauGK2bajlY+iLA4HYPq?=
 =?us-ascii?Q?msgcP5oYiFJ+EaommjnzrHRTBqdqCjkR2pdteY/jskvuwYf0waapTYFSZTvw?=
 =?us-ascii?Q?qrzzp4uvo67FitU6iukeBgy20j7P0WBey5Sn5IMX4v3DsfON4kavhPS7xiDA?=
 =?us-ascii?Q?BJpgbff5JFQteXHITwHGcnfYT3djPHzN3vCLiRH+4ic6GJ+b5ZHOYYM6ZMji?=
 =?us-ascii?Q?uLQLkaqmXCiMJUHZghyb3GZazBcmX+MbP/NU0qahEB9CQ6xQ5c5gwSTJAUjh?=
 =?us-ascii?Q?m8X/IapZbO4pXDPz4LPqotcGQ4nxLRKtRMblb2jubYVa66iWu9GMkZvzWtWI?=
 =?us-ascii?Q?nhE5uPIJlkrbzuBKNwKzHiriG7f+eZ4N2vfLvHcflbQJFlRhAe/ExZoocChS?=
 =?us-ascii?Q?lePl+9sThLOjYN6ICwXPzAAQL42FwfMOCv3enjRTZx+W1ti126/eFj6FGw/U?=
 =?us-ascii?Q?z1crgil+G98KDi84l8C2UEKdiBcTTcYWw/9ycCODAOL98xNCaAWiObn5CkFo?=
 =?us-ascii?Q?jcP5g0WGLHxdhkNx7Af0HBoRCJrHaiNWxu1U0GvDYqX7cJbBWi8O+abQLMc1?=
 =?us-ascii?Q?U87gZM/z/EQhaXpzvjiX1nExw7TP8IqqO0AlNOHHsRVw7FWNYDz8sc1zAJy0?=
 =?us-ascii?Q?FcAPkHsXx8Dn/V3iwDn2+M99StMpGb1fSu41tPSDxhpcwz/K72hPOH2jXBDx?=
 =?us-ascii?Q?vX2bP/GLWffTatZWB2SCKp9kETTw3UFI6HGCFdYOtovk064cVqSkx9Qa2/zb?=
 =?us-ascii?Q?NY6YYM3kfMxZ6L9AdCuljiuxSyGpX+EL3tNQRObYWq4vJwE6qtm5/qZlSGHj?=
 =?us-ascii?Q?/g=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: cde3a836-8fd8-4fbb-e6af-08db3fa0c599
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Apr 2023 00:06:34.9024
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: If4a4IyjghfvPEGSDnUPUAKxZTNVBH0Nd7Hgv41goekKRagiK1/Xy7a09MuEiVU5umUjhOZuxr4EDH/FyQFppX8nIBfthasMOTvTdKNtbSQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7684
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Terry Bowman wrote:
> RCH downstream port error logging is missing in the current CXL driver. The
> missing AER and RAS error logging is needed for communicating driver error
> details to userspace. Update the driver to include PCIe AER and CXL RAS
> error logging.
> 
> Add RCH downstream port error handling into the existing RCiEP handler.
> The downstream port error handler is added to the RCiEP error handler
> because the downstream port is implemented in a RCRB, is not PCI
> enumerable, and as a result is not directly accessible to the PCI AER
> root port driver. The AER root port driver calls the RCiEP handler for
> handling RCD errors and RCH downstream port protocol errors.
> 
> Update mem.c to include RAS and AER setup. This includes AER and RAS
> capability discovery and mapping for later use in the error handler.
> 
> Disable RCH downstream port's root port cmd interrupts.[1]
> 
> Update existing RCiEP correctable and uncorrectable handlers to also call
> the RCH handler. The RCH handler will read the RCH AER registers, check for
> error severity, and if an error exists will log using an existing kernel
> AER trace routine. The RCH handler will also log downstream port RAS errors
> if they exist.

I think this patch wants a lead in refactoring to move the existing
probe of the CXL RAS capability into the cxl_port driver so that the RCH
path and the VH path can be unified for register mapping and error
handling invocation. I do not see a compelling rationale to have 2
separate ways to map the RAS capability. The timing of when
cxl_setup_ras() is called looks problematic relative to when the first
error handler callback might happen.

For example what happens when an error fires after cxl_pci has
registered its error handlers, but before the component registers have
been mapped out of the RCRB?

This implies the need for a callback for cxl_pci to notify the cxl_port
driver of CXL errors to handle relative to a PCI AER event.