Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
Date:   Tue, 12 Dec 2023 09:24:21 -0800
From:   Dan Williams <dan.j.williams@intel.com>
To:     Ira Weiny <ira.weiny@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Jonathan Cameron <jonathan.cameron@huawei.com>,
        Smita Koralahalli <Smita.KoralahalliChannabasappa@amd.com>,
        Shiju Jose <shiju.jose@huawei.com>
CC:     Yazen Ghannam <yazen.ghannam@amd.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Dave Jiang <dave.jiang@intel.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Ard Biesheuvel" <ardb@kernel.org>, <linux-efi@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
        Ira Weiny <ira.weiny@intel.com>
Subject: Re: [PATCH v2 7/7] cxl/memdev: Register for and process CPER events
Message-ID: <657897453dda8_269bd29492@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20231211-cxl-cper-v2-0-c116900ba658@intel.com>
 <20231211-cxl-cper-v2-7-c116900ba658@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20231211-cxl-cper-v2-7-c116900ba658@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?AoR2U4u2DMQAz1zATLNkgZ+7UibuNK+2E8uqzaSwCXQd7QArvxHxKt0ZuafG?=
 =?us-ascii?Q?Yp14o+iIZ0I/Wc1geNMKMyV+y//zosx7facSKhhN+Q4kr6Z80sl6wxKvK8d9?=
 =?us-ascii?Q?VNJcWYhawfdEE9QxPTAoQlgCfpeg1r1jmJ8u7GTd4JnhKgsSO3JnQY0WXUIb?=
 =?us-ascii?Q?OqWkciphG3uzMF5FIAnyjoQ6qBY0AhT9wU6dVXu/VVhvFXz3rPgCvlayMRrf?=
 =?us-ascii?Q?8HuMAcrmA//4EwDWENlFqLE/MMNBqXO3TqoAw6pfWQOoZ17WanhT6spOyPNa?=
 =?us-ascii?Q?/JFiXbJTHTrNii4115T0svnxcH4J/eCmCGj+vDDglNlSTQ9nJwMykxTTTAjo?=
 =?us-ascii?Q?nhmjt1X6/nL2NZ8GEVGJy30pL2XskuP1mrG2WJFqyGA2wt1Yt64nJ5CaN+Va?=
 =?us-ascii?Q?Q3rUOWLJwtWgZcaHtoijXpp7tOfxPcz/BneQdhQPgnBLsB0rIRYfjcMXKnIs?=
 =?us-ascii?Q?JvXsa2k5eMDHZ98gWPo8stOkb3JtJSVuLTrcVtAibAR9tAiKSP3Gqa0hqgVT?=
 =?us-ascii?Q?D678DvtKtajidW2hu4XLJiakLegaMMF2xWaVM2Na/KRc9pFFz5/AmRXmN9gE?=
 =?us-ascii?Q?ob4FEyX8mfQZA9YBneI8GbIer8h06qIdSkGF/tpLfw/zdVQ7OARrL3PvtWkV?=
 =?us-ascii?Q?8Kl9sbvfUP2sBTUPLzPlorXCdLTRrsg/09iXcI2fRefXLGwnvkpds8SRMhTs?=
 =?us-ascii?Q?rcbIErS2GKOkirfXAQVuGS/carhERpbbaR85+LQHBINSMKaiSbzAG2BThDDu?=
 =?us-ascii?Q?kiSjCrSQToMC+Ody4ViXj0VSYMqmt2nSYM97ajZ7yCINTeAJiRiiIfSIgSm+?=
 =?us-ascii?Q?WLsNgSetn4cdsyrsSeXymetEKnfve6K/2wI4OyHCYd1VjmISfZZwtxsLNi7F?=
 =?us-ascii?Q?bKJXU6L9U4bo1ty0qq+MkW/dKgTvPqNqLHDC7+MVPNavc7MaOOH2fCyqi50/?=
 =?us-ascii?Q?SoAFyCAr8S4j+rqyLjQzMrrHApWrqjJU5AT54NtlBrM3wXzkUSGXQxStQs8f?=
 =?us-ascii?Q?lfGYFH5gpGFJ5spcDc/V090JKxwzpqv0fc/EQjQhpIyn5rxIFljhDx7atAhs?=
 =?us-ascii?Q?zGjRvjSvfgoVfJWANL0AR4tf4gOKHevAWmtgRipG65Mr7X/FSJBm/WFsPK4r?=
 =?us-ascii?Q?QP4jIDEvfxs3lDy2RSSMv7w9MWWN+AnussEpdQhv/98RIHzlt4CsuXjZW0tp?=
 =?us-ascii?Q?yHm1pujqDBL+NolzkP1sNiq53zE0Q0+tLPR/9tUjvCAdrRC3uGuvdrbGmeZB?=
 =?us-ascii?Q?r1CaQAdX3H+olfN/kLZkTb7bE8XDrzS/wb1G2/83cmiGjmqcNqTzIzu9q0eh?=
 =?us-ascii?Q?oYqI4yZFLn2l98Pw9nMejxGqjP3oEFbFN4JrlkTaqdwYiW8HaVA6C4l89TY3?=
 =?us-ascii?Q?aHWeEvWrSJZoiMRp3JdrwiGrZiFmPQMCHVJ0NA9ShmworEHN0/tQ3KQjVzmB?=
 =?us-ascii?Q?3W4P1Dr6X2uKuqwi1fbzLVRxcNeGQZlrBj2FKnjrc619n8sYhAo+/quzcaru?=
 =?us-ascii?Q?gxzDEoygICmQ0qX8awx2R3N4GsVJ0QgO7vGKFMdYZDM94PRKDcNQhIQ+LoqQ?=
 =?us-ascii?Q?HN0+Y2UckkRSgCc5LYOZq5tWxdshm8Jzlju18qG4yZBY74eD3178vdygackz?=
 =?us-ascii?Q?Ww=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f1e193a1-5b3f-4d06-62a1-08dbfb3732af
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2023 17:24:29.6536
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Zpmh6IC/RvSlVCnYGo649EYkoO/C4ZokSXR1d/f8QPzMyowO23FnV7Uyg5tDRMbVNsxlTcY8Tj5qyAD/T3254bTBPPiyTyDTM/+OFgGYPIc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5220
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 12 Dec 2023 09:24:51 -0800 (PST)

Ira Weiny wrote:
> If the firmware has configured CXL event support to be firmware first
> the OS can process those events through CPER records.  The CXL layer has
> unique DPA to HPA knowledge and standard event trace parsing in place.
> 
> CPER records contain Bus, Device, Function information which can be used
> to identify the PCI device which is sending the event.
> 
> Change pci driver registration to include registration for a CXL CPER
> callback to process the events through the trace subsystem.
> 
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> 
> ---
> Changes from v1:
> [djbw: use single registration function]
> ---
>  drivers/cxl/core/mbox.c | 31 ++++++++++++++++++++++++-----
>  drivers/cxl/cxlmem.h    |  6 ++++++
>  drivers/cxl/pci.c       | 52 ++++++++++++++++++++++++++++++++++++++++++++++++-
>  3 files changed, 83 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index a67161f8764a..da262bbc3519 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -854,9 +854,30 @@ static const uuid_t dram_event_uuid = CXL_EVENT_DRAM_UUID;
>   */
>  static const uuid_t mem_mod_event_uuid = CXL_EVENT_MEM_MODULE_UUID;
>  
> -static void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
> -				   enum cxl_event_log_type type,
> -				   struct cxl_event_record_raw *record)
> +void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
> +			    enum cxl_event_log_type type,
> +			    enum cxl_event_type event_type,
> +			    union cxl_event *event)
> +{
> +	switch (event_type) {
> +	case CXL_CPER_EVENT_GEN_MEDIA:
> +		trace_cxl_general_media(cxlmd, type, &gen_media_event_uuid,
> +					&event->gen_media);
> +		break;
> +	case CXL_CPER_EVENT_DRAM:
> +		trace_cxl_dram(cxlmd, type, &dram_event_uuid, &event->dram);
> +		break;
> +	case CXL_CPER_EVENT_MEM_MODULE:
> +		trace_cxl_memory_module(cxlmd, type, &mem_mod_event_uuid,
> +					&event->mem_module);
> +		break;
> +	}
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_event_trace_record, CXL);
> +
> +static void __cxl_event_trace_record(const struct cxl_memdev *cxlmd,
> +				     enum cxl_event_log_type type,
> +				     struct cxl_event_record_raw *record)
>  {
>  	union cxl_event *evt = &record->event;
>  	uuid_t *id = &record->id;
> @@ -979,8 +1000,8 @@ static void cxl_mem_get_records_log(struct cxl_memdev_state *mds,
>  			break;
>  
>  		for (i = 0; i < nr_rec; i++)
> -			cxl_event_trace_record(cxlmd, type,
> -					       &payload->records[i]);
> +			__cxl_event_trace_record(cxlmd, type,
> +						 &payload->records[i]);
>  
>  		if (payload->flags & CXL_GET_EVENT_FLAG_OVERFLOW)
>  			trace_cxl_overflow(cxlmd, type, payload);
> diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> index e5d770e26e02..7a891b4641cc 100644
> --- a/drivers/cxl/cxlmem.h
> +++ b/drivers/cxl/cxlmem.h
> @@ -481,6 +481,8 @@ struct cxl_memdev_state {
>  	struct cxl_security_state security;
>  	struct cxl_fw_state fw;
>  
> +	struct notifier_block cxl_cper_nb;
> +
>  	struct rcuwait mbox_wait;
>  	int (*mbox_send)(struct cxl_memdev_state *mds,
>  			 struct cxl_mbox_cmd *cmd);
> @@ -802,6 +804,10 @@ void set_exclusive_cxl_commands(struct cxl_memdev_state *mds,
>  void clear_exclusive_cxl_commands(struct cxl_memdev_state *mds,
>  				  unsigned long *cmds);
>  void cxl_mem_get_event_records(struct cxl_memdev_state *mds, u32 status);
> +void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
> +			    enum cxl_event_log_type type,
> +			    enum cxl_event_type event_type,
> +			    union cxl_event *event);
>  int cxl_set_timestamp(struct cxl_memdev_state *mds);
>  int cxl_poison_state_init(struct cxl_memdev_state *mds);
>  int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len,
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 0155fb66b580..30a98399d013 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -1,5 +1,6 @@
>  // SPDX-License-Identifier: GPL-2.0-only
>  /* Copyright(c) 2020 Intel Corporation. All rights reserved. */
> +#include <asm-generic/unaligned.h>
>  #include <linux/io-64-nonatomic-lo-hi.h>
>  #include <linux/moduleparam.h>
>  #include <linux/module.h>
> @@ -969,6 +970,55 @@ static struct pci_driver cxl_pci_driver = {
>  	},
>  };
>  
> +#define CXL_EVENT_HDR_FLAGS_REC_SEVERITY GENMASK(1, 0)
> +static void cxl_cper_event_call(struct cxl_cper_event_data *ev_data)
> +{
> +	struct cper_cxl_event_devid *device_id = &ev_data->rec->hdr.device_id;
> +	struct cxl_dev_state *cxlds = NULL;
> +	enum cxl_event_log_type log_type;
> +	struct pci_dev *pdev;
> +	unsigned int devfn;
> +	u32 hdr_flags;
> +
> +	devfn = PCI_DEVFN(device_id->device_num, device_id->func_num);
> +	pdev = pci_get_domain_bus_and_slot(device_id->segment_num,
> +					   device_id->bus_num, devfn);

What if pci_get_domain_bus_and_slot() returned NULL?

> +
> +	guard(mutex)(&pdev->dev.mutex);

Lets not open code this since device_lock() is so prevalent it deserves
its own guard() type:

DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T))

> +	if (pdev->driver == &cxl_pci_driver)
> +		cxlds = pci_get_drvdata(pdev);
> +	if (!cxlds)
> +		goto out;

Lets not mix usage of cleanup.h helpers with usage of goto. The helpers
are there to eliminate goto errors. Just add a new helper:

DEFINE_FREE(pci_dev_put, struct pci_dev *, if (_T) pci_dev_put(_T))

...and declare @pdev as:

struct pci_dev *pdev __free(pci_dev_put) = NULL;

> +
> +	/* Fabricate a log type */
> +	hdr_flags = get_unaligned_le24(ev_data->rec->event.generic.hdr.flags);
> +	log_type = FIELD_GET(CXL_EVENT_HDR_FLAGS_REC_SEVERITY, hdr_flags);
> +
> +	cxl_event_trace_record(cxlds->cxlmd, log_type, ev_data->event_type,
> +			       &ev_data->rec->event);
> +out:
> +	pci_dev_put(pdev);
> +}
> +
> +static int __init cxl_pci_driver_init(void)
> +{
> +	int rc;
> +
> +	rc = pci_register_driver(&cxl_pci_driver);
> +	if (rc)
> +		return rc;
> +
> +	cxl_cper_register_notifier(cxl_cper_event_call);
> +	return 0;
> +}
> +
> +static void __exit cxl_pci_driver_exit(void)
> +{
> +	cxl_cper_unregister_notifier();
> +	pci_unregister_driver(&cxl_pci_driver);
> +}
> +
> +module_init(cxl_pci_driver_init);
> +module_exit(cxl_pci_driver_exit);
>  MODULE_LICENSE("GPL v2");
> -module_pci_driver(cxl_pci_driver);
>  MODULE_IMPORT_NS(CXL);
> 
> -- 
> 2.43.0
>