Series status/background
========================
This is another RFC version of processing the CXL CPER records through
the CXL trace mechanisms as Dan mentioned in [1].
I moved forward with eliminating the GUID to UUID conversion I mentioned
in the original RFC thread[2]. Instead a new event type is used once
the GUID or UUID's is used to decode the event.
This remains compile tested with only.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Cover letter
============
CXL Component Events, as defined by EFI 2.10 Section N.2.14, wrap a
mostly CXL event payload in an EFI Common Platform Error Record (CPER)
record. If a device is configured for firmware first CXL event records
are not sent directly to the host.
The CXL sub-system uniquely has DPA to HPA translation information. It
also already properly decodes the event record format. Send the CXL
CPER records to the CXL sub-system for processing.
With CXL event logs the device interrupts the host with events. In the
EFI case events are wrapped with device information which needs to be
matched with memdev devices the CXL driver is tracking.
A number of alternatives were considered to match the memdev with the
CPER record. The most straight forward comparison is via serial number.
CPER records are identified with GUID's while CXL event logs contain
UUID's. The UUID was previously printed for all events. But the UUID
is redundant information which presents unnecessary complexity when
processing CPER data. Remove the UUIDs from known events.
Signed-off-by: Ira Weiny <[email protected]>
---
Changes in RFC v2:
- iweiny: remove uuid from existing known event traces
- iweiny: pass an enum for the event type.
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Ira Weiny (3):
cxl/trace: Remove uuid from event trace known events
firmware/efi: Process CXL Component Events
cxl/memdev: Register for and process CPER events
drivers/cxl/core/mbox.c | 45 +++++++++++++++++++++------
drivers/cxl/core/trace.h | 10 +++---
drivers/cxl/cxlmem.h | 7 +++++
drivers/cxl/pci.c | 69 ++++++++++++++++++++++++++++++++++++++++-
drivers/firmware/efi/cper.c | 16 ++++++++++
drivers/firmware/efi/cper_cxl.c | 40 ++++++++++++++++++++++++
drivers/firmware/efi/cper_cxl.h | 29 +++++++++++++++++
include/linux/efi.h | 59 +++++++++++++++++++++++++++++++++++
8 files changed, 259 insertions(+), 16 deletions(-)
---
base-commit: 1c8b86a3799f7e5be903c3f49fcdaee29fd385b5
change-id: 20230601-cxl-cper-26ffc839c6c6
Best regards,
--
Ira Weiny <[email protected]>
The uuid printed in the well known events is redundant. The uuid
defines what the event was.
Remove the uuid from the known events and only report it in the generic
event as it remains informative there.
Signed-off-by: Ira Weiny <[email protected]>
---
drivers/cxl/core/trace.h | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/cxl/core/trace.h b/drivers/cxl/core/trace.h
index a0b5819bc70b..79ed03637604 100644
--- a/drivers/cxl/core/trace.h
+++ b/drivers/cxl/core/trace.h
@@ -189,7 +189,6 @@ TRACE_EVENT(cxl_overflow,
__string(memdev, dev_name(&cxlmd->dev)) \
__string(host, dev_name(cxlmd->dev.parent)) \
__field(int, log) \
- __field_struct(uuid_t, hdr_uuid) \
__field(u64, serial) \
__field(u32, hdr_flags) \
__field(u16, hdr_handle) \
@@ -203,7 +202,6 @@ TRACE_EVENT(cxl_overflow,
__assign_str(host, dev_name((cxlmd)->dev.parent)); \
__entry->log = (l); \
__entry->serial = (cxlmd)->cxlds->serial; \
- memcpy(&__entry->hdr_uuid, &(hdr).id, sizeof(uuid_t)); \
__entry->hdr_length = (hdr).length; \
__entry->hdr_flags = get_unaligned_le24((hdr).flags); \
__entry->hdr_handle = le16_to_cpu((hdr).handle); \
@@ -212,12 +210,12 @@ TRACE_EVENT(cxl_overflow,
__entry->hdr_maint_op_class = (hdr).maint_op_class
#define CXL_EVT_TP_printk(fmt, ...) \
- TP_printk("memdev=%s host=%s serial=%lld log=%s : time=%llu uuid=%pUb " \
+ TP_printk("memdev=%s host=%s serial=%lld log=%s : time=%llu " \
"len=%d flags='%s' handle=%x related_handle=%x " \
"maint_op_class=%u : " fmt, \
__get_str(memdev), __get_str(host), __entry->serial, \
cxl_event_log_type_str(__entry->log), \
- __entry->hdr_timestamp, &__entry->hdr_uuid, __entry->hdr_length,\
+ __entry->hdr_timestamp, __entry->hdr_length, \
show_hdr_flags(__entry->hdr_flags), __entry->hdr_handle, \
__entry->hdr_related_handle, __entry->hdr_maint_op_class, \
##__VA_ARGS__)
@@ -231,15 +229,17 @@ TRACE_EVENT(cxl_generic_event,
TP_STRUCT__entry(
CXL_EVT_TP_entry
+ __field_struct(uuid_t, hdr_uuid)
__array(u8, data, CXL_EVENT_RECORD_DATA_LENGTH)
),
TP_fast_assign(
CXL_EVT_TP_fast_assign(cxlmd, log, rec->hdr);
+ memcpy(&__entry->hdr_uuid, &rec->hdr.id, sizeof(uuid_t));
memcpy(__entry->data, &rec->data, CXL_EVENT_RECORD_DATA_LENGTH);
),
- CXL_EVT_TP_printk("%s",
+ CXL_EVT_TP_printk("uuid=%pUb %s", &__entry->hdr_uuid,
__print_hex(__entry->data, CXL_EVENT_RECORD_DATA_LENGTH))
);
--
2.41.0
BIOS can configure memory devices as firmware first. This will send CXL
events to the firmware instead of the OS. The firmware can then send
these events to the OS via UEFI.
UEFI v2.10 section N.2.14 defines a Common Platform Error Record (CPER)
format for CXL Component Events. The format is mostly the same as the
CXL Common Event Record Format. The only difference is the UUID is
passed via the Section Type as a GUID and not included as part of the
record data.
Add EFI support to detect CXL CPER records and call a notifier chain
with the record data blobs.
Note that the format of a GUID and UUID are not the same. Therefore the
Section Type GUID defines are duplicated from the CXL code.
Signed-off-by: Ira Weiny <[email protected]>
---
Changes from RFC v1
[iweiny: use an enum for know record types and skip converting GUID to UUID]
[iweiny: commit to the UUID not being part of the event record data]
[iweiny: use defines for GUID definitions]
---
drivers/firmware/efi/cper.c | 16 +++++++++++
drivers/firmware/efi/cper_cxl.c | 40 ++++++++++++++++++++++++++++
drivers/firmware/efi/cper_cxl.h | 29 ++++++++++++++++++++
include/linux/efi.h | 59 +++++++++++++++++++++++++++++++++++++++++
4 files changed, 144 insertions(+)
diff --git a/drivers/firmware/efi/cper.c b/drivers/firmware/efi/cper.c
index 35c37f667781..d6415c94d584 100644
--- a/drivers/firmware/efi/cper.c
+++ b/drivers/firmware/efi/cper.c
@@ -607,6 +607,22 @@ cper_estatus_print_section(const char *pfx, struct acpi_hest_generic_data *gdata
cper_print_prot_err(newpfx, prot_err);
else
goto err_section_too_small;
+ } else if (guid_equal(sec_type, &CPER_SEC_CXL_GEN_MEDIA) ||
+ guid_equal(sec_type, &CPER_SEC_CXL_DRAM) ||
+ guid_equal(sec_type, &CPER_SEC_CXL_MEM_MODULE)) {
+ struct cper_cxl_event_rec *rec = acpi_hest_get_payload(gdata);
+
+ printk("%ssection type: CXL Event\n", newpfx);
+
+ if (rec->hdr.length <= sizeof(rec->hdr))
+ goto err_section_too_small;
+
+ if (rec->hdr.length > sizeof(*rec)) {
+ pr_err(FW_WARN "error section length is too big\n");
+ return;
+ }
+
+ cper_post_cxl_event(newpfx, sec_type, rec);
} else {
const void *err = acpi_hest_get_payload(gdata);
diff --git a/drivers/firmware/efi/cper_cxl.c b/drivers/firmware/efi/cper_cxl.c
index a55771b99a97..04234884898d 100644
--- a/drivers/firmware/efi/cper_cxl.c
+++ b/drivers/firmware/efi/cper_cxl.c
@@ -187,3 +187,43 @@ void cper_print_prot_err(const char *pfx, const struct cper_sec_prot_err *prot_e
sizeof(cxl_ras->header_log), 0);
}
}
+
+/* CXL CPER notifier chain */
+static BLOCKING_NOTIFIER_HEAD(cxl_cper_chain_head);
+
+void cper_post_cxl_event(const char *pfx, guid_t *sec_type,
+ struct cper_cxl_event_rec *rec)
+{
+ struct cxl_cper_notifier_data nd = {
+ .rec = rec,
+ };
+
+ if (!(rec->hdr.validation_bits & CPER_CXL_COMP_EVENT_LOG_VALID)) {
+ pr_err(FW_WARN "cxl event no Component Event Log present\n");
+ return;
+ }
+
+ if (guid_equal(sec_type, &CPER_SEC_CXL_GEN_MEDIA))
+ nd.cper_event = CXL_CPER_EVENT_GEN_MEDIA;
+ else if (guid_equal(sec_type, &CPER_SEC_CXL_DRAM))
+ nd.cper_event = CXL_CPER_EVENT_DRAM;
+ else if (guid_equal(sec_type, &CPER_SEC_CXL_MEM_MODULE))
+ nd.cper_event = CXL_CPER_EVENT_MEM_MODULE;
+
+ if (blocking_notifier_call_chain(&cxl_cper_chain_head, 0, (void *)&nd)
+ == NOTIFY_BAD)
+ pr_err(FW_WARN "cxl event notifier chain failed\n");
+}
+
+int register_cxl_cper_notifier(struct notifier_block *nb)
+{
+ return blocking_notifier_chain_register(&cxl_cper_chain_head, nb);
+}
+EXPORT_SYMBOL(register_cxl_cper_notifier);
+
+void unregister_cxl_cper_notifier(struct notifier_block *nb)
+{
+ blocking_notifier_chain_unregister(&cxl_cper_chain_head, nb);
+}
+EXPORT_SYMBOL(unregister_cxl_cper_notifier);
+
diff --git a/drivers/firmware/efi/cper_cxl.h b/drivers/firmware/efi/cper_cxl.h
index 86bfcf7909ec..ca26126cd9b8 100644
--- a/drivers/firmware/efi/cper_cxl.h
+++ b/drivers/firmware/efi/cper_cxl.h
@@ -10,11 +10,38 @@
#ifndef LINUX_CPER_CXL_H
#define LINUX_CPER_CXL_H
+#include <linux/efi.h>
+
/* CXL Protocol Error Section */
#define CPER_SEC_CXL_PROT_ERR \
GUID_INIT(0x80B9EFB4, 0x52B5, 0x4DE3, 0xA7, 0x77, 0x68, 0x78, \
0x4B, 0x77, 0x10, 0x48)
+/* CXL Event record UUIDs are used as the section type */
+/*
+ * General Media Event Record
+ * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43
+ */
+#define CPER_SEC_CXL_GEN_MEDIA \
+ GUID_INIT(0xfbcd0a77, 0xc260, 0x417f, \
+ 0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6)
+
+/*
+ * DRAM Event Record
+ * CXL rev 3.0 section 8.2.9.2.1.2; Table 8-44
+ */
+#define CPER_SEC_CXL_DRAM \
+ GUID_INIT(0x601dcbb3, 0x9c06, 0x4eab, \
+ 0xb8, 0xaf, 0x4e, 0x9b, 0xfb, 0x5c, 0x96, 0x24)
+
+/*
+ * Memory Module Event Record
+ * CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45
+ */
+#define CPER_SEC_CXL_MEM_MODULE \
+ GUID_INIT(0xfe927475, 0xdd59, 0x4339, \
+ 0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74)
+
#pragma pack(1)
/* Compute Express Link Protocol Error Section, UEFI v2.10 sec N.2.13 */
@@ -62,5 +89,7 @@ struct cper_sec_prot_err {
#pragma pack()
void cper_print_prot_err(const char *pfx, const struct cper_sec_prot_err *prot_err);
+void cper_post_cxl_event(const char *pfx, guid_t *sec_type,
+ struct cper_cxl_event_rec *rec);
#endif //__CPER_CXL_
diff --git a/include/linux/efi.h b/include/linux/efi.h
index 80b21d1c6eaf..b5b8b46c8deb 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -1355,4 +1355,63 @@ bool efi_config_table_is_usable(const efi_guid_t *guid, unsigned long table)
umode_t efi_attr_is_visible(struct kobject *kobj, struct attribute *attr, int n);
+/*
+ * Event log size adjusted for CPER
+ *
+ * Base table from CXL r3.0 Table 8-42: (30h + 50h)
+ * For lack of UUID: - 10h
+ *
+ * (30h + 50h) - 10h = 70h
+ */
+#define CPER_CXL_COMP_EVENT_LOG_SIZE 0x70
+#define CPER_CXL_DEVICE_ID_VALID BIT(0)
+#define CPER_CXL_DEVICE_SN_VALID BIT(1)
+#define CPER_CXL_COMP_EVENT_LOG_VALID BIT(2)
+struct cper_cxl_event_rec {
+ struct {
+ u32 length;
+ u64 validation_bits;
+ struct {
+ u16 vendor_id;
+ u16 device_id;
+ u8 func_num;
+ u8 device_num;
+ u8 bus_num;
+ u16 segment_num;
+ u16 slot_num; /* bits 2:0 reserved */
+ u8 reserved;
+ } device_id;
+ struct {
+ u32 lower_dw;
+ u32 upper_dw;
+ } dev_serial_num;
+ } hdr;
+
+ u8 comp_event_log[CPER_CXL_COMP_EVENT_LOG_SIZE];
+};
+#define CPER_CXL_REC_LEN(rec) (rec->hdr.length - sizeof(rec->hdr))
+
+enum cxl_cper_event {
+ CXL_CPER_EVENT_GEN_MEDIA,
+ CXL_CPER_EVENT_DRAM,
+ CXL_CPER_EVENT_MEM_MODULE,
+};
+
+struct cxl_cper_notifier_data {
+ enum cxl_cper_event cper_event;
+ struct cper_cxl_event_rec *rec;
+};
+
+#ifdef CONFIG_EFI
+int register_cxl_cper_notifier(struct notifier_block *nb);
+void unregister_cxl_cper_notifier(struct notifier_block *nb);
+#else
+static inline int register_cxl_cper_notifier(struct notifier_block *nb)
+{
+ return 0;
+}
+
+static inline void unregister_cxl_cper_notifier(struct notifier_block *nb) { }
+#endif
+
#endif /* _LINUX_EFI_H */
--
2.41.0
If the firmware has configured CXL event support to be firmware first
the OS can process those events through CPER records. Matching memory
devices to the CPER records can be done via the serial number which is
part of the CPER record header.
Detect firmware first, register a notifier callback for each memdev, and
trace events when they match a device registered.
Signed-off-by: Ira Weiny <[email protected]>
---
Changes from RFC v1:
[iweiny: adjust to cper_event enum instead of converting guids]
---
drivers/cxl/core/mbox.c | 45 +++++++++++++++++++++++++-------
drivers/cxl/cxlmem.h | 7 +++++
drivers/cxl/pci.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++-
3 files changed, 110 insertions(+), 11 deletions(-)
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 4df4f614f490..3f760d1d21de 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -860,26 +860,51 @@ static const uuid_t mem_mod_event_uuid =
UUID_INIT(0xfe927475, 0xdd59, 0x4339,
0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74);
-static void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
- enum cxl_event_log_type type,
- struct cxl_event_record_raw *record)
+void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
+ enum cxl_event_log_type type,
+ struct cxl_event_record_raw *record,
+ enum cxl_cper_event cper_event)
{
- uuid_t *id = &record->hdr.id;
-
- if (uuid_equal(id, &gen_media_event_uuid)) {
+ switch (cper_event) {
+ case CXL_CPER_EVENT_GEN_MEDIA: {
struct cxl_event_gen_media *rec =
(struct cxl_event_gen_media *)record;
trace_cxl_general_media(cxlmd, type, rec);
- } else if (uuid_equal(id, &dram_event_uuid)) {
+ break;
+ }
+ case CXL_CPER_EVENT_DRAM: {
struct cxl_event_dram *rec = (struct cxl_event_dram *)record;
trace_cxl_dram(cxlmd, type, rec);
- } else if (uuid_equal(id, &mem_mod_event_uuid)) {
+ break;
+ }
+ case CXL_CPER_EVENT_MEM_MODULE: {
struct cxl_event_mem_module *rec =
(struct cxl_event_mem_module *)record;
trace_cxl_memory_module(cxlmd, type, rec);
+ break;
+ }
+ }
+}
+EXPORT_SYMBOL_NS_GPL(cxl_event_trace_record, CXL);
+
+static void __cxl_event_trace_record(const struct cxl_memdev *cxlmd,
+ enum cxl_event_log_type type,
+ struct cxl_event_record_raw *record)
+{
+ uuid_t *id = &record->hdr.id;
+
+ if (uuid_equal(id, &gen_media_event_uuid)) {
+ cxl_event_trace_record(cxlmd, type, record,
+ CXL_CPER_EVENT_GEN_MEDIA);
+ } else if (uuid_equal(id, &dram_event_uuid)) {
+ cxl_event_trace_record(cxlmd, type, record,
+ CXL_CPER_EVENT_DRAM);
+ } else if (uuid_equal(id, &mem_mod_event_uuid)) {
+ cxl_event_trace_record(cxlmd, type, record,
+ CXL_CPER_EVENT_MEM_MODULE);
} else {
/* For unknown record types print just the header */
trace_cxl_generic_event(cxlmd, type, record);
@@ -991,8 +1016,8 @@ static void cxl_mem_get_records_log(struct cxl_memdev_state *mds,
break;
for (i = 0; i < nr_rec; i++)
- cxl_event_trace_record(cxlmd, type,
- &payload->records[i]);
+ __cxl_event_trace_record(cxlmd, type,
+ &payload->records[i]);
if (payload->flags & CXL_GET_EVENT_FLAG_OVERFLOW)
trace_cxl_overflow(cxlmd, type, payload);
diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
index 706f8a6d1ef4..89bd85e7f51c 100644
--- a/drivers/cxl/cxlmem.h
+++ b/drivers/cxl/cxlmem.h
@@ -6,6 +6,7 @@
#include <linux/cdev.h>
#include <linux/uuid.h>
#include <linux/rcuwait.h>
+#include <linux/efi.h>
#include "cxl.h"
/* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */
@@ -477,6 +478,8 @@ struct cxl_memdev_state {
struct cxl_security_state security;
struct cxl_fw_state fw;
+ struct notifier_block cxl_cper_nb;
+
struct rcuwait mbox_wait;
int (*mbox_send)(struct cxl_memdev_state *mds,
struct cxl_mbox_cmd *cmd);
@@ -863,6 +866,10 @@ void set_exclusive_cxl_commands(struct cxl_memdev_state *mds,
void clear_exclusive_cxl_commands(struct cxl_memdev_state *mds,
unsigned long *cmds);
void cxl_mem_get_event_records(struct cxl_memdev_state *mds, u32 status);
+void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
+ enum cxl_event_log_type type,
+ struct cxl_event_record_raw *record,
+ enum cxl_cper_event cper_event);
int cxl_set_timestamp(struct cxl_memdev_state *mds);
int cxl_poison_state_init(struct cxl_memdev_state *mds);
int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len,
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 44a21ab7add5..36d6f03e55de 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include <asm-generic/unaligned.h>
#include <linux/io-64-nonatomic-lo-hi.h>
#include <linux/moduleparam.h>
#include <linux/module.h>
@@ -10,6 +11,7 @@
#include <linux/pci.h>
#include <linux/aer.h>
#include <linux/io.h>
+#include <linux/efi.h>
#include "cxlmem.h"
#include "cxlpci.h"
#include "cxl.h"
@@ -748,6 +750,69 @@ static bool cxl_event_int_is_fw(u8 setting)
return mode == CXL_INT_FW;
}
+#define CXL_EVENT_HDR_FLAGS_REC_SEVERITY GENMASK(1, 0)
+int cxl_cper_event_call(struct notifier_block *nb, unsigned long action, void *data)
+{
+ struct cxl_cper_notifier_data *nd = data;
+ struct cxl_event_record_raw record = (struct cxl_event_record_raw) {
+ .hdr.id = UUID_INIT(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
+ };
+ enum cxl_event_log_type log_type;
+ struct cxl_memdev_state *mds;
+ u32 hdr_flags;
+
+ mds = container_of(nb, struct cxl_memdev_state, cxl_cper_nb);
+
+ /* Need serial number for device identification */
+ if (!(nd->rec->hdr.validation_bits & CPER_CXL_DEVICE_SN_VALID))
+ return NOTIFY_DONE;
+
+ /* FIXME endianess and bytes of serial number need verification */
+ /* FIXME Should other values be checked? */
+ if (memcmp(&mds->cxlds.serial, &nd->rec->hdr.dev_serial_num,
+ sizeof(mds->cxlds.serial)))
+ return NOTIFY_DONE;
+
+ /* ensure record can always handle the full CPER provided data */
+ BUILD_BUG_ON(sizeof(record) <
+ (CPER_CXL_COMP_EVENT_LOG_SIZE + sizeof(record.hdr.id)));
+
+ /*
+ * UEFI v2.10 defines N.2.14 defines the CXL CPER record as not
+ * including the uuid field.
+ */
+ memcpy(&record.hdr.length, &nd->rec->comp_event_log,
+ CPER_CXL_REC_LEN(nd->rec));
+
+ /* Fabricate a log type */
+ hdr_flags = get_unaligned_le24(record.hdr.flags);
+ log_type = FIELD_GET(CXL_EVENT_HDR_FLAGS_REC_SEVERITY, hdr_flags);
+
+ cxl_event_trace_record(mds->cxlds.cxlmd, log_type, &record,
+ nd->cper_event);
+
+ return NOTIFY_OK;
+}
+
+static void cxl_unregister_cper_events(void *_mds)
+{
+ struct cxl_memdev_state *mds = _mds;
+
+ unregister_cxl_cper_notifier(&mds->cxl_cper_nb);
+}
+
+static void register_cper_events(struct cxl_memdev_state *mds)
+{
+ mds->cxl_cper_nb.notifier_call = cxl_cper_event_call;
+
+ if (register_cxl_cper_notifier(&mds->cxl_cper_nb)) {
+ dev_err(mds->cxlds.dev, "CPER registration failed\n");
+ return;
+ }
+
+ devm_add_action_or_reset(mds->cxlds.dev, cxl_unregister_cper_events, mds);
+}
+
static int cxl_event_config(struct pci_host_bridge *host_bridge,
struct cxl_memdev_state *mds)
{
@@ -758,8 +823,10 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
* When BIOS maintains CXL error reporting control, it will process
* event records. Only one agent can do so.
*/
- if (!host_bridge->native_cxl_error)
+ if (!host_bridge->native_cxl_error) {
+ register_cper_events(mds);
return 0;
+ }
rc = cxl_mem_alloc_event_buf(mds);
if (rc)
--
2.41.0
Ira Weiny wrote:
> The uuid printed in the well known events is redundant. The uuid
> defines what the event was.
>
> Remove the uuid from the known events and only report it in the generic
> event as it remains informative there.
>
> Signed-off-by: Ira Weiny <[email protected]>
Looks good to me:
Reviewed-by: Dan Williams <[email protected]>
Ira Weiny wrote:
> BIOS can configure memory devices as firmware first. This will send CXL
> events to the firmware instead of the OS. The firmware can then send
> these events to the OS via UEFI.
>
> UEFI v2.10 section N.2.14 defines a Common Platform Error Record (CPER)
> format for CXL Component Events. The format is mostly the same as the
> CXL Common Event Record Format. The only difference is the UUID is
> passed via the Section Type as a GUID and not included as part of the
> record data.
>
> Add EFI support to detect CXL CPER records and call a notifier chain
> with the record data blobs.
>
> Note that the format of a GUID and UUID are not the same. Therefore the
> Section Type GUID defines are duplicated from the CXL code.
>
> Signed-off-by: Ira Weiny <[email protected]>
>
> ---
> Changes from RFC v1
> [iweiny: use an enum for know record types and skip converting GUID to UUID]
> [iweiny: commit to the UUID not being part of the event record data]
> [iweiny: use defines for GUID definitions]
> ---
> drivers/firmware/efi/cper.c | 16 +++++++++++
> drivers/firmware/efi/cper_cxl.c | 40 ++++++++++++++++++++++++++++
> drivers/firmware/efi/cper_cxl.h | 29 ++++++++++++++++++++
> include/linux/efi.h | 59 +++++++++++++++++++++++++++++++++++++++++
> 4 files changed, 144 insertions(+)
>
> diff --git a/drivers/firmware/efi/cper.c b/drivers/firmware/efi/cper.c
> index 35c37f667781..d6415c94d584 100644
> --- a/drivers/firmware/efi/cper.c
> +++ b/drivers/firmware/efi/cper.c
> @@ -607,6 +607,22 @@ cper_estatus_print_section(const char *pfx, struct acpi_hest_generic_data *gdata
> cper_print_prot_err(newpfx, prot_err);
> else
> goto err_section_too_small;
> + } else if (guid_equal(sec_type, &CPER_SEC_CXL_GEN_MEDIA) ||
> + guid_equal(sec_type, &CPER_SEC_CXL_DRAM) ||
> + guid_equal(sec_type, &CPER_SEC_CXL_MEM_MODULE)) {
> + struct cper_cxl_event_rec *rec = acpi_hest_get_payload(gdata);
> +
> + printk("%ssection type: CXL Event\n", newpfx);
I would say that since this is going to be hanlded elsewhere the kernel
log can stay silent.
> +
> + if (rec->hdr.length <= sizeof(rec->hdr))
> + goto err_section_too_small;
> +
> + if (rec->hdr.length > sizeof(*rec)) {
> + pr_err(FW_WARN "error section length is too big\n");
> + return;
> + }
> +
> + cper_post_cxl_event(newpfx, sec_type, rec);
> } else {
> const void *err = acpi_hest_get_payload(gdata);
>
> diff --git a/drivers/firmware/efi/cper_cxl.c b/drivers/firmware/efi/cper_cxl.c
> index a55771b99a97..04234884898d 100644
> --- a/drivers/firmware/efi/cper_cxl.c
> +++ b/drivers/firmware/efi/cper_cxl.c
> @@ -187,3 +187,43 @@ void cper_print_prot_err(const char *pfx, const struct cper_sec_prot_err *prot_e
> sizeof(cxl_ras->header_log), 0);
> }
> }
> +
> +/* CXL CPER notifier chain */
> +static BLOCKING_NOTIFIER_HEAD(cxl_cper_chain_head);
> +
> +void cper_post_cxl_event(const char *pfx, guid_t *sec_type,
> + struct cper_cxl_event_rec *rec)
> +{
> + struct cxl_cper_notifier_data nd = {
> + .rec = rec,
> + };
> +
> + if (!(rec->hdr.validation_bits & CPER_CXL_COMP_EVENT_LOG_VALID)) {
> + pr_err(FW_WARN "cxl event no Component Event Log present\n");
> + return;
> + }
> +
> + if (guid_equal(sec_type, &CPER_SEC_CXL_GEN_MEDIA))
> + nd.cper_event = CXL_CPER_EVENT_GEN_MEDIA;
> + else if (guid_equal(sec_type, &CPER_SEC_CXL_DRAM))
> + nd.cper_event = CXL_CPER_EVENT_DRAM;
> + else if (guid_equal(sec_type, &CPER_SEC_CXL_MEM_MODULE))
> + nd.cper_event = CXL_CPER_EVENT_MEM_MODULE;
> +
> + if (blocking_notifier_call_chain(&cxl_cper_chain_head, 0, (void *)&nd)
> + == NOTIFY_BAD)
> + pr_err(FW_WARN "cxl event notifier chain failed\n");
> +}
> +
> +int register_cxl_cper_notifier(struct notifier_block *nb)
> +{
> + return blocking_notifier_chain_register(&cxl_cper_chain_head, nb);
> +}
> +EXPORT_SYMBOL(register_cxl_cper_notifier);
I think this should be EXPORT_SYMBOL_NS_GPL(..., CXL) since I can't
imagine a third-party driver use case for this.
> +
> +void unregister_cxl_cper_notifier(struct notifier_block *nb)
> +{
> + blocking_notifier_chain_unregister(&cxl_cper_chain_head, nb);
> +}
> +EXPORT_SYMBOL(unregister_cxl_cper_notifier);
> +
> diff --git a/drivers/firmware/efi/cper_cxl.h b/drivers/firmware/efi/cper_cxl.h
> index 86bfcf7909ec..ca26126cd9b8 100644
> --- a/drivers/firmware/efi/cper_cxl.h
> +++ b/drivers/firmware/efi/cper_cxl.h
> @@ -10,11 +10,38 @@
> #ifndef LINUX_CPER_CXL_H
> #define LINUX_CPER_CXL_H
>
> +#include <linux/efi.h>
> +
> /* CXL Protocol Error Section */
> #define CPER_SEC_CXL_PROT_ERR \
> GUID_INIT(0x80B9EFB4, 0x52B5, 0x4DE3, 0xA7, 0x77, 0x68, 0x78, \
> 0x4B, 0x77, 0x10, 0x48)
>
> +/* CXL Event record UUIDs are used as the section type */
> +/*
> + * General Media Event Record
> + * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43
> + */
> +#define CPER_SEC_CXL_GEN_MEDIA \
> + GUID_INIT(0xfbcd0a77, 0xc260, 0x417f, \
> + 0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6)
> +
> +/*
> + * DRAM Event Record
> + * CXL rev 3.0 section 8.2.9.2.1.2; Table 8-44
> + */
> +#define CPER_SEC_CXL_DRAM \
> + GUID_INIT(0x601dcbb3, 0x9c06, 0x4eab, \
> + 0xb8, 0xaf, 0x4e, 0x9b, 0xfb, 0x5c, 0x96, 0x24)
> +
> +/*
> + * Memory Module Event Record
> + * CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45
> + */
> +#define CPER_SEC_CXL_MEM_MODULE \
> + GUID_INIT(0xfe927475, 0xdd59, 0x4339, \
> + 0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74)
> +
> #pragma pack(1)
>
> /* Compute Express Link Protocol Error Section, UEFI v2.10 sec N.2.13 */
> @@ -62,5 +89,7 @@ struct cper_sec_prot_err {
> #pragma pack()
>
> void cper_print_prot_err(const char *pfx, const struct cper_sec_prot_err *prot_err);
> +void cper_post_cxl_event(const char *pfx, guid_t *sec_type,
> + struct cper_cxl_event_rec *rec);
>
> #endif //__CPER_CXL_
> diff --git a/include/linux/efi.h b/include/linux/efi.h
> index 80b21d1c6eaf..b5b8b46c8deb 100644
> --- a/include/linux/efi.h
> +++ b/include/linux/efi.h
> @@ -1355,4 +1355,63 @@ bool efi_config_table_is_usable(const efi_guid_t *guid, unsigned long table)
>
> umode_t efi_attr_is_visible(struct kobject *kobj, struct attribute *attr, int n);
>
> +/*
> + * Event log size adjusted for CPER
> + *
> + * Base table from CXL r3.0 Table 8-42: (30h + 50h)
> + * For lack of UUID: - 10h
> + *
> + * (30h + 50h) - 10h = 70h
> + */
> +#define CPER_CXL_COMP_EVENT_LOG_SIZE 0x70
> +#define CPER_CXL_DEVICE_ID_VALID BIT(0)
> +#define CPER_CXL_DEVICE_SN_VALID BIT(1)
> +#define CPER_CXL_COMP_EVENT_LOG_VALID BIT(2)
> +struct cper_cxl_event_rec {
> + struct {
> + u32 length;
> + u64 validation_bits;
> + struct {
> + u16 vendor_id;
> + u16 device_id;
> + u8 func_num;
> + u8 device_num;
> + u8 bus_num;
> + u16 segment_num;
> + u16 slot_num; /* bits 2:0 reserved */
> + u8 reserved;
> + } device_id;
> + struct {
> + u32 lower_dw;
> + u32 upper_dw;
> + } dev_serial_num;
> + } hdr;
> +
> + u8 comp_event_log[CPER_CXL_COMP_EVENT_LOG_SIZE];
Rather than define CPER_CXL_COMP_EVENT_LOG_SIZE I would prefer that CXL
and EFI share a common struct definition for these common fields.
That would also remove the need for BUILD_BUG_ON() since they literally
can not disagree on the size in that case.
> +};
> +#define CPER_CXL_REC_LEN(rec) (rec->hdr.length - sizeof(rec->hdr))
> +
> +enum cxl_cper_event {
> + CXL_CPER_EVENT_GEN_MEDIA,
> + CXL_CPER_EVENT_DRAM,
> + CXL_CPER_EVENT_MEM_MODULE,
> +};
It follows from defining that common data structure above that this enum
would be a generic CXL namespace that drops "_CPER_". I.e. the CPER
notification handler and the native driver translate the event to this
common generic sub-structure that gets emitted.
> +
> +struct cxl_cper_notifier_data {
> + enum cxl_cper_event cper_event;
> + struct cper_cxl_event_rec *rec;
> +};
> +
> +#ifdef CONFIG_EFI
> +int register_cxl_cper_notifier(struct notifier_block *nb);
> +void unregister_cxl_cper_notifier(struct notifier_block *nb);
> +#else
> +static inline int register_cxl_cper_notifier(struct notifier_block *nb)
> +{
> + return 0;
> +}
> +
> +static inline void unregister_cxl_cper_notifier(struct notifier_block *nb) { }
> +#endif
> +
> #endif /* _LINUX_EFI_H */
>
> --
> 2.41.0
>
Ira Weiny wrote:
> If the firmware has configured CXL event support to be firmware first
> the OS can process those events through CPER records. Matching memory
> devices to the CPER records can be done via the serial number which is
> part of the CPER record header.
>
> Detect firmware first, register a notifier callback for each memdev, and
> trace events when they match a device registered.
>
> Signed-off-by: Ira Weiny <[email protected]>
>
[..]
The changes requested in patch2 cover all of the comments I currently
have on this patch, just one more cleanup below:
> +#define CXL_EVENT_HDR_FLAGS_REC_SEVERITY GENMASK(1, 0)
> +int cxl_cper_event_call(struct notifier_block *nb, unsigned long action, void *data)
> +{
> + struct cxl_cper_notifier_data *nd = data;
> + struct cxl_event_record_raw record = (struct cxl_event_record_raw) {
> + .hdr.id = UUID_INIT(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
> + };
Just do:
struct cxl_event_record_raw record = { 0 };
...and the compiler will take care of the rest as initializing any field
automatically initializes everything else to zero.
On Thu, 26 Oct 2023, Ira Weiny wrote:
>The uuid printed in the well known events is redundant. The uuid
>defines what the event was.
>
>Remove the uuid from the known events and only report it in the generic
>event as it remains informative there.
>
>Signed-off-by: Ira Weiny <[email protected]>
Reviewed-by: Davidlohr Bueso <[email protected]>
Hi Ira,
On 10/26/2023 11:21 AM, Ira Weiny wrote:
> If the firmware has configured CXL event support to be firmware first
> the OS can process those events through CPER records. Matching memory
> devices to the CPER records can be done via the serial number which is
> part of the CPER record header.
>
> Detect firmware first, register a notifier callback for each memdev, and
> trace events when they match a device registered.
>
> Signed-off-by: Ira Weiny <[email protected]>
>
> ---
> Changes from RFC v1:
> [iweiny: adjust to cper_event enum instead of converting guids]
> ---
> drivers/cxl/core/mbox.c | 45 +++++++++++++++++++++++++-------
> drivers/cxl/cxlmem.h | 7 +++++
> drivers/cxl/pci.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++-
> 3 files changed, 110 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
> index 4df4f614f490..3f760d1d21de 100644
> --- a/drivers/cxl/core/mbox.c
> +++ b/drivers/cxl/core/mbox.c
> @@ -860,26 +860,51 @@ static const uuid_t mem_mod_event_uuid =
> UUID_INIT(0xfe927475, 0xdd59, 0x4339,
> 0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74);
>
> -static void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
> - enum cxl_event_log_type type,
> - struct cxl_event_record_raw *record)
> +void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
> + enum cxl_event_log_type type,
> + struct cxl_event_record_raw *record,
> + enum cxl_cper_event cper_event)
> {
> - uuid_t *id = &record->hdr.id;
> -
> - if (uuid_equal(id, &gen_media_event_uuid)) {
> + switch (cper_event) {
> + case CXL_CPER_EVENT_GEN_MEDIA: {
> struct cxl_event_gen_media *rec =
> (struct cxl_event_gen_media *)record;
>
> trace_cxl_general_media(cxlmd, type, rec);
> - } else if (uuid_equal(id, &dram_event_uuid)) {
> + break;
> + }
> + case CXL_CPER_EVENT_DRAM: {
> struct cxl_event_dram *rec = (struct cxl_event_dram *)record;
>
> trace_cxl_dram(cxlmd, type, rec);
> - } else if (uuid_equal(id, &mem_mod_event_uuid)) {
> + break;
> + }
> + case CXL_CPER_EVENT_MEM_MODULE: {
> struct cxl_event_mem_module *rec =
> (struct cxl_event_mem_module *)record;
>
> trace_cxl_memory_module(cxlmd, type, rec);
> + break;
> + }
> + }
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_event_trace_record, CXL);
> +
> +static void __cxl_event_trace_record(const struct cxl_memdev *cxlmd,
> + enum cxl_event_log_type type,
> + struct cxl_event_record_raw *record)
> +{
> + uuid_t *id = &record->hdr.id;
> +
> + if (uuid_equal(id, &gen_media_event_uuid)) {
> + cxl_event_trace_record(cxlmd, type, record,
> + CXL_CPER_EVENT_GEN_MEDIA);
> + } else if (uuid_equal(id, &dram_event_uuid)) {
> + cxl_event_trace_record(cxlmd, type, record,
> + CXL_CPER_EVENT_DRAM);
> + } else if (uuid_equal(id, &mem_mod_event_uuid)) {
> + cxl_event_trace_record(cxlmd, type, record,
> + CXL_CPER_EVENT_MEM_MODULE);
> } else {
> /* For unknown record types print just the header */
> trace_cxl_generic_event(cxlmd, type, record);
> @@ -991,8 +1016,8 @@ static void cxl_mem_get_records_log(struct cxl_memdev_state *mds,
> break;
>
> for (i = 0; i < nr_rec; i++)
> - cxl_event_trace_record(cxlmd, type,
> - &payload->records[i]);
> + __cxl_event_trace_record(cxlmd, type,
> + &payload->records[i]);
>
> if (payload->flags & CXL_GET_EVENT_FLAG_OVERFLOW)
> trace_cxl_overflow(cxlmd, type, payload);
> diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> index 706f8a6d1ef4..89bd85e7f51c 100644
> --- a/drivers/cxl/cxlmem.h
> +++ b/drivers/cxl/cxlmem.h
> @@ -6,6 +6,7 @@
> #include <linux/cdev.h>
> #include <linux/uuid.h>
> #include <linux/rcuwait.h>
> +#include <linux/efi.h>
> #include "cxl.h"
>
> /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */
> @@ -477,6 +478,8 @@ struct cxl_memdev_state {
> struct cxl_security_state security;
> struct cxl_fw_state fw;
>
> + struct notifier_block cxl_cper_nb;
> +
> struct rcuwait mbox_wait;
> int (*mbox_send)(struct cxl_memdev_state *mds,
> struct cxl_mbox_cmd *cmd);
> @@ -863,6 +866,10 @@ void set_exclusive_cxl_commands(struct cxl_memdev_state *mds,
> void clear_exclusive_cxl_commands(struct cxl_memdev_state *mds,
> unsigned long *cmds);
> void cxl_mem_get_event_records(struct cxl_memdev_state *mds, u32 status);
> +void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
> + enum cxl_event_log_type type,
> + struct cxl_event_record_raw *record,
> + enum cxl_cper_event cper_event);
> int cxl_set_timestamp(struct cxl_memdev_state *mds);
> int cxl_poison_state_init(struct cxl_memdev_state *mds);
> int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len,
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 44a21ab7add5..36d6f03e55de 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -1,5 +1,6 @@
> // SPDX-License-Identifier: GPL-2.0-only
> /* Copyright(c) 2020 Intel Corporation. All rights reserved. */
> +#include <asm-generic/unaligned.h>
> #include <linux/io-64-nonatomic-lo-hi.h>
> #include <linux/moduleparam.h>
> #include <linux/module.h>
> @@ -10,6 +11,7 @@
> #include <linux/pci.h>
> #include <linux/aer.h>
> #include <linux/io.h>
> +#include <linux/efi.h>
> #include "cxlmem.h"
> #include "cxlpci.h"
> #include "cxl.h"
> @@ -748,6 +750,69 @@ static bool cxl_event_int_is_fw(u8 setting)
> return mode == CXL_INT_FW;
> }
>
> +#define CXL_EVENT_HDR_FLAGS_REC_SEVERITY GENMASK(1, 0)
> +int cxl_cper_event_call(struct notifier_block *nb, unsigned long action, void *data)
> +{
> + struct cxl_cper_notifier_data *nd = data;
> + struct cxl_event_record_raw record = (struct cxl_event_record_raw) {
> + .hdr.id = UUID_INIT(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
> + };
> + enum cxl_event_log_type log_type;
> + struct cxl_memdev_state *mds;
> + u32 hdr_flags;
> +
> + mds = container_of(nb, struct cxl_memdev_state, cxl_cper_nb);
> +
> + /* Need serial number for device identification */
> + if (!(nd->rec->hdr.validation_bits & CPER_CXL_DEVICE_SN_VALID))
> + return NOTIFY_DONE;
For all the event records that I tested so far, this has never been
true. That is CPER_CXL_DEVICE_SN_VALID is never set which might not log
the records at all. Should we be bit more lenient here and include
validating device_id (bdf) instead and check if cxlds exist?
pci_get_domain_bus_and_slot() and pci_get_drvdata()..
> +
> + /* FIXME endianess and bytes of serial number need verification */
> + /* FIXME Should other values be checked? */
> + if (memcmp(&mds->cxlds.serial, &nd->rec->hdr.dev_serial_num,
> + sizeof(mds->cxlds.serial)))
> + return NOTIFY_DONE;
> +
> + /* ensure record can always handle the full CPER provided data */
> + BUILD_BUG_ON(sizeof(record) <
> + (CPER_CXL_COMP_EVENT_LOG_SIZE + sizeof(record.hdr.id)));
> +
> + /*
> + * UEFI v2.10 defines N.2.14 defines the CXL CPER record as not
> + * including the uuid field.
> + */
> + memcpy(&record.hdr.length, &nd->rec->comp_event_log,
> + CPER_CXL_REC_LEN(nd->rec));
I'm doubtful this will do the job. I think we should copy into each
field of struct cxl_event_record_hdr individually starting from length
by pointer arithmetic (which is definitely bad, but I cannot think of a
better way to do this) and then do memcpy for data field in struct
cxl_event_record_raw..
Any other suggestions would be helpful as well.
I can make these changes and validate it on my end if that works..?
Thanks,
Smita
> +
> + /* Fabricate a log type */
> + hdr_flags = get_unaligned_le24(record.hdr.flags);
> + log_type = FIELD_GET(CXL_EVENT_HDR_FLAGS_REC_SEVERITY, hdr_flags);
> +
> + cxl_event_trace_record(mds->cxlds.cxlmd, log_type, &record,
> + nd->cper_event);
> +
> + return NOTIFY_OK;
> +}
> +
> +static void cxl_unregister_cper_events(void *_mds)
> +{
> + struct cxl_memdev_state *mds = _mds;
> +
> + unregister_cxl_cper_notifier(&mds->cxl_cper_nb);
> +}
> +
> +static void register_cper_events(struct cxl_memdev_state *mds)
> +{
> + mds->cxl_cper_nb.notifier_call = cxl_cper_event_call;
> +
> + if (register_cxl_cper_notifier(&mds->cxl_cper_nb)) {
> + dev_err(mds->cxlds.dev, "CPER registration failed\n");
> + return;
> + }
> +
> + devm_add_action_or_reset(mds->cxlds.dev, cxl_unregister_cper_events, mds);
> +}
> +
> static int cxl_event_config(struct pci_host_bridge *host_bridge,
> struct cxl_memdev_state *mds)
> {
> @@ -758,8 +823,10 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
> * When BIOS maintains CXL error reporting control, it will process
> * event records. Only one agent can do so.
> */
> - if (!host_bridge->native_cxl_error)
> + if (!host_bridge->native_cxl_error) {
> + register_cper_events(mds);
> return 0;
> + }
>
> rc = cxl_mem_alloc_event_buf(mds);
> if (rc)
>
Smita Koralahalli wrote:
[..]
> > +#define CXL_EVENT_HDR_FLAGS_REC_SEVERITY GENMASK(1, 0)
> > +int cxl_cper_event_call(struct notifier_block *nb, unsigned long action, void *data)
> > +{
> > + struct cxl_cper_notifier_data *nd = data;
> > + struct cxl_event_record_raw record = (struct cxl_event_record_raw) {
> > + .hdr.id = UUID_INIT(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
> > + };
> > + enum cxl_event_log_type log_type;
> > + struct cxl_memdev_state *mds;
> > + u32 hdr_flags;
> > +
> > + mds = container_of(nb, struct cxl_memdev_state, cxl_cper_nb);
> > +
> > + /* Need serial number for device identification */
> > + if (!(nd->rec->hdr.validation_bits & CPER_CXL_DEVICE_SN_VALID))
> > + return NOTIFY_DONE;
>
> For all the event records that I tested so far, this has never been
> true. That is CPER_CXL_DEVICE_SN_VALID is never set which might not log
> the records at all. Should we be bit more lenient here and include
> validating device_id (bdf) instead and check if cxlds exist?
Agree. While I do think those devices are out of spec given CXL mandates
a valid serial number, I think the robustness priciple applies and Linux
should rely on bdf information. I also expect that with MH-SLDs and
potentially other scenarios, a serial number may be duplicated so bdf is
more reliable in that dimension as well.
> > + /* FIXME endianess and bytes of serial number need verification */
> > + /* FIXME Should other values be checked? */
> > + if (memcmp(&mds->cxlds.serial, &nd->rec->hdr.dev_serial_num,
> > + sizeof(mds->cxlds.serial)))
> > + return NOTIFY_DONE;
> > +
> > + /* ensure record can always handle the full CPER provided data */
> > + BUILD_BUG_ON(sizeof(record) <
> > + (CPER_CXL_COMP_EVENT_LOG_SIZE + sizeof(record.hdr.id)));
> > +
> > + /*
> > + * UEFI v2.10 defines N.2.14 defines the CXL CPER record as not
> > + * including the uuid field.
> > + */
> > + memcpy(&record.hdr.length, &nd->rec->comp_event_log,
> > + CPER_CXL_REC_LEN(nd->rec));
>
> I'm doubtful this will do the job. I think we should copy into each
> field of struct cxl_event_record_hdr individually starting from length
> by pointer arithmetic (which is definitely bad, but I cannot think of a
> better way to do this) and then do memcpy for data field in struct
> cxl_event_record_raw..
>
> Any other suggestions would be helpful as well.
>
> I can make these changes and validate it on my end if that works..?
It sounds like you have a more readily available real world test
environment for this, so that sounds good to me.
Dan Williams wrote:
> Ira Weiny wrote:
[snip]
> >
> > diff --git a/drivers/firmware/efi/cper.c b/drivers/firmware/efi/cper.c
> > index 35c37f667781..d6415c94d584 100644
> > --- a/drivers/firmware/efi/cper.c
> > +++ b/drivers/firmware/efi/cper.c
> > @@ -607,6 +607,22 @@ cper_estatus_print_section(const char *pfx, struct acpi_hest_generic_data *gdata
> > cper_print_prot_err(newpfx, prot_err);
> > else
> > goto err_section_too_small;
> > + } else if (guid_equal(sec_type, &CPER_SEC_CXL_GEN_MEDIA) ||
> > + guid_equal(sec_type, &CPER_SEC_CXL_DRAM) ||
> > + guid_equal(sec_type, &CPER_SEC_CXL_MEM_MODULE)) {
> > + struct cper_cxl_event_rec *rec = acpi_hest_get_payload(gdata);
> > +
> > + printk("%ssection type: CXL Event\n", newpfx);
>
> I would say that since this is going to be hanlded elsewhere the kernel
> log can stay silent.
Yep, bad cargo cult.
Removed.
[snip]
> > +
> > +int register_cxl_cper_notifier(struct notifier_block *nb)
> > +{
> > + return blocking_notifier_chain_register(&cxl_cper_chain_head, nb);
> > +}
> > +EXPORT_SYMBOL(register_cxl_cper_notifier);
>
> I think this should be EXPORT_SYMBOL_NS_GPL(..., CXL) since I can't
> imagine a third-party driver use case for this.
Good point. Done.
[snip]
> >
> > +/*
> > + * Event log size adjusted for CPER
> > + *
> > + * Base table from CXL r3.0 Table 8-42: (30h + 50h)
> > + * For lack of UUID: - 10h
> > + *
> > + * (30h + 50h) - 10h = 70h
> > + */
> > +#define CPER_CXL_COMP_EVENT_LOG_SIZE 0x70
> > +#define CPER_CXL_DEVICE_ID_VALID BIT(0)
> > +#define CPER_CXL_DEVICE_SN_VALID BIT(1)
> > +#define CPER_CXL_COMP_EVENT_LOG_VALID BIT(2)
> > +struct cper_cxl_event_rec {
> > + struct {
> > + u32 length;
> > + u64 validation_bits;
> > + struct {
> > + u16 vendor_id;
> > + u16 device_id;
> > + u8 func_num;
> > + u8 device_num;
> > + u8 bus_num;
> > + u16 segment_num;
> > + u16 slot_num; /* bits 2:0 reserved */
> > + u8 reserved;
> > + } device_id;
> > + struct {
> > + u32 lower_dw;
> > + u32 upper_dw;
> > + } dev_serial_num;
> > + } hdr;
> > +
> > + u8 comp_event_log[CPER_CXL_COMP_EVENT_LOG_SIZE];
>
> Rather than define CPER_CXL_COMP_EVENT_LOG_SIZE I would prefer that CXL
> and EFI share a common struct definition for these common fields.
>
> That would also remove the need for BUILD_BUG_ON() since they literally
> can not disagree on the size in that case.
I don't know if we discussed this publicly or internally but I had
versions which lifted the CXL structs to the core. But your opinion at
that time was it was not needed.
Looking at it again I might get away with the main event record struct
defined here. But it is kind of odd to be in efi.h IMO.
>
> > +};
> > +#define CPER_CXL_REC_LEN(rec) (rec->hdr.length - sizeof(rec->hdr))
> > +
> > +enum cxl_cper_event {
> > + CXL_CPER_EVENT_GEN_MEDIA,
> > + CXL_CPER_EVENT_DRAM,
> > + CXL_CPER_EVENT_MEM_MODULE,
> > +};
>
> It follows from defining that common data structure above that this enum
> would be a generic CXL namespace that drops "_CPER_". I.e. the CPER
> notification handler and the native driver translate the event to this
> common generic sub-structure that gets emitted.
Ok this would be along the lines of promoting the definitions to a common
header. Again I kept things pretty separate because it seemed that was
the direction you wanted to go.
I don't particularly like the memcpy which Smita flagged either. But I
think this will require reworking the trace code to take a 'non-uuid'
structure equal to the payload 'comp_event_log' above.
Based on these comments I'll add some header promotion of common record
structures and see how it works out with the modified trace code.
Do you have any opinions on the name of the core header?
Ira
Dan Williams wrote:
> Ira Weiny wrote:
> > If the firmware has configured CXL event support to be firmware first
> > the OS can process those events through CPER records. Matching memory
> > devices to the CPER records can be done via the serial number which is
> > part of the CPER record header.
> >
> > Detect firmware first, register a notifier callback for each memdev, and
> > trace events when they match a device registered.
> >
> > Signed-off-by: Ira Weiny <[email protected]>
> >
> [..]
>
> The changes requested in patch2 cover all of the comments I currently
> have on this patch, just one more cleanup below:
>
> > +#define CXL_EVENT_HDR_FLAGS_REC_SEVERITY GENMASK(1, 0)
> > +int cxl_cper_event_call(struct notifier_block *nb, unsigned long action, void *data)
> > +{
> > + struct cxl_cper_notifier_data *nd = data;
> > + struct cxl_event_record_raw record = (struct cxl_event_record_raw) {
> > + .hdr.id = UUID_INIT(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
> > + };
>
> Just do:
>
> struct cxl_event_record_raw record = { 0 };
>
> ...and the compiler will take care of the rest as initializing any field
> automatically initializes everything else to zero.
Not quite sure what I was thinking.
This works better but I think I can avoid needing this initialization all
together by reworking the structures. For now I have made the change and
I'll see how it falls out.
Thanks,
Ira
Smita Koralahalli wrote:
> >
[snip]
> > +#define CXL_EVENT_HDR_FLAGS_REC_SEVERITY GENMASK(1, 0)
> > +int cxl_cper_event_call(struct notifier_block *nb, unsigned long action, void *data)
> > +{
> > + struct cxl_cper_notifier_data *nd = data;
> > + struct cxl_event_record_raw record = (struct cxl_event_record_raw) {
> > + .hdr.id = UUID_INIT(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
> > + };
> > + enum cxl_event_log_type log_type;
> > + struct cxl_memdev_state *mds;
> > + u32 hdr_flags;
> > +
> > + mds = container_of(nb, struct cxl_memdev_state, cxl_cper_nb);
> > +
> > + /* Need serial number for device identification */
> > + if (!(nd->rec->hdr.validation_bits & CPER_CXL_DEVICE_SN_VALID))
> > + return NOTIFY_DONE;
>
> For all the event records that I tested so far, this has never been
> true. That is CPER_CXL_DEVICE_SN_VALID is never set which might not log
> the records at all. Should we be bit more lenient here and include
> validating device_id (bdf) instead and check if cxlds exist?
>
> pci_get_domain_bus_and_slot() and pci_get_drvdata()..
Checking BDF is reasonable. Not sure what you mean by 'check if cxlds
exists'?
This will be called on each memdev so the cxlds should be valid. Do you
think we need some locking? I think the unregister will block device
removal if this is running.
>
> > +
> > + /* FIXME endianess and bytes of serial number need verification */
> > + /* FIXME Should other values be checked? */
> > + if (memcmp(&mds->cxlds.serial, &nd->rec->hdr.dev_serial_num,
> > + sizeof(mds->cxlds.serial)))
> > + return NOTIFY_DONE;
> > +
> > + /* ensure record can always handle the full CPER provided data */
> > + BUILD_BUG_ON(sizeof(record) <
> > + (CPER_CXL_COMP_EVENT_LOG_SIZE + sizeof(record.hdr.id)));
> > +
> > + /*
> > + * UEFI v2.10 defines N.2.14 defines the CXL CPER record as not
> > + * including the uuid field.
> > + */
> > + memcpy(&record.hdr.length, &nd->rec->comp_event_log,
> > + CPER_CXL_REC_LEN(nd->rec));
>
> I'm doubtful this will do the job.
I'm not sure why but, see below...
> I think we should copy into each
> field of struct cxl_event_record_hdr individually starting from length
> by pointer arithmetic (which is definitely bad, but I cannot think of a
> better way to do this) and then do memcpy for data field in struct
> cxl_event_record_raw..
>
> Any other suggestions would be helpful as well.
Based on Dan's suggestion to share the structures this memcpy can be
avoided altogether. Let's see how that works.
>
> I can make these changes and validate it on my end if that works..?
Any testing would be welcome. I don't have a test setup readily
available.
Ira
[snip]
On Tue, 31 Oct 2023 10:13:40 -0700
Ira Weiny <[email protected]> wrote:
> Dan Williams wrote:
> > Ira Weiny wrote:
> > > If the firmware has configured CXL event support to be firmware first
> > > the OS can process those events through CPER records. Matching memory
> > > devices to the CPER records can be done via the serial number which is
> > > part of the CPER record header.
> > >
> > > Detect firmware first, register a notifier callback for each memdev, and
> > > trace events when they match a device registered.
> > >
> > > Signed-off-by: Ira Weiny <[email protected]>
> > >
> > [..]
> >
> > The changes requested in patch2 cover all of the comments I currently
> > have on this patch, just one more cleanup below:
> >
> > > +#define CXL_EVENT_HDR_FLAGS_REC_SEVERITY GENMASK(1, 0)
> > > +int cxl_cper_event_call(struct notifier_block *nb, unsigned long action, void *data)
> > > +{
> > > + struct cxl_cper_notifier_data *nd = data;
> > > + struct cxl_event_record_raw record = (struct cxl_event_record_raw) {
> > > + .hdr.id = UUID_INIT(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
> > > + };
> >
> > Just do:
> >
> > struct cxl_event_record_raw record = { 0 };
FWIW (I'm in a pedantic mood), unless you care about that specific 0
= {};
equally valid.
c99
"If there are fewer initializers in a brace-enclosed list than there
are elements or members of an aggregate ... the remainder of the
aggregate shall be initiali\ed implicitly the same as objects
that have static storage duration."
i.e. to 0.
Nothing says that fewer != 0 ;)
> >
> > ...and the compiler will take care of the rest as initializing any field
> > automatically initializes everything else to zero.
>
> Not quite sure what I was thinking.
>
> This works better but I think I can avoid needing this initialization all
> together by reworking the structures. For now I have made the change and
> I'll see how it falls out.
>
> Thanks,
> Ira
>
Ira Weiny wrote:
> Smita Koralahalli wrote:
>
[snip]
> > I think we should copy into each
> > field of struct cxl_event_record_hdr individually starting from length
> > by pointer arithmetic (which is definitely bad, but I cannot think of a
> > better way to do this) and then do memcpy for data field in struct
> > cxl_event_record_raw..
> >
> > Any other suggestions would be helpful as well.
>
> Based on Dan's suggestion to share the structures this memcpy can be
> avoided altogether. Let's see how that works.
memcpy was removed.
>
> >
> > I can make these changes and validate it on my end if that works..?
>
> Any testing would be welcome. I don't have a test setup readily
> available.
Sorry this took so long but I have a new series with the
headers/structures re-aranged along with the other changes.
https://lore.kernel.org/all/[email protected]/
Thanks,
Ira