If a device is configured for firmware first CXL event records are not
sent directly to the host, rather they are reported through the EFI
Common Platform Error Records (CPER). EFI 2.10 Section N.2.14 defines
the CXL CPER to wrap a mostly CXL event payload.
The CXL sub-system uniquely has DPA to HPA translation information.[0]
It also already has event decoding/tracing. Such translations are very
useful for users to determine which system issues may correspond to
specific hardware events.
The restructuring of the event data structures in 6.8 made sharing the
data between CPER/event logs more efficient. Now re-wire the sending of
CPER records to the CXL sub-system.
In addition provide a default RAS event should the CXL module not be
loaded.
Series status/background
========================
Smita and Jonathan have been a great help with this series. Once again
thank you.
Unfortunately, with all the churn surrounding the bug which Dan
Carpenter found the maintainers were force to revert this work.
Testing
=======
A quick hack was added to debugfs patch to facilitate easier testing.[1]
With this it was verified that the bug Dan Carpenter found is fixed.
However, the tp_printk bug Jonathan found remains. Fortunately,
tp_printk is not widely used so it is anticipated this will not be an
issue.
[0]
Link: https://lore.kernel.org/all/[email protected]/
[1]
Link: https://github.com/weiny2/linux-kernel/commit/9b1f33314e8488506dbad63dc1c896386d4803d6
Signed-off-by: Ira Weiny <[email protected]>
---
Changes in v2:
- iweiny: address comments from V1 (noted in the patches themselves)
- iweiny: drop header file clean up patch (only needed for my debugfs test)
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Ira Weiny (3):
acpi/ghes: Process CXL Component Events
cxl/pci: Process CPER events
ras/events: Trace CXL CPER events without CXL stack
drivers/acpi/apei/ghes.c | 128 ++++++++++++++++++++++++++++++++++++++++++++++
drivers/cxl/pci.c | 61 +++++++++++++++++++++-
include/linux/cxl-event.h | 18 +++++++
include/ras/ras_event.h | 51 ++++++++++++++++++
4 files changed, 257 insertions(+), 1 deletion(-)
---
base-commit: 4d2008430ce87061c9cefd4f83daf2d5bb323a96
change-id: 20240220-cxl-cper3-30e55279f936
Best regards,
--
Ira Weiny <[email protected]>
BIOS can configure memory devices as firmware first. This will send CXL
events to the firmware instead of the OS. The firmware can then inform
the OS of these events via UEFI.
UEFI v2.10 section N.2.14 defines a Common Platform Error Record (CPER)
format for CXL Component Events. The format is mostly the same as the
CXL Common Event Record Format. The difference lies in the use of a
GUID as the CPER Section Type which matches the UUID defined in CXL 3.1
Table 8-43.
Currently a configuration such as this will trace a non standard event
in the log omitting useful details of the event. In addition the CXL
sub-system contains additional region and HPA information useful to the
user.[0]
Add GHES support to detect CXL CPER records. Add the ability for the
CXL sub-system to register a callback to receive the events.
The CXL code is required to be called from process context as it needs
to take a device lock. The GHES code may be in interrupt context. This
complicated the use of a callback. Dan Williams suggested the use of
work items as an atomic way of switching between the callback execution
and a default handler.[1]
This patch adds back the functionality which was removed to fix the
report by Dan Carpenter[2].
[0]
Link: https://lore.kernel.org/all/[email protected]/
[1]
Link: https://lore.kernel.org/all/[email protected]/
[2]
Link: https://lore.kernel.org/all/[email protected]/
Cc: Ard Biesheuvel <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>
Cc: Tony Luck <[email protected]>
Cc: Borislav Petkov <[email protected]>
Suggested-by: Dan Carpenter <[email protected]>
Suggested-by: Dan Williams <[email protected]>
Signed-off-by: Ira Weiny <[email protected]>
---
Changes:
[iweiny: clarify commit message]
[djbw: remove local wt]
---
drivers/acpi/apei/ghes.c | 124 ++++++++++++++++++++++++++++++++++++++++++++++
include/linux/cxl-event.h | 18 +++++++
2 files changed, 142 insertions(+)
diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
index 512067cac170..cdcfdf6ebe81 100644
--- a/drivers/acpi/apei/ghes.c
+++ b/drivers/acpi/apei/ghes.c
@@ -26,6 +26,8 @@
#include <linux/interrupt.h>
#include <linux/timer.h>
#include <linux/cper.h>
+#include <linux/cleanup.h>
+#include <linux/cxl-event.h>
#include <linux/platform_device.h>
#include <linux/mutex.h>
#include <linux/ratelimit.h>
@@ -33,6 +35,7 @@
#include <linux/irq_work.h>
#include <linux/llist.h>
#include <linux/genalloc.h>
+#include <linux/kfifo.h>
#include <linux/pci.h>
#include <linux/pfn.h>
#include <linux/aer.h>
@@ -673,6 +676,112 @@ static void ghes_defer_non_standard_event(struct acpi_hest_generic_data *gdata,
schedule_work(&entry->work);
}
+/* CXL Event record UUIDs are formated as GUIDs and reported in section type */
+
+/*
+ * General Media Event Record
+ * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43
+ */
+#define CPER_SEC_CXL_GEN_MEDIA_GUID \
+ GUID_INIT(0xfbcd0a77, 0xc260, 0x417f, \
+ 0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6)
+
+/*
+ * DRAM Event Record
+ * CXL rev 3.0 section 8.2.9.2.1.2; Table 8-44
+ */
+#define CPER_SEC_CXL_DRAM_GUID \
+ GUID_INIT(0x601dcbb3, 0x9c06, 0x4eab, \
+ 0xb8, 0xaf, 0x4e, 0x9b, 0xfb, 0x5c, 0x96, 0x24)
+
+/*
+ * Memory Module Event Record
+ * CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45
+ */
+#define CPER_SEC_CXL_MEM_MODULE_GUID \
+ GUID_INIT(0xfe927475, 0xdd59, 0x4339, \
+ 0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74)
+
+struct cxl_cper_work_data {
+ enum cxl_event_type event_type;
+ struct cxl_cper_event_rec rec;
+};
+
+DEFINE_KFIFO(cxl_cper_fifo, struct cxl_cper_work_data, 32);
+static DEFINE_SPINLOCK(cxl_cper_work_lock);
+static cxl_cper_callback cper_callback;
+static void cxl_cper_cb_fn(struct work_struct *work)
+{
+ struct cxl_cper_work_data wd;
+
+ while (kfifo_get(&cxl_cper_fifo, &wd))
+ cper_callback(wd.event_type, &wd.rec);
+}
+static DECLARE_WORK(cxl_cb_work, cxl_cper_cb_fn);
+struct work_struct *cxl_cper_work = NULL;
+
+static void cxl_cper_post_event(enum cxl_event_type event_type,
+ struct cxl_cper_event_rec *rec)
+{
+ struct cxl_cper_work_data wd;
+
+ if (rec->hdr.length <= sizeof(rec->hdr) ||
+ rec->hdr.length > sizeof(*rec)) {
+ pr_err(FW_WARN "CXL CPER Invalid section length (%u)\n",
+ rec->hdr.length);
+ return;
+ }
+
+ if (!(rec->hdr.validation_bits & CPER_CXL_COMP_EVENT_LOG_VALID)) {
+ pr_err(FW_WARN "CXL CPER invalid event\n");
+ return;
+ }
+
+ guard(spinlock_irqsave)(&cxl_cper_work_lock);
+
+ if (!cxl_cper_work)
+ return;
+
+ wd.event_type = event_type;
+ memcpy(&wd.rec, rec, sizeof(wd.rec));
+
+ if (!kfifo_put(&cxl_cper_fifo, wd)) {
+ pr_err_ratelimited("CXL CPER kfifo overflow\n");
+ return;
+ }
+
+ schedule_work(cxl_cper_work);
+}
+
+int cxl_cper_register_callback(cxl_cper_callback callback)
+{
+ if (cper_callback)
+ return -EINVAL;
+
+ guard(spinlock)(&cxl_cper_work_lock);
+ cper_callback = callback;
+ cxl_cper_work = &cxl_cb_work;
+ return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_cper_register_callback, CXL);
+
+int cxl_cper_unregister_callback(cxl_cper_callback callback)
+{
+ if (callback != cper_callback)
+ return -EINVAL;
+
+ /* Avoid guard() because cancel_work_sync() can sleep */
+ spin_lock(&cxl_cper_work_lock);
+ cxl_cper_work = NULL;
+ spin_unlock(&cxl_cper_work_lock);
+
+ cancel_work_sync(&cxl_cb_work);
+ cper_callback = NULL;
+
+ return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_cper_unregister_callback, CXL);
+
static bool ghes_do_proc(struct ghes *ghes,
const struct acpi_hest_generic_status *estatus)
{
@@ -707,6 +816,21 @@ static bool ghes_do_proc(struct ghes *ghes,
}
else if (guid_equal(sec_type, &CPER_SEC_PROC_ARM)) {
queued = ghes_handle_arm_hw_error(gdata, sev, sync);
+ }
+ else if (guid_equal(sec_type, &CPER_SEC_CXL_GEN_MEDIA_GUID)) {
+ struct cxl_cper_event_rec *rec = acpi_hest_get_payload(gdata);
+
+ cxl_cper_post_event(CXL_CPER_EVENT_GEN_MEDIA, rec);
+ }
+ else if (guid_equal(sec_type, &CPER_SEC_CXL_DRAM_GUID)) {
+ struct cxl_cper_event_rec *rec = acpi_hest_get_payload(gdata);
+
+ cxl_cper_post_event(CXL_CPER_EVENT_DRAM, rec);
+ }
+ else if (guid_equal(sec_type, &CPER_SEC_CXL_MEM_MODULE_GUID)) {
+ struct cxl_cper_event_rec *rec = acpi_hest_get_payload(gdata);
+
+ cxl_cper_post_event(CXL_CPER_EVENT_MEM_MODULE, rec);
} else {
void *err = acpi_hest_get_payload(gdata);
diff --git a/include/linux/cxl-event.h b/include/linux/cxl-event.h
index 03fa6d50d46f..91125eca4c8a 100644
--- a/include/linux/cxl-event.h
+++ b/include/linux/cxl-event.h
@@ -140,4 +140,22 @@ struct cxl_cper_event_rec {
union cxl_event event;
} __packed;
+typedef void (*cxl_cper_callback)(enum cxl_event_type type,
+ struct cxl_cper_event_rec *rec);
+
+#ifdef CONFIG_ACPI_APEI_GHES
+int cxl_cper_register_callback(cxl_cper_callback callback);
+int cxl_cper_unregister_callback(cxl_cper_callback callback);
+#else
+static inline int cxl_cper_register_callback(cxl_cper_callback callback)
+{
+ return 0;
+}
+
+static inline int cxl_cper_unregister_callback(cxl_cper_callback callback)
+{
+ return 0;
+}
+#endif
+
#endif /* _LINUX_CXL_EVENT_H */
--
2.44.0
If the firmware has configured CXL event support to be firmware first
the OS will receive those events through CPER records. The CXL layer has
unique DPA to HPA knowledge and existing event trace parsing in
place.[0]
Add a CXL CPER callback to process CPER events.
[0]
Link: https://lore.kernel.org/all/[email protected]/
Signed-off-by: Ira Weiny <[email protected]>
---
Changes:
[djbw: Clean up error and debug messages]
[iweiny: Clean up commit message]
[iweiny: Add back in after the revert in 6.8]
---
drivers/cxl/pci.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 60 insertions(+), 1 deletion(-)
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 2ff361e756d6..c0bc2e994a90 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -974,6 +974,65 @@ static struct pci_driver cxl_pci_driver = {
},
};
-module_pci_driver(cxl_pci_driver);
+#define CXL_EVENT_HDR_FLAGS_REC_SEVERITY GENMASK(1, 0)
+static void cxl_cper_event_call(enum cxl_event_type ev_type,
+ struct cxl_cper_event_rec *rec)
+{
+ struct cper_cxl_event_devid *device_id = &rec->hdr.device_id;
+ struct pci_dev *pdev __free(pci_dev_put) = NULL;
+ enum cxl_event_log_type log_type;
+ struct cxl_dev_state *cxlds;
+ unsigned int devfn;
+ u32 hdr_flags;
+
+ pr_debug("CPER event %d for device %u:%u:%u.%u\n", ev_type,
+ device_id->segment_num, device_id->bus_num,
+ device_id->device_num, device_id->func_num);
+
+ devfn = PCI_DEVFN(device_id->device_num, device_id->func_num);
+ pdev = pci_get_domain_bus_and_slot(device_id->segment_num,
+ device_id->bus_num, devfn);
+ if (!pdev)
+ return;
+
+ guard(device)(&pdev->dev);
+ if (pdev->driver != &cxl_pci_driver)
+ return;
+
+ cxlds = pci_get_drvdata(pdev);
+ if (!cxlds)
+ return;
+
+ /* Fabricate a log type */
+ hdr_flags = get_unaligned_le24(rec->event.generic.hdr.flags);
+ log_type = FIELD_GET(CXL_EVENT_HDR_FLAGS_REC_SEVERITY, hdr_flags);
+
+ cxl_event_trace_record(cxlds->cxlmd, log_type, ev_type,
+ &uuid_null, &rec->event);
+}
+
+static int __init cxl_pci_driver_init(void)
+{
+ int rc;
+
+ rc = pci_register_driver(&cxl_pci_driver);
+ if (rc)
+ return rc;
+
+ rc = cxl_cper_register_callback(cxl_cper_event_call);
+ if (rc)
+ pci_unregister_driver(&cxl_pci_driver);
+
+ return rc;
+}
+
+static void __exit cxl_pci_driver_exit(void)
+{
+ cxl_cper_unregister_callback(cxl_cper_event_call);
+ pci_unregister_driver(&cxl_pci_driver);
+}
+
+module_init(cxl_pci_driver_init);
+module_exit(cxl_pci_driver_exit);
MODULE_LICENSE("GPL v2");
MODULE_IMPORT_NS(CXL);
--
2.44.0
If CXL is solely managed by firmware (including HDM configuration and
event processing via firmware first) it is possible to run the system
without the CXL software loaded. In this case no CXL callback will be
loaded and CXL CPER errors will not be processed at all.
In this case memory device and region (HPA) information is missing but
omitting the error completely is not friendly. Some device information
is available the event.
Trace CXL CPER events if the CXL stack is not loaded. A balance was
chosen to decode only the CPER header as this configuration is likely
rare.
Cc: Ard Biesheuvel <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>
Cc: Tony Luck <[email protected]>
Cc: Borislav Petkov <[email protected]>
Suggested-by: Dan Williams <[email protected]>
Signed-off-by: Ira Weiny <[email protected]>
---
Changes:
[djbw: remove local work task and call trace directly]
[djbw: strip down the generic trace information]
---
drivers/acpi/apei/ghes.c | 6 +++++-
include/ras/ras_event.h | 51 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+), 1 deletion(-)
diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
index cdcfdf6ebe81..20a0e622f829 100644
--- a/drivers/acpi/apei/ghes.c
+++ b/drivers/acpi/apei/ghes.c
@@ -739,8 +739,12 @@ static void cxl_cper_post_event(enum cxl_event_type event_type,
guard(spinlock_irqsave)(&cxl_cper_work_lock);
- if (!cxl_cper_work)
+ if (!cxl_cper_work) {
+ union cxl_event *evt = &rec->event;
+
+ trace_cper_cxl_event(rec, &evt->generic, event_type);
return;
+ }
wd.event_type = event_type;
memcpy(&wd.rec, rec, sizeof(wd.rec));
diff --git a/include/ras/ras_event.h b/include/ras/ras_event.h
index c011ea236e9b..c1fac66fb44a 100644
--- a/include/ras/ras_event.h
+++ b/include/ras/ras_event.h
@@ -422,6 +422,57 @@ TRACE_EVENT(memory_failure_event,
)
);
#endif /* CONFIG_MEMORY_FAILURE */
+
+#include <linux/cxl-event.h>
+#include <asm-generic/unaligned.h>
+
+TRACE_EVENT(cper_cxl_event,
+
+ TP_PROTO(struct cxl_cper_event_rec *cper_rec,
+ struct cxl_event_generic *gen_rec,
+ enum cxl_event_type event_type),
+
+ TP_ARGS(cper_rec, gen_rec, event_type),
+
+ /* packed record */
+ TP_STRUCT__entry(
+ __field(u16, segment)
+ __field(u8, bus)
+ __field(u8, device)
+ __field(u32, hdr_flags)
+ __field(u64, serial)
+ __field(u64, hdr_timestamp)
+ __field(u8, func)
+ __field(u8, hdr_length)
+ __field(u8, hdr_maint_op_class)
+ __field(u8, event_type)
+ __array(u8, data, CXL_EVENT_RECORD_DATA_LENGTH)
+ ),
+
+ TP_fast_assign(
+ __entry->event_type = event_type;
+ __entry->segment = cper_rec->hdr.device_id.segment_num;
+ __entry->bus = cper_rec->hdr.device_id.bus_num;
+ __entry->device = cper_rec->hdr.device_id.device_num;
+ __entry->func = cper_rec->hdr.device_id.func_num;
+ __entry->serial = (((u64)cper_rec->hdr.dev_serial_num.upper_dw) << 32) |
+ cper_rec->hdr.dev_serial_num.lower_dw;
+ __entry->hdr_length = gen_rec->hdr.length;
+ __entry->hdr_flags = get_unaligned_le24(gen_rec->hdr.flags);
+ __entry->hdr_timestamp = le64_to_cpu(gen_rec->hdr.timestamp);
+ __entry->hdr_maint_op_class = gen_rec->hdr.maint_op_class;
+ memcpy(__entry->data, gen_rec->data, CXL_EVENT_RECORD_DATA_LENGTH);
+ ),
+
+ TP_printk("device=%04x:%02x:%02x.%02x serial=%lld : event_type=%d time=%llu "
+ "len=%d flags=%x maint_op_class=%u : %s",
+ __entry->segment, __entry->bus, __entry->device, __entry->func,
+ __entry->serial, __entry->event_type, __entry->hdr_timestamp,
+ __entry->hdr_length, __entry->hdr_flags, __entry->hdr_maint_op_class,
+ __print_hex(__entry->data, CXL_EVENT_RECORD_DATA_LENGTH))
+
+);
+
#endif /* _TRACE_HW_EVENT_MC_H */
/* This part must be outside protection */
--
2.44.0
Ira Weiny wrote:
> BIOS can configure memory devices as firmware first. This will send CXL
> events to the firmware instead of the OS. The firmware can then inform
> the OS of these events via UEFI.
>
> UEFI v2.10 section N.2.14 defines a Common Platform Error Record (CPER)
> format for CXL Component Events. The format is mostly the same as the
> CXL Common Event Record Format. The difference lies in the use of a
> GUID as the CPER Section Type which matches the UUID defined in CXL 3.1
> Table 8-43.
>
> Currently a configuration such as this will trace a non standard event
> in the log omitting useful details of the event. In addition the CXL
> sub-system contains additional region and HPA information useful to the
> user.[0]
>
> Add GHES support to detect CXL CPER records. Add the ability for the
> CXL sub-system to register a callback to receive the events.
>
> The CXL code is required to be called from process context as it needs
> to take a device lock. The GHES code may be in interrupt context. This
> complicated the use of a callback. Dan Williams suggested the use of
> work items as an atomic way of switching between the callback execution
> and a default handler.[1]
>
> This patch adds back the functionality which was removed to fix the
> report by Dan Carpenter[2].
>
> [0]
> Link: https://lore.kernel.org/all/[email protected]/
> [1]
> Link: https://lore.kernel.org/all/[email protected]/
> [2]
> Link: https://lore.kernel.org/all/[email protected]/
Minor, but this can be reformatted a bit cleaner:
Link: http://lore.kernel.org/r/[email protected] [0]
Link: http://lore.kernel.org/r/[email protected] [1]
Link: http://lore.kernel.org/r/[email protected] [2]
>
> Cc: Ard Biesheuvel <[email protected]>
> Cc: "Rafael J. Wysocki" <[email protected]>
> Cc: Tony Luck <[email protected]>
> Cc: Borislav Petkov <[email protected]>
> Suggested-by: Dan Carpenter <[email protected]>
> Suggested-by: Dan Williams <[email protected]>
> Signed-off-by: Ira Weiny <[email protected]>
>
> ---
> Changes:
> [iweiny: clarify commit message]
> [djbw: remove local wt]
> ---
> drivers/acpi/apei/ghes.c | 124 ++++++++++++++++++++++++++++++++++++++++++++++
> include/linux/cxl-event.h | 18 +++++++
> 2 files changed, 142 insertions(+)
>
> diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
> index 512067cac170..cdcfdf6ebe81 100644
> --- a/drivers/acpi/apei/ghes.c
> +++ b/drivers/acpi/apei/ghes.c
> @@ -26,6 +26,8 @@
> #include <linux/interrupt.h>
> #include <linux/timer.h>
> #include <linux/cper.h>
> +#include <linux/cleanup.h>
> +#include <linux/cxl-event.h>
> #include <linux/platform_device.h>
> #include <linux/mutex.h>
> #include <linux/ratelimit.h>
> @@ -33,6 +35,7 @@
> #include <linux/irq_work.h>
> #include <linux/llist.h>
> #include <linux/genalloc.h>
> +#include <linux/kfifo.h>
> #include <linux/pci.h>
> #include <linux/pfn.h>
> #include <linux/aer.h>
> @@ -673,6 +676,112 @@ static void ghes_defer_non_standard_event(struct acpi_hest_generic_data *gdata,
> schedule_work(&entry->work);
> }
>
> +/* CXL Event record UUIDs are formated as GUIDs and reported in section type */
> +
> +/*
> + * General Media Event Record
> + * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43
> + */
> +#define CPER_SEC_CXL_GEN_MEDIA_GUID \
> + GUID_INIT(0xfbcd0a77, 0xc260, 0x417f, \
> + 0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6)
> +
> +/*
> + * DRAM Event Record
> + * CXL rev 3.0 section 8.2.9.2.1.2; Table 8-44
> + */
> +#define CPER_SEC_CXL_DRAM_GUID \
> + GUID_INIT(0x601dcbb3, 0x9c06, 0x4eab, \
> + 0xb8, 0xaf, 0x4e, 0x9b, 0xfb, 0x5c, 0x96, 0x24)
> +
> +/*
> + * Memory Module Event Record
> + * CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45
> + */
> +#define CPER_SEC_CXL_MEM_MODULE_GUID \
> + GUID_INIT(0xfe927475, 0xdd59, 0x4339, \
> + 0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74)
> +
> +struct cxl_cper_work_data {
> + enum cxl_event_type event_type;
> + struct cxl_cper_event_rec rec;
> +};
> +
> +DEFINE_KFIFO(cxl_cper_fifo, struct cxl_cper_work_data, 32);
Any comment on where that "32" comes from?
> +static DEFINE_SPINLOCK(cxl_cper_work_lock);
Needs a comment on what it is specifically protecting.
> +static cxl_cper_callback cper_callback;
> +static void cxl_cper_cb_fn(struct work_struct *work)
> +{
> + struct cxl_cper_work_data wd;
> +
> + while (kfifo_get(&cxl_cper_fifo, &wd))
> + cper_callback(wd.event_type, &wd.rec);
> +}
> +static DECLARE_WORK(cxl_cb_work, cxl_cper_cb_fn);
> +struct work_struct *cxl_cper_work = NULL;
Initializing global data to NULL is redundant, however this feels like
one too many dynamic things registered.
cxl_cper_work and cper_callback are dynamic, but from the GHES
perspective all it cares about is checking if work is registered and if
so put the data in the kfifo and trigger that work func.
It need not care about what happens after the work is queued. So, lets
just have the CXL driver register its own cxl_cper_work instance and
skip defining one locally here. Export cxl_cper_fifo for the driver to
optionally reference.
> +
> +static void cxl_cper_post_event(enum cxl_event_type event_type,
> + struct cxl_cper_event_rec *rec)
> +{
> + struct cxl_cper_work_data wd;
> +
> + if (rec->hdr.length <= sizeof(rec->hdr) ||
> + rec->hdr.length > sizeof(*rec)) {
> + pr_err(FW_WARN "CXL CPER Invalid section length (%u)\n",
> + rec->hdr.length);
> + return;
> + }
> +
> + if (!(rec->hdr.validation_bits & CPER_CXL_COMP_EVENT_LOG_VALID)) {
> + pr_err(FW_WARN "CXL CPER invalid event\n");
> + return;
> + }
> +
> + guard(spinlock_irqsave)(&cxl_cper_work_lock);
> +
> + if (!cxl_cper_work)
> + return;
> +
> + wd.event_type = event_type;
> + memcpy(&wd.rec, rec, sizeof(wd.rec));
> +
> + if (!kfifo_put(&cxl_cper_fifo, wd)) {
> + pr_err_ratelimited("CXL CPER kfifo overflow\n");
> + return;
> + }
> +
> + schedule_work(cxl_cper_work);
> +}
> +
> +int cxl_cper_register_callback(cxl_cper_callback callback)
> +{
> + if (cper_callback)
> + return -EINVAL;
> +
> + guard(spinlock)(&cxl_cper_work_lock);
> + cper_callback = callback;
> + cxl_cper_work = &cxl_cb_work;
Per above this would just become cxl_cper_register_work(), and then the
lock makes more sense as a cxl_cper_register_lock.
> + return 0;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_cper_register_callback, CXL);
> +
> +int cxl_cper_unregister_callback(cxl_cper_callback callback)
> +{
> + if (callback != cper_callback)
> + return -EINVAL;
> +
> + /* Avoid guard() because cancel_work_sync() can sleep */
> + spin_lock(&cxl_cper_work_lock);
> + cxl_cper_work = NULL;
> + spin_unlock(&cxl_cper_work_lock);
> +
> + cancel_work_sync(&cxl_cb_work);
Also per above, moving the responsibility of cancel_work_sync() to
caller also brings guard() back in play here.
Ira Weiny wrote:
> If CXL is solely managed by firmware (including HDM configuration and
> event processing via firmware first) it is possible to run the system
> without the CXL software loaded. In this case no CXL callback will be
> loaded and CXL CPER errors will not be processed at all.
>
> In this case memory device and region (HPA) information is missing but
> omitting the error completely is not friendly. Some device information
> is available the event.
>
> Trace CXL CPER events if the CXL stack is not loaded. A balance was
> chosen to decode only the CPER header as this configuration is likely
> rare.
I think the justification for this is weak and it pollutes the user ABI.
What environment cares about CXL RAS without the CXL driver? If such a
use case ever pops up it is trivial to revive this otherwise saves
carrying this and all the minor maintenance overhead it causes.
Dan Williams wrote:
> Ira Weiny wrote:
[snip]
> >
> > [0]
> > Link: https://lore.kernel.org/all/[email protected]/
> > [1]
> > Link: https://lore.kernel.org/all/[email protected]/
> > [2]
> > Link: https://lore.kernel.org/all/[email protected]/
>
> Minor, but this can be reformatted a bit cleaner:
>
> Link: http://lore.kernel.org/r/[email protected] [0]
> Link: http://lore.kernel.org/r/[email protected] [1]
> Link: http://lore.kernel.org/r/[email protected] [2]
>
Sure.
[snip]
> > +/*
> > + * Memory Module Event Record
> > + * CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45
> > + */
> > +#define CPER_SEC_CXL_MEM_MODULE_GUID \
> > + GUID_INIT(0xfe927475, 0xdd59, 0x4339, \
> > + 0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74)
> > +
> > +struct cxl_cper_work_data {
> > + enum cxl_event_type event_type;
> > + struct cxl_cper_event_rec rec;
> > +};
> > +
> > +DEFINE_KFIFO(cxl_cper_fifo, struct cxl_cper_work_data, 32);
>
> Any comment on where that "32" comes from?
If anyone has any better queue depth I'm open. It is just a guess.
>
> > +static DEFINE_SPINLOCK(cxl_cper_work_lock);
>
> Needs a comment on what it is specifically protecting.
Ok.
>
> > +static cxl_cper_callback cper_callback;
> > +static void cxl_cper_cb_fn(struct work_struct *work)
> > +{
> > + struct cxl_cper_work_data wd;
> > +
> > + while (kfifo_get(&cxl_cper_fifo, &wd))
> > + cper_callback(wd.event_type, &wd.rec);
> > +}
> > +static DECLARE_WORK(cxl_cb_work, cxl_cper_cb_fn);
> > +struct work_struct *cxl_cper_work = NULL;
>
> Initializing global data to NULL is redundant, however this feels like
> one too many dynamic things registered.
>
> cxl_cper_work and cper_callback are dynamic, but from the GHES
> perspective all it cares about is checking if work is registered and if
> so put the data in the kfifo and trigger that work func.
>
> It need not care about what happens after the work is queued. So, lets
> just have the CXL driver register its own cxl_cper_work instance and
> skip defining one locally here. Export cxl_cper_fifo for the driver to
> optionally reference.
Ok I thought we had decided not to do that. If I recall exporting the
fifo had some difficulties but it may have been my unfamiliarity with it.
Ira
[snip]
Dan Williams wrote:
> Ira Weiny wrote:
> > If CXL is solely managed by firmware (including HDM configuration and
> > event processing via firmware first) it is possible to run the system
> > without the CXL software loaded. In this case no CXL callback will be
> > loaded and CXL CPER errors will not be processed at all.
> >
> > In this case memory device and region (HPA) information is missing but
> > omitting the error completely is not friendly. Some device information
> > is available the event.
> >
> > Trace CXL CPER events if the CXL stack is not loaded. A balance was
> > chosen to decode only the CPER header as this configuration is likely
> > rare.
>
> I think the justification for this is weak and it pollutes the user ABI.
Do you want me to drop this patch or slim the tracepoint down even
further?
Looking at this again I feel like
54ce1927eb78 ("cxl/cper: Fix errant CPER prints for CXL events")
Was a mistake. PCI AER errors are both printed in
cper_estatus_print_section() and again printed/traced in pci_print_aer()
Why should CXL be any different?
Ira
Ira Weiny wrote:
> Dan Williams wrote:
> > Ira Weiny wrote:
>
[snip]
> >
> > > +static cxl_cper_callback cper_callback;
> > > +static void cxl_cper_cb_fn(struct work_struct *work)
> > > +{
> > > + struct cxl_cper_work_data wd;
> > > +
> > > + while (kfifo_get(&cxl_cper_fifo, &wd))
> > > + cper_callback(wd.event_type, &wd.rec);
> > > +}
> > > +static DECLARE_WORK(cxl_cb_work, cxl_cper_cb_fn);
> > > +struct work_struct *cxl_cper_work = NULL;
> >
> > Initializing global data to NULL is redundant, however this feels like
> > one too many dynamic things registered.
> >
> > cxl_cper_work and cper_callback are dynamic, but from the GHES
> > perspective all it cares about is checking if work is registered and if
> > so put the data in the kfifo and trigger that work func.
> >
> > It need not care about what happens after the work is queued. So, lets
> > just have the CXL driver register its own cxl_cper_work instance and
> > skip defining one locally here. Export cxl_cper_fifo for the driver to
> > optionally reference.
>
> Ok I thought we had decided not to do that. If I recall exporting the
> fifo had some difficulties but it may have been my unfamiliarity with it.
Looking more closely I see that AER does something similar but it uses
constructs which I thought we should avoid. For example all of
ghes_handle_aer is predicated on CONFIG_ACPI_APEI_PCIEAER.[0]
What I have is a bit complicated, with the extra pointer. Keeping the
kfifo within the ghes module is much easier to follow IMO.
Let me know if you really want me to pursue this separation but my late
night gut feeling is this is going to be more trouble than it is worth to
keep the separation amongst the modules.
Ira
[0]
static void ghes_handle_aer(struct acpi_hest_generic_data *gdata)
{
#ifdef CONFIG_ACPI_APEI_PCIEAER
...
#endif
}