Platforms with large BERT table data can trigger soft lockup errors
while attempting to print the entire BERT table data to the console at
boot:
watchdog: BUG: soft lockup - CPU#160 stuck for 23s! [swapper/0:1]
Observed on Ampere Altra systems with a single BERT record of ~250KB.
The original bert driver appears to have assumed relatively small table
data. Since it is impractical to reassemble large table data from
interwoven console messages, and the table data is available in
/sys/firmware/acpi/tables/data/BERT
limit the size for tables printed to the console to 1024 (for no reason
other than it seemed like a good place to kick off the discussion, would
appreciate feedback from existing users in terms of what size would
maintain their current usage model).
Alternatively, we could make printing a CONFIG option, use the
bert_disable boot arg (or something similar), or use a debug log level.
However, all those solutions require extra steps or change the existing
behavior for small table data. Limiting the size preserves existing
behavior on existing platforms with small table data, and eliminates the
soft lockups for platforms with large table data, while still making it
available.
Cc: "Rafael J. Wysocki" <[email protected]>
Cc: Len Brown <[email protected]>
Cc: James Morse <[email protected]>
Cc: Tony Luck <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Doug Rady <[email protected]>
Signed-off-by: Darren Hart <[email protected]>
---
drivers/acpi/apei/bert.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/drivers/acpi/apei/bert.c b/drivers/acpi/apei/bert.c
index 19e50fcbf4d6..ad8ab3f12cf3 100644
--- a/drivers/acpi/apei/bert.c
+++ b/drivers/acpi/apei/bert.c
@@ -29,6 +29,7 @@
#undef pr_fmt
#define pr_fmt(fmt) "BERT: " fmt
+#define ACPI_BERT_PRINT_MAX_LEN 1024
static int bert_disable;
@@ -58,8 +59,11 @@ static void __init bert_print_all(struct acpi_bert_region *region,
}
pr_info_once("Error records from previous boot:\n");
-
- cper_estatus_print(KERN_INFO HW_ERR, estatus);
+ if (region_len < ACPI_BERT_PRINT_MAX_LEN)
+ cper_estatus_print(KERN_INFO HW_ERR, estatus);
+ else
+ pr_info_once("Max print length exceeded, table data is available at:\n"
+ "/sys/firmware/acpi/tables/data/BERT");
/*
* Because the boot error source is "one-time polled" type,
--
2.31.1
On Tue, Mar 8, 2022 at 7:51 PM Darren Hart
<[email protected]> wrote:
>
> Platforms with large BERT table data can trigger soft lockup errors
> while attempting to print the entire BERT table data to the console at
> boot:
>
> watchdog: BUG: soft lockup - CPU#160 stuck for 23s! [swapper/0:1]
>
> Observed on Ampere Altra systems with a single BERT record of ~250KB.
>
> The original bert driver appears to have assumed relatively small table
> data. Since it is impractical to reassemble large table data from
> interwoven console messages, and the table data is available in
>
> /sys/firmware/acpi/tables/data/BERT
>
> limit the size for tables printed to the console to 1024 (for no reason
> other than it seemed like a good place to kick off the discussion, would
> appreciate feedback from existing users in terms of what size would
> maintain their current usage model).
>
> Alternatively, we could make printing a CONFIG option, use the
> bert_disable boot arg (or something similar), or use a debug log level.
> However, all those solutions require extra steps or change the existing
> behavior for small table data. Limiting the size preserves existing
> behavior on existing platforms with small table data, and eliminates the
> soft lockups for platforms with large table data, while still making it
> available.
>
> Cc: "Rafael J. Wysocki" <[email protected]>
> Cc: Len Brown <[email protected]>
> Cc: James Morse <[email protected]>
> Cc: Tony Luck <[email protected]>
> Cc: Borislav Petkov <[email protected]>
> Cc: Doug Rady <[email protected]>
> Signed-off-by: Darren Hart <[email protected]>
Not that I have a particularly strong opinion here, but this looks
reasonable to me, so I've queued it up for 5.18.
APEI reviewers, please chime in if you disagree with the above.
> ---
> drivers/acpi/apei/bert.c | 8 ++++++--
> 1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/acpi/apei/bert.c b/drivers/acpi/apei/bert.c
> index 19e50fcbf4d6..ad8ab3f12cf3 100644
> --- a/drivers/acpi/apei/bert.c
> +++ b/drivers/acpi/apei/bert.c
> @@ -29,6 +29,7 @@
>
> #undef pr_fmt
> #define pr_fmt(fmt) "BERT: " fmt
> +#define ACPI_BERT_PRINT_MAX_LEN 1024
>
> static int bert_disable;
>
> @@ -58,8 +59,11 @@ static void __init bert_print_all(struct acpi_bert_region *region,
> }
>
> pr_info_once("Error records from previous boot:\n");
> -
> - cper_estatus_print(KERN_INFO HW_ERR, estatus);
> + if (region_len < ACPI_BERT_PRINT_MAX_LEN)
> + cper_estatus_print(KERN_INFO HW_ERR, estatus);
> + else
> + pr_info_once("Max print length exceeded, table data is available at:\n"
> + "/sys/firmware/acpi/tables/data/BERT");
>
> /*
> * Because the boot error source is "one-time polled" type,
> --
> 2.31.1
>
On Wed, Mar 09, 2022 at 07:42:26PM +0100, Rafael J. Wysocki wrote:
> On Tue, Mar 8, 2022 at 7:51 PM Darren Hart
> Not that I have a particularly strong opinion here, but this looks
> reasonable to me, so I've queued it up for 5.18.
>
> APEI reviewers, please chime in if you disagree with the above.
It looked reasonable to me when I skimmed it in March. But the
reality check now needs cashing because some validation team
here is complaining that they don't see any errors printed from
their BERT tests. :-(
So I looked again. This test inside the loop seems bogus:
if (region_len < ACPI_BERT_PRINT_MAX_LEN) {
because "region_len" isn't updated inside the loop. If it is too big
then it will prevent Linux from printing any/all of the records in the
BERT table (and the test could have been done before the loop).
Maybe below patch is better? It avoids printing individual CPER
records that are too large (checking estatus_len instead of region_len).
I also added a limit to how many records to print (I randomly picked "5" as
the limit ... the specific failing test only want to print one).
-Tony
[I will write up a proper commit message and add a Signed-off-by if
this looks to be a reasonable direction]
diff --git a/drivers/acpi/apei/bert.c b/drivers/acpi/apei/bert.c
index 598fd19b65fa..4e894a728c02 100644
--- a/drivers/acpi/apei/bert.c
+++ b/drivers/acpi/apei/bert.c
@@ -29,6 +29,8 @@
#undef pr_fmt
#define pr_fmt(fmt) "BERT: " fmt
+
+#define ACPI_BERT_PRINT_MAX_RECORDS 5
#define ACPI_BERT_PRINT_MAX_LEN 1024
static int bert_disable;
@@ -39,6 +41,7 @@ static void __init bert_print_all(struct acpi_bert_region *region,
struct acpi_hest_generic_status *estatus =
(struct acpi_hest_generic_status *)region;
int remain = region_len;
+ int ncper = 0, skipped = 0;
u32 estatus_len;
while (remain >= sizeof(struct acpi_bert_region)) {
@@ -46,24 +49,23 @@ static void __init bert_print_all(struct acpi_bert_region *region,
if (remain < estatus_len) {
pr_err(FW_BUG "Truncated status block (length: %u).\n",
estatus_len);
- return;
+ break;
}
/* No more error records. */
if (!estatus->block_status)
- return;
+ break;
if (cper_estatus_check(estatus)) {
pr_err(FW_BUG "Invalid error record.\n");
- return;
+ break;
}
pr_info_once("Error records from previous boot:\n");
- if (region_len < ACPI_BERT_PRINT_MAX_LEN)
+ if (ncper++ < ACPI_BERT_PRINT_MAX_RECORDS && estatus_len < ACPI_BERT_PRINT_MAX_LEN)
cper_estatus_print(KERN_INFO HW_ERR, estatus);
else
- pr_info_once("Max print length exceeded, table data is available at:\n"
- "/sys/firmware/acpi/tables/data/BERT");
+ skipped++;
/*
* Because the boot error source is "one-time polled" type,
@@ -75,6 +77,9 @@ static void __init bert_print_all(struct acpi_bert_region *region,
estatus = (void *)estatus + estatus_len;
remain -= estatus_len;
}
+ if (skipped)
+ pr_info("Skipped %d error records, full table data is available at:\n"
+ "/sys/firmware/acpi/tables/data/BERT", skipped);
}
static int __init setup_bert_disable(char *str)
The fix in commit 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT
table data") does not work as intended on systems where the BIOS has a
fixed size block of memory for the BERT table, relying on s/w to quit
when it finds a record with estatus->block_status == 0. On these systems
all errors are suppressed because the check:
if (region_len < ACPI_BERT_PRINT_MAX_LEN)
always fails.
New scheme skips individual CPER records that are too large, and also
limits the total number of records that will be printed to 5.
Fixes: 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT table data")
Cc: [email protected]
Signed-off-by: Tony Luck <[email protected]>
---
Now in PATCH format with a real commit comment. This version fixes
the issues seen by Intel's validation team.
drivers/acpi/apei/bert.c | 31 +++++++++++++++++++++++--------
1 file changed, 23 insertions(+), 8 deletions(-)
diff --git a/drivers/acpi/apei/bert.c b/drivers/acpi/apei/bert.c
index 598fd19b65fa..45973aa6e06d 100644
--- a/drivers/acpi/apei/bert.c
+++ b/drivers/acpi/apei/bert.c
@@ -29,16 +29,26 @@
#undef pr_fmt
#define pr_fmt(fmt) "BERT: " fmt
+
+#define ACPI_BERT_PRINT_MAX_RECORDS 5
#define ACPI_BERT_PRINT_MAX_LEN 1024
static int bert_disable;
+/*
+ * Print "all" the error records in the BERT table, but avoid huge spam to
+ * the console if the BIOS included oversize records, or too many records.
+ * Skipping some records here does not lose anything because the full
+ * data is available to user tools in:
+ * /sys/firmware/acpi/tables/data/BERT
+ */
static void __init bert_print_all(struct acpi_bert_region *region,
unsigned int region_len)
{
struct acpi_hest_generic_status *estatus =
(struct acpi_hest_generic_status *)region;
int remain = region_len;
+ int printed = 0, skipped = 0;
u32 estatus_len;
while (remain >= sizeof(struct acpi_bert_region)) {
@@ -46,24 +56,26 @@ static void __init bert_print_all(struct acpi_bert_region *region,
if (remain < estatus_len) {
pr_err(FW_BUG "Truncated status block (length: %u).\n",
estatus_len);
- return;
+ break;
}
/* No more error records. */
if (!estatus->block_status)
- return;
+ break;
if (cper_estatus_check(estatus)) {
pr_err(FW_BUG "Invalid error record.\n");
- return;
+ break;
}
- pr_info_once("Error records from previous boot:\n");
- if (region_len < ACPI_BERT_PRINT_MAX_LEN)
+ if (estatus_len < ACPI_BERT_PRINT_MAX_LEN &&
+ printed < ACPI_BERT_PRINT_MAX_RECORDS) {
+ pr_info_once("Error records from previous boot:\n");
cper_estatus_print(KERN_INFO HW_ERR, estatus);
- else
- pr_info_once("Max print length exceeded, table data is available at:\n"
- "/sys/firmware/acpi/tables/data/BERT");
+ printed++;
+ } else {
+ skipped++;
+ }
/*
* Because the boot error source is "one-time polled" type,
@@ -75,6 +87,9 @@ static void __init bert_print_all(struct acpi_bert_region *region,
estatus = (void *)estatus + estatus_len;
remain -= estatus_len;
}
+
+ if (skipped)
+ pr_info(HW_ERR "Skipped %d error records\n", skipped);
}
static int __init setup_bert_disable(char *str)
--
2.35.3
On Wed, Jun 22, 2022 at 10:09:06AM -0700, Tony Luck wrote:
> The fix in commit 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT
> table data") does not work as intended on systems where the BIOS has a
> fixed size block of memory for the BERT table, relying on s/w to quit
> when it finds a record with estatus->block_status == 0. On these systems
> all errors are suppressed because the check:
>
> if (region_len < ACPI_BERT_PRINT_MAX_LEN)
>
> always fails.
>
> New scheme skips individual CPER records that are too large, and also
> limits the total number of records that will be printed to 5.
Apologies for the delay.
This seems like a reasonable approach. Working to confirm new behavior on Ampere
Altra systems (specifically how region_len and estatus_len are related).
--
Darren Hart
Ampere Computing / OS and Kernel
On Wed, Jun 22, 2022 at 7:09 PM Tony Luck <[email protected]> wrote:
>
> The fix in commit 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT
> table data") does not work as intended on systems where the BIOS has a
> fixed size block of memory for the BERT table, relying on s/w to quit
> when it finds a record with estatus->block_status == 0. On these systems
> all errors are suppressed because the check:
>
> if (region_len < ACPI_BERT_PRINT_MAX_LEN)
>
> always fails.
>
> New scheme skips individual CPER records that are too large, and also
> limits the total number of records that will be printed to 5.
>
> Fixes: 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT table data")
> Cc: [email protected]
> Signed-off-by: Tony Luck <[email protected]>
> ---
>
> Now in PATCH format with a real commit comment. This version fixes
> the issues seen by Intel's validation team.
>
> drivers/acpi/apei/bert.c | 31 +++++++++++++++++++++++--------
> 1 file changed, 23 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/acpi/apei/bert.c b/drivers/acpi/apei/bert.c
> index 598fd19b65fa..45973aa6e06d 100644
> --- a/drivers/acpi/apei/bert.c
> +++ b/drivers/acpi/apei/bert.c
> @@ -29,16 +29,26 @@
>
> #undef pr_fmt
> #define pr_fmt(fmt) "BERT: " fmt
> +
> +#define ACPI_BERT_PRINT_MAX_RECORDS 5
> #define ACPI_BERT_PRINT_MAX_LEN 1024
>
> static int bert_disable;
>
> +/*
> + * Print "all" the error records in the BERT table, but avoid huge spam to
> + * the console if the BIOS included oversize records, or too many records.
> + * Skipping some records here does not lose anything because the full
> + * data is available to user tools in:
> + * /sys/firmware/acpi/tables/data/BERT
> + */
> static void __init bert_print_all(struct acpi_bert_region *region,
> unsigned int region_len)
> {
> struct acpi_hest_generic_status *estatus =
> (struct acpi_hest_generic_status *)region;
> int remain = region_len;
> + int printed = 0, skipped = 0;
> u32 estatus_len;
>
> while (remain >= sizeof(struct acpi_bert_region)) {
> @@ -46,24 +56,26 @@ static void __init bert_print_all(struct acpi_bert_region *region,
> if (remain < estatus_len) {
> pr_err(FW_BUG "Truncated status block (length: %u).\n",
> estatus_len);
> - return;
> + break;
> }
>
> /* No more error records. */
> if (!estatus->block_status)
> - return;
> + break;
>
> if (cper_estatus_check(estatus)) {
> pr_err(FW_BUG "Invalid error record.\n");
> - return;
> + break;
> }
>
> - pr_info_once("Error records from previous boot:\n");
> - if (region_len < ACPI_BERT_PRINT_MAX_LEN)
> + if (estatus_len < ACPI_BERT_PRINT_MAX_LEN &&
> + printed < ACPI_BERT_PRINT_MAX_RECORDS) {
> + pr_info_once("Error records from previous boot:\n");
> cper_estatus_print(KERN_INFO HW_ERR, estatus);
> - else
> - pr_info_once("Max print length exceeded, table data is available at:\n"
> - "/sys/firmware/acpi/tables/data/BERT");
> + printed++;
> + } else {
> + skipped++;
> + }
>
> /*
> * Because the boot error source is "one-time polled" type,
> @@ -75,6 +87,9 @@ static void __init bert_print_all(struct acpi_bert_region *region,
> estatus = (void *)estatus + estatus_len;
> remain -= estatus_len;
> }
> +
> + if (skipped)
> + pr_info(HW_ERR "Skipped %d error records\n", skipped);
> }
>
> static int __init setup_bert_disable(char *str)
> --
Applied as 5.20 material, thanks!