In general it isn't very useful to dump the kernel stack in the panic
from a fatal machine check. The problem is almost always hardware
related, so knowing how the kernel got to the routine that triggered the
machine check isn't useful.
But Linux now has the capability to recover from most user mode and a
few kernel mode memory related machine checks. Validation folks are
testing that out and occasionally bring a kernel log like this to me:
[69608.047771] mce: [Hardware Error]: Machine check: Data load in unrecoverable area of kernel
[69608.021729] mce: [Hardware Error]: TSC 7874eb580177 ADDR 43bb84bd00 MISC 86 PPIN 9f061818e1a92082
[69608.047773] Kernel panic - not syncing: Fatal local machine check
[69608.021720] mce: [Hardware Error]: RIP 10:<ffffffff8b767517> {copy_page+0x7/0x10}
All I can tell them is that Linux was copying a page and hit poison in
the source of the copy. But there are lots of reasons why Linux may be
copying a page. A stack trace would help figure out if:
1) the test was bad and just injected an error into the wrong location
2) an injected error sat around in memory and was later consumed
Case 2 will help identify places where Linux might use a "safe" copy
function that returns an error to the caller which may attempt some sort
of recovery.
Patch 1 cleans up the Intel severity calculation by using a new severity
table entry instead of some, now dubious, code to adjust the severity
for errors in kernel context.
Patch 2 adds a new severity level that triggers printing a stack trace.
I've only updated the Intel severity calculation to use this new
severity level. I'm not sure if AMD also has situations where this would
be useful. If so, then mce_severity_amd() would need to be updated too
to return different severity for IN_KERNEL and IN_KERNEL_RECOV cases.
I've tested this out on systems that do both broadcast and local machine
checks.
Tony Luck (2):
x86/mce: Use severity table to handle uncorrected errors in kernel
x86/mce: Dump the stack for recoverable machine checks in kernel
context
arch/x86/kernel/cpu/mce/internal.h | 1 +
arch/x86/kernel/cpu/mce/core.c | 11 +++++++++--
arch/x86/kernel/cpu/mce/severity.c | 10 ++++++----
3 files changed, 16 insertions(+), 6 deletions(-)
base-commit: 521a547ced6477c54b4b0cc206000406c221b4d6
--
2.37.3
On Thu, Sep 22, 2022 at 12:51:34PM -0700, Tony Luck wrote:
> I've only updated the Intel severity calculation to use this new
> severity level. I'm not sure if AMD also has situations where this would
> be useful. If so, then mce_severity_amd() would need to be updated too
> to return different severity for IN_KERNEL and IN_KERNEL_RECOV cases.
I'd look into Yazen's direction for that...
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
On Mon, Oct 31, 2022 at 11:30:03AM +0100, Borislav Petkov wrote:
> On Thu, Sep 22, 2022 at 12:51:34PM -0700, Tony Luck wrote:
> > I've only updated the Intel severity calculation to use this new
> > severity level. I'm not sure if AMD also has situations where this would
> > be useful. If so, then mce_severity_amd() would need to be updated too
> > to return different severity for IN_KERNEL and IN_KERNEL_RECOV cases.
>
> I'd look into Yazen's direction for that...
>
Yes, I think this is something we can look into. I'm not aware of any
situations at the moment. But I'd like to start focusing more on the various
recovery paths and corner cases.
Thanks,
Yazen