2020-01-03 15:08:55

by Jan H. Schönherr

[permalink] [raw]
Subject: [PATCH v2 1/6] x86/mce: Take action on UCNA/Deferred errors again

Commit fa92c5869426 ("x86, mce: Support memory error recovery for both
UCNA and Deferred error in machine_check_poll") added handling of UCNA
and Deferred errors by adding them to the ring for SRAO errors.

Later, commit fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action
Optional errors") switched storage from the SRAO ring to the unified
pool that is still in use today. In order to only act on the intended
errors, a filter for MCE_AO_SEVERITY is used -- effectively removing
handling of UCNA/Deferred errors again.

Extend the severity filter to include UCNA/Deferred errors again.
Also, generalize the naming of the notifier from SRAO to UC to capture
the extended scope.

Note, that this change may cause a message like the following to appear,
as the same address may be reported as SRAO and as UCNA:

Memory failure: 0x5fe3284: already hardware poisoned

Technically, this is a return to previous behavior.

Fixes: fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action Optional errors")
Signed-off-by: Jan H. Schönherr <[email protected]>
---
Changes v1->v2:
- rename notifier from SRAO to UC (as requested by Tony)
- extend commit message (per remark from Tony)
- don't mention Linux versions (per remark from Boris)

There was some discussion on v1, whether the SRAO/UC notifier does
the right thing or not. While it seems to be correct as is for Intel
(per Tony), there were some concerns for AMD (per Yazen). Hence, there
is a new patch 5 in this series, which disables the notifier on AMD.
---
arch/x86/include/asm/mce.h | 2 +-
arch/x86/kernel/cpu/mce/core.c | 31 ++++++++++++++++---------------
2 files changed, 17 insertions(+), 16 deletions(-)

diff --git a/arch/x86/include/asm/mce.h b/arch/x86/include/asm/mce.h
index dc2d4b206ab7..c8ff6f6750ef 100644
--- a/arch/x86/include/asm/mce.h
+++ b/arch/x86/include/asm/mce.h
@@ -144,7 +144,7 @@ struct mce_log_buffer {

enum mce_notifier_prios {
MCE_PRIO_FIRST = INT_MAX,
- MCE_PRIO_SRAO = INT_MAX - 1,
+ MCE_PRIO_UC = INT_MAX - 1,
MCE_PRIO_EXTLOG = INT_MAX - 2,
MCE_PRIO_NFIT = INT_MAX - 3,
MCE_PRIO_EDAC = INT_MAX - 4,
diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
index 8994fe7751a4..16134ce587fd 100644
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -156,10 +156,8 @@ void mce_log(struct mce *m)
}
EXPORT_SYMBOL_GPL(mce_log);

-static struct notifier_block mce_srao_nb;
-
/*
- * We run the default notifier if we have only the SRAO, the first and the
+ * We run the default notifier if we have only the UC, the first and the
* default notifier registered. I.e., the mandatory NUM_DEFAULT_NOTIFIERS
* notifiers registered on the chain.
*/
@@ -580,26 +578,29 @@ static struct notifier_block first_nb = {
.priority = MCE_PRIO_FIRST,
};

-static int srao_decode_notifier(struct notifier_block *nb, unsigned long val,
- void *data)
+static int uc_decode_notifier(struct notifier_block *nb, unsigned long val,
+ void *data)
{
struct mce *mce = (struct mce *)data;
unsigned long pfn;

- if (!mce)
+ if (!mce || !mce_usable_address(mce))
return NOTIFY_DONE;

- if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) {
- pfn = mce->addr >> PAGE_SHIFT;
- if (!memory_failure(pfn, 0))
- set_mce_nospec(pfn);
- }
+ if (mce->severity != MCE_AO_SEVERITY &&
+ mce->severity != MCE_DEFERRED_SEVERITY)
+ return NOTIFY_DONE;
+
+ pfn = mce->addr >> PAGE_SHIFT;
+ if (!memory_failure(pfn, 0))
+ set_mce_nospec(pfn);

return NOTIFY_OK;
}
-static struct notifier_block mce_srao_nb = {
- .notifier_call = srao_decode_notifier,
- .priority = MCE_PRIO_SRAO,
+
+static struct notifier_block mce_uc_nb = {
+ .notifier_call = uc_decode_notifier,
+ .priority = MCE_PRIO_UC,
};

static int mce_default_notifier(struct notifier_block *nb, unsigned long val,
@@ -1970,7 +1971,7 @@ int __init mcheck_init(void)
{
mcheck_intel_therm_init();
mce_register_decode_chain(&first_nb);
- mce_register_decode_chain(&mce_srao_nb);
+ mce_register_decode_chain(&mce_uc_nb);
mce_register_decode_chain(&mce_default_nb);
mcheck_vendor_init_severity();

--
2.22.0.3.gb49bb57c8208.dirty


2020-01-10 09:52:33

by Borislav Petkov

[permalink] [raw]
Subject: Re: [PATCH v2 1/6] x86/mce: Take action on UCNA/Deferred errors again

On Fri, Jan 03, 2020 at 04:07:17PM +0100, Jan H. Schönherr wrote:
> Commit fa92c5869426 ("x86, mce: Support memory error recovery for both
> UCNA and Deferred error in machine_check_poll") added handling of UCNA
> and Deferred errors by adding them to the ring for SRAO errors.
>
> Later, commit fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action
> Optional errors") switched storage from the SRAO ring to the unified
> pool that is still in use today. In order to only act on the intended
> errors, a filter for MCE_AO_SEVERITY is used -- effectively removing
> handling of UCNA/Deferred errors again.
>
> Extend the severity filter to include UCNA/Deferred errors again.
> Also, generalize the naming of the notifier from SRAO to UC to capture
> the extended scope.
>
> Note, that this change may cause a message like the following to appear,
> as the same address may be reported as SRAO and as UCNA:
>
> Memory failure: 0x5fe3284: already hardware poisoned
>
> Technically, this is a return to previous behavior.
>
> Fixes: fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action Optional errors")
> Signed-off-by: Jan H. Schönherr <[email protected]>

Tony, ACK?

Also, do you want it in stable@ so that it gets backported?

> diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
> index 8994fe7751a4..16134ce587fd 100644
> --- a/arch/x86/kernel/cpu/mce/core.c
> +++ b/arch/x86/kernel/cpu/mce/core.c
> @@ -156,10 +156,8 @@ void mce_log(struct mce *m)
> }
> EXPORT_SYMBOL_GPL(mce_log);
>
> -static struct notifier_block mce_srao_nb;
> -
> /*
> - * We run the default notifier if we have only the SRAO, the first and the
> + * We run the default notifier if we have only the UC, the first and the
> * default notifier registered. I.e., the mandatory NUM_DEFAULT_NOTIFIERS
> * notifiers registered on the chain.
> */
> @@ -580,26 +578,29 @@ static struct notifier_block first_nb = {
> .priority = MCE_PRIO_FIRST,
> };
>
> -static int srao_decode_notifier(struct notifier_block *nb, unsigned long val,
> - void *data)
> +static int uc_decode_notifier(struct notifier_block *nb, unsigned long val,
> + void *data)
> {
> struct mce *mce = (struct mce *)data;
> unsigned long pfn;
>
> - if (!mce)
> + if (!mce || !mce_usable_address(mce))
> return NOTIFY_DONE;
>
> - if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) {
> - pfn = mce->addr >> PAGE_SHIFT;
> - if (!memory_failure(pfn, 0))
> - set_mce_nospec(pfn);
> - }
> + if (mce->severity != MCE_AO_SEVERITY &&
> + mce->severity != MCE_DEFERRED_SEVERITY)
> + return NOTIFY_DONE;
> +
> + pfn = mce->addr >> PAGE_SHIFT;
> + if (!memory_failure(pfn, 0))
> + set_mce_nospec(pfn);

I'm wondering if in the memory_failure error() case, we should hand it
down to the remaining notifiers.

Which also begs the question in light of this clumsy notifier counting:

How about we have the default notifier *unconditionally* print the MCE?
I.e., if the error has reached it, it would print it. If not and some
other notifier consumed it, it will get handled differently.

This way we won't need any special counting of notifiers and special
reg/unreg of notifiers etc.

IOW, the logic would be:

If something consumes the error, then it doesn't get printed. Notifier
does NOTIFY_STOP.

If nothing consumes it or something looks at it and decides that it
should still get printed, then the last catch-all notifier callback does
that.

Thoughts?

--
Regards/Gruss,
Boris.

https://people.kernel.org/tglx/notes-about-netiquette

2020-01-10 18:46:47

by Tony Luck

[permalink] [raw]
Subject: Re: [PATCH v2 1/6] x86/mce: Take action on UCNA/Deferred errors again

On Fri, Jan 10, 2020 at 10:50:04AM +0100, Borislav Petkov wrote:
> On Fri, Jan 03, 2020 at 04:07:17PM +0100, Jan H. Sch?nherr wrote:
> > Commit fa92c5869426 ("x86, mce: Support memory error recovery for both
> > UCNA and Deferred error in machine_check_poll") added handling of UCNA
> > and Deferred errors by adding them to the ring for SRAO errors.
> >
> > Later, commit fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action
> > Optional errors") switched storage from the SRAO ring to the unified
> > pool that is still in use today. In order to only act on the intended
> > errors, a filter for MCE_AO_SEVERITY is used -- effectively removing
> > handling of UCNA/Deferred errors again.
> >
> > Extend the severity filter to include UCNA/Deferred errors again.
> > Also, generalize the naming of the notifier from SRAO to UC to capture
> > the extended scope.
> >
> > Note, that this change may cause a message like the following to appear,
> > as the same address may be reported as SRAO and as UCNA:
> >
> > Memory failure: 0x5fe3284: already hardware poisoned
> >
> > Technically, this is a return to previous behavior.
> >
> > Fixes: fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action Optional errors")
> > Signed-off-by: Jan H. Sch?nherr <[email protected]>
>
> Tony, ACK?

Acked-by: Tony Luck <[email protected]>

> Also, do you want it in stable@ so that it gets backported?

That's a tricky question. We have changing behavior (UCNA pages offlined,
then a few kernel versions stopped doing this, now we are going to start
doing it again. But is it really a _BUG_ that needs backporting to stable?
I'm leaning towards "no it isn't". But could perhaps be convinced to change
my mind if somebody has a good reason for wanting it there.

Is there something to put in the tags to stop this being autoselected
for backport because it has a Fixes: tag?

> I'm wondering if in the memory_failure error() case, we should hand it
> down to the remaining notifiers.
>
> Which also begs the question in light of this clumsy notifier counting:
>
> How about we have the default notifier *unconditionally* print the MCE?
> I.e., if the error has reached it, it would print it. If not and some
> other notifier consumed it, it will get handled differently.
>
> This way we won't need any special counting of notifiers and special
> reg/unreg of notifiers etc.
>
> IOW, the logic would be:
>
> If something consumes the error, then it doesn't get printed. Notifier
> does NOTIFY_STOP.
>
> If nothing consumes it or something looks at it and decides that it
> should still get printed, then the last catch-all notifier callback does
> that.

I totally agree that counting notifiers is clumsy. Also less than
ideal is the concept that any notifier on the chain can declare:
"I fixed it"
and prevent any other notifiers from even seeing it. Well the concept
is good, but it is overused.

I think we may do better with a field in the "struct mce" that is being
passed to each where notifiers can wiggle some bits (semantics to be
defined later) which can tell subsequent notifiers what sort of actions
have been taken.
E.g. the SRAO/UCNA notifier can say "I took this page offline"
the dev_mcelog one can say "I think I handed to a process that has /dev/mcelog open"
EDAC drivers can say "I decoded the address and printed something"
CEC can say: "I silently counted this corrected error", or "error exceeded
threshold and I took the page offline".

The default notifier can print to console if nobody set a bit to say
that the error had been somehow logged.

-Tony

2020-01-11 13:08:18

by Borislav Petkov

[permalink] [raw]
Subject: Re: [PATCH v2 1/6] x86/mce: Take action on UCNA/Deferred errors again

On Fri, Jan 10, 2020 at 10:45:33AM -0800, Luck, Tony wrote:
> That's a tricky question. We have changing behavior (UCNA pages offlined,
> then a few kernel versions stopped doing this, now we are going to start
> doing it again. But is it really a _BUG_ that needs backporting to stable?
> I'm leaning towards "no it isn't".

I think the same, so let's not. We can always backport it later, if it
turns out that it is needed.

> But could perhaps be convinced to change my mind if somebody has a
> good reason for wanting it there.

Yah.

> Is there something to put in the tags to stop this being autoselected
> for backport because it has a Fixes: tag?

I don't think so.

What we could do is write the "Fixes:" tag in free text in the commit
message so that tools don't pick up on it.

I'll reply to the other thing below in a separate mail.

Thx.

--
Regards/Gruss,
Boris.

https://people.kernel.org/tglx/notes-about-netiquette

2020-01-11 13:20:57

by Borislav Petkov

[permalink] [raw]
Subject: Re: [PATCH v2 1/6] x86/mce: Take action on UCNA/Deferred errors again

On Fri, Jan 10, 2020 at 10:45:33AM -0800, Luck, Tony wrote:
> I totally agree that counting notifiers is clumsy. Also less than
> ideal is the concept that any notifier on the chain can declare:
> "I fixed it"
> and prevent any other notifiers from even seeing it. Well the concept
> is good, but it is overused.

But why can't we use it?

Don't get me wrong: I'm simply following my KISS approach to do the
simplest scheme required. So, do you see a use case where the whole
error handling chain would need more sophisticated handling?

> I think we may do better with a field in the "struct mce" that is being
> passed to each where notifiers can wiggle some bits (semantics to be
> defined later) which can tell subsequent notifiers what sort of actions
> have been taken.
> E.g. the SRAO/UCNA notifier can say "I took this page offline"
> the dev_mcelog one can say "I think I handed to a process that has /dev/mcelog open"
> EDAC drivers can say "I decoded the address and printed something"
> CEC can say: "I silently counted this corrected error", or "error exceeded
> threshold and I took the page offline".
>
> The default notifier can print to console if nobody set a bit to say
> that the error had been somehow logged.

That idea is good and I'll gladly take patches for it so if you wanna do
it...

Thx.

--
Regards/Gruss,
Boris.

https://people.kernel.org/tglx/notes-about-netiquette