Currently, if printk lock (logbuf_lock) is held by other thread during
crash, there is a chance of deadlocking the crash on next printk, and
blocking a possibly desired kdump.
At the start of default_machine_crash_shutdown, make printk enter
NMI context, as it will use per-cpu buffers to store the message,
and avoid locking logbuf_lock.
Suggested-by: Michael Ellerman <[email protected]>
Signed-off-by: Leonardo Bras <[email protected]>
---
Changes since v1:
- Added in-code comment explaining the need of context change
- Function moved to the start of default_machine_crash_shutdown,
to avoid locking any printk on crashing routine.
- Title was 'Use NMI context for printk after crashing other CPUs'
---
arch/powerpc/kexec/crash.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/arch/powerpc/kexec/crash.c b/arch/powerpc/kexec/crash.c
index d488311efab1..c9a889880214 100644
--- a/arch/powerpc/kexec/crash.c
+++ b/arch/powerpc/kexec/crash.c
@@ -311,6 +311,9 @@ void default_machine_crash_shutdown(struct pt_regs *regs)
unsigned int i;
int (*old_handler)(struct pt_regs *regs);
+ /* Avoid hardlocking with irresponsive CPU holding logbuf_lock */
+ printk_nmi_enter();
+
/*
* This function is only called after the system
* has panicked or is otherwise in a critical state.
--
2.25.4
Excerpts from Leonardo Bras's message of May 13, 2020 7:45 am:
> Currently, if printk lock (logbuf_lock) is held by other thread during
> crash, there is a chance of deadlocking the crash on next printk, and
> blocking a possibly desired kdump.
>
> At the start of default_machine_crash_shutdown, make printk enter
> NMI context, as it will use per-cpu buffers to store the message,
> and avoid locking logbuf_lock.
printk_nmi_enter is used in one other place outside nmi_enter.
Is there a different/better way to handle this? What do other
architectures do?
Other subsystems get put into an nmi-mode when we call nmi_enter
(lockdep, ftrace, rcu etc). It seems like those would be useful for
similar reasons, so at least explaining why that is not used in a
comment would be good.
Aside from that, I welcome any effort to make our crashes more reliable
so thanks for working on this stuff.
Thanks,
Nick
>
> Suggested-by: Michael Ellerman <[email protected]>
> Signed-off-by: Leonardo Bras <[email protected]>
>
> ---
> Changes since v1:
> - Added in-code comment explaining the need of context change
> - Function moved to the start of default_machine_crash_shutdown,
> to avoid locking any printk on crashing routine.
> - Title was 'Use NMI context for printk after crashing other CPUs'
>
> ---
> arch/powerpc/kexec/crash.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/arch/powerpc/kexec/crash.c b/arch/powerpc/kexec/crash.c
> index d488311efab1..c9a889880214 100644
> --- a/arch/powerpc/kexec/crash.c
> +++ b/arch/powerpc/kexec/crash.c
> @@ -311,6 +311,9 @@ void default_machine_crash_shutdown(struct pt_regs *regs)
> unsigned int i;
> int (*old_handler)(struct pt_regs *regs);
>
> + /* Avoid hardlocking with irresponsive CPU holding logbuf_lock */
> + printk_nmi_enter();
> +
> /*
> * This function is only called after the system
> * has panicked or is otherwise in a critical state.
> --
> 2.25.4
>
>
Hello Nick, thanks for your feedback.
Comments inline:
On Wed, 2020-05-13 at 14:36 +1000, Nicholas Piggin wrote:
> Excerpts from Leonardo Bras's message of May 13, 2020 7:45 am:
> > Currently, if printk lock (logbuf_lock) is held by other thread during
> > crash, there is a chance of deadlocking the crash on next printk, and
> > blocking a possibly desired kdump.
> >
> > At the start of default_machine_crash_shutdown, make printk enter
> > NMI context, as it will use per-cpu buffers to store the message,
> > and avoid locking logbuf_lock.
>
> printk_nmi_enter is used in one other place outside nmi_enter.
>
> Is there a different/better way to handle this? What do other
> architectures do?
To be honest, I was unaware of nmi_enter() and I have yet to study what
other architectures do here.
> Other subsystems get put into an nmi-mode when we call nmi_enter
> (lockdep, ftrace, rcu etc). It seems like those would be useful for
> similar reasons, so at least explaining why that is not used in a
> comment would be good.
My reasoning for using printk_nmi_enter() here was only to keep it from
using printk regular buffer (and locking logbuf_lock) at this point of
the crash.
I have yet to see how nmi_enter() extra functions would happen to
interfere with the crash at this point.
(In a quick look at x86, (native_machine_crash_shutdown) I could not
see it using any printk, so it may not be necessary).
> Aside from that, I welcome any effort to make our crashes more reliable
> so thanks for working on this stuff.
>
> Thanks,
> Nick
Thank you, it means a lot.
Leonardo Bras
On Tue, 12 May 2020 18:45:35 -0300, Leonardo Bras wrote:
> Currently, if printk lock (logbuf_lock) is held by other thread during
> crash, there is a chance of deadlocking the crash on next printk, and
> blocking a possibly desired kdump.
>
> At the start of default_machine_crash_shutdown, make printk enter
> NMI context, as it will use per-cpu buffers to store the message,
> and avoid locking logbuf_lock.
Applied to powerpc/next.
[1/1] powerpc/crash: Use NMI context for printk when starting to crash
https://git.kernel.org/powerpc/c/af2876b501e42c3fb5174cac9dd02598436f0fdf
cheers