panic_lock is meant to ensure that panic processing takes
place only on one cpu; if any of the other cpus encounter
a panic, they will spin waiting to be shut down.
However, this causes a regression in this scenario:
1. Cpu 0 encounters a panic and acquires the panic_lock
and proceeds with the panic processing.
2. There is an interrupt on cpu 0 that also encounters
an error condition and invokes panic.
3. This second invocation fails to acquire the panic_lock
and enters the infinite while loop in panic_smp_self_stop.
Thus all panic processing is stopped, and the cpu is stuck
for eternity in the while(1) inside panic_smp_self_stop.
To address this, disable local interrupts with
local_irq_disable before acquiring the panic_lock. This will
prevent interrupt handlers from executing during the panic
processing, thus avoiding this particular problem.
Signed-off-by: Vikram Mulukutla <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
---
kernel/panic.c | 8 ++++++++
1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/kernel/panic.c b/kernel/panic.c
index d2a5f4e..e1b2822 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -75,6 +75,14 @@ void panic(const char *fmt, ...)
int state = 0;
/*
+ * Disable local interrupts. This will prevent panic_smp_self_stop
+ * from deadlocking the first cpu that invokes the panic, since
+ * there is nothing to prevent an interrupt handler (that runs
+ * after the panic_lock is acquired) from invoking panic again.
+ */
+ local_irq_disable();
+
+ /*
* It's possible to come here directly from a panic-assertion and
* not have preempt disabled. Some functions called from here want
* preempt to be disabled. No point enabling it later though...
--
1.7.8.3
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Hello Vikram,
Putting "linux-arch" on cc...
On Thu, 28 Jun 2012 16:43:05 -0700
Vikram Mulukutla <[email protected]> wrote:
> panic_lock is meant to ensure that panic processing takes
> place only on one cpu; if any of the other cpus encounter
> a panic, they will spin waiting to be shut down.
>
> However, this causes a regression in this scenario:
>
> 1. Cpu 0 encounters a panic and acquires the panic_lock
> and proceeds with the panic processing.
> 2. There is an interrupt on cpu 0 that also encounters
> an error condition and invokes panic.
> 3. This second invocation fails to acquire the panic_lock
> and enters the infinite while loop in panic_smp_self_stop.
>
> Thus all panic processing is stopped, and the cpu is stuck
> for eternity in the while(1) inside panic_smp_self_stop.
>
> To address this, disable local interrupts with
> local_irq_disable before acquiring the panic_lock. This will
> prevent interrupt handlers from executing during the panic
> processing, thus avoiding this particular problem.
Looks good to me.
I re-read the panic lock discussion and in fact one version of my patch
also disabled interrupts:
http://lists.infradead.org/pipermail/kexec/2011-October/005695.html
I think the reason why we later took a version with irqs enabled was
that we did not think about the scenario you described above and
we wanted to make the change as less intrusive as possible. But I am
not really sure about that.
Regarding you patch: Perhaps we could use spin_trylock_irq() instead of
local_irq_disable() and spin_lock().
Michael
On Thu, 28 Jun 2012 16:43:05 -0700
Vikram Mulukutla <[email protected]> wrote:
> panic_lock is meant to ensure that panic processing takes
> place only on one cpu; if any of the other cpus encounter
> a panic, they will spin waiting to be shut down.
>
> However, this causes a regression in this scenario:
>
> 1. Cpu 0 encounters a panic and acquires the panic_lock
> and proceeds with the panic processing.
> 2. There is an interrupt on cpu 0 that also encounters
> an error condition and invokes panic.
> 3. This second invocation fails to acquire the panic_lock
> and enters the infinite while loop in panic_smp_self_stop.
>
> Thus all panic processing is stopped, and the cpu is stuck
> for eternity in the while(1) inside panic_smp_self_stop.
>
> To address this, disable local interrupts with
> local_irq_disable before acquiring the panic_lock. This will
> prevent interrupt handlers from executing during the panic
> processing, thus avoiding this particular problem.
>
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -75,6 +75,14 @@ void panic(const char *fmt, ...)
> int state = 0;
>
> /*
> + * Disable local interrupts. This will prevent panic_smp_self_stop
> + * from deadlocking the first cpu that invokes the panic, since
> + * there is nothing to prevent an interrupt handler (that runs
> + * after the panic_lock is acquired) from invoking panic again.
> + */
> + local_irq_disable();
> +
> + /*
> * It's possible to come here directly from a panic-assertion and
> * not have preempt disabled. Some functions called from here want
> * preempt to be disabled. No point enabling it later though...
Seems sane. panic() *should* work correctly when called with
interrupts disabled, so there be no bad effects from internally
disabling interrupts. If there are bad effects, we should fix them up.