2010-02-01 04:15:48

by Anton Blanchard

[permalink] [raw]
Subject: [PATCH] panic: Fix panic_timeout accuracy when running on a hypervisor


I've had some complaints about panic_timeout being wildly innacurate on
shared processor PowerPC partitions (a 3 minute panic_timeout taking 30
minutes).

The problem is we loop on mdelay(1) and with a 1ms in 10ms hypervisor
timeslice each of these will take 10ms (ie 10x) longer. I expect other
platforms with shared processor hypervisors will see the same issue.

This patch keeps the old behaviour if we have a panic_blink (only keyboard
LEDs right now) and does 1 second mdelays if we don't.

Signed-off-by: Anton Blanchard <[email protected]>
---

Index: linux-cpumask/kernel/panic.c
===================================================================
--- linux-cpumask.orig/kernel/panic.c 2010-02-01 14:17:40.140961595 +1100
+++ linux-cpumask/kernel/panic.c 2010-02-01 14:30:45.549711407 +1100
@@ -36,15 +36,36 @@ ATOMIC_NOTIFIER_HEAD(panic_notifier_list

EXPORT_SYMBOL(panic_notifier_list);

-static long no_blink(long time)
-{
- return 0;
-}
-
/* Returns how long it waited in ms */
long (*panic_blink)(long time);
EXPORT_SYMBOL(panic_blink);

+static void panic_blink_one_second(void)
+{
+ static long i = 0, end;
+
+ if (panic_blink) {
+ end = i + MSEC_PER_SEC;
+
+ while (i < end) {
+ i += panic_blink(i);
+ mdelay(1);
+ i++;
+ }
+ } else {
+ /*
+ * When running under a hypervisor a small mdelay may get
+ * rounded up to the hypervisor timeslice. For example, with
+ * a 1ms in 10ms hypervisor timeslice we might inflate a
+ * mdelay(1) loop by 10x.
+ *
+ * If we have nothing to blink, spin on 1 second calls to
+ * mdelay to avoid this.
+ */
+ mdelay(MSEC_PER_SEC);
+ }
+}
+
/**
* panic - halt the system
* @fmt: The text string to print
@@ -95,9 +116,6 @@ NORET_TYPE void panic(const char * fmt,

bust_spinlocks(0);

- if (!panic_blink)
- panic_blink = no_blink;
-
if (panic_timeout > 0) {
/*
* Delay timeout seconds before rebooting the machine.
@@ -105,11 +123,9 @@ NORET_TYPE void panic(const char * fmt,
*/
printk(KERN_EMERG "Rebooting in %d seconds..", panic_timeout);

- for (i = 0; i < panic_timeout*1000; ) {
+ for (i = 0; i < panic_timeout; i++) {
touch_nmi_watchdog();
- i += panic_blink(i);
- mdelay(1);
- i++;
+ panic_blink_one_second();
}
/*
* This will not be a clean reboot, with everything
@@ -135,11 +151,9 @@ NORET_TYPE void panic(const char * fmt,
}
#endif
local_irq_enable();
- for (i = 0; ; ) {
+ while (1) {
touch_softlockup_watchdog();
- i += panic_blink(i);
- mdelay(1);
- i++;
+ panic_blink_one_second();
}
}


2010-02-01 22:03:00

by Andrew Morton

[permalink] [raw]
Subject: Re: [PATCH] panic: Fix panic_timeout accuracy when running on a hypervisor

On Mon, 1 Feb 2010 15:14:30 +1100
Anton Blanchard <[email protected]> wrote:

>
> I've had some complaints about panic_timeout being wildly innacurate on
> shared processor PowerPC partitions (a 3 minute panic_timeout taking 30
> minutes).
>
> The problem is we loop on mdelay(1) and with a 1ms in 10ms hypervisor
> timeslice each of these will take 10ms (ie 10x) longer. I expect other
> platforms with shared processor hypervisors will see the same issue.
>
> This patch keeps the old behaviour if we have a panic_blink (only keyboard
> LEDs right now) and does 1 second mdelays if we don't.
>
> ...
>
> +static void panic_blink_one_second(void)
> +{
> + static long i = 0, end;

I assumed the `static' was a brainfart and removed it?

> + if (panic_blink) {
> + end = i + MSEC_PER_SEC;
> +
> + while (i < end) {
> + i += panic_blink(i);
> + mdelay(1);
> + i++;
> + }
> + } else {
> + /*
> + * When running under a hypervisor a small mdelay may get
> + * rounded up to the hypervisor timeslice. For example, with
> + * a 1ms in 10ms hypervisor timeslice we might inflate a
> + * mdelay(1) loop by 10x.
> + *
> + * If we have nothing to blink, spin on 1 second calls to
> + * mdelay to avoid this.
> + */
> + mdelay(MSEC_PER_SEC);
> + }
> +}

In fact we can simplify it a bit:

--- a/kernel/panic.c~panic-fix-panic_timeout-accuracy-when-running-on-a-hypervisor-fix
+++ a/kernel/panic.c
@@ -42,12 +42,10 @@ EXPORT_SYMBOL(panic_blink);

static void panic_blink_one_second(void)
{
- static long i = 0, end;
-
if (panic_blink) {
- end = i + MSEC_PER_SEC;
+ long i = 0;

- while (i < end) {
+ while (i < MSEC_PER_SEC) {
i += panic_blink(i);
mdelay(1);
i++;
_

Why does it do the mdelay() as well as calling panic_blink()? hm,
because the old code did. I guess it doesn't trust panic_blink().

2010-02-02 00:53:18

by Anton Blanchard

[permalink] [raw]
Subject: Re: [PATCH] panic: Fix panic_timeout accuracy when running on a hypervisor


Hi Andrew,

> > +static void panic_blink_one_second(void)
> > +{
> > + static long i = 0, end;
>
> I assumed the `static' was a brainfart and removed it?

...

> In fact we can simplify it a bit:
>
> --- a/kernel/panic.c~panic-fix-panic_timeout-accuracy-when-running-on-a-hypervisor-fix
> +++ a/kernel/panic.c
> @@ -42,12 +42,10 @@ EXPORT_SYMBOL(panic_blink);
>
> static void panic_blink_one_second(void)
> {
> - static long i = 0, end;
> -
> if (panic_blink) {
> - end = i + MSEC_PER_SEC;
> + long i = 0;
>
> - while (i < end) {
> + while (i < MSEC_PER_SEC) {
> i += panic_blink(i);
> mdelay(1);
> i++;

Unfortunately the panic_blink users seem to rely on count ever increasing:

static long i8042_panic_blink(long count)
{
...
static long last_blink;
...
if (count - last_blink < i8042_blink_frequency)
return 0;

If we reset to 0 each second, this is going to always be true. Ugly interface.

Anton