2007-05-02 22:38:34

by Zachary Amsden

[permalink] [raw]
Subject: Mysterious RTC hangs on x86_64 - fixed, sort of

CONFIG_HPET_EMULATE_RTC=y
CONFIG_RTC=y
# CONFIG_HPET_RTC_IRQ is not set
CONFIG_SND_RTCTIMER=m
CONFIG_SND_SEQ_RTCTIMER_DEFAULT=y
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
CONFIG_RTC_DEBUG=y
# RTC interfaces
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y


Attachments:
x86_64-rtc-mystery.patch (810.00 B)
rtc.config (323.00 B)
Download all attachments

2007-05-02 22:56:37

by Chuck Ebbert

[permalink] [raw]
Subject: Re: Mysterious RTC hangs on x86_64 - fixed, sort of

Zachary Amsden wrote:
> With this patch, /sbin/hwclock no longer hangs my AMD64 machine when run
> after reaching multiuser. What I don't understand is why. I have the
> RTC based sound sequencer timer as a module, but not loaded, and the
> error message I added to indicate broken rtc control does not fire.
>
> So why is it that if I stop taking the rtc_task_lock and issuing the
> callbacks which should never be held or exist that my system no longer
> hard freezes?
>
> --- /tmp/a 2007-05-03 15:36:07.451256181 -0700
> +++ drivers/char/rtc.c 2007-05-03 15:27:49.000000000 -0700
> @@ -265,10 +265,10 @@
> spin_unlock (&rtc_lock);
>
> /* Now do the rest of the actions */
> - spin_lock(&rtc_task_lock);
> - if (rtc_callback)
> - rtc_callback->func(rtc_callback->private_data);
> - spin_unlock(&rtc_task_lock);
> +/* spin_lock(&rtc_task_lock); */
> +// if (rtc_callback)
> +// rtc_callback->func(rtc_callback->private_data);
> +/* spin_unlock(&rtc_task_lock); */
> wake_up_interruptible(&rtc_wait);

Try leaving the spinlocks and just disabling the callbacks. And maybe
enable spinlock debugging...

>
> CONFIG_HPET_EMULATE_RTC=y

Did you try without that?

2007-05-03 00:08:13

by Zachary Amsden

[permalink] [raw]
Subject: Re: Mysterious RTC hangs on x86_64 - fixed, sort of

Chuck Ebbert wrote:

Well, turns out this is a heisenbug. Which is good, since it means the
nop patch didn't change anything.

> Try leaving the spinlocks and just disabling the callbacks. And maybe
> enable spinlock debugging...
>

I tried removing all the spinlocks inside the interrupt handler. Seemed
to work fine for a while, but still hung (at worst, it looks missing
locks means we might screw up and read / write the wrong CMOS register,
not hang or crash).

So I took down 2nd CPU with hotplug (did not yet try UP kernel though).
It took a longer time, but still hung. Seems not to be a spinlock
problem, but I'll turn on debugging anyway.

>
>> CONFIG_HPET_EMULATE_RTC=y
>>
>
> Did you try without that?
>

Will do. That looks much more suspicious like. I thought I killed it
already, but had only got this:

# CONFIG_HPET_RTC_IRQ is not set

If that still crashes, I'll try running cmos access in a loop in userspace to see if maybe the port I/O is tickling a chipset bug (the only other report I know of is on same chipset, nVidia MCP51). Maybe SMM handler is accessing CMOS or something wacked out. <laughs hysterical... stops laughing when the theory actually sounds plausible>. Stuck in SMM is not good for CPU thermal throttling ... hopefully Turion's don't reach nuclear emission point.

Would also explain maybe why NMI watchdog doesn't seem to notice anything wrong.


Thanks,
Zach

2007-05-03 04:25:42

by Zachary Amsden

[permalink] [raw]
Subject: Re: Mysterious RTC hangs on x86_64 - fixed, sort of

Chuck Ebbert wrote:
>
>
>> CONFIG_HPET_EMULATE_RTC=y
>>
>
> Did you try without that?
>

Just did. Still hangs same way; strace shows /sbin/hwclock dying after
hundreds of RTC_RD_TIME. And now /proc/interrupts shows no rtc
interrupts being generated (expected, I gues). Seems to take longer to
crash, but this is a heisenbug.

Enough crashing for today. Strangest thing is the NMI watchdog does not
fire...

Zach

2007-05-03 09:00:31

by Andi Kleen

[permalink] [raw]
Subject: Re: Mysterious RTC hangs on x86_64 - fixed, sort of


> Enough crashing for today. Strangest thing is the NMI watchdog does not
> fire...

It's disabled now by default.

-Andi