2005-12-31 15:01:22

by Mike Galbraith

[permalink] [raw]
Subject: [2.6.15-rc7-rt1] check_monotonic_clock: monotonic inconsistency detected!

Greetings,

After running just fine for most of the day, my PIII/500 coughed up
attached fur-ball as I tried to start my ancient (au) sound
server. Subsequent repeated starting and stopping of same did not induce a
repeat (but then all zillion sound modules were loaded). If any other info
is needed, just holler.

-Mike


Attachments:
xx (803.00 B)

2005-12-31 15:50:03

by Steven Rostedt

[permalink] [raw]
Subject: Re: [2.6.15-rc7-rt1] check_monotonic_clock: monotonic inconsistency detected!

On Sat, 2005-12-31 at 16:00 +0100, Mike Galbraith wrote:
> Greetings,
>
> After running just fine for most of the day, my PIII/500 coughed up
> attached fur-ball as I tried to start my ancient (au) sound
> server. Subsequent repeated starting and stopping of same did not induce a
> repeat (but then all zillion sound modules were loaded). If any other info
> is needed, just holler.

[inlining the attachment]

check_monotonic_clock: monotonic inconsistency detected!
from 191c7872fef6 (27610070580982) to 191c7872dd13 (27610070572307).
softirq-hrtimer/8[CPU#0]: BUG in check_monotonic_clock at kernel/time/timeofday.c:160
[<c0104063>] dump_stack+0x23/0x30 (20)
[<c011db1a>] __WARN_ON+0x6a/0xa0 (44)
[<c013c8d2>] check_monotonic_clock+0xe2/0x100 (52)
[<c013cd85>] get_monotonic_clock+0xe5/0x120 (80)
[<c0135b16>] hrtimer_forward+0x36/0xf0 (80)
[<c0120cbb>] it_real_fn+0x6b/0x80 (28)
[<c0136345>] run_hrtimer_softirq+0x65/0x150 (52)
[<c0122826>] ksoftirqd+0x116/0x1c0 (60)
[<c01325b9>] kthread+0xa9/0xf0 (52)
[<c0101245>] kernel_thread_helper+0x5/0x10 (671293468)
------------------------------
| showing all locks held by: | (softirq-hrtimer/8 [d7fcabb0, 98]):
------------------------------

[end inline]

This looks like it probably has nothing to do with your module. The
monotonic clock went backwards. Also, it will only print once per boot,
even if the event happens again.

Is your system SMP? Could you also send your config. And John will
probably want to see your dmesg output.

-- Steve


2005-12-31 16:36:58

by Mike Galbraith

[permalink] [raw]
Subject: Re: [2.6.15-rc7-rt1] check_monotonic_clock: monotonic inconsistency detected!

At 10:49 AM 12/31/2005 -0500, Steven Rostedt wrote:
>On Sat, 2005-12-31 at 16:00 +0100, Mike Galbraith wrote:
> > Greetings,
> >
> > After running just fine for most of the day, my PIII/500 coughed up
> > attached fur-ball as I tried to start my ancient (au) sound
> > server. Subsequent repeated starting and stopping of same did not
> induce a
> > repeat (but then all zillion sound modules were loaded). If any other
> info
> > is needed, just holler.
>
>[inlining the attachment]
>
>check_monotonic_clock: monotonic inconsistency detected!
> from 191c7872fef6 (27610070580982) to 191c7872dd13
> (27610070572307).
>softirq-hrtimer/8[CPU#0]: BUG in check_monotonic_clock at
>kernel/time/timeofday.c:160
> [<c0104063>] dump_stack+0x23/0x30 (20)
> [<c011db1a>] __WARN_ON+0x6a/0xa0 (44)
> [<c013c8d2>] check_monotonic_clock+0xe2/0x100 (52)
> [<c013cd85>] get_monotonic_clock+0xe5/0x120 (80)
> [<c0135b16>] hrtimer_forward+0x36/0xf0 (80)
> [<c0120cbb>] it_real_fn+0x6b/0x80 (28)
> [<c0136345>] run_hrtimer_softirq+0x65/0x150 (52)
> [<c0122826>] ksoftirqd+0x116/0x1c0 (60)
> [<c01325b9>] kthread+0xa9/0xf0 (52)
> [<c0101245>] kernel_thread_helper+0x5/0x10 (671293468)
>------------------------------
>| showing all locks held by: | (softirq-hrtimer/8 [d7fcabb0, 98]):
>------------------------------
>
>[end inline]
>
>This looks like it probably has nothing to do with your module. The
>monotonic clock went backwards. Also, it will only print once per boot,
>even if the event happens again.
>
>Is your system SMP?

No.

> Could you also send your config.

Attached. (this is winbloze eudora, no inlining possible)

> And John will
>probably want to see your dmesg output.

I'll send that off-line upon request. FWIW, it's not the original boot
dmesg output however, but one a freshly booted system...

As luck would have it, just as I was collecting the data, there was a major
explosion. Box was slogging through glibc make check, and when it hit the
mutex tests in nptl, KaBOOM. Ding-dong-dead box... [reboot] oh my, seems
repeatable. Guess I'll see if I can find my serial console cable instead
of typing make install as originally planned :)

-Mike


Attachments:
.config (30.96 kB)

2005-12-31 17:10:56

by Mike Galbraith

[permalink] [raw]
Subject: [2.6.15-rc7-rt1] explosions in nptl mutex tests

At 05:36 PM 12/31/2005 +0100, Mike Galbraith wrote:

>As luck would have it, just as I was collecting the data, there was a
>major explosion. Box was slogging through glibc make check, and when it
>hit the mutex tests in nptl, KaBOOM. Ding-dong-dead box... [reboot] oh
>my, seems repeatable. Guess I'll see if I can find my serial console
>cable instead of typing make install as originally planned :)
>
> -Mike

Oopsen attached. First two times it exploded on tst-mutex7, this time that
passed, but it exploded on tst-mutex7a.

-Mike


Attachments:
oops.txt (4.45 kB)

2006-01-02 04:46:37

by Mike Galbraith

[permalink] [raw]
Subject: Re: [2.6.15-rc7-rt1] explosions in nptl mutex tests

At 06:10 PM 12/31/2005 +0100, Mike Galbraith wrote:
>At 05:36 PM 12/31/2005 +0100, Mike Galbraith wrote:
>
>>As luck would have it, just as I was collecting the data, there was a
>>major explosion. Box was slogging through glibc make check, and when it
>>hit the mutex tests in nptl, KaBOOM. Ding-dong-dead box... [reboot] oh
>>my, seems repeatable. Guess I'll see if I can find my serial console
>>cable instead of typing make install as originally planned :)
>>
>> -Mike
>
>Oopsen attached. First two times it exploded on tst-mutex7, this time
>that passed, but it exploded on tst-mutex7a.

Further datapoint: on a lark, I tried 8k stacks, and my symptoms changed -
tst-mutex7 never failed, but running tst-mutex7a in a tight loop crashed
the box reliably though differently. I then recompiled with no debug
options enabled, and ran all mutex tests in a loop for a couple of hours
with no crash.

-Mike

2006-01-04 20:45:44

by john stultz

[permalink] [raw]
Subject: Re: [2.6.15-rc7-rt1] check_monotonic_clock: monotonic inconsistency detected!

On Sat, 2005-12-31 at 17:36 +0100, Mike Galbraith wrote:
> At 10:49 AM 12/31/2005 -0500, Steven Rostedt wrote:
> > And John will
> >probably want to see your dmesg output.

Am I so predictable? :)

> I'll send that off-line upon request. FWIW, it's not the original boot
> dmesg output however, but one a freshly booted system...

Please do send a dmesg to me. I'm looking to find which clocksource
caused this warning. Its probably the TSC, which isn't surprising, but I
want to be sure.

thanks
-john


2006-01-04 20:53:25

by Steven Rostedt

[permalink] [raw]
Subject: Re: [2.6.15-rc7-rt1] check_monotonic_clock: monotonic inconsistency detected!


On Wed, 4 Jan 2006, john stultz wrote:

> On Sat, 2005-12-31 at 17:36 +0100, Mike Galbraith wrote:
> > At 10:49 AM 12/31/2005 -0500, Steven Rostedt wrote:
> > > And John will
> > >probably want to see your dmesg output.
>
> Am I so predictable? :)
>

Oh, only that every problem thread, you start out with "can you send me
your dmesg?". ;)

-- Steve