2010-01-10 21:48:18

by Michael Breuer

[permalink] [raw]
Subject: 2.6.33RC3 hung task - jbd2/dm-1-8 (during raid rebuild)

System crashed (watchdog reset).

Was doing a raid rebuild at the time (raid6).

I typed, 'yumdownloader --source virtlib' ... system froze, then
rebooted on the watchdog reset.

I've attached the log extract from the event as well as dmesg.



Attachments:
messages (22.44 kB)
dmesg.old (56.45 kB)
Download all attachments

2010-01-30 19:21:05

by Michael Breuer

[permalink] [raw]
Subject: Regressionhung task during raid rebuild 2.6.33 rc5 was: Re: 2.6.33RC3 hung task - jbd2/dm-1-8 (during raid rebuild)

On 1/10/2010 4:47 PM, Michael Breuer wrote:
> System crashed (watchdog reset).
>
> Was doing a raid rebuild at the time (raid6).
>
> I typed, 'yumdownloader --source virtlib' ... system froze, then
> rebooted on the watchdog reset.
>
> I've attached the log extract from the event as well as dmesg.
>
>
Hit this again - system doesn't crash now - just get the hung task
warning (log extract attached). System seems to keep running (albeit the
rebuild is incredibly slow on 2.6.33 rc5 - approx 6500K/sec vs. about
80,000K/sec on 2.6.32.4.

Looks like it may be triggered by disk io during the rebuild. This time
I had typed, "sync." "sync" hung for a long while but did eventually
complete.


Attachments:
messages (22.94 kB)

2010-01-30 19:26:40

by Michael Breuer

[permalink] [raw]
Subject: Re: Regression: hung task during raid rebuild 2.6.33 rc5 was: Re: 2.6.33RC3 hung task - jbd2/dm-1-8 (during raid rebuild)

On 1/30/2010 2:21 PM, Michael Breuer wrote:
> On 1/10/2010 4:47 PM, Michael Breuer wrote:
>> System crashed (watchdog reset).
>>
>> Was doing a raid rebuild at the time (raid6).
>>
>> I typed, 'yumdownloader --source virtlib' ... system froze, then
>> rebooted on the watchdog reset.
>>
>> I've attached the log extract from the event as well as dmesg.
>>
>>
> Hit this again - system doesn't crash now - just get the hung task
> warning (log extract attached). System seems to keep running (albeit
> the rebuild is incredibly slow on 2.6.33 rc5 - approx 6500K/sec vs.
> about 80,000K/sec on 2.6.32.4.
>
> Looks like it may be triggered by disk io during the rebuild. This
> time I had typed, "sync." "sync" hung for a long while but did
> eventually complete.
Also - when I attempted to reboot, I got the following (only on an ssh
terminal window - nothing in dmesg or log after reboot):

Message from syslogd@mail at Jan 30 14:16:42 ...
kernel:------------[ cut here ]------------

Message from syslogd@mail at Jan 30 14:16:42 ...
kernel:invalid opcode: 0000 [#1] SMP

Message from syslogd@mail at Jan 30 14:16:42 ...
kernel:last sysfs file: /sys/devices/virtual/block/md0/size

Message from syslogd@mail at Jan 30 14:16:42 ...
kernel:Stack:

Message from syslogd@mail at Jan 30 14:16:42 ...
kernel:Call Trace:

Message from syslogd@mail at Jan 30 14:16:42 ...
kernel:Code: 41 5f c9 c3 55 48 89 e5 41 55 41 54 53 48 83 ec 38 0f 1f
44 00 00 48 89 fb f6 46 20 01 0f 84 26 01 00 00 8b 47 38 83 f8 01 75 04
<0f> 0b eb fe 45 31 e4 83 f8 02 75 2a c7 47 38 00 00 00 00 f0 80

[ends here]

2010-01-30 19:57:09

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: Regressionhung task during raid rebuild 2.6.33 rc5 was: Re: 2.6.33RC3 hung task - jbd2/dm-1-8 (during raid rebuild)

On Saturday 30 January 2010, Michael Breuer wrote:
> On 1/10/2010 4:47 PM, Michael Breuer wrote:
> > System crashed (watchdog reset).
> >
> > Was doing a raid rebuild at the time (raid6).
> >
> > I typed, 'yumdownloader --source virtlib' ... system froze, then
> > rebooted on the watchdog reset.
> >
> > I've attached the log extract from the event as well as dmesg.
> >
> >
> Hit this again - system doesn't crash now - just get the hung task
> warning (log extract attached). System seems to keep running (albeit the
> rebuild is incredibly slow on 2.6.33 rc5 - approx 6500K/sec vs. about
> 80,000K/sec on 2.6.32.4.
>
> Looks like it may be triggered by disk io during the rebuild. This time
> I had typed, "sync." "sync" hung for a long while but did eventually
> complete.

Is this the bug tracked as http://bugzilla.kernel.org/show_bug.cgi?id=15125 ?

Rafael

2010-01-30 20:00:14

by Michael Breuer

[permalink] [raw]
Subject: Re: Regressionhung task during raid rebuild 2.6.33 rc5 was: Re: 2.6.33RC3 hung task - jbd2/dm-1-8 (during raid rebuild)

On 1/30/2010 2:57 PM, Rafael J. Wysocki wrote:
> On Saturday 30 January 2010, Michael Breuer wrote:
>
>> On 1/10/2010 4:47 PM, Michael Breuer wrote:
>>
>>> System crashed (watchdog reset).
>>>
>>> Was doing a raid rebuild at the time (raid6).
>>>
>>> I typed, 'yumdownloader --source virtlib' ... system froze, then
>>> rebooted on the watchdog reset.
>>>
>>> I've attached the log extract from the event as well as dmesg.
>>>
>>>
>>>
>> Hit this again - system doesn't crash now - just get the hung task
>> warning (log extract attached). System seems to keep running (albeit the
>> rebuild is incredibly slow on 2.6.33 rc5 - approx 6500K/sec vs. about
>> 80,000K/sec on 2.6.32.4.
>>
>> Looks like it may be triggered by disk io during the rebuild. This time
>> I had typed, "sync." "sync" hung for a long while but did eventually
>> complete.
>>
> Is this the bug tracked as http://bugzilla.kernel.org/show_bug.cgi?id=15125 ?
>
> Rafael
>
Yes - although the behavior is different on rc5 vs. rc3. I think it's
the same.