Hi,
ever since I moved to 2.6.25-rcY (Y should be between 2 and 5!).
I've seen several, but really hard-to-catch instant freezes on my
AMD64 Athlon X2 4200+ system...
Most of them happend in X.org so at first I thought it had something to do
with the NVIDIA module... BUT, one time it froze "a way before" the module
could get loaded (and desynced my raid.......).
---
SYSRQ-P revealed that the CPU were looping inside:
smp_64.c native_flush_tlb_others:
assembler code:
< 1ee: f3 90 pause
< 1f0: f6 45 00 03 testb $0x3,0x0(%rbp)
< 1f4: 75 f8 jne 1ee <native_flush_tlb_others+0x5f>
also known as: (in C)
while (!cpus_empty(f->flush_cpumask))
cpu_relax();
So... has anyone a good idea what's happening here exactly? Or is there
already another topic or even a patch available?
Regards,
Chr.
CC added
On Tue, Mar 11, 2008 at 6:16 AM, Chr <[email protected]> wrote:
> Hi,
Hi Chr,
>
> ever since I moved to 2.6.25-rcY (Y should be between 2 and 5!).
> I've seen several, but really hard-to-catch instant freezes on my
> AMD64 Athlon X2 4200+ system...
Here I guess 2.6.24 is fine for you?
>
> Most of them happend in X.org so at first I thought it had something to do
> with the NVIDIA module... BUT, one time it froze "a way before" the module
> could get loaded (and desynced my raid.......).
>
> ---
> SYSRQ-P revealed that the CPU were looping inside:
>
> smp_64.c native_flush_tlb_others:
> assembler code:
> < 1ee: f3 90 pause
> < 1f0: f6 45 00 03 testb $0x3,0x0(%rbp)
> < 1f4: 75 f8 jne 1ee <native_flush_tlb_others+0x5f>
>
> also known as: (in C)
>
> while (!cpus_empty(f->flush_cpumask))
> cpu_relax();
>
> So... has anyone a good idea what's happening here exactly? Or is there
> already another topic or even a patch available?
>
> Regards,
> Chr.
> --
Would you please attach your config file? Do you have CONFIG_CPU_IDLE set?
Regards,
On Tuesday 11 March 2008 10:55:40 Jike Song wrote:
> On Tue, Mar 11, 2008 at 6:16 AM, Chr <[email protected]> wrote:
> > Hi,
> > ever since I moved to 2.6.25-rcY (Y should be between 2 and 5!).
> > I've seen several, but really hard-to-catch instant freezes on my
> > AMD64 Athlon X2 4200+ system...
>
> Here I guess 2.6.24 is fine for you?
Yes, 2.6.24(.3) is fine!
> > Most of them happend in X.org so at first I thought it had something to
> > do with the NVIDIA module... BUT, one time it froze "a way before" the
> > module could get loaded...
> >
> > ---
> > SYSRQ-P revealed that the CPU were looping inside:
> >
> > smp_64.c native_flush_tlb_others:
> > assembler code:
> > < 1ee: f3 90 pause
> > < 1f0: f6 45 00 03 testb $0x3,0x0(%rbp)
> > < 1f4: 75 f8 jne 1ee
> > <native_flush_tlb_others+0x5f>
> >
> > also known as: (in C)
> >
> > while (!cpus_empty(f->flush_cpumask))
> > cpu_relax();
> >
> > So... has anyone a good idea what's happening here exactly? Or is there
> > already another topic or even a patch available?
>
> Would you please attach your config file? Do you have CONFIG_CPU_IDLE set?
(Attached). Yes, CONFIG_CPU_IDLE is enabled! I guess I should now disable it,
try again and report back, right?! ;-)
Regards,
Chr.
On Tue, 11 Mar 2008, Jike Song wrote:
> > Most of them happend in X.org so at first I thought it had something to do
> > with the NVIDIA module... BUT, one time it froze "a way before" the module
> > could get loaded (and desynced my raid.......).
> >
> > ---
> > SYSRQ-P revealed that the CPU were looping inside:
> >
> > smp_64.c native_flush_tlb_others:
> > assembler code:
> > < 1ee: f3 90 pause
> > < 1f0: f6 45 00 03 testb $0x3,0x0(%rbp)
> > < 1f4: 75 f8 jne 1ee <native_flush_tlb_others+0x5f>
> >
> > also known as: (in C)
> >
> > while (!cpus_empty(f->flush_cpumask))
> > cpu_relax();
> >
> > So... has anyone a good idea what's happening here exactly? Or is there
> > already another topic or even a patch available?
Any chance that you can capture SYSRQ-T output via serial or
netconsole, so we can see the stacktrace and what the other CPUs are
doing, if they are doing anything.
Thanks,
tglx
>-----Original Message-----
>From: [email protected] [mailto:[email protected]]
>Sent: Tuesday, March 11, 2008 3:31 AM
>To: Jike Song
>Cc: Linux Kernel; Pallipadi, Venkatesh; Ingo Molnar; Thomas
>Gleixner; Brown, Len
>Subject: Re: endless loop in native_flush_tlb_others in smp_64.c
>
>On Tuesday 11 March 2008 10:55:40 Jike Song wrote:
>> On Tue, Mar 11, 2008 at 6:16 AM, Chr <[email protected]> wrote:
>> > Hi,
>> > ever since I moved to 2.6.25-rcY (Y should be between 2 and 5!).
>> > I've seen several, but really hard-to-catch instant freezes on my
>> > AMD64 Athlon X2 4200+ system...
>>
>> Here I guess 2.6.24 is fine for you?
>Yes, 2.6.24(.3) is fine!
>
>> > Most of them happend in X.org so at first I thought it
>had something to
>> > do with the NVIDIA module... BUT, one time it froze "a way
>before" the
>> > module could get loaded...
>> >
>> > ---
>> > SYSRQ-P revealed that the CPU were looping inside:
>> >
>> > smp_64.c native_flush_tlb_others:
>> > assembler code:
>> > < 1ee: f3 90 pause
>> > < 1f0: f6 45 00 03 testb $0x3,0x0(%rbp)
>> > < 1f4: 75 f8 jne 1ee
>> > <native_flush_tlb_others+0x5f>
>> >
>> > also known as: (in C)
>> >
>> > while (!cpus_empty(f->flush_cpumask))
>> > cpu_relax();
>> >
>> > So... has anyone a good idea what's happening here
>exactly? Or is there
>> > already another topic or even a patch available?
>>
>> Would you please attach your config file? Do you have
>CONFIG_CPU_IDLE set?
>(Attached). Yes, CONFIG_CPU_IDLE is enabled! I guess I should
>now disable it,
>try again and report back, right?! ;-)
I don't think it is related to CPU_IDLE. SYSRQ-T that Thomas asked
should give more clues.
Thanks,
Venki
On Tuesday 11 March 2008 12:09:24 you wrote:
> On Tue, 11 Mar 2008, Jike Song wrote:
>
> Any chance that you can capture SYSRQ-T output via serial or
> netconsole, so we can see the stacktrace and what the other CPUs are
> doing, if they are doing anything.
this time with a 2.6.25-rc4-wl: (unfortunatly tainted again)
the serial console seems to work: GFPs all over the place...
take a look here:
http://www.pastebin.ca/938757
Since I get so many different Oopses. I'm beginning to suspect my
fancy JFS/ReiserFS/Ext3:DM-Crypt:LVM2:MD(Raid1) combo causes
memory corruptions/leaks/voodoo...
like this other tragic incident:
loop0 D ffff810079331bd0 0 15716 2
ffff810079331b40 0000000000000046 ffff810062295c90 ffffffff804028e0
ffff810069608800 ffff810079331af0 ffffc20010af7040 ffffffff805f6700
ffffffff805f6700 ffffffff805f2f50 ffffffff805f6700 ffff81007a7df830
Call Trace:
[<ffffffff804028e0>] __split_bio+0x367/0x378
[<ffffffff8033e442>] generic_unplug_device+0x18/0x24
[<ffffffff804040b5>] dm_table_unplug_all+0x2a/0x3d
[<ffffffff802930c5>] sync_buffer+0x0/0x3f
[<ffffffff8048476d>] io_schedule+0x28/0x34
[<ffffffff80293100>] sync_buffer+0x3b/0x3f
[<ffffffff8048499e>] __wait_on_bit+0x40/0x6e
[<ffffffff802930c5>] sync_buffer+0x0/0x3f
[<ffffffff80484a38>] out_of_line_wait_on_bit+0x6c/0x78
[<ffffffff8023eb3d>] wake_bit_function+0x0/0x23
[<ffffffff802932b7>] ll_rw_block+0x8c/0xaf
[<ffffffff8029385b>] __block_prepare_write+0x366/0x3b9
[<ffffffff802e2a1c>] ext3_get_block+0x0/0xf9
[<ffffffff8029394b>] block_write_begin+0x78/0xc9
[<ffffffff802e3f1f>] ext3_write_begin+0xeb/0x1aa
[<ffffffff802e2a1c>] ext3_get_block+0x0/0xf9
[<ffffffff803b5928>] do_lo_send_aops+0x9f/0x177
[<ffffffff803b5889>] do_lo_send_aops+0x0/0x177
[<ffffffff803b5732>] loop_thread+0x2ce/0x425
[<ffffffff803b5464>] loop_thread+0x0/0x425
[<ffffffff8023e9ed>] kthread+0x47/0x76
[<ffffffff80229404>] schedule_tail+0x28/0x5c
[<ffffffff8020be68>] child_rip+0xa/0x12
[<ffffffff8023e9a6>] kthread+0x0/0x76
[<ffffffff8020be5e>] child_rip+0x0/0x12
situation: the system died after writing >2 Gb from /dev/zero
(gosh, about only 1Mb/s-500kb/s!!) into a file in a _mounted_
loopdevice of a old-hdd-image-file on a jfs/dm-crypt/lvm2 combo.
BTW: bisect is still running... the regression seems to have sneaked in
between 2.6.24 and 2.6.25-rc1 however 4000 diffs will take a while...
(it takes so long since the raid has to resync each reboot...
Thank *** that this is just a stress-testing system that can take some
beating without _failing_ apart. ;-) )
Regards,
Chr