hi,
kernel panic on mainline(commit: 48d212a2eecaca) with a large system, which
has 120Gb RAM & 8 numa nodes:
... [cut here] ...
[ 3.404017] Call Trace:
[ 3.404553] [<ffffffff810afd19>] find_busiest_group+0x39/0x4b0
[ 3.406188] [<ffffffff810b0295>] load_balance+0x105/0xa50
[ 3.407444] [<ffffffff810ce2ed>] ? trace_hardirqs_off+0xd/0x10
[ 3.408695] [<ffffffff810aa1cf>] ? local_clock+0x6f/0x80
[ 3.409789] [<ffffffff810b11b0>] idle_balance+0x130/0x2d0
[ 3.410879] [<ffffffff810b10d0>] ? idle_balance+0x50/0x2d0
[ 3.411996] [<ffffffff8167f340>] __schedule+0x910/0xa00
[ 3.413204] [<ffffffff8167f769>] schedule+0x29/0x70
[ 3.414324] [<ffffffff8102352f>] cpu_idle+0x12f/0x140
[ 3.415433] [<ffffffff81667765>] start_secondary+0x262/0x264
[ 3.416763] Code: 44 8b bd 7c ff ff ff 45 85 ff 0f 85 30 02 00 00 48
8b bd 48 ff ff ff 48 8b 4f 10 4c 8b 45 98 8b 71 04 31 d2 4c 89 c0 48 c1
e0 0a <48> f7 f6 48 8b 75 a0 48 85 f6 48 89 c7 49 89 c1 48 89 45 90 0f
[ 3.420335] RIP [<ffffffff810af93b>] update_sd_lb_stats+0x27b/0x620
[ 3.421664] RSP <ffff88041922fb48>
[ 3.422473] ---[ end trace 04b848dd1c06d585 ]---
[ 3.423472] Kernel panic - not syncing: Attempted to kill the idle task!
whole console log is here:
http://www.sanweiying.org/download/debug_kernel/kernel_panic_48d212a2eecaca
and config file is
http://www.sanweiying.org/download/debug_kernel/config_48d212a2eecaca
and I have tested different branches, more details is available in
https://lkml.org/lkml/2012/6/6/704
here I do a simple summary:
a), mainline,
v3.4: no such panic
b), mainline, v3.5-rc1(commit: 48d212a2eecaca), yes
c), Andrea's tree, autonuma15 yes
d), tip/master(commit:b2f5ce55c4e68370) no such panic
let me know if you need any info.
Thanks,
Zhouping
On Sat, 2012-06-09 at 01:13 +0800, ZhouPing Liu wrote:
> kernel panic on mainline(commit: 48d212a2eecaca) with a large system, which
> has 120Gb RAM & 8 numa nodes:
>
> ... [cut here] ...
> [ 3.404017] Call Trace:
> [ 3.404553] [<ffffffff810afd19>] find_busiest_group+0x39/0x4b0
> [ 3.406188] [<ffffffff810b0295>] load_balance+0x105/0xa50
> [ 3.407444] [<ffffffff810ce2ed>] ? trace_hardirqs_off+0xd/0x10
> [ 3.408695] [<ffffffff810aa1cf>] ? local_clock+0x6f/0x80
> [ 3.409789] [<ffffffff810b11b0>] idle_balance+0x130/0x2d0
> [ 3.410879] [<ffffffff810b10d0>] ? idle_balance+0x50/0x2d0
> [ 3.411996] [<ffffffff8167f340>] __schedule+0x910/0xa00
> [ 3.413204] [<ffffffff8167f769>] schedule+0x29/0x70
> [ 3.414324] [<ffffffff8102352f>] cpu_idle+0x12f/0x140
> [ 3.415433] [<ffffffff81667765>] start_secondary+0x262/0x264
> [ 3.416763] Code: 44 8b bd 7c ff ff ff 45 85 ff 0f 85 30 02 00 00 48
> 8b bd 48 ff ff ff 48 8b 4f 10 4c 8b 45 98 8b 71 04 31 d2 4c 89 c0 48 c1
> e0 0a <48> f7 f6 48 8b 75 a0 48 85 f6 48 89 c7 49 89 c1 48 89 45 90 0f
> [ 3.420335] RIP [<ffffffff810af93b>] update_sd_lb_stats+0x27b/0x620
> [ 3.421664] RSP <ffff88041922fb48>
> [ 3.422473] ---[ end trace 04b848dd1c06d585 ]---
> [ 3.423472] Kernel panic - not syncing: Attempted to kill the idle task!
> here I do a simple summary:
> b), mainline, v3.5-rc1(commit: 48d212a2eecaca), yes
> d), tip/master(commit:b2f5ce55c4e68370) no such panic
>
> let me know if you need any info.
There's a number of patches in tip/sched/urgent that I think fix this
(hence your D) and these should make their way to Linus shortly.
That said, can you provide me your node distance table so I can verify
locally?
cat /sys/devices/system/node/node*/distance
----- Original Message -----
> From: "Peter Zijlstra" <[email protected]>
> To: "ZhouPing Liu" <[email protected]>
> Cc: "Andrea Arcangeli" <[email protected]>, "Linus Torvalds" <[email protected]>, "Hillf Danton"
> <[email protected]>, [email protected], "LKML" <[email protected]>
> Sent: Saturday, June 9, 2012 2:46:57 AM
> Subject: Re: Kernel panic - not syncing: Attempted to kill the idle task!
>
> On Sat, 2012-06-09 at 01:13 +0800, ZhouPing Liu wrote:
>
> > kernel panic on mainline(commit: 48d212a2eecaca) with a large
> > system, which
> > has 120Gb RAM & 8 numa nodes:
> >
> > ... [cut here] ...
> > [ 3.404017] Call Trace:
> > [ 3.404553] [<ffffffff810afd19>] find_busiest_group+0x39/0x4b0
> > [ 3.406188] [<ffffffff810b0295>] load_balance+0x105/0xa50
> > [ 3.407444] [<ffffffff810ce2ed>] ? trace_hardirqs_off+0xd/0x10
> > [ 3.408695] [<ffffffff810aa1cf>] ? local_clock+0x6f/0x80
> > [ 3.409789] [<ffffffff810b11b0>] idle_balance+0x130/0x2d0
> > [ 3.410879] [<ffffffff810b10d0>] ? idle_balance+0x50/0x2d0
> > [ 3.411996] [<ffffffff8167f340>] __schedule+0x910/0xa00
> > [ 3.413204] [<ffffffff8167f769>] schedule+0x29/0x70
> > [ 3.414324] [<ffffffff8102352f>] cpu_idle+0x12f/0x140
> > [ 3.415433] [<ffffffff81667765>] start_secondary+0x262/0x264
> > [ 3.416763] Code: 44 8b bd 7c ff ff ff 45 85 ff 0f 85 30 02 00
> > 00 48
> > 8b bd 48 ff ff ff 48 8b 4f 10 4c 8b 45 98 8b 71 04 31 d2 4c 89 c0
> > 48 c1
> > e0 0a <48> f7 f6 48 8b 75 a0 48 85 f6 48 89 c7 49 89 c1 48 89 45 90
> > 0f
> > [ 3.420335] RIP [<ffffffff810af93b>]
> > update_sd_lb_stats+0x27b/0x620
> > [ 3.421664] RSP <ffff88041922fb48>
> > [ 3.422473] ---[ end trace 04b848dd1c06d585 ]---
> > [ 3.423472] Kernel panic - not syncing: Attempted to kill the
> > idle task!
>
> > here I do a simple summary:
>
> > b), mainline, v3.5-rc1(commit: 48d212a2eecaca), yes
> > d), tip/master(commit:b2f5ce55c4e68370) no such
> > panic
> >
> > let me know if you need any info.
>
> There's a number of patches in tip/sched/urgent that I think fix this
> (hence your D) and these should make their way to Linus shortly.
>
> That said, can you provide me your node distance table so I can
> verify
> locally?
>
> cat /sys/devices/system/node/node*/distance
# cat /sys/devices/system/node/node*/distance
10 17 17 24 24 24 30 30
18 10 30 18 18 24 24 24
18 24 10 24 24 17 30 30
24 18 23 10 24 17 17 30
24 17 24 24 10 18 30 18
31 24 17 18 18 10 24 24
30 24 30 17 24 24 10 18
30 24 30 24 17 24 17 10
--
Thanks,
Zhouping
On Fri, 2012-06-08 at 21:38 -0400, Zhouping Liu wrote:
> # cat /sys/devices/system/node/node*/distance
> 10 17 17 24 24 24 30 30
> 18 10 30 18 18 24 24 24
> 18 24 10 24 24 17 30 30
> 24 18 23 10 24 17 17 30
> 24 17 24 24 10 18 30 18
> 31 24 17 18 18 10 24 24
> 30 24 30 17 24 24 10 18
> 30 24 30 24 17 24 17 10
You have to be kidding me right? That thing is a complete trainwreck,
what idiot vendor did this?
If you boot that machine again, does it have the same stupid table or
are we staring at white-noise?
On Mon, Jun 11, 2012 at 03:27:48PM +0200, Peter Zijlstra wrote:
> On Fri, 2012-06-08 at 21:38 -0400, Zhouping Liu wrote:
> > # cat /sys/devices/system/node/node*/distance
> > 10 17 17 24 24 24 30 30
> > 18 10 30 18 18 24 24 24
> > 18 24 10 24 24 17 30 30
> > 24 18 23 10 24 17 17 30
> > 24 17 24 24 10 18 30 18
> > 31 24 17 18 18 10 24 24
> > 30 24 30 17 24 24 10 18
> > 30 24 30 24 17 24 17 10
>
> You have to be kidding me right? That thing is a complete trainwreck,
> what idiot vendor did this?
HP ProLiant DL785 G6
http://www.sanweiying.org/download/debug_kernel/kernel_panic_48d212a2eecaca
contains dmesg and there's some serious fun with how the box boots:
node 0, cpus: 0, 8, 16...
node 1, cpus: 1, 9, 17...
> If you boot that machine again, does it have the same stupid table or
> are we staring at white-noise?
I'll bet it does.
--
Regards/Gruss,
Boris.
On 06/11/2012 09:27 PM, Peter Zijlstra wrote:
> On Fri, 2012-06-08 at 21:38 -0400, Zhouping Liu wrote:
>> # cat /sys/devices/system/node/node*/distance
>> 10 17 17 24 24 24 30 30
>> 18 10 30 18 18 24 24 24
>> 18 24 10 24 24 17 30 30
>> 24 18 23 10 24 17 17 30
>> 24 17 24 24 10 18 30 18
>> 31 24 17 18 18 10 24 24
>> 30 24 30 17 24 24 10 18
>> 30 24 30 24 17 24 17 10
> You have to be kidding me right? That thing is a complete trainwreck,
> what idiot vendor did this?
it's a HP's machine.
If I understand correctly, you meant the hardware has a bad configuration,
just serious :) could you explain the reason? (you can ignore it if
it's a stupid question)
>
> If you boot that machine again, does it have the same stupid table or
> are we staring at white-noise?
indeed, it's always the same table.
And the issues is gone in the latest Linus tree(commit b84297197ce60),
I'm not sure which patches fixed the issue.
Thanks,
Zhouping
On 06/12/2012 11:54 AM, Zhouping Liu wrote:
> On 06/11/2012 09:27 PM, Peter Zijlstra wrote:
>> On Fri, 2012-06-08 at 21:38 -0400, Zhouping Liu wrote:
>>> # cat /sys/devices/system/node/node*/distance
>>> 10 17 17 24 24 24 30 30
>>> 18 10 30 18 18 24 24 24
>>> 18 24 10 24 24 17 30 30
>>> 24 18 23 10 24 17 17 30
>>> 24 17 24 24 10 18 30 18
>>> 31 24 17 18 18 10 24 24
>>> 30 24 30 17 24 24 10 18
>>> 30 24 30 24 17 24 17 10
>> You have to be kidding me right? That thing is a complete trainwreck,
>> what idiot vendor did this?
>
> it's a HP's machine.
> If I understand correctly, you meant the hardware has a bad
> configuration,
> just serious :) could you explain the reason? (you can ignore it if
> it's a stupid question)
^^^^it's a typo, sorry, I was willing to express my
*curious* (I recognized my bad English)
and I have one more machine which has the similar nodes table.
Thanks,
Zhouping
On Tue, 2012-06-12 at 11:54 +0800, Zhouping Liu wrote:
> On 06/11/2012 09:27 PM, Peter Zijlstra wrote:
> > On Fri, 2012-06-08 at 21:38 -0400, Zhouping Liu wrote:
> >> # cat /sys/devices/system/node/node*/distance
> >> 10 17 17 24 24 24 30 30
> >> 18 10 30 18 18 24 24 24
> >> 18 24 10 24 24 17 30 30
> >> 24 18 23 10 24 17 17 30
> >> 24 17 24 24 10 18 30 18
> >> 31 24 17 18 18 10 24 24
> >> 30 24 30 17 24 24 10 18
> >> 30 24 30 24 17 24 17 10
> > You have to be kidding me right? That thing is a complete trainwreck,
> > what idiot vendor did this?
>
> it's a HP's machine.
> If I understand correctly, you meant the hardware has a bad configuration,
> just serious :) could you explain the reason? (you can ignore it if
> it's a stupid question)
Not only a weird hardware setup (although if we go by this SLIT table,
then that too).
The table itself has various problems:
1) the table isn't symmetric; T(i,j) != T(j,i), for instance, the
distance from 0->1 is different from 1->0 (17 vs 18).
2) 4 nodes have 2 connections, 4 nodes have 3 connections. Which have 2
connections seems completely without pattern, see 4).
3) if we read it like: 10 (self), {17,18} 1 hop, {23,24} 2 hops,
{30,31} 3 hops, its still obviously wrong, see 1->2 and 2->1 (this goes
back to point 1 as well). 1->2 takes 3 hops while 2->1 takes 2 hops.
Going by the 1 hop connections (which aside from the 17 vs 18 mess) are
symmetric, both these should be 2 hops (1<->0<->2 in fact). There's
multiple such 'mistakes'.
4) take a piece of paper and draw a cube, mark each corner as a node,
then high-light the single hop edges and be awestruck by the creative
wiring.
This is by far the most 'creative' SLIT table I have ever seen and of
course its HP again.. those guys have the most shitty BIOS record ever.
Now we could 'fix' up this table by doing a min-symmetry filter over it,
but I'm tempted to just give up and do a single machine wide fall-back
domain when we find crappy tables like this.
On Tue, 2012-06-12 at 13:20 +0300, Linus Torvalds wrote:
> Peter, you are very fundamentally wrong if you think the distance
> array has to be symmetric. That is fundamentally not true for many
> typologies.
>
> The trivial example of a non symmetric case is just a simple
> unidirectional ring. The distance from n to n+1 is just one, but the
> distance from n+1 to n is n-1 hops.
>
> So don't try to say that distances have to be symmetric. That's just
> garbage.
Sure, I realize this, but the 17,18 thing isn't a ring. It looks like
something that should be symmetric but isn't.
On Tue, 2012-06-12 at 12:27 +0200, Peter Zijlstra wrote:
> On Tue, 2012-06-12 at 13:20 +0300, Linus Torvalds wrote:
> > Peter, you are very fundamentally wrong if you think the distance
> > array has to be symmetric. That is fundamentally not true for many
> > typologies.
> >
> > The trivial example of a non symmetric case is just a simple
> > unidirectional ring. The distance from n to n+1 is just one, but the
> > distance from n+1 to n is n-1 hops.
> >
> > So don't try to say that distances have to be symmetric. That's just
> > garbage.
>
> Sure, I realize this, but the 17,18 thing isn't a ring. It looks like
> something that should be symmetric but isn't.
Not a unidirectional one that is.. I played around a bit more and found
a shape that isn't too odd:
2
/ \
0 5
/ / \
1 --- / --- 3
\ / /
4 6
\ /
7
I would've crossed 0<->6 instead of 1<->3 so the 2/3 connected nodes are
spread better, but what do I know.
The only really odd thing is the numbering, which is what threw me.
But yeah, the possibility of uni-directional rings makes detecting
obvious crack tables harder, which is why I haven't got it -- yet.
On Tue, 2012-06-12 at 13:20 +0300, Linus Torvalds wrote:
> you are very fundamentally wrong if you think the distance array has
> to be symmetric.
I just checked, both QPI and HT are bi-directional links so anything
mid-sized x86 had better be symmetric. Big numa can of course stick in
their own interconnects and might do uni-directional stuff, although I
doubt anybody still does that.
The distance table I have for a 4096 cpu SGI machine is fully symmetric
for instance.
This of course doesn't mean we should make hard assumptions about this
in the scheduler -- and we don't. But in this case it does mean the HP
table is very funny indeed.