Hmmm ... 2.5.43-mm2 and 2.5.44 work fine, but 2.5.44-mm1 (and mm2)
panic consistently on boot for a 16 way NUMA-Q.
Normally this box will boot with TSC on or off. If anyone has any pointers as
to what's changed in the mm patchset going from 43-mm2 to 44-mm1 that
might have touched this area (I can't see anything), please poke me in the
eye. Otherwise I'll just keep digging into it ....
Diff of bootlogs with TSC on below. As per usual for SMP panics, it's garbage ;-(
BIOS bug, IO-APIC#7 ID 13 is already used!...
... fixing up to 9. (tell your hw vendor)
...changing IO-APIC physical APIC ID to 9 ... ok.
-..TIMER: vector=0x31 pin1=2 pin2=0
-testing the IO APIC.......................
-
-
-
-
-
-
-
-
+ge<4>al prot, 5-19 fault: 0000
+1 <4>C 5-22 3
+ EIP: , 6-060:[<0, 7-008>], 7-8ot ta, 7-18, 7-19S, 7-200, 2
+1e<4>, 7-000, 7-2eb, 8-0321c60 ecx: c0320, 9-8 edx: 0, 9-180, 9-19:<4>, 9-20
, 9-21 c3a38000 ebp: 00000001 esp: c3a39f14
+ector=0x31 pin1=2 pin2=0
+ 0068
+Process (pid: 49194, threadinf<3>..38000 task=f01c07f6)mer notStack: ted to IO
APIC
+00...t00000011 set up timer (IRQ0) through t e 8259A ... 0320c24 . found n 0)
...a c02dc960 c02c4800 00000000 00000000 <4>c3d.
+64..00000046 set uf40 mer ual Wire IRQ0000000 00128868 c02b1e3c 0004200
0 c01078cc k<4>00nu0003 <4>c02b2128 ourcCall160.
+:number [<c011d745>13 tasklet_hi_action+0x85/0xe0
+O-APIC #14 re] ido_ssft24.
+xnumber of [-AP108f40>] ido_IRQ+0x100/0x1mber [<c01078c #5 registn_int 24.
++numb/0x20
+O-APIC #8d72>] trelease.
+nnumberem+0xIO-APIC #7 [<c0118ccd 24.prinumber o5f IO-APIC
+8Code: ters: 24.
+anue.er of IO-A0IC #9 r panic: Aiee, killingngnterrup APIC...............rrupt
handl
+ - not syncing
+. .
+.... register #00: 0D000000
If I patch it so that I can turn TSC off, and do so, I now get:
..TIMER: vector=0x31 pin1=2 pin2=0
..MP-BIOS bug: 8254 timer not connected to IO-APIC
...trying to set up timer (IRQ0) through the 8259A ...
..... (found pin 0) ... failed.
...trying to set up timer as Virtual Wire IRQ... failed.
...trying to set up timer as ExtINT IRQ... failed :(.
Kernel panic: IO-APIC + timer doesn't work! pester [email protected]
which never used to happen before ... a good boot does this
... fixing up to 9. (tell your hw vendor)
...changing IO-APIC physical APIC ID to 9 ... ok.
..TIMER: vector=0x31 pin1=2 pin2=0
testing the IO APIC.......................
.................................... done.
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 699.0999 MHz.
..... host bus clock speed is 99.0999 MHz.
"Martin J. Bligh" wrote:
>
> Hmmm ... 2.5.43-mm2 and 2.5.44 work fine, but 2.5.44-mm1 (and mm2)
> panic consistently on boot for a 16 way NUMA-Q.
>
> Normally this box will boot with TSC on or off. If anyone has any pointers as
> to what's changed in the mm patchset going from 43-mm2 to 44-mm1 that
> might have touched this area (I can't see anything), please poke me in the
> eye. Otherwise I'll just keep digging into it ....
>
Is possibly the code which defers the allocation of the per-cpu
memory until the secondary processors are being brought online.
I've decided to toss that. It's causing some grief for architectures,
and only buys us 10k * (NR_CPUS - nr-cpus) worth of memory anyway.
Well. It would be useful for NUMA to be able to place the per-cpu storage
into node-local memory. But the code doesn't do that. It just uses
kmalloc on the boot cpu, and we don't have an alloc_pages_for_another_cpu()
API..
>> Hmmm ... 2.5.43-mm2 and 2.5.44 work fine, but 2.5.44-mm1 (and mm2)
>> panic consistently on boot for a 16 way NUMA-Q.
>>
>> Normally this box will boot with TSC on or off. If anyone has any pointers as
>> to what's changed in the mm patchset going from 43-mm2 to 44-mm1 that
>> might have touched this area (I can't see anything), please poke me in the
>> eye. Otherwise I'll just keep digging into it ....
>>
>
>
> Is possibly the code which defers the allocation of the per-cpu
> memory until the secondary processors are being brought online.
Aha ... ;-) thanks for the pointer. Will poke at that.
> I've decided to toss that. It's causing some grief for architectures,
> and only buys us 10k * (NR_CPUS - nr-cpus) worth of memory anyway.
Hmmm ... I guess people with SMP normally have lots of RAM anyway,
and those few that care could just config NR_CPUS down. Shame though,
automagical is much nicer. Maybe we can work out exactly why it's broken
instead.
> Well. It would be useful for NUMA to be able to place the per-cpu storage
> into node-local memory. But the code doesn't do that. It just uses
> kmalloc on the boot cpu, and we don't have an alloc_pages_for_another_cpu()
> API..
Well at a page granularity level you have alloc_pages_node(cpu_to_node).
I suppose technically that's not guaranteed, but on boot it'll never fall back.
If you need it in permanent KVA, you can remap it into the vmalloc reserve
area.
Why do you need to do it that late, should be able to do it once you've parsed
mptables? alloc_bootmem / alloc_bootmem_node ?
M.
"Martin J. Bligh" wrote:
>
> >> Hmmm ... 2.5.43-mm2 and 2.5.44 work fine, but 2.5.44-mm1 (and mm2)
> >> panic consistently on boot for a 16 way NUMA-Q.
> >>
> >> Normally this box will boot with TSC on or off. If anyone has any pointers as
> >> to what's changed in the mm patchset going from 43-mm2 to 44-mm1 that
> >> might have touched this area (I can't see anything), please poke me in the
> >> eye. Otherwise I'll just keep digging into it ....
> >>
> >
> >
> > Is possibly the code which defers the allocation of the per-cpu
> > memory until the secondary processors are being brought online.
>
> Aha ... ;-) thanks for the pointer. Will poke at that.
The kgdb code plays around at that level too. A patch -R of
kgdb.patch would be interesting.
> > I've decided to toss that. It's causing some grief for architectures,
> > and only buys us 10k * (NR_CPUS - nr-cpus) worth of memory anyway.
>
> Hmmm ... I guess people with SMP normally have lots of RAM anyway,
> and those few that care could just config NR_CPUS down. Shame though,
> automagical is much nicer. Maybe we can work out exactly why it's broken
> instead.
>
> > Well. It would be useful for NUMA to be able to place the per-cpu storage
> > into node-local memory. But the code doesn't do that. It just uses
> > kmalloc on the boot cpu, and we don't have an alloc_pages_for_another_cpu()
> > API..
>
> Well at a page granularity level you have alloc_pages_node(cpu_to_node).
> I suppose technically that's not guaranteed, but on boot it'll never fall back.
> If you need it in permanent KVA, you can remap it into the vmalloc reserve
> area.
That would suit.
> Why do you need to do it that late, should be able to do it once you've parsed
> mptables? alloc_bootmem / alloc_bootmem_node ?
Lack of architecture hooks, maybe? That'd be one for Rusty.
>> > Is possibly the code which defers the allocation of the per-cpu
>> > memory until the secondary processors are being brought online.
>>
>> Aha ... ;-) thanks for the pointer. Will poke at that.
Humpf. Well I tried -mm3 (after backing out Rusty's patch), and
it still has the problem ;-( Must be something else.
> The kgdb code plays around at that level too. A patch -R of
> kgdb.patch would be interesting.
OK ... will try that then brute force and ignorance (binary chop
search) I guess.
M.
>> The kgdb code plays around at that level too. A patch -R of
>> kgdb.patch would be interesting.
>
> OK ... will try that then brute force and ignorance (binary chop
> search) I guess.
Your psychic powers seem to be on form. Was kgdb. Worked in
41-mm1 for me on the same machine though.
M.