Even though I booted with notsc, rdtsc happens here and appears to
generate #GP or some such nonsense:
Restoring NMI vector
Booting processor 14/52 eip 2000
Initializing CPU#14
masked ExtINT on CPU#14
Leaving ESR disabled.
Calibrating delay loop...
Program received signal SIGEMT, Emulation trap.
__rdtsc_delay (loops=69700) at delay.c:40
40 delay.c: No such file or directory.
in delay.c
(gdb) bt
#0 __rdtsc_delay (loops=69700) at delay.c:40
#1 0xc019f31d in __delay (loops=69700) at delay.c:63
#2 0xc019f384 in __const_udelay (xloops=429400) at delay.c:74
#3 0xc02c1709 in do_boot_cpu (apicid=52) at smpboot.c:865
#4 0xc02c1a4e in smp_boot_cpus (max_cpus=4294967295) at smpboot.c:1095
#5 0xc02c1c70 in smp_prepare_cpus (max_cpus=4294967295) at smpboot.c:1199
#6 0xc01050d7 in init (unused=0x0) at main.c:547
(gdb) p $eip
$3 = (void *) 0xc019f2d4
(gdb) disassemble $eip
Dump of assembler code for function __rdtsc_delay:
0xc019f2c0 <__rdtsc_delay>: push %ebp
0xc019f2c1 <__rdtsc_delay+1>: mov %esp,%ebp
0xc019f2c3 <__rdtsc_delay+3>: push %ebx
0xc019f2c4 <__rdtsc_delay+4>: mov 0x8(%ebp),%ebx
0xc019f2c7 <__rdtsc_delay+7>: rdtsc
0xc019f2c9 <__rdtsc_delay+9>: mov %eax,%ecx
0xc019f2cb <__rdtsc_delay+11>: nop
0xc019f2cc <__rdtsc_delay+12>: lea 0x0(%esi,1),%esi
0xc019f2d0 <__rdtsc_delay+16>: repz nop
0xc019f2d2 <__rdtsc_delay+18>: rdtsc
0xc019f2d4 <__rdtsc_delay+20>: sub %ecx,%eax
0xc019f2d6 <__rdtsc_delay+22>: cmp %ebx,%eax
0xc019f2d8 <__rdtsc_delay+24>: jb 0xc019f2d0 <__rdtsc_delay+16>
0xc019f2da <__rdtsc_delay+26>: pop %ebx
0xc019f2db <__rdtsc_delay+27>: mov %ebp,%esp
0xc019f2dd <__rdtsc_delay+29>: pop %ebp
0xc019f2de <__rdtsc_delay+30>: ret
End of assembler dump.
Does this help (on top of John's TSC patch in rollup 0)?
diff -urN -X /home/fletch/.diff.exclude virgin/include/asm-i386/cpufeature.h cpu_has_tsc/include/asm-i386/cpufeature.h
--- virgin/include/asm-i386/cpufeature.h Mon Sep 9 10:35:04 2002
+++ cpu_has_tsc/include/asm-i386/cpufeature.h Thu Sep 19 20:07:11 2002
@@ -70,7 +70,11 @@
#define cpu_has_vme boot_cpu_has(X86_FEATURE_VME)
#define cpu_has_de boot_cpu_has(X86_FEATURE_DE)
#define cpu_has_pse boot_cpu_has(X86_FEATURE_PSE)
-#define cpu_has_tsc boot_cpu_has(X86_FEATURE_TSC)
+#ifdef CONFIG_X86_TSC
+#define cpu_has_tsc boot_cpu_has(X86_FEATURE_TSC)
+#else
+#define cpu_has_tsc (0)
+#endif
#define cpu_has_pae boot_cpu_has(X86_FEATURE_PAE)
#define cpu_has_pge boot_cpu_has(X86_FEATURE_PGE)
#define cpu_has_apic
boot_cpu_has(X86_FEATURE_APIC)
--On Thursday, September 19, 2002 5:10 PM -0700 William Lee Irwin III <[email protected]> wrote:
> Even though I booted with notsc, rdtsc happens here and appears to
> generate #GP or some such nonsense:
>
> Restoring NMI vector
> Booting processor 14/52 eip 2000
> Initializing CPU#14
> masked ExtINT on CPU#14
> Leaving ESR disabled.
> Calibrating delay loop...
> Program received signal SIGEMT, Emulation trap.
> __rdtsc_delay (loops=69700) at delay.c:40
> 40 delay.c: No such file or directory.
> in delay.c
> (gdb) bt
># 0 __rdtsc_delay (loops=69700) at delay.c:40
># 1 0xc019f31d in __delay (loops=69700) at delay.c:63
># 2 0xc019f384 in __const_udelay (xloops=429400) at delay.c:74
># 3 0xc02c1709 in do_boot_cpu (apicid=52) at smpboot.c:865
># 4 0xc02c1a4e in smp_boot_cpus (max_cpus=4294967295) at smpboot.c:1095
># 5 0xc02c1c70 in smp_prepare_cpus (max_cpus=4294967295) at smpboot.c:1199
># 6 0xc01050d7 in init (unused=0x0) at main.c:547
> (gdb) p $eip
> $3 = (void *) 0xc019f2d4
> (gdb) disassemble $eip
> Dump of assembler code for function __rdtsc_delay:
> 0xc019f2c0 <__rdtsc_delay>: push %ebp
> 0xc019f2c1 <__rdtsc_delay+1>: mov %esp,%ebp
> 0xc019f2c3 <__rdtsc_delay+3>: push %ebx
> 0xc019f2c4 <__rdtsc_delay+4>: mov 0x8(%ebp),%ebx
> 0xc019f2c7 <__rdtsc_delay+7>: rdtsc
> 0xc019f2c9 <__rdtsc_delay+9>: mov %eax,%ecx
> 0xc019f2cb <__rdtsc_delay+11>: nop
> 0xc019f2cc <__rdtsc_delay+12>: lea 0x0(%esi,1),%esi
> 0xc019f2d0 <__rdtsc_delay+16>: repz nop
> 0xc019f2d2 <__rdtsc_delay+18>: rdtsc
> 0xc019f2d4 <__rdtsc_delay+20>: sub %ecx,%eax
> 0xc019f2d6 <__rdtsc_delay+22>: cmp %ebx,%eax
> 0xc019f2d8 <__rdtsc_delay+24>: jb 0xc019f2d0 <__rdtsc_delay+16>
> 0xc019f2da <__rdtsc_delay+26>: pop %ebx
> 0xc019f2db <__rdtsc_delay+27>: mov %ebp,%esp
> 0xc019f2dd <__rdtsc_delay+29>: pop %ebp
> 0xc019f2de <__rdtsc_delay+30>: ret
> End of assembler dump.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
>
>> Does this help (on top of John's TSC patch in rollup 0)?
>
> Nope. But I believe I found the root cause: it apparently takes
> long enough to kick all the cpus the NMI oopser goes off during
> one of the many long ints-off sections in the bootstrap phase. My
> burning question now is why this only showed up in 2.5.36. Somehow
> I mistook a rather blatant (c.f. SIGEMT) NMI oops for TSD %cr4 #GP.
>
> I'm going to guess the NMI oopser was not eager enough to trip
> beforehand and recent changes repaired that. Is this close?
>
> If so, it's probably not worth mucking around with the bootstrap
> sequence to deal with something this minor. It's not like it can
> be mistaken for having hung, as console output is very consistent.
> Maybe we should give NUMA-Q a couple of minutes instead of 5s?
Nah, just recode the boot sequence to make them all boot in
parallel ;-)
M.
On Thu, Sep 19, 2002 at 08:07:06PM -0700, Martin J. Bligh wrote:
> Does this help (on top of John's TSC patch in rollup 0)?
Nope. But I believe I found the root cause: it apparently takes
long enough to kick all the cpus the NMI oopser goes off during
one of the many long ints-off sections in the bootstrap phase. My
burning question now is why this only showed up in 2.5.36. Somehow
I mistook a rather blatant (c.f. SIGEMT) NMI oops for TSD %cr4 #GP.
I'm going to guess the NMI oopser was not eager enough to trip
beforehand and recent changes repaired that. Is this close?
If so, it's probably not worth mucking around with the bootstrap
sequence to deal with something this minor. It's not like it can
be mistaken for having hung, as console output is very consistent.
Maybe we should give NUMA-Q a couple of minutes instead of 5s?
Thanks,
Bill
At some point in the past, I wrote:
>> If so, it's probably not worth mucking around with the bootstrap
>> sequence to deal with something this minor. It's not like it can
>> be mistaken for having hung, as console output is very consistent.
>> Maybe we should give NUMA-Q a couple of minutes instead of 5s?
On Thu, Sep 19, 2002 at 09:00:09PM -0700, Martin J. Bligh wrote:
> Nah, just recode the boot sequence to make them all boot in
> parallel ;-)
> M.
Do you think cpu wakeup alone could be doing this? If so, then doing
that bit would be relatively isolated (though a slightly larger diff
than changing an NMI oopser timeout).
Does 0xFF broadcast cluster, broadcast low nybble work or is waking
them a cluster at a time required? This thing is not swift to boot...
Thanks,
Bill
> At some point in the past, I wrote:
>>> If so, it's probably not worth mucking around with the bootstrap
>>> sequence to deal with something this minor. It's not like it can
>>> be mistaken for having hung, as console output is very consistent.
>>> Maybe we should give NUMA-Q a couple of minutes instead of 5s?
>
> On Thu, Sep 19, 2002 at 09:00:09PM -0700, Martin J. Bligh wrote:
>> Nah, just recode the boot sequence to make them all boot in
>> parallel ;-)
>
> Do you think cpu wakeup alone could be doing this? If so, then doing
> that bit would be relatively isolated (though a slightly larger diff
> than changing an NMI oopser timeout).
Seems odd that you didn't get it in previous versions ... ? But
running the NMI oops detection whilst stealing NMIs in order to
bootstrap the CPUs is probably a Bad Idea (tm).
> Does 0xFF broadcast cluster, broadcast low nybble work or is waking
> them a cluster at a time required? This thing is not swift to boot...
Well, you don't want to send an NMI to yourself (the BSP), I presume?
That would probably make it unhappy ;-) The NMIs have to be logically addressed, which precludes the allbutself thingy, IIRC. So you have
a choice of serialised unicast (which is probably quite fast enough)
or cluster at a time, being careful to exclude yourself when you do
the BSP's cluster.
But I think you'd be far better off just disabling the NMI oopser
until we've booted - the dual use is too much nefarious incest for
my liking.
M.