Here's a patch for those who have been plagued by APIC errors starting around
-rc1-ac6. I've submitted it to Alan, but since it has been affecting a
number of folks, I'm also posting it here for your consideration and review.
This fixes the APIC receive accept errors on the two machines we have that
were subject to it. Let me know if it doesn't work for you.
--
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
> This fixes the APIC receive accept errors on the two machines we have that
> were subject to it. Let me know if it doesn't work for you.
This patch works for my Asus P2B-DS with 2xPII(Deschutes)-400.
--
Leszek.
-- [email protected] 2:480/33.7 -- REAL programmers use INTEGERS --
-- speaking just for myself...
Hi Alan, James
This is what i have so far, i'll probably have time to really
debug it when i get home later today. The problem persists with Jame's
Summit patch applied too (just in case there were other fixes there).
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: OEM00000 Product ID: 0.1 APIC at: 0xFEE00000
Processor #0 Pentium(tm) Pro APIC version 17
Processor #1 Pentium(tm) Pro APIC version 17
Processor #2 Pentium(tm) Pro APIC version 17
Processor #3 Pentium(tm) Pro APIC version 17
I/O APIC #4 Version 17 at 0xFEC00000.
Enabling APIC mode: Flat. Using 1 I/O APICs
Processors: 4
[...]
ENABLING IO-APIC IRQs
Setting 4 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 4 ... ok.
..TIMER: vector=0x31 pin1=0 pin2=-1
<dead>
Around here the machine gets a vector 0x31 (timer) interrupt on CPU0 then
locks up since the destination cpu bitmask is 0, It also seems that the
code is trying to use logical apic id in places instead of the physical
apic id, i saw attempted deliveries to physical apic id 4 and 8, this can
possibly explain the APIC receive errors people were reporting?
Unfortunately this is the only info i have right now, i thought i'd be
able to gather more but looks like i'll have to wait till i get home.
As a control, the machine boots and runs 2.4.19-rc2
Regards,
Zwane Mwaikambo
Alan on a side note, would you like a patch to bring around some fixes
from 2.5 irq_balance? Your kernel can't boot UP w/ IOAPIC
--
function.linuxpower.ca
On Tue, 23 Jul 2002, Zwane Mwaikambo wrote:
> Around here the machine gets a vector 0x31 (timer) interrupt on CPU0 then
> locks up since the destination cpu bitmask is 0, It also seems that the
> code is trying to use logical apic id in places instead of the physical
> apic id, i saw attempted deliveries to physical apic id 4 and 8, this can
> possibly explain the APIC receive errors people were reporting?
Correction, the logical/physical apic id problem doesn't appear to be
there with the summit patch. What i'm currently seeing is a destination of
0 with a non flat/physical destination format.
--
function.linuxpower.ca
On Mon, 22 Jul 2002, James Cleverdon wrote:
> Here's a patch for those who have been plagued by APIC errors starting around
> -rc1-ac6. I've submitted it to Alan, but since it has
> been affecting a number of folks, I'm also posting it here for
> your consideration and review.
>
> This fixes the APIC receive accept errors on the two machines we have
> that were subject to it. Let me know if it doesn't work for you.
I does work with the dell 2400.
Same config I forwarded to the list, minus the RZ1000, PIIX, CMD640, and
pcmcia bits.
Regards
James Bourne
--
James Bourne, Supervisor Data Centre Operations
Mount Royal College, Calgary, AB, CA
http://www.mtroyal.ab.ca
******************************************************************************
This communication is intended for the use of the recipient to which it is
addressed, and may contain confidential, personal, and or privileged
information. Please contact the sender immediately if you are not the
intended recipient of this communication, and do not copy, distribute, or
take action relying on it. Any communication received in error, or
subsequent reply, should be deleted or destroyed.
******************************************************************************
"There are only 10 types of people in this world: those who
understand binary and those who don't."
On Mon, 2002-07-22 at 22:21, James Cleverdon wrote:
> Here's a patch for those who have been plagued by APIC errors starting around
> -rc1-ac6. I've submitted it to Alan, but since it has been affecting a
> number of folks, I'm also posting it here for your consideration and review.
>
> This fixes the APIC receive accept errors on the two machines we have that
> were subject to it. Let me know if it doesn't work for you.
Thanks. That worked for my Intel STL2 with 2 x P-III (Coppermine).
Steven
On Mon, 22 Jul 2002 21:21:04 -0700
James Cleverdon <[email protected]> wrote:
Thanks, it works now on both DELL MT 530 (2xPIII Xeon 1.5Ghz) and DELL 2450 ( 2xPIII Copermine 1gHZ).
Before, the MT 530 was halted on the SCSI probe while the 2450 was stucked with the APIC error message
on CPU #0
Thanks,
Philippe.
| This fixes the APIC receive accept errors on the two machines we have that
| were subject to it. Let me know if it doesn't work for you.
|
| --
| James Cleverdon
| IBM xSeries Linux Solutions
| {jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
|
On Tuesday 23 July 2002 05:11 am, Zwane Mwaikambo wrote:
> On Tue, 23 Jul 2002, Zwane Mwaikambo wrote:
> > Around here the machine gets a vector 0x31 (timer) interrupt on CPU0 then
> > locks up since the destination cpu bitmask is 0, It also seems that the
> > code is trying to use logical apic id in places instead of the physical
> > apic id, i saw attempted deliveries to physical apic id 4 and 8, this can
> > possibly explain the APIC receive errors people were reporting?
>
> Correction, the logical/physical apic id problem doesn't appear to be
> there with the summit patch. What i'm currently seeing is a destination of
> 0 with a non flat/physical destination format.
Drat! I thought I had all the logical vs. physical stuff straightened out.
Could you give this patch a try? It dumps all kinds of APIC state info.
You'll need to put a call to apic_state_dump() into check_timer() just after
the TIMER: printk.
(Hmmm.... Must clean up this patch and submit it to kdb as two new commands,
one for I/O APICs and one for local APICs....)
--
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
On Tue, 23 Jul 2002, James Cleverdon wrote:
> Drat! I thought I had all the logical vs. physical stuff straightened out.
> Could you give this patch a try? It dumps all kinds of APIC state info.
> You'll need to put a call to apic_state_dump() into check_timer() just after
> the TIMER: printk.
ID=0x02000000, LVR=0x00170011, TPR=0x00000000, ARB=0x00000002, PROCPRI=0x000000F0
DFR=0x0FFFFFFF, LDR=0x01000000, ICR=0x00088500, SPIV=0x000001FF, ICR=0x00088500,
ICR2=0x03000000, LVTT=0x00000000, LVTPC=0x00000000
LVT0=0x00010700, LVT1=0x00000400, LVTERR=0x000000FE
ISR: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
TMR: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
IRR: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
clustered_apic_mode=0, esr_disable=0, target_cpus=0x00 apic_broadcast_id=0x0F
raw_phys_apicid[]= 00 01 02 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00
cpu_2_logical_apicid[]= 01 01 02 08 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF
cpu_2_physical_apicid[]= 02 00 01 03 FF FF FF FF FF FF FF FF FF FF FF FF FF FF F
F FF FF FF FF FF FF FF FF FF FF FF FF FF
I/O APIC # 0:
00: 00000931:04000000 00010939:02000000 00010000:00000000 00000941:0F000000
04: 00000949:0F000000 00000951:0F000000 00010959:02000000 00000961:0F000000
08: 00000969:0F000000 00000971:0F000000 00000979:0F000000 00000981:0F000000
0C: 00000989:0F000000 00000991:0F000000 00000999:0F000000 000009A1:0F000000
10: 00010000:00000000 00010000:00000000 00010000:00000000 00010000:00000000
14: 00010000:00000000 00010000:00000000 00010000:00000000 00010000:00000000
> (Hmmm.... Must clean up this patch and submit it to kdb as two new commands,
> one for I/O APICs and one for local APICs....)
i'd vote for that =) except for one thing.
+ /* APICs tend to spasm when they get errors. Disable the error intr. */
+ apic_write_around(APIC_LVTERR, ERROR_APIC_VECTOR | APIC_LVT_MASKED);
Isn't that a bit drastic?
Regards,
Zwane
--
function.linuxpower.ca
On Wed, 24 Jul 2002 17:26:49 +0200 (SAST), Zwane Mwaikambo wrote:
>i'd vote for that =) except for one thing.
>
>+ /* APICs tend to spasm when they get errors. Disable the error intr. */
>+ apic_write_around(APIC_LVTERR, ERROR_APIC_VECTOR | APIC_LVT_MASKED);
>
>Isn't that a bit drastic?
Drastic is an understatement. Try "gross". Sane machines running correct
code shouldn't throw local APIC errors. If something's causing errors,
that something should be fixed, not hidden.
I hope that was just a temporary debug hack and not part of the design...
/Mikael
On Wednesday 24 July 2002 08:26 am, Zwane Mwaikambo wrote:
> On Tue, 23 Jul 2002, James Cleverdon wrote:
> > Drat! I thought I had all the logical vs. physical stuff straightened
> > out. Could you give this patch a try? It dumps all kinds of APIC state
> > info. You'll need to put a call to apic_state_dump() into check_timer()
> > just after the TIMER: printk.
>
> ID=0x02000000, LVR=0x00170011, TPR=0x00000000, ARB=0x00000002,
> PROCPRI=0x000000F0 DFR=0x0FFFFFFF, LDR=0x01000000, ICR=0x00088500,
> SPIV=0x000001FF, ICR=0x00088500, ICR2=0x03000000, LVTT=0x00000000,
> LVTPC=0x00000000
> LVT0=0x00010700, LVT1=0x00000400, LVTERR=0x000000FE
> ISR: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> 00000000 TMR: 00000000 00000000 00000000 00000000 00000000 00000000
> 00000000 00000000 IRR: 00000000 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 clustered_apic_mode=0, esr_disable=0,
> target_cpus=0x00 apic_broadcast_id=0x0F raw_phys_apicid[]= 00 01 02
> 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 00 00 00 00
> cpu_2_logical_apicid[]= 01 01 02 08 FF FF FF FF FF FF FF FF FF FF FF FF FF
> FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
The logical numbers are bad. They should go 01 02 04 08... on a flat box.
(NUMA boxes are different.) I'll double check the code; am probably making
stupid assumptions, especially given the physical IDs below.
> cpu_2_physical_apicid[]= 02 00 01 03 FF FF FF FF FF FF FF FF FF FF FF FF FF
> FF F F FF FF FF FF FF FF FF FF FF FF FF FF FF
> I/O APIC # 0:
> 00: 00000931:04000000 00010939:02000000 00010000:00000000 00000941:0F000000
> 04: 00000949:0F000000 00000951:0F000000 00010959:02000000 00000961:0F000000
> 08: 00000969:0F000000 00000971:0F000000 00000979:0F000000 00000981:0F000000
> 0C: 00000989:0F000000 00000991:0F000000 00000999:0F000000 000009A1:0F000000
> 10: 00010000:00000000 00010000:00000000 00010000:00000000 00010000:00000000
> 14: 00010000:00000000 00010000:00000000 00010000:00000000 00010000:00000000
>
> > (Hmmm.... Must clean up this patch and submit it to kdb as two new
> > commands, one for I/O APICs and one for local APICs....)
>
> i'd vote for that =) except for one thing.
>
> + /* APICs tend to spasm when they get errors. Disable the error
> intr. */ + apic_write_around(APIC_LVTERR, ERROR_APIC_VECTOR |
> APIC_LVT_MASKED);
>
> Isn't that a bit drastic?
No. ;^)
When a local APIC weirds out and starts spewing interrupts as fast as it can
generate them, it can completely paralyze the system. I suspect that the
APIC error states aren't as well tested as I'd like. Anyway, turning off the
error interrupt is the only way to make enough forward progress to get a
clean shutdown. We had these problems with NUMA P6 boxes. Despite an APIC
bus analyzer pod on the logic analyzers, we never could find out what was
making them spasm or find any other halfway satisfactory solution, other than
to turn off the error interrupt.
> Regards,
> Zwane
--
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
On Wednesday 24 July 2002 08:26 am, Zwane Mwaikambo wrote:
[ Snip! ]
>raw_phys_apicid[]= 00 01 02 03 00 00 00 00 00 00 00 00 00 00 00 00 00
>00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> cpu_2_logical_apicid[]= 01 01 02 08 FF FF FF FF FF FF FF FF FF FF FF FF FF
> FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
> cpu_2_physical_apicid[]= 02 00 01 03 FF FF FF FF FF FF FF FF FF FF FF FF FF
> FF F F FF FF FF FF FF FF FF FF FF FF FF FF FF
Ah ha! Note that while the CPU records in the {MPS,ACPI/MADT} table are in
numerical order (as preserved in raw_phys_apicid), the boot CPU is # 02. The
flat code in smp_boot_cpus assumes that the boot CPU will be the first record
in the list. Oops.
Try the attached patch and see if it helps.
[ Snip! ]
>
> Regards,
> Zwane
--
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
On Wed, 24 Jul 2002, James Cleverdon wrote:
> Ah ha! Note that while the CPU records in the {MPS,ACPI/MADT} table are in
> numerical order (as preserved in raw_phys_apicid), the boot CPU is # 02. The
> flat code in smp_boot_cpus assumes that the boot CPU will be the first record
> in the list. Oops.
Ok i'll give it a whirl, in that case how about the following code to do
the BSP check in another area too?
Index: linux-2.4.19-rc3-ac2/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/zwane/source/cvs_rep/linux-2.4.19-rc3-ac2/arch/i386/kernel/smpboot.c,v
retrieving revision 1.2
diff -u -r1.2 smpboot.c
--- linux-2.4.19-rc3-ac2/arch/i386/kernel/smpboot.c 2002/07/25 06:06:56 1.2
+++ linux-2.4.19-rc3-ac2/arch/i386/kernel/smpboot.c 2002/07/25 06:15:05
@@ -46,6 +46,7 @@
#include <asm/mtrr.h>
#include <asm/pgalloc.h>
#include <asm/smpboot.h>
+#include <asm/msr.h>
/* Set if we find a B stepping CPU */
static int smp_b_stepping;
@@ -229,6 +230,14 @@
return res;
}
+int smp_cpu_is_bsp (void)
+{
+ unsigned long l, h;
+
+ rdmsr(MSR_IA32_APICBASE, l, h);
+ return (l & MSR_IA32_APICBASE_BSP);
+}
+
static void __init synchronize_tsc_bp (void)
{
int i;
@@ -1067,7 +1076,7 @@
connect_bsp_APIC();
setup_local_APIC();
- if (GET_APIC_ID(apic_read(APIC_ID)) != boot_cpu_physical_apicid)
+ if (!smp_cpu_is_bsp())
BUG();
/*
--
function.linuxpower.ca
On Wed, 24 Jul 2002, James Cleverdon wrote:
> Ah ha! Note that while the CPU records in the {MPS,ACPI/MADT} table are in
> numerical order (as preserved in raw_phys_apicid), the boot CPU is # 02. The
> flat code in smp_boot_cpus assumes that the boot CPU will be the first record
> in the list. Oops.
>
> Try the attached patch and see if it helps.
Ok that one goes all the way, but i don't think i've covered everything
(e.g. tested all the IPI functions). But otherwise looks good, i'll give
it a go on a bigger box later (tested on 4-way, i'll try 12)
Cheers,
Zwane
--
function.linuxpower.ca
On Thursday 25 July 2002 12:11 am, Zwane Mwaikambo wrote:
> On Wed, 24 Jul 2002, James Cleverdon wrote:
> > Ah ha! Note that while the CPU records in the {MPS,ACPI/MADT} table are
> > in numerical order (as preserved in raw_phys_apicid), the boot CPU is #
> > 02. The flat code in smp_boot_cpus assumes that the boot CPU will be the
> > first record in the list. Oops.
>
> Ok i'll give it a whirl, in that case how about the following code to do
> the BSP check in another area too?
That would work, too. The bug was not in recognizing the boot cpu, but in
assuming that we would continue (for one reason or another) the first time
around the loop, because logical ID 0x01 was already assigned.
The previous code got around this dependency in a weird and rather kludgey
way. Check -rc3 for the two instances of boot_cpu_logical_apicid in
mpparse.c and smpboot.c, and the two entirely different sources of
boot_cpu_logical_apicid's value. Bizarre.
> Index: linux-2.4.19-rc3-ac2/arch/i386/kernel/smpboot.c
> ===================================================================
> RCS file:
> /home/zwane/source/cvs_rep/linux-2.4.19-rc3-ac2/arch/i386/kernel/smpboot.c,
>v retrieving revision 1.2
> diff -u -r1.2 smpboot.c
> --- linux-2.4.19-rc3-ac2/arch/i386/kernel/smpboot.c 2002/07/25 06:06:56 1.2
> +++ linux-2.4.19-rc3-ac2/arch/i386/kernel/smpboot.c 2002/07/25 06:15:05
> @@ -46,6 +46,7 @@
> #include <asm/mtrr.h>
> #include <asm/pgalloc.h>
> #include <asm/smpboot.h>
> +#include <asm/msr.h>
>
> /* Set if we find a B stepping CPU */
> static int smp_b_stepping;
> @@ -229,6 +230,14 @@
> return res;
> }
>
> +int smp_cpu_is_bsp (void)
> +{
> + unsigned long l, h;
> +
> + rdmsr(MSR_IA32_APICBASE, l, h);
> + return (l & MSR_IA32_APICBASE_BSP);
> +}
> +
> static void __init synchronize_tsc_bp (void)
> {
> int i;
> @@ -1067,7 +1076,7 @@
> connect_bsp_APIC();
> setup_local_APIC();
>
> - if (GET_APIC_ID(apic_read(APIC_ID)) != boot_cpu_physical_apicid)
> + if (!smp_cpu_is_bsp())
> BUG();
>
> /*
--
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
On Wednesday 24 July 2002 10:28 am, Mikael Pettersson wrote:
> On Wed, 24 Jul 2002 17:26:49 +0200 (SAST), Zwane Mwaikambo wrote:
> >i'd vote for that =) except for one thing.
> >
> >+ /* APICs tend to spasm when they get errors. Disable the error
> > intr. */ + apic_write_around(APIC_LVTERR, ERROR_APIC_VECTOR |
> > APIC_LVT_MASKED);
> >
> >Isn't that a bit drastic?
>
> Drastic is an understatement. Try "gross". Sane machines running correct
> code shouldn't throw local APIC errors. If something's causing errors,
> that something should be fixed, not hidden.
>
> I hope that was just a temporary debug hack and not part of the design...
>
> /Mikael
On the contrary, when Intel moved the local APIC from a separate chip onto the
CPU around the time of the P54C, they hobbled it. Formerly, it could accept
and latch any number of interrupts because it contained three bit vectors
that could store all the necessary state info. The P54C (and later) version
had two latches per interrupt level. The level was defined as the top nibble
of the interrupt vector. So, P54Cs could only latch two interrupts for, say,
the 0x31-0x3F range for ISA IRQs. Too bad if three 0x3X interrupts arrive.
Number 3 cannot be latched.
Intel added new error states to the local APIC and the bus protocol to allow
for interrupts to _not_ be delivered, thanks to the latch limit. On a busy
system with lots of interrupts, you will sometimes see several of these
receive accept errors per day. There is nothing you can do to fix the
condition, aside from processing all . It really is more of a warning than
an error.
On our NUMA P6 box, we found that the local APICs would occasionally start
spasming with error interrupts. An APIC bus analyzer didn't show any kind of
errors on the APIC bus. They would just weird out and all attempts to clear
the error had no effect. We never did find a solution to that one or get an
adequate explanation from Intel. The only kludge that worked was to turn off
the APIC error interrupt.
Naturally, the cleaned up version of the apic_state_dump patch wouldn't do
that, or would make it an option.
--
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com
Hi James,
On Thu, 25 Jul 2002, James Cleverdon wrote:
> On our NUMA P6 box, we found that the local APICs would occasionally start
> spasming with error interrupts. An APIC bus analyzer didn't show any kind of
> errors on the APIC bus. They would just weird out and all attempts to clear
> the error had no effect. We never did find a solution to that one or get an
> adequate explanation from Intel. The only kludge that worked was to turn off
> the APIC error interrupt.
>
> Naturally, the cleaned up version of the apic_state_dump patch wouldn't do
> that, or would make it an option.
Since you have the bus analyzer, how frequent (if at all) have you seen
the EOI register being written to without any bit set in the ISR?
Thanks,
Zwane
--
function.linuxpower.ca