Hi! I just wanted to let you know that one of the desktops I use for
testing no longer seems to boot after this commit (just finished
bisecting and confirming). The machine hangs before it gets to fbcon,
and the error I'm seeing in the early boot console is as such:
Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.8.0-rc5Lyude-Test+ #20
Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.10 01/22/2019
Call trace:
<TASK>
dump_stack_lvl+0x47/0x60
panic+0x340/0x370
? timer_irq_works+0x67/0x130
panic_if_irq_remap+0x1d/0x20
setup_IO_APIC+0x82d/0x950
? _raw_spin_unlock_irqrestore+0x1d/0x40
? clear_IO_APIC_pin+0x16c/0x260
apic_intr_mode_init+0x5d/0xf0
x86_late_time_init+0x24/0x40
start_kernel+0x673/0xa90
x86_64_start_reservations+0x18/0x30
x86_64_start_kernel+0x96/0xa0
secondary_startup_64_no_verify+0x180/0x18b
</TASK>
--- [ end Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC ]---
Assuming I copied this over by hand to my computer correctly, the
decoded backtrace should be:
Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.10 01/22/2019
Call trace:
<TASK>
dump_stack_lvl (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/lib/dump_stack.c:107)
panic (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/kernel/panic.c:344)
? timer_irq_works (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/./arch/x86/include/asm/msr.h:186 /home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/apic/io_apic.c:1595 /home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/apic/io_apic.c:1634)
panic_if_irq_remap (??:?)
setup_IO_APIC (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/apic/io_apic.c:2241 /home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/apic/io_apic.c:2413)
? _raw_spin_unlock_irqrestore (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/./arch/x86/include/asm/preempt.h:94 (discriminator 1) /home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/./include/linux/spinlock_api_smp.h:152 (discriminator 1) /home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/kernel/locking/spinlock.c:194 (discriminator 1))
? clear_IO_APIC_pin (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/apic/io_apic.c:563)
apic_intr_mode_init (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/apic/apic.c:2330 /home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/apic/apic.c:1374)
x86_late_time_init (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/time.c:101)
start_kernel (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/init/main.c:1035)
x86_64_start_reservations (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/head64.c:543)
x86_64_start_kernel (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/head64.c:485 (discriminator 5))
secondary_startup_64_no_verify (/home/lyudess/Projects/linux/worktrees/nouveau-aux-fixes/arch/x86/kernel/head_64.S:459)
</TASK>
--- [ end Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC ]---
Happy to provide any more information from this machine if you need it
:). And hopefully i'm not just late to the party and reporting a
regression someone else found already lol
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
On Wed, Apr 17, 2024 at 05:21:43PM -0400, Lyude Paul wrote:
> Hi! I just wanted to let you know that one of the desktops I use for
> testing no longer seems to boot after this commit (just finished
> bisecting and confirming). The machine hangs before it gets to fbcon,
> and the error I'm seeing in the early boot console is as such:
>
> Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.8.0-rc5Lyude-Test+ #20
> Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.10 01/22/2019
Looks like an AMD chipset. Thomas did fix some fallout from the topo
rework on AMD, can you test the tip/master branch pls?
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
Just gave it a try, unfortunately I'm still seeing the same result on
that branch.
One more piece of information I apparently missed when reporting this
yesterday btw: I noticed one more kernel message that comes before the
panic that's probably relevant:
.TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
On Thu, 2024-04-18 at 10:27 +0200, Borislav Petkov wrote:
> On Wed, Apr 17, 2024 at 05:21:43PM -0400, Lyude Paul wrote:
> > Hi! I just wanted to let you know that one of the desktops I use
> > for
> > testing no longer seems to boot after this commit (just finished
> > bisecting and confirming). The machine hangs before it gets to
> > fbcon,
> > and the error I'm seeing in the early boot console is as such:
> >
> > Kernel panic - not syncing: timer doesn't work through
> > Interrupt-remapped IO-APIC
> > CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.8.0-rc5Lyude-Test+
> > #20
> > Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.10
> > 01/22/2019
>
> Looks like an AMD chipset. Thomas did fix some fallout from the topo
> rework on AMD, can you test the tip/master branch pls?
>
> https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/
>
> Thx.
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
On Thu, Apr 18 2024 at 13:20, Lyude Paul wrote:
> Just gave it a try, unfortunately I'm still seeing the same result on
> that branch.
>
> One more piece of information I apparently missed when reporting this
> yesterday btw: I noticed one more kernel message that comes before the
> panic that's probably relevant:
>
> ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Can you please apply the debug patch below which should make it boot
again.
Please also provide the output of the files underneath of
/sys/kernel/debug/x86/topo/
Thanks,
tglx
---
arch/x86/kernel/cpu/topology.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -176,6 +176,8 @@ static __init void topo_register_apic(u3
{
int cpu, dom;
+ pr_info("APIC ID %x present %d\n", apic_id, present);
+
if (present) {
set_bit(apic_id, phys_cpu_present_map);
@@ -201,10 +203,7 @@ static __init void topo_register_apic(u3
*/
if (hypervisor_is_type(X86_HYPER_NATIVE) &&
topo_unit_count(pkgid, TOPO_PKG_DOMAIN, phys_cpu_present_map)) {
- pr_info_once("Ignoring hot-pluggable APIC ID %x in present package.\n",
- apic_id);
- topo_info.nr_rejected_cpus++;
- return;
+ pr_info("Hot-pluggable APIC ID %x in present package.\n", apic_id);
}
topo_info.nr_disabled_cpus++;
On Thu, Apr 18 2024 at 21:13, Thomas Gleixner wrote:
> On Thu, Apr 18 2024 at 13:20, Lyude Paul wrote:
>
>> Just gave it a try, unfortunately I'm still seeing the same result on
>> that branch.
>>
>> One more piece of information I apparently missed when reporting this
>> yesterday btw: I noticed one more kernel message that comes before the
>> panic that's probably relevant:
>>
>> ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>
> Can you please apply the debug patch below which should make it boot
> again.
And provide the resulting dmesg obviously.
> Please also provide the output of the files underneath of
>
> /sys/kernel/debug/x86/topo/
>
> Thanks,
>
> tglx
Awesome - can confirm the patch does indeed make the machine boot. Full
dmesg from boot attached. And the contents of
/sys/kernel/debug/x86/topo/ is as follows:
domain: Thread shift: 1 dom_size: 2 max_threads: 2
domain: Core shift: 4 dom_size: 8 max_threads: 16
domain: Module shift: 4 dom_size: 1 max_threads: 16
domain: Tile shift: 4 dom_size: 1 max_threads: 16
domain: Die shift: 4 dom_size: 1 max_threads: 16
domain: DieGrp shift: 4 dom_size: 1 max_threads: 16
domain: Package shift: 4 dom_size: 1 max_threads: 16
On Thu, 2024-04-18 at 21:13 +0200, Thomas Gleixner wrote:
> On Thu, Apr 18 2024 at 13:20, Lyude Paul wrote:
>
> > Just gave it a try, unfortunately I'm still seeing the same result
> > on
> > that branch.
> >
> > One more piece of information I apparently missed when reporting
> > this
> > yesterday btw: I noticed one more kernel message that comes before
> > the
> > panic that's probably relevant:
> >
> > ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>
> Can you please apply the debug patch below which should make it boot
> again.
>
> Please also provide the output of the files underneath of
>
> /sys/kernel/debug/x86/topo/
>
> Thanks,
>
> tglx
> ---
> arch/x86/kernel/cpu/topology.c | 7 +++----
> 1 file changed, 3 insertions(+), 4 deletions(-)
>
> --- a/arch/x86/kernel/cpu/topology.c
> +++ b/arch/x86/kernel/cpu/topology.c
> @@ -176,6 +176,8 @@ static __init void topo_register_apic(u3
> {
> int cpu, dom;
>
> + pr_info("APIC ID %x present %d\n", apic_id, present);
> +
> if (present) {
> set_bit(apic_id, phys_cpu_present_map);
>
> @@ -201,10 +203,7 @@ static __init void topo_register_apic(u3
> */
> if (hypervisor_is_type(X86_HYPER_NATIVE) &&
> topo_unit_count(pkgid, TOPO_PKG_DOMAIN,
> phys_cpu_present_map)) {
> - pr_info_once("Ignoring hot-pluggable APIC ID
> %x in present package.\n",
> - apic_id);
> - topo_info.nr_rejected_cpus++;
> - return;
> + pr_info("Hot-pluggable APIC ID %x in present
> package.\n", apic_id);
> }
>
> topo_info.nr_disabled_cpus++;
>
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
Paul!
On Fri, Apr 19 2024 at 13:38, Lyude Paul wrote:
> Awesome - can confirm the patch does indeed make the machine boot. Full
> dmesg from boot attached.
Thanks for providing the data.
[ 0.089286] CPU topo: APIC ID 0 present 1
[ 0.089294] CPU topo: APIC ID 0 present 0
[ 0.089296] CPU topo: Hot-pluggable APIC ID 0 in present package.
ACPI is really a wonderland.
Can you please test the patch below?
Thanks,
tglx
---
Subject: x86/topology: Deal with more broken ACPI tables
From: Thomas Gleixner <[email protected]>
Date: Thu, 18 Apr 2024 21:02:39 +0200
Paul reported a regression which waas caused by the handling of non-present
CPUs in a present package. It's caused by the ACPI table on the system
which advertises APICs twice, present and non-present:
CPU topo: APIC ID 0 present 1
CPU topo: APIC ID 0 present 0
CPU topo: Hot-pluggable APIC ID 0 in present package.
Which causes the topology to get confused to the point that it fails to
bring the system up because the target APIC for the IOAPIC is not
available.
Prevent this by checking whether a non-present CPU has been already
registered as present before. If so emit a firmware warning and ignore the
registration request.
Fixes: f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a present package")
Reported-by: Lyude Paul <[email protected]>
Signed-off-by: Thomas Gleixner <[email protected]>
---
arch/x86/kernel/cpu/topology.c | 6 ++++++
1 file changed, 6 insertions(+)
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -195,6 +195,12 @@ static __init void topo_register_apic(u3
} else {
u32 pkgid = topo_apicid(apic_id, TOPO_PKG_DOMAIN);
+ if (test_bit(apic_id, phys_cpu_present_map)) {
+ pr_warn_once(FW_BUG "Already present APIC ID %x registered again as non-present\n",
+ apic_id);
+ return;
+ }
+
/*
* Check for present APICs in the same package when running
* on bare metal. Allow the bogosity in a guest.
On Sat, Apr 20 2024 at 00:15, Thomas Gleixner wrote:
> Paul!
>
> On Fri, Apr 19 2024 at 13:38, Lyude Paul wrote:
>> Awesome - can confirm the patch does indeed make the machine boot. Full
>> dmesg from boot attached.
>
> Thanks for providing the data.
>
> [ 0.089286] CPU topo: APIC ID 0 present 1
> [ 0.089294] CPU topo: APIC ID 0 present 0
> [ 0.089296] CPU topo: Hot-pluggable APIC ID 0 in present package.
>
> ACPI is really a wonderland.
Second thoughts. I just stared at this some more and I really cannot
figure out why any of this (including the debug patch) makes a
difference or even sense at all.
All the commit you bisected to does is to reject the non-present APIC
IDs, but that's just an accounting thing. Instead of having them
accounted as disabled they are accounted as rejected.
So no. None of this makes any sense at all.
On Sat, 2024-04-20 at 00:15 +0200, Thomas Gleixner wrote:
> Paul!
Lyude is fine BTW :P (I get the confusion though, Paul is usually not a
last name lol)
Anyway - unfortunately it doesn't seem like this patch helps :s, I'm
still not seeing any difference and the backtrace I'm seeing at early
boot looks the same. Any more information I can provide?
>
> On Fri, Apr 19 2024 at 13:38, Lyude Paul wrote:
> > Awesome - can confirm the patch does indeed make the machine boot.
> > Full
> > dmesg from boot attached.
>
> Thanks for providing the data.
>
> [ 0.089286] CPU topo: APIC ID 0 present 1
> [ 0.089294] CPU topo: APIC ID 0 present 0
> [ 0.089296] CPU topo: Hot-pluggable APIC ID 0 in present package.
>
> ACPI is really a wonderland.
>
> Can you please test the patch below?
>
> Thanks,
>
> tglx
> ---
> Subject: x86/topology: Deal with more broken ACPI tables
> From: Thomas Gleixner <[email protected]>
> Date: Thu, 18 Apr 2024 21:02:39 +0200
>
> Paul reported a regression which waas caused by the handling of non-
> present
> CPUs in a present package. It's caused by the ACPI table on the
> system
> which advertises APICs twice, present and non-present:
>
> CPU topo: APIC ID 0 present 1
> CPU topo: APIC ID 0 present 0
> CPU topo: Hot-pluggable APIC ID 0 in present
> package.
>
>
> Which causes the topology to get confused to the point that it fails
> to
> bring the system up because the target APIC for the IOAPIC is not
> available.
>
> Prevent this by checking whether a non-present CPU has been already
> registered as present before. If so emit a firmware warning and
> ignore the
> registration request.
>
> Fixes: f0551af0213 ("x86/topology: Ignore non-present APIC IDs in a
> present package")
> Reported-by: Lyude Paul <[email protected]>
> Signed-off-by: Thomas Gleixner <[email protected]>
> ---
> arch/x86/kernel/cpu/topology.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> --- a/arch/x86/kernel/cpu/topology.c
> +++ b/arch/x86/kernel/cpu/topology.c
> @@ -195,6 +195,12 @@ static __init void topo_register_apic(u3
> } else {
> u32 pkgid = topo_apicid(apic_id, TOPO_PKG_DOMAIN);
>
> + if (test_bit(apic_id, phys_cpu_present_map)) {
> + pr_warn_once(FW_BUG "Already present APIC ID
> %x registered again as non-present\n",
> + apic_id);
> + return;
> + }
> +
> /*
> * Check for present APICs in the same package when
> running
> * on bare metal. Allow the bogosity in a guest.
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
Lyude!
On Wed, Apr 24 2024 at 16:56, Lyude Paul wrote:
> On Sat, 2024-04-20 at 00:15 +0200, Thomas Gleixner wrote:
> Lyude is fine BTW :P (I get the confusion though, Paul is usually not a
> last name lol)
:)
> Anyway - unfortunately it doesn't seem like this patch helps :s, I'm
> still not seeing any difference and the backtrace I'm seeing at early
> boot looks the same. Any more information I can provide?
Can you please boot a kernel with the commit in question reverted and
add 'possible_cpus=8' to the kernel command line?
In theory this should fail too.
Thanks,
tglx
Yep - tried booting a kernel with f0551af0213 reverted and
possible_cpus=8, it definitely looks like that crashes things as well
in the same way. Also - it scrolled off the screen before I had a
chance to write it down, but I'm -fairly- sure I saw some sort of
complaint about "16 [or some double digit number] processors exceeds
max number of 8". Which is quite interesting, as this is definitely
just a quad core ryzen processor with hyperthreading - so there should
only be 8 threads.
On Thu, 2024-04-25 at 04:11 +0200, Thomas Gleixner wrote:
> Lyude!
>
> On Wed, Apr 24 2024 at 16:56, Lyude Paul wrote:
> > On Sat, 2024-04-20 at 00:15 +0200, Thomas Gleixner wrote:
> > Lyude is fine BTW :P (I get the confusion though, Paul is usually
> > not a
> > last name lol)
>
> :)
>
> > Anyway - unfortunately it doesn't seem like this patch helps :s,
> > I'm
> > still not seeing any difference and the backtrace I'm seeing at
> > early
> > boot looks the same. Any more information I can provide?
>
> Can you please boot a kernel with the commit in question reverted and
> add 'possible_cpus=8' to the kernel command line?
>
> In theory this should fail too.
>
> Thanks,
>
> tglx
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
Lyude!
On Thu, Apr 25 2024 at 11:56, Lyude Paul wrote:
> On Thu, 2024-04-25 at 04:11 +0200, Thomas Gleixner wrote:
>>
>> Can you please boot a kernel with the commit in question reverted and
>> add 'possible_cpus=8' to the kernel command line?
>>
>> In theory this should fail too.
>
> Yep - tried booting a kernel with f0551af0213 reverted and
> possible_cpus=8, it definitely looks like that crashes things as well
> in the same way.
Good. That means it's a problem which existed before but went unnoticed.
> Also - it scrolled off the screen before I had a chance to write it
> down, but I'm -fairly- sure I saw some sort of complaint about "16 [or
> some double digit number] processors exceeds max number of 8". Which
> is quite interesting, as this is definitely just a quad core ryzen
> processor with hyperthreading - so there should only be 8 threads.
Right, that's what we saw with the debug patch. The ACPI/MADT table
is clearly bonkers. The effect of it is that it pretends that the system
has 16 possible CPUs:
[ 0.089381] CPU topo: Allowing 8 present CPUs plus 8 hotplug CPUs
Which in turn changes the sizing of the per CPU data and affects some
other details which depend on the number of possible CPUs.
But that should not matter at all because the system scaling should be
sufficient with 8 CPUs, but it does not for some completely non-obvious
reasons.
Can you please try to increase possible_cpus=N on the command line one
by one and check when it actually starts to "work" again.
One other thing to try is to boot with 'possible_cpus=8' and
'intremap=off' and see whether that makes a difference.
I really have no idea where to look and not having the early boot
messages in case of the fail is not helpful as I can't add meaningful
debug to it.
I just checked: the motherboard has a serial port, so it would be
extremly helpful to hook up a serial cable to this thing and enable
serial console on the kernel command line. That way we might eventually
see information which is emitted before it fails to validate the timer
interrupt.
Thanks,
tglx
On 4/25/2024 16:42, Thomas Gleixner wrote:
> Lyude!
>
> On Thu, Apr 25 2024 at 11:56, Lyude Paul wrote:
>> On Thu, 2024-04-25 at 04:11 +0200, Thomas Gleixner wrote:
>>>
>>> Can you please boot a kernel with the commit in question reverted and
>>> add 'possible_cpus=8' to the kernel command line?
>>>
>>> In theory this should fail too.
>>
>> Yep - tried booting a kernel with f0551af0213 reverted and
>> possible_cpus=8, it definitely looks like that crashes things as well
>> in the same way.
>
> Good. That means it's a problem which existed before but went unnoticed.
>
>> Also - it scrolled off the screen before I had a chance to write it
>> down, but I'm -fairly- sure I saw some sort of complaint about "16 [or
>> some double digit number] processors exceeds max number of 8". Which
>> is quite interesting, as this is definitely just a quad core ryzen
>> processor with hyperthreading - so there should only be 8 threads.
>
> Right, that's what we saw with the debug patch. The ACPI/MADT table
> is clearly bonkers. The effect of it is that it pretends that the system
> has 16 possible CPUs:
>
> [ 0.089381] CPU topo: Allowing 8 present CPUs plus 8 hotplug CPUs
>
> Which in turn changes the sizing of the per CPU data and affects some
> other details which depend on the number of possible CPUs.
At least this aspect of this I suspect is caused by commit
fed8d8773b8ea68ad99d9eee8c8343bef9da2c2c.
If you try reverting that I expect the "hotplug CPUs" disappear.
>
> But that should not matter at all because the system scaling should be
> sufficient with 8 CPUs, but it does not for some completely non-obvious
> reasons.
>
> Can you please try to increase possible_cpus=N on the command line one
> by one and check when it actually starts to "work" again.
>
> One other thing to try is to boot with 'possible_cpus=8' and
> 'intremap=off' and see whether that makes a difference.
>
> I really have no idea where to look and not having the early boot
> messages in case of the fail is not helpful as I can't add meaningful
> debug to it.
>
> I just checked: the motherboard has a serial port, so it would be
> extremly helpful to hook up a serial cable to this thing and enable
> serial console on the kernel command line. That way we might eventually
> see information which is emitted before it fails to validate the timer
> interrupt.
>
> Thanks,
>
> tglx
>
H! Lyude, Thomas, what's the status here? From here it looks like we
were close to a fix, but then it turned out to be a bad fix -- and
afterwards nothing much seems to have happened. Did it fall through the
cracks, or was this already fixed and I just missed that?
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
On 02.05.24 12:33, Mario Limonciello wrote:
> On 4/25/2024 16:42, Thomas Gleixner wrote:
>> On Thu, Apr 25 2024 at 11:56, Lyude Paul wrote:
>>> On Thu, 2024-04-25 at 04:11 +0200, Thomas Gleixner wrote:
>>>>
>>>> Can you please boot a kernel with the commit in question reverted and
>>>> add 'possible_cpus=8' to the kernel command line?
>>>>
>>>> In theory this should fail too.
>>>
>>> Yep - tried booting a kernel with f0551af0213 reverted and
>>> possible_cpus=8, it definitely looks like that crashes things as well
>>> in the same way.
>>
>> Good. That means it's a problem which existed before but went unnoticed.
>>
>>> Also - it scrolled off the screen before I had a chance to write it
>>> down, but I'm -fairly- sure I saw some sort of complaint about "16 [or
>>> some double digit number] processors exceeds max number of 8". Which
>>> is quite interesting, as this is definitely just a quad core ryzen
>>> processor with hyperthreading - so there should only be 8 threads.
>>
>> Right, that's what we saw with the debug patch. The ACPI/MADT table
>> is clearly bonkers. The effect of it is that it pretends that the system
>> has 16 possible CPUs:
>>
>> [ 0.089381] CPU topo: Allowing 8 present CPUs plus 8 hotplug CPUs
>>
>> Which in turn changes the sizing of the per CPU data and affects some
>> other details which depend on the number of possible CPUs.
>
> At least this aspect of this I suspect is caused by commit
> fed8d8773b8ea68ad99d9eee8c8343bef9da2c2c.
>
> If you try reverting that I expect the "hotplug CPUs" disappear.
>
>>
>> But that should not matter at all because the system scaling should be
>> sufficient with 8 CPUs, but it does not for some completely non-obvious
>> reasons.
>>
>> Can you please try to increase possible_cpus=N on the command line one
>> by one and check when it actually starts to "work" again.
>>
>> One other thing to try is to boot with 'possible_cpus=8' and
>> 'intremap=off' and see whether that makes a difference.
>>
>> I really have no idea where to look and not having the early boot
>> messages in case of the fail is not helpful as I can't add meaningful
>> debug to it.
>>
>> I just checked: the motherboard has a serial port, so it would be
>> extremly helpful to hook up a serial cable to this thing and enable
>> serial console on the kernel command line. That way we might eventually
>> see information which is emitted before it fails to validate the timer
>> interrupt.
>>
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
#regzbot poke
On Wed, May 08 2024 at 10:38, Linux regression tracking (Thorsten Leemhuis) wrote:
> H! Lyude, Thomas, what's the status here? From here it looks like we
> were close to a fix, but then it turned out to be a bad fix -- and
> afterwards nothing much seems to have happened. Did it fall through the
> cracks, or was this already fixed and I just missed that?
I'm waiting for more data still.
Thanks,
tglx
Yes sorry - I was out of work for a bit but I'm back now and can get
you the info today - thanks for the patience ♥
On Wed, 2024-05-08 at 12:30 +0200, Thomas Gleixner wrote:
> On Wed, May 08 2024 at 10:38, Linux regression tracking (Thorsten
> Leemhuis) wrote:
> > H! Lyude, Thomas, what's the status here? From here it looks like
> > we
> > were close to a fix, but then it turned out to be a bad fix -- and
> > afterwards nothing much seems to have happened. Did it fall through
> > the
> > cracks, or was this already fixed and I just missed that?
>
> I'm waiting for more data still.
>
> Thanks,
>
> tglx
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
Mario!
On Thu, May 02 2024 at 05:33, Mario Limonciello wrote:
> On 4/25/2024 16:42, Thomas Gleixner wrote:
>> Right, that's what we saw with the debug patch. The ACPI/MADT table
>> is clearly bonkers. The effect of it is that it pretends that the system
>> has 16 possible CPUs:
>>
>> [ 0.089381] CPU topo: Allowing 8 present CPUs plus 8 hotplug CPUs
>>
>> Which in turn changes the sizing of the per CPU data and affects some
>> other details which depend on the number of possible CPUs.
>
> At least this aspect of this I suspect is caused by commit
> fed8d8773b8ea68ad99d9eee8c8343bef9da2c2c.
>
> If you try reverting that I expect the "hotplug CPUs" disappear.
That does not solve anything.
The topology core already rejects those CPUs and accounts only for 8,
which in turn causes the boot to fail as also demonstrated by limiting
the number of possible CPUs to 8.
There is some other problem with this broken BIOS/ACPI.
On 5/8/2024 16:47, Thomas Gleixner wrote:
> Mario!
>
> On Thu, May 02 2024 at 05:33, Mario Limonciello wrote:
>> On 4/25/2024 16:42, Thomas Gleixner wrote:
>>> Right, that's what we saw with the debug patch. The ACPI/MADT table
>>> is clearly bonkers. The effect of it is that it pretends that the system
>>> has 16 possible CPUs:
>>>
>>> [ 0.089381] CPU topo: Allowing 8 present CPUs plus 8 hotplug CPUs
>>>
>>> Which in turn changes the sizing of the per CPU data and affects some
>>> other details which depend on the number of possible CPUs.
>>
>> At least this aspect of this I suspect is caused by commit
>> fed8d8773b8ea68ad99d9eee8c8343bef9da2c2c.
>>
>> If you try reverting that I expect the "hotplug CPUs" disappear.
>
> That does not solve anything.
>
> The topology core already rejects those CPUs and accounts only for 8,
> which in turn causes the boot to fail as also demonstrated by limiting
> the number of possible CPUs to 8.
>
> There is some other problem with this broken BIOS/ACPI.
Something very commonly done in BIOSes on AMD systems is that the FADT
has "entries" for the maximum number of CPUs that can be present. For
example if the system can support up to 12 cores and you buy an 8 core
vs 12 core the BIOS will have the same number of entries (probably 24
considering SMT) either way. In the case of 8 cores only 16 would end
up populated.
Looking at Lyude's logs that system is from before ACPI 6.3 was even
introduced so that's why I was suggesting that reverting that commit
might help at least the kernel claiming that it saw a number of hotplug
CPUs.
But yes, I agree it probably won't help the overall issue that started
this thread.
Ok - so re the serial port comment (which I missed the first time I
read your message, oops!): I can totally get a serial console on this
machine, I happen to already have a serial port hooked up to the
motherboard. I'm not sure where my USB to RS232 adapter to connect to
it on my laptop is though, so I put in an order for a new one and it
should arrive tomorrow.
Regarding the test results - I tried possible_cpus all the way up to 17
and nothing got the machine to boot. However, possible_cpus=8
intremap=off did get the machine to boot successfully
On Wed, 2024-05-08 at 17:02 -0400, Lyude Paul wrote:
> Yes sorry - I was out of work for a bit but I'm back now and can get
> you the info today - thanks for the patience ♥
>
> On Wed, 2024-05-08 at 12:30 +0200, Thomas Gleixner wrote:
> > On Wed, May 08 2024 at 10:38, Linux regression tracking (Thorsten
> > Leemhuis) wrote:
> > > H! Lyude, Thomas, what's the status here? From here it looks like
> > > we
> > > were close to a fix, but then it turned out to be a bad fix --
> > > and
> > > afterwards nothing much seems to have happened. Did it fall
> > > through
> > > the
> > > cracks, or was this already fixed and I just missed that?
> >
> > I'm waiting for more data still.
> >
> > Thanks,
> >
> > tglx
> >
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
So uh, apparently amazon decided that instead of using the shipping
address I have in their address book that is marked as "Default
Shipping Address" that it makes much more sense to sort all addresses
in the address book alphabetically, and then surprise me by sending it
to the first address in that list. which means the adapter got shipped
half way across the country and it's probably going to take me until at
least tomorrow to get a different one here, sorry :(.
Will get back to you with this info as soon as I can
On Wed, 2024-05-08 at 12:30 +0200, Thomas Gleixner wrote:
> On Wed, May 08 2024 at 10:38, Linux regression tracking (Thorsten
> Leemhuis) wrote:
> > H! Lyude, Thomas, what's the status here? From here it looks like
> > we
> > were close to a fix, but then it turned out to be a bad fix -- and
> > afterwards nothing much seems to have happened. Did it fall through
> > the
> > cracks, or was this already fixed and I just missed that?
>
> I'm waiting for more data still.
>
> Thanks,
>
> tglx
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
On Wed, May 08 2024 at 19:21, Lyude Paul wrote:
> Regarding the test results - I tried possible_cpus all the way up to 17
> and nothing got the machine to boot. However, possible_cpus=8
> intremap=off did get the machine to boot successfully
Oh. That's interesting.
Does v6.9 (released yesterday) boot with just 'intremap=off' too?
Thanks,
tglx
Oh! I am not sure what changed, but I realized that you might also want to see
the serial output from the debugging patch that you had sent me a while ago.
So I just built that against the 6.9 kernel and it actually seems to boot
without needing me to pass intremap=off. So it seems like that might actually
be a fix!
On Mon, 2024-05-13 at 19:18 -0400, Lyude Paul wrote:
> On Mon, 2024-05-13 at 16:08 +0200, Thomas Gleixner wrote:
> > On Wed, May 08 2024 at 19:21, Lyude Paul wrote:
> > > Regarding the test results - I tried possible_cpus all the way up to 17
> > > and nothing got the machine to boot. However, possible_cpus=8
> > > intremap=off did get the machine to boot successfully
> >
> > Oh. That's interesting.
> >
> > Does v6.9 (released yesterday) boot with just 'intremap=off' too?
>
> Yes - it still boots. As well I finally got the serial console adapter in, but
> I don't see any additional output:
>
> [ 0.146896] printk: legacy console [tty0] enabled
> [ 0.147433] printk: legacy console [ttyS0] enabled
> [ 1.951453] ACPI: Core revision 20230628
> [ 1.955466] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns
> [ 1.964698] APIC: Switch to symmetric I/O mode setup
> [ 1.970973] AMD-Vi: Using global IVHD EFR:0xf77ef22294ada, EFR2:0x0
> [ 1.978491] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> [ 1.996945] Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
> [ 2.005376] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.9.0Lyude-Test+ #7
> [ 2.012248] Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.I0 01/22/2019
> [ 2.020160] Call Trace:
> [ 2.022584] <TASK>
> [ 2.024666] dump_stack_lvl+0x5d/0x80
> [ 2.028499] panic+0x118/0x2c8
> [ 2.031522] ? timer_irq_works+0x54/0xf0
> [ 2.035408] panic_if_irq_remap.cold+0x5/0x9
> [ 2.039631] setup_IO_APIC+0x398/0x860
> [ 2.043475] ? _raw_spin_unlock_irqrestore+0x1d/0x40
> [ 2.048393] ? clear_IO_APIC_pin+0x125/0x1e0
> [ 2.052617] apic_intr_mode_init+0xcc/0xe0
> [ 2.056672] x86_late_time_init+0x24/0x40
> [ 2.060820] start_kernel+0x8be/0x960
> [ 2.064446] x86_64_start_reservations+0x24/0x30
> [ 2.069015] x86_64_start_kernel+0x95/0xa0
> [ 2.073069] common_startup_64+0x13e/0x141
> [ 2.077280] </TASK>
> [ 2.079447] ---[ end Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC ]---
>
>
> >
> > Thanks,
> >
> > tglx
> >
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
On Mon, 2024-05-13 at 16:08 +0200, Thomas Gleixner wrote:
> On Wed, May 08 2024 at 19:21, Lyude Paul wrote:
> > Regarding the test results - I tried possible_cpus all the way up to 17
> > and nothing got the machine to boot. However, possible_cpus=8
> > intremap=off did get the machine to boot successfully
>
> Oh. That's interesting.
>
> Does v6.9 (released yesterday) boot with just 'intremap=off' too?
Yes - it still boots. As well I finally got the serial console adapter in, but
I don't see any additional output:
[ 0.146896] printk: legacy console [tty0] enabled
[ 0.147433] printk: legacy console [ttyS0] enabled
[ 1.951453] ACPI: Core revision 20230628
[ 1.955466] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns
[ 1.964698] APIC: Switch to symmetric I/O mode setup
[ 1.970973] AMD-Vi: Using global IVHD EFR:0xf77ef22294ada, EFR2:0x0
[ 1.978491] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 1.996945] Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
[ 2.005376] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.9.0Lyude-Test+ #7
[ 2.012248] Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.I0 01/22/2019
[ 2.020160] Call Trace:
[ 2.022584] <TASK>
[ 2.024666] dump_stack_lvl+0x5d/0x80
[ 2.028499] panic+0x118/0x2c8
[ 2.031522] ? timer_irq_works+0x54/0xf0
[ 2.035408] panic_if_irq_remap.cold+0x5/0x9
[ 2.039631] setup_IO_APIC+0x398/0x860
[ 2.043475] ? _raw_spin_unlock_irqrestore+0x1d/0x40
[ 2.048393] ? clear_IO_APIC_pin+0x125/0x1e0
[ 2.052617] apic_intr_mode_init+0xcc/0xe0
[ 2.056672] x86_late_time_init+0x24/0x40
[ 2.060820] start_kernel+0x8be/0x960
[ 2.064446] x86_64_start_reservations+0x24/0x30
[ 2.069015] x86_64_start_kernel+0x95/0xa0
[ 2.073069] common_startup_64+0x13e/0x141
[ 2.077280] </TASK>
[ 2.079447] ---[ end Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC ]---
>
> Thanks,
>
> tglx
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
Lyude!
On Mon, May 13 2024 at 19:32, Lyude Paul wrote:
> Oh! I am not sure what changed, but I realized that you might also want to see
> the serial output from the debugging patch that you had sent me a while ago.
> So I just built that against the 6.9 kernel and it actually seems to boot
> without needing me to pass intremap=off. So it seems like that might actually
> be a fix!
Which one of the debug patches did you use?
>> Yes - it still boots. As well I finally got the serial console adapter in, but
>> I don't see any additional output:
That's fine, but now I can provide you debug patches which dump
information during early boot.
Can you please provide the output of 'cpuid -r' ?
Thanks,
tglx
On Tue, 2024-05-14 at 10:25 +0200, Thomas Gleixner wrote:
> Lyude!
>
> Which one of the debug patches did you use?
The one you sent on 4/18, when you also asked me for the output of
/sys/kernel/debug/x86/topo/
(I thought I remembered that patch not booting on previous kernels but maybe
I'm misremembering)
>
> > > Yes - it still boots. As well I finally got the serial console adapter in, but
> > > I don't see any additional output:
>
> That's fine, but now I can provide you debug patches which dump
> information during early boot.
>
> Can you please provide the output of 'cpuid -r' ?
>
CPU 0:
0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x00000001 0x00: eax=0x00800f11 ebx=0x00080800 ecx=0x7ed8320b edx=0x178bfbff
0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
0x8000001e 0x00: eax=0x00000000 ebx=0x00000100 ecx=0x00000000 edx=0x00000000
0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 1:
0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x00000001 0x00: eax=0x00800f11 ebx=0x01080800 ecx=0x7ed8320b edx=0x178bfbff
0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
0x8000001e 0x00: eax=0x00000001 ebx=0x00000100 ecx=0x00000000 edx=0x00000000
0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 2:
0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x00000001 0x00: eax=0x00800f11 ebx=0x02080800 ecx=0x7ed8320b edx=0x178bfbff
0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
0x8000001e 0x00: eax=0x00000002 ebx=0x00000101 ecx=0x00000000 edx=0x00000000
0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 3:
0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x00000001 0x00: eax=0x00800f11 ebx=0x03080800 ecx=0x7ed8320b edx=0x178bfbff
0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
0x8000001e 0x00: eax=0x00000003 ebx=0x00000101 ecx=0x00000000 edx=0x00000000
0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 4:
0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x00000001 0x00: eax=0x00800f11 ebx=0x08080800 ecx=0x7ed8320b edx=0x178bfbff
0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
0x8000001e 0x00: eax=0x00000008 ebx=0x00000104 ecx=0x00000000 edx=0x00000000
0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 5:
0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x00000001 0x00: eax=0x00800f11 ebx=0x09080800 ecx=0x7ed8320b edx=0x178bfbff
0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
0x8000001e 0x00: eax=0x00000009 ebx=0x00000104 ecx=0x00000000 edx=0x00000000
0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 6:
0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x00000001 0x00: eax=0x00800f11 ebx=0x0a080800 ecx=0x7ed8320b edx=0x178bfbff
0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
0x8000001e 0x00: eax=0x0000000a ebx=0x00000105 ecx=0x00000000 edx=0x00000000
0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
CPU 7:
0x00000000 0x00: eax=0x0000000d ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x00000001 0x00: eax=0x00800f11 ebx=0x0b080800 ecx=0x7ed8320b edx=0x178bfbff
0x00000002 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000004 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00000000
0x00000006 0x00: eax=0x00000004 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
0x00000007 0x00: eax=0x00000000 ebx=0x209c01a9 ecx=0x00000000 edx=0x00000000
0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000a 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x0000000d 0x00: eax=0x00000007 ebx=0x00000340 ecx=0x00000340 edx=0x00000000
0x0000000d 0x01: eax=0x0000000f ebx=0x00000340 ecx=0x00000000 edx=0x00000000
0x0000000d 0x02: eax=0x00000100 ebx=0x00000240 ecx=0x00000000 edx=0x00000000
0x20000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000000 0x00: eax=0x8000001f ebx=0x68747541 ecx=0x444d4163 edx=0x69746e65
0x80000001 0x00: eax=0x00800f11 ebx=0x20000000 ecx=0x35c233ff edx=0x2fd3fbff
0x80000002 0x00: eax=0x20444d41 ebx=0x657a7952 ecx=0x2035206e edx=0x30303431
0x80000003 0x00: eax=0x61755120 ebx=0x6f432d64 ecx=0x50206572 edx=0x65636f72
0x80000004 0x00: eax=0x726f7373 ebx=0x20202020 ecx=0x20202020 edx=0x00202020
0x80000005 0x00: eax=0xff40ff40 ebx=0xff40ff40 ecx=0x20080140 edx=0x40040140
0x80000006 0x00: eax=0x26006400 ebx=0x66006400 ecx=0x02006140 edx=0x00408140
0x80000007 0x00: eax=0x00000000 ebx=0x0000001b ecx=0x00000000 edx=0x00006599
0x80000008 0x00: eax=0x00003030 ebx=0x00001007 ecx=0x00004007 edx=0x00000000
0x80000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000a 0x00: eax=0x00000001 ebx=0x00008000 ecx=0x00000000 edx=0x0001bcff
0x8000000b 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000d 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000e 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000000f 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000010 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000011 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000012 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000013 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000014 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000015 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000016 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000017 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000018 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x80000019 0x00: eax=0xf040f040 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001a 0x00: eax=0x00000003 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001b 0x00: eax=0x000003ff ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001c 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0x8000001d 0x00: eax=0x00004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000
0x8000001d 0x01: eax=0x00004122 ebx=0x00c0003f ecx=0x000000ff edx=0x00000000
0x8000001d 0x02: eax=0x00004143 ebx=0x01c0003f ecx=0x000003ff edx=0x00000002
0x8000001d 0x03: eax=0x0000c163 ebx=0x03c0003f ecx=0x00000fff edx=0x00000001
0x8000001e 0x00: eax=0x0000000b ebx=0x00000105 ecx=0x00000000 edx=0x00000000
0x8000001f 0x00: eax=0x00000007 ebx=0x0000016f ecx=0x0000000f edx=0x00000000
0x80860000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
0xc0000000 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000
> Thanks,
>
> tglx
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
Lyude!
On Wed, May 15 2024 at 19:15, Lyude Paul wrote:
> On Tue, 2024-05-14 at 10:25 +0200, Thomas Gleixner wrote:
>>
>> Which one of the debug patches did you use?
>
> The one you sent on 4/18, when you also asked me for the output of
> /sys/kernel/debug/x86/topo/
>
> (I thought I remembered that patch not booting on previous kernels but maybe
> I'm misremembering)
It booted with the -rc kernel too. :)
I found an interesting and probably related difference in the boot logs
though. Up to 8 possible CPUs the kernel uses logical destination mode
for the APIC. With more than 8 it uses physical destination mode.
Can you please test the patch below on top of 6.9 and validate that it
boots w/o any magic command line parameter?
Thanks,
tglx
---
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 66fd4b2a37a3..db5e93a7c194 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -1883,6 +1883,8 @@ static inline void try_to_enable_x2apic(int remap_mode) { }
static inline void __x2apic_enable(void) { }
#endif /* !CONFIG_X86_X2APIC */
+int irq_remap_mode = -1;
+
void __init enable_IR_x2apic(void)
{
unsigned long flags;
@@ -1915,6 +1917,8 @@ void __init enable_IR_x2apic(void)
if (ir_stat < 0)
restore_ioapic_entries();
+ else
+ irq_remap_mode = ir_stat;
legacy_pic->restore_mask();
local_irq_restore(flags);
}
diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c
index f37ad3392fec..5cc386db3557 100644
--- a/arch/x86/kernel/apic/apic_flat_64.c
+++ b/arch/x86/kernel/apic/apic_flat_64.c
@@ -12,6 +12,7 @@
#include <linux/export.h>
#include <linux/acpi.h>
+#include <asm/irq_remapping.h>
#include <asm/jailhouse_para.h>
#include <asm/apic.h>
@@ -130,7 +131,8 @@ static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
static int physflat_probe(void)
{
- return apic == &apic_physflat || num_possible_cpus() > 8 || jailhouse_paravirt();
+ return apic == &apic_physflat || irq_remap_mode == IRQ_REMAP_XAPIC_MODE ||
+ num_possible_cpus() > 8 || jailhouse_paravirt();
}
static struct apic apic_physflat __ro_after_init = {
diff --git a/arch/x86/kernel/apic/local.h b/arch/x86/kernel/apic/local.h
index 842fe28496be..f633ab6dfa9f 100644
--- a/arch/x86/kernel/apic/local.h
+++ b/arch/x86/kernel/apic/local.h
@@ -22,6 +22,8 @@ void x2apic_send_IPI_allbutself(int vector);
void x2apic_send_IPI_self(int vector);
extern u32 x2apic_max_apicid;
+extern int irq_remap_mode;
+
/* IPI */
DECLARE_STATIC_KEY_FALSE(apic_use_ipi_shorthand);
Awesome! This patch does seem to make the system boot, thank you for your help
<3
Tested-by: Lyude Paul <[email protected]>
On Thu, 2024-05-16 at 15:38 +0200, Thomas Gleixner wrote:
> Lyude!
>
> On Wed, May 15 2024 at 19:15, Lyude Paul wrote:
> > On Tue, 2024-05-14 at 10:25 +0200, Thomas Gleixner wrote:
> > >
> > > Which one of the debug patches did you use?
> >
> > The one you sent on 4/18, when you also asked me for the output of
> > /sys/kernel/debug/x86/topo/
> >
> > (I thought I remembered that patch not booting on previous kernels but maybe
> > I'm misremembering)
>
> It booted with the -rc kernel too. :)
>
> I found an interesting and probably related difference in the boot logs
> though. Up to 8 possible CPUs the kernel uses logical destination mode
> for the APIC. With more than 8 it uses physical destination mode.
>
> Can you please test the patch below on top of 6.9 and validate that it
> boots w/o any magic command line parameter?
>
> Thanks,
>
> tglx
> ---
> diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
> index 66fd4b2a37a3..db5e93a7c194 100644
> --- a/arch/x86/kernel/apic/apic.c
> +++ b/arch/x86/kernel/apic/apic.c
> @@ -1883,6 +1883,8 @@ static inline void try_to_enable_x2apic(int remap_mode) { }
> static inline void __x2apic_enable(void) { }
> #endif /* !CONFIG_X86_X2APIC */
>
> +int irq_remap_mode = -1;
> +
> void __init enable_IR_x2apic(void)
> {
> unsigned long flags;
> @@ -1915,6 +1917,8 @@ void __init enable_IR_x2apic(void)
>
> if (ir_stat < 0)
> restore_ioapic_entries();
> + else
> + irq_remap_mode = ir_stat;
> legacy_pic->restore_mask();
> local_irq_restore(flags);
> }
> diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c
> index f37ad3392fec..5cc386db3557 100644
> --- a/arch/x86/kernel/apic/apic_flat_64.c
> +++ b/arch/x86/kernel/apic/apic_flat_64.c
> @@ -12,6 +12,7 @@
> #include <linux/export.h>
> #include <linux/acpi.h>
>
> +#include <asm/irq_remapping.h>
> #include <asm/jailhouse_para.h>
> #include <asm/apic.h>
>
> @@ -130,7 +131,8 @@ static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
>
> static int physflat_probe(void)
> {
> - return apic == &apic_physflat || num_possible_cpus() > 8 || jailhouse_paravirt();
> + return apic == &apic_physflat || irq_remap_mode == IRQ_REMAP_XAPIC_MODE ||
> + num_possible_cpus() > 8 || jailhouse_paravirt();
> }
>
> static struct apic apic_physflat __ro_after_init = {
> diff --git a/arch/x86/kernel/apic/local.h b/arch/x86/kernel/apic/local.h
> index 842fe28496be..f633ab6dfa9f 100644
> --- a/arch/x86/kernel/apic/local.h
> +++ b/arch/x86/kernel/apic/local.h
> @@ -22,6 +22,8 @@ void x2apic_send_IPI_allbutself(int vector);
> void x2apic_send_IPI_self(int vector);
> extern u32 x2apic_max_apicid;
>
> +extern int irq_remap_mode;
> +
> /* IPI */
>
> DECLARE_STATIC_KEY_FALSE(apic_use_ipi_shorthand);
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
Lyude!
On Wed, May 22 2024 at 15:35, Lyude Paul wrote:
Thank for testing!
> Awesome! This patch does seem to make the system boot, thank you for
> your help
The only thing what's awesome here is that it confirms my analysis of
the underlying problem. I offered Borislav a bet on that, but he
politely declined :(
The not so awesome part is the question what to do with that insight.
The first issue is that we don't know whether that's only a problem on
your particular system or if there is an underlying systematic problem
on that particular CPU variant (model/stepping).
Unless the AMD folks can give an authoritative answer we have three options:
1) Targeted via quirk
As you are so far the only one complaining about this, it might be
sufficient to enforce the physical flat mode for your particular
machine via a DMI quirk or on the actual CPU model/stepping.
2) Tie it to interrupt remapping
That's the patch I provided you for testing
3) Remove the default logical destination mode on 64bit completely
My favourite
#1 is stupid IMO because it's likely that other systems are affected by
this nonsense and I don't want to end up adding quirks over and over
#2 is silly because it effectively enforces physical destination mode on
any system which has interrupt remapping available in hardware.
That's pretty much everything halfways modern.
#3 makes a lot of sense because:
- it reduces the amount of code
Given the trend of the last decade this actually removes code which
will be used less frequently as the number of logical CPUs keeps
increasing.
- the only benefit of logical destination mode over physical
destination mode is the ability to send IPIs to multiple CPUs in
one operation.
The question is whether this still matters.
IMO it does not matter because anything which is IPI sensitive is
running on machines which have more than 8 CPUs today. The time
where 8 CPU (threads) workstations and servers were state of the
art are long gone.
- physical destination mode is guaranteed to work because it's the
only way to get a CPU up and running via the INIT/INIT/STARTUP
sequence, while obvioulsy logical destination mode has its issues
not only on the system at hand (see physflat_acpi_madt_oem_check()).
Patch for this below.
Thanks,
tglx
---
arch/x86/kernel/apic/apic_flat_64.c | 116 ------------------------------------
1 file changed, 3 insertions(+), 113 deletions(-)
--- a/arch/x86/kernel/apic/apic_flat_64.c
+++ b/arch/x86/kernel/apic/apic_flat_64.c
@@ -18,126 +18,19 @@
#include "local.h"
static struct apic apic_physflat;
-static struct apic apic_flat;
-struct apic *apic __ro_after_init = &apic_flat;
+struct apic *apic __ro_after_init = &apic_phys_flat;
EXPORT_SYMBOL_GPL(apic);
-static int flat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
-{
- return 1;
-}
-
-static void _flat_send_IPI_mask(unsigned long mask, int vector)
-{
- unsigned long flags;
-
- local_irq_save(flags);
- __default_send_IPI_dest_field(mask, vector, APIC_DEST_LOGICAL);
- local_irq_restore(flags);
-}
-
-static void flat_send_IPI_mask(const struct cpumask *cpumask, int vector)
-{
- unsigned long mask = cpumask_bits(cpumask)[0];
-
- _flat_send_IPI_mask(mask, vector);
-}
-
-static void
-flat_send_IPI_mask_allbutself(const struct cpumask *cpumask, int vector)
-{
- unsigned long mask = cpumask_bits(cpumask)[0];
- int cpu = smp_processor_id();
-
- if (cpu < BITS_PER_LONG)
- __clear_bit(cpu, &mask);
-
- _flat_send_IPI_mask(mask, vector);
-}
-
-static u32 flat_get_apic_id(u32 x)
-{
- return (x >> 24) & 0xFF;
-}
-
-static int flat_probe(void)
-{
- return 1;
-}
-
-static struct apic apic_flat __ro_after_init = {
- .name = "flat",
- .probe = flat_probe,
- .acpi_madt_oem_check = flat_acpi_madt_oem_check,
-
- .dest_mode_logical = true,
-
- .disable_esr = 0,
-
- .init_apic_ldr = default_init_apic_ldr,
- .cpu_present_to_apicid = default_cpu_present_to_apicid,
-
- .max_apic_id = 0xFE,
- .get_apic_id = flat_get_apic_id,
-
- .calc_dest_apicid = apic_flat_calc_apicid,
-
- .send_IPI = default_send_IPI_single,
- .send_IPI_mask = flat_send_IPI_mask,
- .send_IPI_mask_allbutself = flat_send_IPI_mask_allbutself,
- .send_IPI_allbutself = default_send_IPI_allbutself,
- .send_IPI_all = default_send_IPI_all,
- .send_IPI_self = default_send_IPI_self,
- .nmi_to_offline_cpu = true,
-
- .read = native_apic_mem_read,
- .write = native_apic_mem_write,
- .eoi = native_apic_mem_eoi,
- .icr_read = native_apic_icr_read,
- .icr_write = native_apic_icr_write,
- .wait_icr_idle = apic_mem_wait_icr_idle,
- .safe_wait_icr_idle = apic_mem_wait_icr_idle_timeout,
-};
-
-/*
- * Physflat mode is used when there are more than 8 CPUs on a system.
- * We cannot use logical delivery in this case because the mask
- * overflows, so use physical mode.
- */
-static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
-{
-#ifdef CONFIG_ACPI
- /*
- * Quirk: some x86_64 machines can only use physical APIC mode
- * regardless of how many processors are present (x86_64 ES7000
- * is an example).
- */
- if (acpi_gbl_FADT.header.revision >= FADT2_REVISION_ID &&
- (acpi_gbl_FADT.flags & ACPI_FADT_APIC_PHYSICAL)) {
- printk(KERN_DEBUG "system APIC only can use physical flat");
- return 1;
- }
-
- if (!strncmp(oem_id, "IBM", 3) && !strncmp(oem_table_id, "EXA", 3)) {
- printk(KERN_DEBUG "IBM Summit detected, will use apic physical");
- return 1;
- }
-#endif
-
- return 0;
-}
-
static int physflat_probe(void)
{
- return apic == &apic_physflat || num_possible_cpus() > 8 || jailhouse_paravirt();
+ return 1;
}
static struct apic apic_physflat __ro_after_init = {
.name = "physical flat",
.probe = physflat_probe,
- .acpi_madt_oem_check = physflat_acpi_madt_oem_check,
.dest_mode_logical = false,
@@ -167,7 +60,4 @@ static struct apic apic_physflat __ro_af
.safe_wait_icr_idle = apic_mem_wait_icr_idle_timeout,
};
-/*
- * We need to check for physflat first, so this order is important.
- */
-apic_drivers(apic_physflat, apic_flat);
+apic_drivers(apic_physflat);
On 23.05.24 00:12, Thomas Gleixner wrote:
> On Wed, May 22 2024 at 15:35, Lyude Paul wrote:
>
>> Awesome! This patch does seem to make the system boot, thank you for
>> your help
+1
> The only thing what's awesome here is that it confirms my analysis of
> the underlying problem. I offered Borislav a bet on that, but he
> politely declined :(
>
> The not so awesome part is the question what to do with that insight.
>
> The first issue is that we don't know whether that's only a problem on
> your particular system or if there is an underlying systematic problem
> on that particular CPU variant (model/stepping).
FWIW, Arch Linux and openSUSE Tumbleweed switched to 6.9.1 in the past
few days (Fedora will likely follow in ~two weeks) and I have not yet
seen any reports about problems like this, which is a first small
indicator that this might be a system specific problem; if any such
reports come up I'll mention it here.
Ciao, Thorsten
Lyude!
On Thu, May 23 2024 at 00:12, Thomas Gleixner wrote:
> The first issue is that we don't know whether that's only a problem on
> your particular system or if there is an underlying systematic problem
> on that particular CPU variant (model/stepping).
Something I wanted to ask before. Is the BIOS of that machine up to
date?
Thanks,
tglx
Lyude!
On Thu, May 23 2024 at 12:47, Thomas Gleixner wrote:
> Something I wanted to ask before. Is the BIOS of that machine up to
> date?
Any update on this? I really don't want to add magic workarounds if
there is a BIOS update which fixes it.
Thanks,
tglx
Ah whoops! Sorry I totally missed this question, thank you for poking
again
I'm not sure it is but I will double check tomorrow and let you know.
On Wed, 2024-05-29 at 00:43 +0200, Thomas Gleixner wrote:
> Lyude!
>
> On Thu, May 23 2024 at 12:47, Thomas Gleixner wrote:
> > Something I wanted to ask before. Is the BIOS of that machine up to
> > date?
>
> Any update on this? I really don't want to add magic workarounds if
> there is a BIOS update which fixes it.
>
> Thanks,
>
> tglx
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
On Wed, 2024-05-29 at 00:43 +0200, Thomas Gleixner wrote:
> Lyude!
>
> On Thu, May 23 2024 at 12:47, Thomas Gleixner wrote:
> > Something I wanted to ask before. Is the BIOS of that machine up to
> > date?
>
> Any update on this? I really don't want to add magic workarounds if
> there is a BIOS update which fixes it.
OK - just booted a normal kernel up after updating the BIOS and you're
right - that does appear to solve the problem :). Thank you!
>
> Thanks,
>
> tglx
>
--
Cheers,
Lyude Paul (she/her)
Software Engineer at Red Hat
On Wed, Jun 05 2024 at 19:15, Lyude Paul wrote:
> On Wed, 2024-05-29 at 00:43 +0200, Thomas Gleixner wrote:
>> Lyude!
>>
>> On Thu, May 23 2024 at 12:47, Thomas Gleixner wrote:
>> > Something I wanted to ask before. Is the BIOS of that machine up to
>> > date?
>>
>> Any update on this? I really don't want to add magic workarounds if
>> there is a BIOS update which fixes it.
>
> OK - just booted a normal kernel up after updating the BIOS and you're
> right - that does appear to solve the problem :). Thank you!
Perfect!
Thanks
Thomas