2011-06-11 15:54:19

by Robert Uhl

[permalink] [raw]
Subject: Access to local APIC registers during an interrupt handler

Hi,

I wrote a small interrupt handler (for a 64 bit system) which is
installed by a kernel module in the IDT. My handler just increments a
global variable and then jumps to the original interrupt handler. So
Linux does (hopefully ;-) not notice my handler. With sysfs I can read
the value of these interrupt counter and know exactly how often a
specific interrupt occured.
Of course on a multicore system the interrupts of all cores are counted
together, but I want to separate between the cores. On newer CPUs I can
use the instruction RDTSCP to get the CPU number in ECX, but on older
CPUs it's unsupported.
So I had the idea to use the local APIC ID to check on which core my
handler is executed, even though sometimes local APIC ID != core number,
but the ID should be at least unique.
I get the address of the local APIC ID register at module init with

u64 lapic_idregister = (u64) fix_to_virt(FIX_APIC_BASE) + 0x20;

and use it in my interrupt handler (of course I push/pop all used
registers):

movq (lapic_idregister), %rcx
movq (%rcx), %rcx

But on real hardware the last instruction seems to cause a page fault or
something (SUSE with 2.6.37.6, Fedora with 2.6.38.6), the system simply
reboots. Without this instruction, the handler is executed without any
problems.
And in qemu with vanilla 2.6.37.6 and a buildroot system everything
works fine!

I already had a look with qemu which instructions are executed at a
local APIC timer interrupt (0xEF) until it writes 0x00 to the local APIC
EOI register (same page), but I still can't figure out what's the
problem with my code.

Maybe someone knows what is missing or if there is any other fast way to
figure out on which core the handler is running?

Kind regards,
Robert


2011-06-12 11:01:31

by Mikael Pettersson

[permalink] [raw]
Subject: Re: Access to local APIC registers during an interrupt handler

Robert Uhl writes:
> Hi,
>
> I wrote a small interrupt handler (for a 64 bit system) which is
> installed by a kernel module in the IDT. My handler just increments a
> global variable and then jumps to the original interrupt handler. So
> Linux does (hopefully ;-) not notice my handler. With sysfs I can read
> the value of these interrupt counter and know exactly how often a
> specific interrupt occured.
> Of course on a multicore system the interrupts of all cores are counted
> together, but I want to separate between the cores. On newer CPUs I can
> use the instruction RDTSCP to get the CPU number in ECX, but on older
> CPUs it's unsupported.
> So I had the idea to use the local APIC ID to check on which core my
> handler is executed, even though sometimes local APIC ID != core number,
> but the ID should be at least unique.
> I get the address of the local APIC ID register at module init with
>
> u64 lapic_idregister = (u64) fix_to_virt(FIX_APIC_BASE) + 0x20;
>
> and use it in my interrupt handler (of course I push/pop all used
> registers):
>
> movq (lapic_idregister), %rcx
> movq (%rcx), %rcx

You're doing a 64-bit load from a 32-bit lapic register.

> But on real hardware the last instruction seems to cause a page fault or
> something (SUSE with 2.6.37.6, Fedora with 2.6.38.6), the system simply
> reboots. Without this instruction, the handler is executed without any
> problems.
> And in qemu with vanilla 2.6.37.6 and a buildroot system everything
> works fine!

If changing the above movq (%rcx), %rcx to use movl instead makes it
work in real HW, then you've found an accepts-invalid bug in qemu.

2011-06-12 12:22:19

by Robert Uhl

[permalink] [raw]
Subject: Re: Access to local APIC registers during an interrupt handler

On 12.06.2011 13:01, Mikael Pettersson wrote:
> > movq (lapic_idregister), %rcx
> > movq (%rcx), %rcx
>
> You're doing a 64-bit load from a 32-bit lapic register.
>
> If changing the above movq (%rcx), %rcx to use movl instead makes it
> work in real HW, then you've found an accepts-invalid bug in qemu.

It works with movl, thank you very much!
I will send a bug report to the qemu team.