Here is the patch to turn kirqd into a config option if it is really
needed. I don't see why the noirqbalance functionality isn't enough for
now. Is there anything currently keeping a userspace irq balancer from
working as 2.5 stands today? It dosen't look like it to me.
Keith
On Wed, May 21, 2003 at 11:28:41AM -0700, Keith Mannthey wrote:
> Here is the patch to turn kirqd into a config option if it is really
> needed. I don't see why the noirqbalance functionality isn't enough for
> now. Is there anything currently keeping a userspace irq balancer from
> working as 2.5 stands today? It dosen't look like it to me.
> Keith
This will do, though my preference is to make the code actually
understand what DESTMOD means in IO-APIC RTE's and what DFR means
for local APIC's instead of the rather ridiculous workarounds for
not doing so currently present.
There are a couple of obstacles to doing this:
(1) There is no true mechanism for correlating IO-APIC's with the
APIC buses corresponding to a given cluster for APIC. The
assumption is largely global addressibility a la xAPIC.
(2) DESTMOD is not a static property. Dynamically switching between
logical and physical DESTMOD is fully possible and allows a
somewhat greater variety of cpu sets to be handled on APIC.
I'd also like for there to be validity checking and explicit error
returns from the affinity setting API.
I'm not entirely happy with the genapic bits. Basically the APIC
is relatively well-standardized, and I'd rather the point-by-point
"this codepath must differ" abstraction be built atop such an APIC
manipulation "library" as it were. For instance:
(1) cpu wakeup via NMI is possible on ordinary machines; INIT merely
cannot address cpus above the limit of APIC's physical
addressing scheme (which is 4 bits) and so is required for
pre-xAPIC machines with > 16 cpus or machines where only
logical interrupts are routed across bus boundaries (be they
APIC buses or memory buses).
(2) clustered hierarchical DFR is usable on single APIC bus boxen and
xAPIC boxen with provisos for cluster ID's being misrouted.
(3) IO-APIC RTE formats are not magical properties of the machine;
there is just logical and physical DESTMOD and representability
of target cpu sets in the logical format and physical format
and the dependence of the logical format on the cpus' DFR's.
These somewhat obvious observations imply to me that common code should
be used to manipulate the local APIC and IO-APIC and the machine-
specific code should choose its preferred modes when calling it, not
provide a private implementation or magic values to stuff into various
registers that specialize the APIC handling to a particular mode.
OTOH I don't see much (if any) chance of any of this happening since
"just barely works" suffices for most people's purposes and the
moderately large amount of work required to do all this ends up with
approximately zero functional difference in the end.
Thanks.
-- wli
> You can build masks of capable clusters easily, even for NUMAQ
Only kinda. Boxes with Hyperthreaded cpus have an odd ordering
scheme. The BIOS is free to assign apicids at will to any cpu. It is
not forced to any certain scheme. On some hyperthreaded boxes the 2nd
cpu is on the same apicid cluster even thought the cpu numbers are far
apart.
This makes building meaningful apicid masks (more than one cpu) a bit
tricky. For example a user would have to know that cpus 1,2,9,10 were
on the same cluster not (1,2,3,4) as you would expect. Since the bios
can do what it will it makes it hard to build masks of capable clusters
easily in all situations.
Keith
From: Keith Mannthey <[email protected]>
Date: 21 May 2003 16:39:29 -0700
For example a user would have to know that cpus 1,2,9,10 were
on the same cluster not (1,2,3,4) as you would expect.
Nothing prevents us from exporting these mappings to userspace.
Just like we can export a "possible mask" for each interrupt
source.
On Wed, May 21, 2003 at 04:39:29PM -0700, Keith Mannthey wrote:
> Only kinda. Boxes with Hyperthreaded cpus have an odd ordering
> scheme. The BIOS is free to assign apicids at will to any cpu. It is
> not forced to any certain scheme. On some hyperthreaded boxes the 2nd
> cpu is on the same apicid cluster even thought the cpu numbers are far
> apart.
> This makes building meaningful apicid masks (more than one cpu) a bit
> tricky. For example a user would have to know that cpus 1,2,9,10 were
> on the same cluster not (1,2,3,4) as you would expect. Since the bios
> can do what it will it makes it hard to build masks of capable clusters
> easily in all situations.
APIC issues can be dealt with very, very simply.
(1) for each cpu, report the physical APIC ID
(2) for each cpu, report the logical APIC ID
(or if using only physical IPI's whatever the BIOS left in the LDR)
(3) report the DFR setting used globally across the system
(4) for each IO-APIC, report where it's attached (bus and node)
(5) report the contents of each IO-APIC RTE
(5a) report the destination (interpretation depends on DESTMOD)
(5b) report DESTMOD as either logical or physical
(5c) report what it's connected to (irq, possibly driver name)
(6) report the APIC revision(s) (to distinguish APIC from xAPIC)
(7) report the IO-APIC revision(s) (for completeness)
The cpus a given IO-APIC RTE can address with physical DESTMOD can then
be determined from the APIC revision, and the cpus a given IO-APIC RTE
can address with logical DESTMOD can then be determined from the APIC
revision and (global and immutable, though the register is per- local
APIC; there's no good way to switch over, and no reason to) DFR setting.
The logical CPU number used to refer to CPUs by the kernel bears no
relation to APIC ID's apart from arithmetic schemes artificially
imposed by the implementation. Fully tabulating the APIC ID's for all
the CPUs as in (1) and (2) is sufficient information for userspace to
construct and invert the relation as required to determine APIC cluster
membership. It is also possible to directly export APIC clusters as
sysfs objects and enumerate the cpus, though (IMHO) it's best to merely
expose the information the kernel acts on as it stands now and let
userspace infer the rest.
In principle one could also export the ability to set IO-APIC RTE's
DESTMOD bits on the fly, given proper validity checks for
addressibility and the like (I'm assuming one would rather barf than
deadlock the box even if some additional code were required). The one
box where it matters doesn't care to use irqbalance anyway, though.
Basically, spill your guts as to what you've got and let userspace
think about how to do the right thing with it.
-- wli
On Wed, May 21, 2003 at 04:39:29PM -0700, Keith Mannthey wrote:
> This makes building meaningful apicid masks (more than one cpu) a bit
> tricky. For example a user would have to know that cpus 1,2,9,10 were
> on the same cluster not (1,2,3,4) as you would expect. Since the bios
> can do what it will it makes it hard to build masks of capable clusters
> easily in all situations.
with sysfs the kernel can export some topology info; iirc that was desired
anyway for other HPC applications anyway ?