>>> > Yes, our feeling it is possible to handle all non-NUMAQ
>>> systems pretty
>>> > generically in terms of APIC setup and interrupt routing. We can use
>>> > either logical clustered or physical destination modes. But
>>> for NUMAQ
>>> > systems, interrupt routing has to know about the local
>>> nodes and have
>>> > necessary logic to do the routing withing local node.
>>>
>>> NUMA-Q doesn't have to know about the local nodes. I set it up to use
>>> physical delivery broadcast, which is a node-local broadcast ... gave
>>> me NUMA affinity for free. I could also use logical clustered
>>> (p3 style)
>>> addressing, and work out all the node locality, but I don't
>>> see the point.
>>
>> I actually meant interrupt distribution (rather than interrupt routing).
>> AFAIK, interrupt distribution right now assumes flat logical setup and
>> tries to distribute the interrupt. And is disabled in case of clustered
>> APIC mode. I was just thinking loud, about the changes interrupt
>> distribution code should have for systems using clustered APIC/physical
>> mode (NUMAQ and non-NUMAQ).
>Actually, if you're talking about irq_balance, that needs fixing for all
>NUMA systems to get affininity, not just NUMA-Q. It then needs an
>abstraction layer to do "program the IO-APIC with a cpu_bitmask" that's
>different for each apic addressing mode used.
Some platforms (like certain ES7000s) won't tolerate any bit masks
programmed into the RTE because their balancing is done entirely in
hardware, similar to XTPR mechanism for Fosters. For those I suggest to have
an escape door, in the form of boot parameter such as "irq_balance=no". It
was suggested to us by SuSE and worked great - I could turn it off in our
platform code unconditionally. It could also help those who can use irq
balancing as is but might want to implement their own balancing schema.
--Natalie
>M.
> Some platforms (like certain ES7000s) won't tolerate any bit masks
> programmed into the RTE because their balancing is done entirely in
> hardware, similar to XTPR mechanism for Fosters. For those I suggest to
> have an escape door, in the form of boot parameter such as
> "irq_balance=no". It was suggested to us by SuSE and worked great - I
> could turn it off in our platform code unconditionally. It could also
> help those who can use irq balancing as is but might want to implement
> their own balancing schema.
Having a boot-time parameter is useful, but I'd like it to default to off
without a paramater for the platforms where it's just broken. At the
moment there's an "if (clustered_apic_mode) return;" stuck at the top
of balance_irq, the latest series of patches changes that to
"if (no_balance_irq) return;", which is set by NUMA-Q. If we can set
up no_balance_irq to default correctly, but be possibly overridden by
the boot-time parameter, I think we'd have the best of both worlds.
M.