We recently ran into some issues caused by APIC bus latency. I was
reminded of that by the recent discussion of NAPI and related
interrupt-performance matters.
Intel processors that predate Pentium 4 but use an APIC transmit APIC
messages over a serial APIC bus, typically at 16.7 MHz. (Pentium 4
uses the system bus for APIC messages.)
A message exchange (IO-APIC sends an interrupt message; CPU sends
back an EOI message) requires from 35 to 48 APIC bus clocks, or 2-3
microseconds. That gets to be pretty significant compared to packet
times, especially at Gbit speeds, but even at 100 MHz, and is the
time required to burst a thousand bytes or more at faster PCI rates.
It's also likely to be significant for inter-processor interrupts,
though I don't know what the implications are here.
--
/Jonathan Lundell.
> A message exchange (IO-APIC sends an interrupt message; CPU sends
> back an EOI message) requires from 35 to 48 APIC bus clocks, or 2-3
> microseconds. That gets to be pretty significant compared to packet
> times, especially at Gbit speeds, but even at 100 MHz, and is the
> time required to burst a thousand bytes or more at faster PCI rates.
>
> It's also likely to be significant for inter-processor interrupts,
> though I don't know what the implications are here.
The big implication so far has been some extremely horrible to debug
irq handling bugs where drivers such as the i810 audio assumed that the
disable of an irq on the pci device was immediate once then pci write
and a pci read to force posting completed.
There are impacts on things like TLB shootdowns where the latency impacts
an SMP crosscall. I'm not sure how bad the impact is on the bigger numa
boxes as I notice Martin uses multiple sends for that
Alan