On Sunday 29 February 2004 03:43, Michael Frank wrote:
> On Fri, 27 Feb 2004 14:53:45 -0600, Jesse Pollard <[email protected]>
wrote:
> > On Friday 27 February 2004 13:19, Michael Frank wrote:
> >> On Fri, 27 Feb 2004 12:55:55 -0600, Matt Mackall <[email protected]> wrote:
> >> > On Fri, Feb 27, 2004 at 09:44:44AM -0800, Grover, Andrew wrote:
> >> >> > From: Helge Hafting [mailto:[email protected]]
> >> >> >
> >> >> > Grover, Andrew wrote:
> >> >> > > Is the assumption that hardirq handlers are superfast also
> >> >> >
> >> >> > the reason
> >> >> >
> >> >> > > why Linux calls all handlers on a shared interrupt, even if
> >> >> >
> >> >> > the first
> >> >> >
> >> >> > > handler reports it was for its device?
> >> >> >
> >> >> > No, it is the other way around. hardirq handlers have to be
> >> >> > superfast because linux usually _have to_ call all the handlers of
> >> >> > a shared irq.
> >> >> >
> >> >> > The fact that one device did indeed have an interrupt for us
> >> >> > doesn't mean
> >> >> > that the others didn't. So all of them have to be checked to be
> >> >> > safe.
> >> >>
> >> >> If a device later in the handler chain is also interrupting, then the
> >> >> interrupt will immediately trigger again. The irq line will remain
> >> >> asserted until nobody is asserting it.
> >> >>
> >> >> If the LAST guy in the chain is the one with the interrupt, then you
> >> >> basically get today's ISR "call each handler" behavior, but it should
> >> >> be possible to in some cases to get less time spent in do_IRQ.
> >> >
> >> > Let's imagine you have n sources simultaneously interrupting on a
> >> > given descriptor. Check the first, it's happening, acknowledge it,
> >> > exit, notice interrupt still asserted, check the first, nope, check
> >> > the second, yep, exit, etc. By the time we've made it to the nth ISR,
> >> > we've banged on the first one n times, the second n-1 times, etc. In
> >> > other words, early chain termination has an O(n^2) worst case.
> >>
> >> With level triggered you can just walk the chain, exit at the end of the
> >> first cycle and should the IRQ still be asserted you just incur the
> >> overhead of exit and reentry of the ISR.
> >>
> >> Even with edge, I would not check alwasy from the beginning of the
> >> chain...
> >
> > You should... after all that first entry in the chain has the highest
> > priority
>
> Please also consider that physcial IRQ's are in practice assigned "semi
> randomly".
I think that would be up to the installer.
> In a way IRQ priorities in general purpose computing applications are
> irrelevant :)
I would say that they have a priority, just that the priority isn't being
used.
> Lets say you have a bunch of devices demand service, all have to be
> serviced but is either not significant which gets done first or the
> practival IRQ priorities are "less than optimal":
>
> Keyboard IRQ1 Well buffered
> NIC1 IRQ10 Buffered
> NIC2 IRQ10 Buffered
> USB IRQ11 Buffered
> serial port IRQ3 Small FIFO (assuming '550)
> serial port IRQ4 "-"
>
> Here, serial ports would would be most critical, however the priorities of
> IRQ3 and IRQ4 are below priorities of IRQ10 and IRQ11...
I don't think these are all using "shared IRQ...". I was my understanding that
each IRQ had its own chain. And the priority of the drivers in that chain
is determined by the order.
> IMO, the best practical approach is to keep efficiency by walking the chain
> only once. If the chain is longer, it may be worthwhile to check the IRQ
> and exit walking the chain if it is inactive.
That depends entirely on the chain. If there is one or more high speed devices
such that walking the chain is faster that responding to an interrupt, then it
is DEFINITELY worth it to walk the chain.
> Priorities in chains would make sense only in specialized applications
> under controlled circumstances wrt IRQ and linking devices into chain at
> the priority desired.
No necessarily specialized applications - just ones that should/must not
loose data due to starvation. Consider the problems caused by relatively
slow devices such as a mouse. Resyncronization is a real problem, which is
one of the reasons to give it a nonshared IRQ. The same could be said of
a PPP connection over a serial line...
If they did share an interrupt, you make the PPP serial line higher priority
than the mouse... Yet you still don't want to loose mouse syncronization - so
check it anyway. Handling it is still faster than an additional interrupt,
and with less overhead.
Granted, this is more important on slower hardware or basic interactive
service, but it all contributes to overhead. Properly organized (and used)
chains will reduce the overhead.
On Mon, 1 Mar 2004 10:57:42 -0600, Jesse Pollard <[email protected]> wrote:
> On Sunday 29 February 2004 03:43, Michael Frank wrote:
>> On Fri, 27 Feb 2004 14:53:45 -0600, Jesse Pollard <[email protected]>
> wrote:
>> > On Friday 27 February 2004 13:19, Michael Frank wrote:
>> >> On Fri, 27 Feb 2004 12:55:55 -0600, Matt Mackall <[email protected]> wrote:
>> >> > On Fri, Feb 27, 2004 at 09:44:44AM -0800, Grover, Andrew wrote:
>> >> >> > From: Helge Hafting [mailto:[email protected]]
>> >> >> >
>> >> >> > Grover, Andrew wrote:
>> >> >> > > Is the assumption that hardirq handlers are superfast also
>> >> >> >
>> >> >> > the reason
>> >> >> >
>> >> >> > > why Linux calls all handlers on a shared interrupt, even if
>> >> >> >
>> >> >> > the first
>> >> >> >
>> >> >> > > handler reports it was for its device?
>> >> >> >
>> >> >> > No, it is the other way around. hardirq handlers have to be
>> >> >> > superfast because linux usually _have to_ call all the handlers of
>> >> >> > a shared irq.
>> >> >> >
>> >> >> > The fact that one device did indeed have an interrupt for us
>> >> >> > doesn't mean
>> >> >> > that the others didn't. So all of them have to be checked to be
>> >> >> > safe.
>> >> >>
>> >> >> If a device later in the handler chain is also interrupting, then the
>> >> >> interrupt will immediately trigger again. The irq line will remain
>> >> >> asserted until nobody is asserting it.
>> >> >>
>> >> >> If the LAST guy in the chain is the one with the interrupt, then you
>> >> >> basically get today's ISR "call each handler" behavior, but it should
>> >> >> be possible to in some cases to get less time spent in do_IRQ.
>> >> >
>> >> > Let's imagine you have n sources simultaneously interrupting on a
>> >> > given descriptor. Check the first, it's happening, acknowledge it,
>> >> > exit, notice interrupt still asserted, check the first, nope, check
>> >> > the second, yep, exit, etc. By the time we've made it to the nth ISR,
>> >> > we've banged on the first one n times, the second n-1 times, etc. In
>> >> > other words, early chain termination has an O(n^2) worst case.
>> >>
>> >> With level triggered you can just walk the chain, exit at the end of the
>> >> first cycle and should the IRQ still be asserted you just incur the
>> >> overhead of exit and reentry of the ISR.
>> >>
>> >> Even with edge, I would not check alwasy from the beginning of the
>> >> chain...
>> >
>> > You should... after all that first entry in the chain has the highest
>> > priority
>>
>> Please also consider that physcial IRQ's are in practice assigned "semi
>> randomly".
>
> I think that would be up to the installer.
How?
>
>> In a way IRQ priorities in general purpose computing applications are
>> irrelevant :)
>
> I would say that they have a priority, just that the priority isn't being
> used.
Not usable in 2 dimensions - A) the IRQ can't be controlled, B) the location
in the chain can't be contrlolled.
>
>> Lets say you have a bunch of devices demand service, all have to be
>> serviced but is either not significant which gets done first or the
>> practival IRQ priorities are "less than optimal":
>>
>> Keyboard IRQ1 Well buffered
>> NIC1 IRQ10 Buffered
>> NIC2 IRQ10 Buffered
>> USB IRQ11 Buffered
MOUSE IRQ12 ?
>> serial port IRQ3 Small FIFO (assuming '550)
>> serial port IRQ4 "-"
>>
>> Here, serial ports would would be most critical, however the priorities of
>> IRQ3 and IRQ4 are below priorities of IRQ10 and IRQ11...
>
> I don't think these are all using "shared IRQ...". I was my understanding that
> each IRQ had its own chain. And the priority of the drivers in that chain
> is determined by the order.
The NICs's are both shared on IRQ10.
>
>> IMO, the best practical approach is to keep efficiency by walking the chain
>> only once. If the chain is longer, it may be worthwhile to check the IRQ
>> and exit walking the chain if it is inactive.
>
> That depends entirely on the chain. If there is one or more high speed devices
> such that walking the chain is faster that responding to an interrupt, then it
> is DEFINITELY worth it to walk the chain.
>
>> Priorities in chains would make sense only in specialized applications
>> under controlled circumstances wrt IRQ and linking devices into chain at
>> the priority desired.
>
> No necessarily specialized applications - just ones that should/must not
> loose data due to starvation. Consider the problems caused by relatively
> slow devices such as a mouse. Resyncronization is a real problem, which is
> one of the reasons to give it a nonshared IRQ. The same could be said of
> a PPP connection over a serial line...
>
> If they did share an interrupt, you make the PPP serial line higher priority
> than the mouse... Yet you still don't want to loose mouse syncronization - so
> check it anyway. Handling it is still faster than an additional interrupt,
> and with less overhead.
Right, the mouse is important too. A major practical system level issue
is shown in above example: The PPP serial line could be easily starved by
those 2 NIC's+USB+Mouse with higher priority.
Baudrate 115,200 / 11bit is about 10000 chars/s peak. Given a FIFO size
of 16 + 1 RxHR (best case), after 1ms-1.5ms characters get lost.
A fix would require reprogram PIC IRQ priorities here to make IRQ3 and IRQ4
higher priority than what comes in from the cascaded controller (via IRQ2).
Obviously more APIC's abound but there still is need for a facility to
manage priorities, and this (primary priority) is more significant than
what happens within a chain.
>
> Granted, this is more important on slower hardware or basic interactive
> service, but it all contributes to overhead. Properly organized (and used)
> chains will reduce the overhead.
>
More control over physical IRQ priority would be an initial step helping
with non-shared legacy hardware and device with small buffers.
More modern hardware like USB is using large buffers, DMA and is less critical.
And perhaps a config-bit per chain to tell it whether to rescan from scratch
would be useful to address the concerns mentioned, _and_ then there should
be a facility to link a interrupt into a specific location in a chain.
Regards
Michael
On Monday 01 March 2004 11:35, Michael Frank wrote:
> On Mon, 1 Mar 2004 10:57:42 -0600, Jesse Pollard <[email protected]>
wrote:
> > On Sunday 29 February 2004 03:43, Michael Frank wrote:
> >> On Fri, 27 Feb 2004 14:53:45 -0600, Jesse Pollard
> >> <[email protected]>
> >
> > wrote:
[snip]
> >> >
> >> > You should... after all that first entry in the chain has the highest
> >> > priority
> >>
> >> Please also consider that physcial IRQ's are in practice assigned "semi
> >> randomly".
> >
> > I think that would be up to the installer.
>
> How?
Usually (at least in my boxes, anyway) the IRQ tends to be associated with the
PCI slot number. suitable ordering of the interfaces in the slots was what I
was thinking of. But then, I don't overload the IRQs either.
>
> >> In a way IRQ priorities in general purpose computing applications are
> >> irrelevant :)
> >
> > I would say that they have a priority, just that the priority isn't being
> > used.
>
> Not usable in 2 dimensions - A) the IRQ can't be controlled, B) the
> location in the chain can't be contrlolled.
The location in the chain should be controled - even if it is nothing
other than a driver based priority number used to insert in the chain
at the appropriate location.
> >> Lets say you have a bunch of devices demand service, all have to be
> >> serviced but is either not significant which gets done first or the
> >> practival IRQ priorities are "less than optimal":
> >>
> >> Keyboard IRQ1 Well buffered
> >>
> >> NIC1 IRQ10 Buffered
> >> NIC2 IRQ10 Buffered
> >> USB IRQ11 Buffered
>
> MOUSE IRQ12 ?
>
> >> serial port IRQ3 Small FIFO (assuming '550)
> >> serial port IRQ4 "-"
> >>
> >> Here, serial ports would would be most critical, however the priorities
> >> of IRQ3 and IRQ4 are below priorities of IRQ10 and IRQ11...
> >
> > I don't think these are all using "shared IRQ...". I was my understanding
> > that each IRQ had its own chain. And the priority of the drivers in that
> > chain is determined by the order.
>
> The NICs's are both shared on IRQ10.
>
[snip]
> >
> > If they did share an interrupt, you make the PPP serial line higher
> > priority than the mouse... Yet you still don't want to loose mouse
> > syncronization - so check it anyway. Handling it is still faster than an
> > additional interrupt, and with less overhead.
>
> Right, the mouse is important too. A major practical system level issue
> is shown in above example: The PPP serial line could be easily starved by
> those 2 NIC's+USB+Mouse with higher priority.
>
> Baudrate 115,200 / 11bit is about 10000 chars/s peak. Given a FIFO size
> of 16 + 1 RxHR (best case), after 1ms-1.5ms characters get lost.
>
> A fix would require reprogram PIC IRQ priorities here to make IRQ3 and IRQ4
> higher priority than what comes in from the cascaded controller (via IRQ2).
>
> Obviously more APIC's abound but there still is need for a facility to
> manage priorities, and this (primary priority) is more significant than
> what happens within a chain.
Yes.
>
> > Granted, this is more important on slower hardware or basic interactive
> > service, but it all contributes to overhead. Properly organized (and
> > used) chains will reduce the overhead.
>
> More control over physical IRQ priority would be an initial step helping
> with non-shared legacy hardware and device with small buffers.
>
> More modern hardware like USB is using large buffers, DMA and is less
> critical.
>
> And perhaps a config-bit per chain to tell it whether to rescan from
> scratch would be useful to address the concerns mentioned, _and_ then there
> should be a facility to link a interrupt into a specific location in a
> chain.
A round-robin IRQ scheduler would also help those at the same priority. Again,
software implementation sucks (I used to do embeded systems).
It really looks like the old arch is being pushed way beyond what is
reasonable. It is really time for vectored interrupts to resurface (re: the
old PDP 11 style, allowing each interrupt to have it's own software priority,
AND interrupt routine... which also happens to be where "spl" started from :-)
It really looks like the PCI can't handle more than about 3 interrupt sources
at a time, in spite of the APIC/cascade faking it to look like more.
The busyist box I have (a server) has two SCSI controllers and two IDE (one
IRQ each, and the IDE is used only for primary boot, and backup), and an
ethernet. The PS2 mouse and keyboard don't get used (much). The second
busyist has three ethernet connections (1 from DSL, 1 from wireless, 1 from
the internal net - it is a firewall) and an IDE boot. Again, the
keyboard/mouse are not used.