Hi all.
I've noticed this problem for a while, but only now decided to ask.
Interrupt balancing doesn't do anything on my system.
CPU0 CPU1
0: 31931808 0 IO-APIC-edge timer
1: 76595 0 IO-APIC-edge i8042
8: 1 0 IO-APIC-edge rtc
9: 1 0 IO-APIC-level acpi
14: 122 1 IO-APIC-edge ide0
16: 4074456 0 IO-APIC-level uhci_hcd, uhci_hcd, radeon@PCI:1:0:0
17: 4295132 0 IO-APIC-level Intel ICH5
18: 2070933 0 IO-APIC-level libata, uhci_hcd, eth0
19: 887311 0 IO-APIC-level uhci_hcd
22: 572530 0 IO-APIC-level ath0
NMI: 31931749 31931636 (I've since disabled the nmi_watchdog)
LOC: 31931252 31931251
ERR: 0
MIS: 0
I enabled the debugging and found that it doesn't think it's worth the
effort. Is that correct? Not a complaint, just curious!
Regards,
Nigel
--
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Ph: +61 (2) 6292 8028 Mob: +61 (417) 100 574
My guess is that irqbalance is not running.
Nigel Cunningham wrote:
>Hi all.
>
>I've noticed this problem for a while, but only now decided to ask.
>Interrupt balancing doesn't do anything on my system.
>
> CPU0 CPU1
> 0: 31931808 0 IO-APIC-edge timer
> 1: 76595 0 IO-APIC-edge i8042
> 8: 1 0 IO-APIC-edge rtc
> 9: 1 0 IO-APIC-level acpi
> 14: 122 1 IO-APIC-edge ide0
> 16: 4074456 0 IO-APIC-level uhci_hcd, uhci_hcd, radeon@PCI:1:0:0
> 17: 4295132 0 IO-APIC-level Intel ICH5
> 18: 2070933 0 IO-APIC-level libata, uhci_hcd, eth0
> 19: 887311 0 IO-APIC-level uhci_hcd
> 22: 572530 0 IO-APIC-level ath0
>NMI: 31931749 31931636 (I've since disabled the nmi_watchdog)
>LOC: 31931252 31931251
>ERR: 0
>MIS: 0
>
>I enabled the debugging and found that it doesn't think it's worth the
>effort. Is that correct? Not a complaint, just curious!
>
>Regards,
>
>Nigel
>
>
Nigel Cunningham wrote:
> Hi all.
>
> I've noticed this problem for a while, but only now decided to ask.
> Interrupt balancing doesn't do anything on my system.
>
> CPU0 CPU1
> 0: 31931808 0 IO-APIC-edge timer
> 1: 76595 0 IO-APIC-edge i8042
> 8: 1 0 IO-APIC-edge rtc
> 9: 1 0 IO-APIC-level acpi
> 14: 122 1 IO-APIC-edge ide0
> 16: 4074456 0 IO-APIC-level uhci_hcd, uhci_hcd, radeon@PCI:1:0:0
> 17: 4295132 0 IO-APIC-level Intel ICH5
> 18: 2070933 0 IO-APIC-level libata, uhci_hcd, eth0
> 19: 887311 0 IO-APIC-level uhci_hcd
> 22: 572530 0 IO-APIC-level ath0
> NMI: 31931749 31931636 (I've since disabled the nmi_watchdog)
> LOC: 31931252 31931251
> ERR: 0
> MIS: 0
>
> I enabled the debugging and found that it doesn't think it's worth the
> effort. Is that correct? Not a complaint, just curious!
What are the results of running irqbalanced?
Jeff
Hi.
On Sat, 2005-02-19 at 17:44, Kwijibo wrote:
> My guess is that irqbalance is not running.
No. It is.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 301 0.0 0.0 0 0 ? SW 16:52 0:00 [kirqd]
The debugging info reports that it doesn't think it's worth doing the
balancing.
Regards,
Nigel
--
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Ph: +61 (2) 6292 8028 Mob: +61 (417) 100 574
Hi Jeff.
On Sat, 2005-02-19 at 17:52, Jeff Garzik wrote:
> Nigel Cunningham wrote:
> What are the results of running irqbalanced?
You mean the debugging output? I can reenable it and record the results
if that's what you mean.
>From memory though, it got to not_worth_the_effort via this code:
if (tmp_loaded == -1) {
/* In the case of small number of heavy interrupt sources,
* loading some of the cpus too much. We use Ingo's original
* approach to rotate them around.
*/
if (!first_attempt && imbalance >= useful_load_threshold) {
rotate_irqs_among_cpus(useful_load_threshold);
return;
}
goto not_worth_the_effort;
}
(arch/i386/kernel/io_apic.c:449.
Regards,
Nigel
--
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Ph: +61 (2) 6292 8028 Mob: +61 (417) 100 574
On Sat, 2005-02-19 at 20:07 +1100, Nigel Cunningham wrote:
> Hi Jeff.
>
> On Sat, 2005-02-19 at 17:52, Jeff Garzik wrote:
> > Nigel Cunningham wrote:
> > What are the results of running irqbalanced?
>
> You mean the debugging output? I can reenable it and record the results
> if that's what you mean.
no Jeff meant
http://people.redhat.com/arjanv/irqbalance/
that app most likely....
On Sat, 2005-02-19 at 20:02 +1100, Nigel Cunningham wrote:
> Hi.
>
> On Sat, 2005-02-19 at 17:44, Kwijibo wrote:
> > My guess is that irqbalance is not running.
>
> No. It is.
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 301 0.0 0.0 0 0 ? SW 16:52 0:00 [kirqd]
>
> The debugging info reports that it doesn't think it's worth doing the
> balancing.
I guess the question was about the userspace irqbalance....
Hi,
On Sat, 2005-02-19 at 20:36, Arjan van de Ven wrote:
> On Sat, 2005-02-19 at 20:07 +1100, Nigel Cunningham wrote:
> > Hi Jeff.
> >
> > On Sat, 2005-02-19 at 17:52, Jeff Garzik wrote:
> > > Nigel Cunningham wrote:
> > > What are the results of running irqbalanced?
> >
> > You mean the debugging output? I can reenable it and record the results
> > if that's what you mean.
>
> no Jeff meant
> http://people.redhat.com/arjanv/irqbalance/
> that app most likely....
Oh... Ok. Never heard of it before.
I'll download it now, but I'm about to go to bed now so the answer will
be a while.
Thanks for the reply.
Nigel
--
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Ph: +61 (2) 6292 8028 Mob: +61 (417) 100 574
Hi again.
Didn't realise it was going to take nothing to install, so I've done it.
IRQs are running on cpu 1 now. Is there some documentation somewhere?
I'm wondering whether I can compile kirqd out.
Thanks and regards,
Nigel
On Sat, 2005-02-19 at 20:36, Arjan van de Ven wrote:
> On Sat, 2005-02-19 at 20:07 +1100, Nigel Cunningham wrote:
> > Hi Jeff.
> >
> > On Sat, 2005-02-19 at 17:52, Jeff Garzik wrote:
> > > Nigel Cunningham wrote:
> > > What are the results of running irqbalanced?
> >
> > You mean the debugging output? I can reenable it and record the results
> > if that's what you mean.
>
> no Jeff meant
> http://people.redhat.com/arjanv/irqbalance/
> that app most likely....
>
--
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Ph: +61 (2) 6292 8028 Mob: +61 (417) 100 574
On Sat, 2005-02-19 at 21:58 +1100, Nigel Cunningham wrote:
> Hi again.
>
> Didn't realise it was going to take nothing to install, so I've done it.
> IRQs are running on cpu 1 now. Is there some documentation somewhere?
> I'm wondering whether I can compile kirqd out.
with irqbalance running yes you can compile kirqd out..
You are right. Kernel balancer doesn't move around the irqs, unless it
has too many interrupts. The logic is moving around interrupts all the
time will not be good on caches. So, there is a threshold above which
the balancer start moving things around.
You should see them moving around if you do 'ping -f' or a big 'dd' from
the disk.
Thanks,
Venki
>-----Original Message-----
>From: [email protected]
>[mailto:[email protected]] On Behalf Of Nigel
>Cunningham
>Sent: Saturday, February 19, 2005 1:02 AM
>To: [email protected]
>Cc: Linux Kernel Mailing List
>Subject: Re: Should kirqd work on HT?
>
>Hi.
>
>On Sat, 2005-02-19 at 17:44, Kwijibo wrote:
>> My guess is that irqbalance is not running.
>
>No. It is.
>
>USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>root 301 0.0 0.0 0 0 ? SW 16:52 0:00 [kirqd]
>
>The debugging info reports that it doesn't think it's worth doing the
>balancing.
>
>Regards,
>
>Nigel
>--
>Nigel Cunningham
>Software Engineer, Canberra, Australia
>http://www.cyclades.com
>
>Ph: +61 (2) 6292 8028 Mob: +61 (417) 100 574
>
>-
>To unsubscribe from this list: send the line "unsubscribe
>linux-kernel" in
>the body of a message to [email protected]
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at http://www.tux.org/lkml/
>
Pallipadi, Venkatesh wrote:
> You are right. Kernel balancer doesn't move around the irqs, unless it
> has too many interrupts. The logic is moving around interrupts all the
> time will not be good on caches. So, there is a threshold above which
> the balancer start moving things around.
>
> You should see them moving around if you do 'ping -f' or a big 'dd' from
> the disk.
If kirqd is moving NIC interrupts, it's broken.
(and another reason why irqbalanced is preferable)
Jeff
--Jeff Garzik <[email protected]> wrote (on Saturday, February 19, 2005 11:30:53 -0500):
> Pallipadi, Venkatesh wrote:
>> You are right. Kernel balancer doesn't move around the irqs, unless it
>> has too many interrupts. The logic is moving around interrupts all the
>> time will not be good on caches. So, there is a threshold above which
>> the balancer start moving things around.
>>
>> You should see them moving around if you do 'ping -f' or a big 'dd' from
>> the disk.
>
> If kirqd is moving NIC interrupts, it's broken.
>
> (and another reason why irqbalanced is preferable)
Why is it broken to move NIC interrupts? Obviously you don't want to
rotate them around a lot, but in the interests of fairness to other
processes, it seems reasonable to migrate them occasionally (IIRC, kirqd
rate limits to once a second or something).
M.
> I've noticed this problem for a while, but only now decided to ask.
> Interrupt balancing doesn't do anything on my system.
>
> CPU0 CPU1
> 0: 31931808 0 IO-APIC-edge timer
> 1: 76595 0 IO-APIC-edge i8042
> 8: 1 0 IO-APIC-edge rtc
> 9: 1 0 IO-APIC-level acpi
> 14: 122 1 IO-APIC-edge ide0
> 16: 4074456 0 IO-APIC-level uhci_hcd, uhci_hcd, radeon@PCI:1:0:0
> 17: 4295132 0 IO-APIC-level Intel ICH5
> 18: 2070933 0 IO-APIC-level libata, uhci_hcd, eth0
> 19: 887311 0 IO-APIC-level uhci_hcd
> 22: 572530 0 IO-APIC-level ath0
> NMI: 31931749 31931636 (I've since disabled the nmi_watchdog)
> LOC: 31931252 31931251
> ERR: 0
> MIS: 0
>
> I enabled the debugging and found that it doesn't think it's worth the
> effort. Is that correct? Not a complaint, just curious!
I think it's nothing to do with HT, just the rate you're sending intterrupts
at isn't high enough to bother rotating.
M.
Martin J. Bligh wrote:
> --Jeff Garzik <[email protected]> wrote (on Saturday, February 19, 2005 11:30:53 -0500):
>
>
>>Pallipadi, Venkatesh wrote:
>>
>>>You are right. Kernel balancer doesn't move around the irqs, unless it
>>>has too many interrupts. The logic is moving around interrupts all the
>>>time will not be good on caches. So, there is a threshold above which
>>>the balancer start moving things around.
>>>
>>>You should see them moving around if you do 'ping -f' or a big 'dd' from
>>>the disk.
>>
>>If kirqd is moving NIC interrupts, it's broken.
>>
>>(and another reason why irqbalanced is preferable)
>
>
> Why is it broken to move NIC interrupts? Obviously you don't want to
> rotate them around a lot, but in the interests of fairness to other
> processes, it seems reasonable to migrate them occasionally (IIRC, kirqd
> rate limits to once a second or something).
This has been explained to you before, search your email archives...
The main problem that has been seen in the field SMP packet ordering,
but a secondary problem observed is cache misses. Just NAPI mitigates
this somewhat (no pun intended).
Overall, kirqd should be avoided except in special situations. It
doesn't know about such policy things as network-specific or
storage-specific irq balancing (and shouldn't).
Jeff