Hi everyone,
I faced the same problem as this URL:
http://stackoverflow.com/questions/13641440/unable-to-distrubute-the-interrupts-over-multiple-cores
The question have been closed with no answer. :(
My machine have 2x Xeon E5504 processors and running vanilla kernel
3.6.11 in CentOS 6.0.
Could anyone give me some tips?
PS:
[root@localhost ~]# cat /proc/cmdline
ro root=/dev/mapper/vg_localhost-lv_root rd_NO_LUKS LANG=en_US.UTF-8
rd_NO_MD quiet rd_LVM_LV=vg_localhost/lv_swap
SYSFONT=latarcyrheb-sun16 rhgb rd_LVM_LV=vg_localhost/lv_root
KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM apic=debug pci=nomsi
[root@localhost ~]# uname -a
Linux localhost.lenkydomain 3.6.11 #2 SMP Fri Mar 22 21:13:41 CST 2013
x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# echo 6 > /proc/irq/25/smp_affinity
[root@localhost ~]# cat /proc/irq/25/smp_affinity
000006
[root@localhost ~]# cat /proc/interrupts | grep eth9
25: 5824 13914 159 128 210
0 0 0 IR-IO-APIC-fasteoi eth9
[root@localhost ~]# cat /proc/interrupts | grep eth9
25: 5824 13927 159 128 210
0 0 0 IR-IO-APIC-fasteoi eth9
[root@localhost ~]# cat /proc/interrupts | grep eth9
25: 5824 13939 159 128 210
0 0 0 IR-IO-APIC-fasteoi eth9
[root@localhost ~]# cat /proc/interrupts | grep eth9
25: 5824 13973 159 128 210
0 0 0 IR-IO-APIC-fasteoi eth9
--
Regards,
Lenky
Hi Gao,
On 03/25/2013 10:33 AM, Lenky Gao wrote:
> [root@localhost ~]# echo 6 > /proc/irq/25/smp_affinity
> [root@localhost ~]# cat /proc/irq/25/smp_affinity
> 000006
Seems you bind the nic irq to second and third cpu for the bit mask
you set is 110, so now eth9's irq is working on the 3rd cpu.
Have you ever tried irqbalance service? It may help to balance irq if the
irq workload of 3rd cpu is too heavy.
thanks,
linfeng
> [root@localhost ~]# cat /proc/interrupts | grep eth9
> 25: 5824 13914 159 128 210
> 0 0 0 IR-IO-APIC-fasteoi eth9
> [root@localhost ~]# cat /proc/interrupts | grep eth9
> 25: 5824 13927 159 128 210
> 0 0 0 IR-IO-APIC-fasteoi eth9
> [root@localhost ~]# cat /proc/interrupts | grep eth9
> 25: 5824 13939 159 128 210
> 0 0 0 IR-IO-APIC-fasteoi eth9
> [root@localhost ~]# cat /proc/interrupts | grep eth9
> 25: 5824 13973 159 128 210
> 0 0 0 IR-IO-APIC-fasteoi eth9
> Seems you bind the nic irq to second and third cpu for the bit mask
> you set is 110, so now eth9's irq is working on the 3rd cpu.
> Have you ever tried irqbalance service? It may help to balance irq if the
> irq workload of 3rd cpu is too heavy.
I need the interrupts distribute over multiple cores, such as the
second and third, so i set the mask parameter to 0x3. But only one
core is handling the interrupts. As you can see, just the second cpu's
irq number has increased.
The irqbalance service has been stopped.
--
Regards,
Lenky
Hi,
On 03/25/2013 11:18 AM, Lenky Gao wrote:
> The irqbalance service has been stopped.
So try start irqbalance to see what happen?
It should help to give what you want ;-)
thanks,
linfeng
> On 03/25/2013 11:18 AM, Lenky Gao wrote:
>> The irqbalance service has been stopped.
> So try start irqbalance to see what happen?
> It should help to give what you want ;-)
Using the irqbalance service to dynamically change the IRQ-bound? It's
seems a software solution. In my old machine, there is no this
problem. I need some information to explain why. Maybe is the
apic_physflat mode only support distribute the interrupts to one
processor, but i am not exactly sure if this is right? Or my E5504
Machine's hardware issue?
--
Regards,
Lenky
Hi,
On 03/25/2013 11:44 AM, Lenky Gao wrote:
>> On 03/25/2013 11:18 AM, Lenky Gao wrote:
>>> The irqbalance service has been stopped.
>> So try start irqbalance to see what happen?
>> It should help to give what you want ;-)
>
> Using the irqbalance service to dynamically change the IRQ-bound? It's
> seems a software solution. In my old machine, there is no this
> problem. I need some information to explain why. Maybe is the
Do you mean on your old machine the irq will be distributed automatically
among the cpus set by smp_affinity?
Sorry, I forgot one thing that the irqbalance service will break the mask
set by smp_affinity. So I don't know if now we support the feature
that balance among the cpus set by smp_affinity.
In my old conception when we set smp_affinity the irqbalance service
should be stopped, so you are right before, and the kernel will choose
one cpu in your mask for the IRQ and bounded from now on if there is no
irqbalance service.
thanks,
linfeng
> Do you mean on your old machine the irq will be distributed automatically
> among the cpus set by smp_affinity?
>
Yes. My another machine's interrupts are as follows:
[root@localhost ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Pentium(R) Dual-Core CPU E6700 @ 3.20GHz
...
[root@localhost ~]# echo 3 > /proc/irq/18/smp_affinity
[root@localhost ~]# cat /proc/irq/18/smp_affinity
3
[root@localhost ~]# cat /proc/interrupts | grep eth2
18: 455 458 IO-APIC-fasteoi eth2
[root@localhost ~]# cat /proc/interrupts | grep eth2
18: 463 467 IO-APIC-fasteoi eth2
[root@localhost ~]# cat /proc/interrupts | grep eth2
18: 471 476 IO-APIC-fasteoi eth2
--
Regards,
Lenky
On 03/25/2013 02:46 PM, Lenky Gao wrote:
>> Do you mean on your old machine the irq will be distributed automatically
>> among the cpus set by smp_affinity?
>>
>
> Yes. My another machine's interrupts are as follows:
And without irqbalance service? It sounds weird to me..
thanks,
linfeng
>
> [root@localhost ~]# cat /proc/cpuinfo
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 23
> model name : Pentium(R) Dual-Core CPU E6700 @ 3.20GHz
> ...
> [root@localhost ~]# echo 3 > /proc/irq/18/smp_affinity
> [root@localhost ~]# cat /proc/irq/18/smp_affinity
> 3
> [root@localhost ~]# cat /proc/interrupts | grep eth2
> 18: 455 458 IO-APIC-fasteoi eth2
> [root@localhost ~]# cat /proc/interrupts | grep eth2
> 18: 463 467 IO-APIC-fasteoi eth2
> [root@localhost ~]# cat /proc/interrupts | grep eth2
> 18: 471 476 IO-APIC-fasteoi eth2
>
>
> And without irqbalance service? It sounds weird to me..
The irqbalance service has been stopped.
--
Regards,
Lenky
I have found a comment in function physflat_cpu_mask_to_apicid to explain why.
static unsigned int physflat_cpu_mask_to_apicid(const struct cpumask *cpumask)
{
int cpu;
/*
* We're using fixed IRQ delivery, can only return one phys APIC ID.
* May as well be the first.
*/
...
This is mean i can not distribute the interrupts over multiple cores
when the machine have more than 8 cores?
--
Regards,
Lenky
Hi,
On 03/25/2013 05:00 PM, Lenky Gao wrote:
> I have found a comment in function physflat_cpu_mask_to_apicid to explain why.
>
> static unsigned int physflat_cpu_mask_to_apicid(const struct cpumask *cpumask)
> {
> int cpu;
>
> /*
> * We're using fixed IRQ delivery, can only return one phys APIC ID.
> * May as well be the first.
> */
> ...
>
> This is mean i can not distribute the interrupts over multiple cores
> when the machine have more than 8 cores?
>
>
Sorry I can't find physflat_cpu_mask_to_apicid() in latest Linux tree, seems it has
been removed out.. But just consider what you said that it's limited by "8 cores"
IMHO it's impossible or it is a bug.
thanks,
linfeng
> Sorry I can't find physflat_cpu_mask_to_apicid() in latest Linux tree, seems it has
> been removed out.. But just consider what you said that it's limited by "8 cores"
> IMHO it's impossible or it is a bug.
Here:
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/arch/x86/kernel/apic/apic_flat_64.c?id=v2.6.30.8#n281
--
Regards,
Lenky