Hi @all
we have detected some problems on our live systems and so i have build a test
setup in our lab as follow:
3 Core 2 duo servers, each with 2 CPUs, with GE interfaces. 2 of them are
only for generating network traffic. the 3rd server is the one i want to
test. it is connected over two GE links to the other servers. the testserver
is configured as an ip router. running kernel 2.6.20.
now if i let traffic flow over the box, about 600Mbit/s and about 120k
packets/s all seems to be ok. the load is balanced over all cpus. if now
insert some iptables rules, about 500, the softirq load increases, but all
seems to be ok. now i insert some rules more, and suddenly 1 CPU is 100%
loaded and the other ones are 99% idle. the load toggles now between the cpus
in intervals.
i have made some graphs http://www.poweraudio.de/linux/cpu.html
this seems to me to be not normal.
any suggestions?
regards
Marc
On Tue, 2007-02-06 at 18:32 +0100, Marc Donner wrote:
> Hi @all
>
> we have detected some problems on our live systems and so i have build a test
> setup in our lab as follow:
>
> 3 Core 2 duo servers, each with 2 CPUs, with GE interfaces. 2 of them are
> only for generating network traffic. the 3rd server is the one i want to
> test. it is connected over two GE links to the other servers. the testserver
> is configured as an ip router. running kernel 2.6.20.
>
> now if i let traffic flow over the box, about 600Mbit/s and about 120k
> packets/s all seems to be ok. the load is balanced over all cpus. if now
> insert some iptables rules, about 500, the softirq load increases, but all
> seems to be ok. now i insert some rules more, and suddenly 1 CPU is 100%
> loaded and the other ones are 99% idle. the load toggles now between the cpus
> in intervals.
I wonder if you are using irqbalance.. if not you probably want to...
(this should at least split it over 2 cpus)
see http://www.irqbalance.org to get irqbalance
On Tuesday 06 February 2007 19:09, you wrote:
> On Tue, 2007-02-06 at 18:32 +0100, Marc Donner wrote:
> > Hi @all
> >
> > we have detected some problems on our live systems and so i have build a
> > test setup in our lab as follow:
> >
> > 3 Core 2 duo servers, each with 2 CPUs, with GE interfaces. 2 of them
> > are only for generating network traffic. the 3rd server is the one i want
> > to test. it is connected over two GE links to the other servers. the
> > testserver is configured as an ip router. running kernel 2.6.20.
> >
> > now if i let traffic flow over the box, about 600Mbit/s and about 120k
> > packets/s all seems to be ok. the load is balanced over all cpus. if now
> > insert some iptables rules, about 500, the softirq load increases, but
> > all seems to be ok. now i insert some rules more, and suddenly 1 CPU is
> > 100% loaded and the other ones are 99% idle. the load toggles now between
> > the cpus in intervals.
>
> I wonder if you are using irqbalance.. if not you probably want to...
> (this should at least split it over 2 cpus)
>
> see http://www.irqbalance.org to get irqbalance
I now have tried irqloadbalance, but the same problem.
Marc Donner wrote:
>>
>> see http://www.irqbalance.org to get irqbalance
>
> I now have tried irqloadbalance, but the same problem.
can you send me the output of
cat /proc/interrupts
(taken when you are or have been loading the network)
maybe there's something fishy going on
2.6.20-rc3
[root@squid ~]# rpm -q irqbalance
irqbalance-0.55-2.fc6
[root@squid ~]# uptime
11:51:50 up 6 days, 30 min, 3 users, load average: 5.31, 5.08, 4.02
[root@squid ~]# service irqbalance status
irqbalance (pid 2310) is running...
[root@squid ~]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 520209517 0 0 0 IO-APIC-edge timer
1: 12 0 0 0 IO-APIC-edge i8042
8: 1 0 0 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 103 0 0 0 IO-APIC-edge i8042
14: 0 0 0 0 IO-APIC-edge libata
15: 0 0 0 0 IO-APIC-edge libata
20: 138736 188194096 0 6797630 IO-APIC-fasteoi libata
22: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2, uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, uhci_hcd:usb3, ehci_hcd:usb5
2296: 1367 0 0 849270653 PCI-MSI-edge eth1
2297: 1022 835083968 0 0 PCI-MSI-edge eth0
NMI: 47756 146249 47617 146186
LOC: 516828752 517331906 516828611 517331771
ERR: 0
2.6.20-rc3-mm1
[root@squid kernel]# uptime
12:17:54 up 1 day, 21:58, 2 users, load average: 9.47, 9.79, 10.28
[root@squid kernel]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 60031592 61350247 22273772 21780215 IO-APIC-edge timer
1: 0 6 1 1 IO-APIC-edge i8042
8: 0 0 1 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 148 283 104 136 IO-APIC-edge i8042
14: 0 0 0 0 IO-APIC-edge libata
15: 0 0 0 0 IO-APIC-edge libata
20: 10482795 1477821 93306 641628 IO-APIC-fasteoi libata
22: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2, uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, uhci_hcd:usb3, ehci_hcd:usb5
2296: 427 426 436 134563009 PCI-MSI-edge eth1
2297: 252 252 135926471 257 PCI-MSI-edge eth0
NMI: 0 0 0 0
LOC: 164661140 165163503 164660992 165163305
ERR: 0
Pablo Sebastian Greco wrote:
> 2296: 427 426 436 134563009 PCI-MSI-edge eth1
> 2297: 252 252 135926471 257 PCI-MSI-edge eth0
this suggests that cores would be busy rather than only one....
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Intel(R) Xeon(TM) CPU 2.66GHz
stepping : 4
cpu MHz : 2656.000
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 6
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est cid cx16 xtpr lahf_lm
bogomips : 5324.82
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Intel(R) Xeon(TM) CPU 2.66GHz
stepping : 4
cpu MHz : 2656.000
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 6
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est cid cx16 xtpr lahf_lm
bogomips : 5320.06
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Intel(R) Xeon(TM) CPU 2.66GHz
stepping : 4
cpu MHz : 2656.000
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 6
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est cid cx16 xtpr lahf_lm
bogomips : 5320.20
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 6
model name : Intel(R) Xeon(TM) CPU 2.66GHz
stepping : 4
cpu MHz : 2656.000
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 6
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est cid cx16 xtpr lahf_lm
bogomips : 5320.16
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 48 bits virtual
power management:
> can you send me the output of
>
> cat /proc/interrupts
here it is:
irqblance is running.
network loaded with 600Mbit/s for about 5minutes.
CPU0 CPU1 CPU2 CPU3
0: 37713 41667 41673 49914 IO-APIC-edge timer
1: 0 0 2 0 IO-APIC-edge i8042
8: 0 0 1 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 2 0 2 0 IO-APIC-edge i8042
14: 11 9 9 8 IO-APIC-edge ide0
20: 0 0 0 0 IO-APIC-fasteoi
uhci_hcd:usb3
21: 62 52 37 46 IO-APIC-fasteoi
ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb4
78: 665 581 344 351 IO-APIC-fasteoi megasas
501: 215717 209388 209430 202514 PCI-MSI-edge eth10
502: 927 1019 1053 888 PCI-MSI-edge eth11
NMI: 451 39 42 46
LOC: 170899 170864 170846 170788
ERR: 0
top output:
top - 01:45:32 up 16 min, 2 users, load average: 1.04, 0.92, 0.50
Tasks: 81 total, 3 running, 78 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 100.0% si
Cpu1 : 0.0% us, 0.0% sy, 0.0% ni, 99.0% id, 1.0% wa, 0.0% hi, 0.0% si
Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.3% si
Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.3% hi, 0.0% si
regards
Marc
Marc Donner wrote:
> 501: 215717 209388 209430 202514 PCI-MSI-edge eth10
> 502: 927 1019 1053 888 PCI-MSI-edge eth11
this is odd, this is not an irq distribution that irqbalance should
give you
1
> NMI: 451 39 42 46
> LOC: 170899 170864 170846 170788
> ERR: 0
>
> top output:
>
> top - 01:45:32 up 16 min, 2 users, load average: 1.04, 0.92, 0.50
> Tasks: 81 total, 3 running, 78 sleeping, 0 stopped, 0 zombie
> Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 100.0% si
>
and this doesn't match the irq output...
sounds as if something has a real bug; can you send an lsmod ? maybe
some driver keeps doing si's
On Wednesday 07 February 2007 06:59, you wrote:
> Marc Donner wrote:
> > 501: 215717 209388 209430 202514 PCI-MSI-edge
> > eth10 502: 927 1019 1053 888 PCI-MSI-edge
> > eth11
>
> this is odd, this is not an irq distribution that irqbalance should
> give you
i think this is ok, because only eth10 is receiving packets. traffic is
flowing from eth10 to eth11
> and this doesn't match the irq output...
> sounds as if something has a real bug; can you send an lsmod ? maybe
> some driver keeps doing si's
lsmod
Module Size Used by
thermal 16780 0
fan 6280 0
button 9696 0
processor 29576 1 thermal
ac 6664 0
battery 11720 0
drivers are build directly in the kernel. i have attached the config file.
i can also give access to the test setup, if you want.
i have also tested kernel 2.6.18.3 and 2.6.19.2 on other hardware. same
effect.
regards
marc
On Wed, 7 Feb 2007, Arjan van de Ven wrote:
>
> Marc Donner wrote:
>> 501: 215717 209388 209430 202514 PCI-MSI-edge eth10
>> 502: 927 1019 1053 888 PCI-MSI-edge eth11
>
> this is odd, this is not an irq distribution that irqbalance should give you
> 1
>> NMI: 451 39 42 46
>> LOC: 170899 170864 170846 170788
>> ERR: 0
>>
>> top output:
>>
>> top - 01:45:32 up 16 min, 2 users, load average: 1.04, 0.92, 0.50
>> Tasks: 81 total, 3 running, 78 sleeping, 0 stopped, 0 zombie
>> Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 100.0%
>> si
>>
> and this doesn't match the irq output...
> sounds as if something has a real bug; can you send an lsmod ? maybe some
> driver keeps doing si's
since this only happens when he adds more iptables rules, is it possible that
there sis some locking, or otherdata structure access that's serializing things
under load?
Marc, since you don't use modules, send your .config.
David Lang
On Thursday 08 February 2007 09:42, you wrote:
> On Wed, 7 Feb 2007, Arjan van de Ven wrote:
> > Marc Donner wrote:
> >> 501: 215717 209388 209430 202514 PCI-MSI-edge
> >> eth10 502: 927 1019 1053 888 PCI-MSI-edge
> >> eth11
> >
> > this is odd, this is not an irq distribution that irqbalance should give
> > you 1
> >
> >> NMI: 451 39 42 46
> >> LOC: 170899 170864 170846 170788
> >> ERR: 0
> >>
> >> top output:
> >>
> >> top - 01:45:32 up 16 min, 2 users, load average: 1.04, 0.92, 0.50
> >> Tasks: 81 total, 3 running, 78 sleeping, 0 stopped, 0 zombie
> >> Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi,
> >> 100.0% si
> >
> > and this doesn't match the irq output...
> > sounds as if something has a real bug; can you send an lsmod ? maybe some
> > driver keeps doing si's
>
> since this only happens when he adds more iptables rules, is it possible
> that there sis some locking, or otherdata structure access that's
> serializing things under load?
>
> Marc, since you don't use modules, send your .config.
>
> David Lang
i've inserted some more iptables rules, and got 'softlockup detected'
messages.
BUG: soft lockup detected on CPU#1!
Call Trace:
<IRQ> [<ffffffff8023fbdf>] softlockup_tick+0xda/0xec
[<ffffffff80229ba5>] update_process_times+0x42/0x68
[<ffffffff8021421e>] smp_local_timer_interrupt+0x32/0x55
[<ffffffff804f162a>] ip_rcv+0x0/0x48e
[<ffffffff8021484b>] smp_apic_timer_interrupt+0x4f/0x67
[<ffffffff8020a206>] apic_timer_interrupt+0x66/0x70
[<ffffffff804f162a>] ip_rcv+0x0/0x48e
[<ffffffff8051bf4e>] ipt_do_table+0x2cd/0x315
[<ffffffff804eb445>] nf_iterate+0x3f/0x7b
[<ffffffff804f2aa0>] ip_forward_finish+0x0/0x3e
[<ffffffff804eb4e0>] nf_hook_slow+0x5f/0xca
[<ffffffff804f2aa0>] ip_forward_finish+0x0/0x3e
[<ffffffff804f2c4c>] ip_forward+0x16e/0x212
[<ffffffff804f1a71>] ip_rcv+0x447/0x48e
[<ffffffff803de54c>] e1000_clean_rx_irq+0x41e/0x4ea
[<ffffffff803e1dea>] e1000_clean+0x2f7/0x4b0
[<ffffffff8021bf72>] task_rq_lock+0x3d/0x6f
[<ffffffff804dcb1a>] net_rx_action+0x78/0x14a
[<ffffffff802268b0>] __do_softirq+0x56/0xd3
[<ffffffff80226699>] ksoftirqd+0x0/0x8f
[<ffffffff8020a75c>] call_softirq+0x1c/0x28
[<ffffffff8020c20a>] do_softirq+0x2c/0x7d
[<ffffffff80214850>] smp_apic_timer_interrupt+0x54/0x67
[<ffffffff8020a206>] apic_timer_interrupt+0x66/0x70
<EOI> [<ffffffff8020c259>] do_softirq+0x7b/0x7d
[<ffffffff802266e8>] ksoftirqd+0x4f/0x8f
[<ffffffff8023312a>] kthread+0xcb/0xf5
[<ffffffff8020a3e8>] child_rip+0xa/0x12
[<ffffffff8023305f>] kthread+0x0/0xf5
[<ffffffff8020a3de>] child_rip+0x0/0x12
Marc