Hi,
I happened to look at a system which was exhibiting poor ping
performance with e1000 driver (in 2.6.25) and had some questions regarding that.
Ping test was done between the system and a laptop, which were connected
using a straight ethernet cable. Ping reported round trip times running
into seconds (!) and also packet loss.
Upon some investigation, I found that the interrupt count field in
/proc/interrupts (associated with eth1) is not incrementing as fast as
it should. Moreover eth1 interrupt line is shared with the hard disk
interrupt (ata_piix) as below:
# cat /proc/interrupts
..
10: 2296 XT-PIC-XT ata_piix, eth0, eth1
..
IRQ10 is thus being shared by both the hard disk and eth0/eth1.
Here's the strange observation I made:
When I initiate some disk activity (ex: dd if=/dev/zero of=/tmp/file), ping
performance suddently shot up (round trip time in double digits ms, 0% packet
loss)! I presume this is because that e1000 intr handler is called
whenever there was a interrupt from hard disk on IRQ10, which polled
NIC and processed packets immediately.
As soon as I kill the background disk-write intensive job, ping
performance again dropped.
This meant that e1000 NIC is having trouble interrupting the OS.
Before I could jump up and say this is a hardware issue, I was told
that Windows works just fine on the server (and as well as 2.4 kernel,
which I couldnt verify) :(
Some more observations:
1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0,
TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of
them helped.
2. When ping performance was poor, readprofile showed that system
is mostly idle. This confirms that OS is not getting very
frequenty interrupts from eth1 and hence idling.
3. When ping performance was poor, ethtool -S eth1 showed that
rx_bytes was incrementing at a good pace, showing that the
NIC was receiving ping responses back, but not handing them over
to OS for further processing
4. e1000 chipset is 82546GB
5. e1000e driver didnt work at all (it doesnt recognize the cards).
Any advice on how to fix this problem?
--
Regards,
vatsa
Srivatsa Vaddagiri wrote:
> Hi,
> I happened to look at a system which was exhibiting poor ping
> performance with e1000 driver (in 2.6.25) and had some questions
> regarding that.
> ...
> Upon some investigation, I found that the interrupt count field in
> /proc/interrupts (associated with eth1) is not incrementing as fast as
> it should. Moreover eth1 interrupt line is shared with the hard disk
> interrupt (ata_piix) as below:
>
> # cat /proc/interrupts
> 10: 2296 XT-PIC-XT ata_piix, eth0, eth1
whats wrong with your system that you can't use acpi and/or apic? It
would probably orthoginally solve the problem by unsharing your
interrupt.
> IRQ10 is thus being shared by both the hard disk and eth0/eth1.
bad for performance but should really work okay.
> Here's the strange observation I made:
>
> When I initiate some disk activity (ex: dd if=/dev/zero
> ...
> This meant that e1000 NIC is having trouble interrupting the OS.
you're correct here, there appears to be some problem on your system
either with interrupt delivery or with the driver masking off interrupts
and leaving them disabled.
> Before I could jump up and say this is a hardware issue, I was told
> that Windows works just fine on the server (and as well as 2.4 kernel,
> which I couldnt verify) :(
well it might be a bios issue, but would likely be solved by using boot
option acpi=force and/or lapci (see kernel-parameters.txt
> Some more observations:
>
> 1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0,
> TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of
> them helped.
these won't help you get an interrupt delivered or re-enabled
> 2. When ping performance was poor, readprofile showed that system
> is mostly idle. This confirms that OS is not getting very
> frequenty interrupts from eth1 and hence idling.
expected, thanks for checking.
> 3. When ping performance was poor, ethtool -S eth1 showed that
> rx_bytes was incrementing at a good pace, showing that the
> NIC was receiving ping responses back, but not handing them over
> to OS for further processing
also expected for an interrupt problem.
> 4. e1000 chipset is 82546GB
>
> 5. e1000e driver didnt work at all (it doesnt recognize the cards).
expected, this is a PCI-X adapter.
> Any advice on how to fix this problem?
try the boot options first, then if that doesn't work for you, download
ethregs from e1000.sourceforge.net download area and compile/run it and
send me the output in private email.
if you have a spare moment, you can try the e1000-8.X driver from
sourceforge and let me know if it works okay, that would imply we just
need to patch the in-kernel driver to fix an already known issue.
Jesse
On Wed, Jun 18, 2008 at 12:18:30PM -0700, Brandeburg, Jesse wrote:
> > # cat /proc/interrupts
> > 10: 2296 XT-PIC-XT ata_piix, eth0, eth1
>
> whats wrong with your system that you can't use acpi and/or apic? It
> would probably orthoginally solve the problem by unsharing your
> interrupt.
Nothing wrong with acpi/apic. It just wasnt helping solve the problem.
I booted with noapic to check if it helps resolve, but found
it didnt.
> > IRQ10 is thus being shared by both the hard disk and eth0/eth1.
>
> bad for performance but should really work okay.
Is there a way we can force unsharing of the IRQ (between harddisk and
eth1) in software?
> > Here's the strange observation I made:
> >
> > When I initiate some disk activity (ex: dd if=/dev/zero
> > ...
>
> > This meant that e1000 NIC is having trouble interrupting the OS.
>
> you're correct here, there appears to be some problem on your system
> either with interrupt delivery
Note that other interrupts (timer, hard disk) are fine. Even eth1 interrupt
"works", just that it comes lazily (once in few seconds - when I am pumping
potentially hundreds of ping packets to it every second).
# watch "grep 10: /proc/interrupts"
shows the interrupt count associated with eth1 increment at the rate of
1-2 every 2-3 seconds (<1 interrupt per second).
Is there some interrupt-related statistics that we can obtain from e1000
card which shows how many times e1000 NIC tried "interrupting" the system?
> or with the driver masking off interrupts and leaving them disabled.
Hmm ..shouldnt that affect ata disk functionality too? hard disk I/O
works fine when ping performance is bad.
> > Before I could jump up and say this is a hardware issue, I was told
> > that Windows works just fine on the server (and as well as 2.4 kernel,
> > which I couldnt verify) :(
>
> well it might be a bios issue,
Again, if it was a bios issue, the question i am faced with is "how is
Windows working fine on it?".
> but would likely be solved by using boot
> option acpi=force and/or lapci (see kernel-parameters.txt
I had tried these other boot options in vain: noapic, acpi=off, acpi=noirq, pci=noacpi
If you recommend any other boot option, we'd be glad to try it out.
> > Some more observations:
> >
> > 1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0,
> > TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of
> > them helped.
>
> these won't help you get an interrupt delivered or re-enabled
ok.
> > 2. When ping performance was poor, readprofile showed that system
> > is mostly idle. This confirms that OS is not getting very
> > frequenty interrupts from eth1 and hence idling.
>
> expected, thanks for checking.
>
> > 3. When ping performance was poor, ethtool -S eth1 showed that
> > rx_bytes was incrementing at a good pace, showing that the
> > NIC was receiving ping responses back, but not handing them over
> > to OS for further processing
>
> also expected for an interrupt problem.
>
> > 4. e1000 chipset is 82546GB
> >
> > 5. e1000e driver didnt work at all (it doesnt recognize the cards).
>
> expected, this is a PCI-X adapter.
>
>
> > Any advice on how to fix this problem?
>
> try the boot options first, then if that doesn't work for you, download
> ethregs from e1000.sourceforge.net download area and compile/run it and
> send me the output in private email.
sure ..i will send you them after sometime.
> if you have a spare moment, you can try the e1000-8.X driver from
> sourceforge and let me know if it works okay, that would imply we just
> need to patch the in-kernel driver to fix an already known issue.
ok, we will test that out as well.
Thanks for all your inputs!
--
Regards,
vatsa
Srivatsa Vaddagiri wrote:
> Hi,
> I happened to look at a system which was exhibiting poor ping
> performance with e1000 driver (in 2.6.25) and had some questions regarding that.
>
> Ping test was done between the system and a laptop, which were connected
> using a straight ethernet cable. Ping reported round trip times running
> into seconds (!) and also packet loss.
>
> Upon some investigation, I found that the interrupt count field in
> /proc/interrupts (associated with eth1) is not incrementing as fast as
> it should. Moreover eth1 interrupt line is shared with the hard disk
> interrupt (ata_piix) as below:
>
> # cat /proc/interrupts
>
> .
>
> 10: 2296 XT-PIC-XT ata_piix, eth0, eth1
>
> .
>
> IRQ10 is thus being shared by both the hard disk and eth0/eth1.
>
> Here's the strange observation I made:
>
> When I initiate some disk activity (ex: dd if=/dev/zero of=/tmp/file), ping
> performance suddently shot up (round trip time in double digits ms, 0% packet
> loss)! I presume this is because that e1000 intr handler is called
> whenever there was a interrupt from hard disk on IRQ10, which polled
> NIC and processed packets immediately.
>
> As soon as I kill the background disk-write intensive job, ping
> performance again dropped.
>
> This meant that e1000 NIC is having trouble interrupting the OS.
>
> Before I could jump up and say this is a hardware issue, I was told
> that Windows works just fine on the server (and as well as 2.4 kernel,
> which I couldnt verify) :(
>
>
> Some more observations:
>
> 1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0,
> TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of
> them helped.
>
> 2. When ping performance was poor, readprofile showed that system
> is mostly idle. This confirms that OS is not getting very
> frequenty interrupts from eth1 and hence idling.
>
> 3. When ping performance was poor, ethtool -S eth1 showed that
> rx_bytes was incrementing at a good pace, showing that the
> NIC was receiving ping responses back, but not handing them over
> to OS for further processing
>
> 4. e1000 chipset is 82546GB
>
> 5. e1000e driver didnt work at all (it doesnt recognize the cards).
>
>
> Any advice on how to fix this problem?
Can you post your dmesg output from bootup with no special options
(noacpi, etc.) enabled?
On 6/19/08, Robert Hancock <[email protected]> wrote:
> Srivatsa Vaddagiri wrote:
>
> > Hi,
> > I happened to look at a system which was exhibiting poor ping
> > performance with e1000 driver (in 2.6.25) and had some questions regarding
> that.
> >
> > Ping test was done between the system and a laptop, which were connected
> > using a straight ethernet cable. Ping reported round trip times running
> > into seconds (!) and also packet loss.
> >
> > Upon some investigation, I found that the interrupt count field in
> > /proc/interrupts (associated with eth1) is not incrementing as fast as
> > it should. Moreover eth1 interrupt line is shared with the hard disk
> > interrupt (ata_piix) as below:
Just FYI, it seems that more people have the same (?) problem:
http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/0499.html
It might be worth joining forces to fight down the bug once and for
all if it's still not fixed after one and a half years? Sorry, I
really meant error instead of bug.
Vegard
--
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
-- E. W. Dijkstra, EWD1036
On Thu, Jun 19, 2008 at 11:25:12AM -0600, Robert Hancock wrote:
> Can you post your dmesg output from bootup with no special options (noacpi,
> etc.) enabled?
Here it is (from RHEL5 kernel, no special boot options):
Linux version 2.6.18-53.el5 ([email protected]) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Wed Oct 10 16:34:02 EDT 2007
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000001f7c0000 (usable)
BIOS-e820: 000000001f7c0000 - 000000001f7ce000 (ACPI data)
BIOS-e820: 000000001f7ce000 - 000000001f7ff8f0 (ACPI NVS)
BIOS-e820: 000000001f7ff8f0 - 000000001f800000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fec10000 - 00000000fec11000 (reserved)
BIOS-e820: 00000000fed20000 - 00000000fed90000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000feff0000 - 00000000ff000000 (reserved)
BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
0MB HIGHMEM available.
503MB LOWMEM available.
Memory for crash kernel (0x0 to 0x0) notwithin permissible range
disabling kdump
Using x86 segment limits to approximate NX protection
On node 0 totalpages: 128960
DMA zone: 4096 pages, LIFO batch:0
Normal zone: 124864 pages, LIFO batch:31
DMI present.
Using APIC driver default
ACPI: RSDP (v000 ACPIAM ) @ 0x000f7ea0
ACPI: RSDT (v001 A M I OEMRSDT 0x01000725 MSFT 0x00000097) @ 0x1f7c0000
ACPI: FADT (v002 A M I OEMFACP 0x01000725 MSFT 0x00000097) @ 0x1f7c0200
ACPI: MADT (v001 A M I OEMAPIC 0x01000725 MSFT 0x00000097) @ 0x1f7c0390
ACPI: OEMB (v001 A M I AMI_OEM 0x01000725 MSFT 0x00000097) @ 0x1f7ce040
ACPI: SSDT (v001 AMI CPU1PM 0x00000001 INTL 0x02002026) @ 0x1f7c4170
ACPI: DSDT (v001 0ABEC 0ABEC004 0x00000004 INTL 0x02002026) @ 0x00000000
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 6:13 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x81] disabled)
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x02] address[0xfec10000] gsi_base[24])
IOAPIC[1]: apic_id 2, version 32, address 0xfec10000, GSI 24-47
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode: Flat. Using 2 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 20000000 (gap: 1f800000:df400000)
Detected 1400.147 MHz processor.
Built 1 zonelists. Total pages: 128960
Kernel command line: ro root=LABEL=/ rhgb quiet
mapped APIC to ffffd000 (fee00000)
mapped IOAPIC to ffffc000 (fec00000)
mapped IOAPIC to ffffb000 (fec10000)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c0743000 soft=c0723000
PID hash table entries: 2048 (order: 11, 8192 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 504468k/515840k available (2080k kernel code, 10836k reserved, 869k data, 220k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 2801.05 BogoMIPS (lpj=1400527)
Security Framework v1.0.0 initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512
CPU: After generic identify, caps: afe9fbff 00100000 00000000 00000000 00000180 00000000 00000000
CPU: After vendor identify, caps: afe9fbff 00100000 00000000 00000000 00000180 00000000 00000000
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU: After all inits, caps: afe9f3ff 00100000 00000000 00000040 00000180 00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
Checking 'hlt' instruction... OK.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 14k freed
ACPI: Core revision 20060707
CPU0: Intel(R) Pentium(R) M processor 1.40GHz stepping 08
Total of 1 processors activated (2801.05 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
Brought up 1 CPUs
sizeof(vma)=84 bytes
sizeof(page)=32 bytes
sizeof(inode)=340 bytes
sizeof(dentry)=136 bytes
sizeof(ext3inode)=492 bytes
sizeof(buffer_head)=52 bytes
sizeof(skbuff)=172 bytes
checking if image is initramfs... it is
Freeing initrd memory: 2350k freed
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
Boot video device is 0000:00:02.0
PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0480-04bf claimed by ICH4 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.2
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P4._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P6._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 *9 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 9 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 *10 11 12 14 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 16 devices
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
pnp: 00:06: ioport range 0x540-0x55f has been reserved
pnp: 00:0e: ioport range 0xa00-0xa7f has been reserved
PCI: Ignore bogus resource 6 [0:0] of 0000:00:02.0
PCI: Bridge: 0000:02:00.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1c.0
IO window: d000-dfff
MEM window: ff700000-ff7fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Setting latency timer of device 0000:00:1e.0 to 64
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 16384 bind 8192)
TCP reno registered
apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac)
apm: overridden by ACPI.
audit: initializing netlink socket (disabled)
audit(1213972188.734:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
SELinux: Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key FF48355AC6B12D27
- User ID: Red Hat, Inc. (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: CPU0 (power states: C1[C1] C2[C2])
ACPI: Processor [CPU1] (supports 8 throttling states)
ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present [20060707]
ACPI: Getting cpuindex for acpiid 0x2
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
agpgart: Detected an Intel 855 Chipset.
agpgart: Detected 8060K stolen memory.
agpgart: AGP aperture is 128M @ 0xf0000000
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:07: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Probing IDE interface ide0...
ide1: I/O resource 0x170-0x177 not free.
ide1: ports already in use, skipping probe
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
Using IPI No-Shortcut mode
Time: tsc clocksource has been installed.
ACPI: (supports S0 S1 S3 S4 S5)
Freeing unused kernel memory: 220k freed
Write protecting the kernel read-only data: 388k
Time: acpi_pm clocksource has been installed.
input: AT Translated Set 2 keyboard as /class/input/input0
logips2pp: Detected unknown logitech mouse model 89
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1d.0: irq 169, io base 0x0000e800
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 177
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.1: irq 177, io base 0x0000e880
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 185
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 3
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 32 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 185, io mem 0xffa7bc00
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 4 ports detected
SCSI subsystem initialized
libata version 2.21 loaded.
ata_piix 0000:00:1f.2: version 2.11
ata_piix 0000:00:1f.2: MAP [ IDE IDE P0 P1 ]
ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 193
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max UDMA/100 cmd 0x000101f0 ctl 0x000103f6 bmdma 0x0001fc00 irq 14
ata2: SATA max UDMA/133 cmd 0x00010170 ctl 0x00010376 bmdma 0x0001fc08 irq 15
input: ImExPS/2 Logitech Explorer Mouse as /class/input/input1
logips2pp: Detected unknown logitech mouse model 89
ata2.01: ATA-7: ST380215AS, 3.AAC, max UDMA/133
ata2.01: 156301488 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata2.01: configured for UDMA/133
Vendor: ATA Model: ST380215AS Rev: 3.AA
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3
sd 1:0:1:0: Attached scsi disk sda
Initializing USB Mass Storage driver...
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
input: ImExPS/2 Logitech Explorer Mouse as /class/input/input2
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
audit(1213972201.588:2): enforcing=1 old_enforcing=0 auid=4294967295
security: 3 users, 6 roles, 1658 types, 210 bools, 1 sens, 1024 cats
security: 61 classes, 57675 rules
SELinux: Completing initialization.
SELinux: Setting up existing superblocks.
SELinux: initialized (dev sda2, type ext3), uses xattr
SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses genfs_contexts
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev eventpollfs, type eventpollfs), uses task SIDs
SELinux: initialized (dev inotifyfs, type inotifyfs), uses genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev futexfs, type futexfs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev cpuset, type cpuset), not configured for labeling
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
audit(1213972202.305:3): policy loaded auid=4294967295
Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
Copyright (c) 1999-2006 Intel Corporation.
ACPI: PCI Interrupt 0000:02:02.0[B] -> GSI 18 (level, low) -> IRQ 193
e1000: 0000:02:02.0: e1000_probe: (PCI-X:66MHz:64-bit) 00:04:23:00:90:ab
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ACPI: PCI Interrupt 0000:02:02.1[B] -> GSI 18 (level, low) -> IRQ 193
e1000: 0000:02:02.1: e1000_probe: (PCI-X:66MHz:64-bit) 00:04:23:00:90:aa
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
i6300ESB timer: initialized (0xe0038800). heartbeat=30 sec (nowayout=0)
input: PC Speaker as /class/input/input3
Floppy drive(s): fd0 is 1.44M
intel_rng: FWH not detected
FDC 0 is a post-1991 82077
sd 1:0:1:0: Attached scsi generic sg0 type 0
lp: driver loaded but no devices found
SELinux: initialized (dev ramfs, type ramfs), uses genfs_contexts
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
ibm_acpi: ec object not found
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
device-mapper: ioctl: 4.11.0-ioctl (2006-09-14) initialised: [email protected]
device-mapper: multipath: version 1.0.5 loaded
EXT3 FS on sda2, internal journal
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
IA-32 Microcode Update Driver: v1.14a <[email protected]>
ip6_tables: (C) 2000-2006 Netfilter Core Team
ip_tables: (C) 2000-2006 Netfilter Core Team
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (4030 buckets, 32240 max) - 228 bytes per conntrack
ADDRCONF(NETDEV_UP): eth0: link is not ready
ADDRCONF(NETDEV_UP): eth1: link is not ready
SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
Bluetooth: Core ver 2.10
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP ver 2.8
Bluetooth: L2CAP socket layer initialized
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM TTY layer initialized
Bluetooth: RFCOMM ver 1.8
Bluetooth: HIDP (Human Interface Emulation) ver 1.1
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
[drm] Initialized drm 1.0.1 20051102
ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 169
[drm] Initialized i915 1.8.0 20060929 on minor 0
[drm] Initialized i915 1.8.0 20060929 on minor 1
--
Regards,
vatsa
Srivatsa Vaddagiri wrote:
> On Thu, Jun 19, 2008 at 11:25:12AM -0600, Robert Hancock wrote:
>> Can you post your dmesg output from bootup with no special options (noacpi,
>> etc.) enabled?
>
> Here it is (from RHEL5 kernel, no special boot options):
Looks reasonable to me.. and what does /proc/interrupts show with this
setup?
cc'ing netdev
Robert Hancock wrote:
> Srivatsa Vaddagiri wrote:
>> Hi,
>> I happened to look at a system which was exhibiting poor ping
>> performance with e1000 driver (in 2.6.25) and had some questions
>> regarding that.
>>
>> Ping test was done between the system and a laptop, which were connected
>> using a straight ethernet cable. Ping reported round trip times running
>> into seconds (!) and also packet loss.
>>
>> Upon some investigation, I found that the interrupt count field in
>> /proc/interrupts (associated with eth1) is not incrementing as fast as
>> it should. Moreover eth1 interrupt line is shared with the hard disk
>> interrupt (ata_piix) as below:
>>
>> # cat /proc/interrupts
>>
>> .
>>
>> 10: 2296 XT-PIC-XT ata_piix, eth0, eth1
>>
>> .
>>
>> IRQ10 is thus being shared by both the hard disk and eth0/eth1.
>>
>> Here's the strange observation I made:
>>
>> When I initiate some disk activity (ex: dd if=/dev/zero
>> of=/tmp/file), ping performance suddently shot up (round trip time in
>> double digits ms, 0% packet loss)! I presume this is because that
>> e1000 intr handler is called
>> whenever there was a interrupt from hard disk on IRQ10, which polled
>> NIC and processed packets immediately.
>>
>> As soon as I kill the background disk-write intensive job, ping
>> performance again dropped.
>>
>> This meant that e1000 NIC is having trouble interrupting the OS.
>>
>> Before I could jump up and say this is a hardware issue, I was told
>> that Windows works just fine on the server (and as well as 2.4 kernel,
>> which I couldnt verify) :(
>>
>>
>> Some more observations:
>>
>> 1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0,
>> TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of
>> them helped.
>>
>> 2. When ping performance was poor, readprofile showed that system
>> is mostly idle. This confirms that OS is not getting very
>> frequenty interrupts from eth1 and hence idling.
>>
>> 3. When ping performance was poor, ethtool -S eth1 showed that
>> rx_bytes was incrementing at a good pace, showing that the NIC
>> was receiving ping responses back, but not handing them over
>> to OS for further processing
>>
>> 4. e1000 chipset is 82546GB
>>
>> 5. e1000e driver didnt work at all (it doesnt recognize the cards).
>>
>>
>> Any advice on how to fix this problem?
>
> Can you post your dmesg output from bootup with no special options
> (noacpi, etc.) enabled?
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On Fri, Jun 20, 2008 at 08:40:47AM -0600, Robert Hancock wrote:
> Looks reasonable to me.. and what does /proc/interrupts show with this
> setup?
Sorry about the delay, but here's /proc/interrupts:
CPU0
0: 1342021 IO-APIC-edge timer
1: 416 IO-APIC-edge i8042
6: 6 IO-APIC-edge floppy
8: 1 IO-APIC-edge rtc
12: 13367 IO-APIC-edge i8042
169: 1 IO-APIC-level acpi
177: 8882 IO-APIC-level libata, eth1
185: 75633 IO-APIC-level uhci_hcd:usb1, i915@pci:0000:00:02.0
193: 0 IO-APIC-level uhci_hcd:usb2
NMI: 0
LOC: 1341970
ERR: 0
MIS: 0
--
Regards,
vatsa
Vatsa,
This is not legacy mode that causes long ping time.
---henry
--- On Wed, 6/25/08, Srivatsa Vaddagiri <[email protected]> wrote:
> From: Srivatsa Vaddagiri <[email protected]>
> Subject: Re: [E1000-devel] Strange problem with e1000 driver - ping packet loss
> To: "Robert Hancock" <[email protected]>
> Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
> Date: Wednesday, June 25, 2008, 9:32 AM
> On Fri, Jun 20, 2008 at 08:40:47AM -0600, Robert Hancock
> wrote:
> > Looks reasonable to me.. and what does
> /proc/interrupts show with this
> > setup?
>
> Sorry about the delay, but here's /proc/interrupts:
>
> CPU0
> 0: 1342021 IO-APIC-edge timer
> 1: 416 IO-APIC-edge i8042
> 6: 6 IO-APIC-edge floppy
> 8: 1 IO-APIC-edge rtc
> 12: 13367 IO-APIC-edge i8042
> 169: 1 IO-APIC-level acpi
> 177: 8882 IO-APIC-level libata, eth1
> 185: 75633 IO-APIC-level uhci_hcd:usb1,
> i915@pci:0000:00:02.0
> 193: 0 IO-APIC-level uhci_hcd:usb2
> NMI: 0
> LOC: 1341970
> ERR: 0
> MIS: 0
>
>
>
> --
> Regards,
> vatsa
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> E1000-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
On Wed, 25 Jun 2008, Len Brown wrote:
> > Linux version 2.6.18-53.el5
can you find out if the latest upstream kernel still has this problem?
> > ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 177
> > PCI: Setting latency timer of device 0000:00:1d.1 to 64
> > uhci_hcd 0000:00:1d.1: UHCI Host Controller
> > uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
> > uhci_hcd 0000:00:1d.1: irq 177, io base 0x0000e880
>
> > SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
> > audit(1213972202.305:3): policy loaded auid=4294967295
> > Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
> > Copyright (c) 1999-2006 Intel Corporation.
> > ACPI: PCI Interrupt 0000:02:02.0[B] -> GSI 18 (level, low) -> IRQ 193
> > e1000: 0000:02:02.0: e1000_probe: (PCI-X:66MHz:64-bit) 00:04:23:00:90:ab
> > e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
> > ACPI: PCI Interrupt 0000:02:02.1[B] -> GSI 18 (level, low) -> IRQ 193
> > e1000: 0000:02:02.1: e1000_probe: (PCI-X:66MHz:64-bit) 00:04:23:00:90:aa
> > e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
your /proc/interrutps showed just eth1, and it was on IRQ 177 --
was that from a different boot than this dmesg?
in any case, when you boot these on a new kernel
they should both show up on IRQ 18 (because they're on GSI 18).
Also, these IOAPIC interrupts are not programmable -- they
are hard-coded, so the issue is not in the ACPI PCI interrupt
link programming code.
-Len
On Wed, Jun 25, 2008 at 03:04:59PM -0400, Len Brown wrote:
> On Wed, 25 Jun 2008, Len Brown wrote:
>
> > > Linux version 2.6.18-53.el5
>
> can you find out if the latest upstream kernel still has this problem?
Len,
We'd tried 2.6.25 and found it doesnt help.
> > > ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 177
> > > PCI: Setting latency timer of device 0000:00:1d.1 to 64
> > > uhci_hcd 0000:00:1d.1: UHCI Host Controller
> > > uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
> > > uhci_hcd 0000:00:1d.1: irq 177, io base 0x0000e880
> >
> > > SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
> > > audit(1213972202.305:3): policy loaded auid=4294967295
> > > Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
> > > Copyright (c) 1999-2006 Intel Corporation.
> > > ACPI: PCI Interrupt 0000:02:02.0[B] -> GSI 18 (level, low) -> IRQ 193
> > > e1000: 0000:02:02.0: e1000_probe: (PCI-X:66MHz:64-bit) 00:04:23:00:90:ab
> > > e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
> > > ACPI: PCI Interrupt 0000:02:02.1[B] -> GSI 18 (level, low) -> IRQ 193
> > > e1000: 0000:02:02.1: e1000_probe: (PCI-X:66MHz:64-bit) 00:04:23:00:90:aa
> > > e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
>
> your /proc/interrutps showed just eth1, and it was on IRQ 177 --
> was that from a different boot than this dmesg?
Yes the dmesg and /proc/interrupts output I'd sent was from different
boot instances.
> in any case, when you boot these on a new kernel
> they should both show up on IRQ 18 (because they're on GSI 18).
hmm ok. we can verify.
> Also, these IOAPIC interrupts are not programmable -- they
> are hard-coded, so the issue is not in the ACPI PCI interrupt
> link programming code.
So is the issue likely with the e1000 device itself then? I have sent
ethregs output to Jesse ...hopefully that will give us some clues.
--
Regards,
vatsa