Conditions:
Intel NIC e100 device driver. Two identical machines.
Private network, no other devices. Connected using a Netgear switch.
Test data is the same thing sent from memory on one machine
to a discard server on another, using TCP/IP SOCK_STREAM.
If I set both machines to auto-negotiation OFF and half duplex,
I get about 9 to 9.5 megabytes/second across the private wire
network.
If I set one machine to full duplex and the other to half-duplex
I get 10 to 11 megabytes/second transfer across the network,
regardless of direction.
If I set both machines to auto-negotiation OFF and full duplex,
I get 300 to 400 kilobytes/second regardless of the direction.
I thought the problem must be the switch so I substituted a
cross-over wire. The problem gets worse, maybe 50 to 100
kilobytes (random bursts) per second.
`lspci`
00:00.0 Host bridge: Intel Corp. 82865G/PE/P DRAM Controller/Host-Hub Interface (rev 02)
00:01.0 PCI bridge: Intel Corp. 82865G/PE/P PCI to AGP Controller (rev 02)
00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1 (rev 02)
00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2 (rev 02)
00:1d.2 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #3 (rev 02)
00:1d.3 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #4 (rev 02)
00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI Bridge (rev c2)
00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev 02)
00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) Ultra ATA 100 Storage Controller (rev 02)
00:1f.2 IDE interface: Intel Corp. 82801EB (ICH5) Serial ATA 150 Storage Controller (rev 02)
00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02)
00:1f.5 Multimedia audio controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) AC'97 Audio Controller (rev 02)
01:00.0 VGA compatible controller: nVidia Corporation NV17 [GeForce4 MX 440] (rev a3)
02:00.0 SCSI storage controller: Adaptec AHA-2940U/UW/D / AIC-7881U (rev 01)
02:01.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 30)
02:04.0 Class 02b0: Analogic Corp: Unknown device 8004 (rev 07)
02:07.0 FireWire (IEEE 1394): Lucent Microelectronics FW323 (rev 61)
02:08.0 Ethernet controller: Intel Corp. 82562EZ 10/100 Ethernet Controller (rev 01)
proc/interrupts
CPU0
0: 507789786 IO-APIC-edge timer
1: 459692 IO-APIC-edge i8042
7: 0 IO-APIC-edge parport0
8: 1 IO-APIC-edge rtc
9: 0 IO-APIC-level acpi
12: 66 IO-APIC-edge i8042
14: 381384 IO-APIC-edge ide0
16: 0 IO-APIC-level uhci_hcd, uhci_hcd
17: 0 IO-APIC-level Intel ICH5
18: 0 IO-APIC-level libata, uhci_hcd
19: 0 IO-APIC-level uhci_hcd
20: 5748440 IO-APIC-level eth0
21: 278400 IO-APIC-level aic7xxx
23: 0 IO-APIC-level ehci_hcd
NMI: 0
LOC: 507803974
ERR: 0
MIS: 0
These are identical machines/motherboards. Anybody have a clue
why full-duplex sucks and mismatching FD/HD actually makes
things faster? Is it possible that full-duplex is actually
half-duplex of something strange like that?
Ethtool statistics:
NIC statistics:
rx_packets: 1732112
tx_packets: 3612310
rx_bytes: 162571267
tx_bytes: 4119843621
rx_errors: 1094
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 0
collisions: 50986
rx_length_errors: 933
rx_over_errors: 0
rx_crc_errors: 84
rx_frame_errors: 77
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_deferred: 298989
tx_single_collisions: 34534
tx_multi_collisions: 6779
tx_flow_control_pause: 122901
rx_flow_control_pause: 0
rx_flow_control_unsupported: 0
tx_tco_packets: 0
rx_tco_packets: 0
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
linux-os wrote:
>
> Conditions:
>
> Intel NIC e100 device driver. Two identical machines.
> Private network, no other devices. Connected using a Netgear switch.
> Test data is the same thing sent from memory on one machine
> to a discard server on another, using TCP/IP SOCK_STREAM.
>
> If I set both machines to auto-negotiation OFF and half duplex,
> I get about 9 to 9.5 megabytes/second across the private wire
> network.
>
> If I set one machine to full duplex and the other to half-duplex
> I get 10 to 11 megabytes/second transfer across the network,
> regardless of direction.
That is asking for all sorts of trouble.
> If I set both machines to auto-negotiation OFF and full duplex,
> I get 300 to 400 kilobytes/second regardless of the direction.
Check for errors in the NICs counters (/proc/net/dev/) in this case.
It appears it is not actually set to full-duplex, or maybe it's
10Mbps-FD. Use ethtool to see the actual settings.
What happens if you just don't muck with the NIC and let it auto-negotiate
on it's own?
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
> What happens if you just don't muck with the NIC and let it auto-negotiate
> on it's own?
This can be asking for trouble too (auto negotiation is often buggy).
What if you hard set them both to 100/full?
Lee
On Tue, 1 Mar 2005, Ben Greear wrote:
> linux-os wrote:
>>
>> Conditions:
>>
>> Intel NIC e100 device driver. Two identical machines.
>> Private network, no other devices. Connected using a Netgear switch.
>> Test data is the same thing sent from memory on one machine
>> to a discard server on another, using TCP/IP SOCK_STREAM.
>>
>> If I set both machines to auto-negotiation OFF and half duplex,
>> I get about 9 to 9.5 megabytes/second across the private wire
>> network.
>>
>> If I set one machine to full duplex and the other to half-duplex
>> I get 10 to 11 megabytes/second transfer across the network,
>> regardless of direction.
>
> That is asking for all sorts of trouble.
>
>> If I set both machines to auto-negotiation OFF and full duplex,
>> I get 300 to 400 kilobytes/second regardless of the direction.
>
> Check for errors in the NICs counters (/proc/net/dev/) in this case.
> It appears it is not actually set to full-duplex, or maybe it's
> 10Mbps-FD. Use ethtool to see the actual settings.
>
I supplied the actual settings.
> What happens if you just don't muck with the NIC and let it auto-negotiate
> on it's own?
>
It goes to half-duplex and runs 9 to 9.5 megabytes/second as stated
above.
That's why I think 1/2 duplex is __really__ full-duplex.
> Ben
>
>
> --
> Ben Greear <[email protected]>
> Candela Technologies Inc http://www.candelatech.com
>
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
On Tue, 1 Mar 2005, Lee Revell wrote:
> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>> What happens if you just don't muck with the NIC and let it auto-negotiate
>> on it's own?
>
> This can be asking for trouble too (auto negotiation is often buggy).
> What if you hard set them both to 100/full?
>
> Lee
>
As previously stated, the through-put is awful.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
In article <1109708691.14272.8.camel@mindpipe>,
Lee Revell <[email protected]> wrote:
>On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>> What happens if you just don't muck with the NIC and let it auto-negotiate
>> on it's own?
>
>This can be asking for trouble too (auto negotiation is often buggy).
>What if you hard set them both to 100/full?
If you do that you also need to force the switchports to full duplex.
Lots of switches default to half-duplex without auto-negotiation.
Mike.
Lee Revell wrote:
> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>
>>What happens if you just don't muck with the NIC and let it auto-negotiate
>>on it's own?
>
>
> This can be asking for trouble too (auto negotiation is often buggy).
> What if you hard set them both to 100/full?
I have not noticed any buggy autonegotiation with the e100 driver in several
years...
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
linux-os wrote:
> On Tue, 1 Mar 2005, Ben Greear wrote:
> I supplied the actual settings.
>
>> What happens if you just don't muck with the NIC and let it
>> auto-negotiate
>> on it's own?
>>
>
> It goes to half-duplex and runs 9 to 9.5 megabytes/second as stated
> above.
>
> That's why I think 1/2 duplex is __really__ full-duplex.
half-duplex will run near line speed in one direction. Try sending in
both directions at the same time and you'll get closer to 40% of the
link utilization... Also, if you see any collisions you are in half-duplex
mode.
You could try connecting the NICs back-to-back with a cross-over
cable to take your switch out of the loop?
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Tue, 2005-03-01 at 20:30 +0000, Miquel van Smoorenburg wrote:
> In article <1109708691.14272.8.camel@mindpipe>,
> Lee Revell <[email protected]> wrote:
> >On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
> >> What happens if you just don't muck with the NIC and let it auto-negotiate
> >> on it's own?
> >
> >This can be asking for trouble too (auto negotiation is often buggy).
> >What if you hard set them both to 100/full?
>
> If you do that you also need to force the switchports to full duplex.
> Lots of switches default to half-duplex without auto-negotiation.
>
Yup, that was exactly what we ended up doing.
Lee
On Tue, 2005-03-01 at 12:30 -0800, Ben Greear wrote:
> Lee Revell wrote:
> > On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
> >
> >>What happens if you just don't muck with the NIC and let it auto-negotiate
> >>on it's own?
> >
> >
> > This can be asking for trouble too (auto negotiation is often buggy).
> > What if you hard set them both to 100/full?
>
> I have not noticed any buggy autonegotiation with the e100 driver in several
> years...
>
Sorry, I misread the post. He tried this.
I was under the impression this was due to inconsistent implementation
of autonegotiation in hardware. When I was an ISP sysadmin we had this
problem with various devices (Cisco switches, Linux and BSD/OS
machines). A device would get power cycled and one side would come up
100/full, the other 100/half. We ended up hard setting everything.
Lee
On Tue, 1 Mar 2005 14:29:24 -0500 (EST), linux-os wrote:
> Intel NIC e100 device driver. Two identical machines.
> Private network, no other devices. Connected using a Netgear switch.
> Test data is the same thing sent from memory on one machine
> to a discard server on another, using TCP/IP SOCK_STREAM.
>
> If I set both machines to auto-negotiation OFF and half duplex,
> I get about 9 to 9.5 megabytes/second across the private wire
> network.
>
> If I set one machine to full duplex and the other to half-duplex
> I get 10 to 11 megabytes/second transfer across the network,
> regardless of direction.
>
> If I set both machines to auto-negotiation OFF and full duplex,
> I get 300 to 400 kilobytes/second regardless of the direction.
Might this be related to the broken BicTCP implementations in the 2.6.6+
kernels? A fix was added around 2.6.11-rc3 or 4.
-Paul
Paul Dickson wrote:
> On Tue, 1 Mar 2005 14:29:24 -0500 (EST), linux-os wrote:
>>Intel NIC e100 device driver. Two identical machines.
>>Private network, no other devices. Connected using a Netgear switch.
>>Test data is the same thing sent from memory on one machine
>>to a discard server on another, using TCP/IP SOCK_STREAM.
>>
>>If I set both machines to auto-negotiation OFF and half duplex,
>>I get about 9 to 9.5 megabytes/second across the private wire
>>network.
>>
>>If I set one machine to full duplex and the other to half-duplex
>>I get 10 to 11 megabytes/second transfer across the network,
>>regardless of direction.
>>
>>If I set both machines to auto-negotiation OFF and full duplex,
>>I get 300 to 400 kilobytes/second regardless of the direction.
>
> Might this be related to the broken BicTCP implementations in the 2.6.6+
> kernels? A fix was added around 2.6.11-rc3 or 4.
Unlikely, the problem with BIC would have shown itself only at high
speeds over long latency links, not over a lan connection.
Baruch
On Wed, 02 Mar 2005 01:02:50 +0000, Baruch Even wrote:
> > Might this be related to the broken BicTCP implementations in the 2.6.6+
> > kernels? A fix was added around 2.6.11-rc3 or 4.
>
> Unlikely, the problem with BIC would have shown itself only at high
> speeds over long latency links, not over a lan connection.
I only mentioned the possibility because I saw the same profile given by
the PDF (the link was mentioned in the patch) while downloading gnoppix
via my cable modem. The oscillations of speed varied from 40K to 500+K.
The average ended up around 270K. (I was using wget for the download).
-Paul
Paul Dickson wrote:
> On Wed, 02 Mar 2005 01:02:50 +0000, Baruch Even wrote:
>
>>>Might this be related to the broken BicTCP implementations in the 2.6.6+
>>>kernels? A fix was added around 2.6.11-rc3 or 4.
>>
>>Unlikely, the problem with BIC would have shown itself only at high
>>speeds over long latency links, not over a lan connection.
>
> I only mentioned the possibility because I saw the same profile given by
> the PDF (the link was mentioned in the patch) while downloading gnoppix
> via my cable modem. The oscillations of speed varied from 40K to 500+K.
> The average ended up around 270K. (I was using wget for the download).
If it is indeed BIC than we have a bug where it doesn't shut itself off
for low latencies. Since we don't test this case extensively here (we
work to improve high-speed and just make sure we don't ruin slower
speeds) I can't say it's impossible, try turning BIC off and see if it
helps.
Due to the scenario that the OP gave it is more likely something to do
with auto-detection somewhere along the way or a driver bug. It is also
possible that I'm mistaken and it is BIC, never hurts to check.
Baruch
I wonder if switch ports are configured as 100FDX auto=off or
100HDX auto=off.
from the report I saw it seems that switch ports are 100HDX auto=off
instead of 100FDX auto=off.
On Tue, 1 Mar 2005, Lee Revell wrote:
> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>> What happens if you just don't muck with the NIC and let it auto-negotiate
>> on it's own?
>
> This can be asking for trouble too (auto negotiation is often buggy).
> What if you hard set them both to 100/full?
>
> Lee
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
not the e100 driver, but some switch, (e.g. some matrix) has a buggy
autonegotiation.
On Tue, 1 Mar 2005, Ben Greear wrote:
> Lee Revell wrote:
>> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>>
>>> What happens if you just don't muck with the NIC and let it auto-negotiate
>>> on it's own?
>>
>>
>> This can be asking for trouble too (auto negotiation is often buggy).
>> What if you hard set them both to 100/full?
>
> I have not noticed any buggy autonegotiation with the e100 driver in several
> years...
>
> Ben
>
> --
> Ben Greear <[email protected]>
> Candela Technologies Inc http://www.candelatech.com
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Lee Revell <[email protected]> said:
> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
[...]
> > What happens if you just don't muck with the NIC and let it auto-negotiate
> > on it's own?
> This can be asking for trouble too (auto negotiation is often buggy).
I'be seen much more broken networks than buggy autonegotiation. If they
negotiate something funny, check and fix the network.
--
Dr. Horst H. von Brand User #22616 counter.li.org
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
On Wed, 2 Mar 2005, Horst von Brand wrote:
> Lee Revell <[email protected]> said:
>> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>
> [...]
>
>>> What happens if you just don't muck with the NIC and let it auto-negotiate
>>> on it's own?
>
>> This can be asking for trouble too (auto negotiation is often buggy).
>
> I'be seen much more broken networks than buggy autonegotiation. If they
> negotiate something funny, check and fix the network.
> --
In this case it's going to negotiate with the exact same device
in the other box when connected with a X-over cable, and with
a Netgear switch on my desk when not. In both cases, I have
complete control of the "network".
FYI I just upgraded to Linux-2.6.11 I'm going to repeat my
experiment(s) later today after I put the same kernel on
my other machine.
> Dr. Horst H. von Brand User #22616 counter.li.org
> Departamento de Informatica Fono: +56 32 654431
> Universidad Tecnica Federico Santa Maria +56 32 654239
> Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Cheers,
Dick Johnson
Penguin : Linux version 2.6.11 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.