This is to announce the first test release of a new Broadcom Tigon3
driver written by Jeff and myself. Get it at:
ftp://ftp.kernel.org/pub/linux/kernel/people/davem/TIGON3/tg3.patch.gz
The patch is against 2.4.18 but there is no reason it should be
difficult to apply this patch to any other tree.
It is meant to replace Broadcom's driver because frankly their driver
is junk and would never be accepted into the tree. For an example of
why their driver is junk, note that the resulting object file from our
driver is less than half the size of Broadcom's. That kind of bloat
is simply unacceptable. Next, Broadcom's driver is still way
non-portable, ioremap() pointers are still dereferenced directly among
other things. Finally, their driver is just plain buggy, they have
code which tries to use page_address() on pages which are potentially
in highmem and that is guarenteed to oops.
There are other problems with their driver too, just ask as I'm in a
good mood to grind my axe about this stuff :-)
We would also like to give Broadcom a big "no thanks" because
their lawyers refused to give Jeff the documentation for the Tigon3
chipset using an NDA that would allow him to write a GPL'd driver
based upon said documentation. This means that all that we know about
the hardware has been reverse engineered from Broadcom's GPL'd driver
plus some experimenting. It is why this driver has taken so long to
finish, because it is hard to find incentive for this kind of brack
breaking work when the vendor is totally uncooperative.
I personally think Broadcom has some hidden agenda that requires that
their driver be "the one". So instead of just complaining about such
stupid policy, we've done something about it by doing our own driver
with all the problems fixed.
We'd really appreciate if people would test this thing out as hard
as they can. I can tell you that I've personally only been able to
test out the following scenerios so far:
1) Both big-endian and little-endian platforms.
2) 64-bit on big-endian side, 32-bit on little-endian side.
3) IOMMU based PCI dma platform on 64-bit/big-endian side,
non-IOMMU based PCI dma platform on 32-bit little-endian side.
4) 10baseT half duplex, 100baseT full duplex
5) Copper based connectors, as that is all that Jeff and I have.
Gigabit and fibre are the big areas where our testing has been
lacking. We have also not done any tuning of the driver at all,
although we do consider the driver feature complete.
Can you list what cards/systems use this driver? I at least am totally
unsure.
Thanks
Nick
On Mon, 25 Feb 2002, David S. Miller wrote:
>
> This is to announce the first test release of a new Broadcom Tigon3
> driver written by Jeff and myself. Get it at:
>
> ftp://ftp.kernel.org/pub/linux/kernel/people/davem/TIGON3/tg3.patch.gz
>
> The patch is against 2.4.18 but there is no reason it should be
> difficult to apply this patch to any other tree.
>
> It is meant to replace Broadcom's driver because frankly their driver
> is junk and would never be accepted into the tree. For an example of
> why their driver is junk, note that the resulting object file from our
> driver is less than half the size of Broadcom's. That kind of bloat
> is simply unacceptable. Next, Broadcom's driver is still way
> non-portable, ioremap() pointers are still dereferenced directly among
> other things. Finally, their driver is just plain buggy, they have
> code which tries to use page_address() on pages which are potentially
> in highmem and that is guarenteed to oops.
>
> There are other problems with their driver too, just ask as I'm in a
> good mood to grind my axe about this stuff :-)
>
> We would also like to give Broadcom a big "no thanks" because
> their lawyers refused to give Jeff the documentation for the Tigon3
> chipset using an NDA that would allow him to write a GPL'd driver
> based upon said documentation. This means that all that we know about
> the hardware has been reverse engineered from Broadcom's GPL'd driver
> plus some experimenting. It is why this driver has taken so long to
> finish, because it is hard to find incentive for this kind of brack
> breaking work when the vendor is totally uncooperative.
>
> I personally think Broadcom has some hidden agenda that requires that
> their driver be "the one". So instead of just complaining about such
> stupid policy, we've done something about it by doing our own driver
> with all the problems fixed.
>
> We'd really appreciate if people would test this thing out as hard
> as they can. I can tell you that I've personally only been able to
> test out the following scenerios so far:
>
> 1) Both big-endian and little-endian platforms.
> 2) 64-bit on big-endian side, 32-bit on little-endian side.
> 3) IOMMU based PCI dma platform on 64-bit/big-endian side,
> non-IOMMU based PCI dma platform on 32-bit little-endian side.
> 4) 10baseT half duplex, 100baseT full duplex
> 5) Copper based connectors, as that is all that Jeff and I have.
>
> Gigabit and fibre are the big areas where our testing has been
> lacking. We have also not done any tuning of the driver at all,
> although we do consider the driver feature complete.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
From: [email protected]
Date: Mon, 25 Feb 2002 22:43:53 -0500 (EST)
Can you list what cards/systems use this driver? I at least am totally
unsure.
I can't tell you much more than the list of PCI device IDs and those
are contained in the driver sources.
The first two cards I have for testing are 3COM copper gigabit cards
but I have no idea what the official model numbers are for these as
they came with no packaging.
On Mon, Feb 25, 2002 at 08:40:22PM -0800, David S. Miller wrote:
> From: [email protected]
> Date: Mon, 25 Feb 2002 22:43:53 -0500 (EST)
>
> Can you list what cards/systems use this driver? I at least am totally
> unsure.
>
> I can't tell you much more than the list of PCI device IDs and those
> are contained in the driver sources.
>
> The first two cards I have for testing are 3COM copper gigabit cards
> but I have no idea what the official model numbers are for these as
> they came with no packaging.
I think these are 3C996-T. Is there any information about the compatibility of
TG3 to TG2 ?
_sh_
From: Sebastian Heidl <[email protected]>
Date: Tue, 26 Feb 2002 12:22:05 +0100
I think these are 3C996-T. Is there any information about the
compatibility of TG3 to TG2 ?
Totally different chip, but there are subtle similarities all over
the place. Ie. if you knew how to program the tg2, or even were just
familiar with the acenic driver, the tg3 stuff looks familiar.
On Tue, Feb 26, 2002 at 03:24:53AM -0800, David S. Miller wrote:
> From: Sebastian Heidl <[email protected]>
> Date: Tue, 26 Feb 2002 12:22:05 +0100
>
> I think these are 3C996-T. Is there any information about the
> compatibility of TG3 to TG2 ?
>
> Totally different chip, but there are subtle similarities all over
> the place. Ie. if you knew how to program the tg2, or even were just
> familiar with the acenic driver, the tg3 stuff looks familiar.
That's what I noticed looking at tg3.c ;-)
I guess nobody was crazy enough yet to try to load a tg2-firmware on a tg3 ?
Only as there are some utils to build a customized firmware for the tg2.
just guessing wildly ;-)
_sh_
From: Sebastian Heidl <[email protected]>
Date: Tue, 26 Feb 2002 12:39:49 +0100
I guess nobody was crazy enough yet to try to load a tg2-firmware on a tg3 ?
Only as there are some utils to build a customized firmware for the tg2.
Different cpus, tg3 uses StrongARM and tg2 uses a lobotomized MIPS.
David S. Miller:
> Gigabit and fibre are the big areas where our testing has been
> lacking. We have also not done any tuning of the driver at all,
> although we do consider the driver feature complete.
I've tested the driver from broadcom (yeah, yeah, the sucky driver :) and
now I've tested yours, and this is the result (from dmesg):
tg3.c:v0.90 (Feb 25, 2002)
tg3: Problem fetching invariants of chip, aborting.
This is lspci -vvvxx:
01:08.0 Ethernet controller: BROADCOM Corporation BCM5700 1000BaseTX (rev 12)
Subsystem: Dell Computer Corporation: Unknown device 00d1
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin A routed to IRQ 17
Region 0: Memory at feb00000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM- Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
Address: 9b8dc9bb84b8f3a8 Data: bed9
00: e4 14 44 16 02 01 b0 02 12 00 00 02 08 20 00 00
10: 04 00 b0 fe 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 d1 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 40 00
If you need any more info, I'll be happy to provide.
--
Thomas
From: Thomas Lang?s <[email protected]>
Date: Tue, 26 Feb 2002 14:57:30 +0100
tg3.c:v0.90 (Feb 25, 2002)
tg3: Problem fetching invariants of chip, aborting.
Please apply this patch and tell me what it spits
out, thanks.
--- drivers/net/tg3.c.~1~ Mon Feb 25 17:50:17 2002
+++ drivers/net/tg3.c Tue Feb 26 06:58:39 2002
@@ -4424,12 +4424,20 @@
if (grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5700 &&
grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5701 &&
grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5703 &&
- grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5703S)
+ grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5703S) {
+ printk("DEBUG: Yikes grc_misc_cfg is %08x\n",
+ grc_misc_cfg);
return -ENODEV;
+ }
err = tg3_phy_probe(tp);
- if (!err)
+ if (err)
+ printk("DEBUG: phy_probe returns with %d\n", err);
+ if (!err) {
err = tg3_read_partno(tp);
+ if (err)
+ printk("DEBUG: read_partno returns with %d\n", err);
+ }
if (tp->phy_id == PHY_ID_SERDES) {
tp->tg3_flags &= ~TG3_FLAG_USE_PHY_IRQ;
David S. Miller:
> Please apply this patch and tell me what it spits
> out, thanks.
tg3.c:v0.90 (Feb 25, 2002)
DEBUG: read_partno returns -19
tg3: Problem fetching invariants of chip, aborting.
--
Thomas
I'm trying to debug what looks like a race-condition I reach with heavy
interrupt traffic, including heavy ethernet traffic.
The result of the race (if it is one) is that the pointers in
skb_head_pool in the network code get corrupted, like the list heads
->next pointer points to the zero-page and other nasty things.
I've gotten an oops from different codepaths but one caught my eye
especially in that there seems to be a codepath with IRQ's enabled which
calls __skb_dequeue, namely do_softirq calls net_tx_action which uses
qdisc_restart which calls q->dequeue, which contains pfifo_fast_dequeue
which calls __skb_dequeue directly.
Trace; c009a508 <pfifo_fast_dequeue+22/5a>
Trace; c009a16e <qdisc_restart+16/b6>
Trace; c0094b44 <net_tx_action+7c/8c>
Trace; c0008a8c <do_softirq+4c/94>
Before diving deeper into the fray, is this normal ? If __skb_dequeue is
called with irq's enabled, couldn't another network interrupt cause a
race, corrupting the skb_head_pool linked structure ?
/BW
Just wanted to say thanks for doing this work, the driver works great
for me for this device:
01:04.0 Ethernet controller: BROADCOM Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 12)
Subsystem: BROADCOM Corporation NetXtreme BCM5700 1000BaseTX
Flags: bus master, 66Mhz, medium devsel, latency 240, IRQ 42
Memory at f0e00000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] PCI-X non-bridge device.
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
But I'm only able to run it at 10Mbit :)
thanks,
greg k-h
On Mon, 25 Feb 2002, David S. Miller wrote:
>
> This is to announce the first test release of a new Broadcom Tigon3
> driver written by Jeff and myself. Get it at:
>
Does/Will this driver support NICE-extensions as does the Broadcom's
driver? I've been successfully using broadcom's driver with the
nice-patched vlan-code for a couple of months now..
I think Ben Greear is about to submit NICE patch for VLAN to be included
in the kernel soon..
(NICE extensions enable hardware vlan-tagging etc for linux VLAN code).
btw. Does any other driver support NICE-extensions?
- Pasi K?rkk?inen
^
. .
Linux
/ - \
Choice.of.the
.Next.Generation.
From: Pasi K?rkk?inen <[email protected]>
Date: Tue, 26 Feb 2002 20:08:51 +0200 (EET)
Does/Will this driver support NICE-extensions as does the Broadcom's
driver? I've been successfully using broadcom's driver with the
nice-patched vlan-code for a couple of months now..
NICE is a gross hack and I would hope that Ben Greear would have
better taste in the interface he chooses for hw VLAN acceleration in
the 802.1q layer :-)
NICE was explicitly left out and this was done on purpose.
From: Thomas Lang?s <[email protected]>
Date: Tue, 26 Feb 2002 16:40:44 +0100
tg3.c:v0.90 (Feb 25, 2002)
DEBUG: read_partno returns -19
tg3: Problem fetching invariants of chip, aborting.
Great, now add the following patch below and let me know what it does
and prints out now.
If the module still fails to load because of the -EBUSY error (ie. the
"read_partno returns -19" thing happens again), bring
drivers/net/tg3.c into an editor and go to around line 4185 and change
the line that reads:
while (--limit) {
into:
while (1) {
And see if it works then. PLEASE type sync a few times before trying
to load the module in this case as it could very well hang your
machine.
Thanks again for all the testing so far:
--- drivers/net/tg3.c.~2~ Mon Feb 25 17:51:41 2002
+++ drivers/net/tg3.c Tue Feb 26 18:51:33 2002
@@ -4168,16 +4168,18 @@
static int __devinit tg3_read_partno(struct tg3 *tp)
{
unsigned char vpd_data[256];
+ unsigned int smallest_limit = ~0U;
int i;
/* Enable seeprom accesses. */
- tw32(GRC_LOCAL_CTRL, GRC_LCLCTRL_AUTO_SEEPROM);
+ tw32(GRC_LOCAL_CTRL,
+ tr32(GRC_LOCAL_CTRL) | GRC_LCLCTRL_AUTO_SEEPROM);
udelay(100);
for (i = 0; i < 256; i += 4) {
u32 tmp;
u16 stat;
- int limit = 5000;
+ unsigned int limit = 100000;
pci_write_config_word(tp->pdev, TG3PCI_VPD_ADDR_FLAG, i);
while (--limit) {
@@ -4188,6 +4190,8 @@
}
if (!limit)
return -EBUSY;
+ if (limit < smallest_limit)
+ smallest_limit = limit;
pci_read_config_dword(tp->pdev, TG3PCI_VPD_DATA, &tmp);
@@ -4196,6 +4200,8 @@
vpd_data[i + 2] = ((tmp >> 16) & 0xff);
vpd_data[i + 3] = ((tmp >> 24) & 0xff);
}
+
+ printk("DEBUG: smallest_limit is %u\n", smallest_limit);
/* Now parse and find the part number. */
for (i = 0; i < 256; ) {
From: Greg KH <[email protected]>
Date: Tue, 26 Feb 2002 09:22:51 -0800
Just wanted to say thanks for doing this work, the driver works great
for me for this device:
01:04.0 Ethernet controller: BROADCOM Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 12)
Subsystem: BROADCOM Corporation NetXtreme BCM5700 1000BaseTX
...
But I'm only able to run it at 10Mbit :)
Thanks for the report, but can you also show us the "dmesg" lines
that the driver printed out when the module was loaded? That provides
more detailed probing information for us.
From: Thomas Lang?s <[email protected]>
Date: Wed, 27 Feb 2002 10:24:50 +0100
I'll test all this, but if you want, I can provide you with root on that
machine, it's only a test-box for now, hooked up on one 100 Mbps and one
1Gbps (that's the broadcom card).
It shouldn't be necessary. :)
I'll report back with results within the hour :)
Thank you.
David S. Miller:
> Great, now add the following patch below and let me know what it does
> and prints out now.
tg3.c:v0.90 (Feb 25, 2002)
DEBUG: smallest_limit is 9897
DEBUG: read_partno returned -19
tg3: Problem fetching invariants of chip, aborting.
> If the module still fails to load because of the -EBUSY error (ie. the
> "read_partno returns -19" thing happens again), bring
> drivers/net/tg3.c into an editor and go to around line 4185 and change
> the line that reads:
> [snipp]
> And see if it works then. PLEASE type sync a few times before trying
> to load the module in this case as it could very well hang your
> machine.
Didn't work, didn't hang the box either :)
Any more ideas? :)
--
Thomas
From: Thomas Lang?s <[email protected]>
Date: Wed, 27 Feb 2002 12:05:49 +0100
> If the module still fails to load because of the -EBUSY error (ie. the
> "read_partno returns -19" thing happens again), bring
> drivers/net/tg3.c into an editor and go to around line 4185 and change
> the line that reads:
> [snipp]
> And see if it works then. PLEASE type sync a few times before trying
> to load the module in this case as it could very well hang your
> machine.
Didn't work, didn't hang the box either :)
What did it print out when you changed the code to
be "while (1)"? It must print something different.
David S. Miller:
> What did it print out when you changed the code to
> be "while (1)"? It must print something different.
Here's the output (counter is something I added inside the while (1)
{}-loop, on top in that loop, it's unsigned int, set to 0 to start with):
tg3.c:v0.90 (Feb 25, 2002)
DEBUG: counter 6592
DEBUG: smallest_limit is 10000
DEBUG: read_partno returned -19
tg3: Problem fetching invariants of chip, aborting.
smallest_limit is naturally 10k since limit isn't decremented...
--
Thomas
From: Thomas Lang?s <[email protected]>
Date: Wed, 27 Feb 2002 12:56:11 +0100
tg3.c:v0.90 (Feb 25, 2002)
DEBUG: counter 6592
DEBUG: smallest_limit is 10000
DEBUG: read_partno returned -19
tg3: Problem fetching invariants of chip, aborting.
smallest_limit is naturally 10k since limit isn't decremented...
Oh I see... ok so it gets the vital product data, but it
cannot find the part number string.
Please, simply make tg3_read_partno always return 0 (or,
alternatively, make tg3_get_invariants() ignore tg3_read_partno's
return value). It should just work, this string is not critical to
the driver's operation.
David S. Miller:
> Please, simply make tg3_read_partno always return 0 (or,
> alternatively, make tg3_get_invariants() ignore tg3_read_partno's
> return value). It should just work, this string is not critical to
> the driver's operation.
Do you want a dump of the vpd_data it receives? Here's the dmesg line after
the driver is loaded:
tg3.c:v0.90 (Feb 25, 2002)
eth1: Tigon3 [partno() rev 7102 PHY(5401)] (PCI:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:06:5b:05:f9:8e
The machine is a Dell PowerEdge 2550 machine, running RedHat Linux 7.2 (with
2.4.18 kernel). Let me know if you want me to run any specific tests, it's
connected to a Cisco switch with gigabit-ports (so I can actually test
things like jumboframes if needed).
--
Thomas
From: Thomas Lang?s <[email protected]>
Date: Wed, 27 Feb 2002 13:24:54 +0100
Do you want a dump of the vpd_data it receives?
It isn't interesting, so no.
Here's the dmesg line after
the driver is loaded:
Thank you.
The machine is a Dell PowerEdge 2550 machine, running RedHat Linux 7.2 (with
2.4.18 kernel). Let me know if you want me to run any specific tests, it's
connected to a Cisco switch with gigabit-ports (so I can actually test
things like jumboframes if needed).
At this point I'm mostly interested in if it works at all :-)
If the answer is yes, tell me that and then you can feel
free to experiment with jumbo frames et al. to discover
other bugs in the driver :-)
David S. Miller:
> At this point I'm mostly interested in if it works at all :-)
> If the answer is yes, tell me that and then you can feel
> free to experiment with jumbo frames et al. to discover
> other bugs in the driver :-)
Just tested with MTU set at 1500 for now, but it seems to work fine, did a
netcat between two boxes on the same switch and got around 80MB/sec.
Any programs or anything that could do a serious stresstest? (Both hosts
are Dell PowerEdge 2550, RedHat Linux 7.2).
--
Thomas
Thomas Lang?s wrote:
>
> David S. Miller:
> > At this point I'm mostly interested in if it works at all :-)
> > If the answer is yes, tell me that and then you can feel
> > free to experiment with jumbo frames et al. to discover
> > other bugs in the driver :-)
>
> Just tested with MTU set at 1500 for now, but it seems to work fine, did a
> netcat between two boxes on the same switch and got around 80MB/sec.
>
> Any programs or anything that could do a serious stresstest? (Both hosts
> are Dell PowerEdge 2550, RedHat Linux 7.2).
google for nttcp
runs a nice tcp performance test...
Thomas Lang?s wrote:
>
> David S. Miller:
> > At this point I'm mostly interested in if it works at all :-)
> > If the answer is yes, tell me that and then you can feel
> > free to experiment with jumbo frames et al. to discover
> > other bugs in the driver :-)
>
> Just tested with MTU set at 1500 for now, but it seems to work fine, did a
> netcat between two boxes on the same switch and got around 80MB/sec.
>
> Any programs or anything that could do a serious stresstest? (Both hosts
> are Dell PowerEdge 2550, RedHat Linux 7.2).
I've had good luck with an infinite loop on sendto(). Can easily saturate a
link, and depending on the receiving host, can essentially DOS the machine.
As an example, a Dell OptiPlex GX1 running 2.4.17 becomes completely unusable
when receiving 80000 packets/sec.
Chris
--
Chris Friesen | MailStop: 043/33/F10
Nortel Networks | work: (613) 765-0557
3500 Carling Avenue | fax: (613) 765-2986
Nepean, ON K2H 8E9 Canada | email: [email protected]
Arjan van de Ven:
> google for nttcp
> runs a nice tcp performance test...
Ok, here's some numbers:
UDP:
kiwi:~/nttcp-1.47# ./nttcp -T -v -r -u 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=3702
nttcp-l: send window size = 65535
nttcp-l: receive window size = 65535
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=1820, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-u@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: send window size = 65535
nttcp-1: receive window size = 65535
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: got EOF
nttcp-l: received 7725056 bytes
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s CPU-C/s
l 7725056 0.07 0.07 878.2714 882.8635 1887 26816.93 26957.1
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
nttcp-l: try to get outstanding messages from 1 remote clients
1 8388608 0.07 0.05 966.3321 1342.1773 2051 29533.31 41020.0
nttcp-1: exiting
TCP:
kiwi:~/nttcp-1.47# ./nttcp -T -v -r 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=3705
nttcp-l: accept from 129.241.57.161
nttcp-l: send window size = 16384
nttcp-l: receive window size = 87380
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=1821, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-1: send window size = 16384
nttcp-1: receive window size = 87380
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: received 8388608 bytes
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s CPU-C/s
l 8388608 0.08 0.05 865.7309 1342.1773 2177 28084.16 43540.0
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
1 8388608 0.08 0.04 864.4930 1677.7216 2048 26382.23 51200.0
nttcp-1: exiting
nttcp on the receiving host is running throuhg xinetd, I don't think that
impacts the performance, tho.
--
Thomas
> > Any programs or anything that could do a serious stresstest? (Both hosts
> > are Dell PowerEdge 2550, RedHat Linux 7.2).
>
> google for nttcp
> runs a nice tcp performance test...
I dig on gensink:
http://jes.home.cern.ch/jes/gensink/
the attached hack gives it the ability to tx with sendfile() from a
ftruncate()ed tmpfile based on the presence of a 5th argument, which can
be fun with scatter/csum capable cards. I've put up some rpms for the
lazy:
http://www.zabbo.net/rpms/gensink/
5aa66dfd7749c77cb58de22fad96de gensink-4.1-1sendfile.i386.rpm
04912a63bda95a79b6b835f340c924c2 gensink-4.1-1sendfile.src.rpm
enjoy.
- z
diff -urN gensink-4.1.orig/gen4.c gensink-4.1/gen4.c
--- gensink-4.1.orig/gen4.c Wed May 16 10:08:01 2001
+++ gensink-4.1/gen4.c Wed Feb 27 10:24:27 2002
@@ -48,20 +48,101 @@
#include <sys/ioctl.h>
+#include <sys/sendfile.h>
+
+void sendfile_loop(int s, int rec_len)
+{
+ char *tmpdir_default = "/tmp";
+ char template[PATH_MAX];
+ char *tmpdir;
+ int fd;
+ int i;
+ ssize_t bytes_written;
+ off_t offset;
+
+ tmpdir = getenv("TMPDIR");
+ if ( tmpdir == NULL )
+ tmpdir = tmpdir_default;
+
+ snprintf(template, PATH_MAX, "%s/XXXXXX", tmpdir);
+
+ fd = mkstemp(template);
+ if ( fd < 0 ) {
+ perror("couldn't mkstemp() sendfile() backing file");
+ exit(-1);
+ }
+
+ unlink(template);
+
+ if ( ftruncate(fd, RECORD_BUFFER_SIZE + rec_len) < 0 ) {
+ perror("frunctate() backing file");
+ exit(-1);
+ }
+
+ offset = 0;
+
+ for (i = 1; i < 4000000; i++)
+ {
+ /* loop writing blocks */
+ if ((bytes_written = sendfile(s,fd,&offset,rec_len)) < 0)
+ oops("write");
+ if ((bytes = bytes + bytes_written) >= REPORT_FREQ)
+ {
+ report(1, "source", bytes);
+ bytes = 0;
+ }
+
+ if ( offset >= RECORD_BUFFER_SIZE )
+ offset = 0;
+ }
+
+ close(fd);
+}
+
+void write_loop(int s, int rec_len)
+{
+ char *record_buffer, *orig_record_ptr;
+ int i, bytes_written;
+
+ record_buffer = (char *)malloc(RECORD_BUFFER_SIZE + rec_len);
+ if (!record_buffer) {
+ fprintf(stderr, "Unable to allocate memory for transmit buffer\n");
+ exit(-1);
+ }
+
+ orig_record_ptr = record_buffer;
+ memset(record_buffer, 0, RECORD_BUFFER_SIZE);
+
+ for (i = 1; i < 4000000; i++)
+ {
+ /* loop writing blocks */
+ if ((bytes_written = write(s,record_buffer,rec_len)) < 0)
+ oops("write");
+ if ((bytes = bytes + bytes_written) >= REPORT_FREQ)
+ {
+ report(1, "source", bytes);
+ bytes = 0;
+ }
+ record_buffer += rec_len;
+ if ((unsigned long)record_buffer > (unsigned long)(orig_record_ptr + RECORD_BUFFER_SIZE))
+ record_buffer = orig_record_ptr;
+ }
+ free(orig_record_ptr);
+}
+
int main(argc, argv)
int argc;
char *argv[];
{
struct sockaddr_in saddr;
- char *record_buffer, *orig_record_ptr;
char reptext[256], *hostname;
- int portnum, rec_len, bytes_written;
- int i;
+ int portnum, rec_len;
+ int use_sendfile = 0;
/* check argument count */
- if(argc != 5)
+ if(argc < 5 || argc > 6)
{
- fprintf(stderr,"Usage: %s server_host server_port record_length setsockopt\n", argv[0]);
+ fprintf(stderr, "Usage: %s server_host server_port record_length setsockopt [use_sendfile_boolean]\n", argv[0]);
exit(-1);
}
@@ -73,16 +154,11 @@
exit(-1);
}
- record_buffer = (char *)malloc(RECORD_BUFFER_SIZE + rec_len);
- if (!record_buffer) {
- fprintf(stderr, "Unable to allocate memory for transmit buffer\n");
- exit(-1);
- }
- orig_record_ptr = record_buffer;
- memset(record_buffer, 0, RECORD_BUFFER_SIZE);
-
max = atoi(argv[4]);
+ if ( argc == 6 )
+ use_sendfile = atoi(argv[5]);
+
strcpy(reptext,"");
/* get the record length */
strncat(reptext, " reclen=", 255 - strlen(reptext));
@@ -122,24 +198,17 @@
printf("Max seg TCP : %d\n",maxseg);
+ printf("sending with %s\n",
+ use_sendfile ? "sendfile(2)" : "write(2)");
+
/* initialise reporting */
report_start(reptext,1,0,1,argv);
report(1, "source", 0);
- for (i = 1; i < 4000000; i++)
- {
- /* loop writing blocks */
- if ((bytes_written = write(s,record_buffer,rec_len)) < 0)
- oops("write");
- if ((bytes = bytes + bytes_written) >= REPORT_FREQ)
- {
- report(1, "source", bytes);
- bytes = 0;
- }
- record_buffer += rec_len;
- if ((unsigned long)record_buffer > (unsigned long)(orig_record_ptr + RECORD_BUFFER_SIZE))
- record_buffer = orig_record_ptr;
- }
- free(orig_record_ptr);
+ if ( use_sendfile )
+ sendfile_loop(s, rec_len);
+ else
+ write_loop(s, rec_len);
+
return 0;
}
> 5aa66dfd7749c77cb58de22fad96de gensink-4.1-1sendfile.i386.rpm
> 04912a63bda95a79b6b835f340c924c2 gensink-4.1-1sendfile.src.rpm
that should be..
3a5aa66dfd7749c77cb58de22fad96de gensink-4.1-1sendfile.i386.rpm
04912a63bda95a79b6b835f340c924c2 gensink-4.1-1sendfile.src.rpm
*sigh*
- z
[ whats the point of md5sums in an unsigned mail anyway? Who knows. ]
Arjan van de Ven:
> google for nttcp
> runs a nice tcp performance test...
I did some more benchmarking as Jeff Garzik requested:
[S: sending host, R: receiving host]
1. S: Broadcom's GPLed driver
R: Miller/Garzik's GPLed driver
2. S: Broadcom's GPLed driver
R: Broadcom's GPLed driver
3. S: Miller/Garzik's GPLed driver
R: Broadcom's GPLed driver
(scenario 4, Miller/Garzik on both ends has already been tested, see another
email in this thread).
__Scenario 1__
UDP:
kiwi:~/nttcp-1.47# ./nttcp -T -v -r -u 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4354
nttcp-l: send window size = 65535
nttcp-l: receive window size = 65535
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=1930, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-u@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: send window size = 65535
nttcp-1: receive window size = 65535
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: got EOF
nttcp-l: received 5910528 bytes
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s CPU-C/s
l 5910528 0.07 0.08 656.3334 591.0528 1444 20043.59 18050.0
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
nttcp-l: try to get outstanding messages from 1 remote clients
1 8388608 0.07 0.07 962.3826 958.6981 2051 29412.61 29300.0
nttcp-1: exiting
TCP:
kiwi:~/nttcp-1.47# ./nttcp -T -v -r 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4355
nttcp-l: accept from 129.241.57.161
nttcp-l: send window size = 16384
nttcp-l: receive window size = 87380
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=1931, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-1: send window size = 16384
nttcp-1: receive window size = 87380
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: received 8388608 bytes
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s CPU-C/s
l 8388608 0.09 0.09 754.8719 745.6540 2073 23318.07 23033.3
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
1 8388608 0.09 0.07 759.8205 958.6981 2048 23187.88 29257.1
nttcp-1: exiting
__Scenario 2__
UDP:
kiwi:~/nttcp-1.47# ./nttcp -T -v -r -u 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4409
nttcp-l: send window size = 65535
nttcp-l: receive window size = 65535
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=2066, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-u@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: send window size = 65535
nttcp-1: receive window size = 65535
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: got EOF
nttcp-l: received 6094848 bytes
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s CPU-C/s
l 6094848 0.07 0.07 678.4114 696.5541 1489 20717.39 21271.4
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
nttcp-l: try to get outstanding messages from 1 remote clients
1 8388608 0.07 0.07 936.5683 958.6981 2051 28623.66 29300.0
nttcp-1: exiting
TCP:
kiwi:~/nttcp-1.47# ./nttcp -T -v -r 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4410
nttcp-l: accept from 129.241.57.161
nttcp-l: send window size = 16384
nttcp-l: receive window size = 87380
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=2067, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-1: send window size = 16384
nttcp-1: receive window size = 87380
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: received 8388608 bytes
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s CPU-C/s
l 8388608 0.09 0.08 758.2323 838.8608 2061 23286.29 25762.5
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
1 8388608 0.09 0.04 763.3294 1677.7216 2048 23294.96 51200.0
nttcp-1: exiting
__Scenario 3__
UDP:
kiwi:~/nttcp-1.47# ./nttcp -T -v -r -u 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4509
nttcp-l: send window size = 65535
nttcp-l: receive window size = 65535
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=2106, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-u@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: send window size = 65535
nttcp-1: receive window size = 65535
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: got EOF
nttcp-l: received 8192000 bytes
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s CPU-C/s
l 8192000 0.07 0.07 914.3367 936.2286 2001 27917.29 28585.7
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
nttcp-l: try to get outstanding messages from 1 remote clients
1 8388608 0.07 0.06 939.5448 1118.4811 2051 28714.63 34183.3
nttcp-1: exiting
TCP:
kiwi:~/nttcp-1.47# ./nttcp -T -v -r 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4512
nttcp-l: accept from 129.241.57.161
nttcp-l: send window size = 16384
nttcp-l: receive window size = 87380
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=2107, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-1: send window size = 16384
nttcp-1: receive window size = 87380
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: received 8388608 bytes
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s CPU-C/s
l 8388608 0.08 0.06 877.2286 1118.4811 2179 28483.29 36316.7
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
1 8388608 0.08 0.04 872.2687 1677.7216 2048 26619.53 51200.0
nttcp-1: exiting
Hope this helps :)
--
Thomas
This statement is a response to the recent negative comments about the
Broadcom Gigabit
ethernet driver by the Linux community.
Our Gigabit ethernet driver was written as a GPL'ed open source driver to
support various 2.2.x and 2.4.x Linux kernels. The primary objectives were
to support Intel
i386 and ia64 platforms and provide high performance especially in server
type
applications. The driver was also written in a way that proprietary
information about the hardware would not be unnecessarily disclosed. This is
necessary to protect our intellectual property and to keep a competitive
edge in the highly competitive Gigabit NIC marketplace.
We don't claim to be Linux experts but no one knows about our hardware more
than we do. At this point, we cannot support the Linux open source community
to write their
own driver. Doing so would require us to disclose too much proprietary
information about
our hardware and put us in a competitive disadvantage in the Gigabit
marketplace. We stronly
believe our driver is solid and provides high performance for our customers.
In one
benchmark test, we've achieved better than 1.8 Gigabit total throughput
using jumbo frames. While
our emphasis has been on Intel i386 and ia64 platforms, our recent versions
of the driver are also
known to work on PowerPC, Sparc, and alpha platforms. While we welcome any
constructive suggestions on improving our driver in anyway, we want to point
out that there are
different styles to writing a device driver, not just the style advocated by
a couple of
arrogant Linux people.
Here more details comments to the Dave Miller's email on Broadcom Driver.
> >It is meant to replace Broadcom's driver because frankly their driver
> > is junk and would never be accepted into the tree. For an example of
> > why their driver is junk, note that the resulting object file from our
> > driver is less than half the size of Broadcom's. That kind of bloat
> > is simply unacceptable.
[BRCM] Our driver is 117K, Intel's driver is 82K, the Altima driver is 82K.
It has
a lot of features and carries all backward compatible for all chips
including
firmware.
> > Next, Broadcom's driver is still way
> > non-portable, ioremap() pointers are still dereferenced directly among
> > other things.
[BRCM] The driver was changed to use readl/writel macro in version 2.0.31 on
12/14/01 when we started testing on other non-Intel platforms and added
big-endian support. Prior to that we only supported Intel platforms and
direct access is not a problem on Intel platforms.
> > Finally, their driver is just plain buggy, they have
> > code which tries to use page_address() on pages which are potentially
> > in highmem and that is guarenteed to oops.
[BRCM] It is true that the driver uses page_address() in one subroutine that
is
used to workaround a problem on the very early 5700 chip. But this routine
is not used at all, it was there intially to support early rev of the
silicon.
It was removed in later version of the Broadcom driver.
Regards,
Timothy Ngo
> different styles to writing a device driver, not just the style advocated by
> a couple of arrogant Linux people.
Documentation/CodingStyle
Its the standard everyone holds to, be they Dave Miller or IBM Corp. If
you wan't your driver in the main tree then its not something you need
to care about
The standard held for the mainstream kernel are very high - and they need
to be. Nobody makes you hold to it, and in fact that you have written a
GPL driver is great - whatever it looks like. Its allowed those people who
do care to not only bitch about it but to actually go and try and do a
better job - at their own expense not yours.
Alan
From: "Timothy Ngo" <[email protected]>
Date: Mon, 11 Mar 2002 16:57:44 -0800
The driver was also written in a way that proprietary information
about the hardware would not be unnecessarily disclosed. This is
necessary to protect our intellectual property and to keep a
competitive edge in the highly competitive Gigabit NIC marketplace.
Then please tell us why other gigabit vendors are able to work with
the Linux developer community and Broadcom is not able to? :-)
At this point, we cannot support the Linux open source community
to write their own driver.
Frankly, we don't need your support anymore. Once it was realized
that we would not receive any help from you we did everything in our
power to produce a driver and situation that did not require
Broadcom's help. We couldn't depend upon you, so we took appropriate
actions.
In one benchmark test, we've achieved better than 1.8 Gigabit total
throughput using jumbo frames.
Our driver is faster than yours, and this is backed up by reports done
by public testers of our driver. Our cases are well documented,
whereas all you can do is mention is a magic "one benchmark test" and
a magic performance result which contains no details. It seems that
knowing more about the hardware than anyone else doesn't seem to help
you all that much in this area. This I find amusing. :-)
We make regular and detailed releases. Our work is publicly
documented and very accessible to the community. Whereas Broadcom
does releases behind closed doors and only distribution vendors even
hear about when this happens.
[BRCM] Our driver is 117K, Intel's driver is 82K, the Altima driver is 82K.
It has
a lot of features and carries all backward compatible for all chips
including
firmware.
Our driver fully supports not only the same exact hardware and
revisions as your driver, it supports MORE chipsets (and MORE
features) than your driver including but not limited to the Syskonnect
Tigon3 based boards.
Your argument for having a huge driver binary size is garbage.
Our driver supports all of your hardware fully, and is HALF the
size.
Your arguments are not only erroneous, they show how deeply you do
not understand the problems the Linux developer community has with
your driver.
> > Finally, their driver is just plain buggy, they have
> > code which tries to use page_address() on pages which are potentially
> > in highmem and that is guarenteed to oops.
[BRCM] It is true that the driver uses page_address() in one subroutine that
is used to workaround a problem on the very early 5700 chip. But this routine
is not used at all, it was there intially to support early rev of the
silicon. It was removed in later version of the Broadcom driver.
Removing it removes support for HIGHMEM zerocopy in your driver for
those revisions when, in fact, you could have simply used calls to
skb_copy() to fix the bug. Our driver handles this correctly, not
penalizing performance on this card revision regardless of how "rare"
or "unimportant" it is. What you have done to fix this bug in your
driver is precisely the kind of garbage change we would never accept
into a driver found in the Linux sources.
Frankly I am disgusted with Broadcom's attitude here. No other vendor
gives us this much of a problem with their gigabit Linux drivers.
Intel is not crying about "competitive gigabit market" to us, they
have cleaned up their driver and allow Jeff Garzik to co-maintain it
with them. Same story for NatSemi's gigabit parts for which Ben
Lahaise (another one of those "arrogant" Linux developers :-) is the
sole driver author.
So please, take your sob story and excuses elsewhere. The world is
crumbling from underneath you, and luckily the Linux community has a
solution to your attitude and ignorance in this situation. We have
our own driver and thus we can and will choose to ignore you. And
ignore you we will if you continue to act this way.
One of the things that Broadcom really, and truly, wants to protect is
their broken NICE extensions in their drivers. They want to retain
this so they can ship their proprietary VLAN trunking and load
balancing kernel module. Not only are the NICE extentions truly
disgusting, making use of them from binary-only Linux kernel modules
is of questionable legality.
Jeff and I have made numerous efforts to fix this relationship, both
publicly, directly with Broadcom, and more recently directly with
vendors who use Broadcom's parts. It is a real shame (and IMHO a PR
disaster for Broadcom, considering how other gigabit vendors interact
with us) that they can't figure out that they are fighting a losing
battle and that it is in their interest to work together and not
against us on these matters.