Something else,
if you are dd a 10mb file on the local machine that is all going to be =
in cache, so
the rate will be horribly incorrect.
Also these are gigabit copper cards correct? Gigabit copper cards only =
run=20
a max of 50MB/second, so getting 35MB/second over one of them is pretty =
good.
I am getting 1500cs/second and 6500ints/second at about 8MB/second to
the network. =20
I get 3600cs/second and 15000ints/second at about 20MB/second to the
network.
How fast of IO are you getting on the local machine?
Roger
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
I'm running 93MBytes/s on gigE copper (testing using a 1000M file).
(jumbo frames enabled)
eric
"Heflin, Roger A." wrote:
>
> Something else,
>
> if you are dd a 10mb file on the local machine that is all going to be in cache, so
> the rate will be horribly incorrect.
>
> Also these are gigabit copper cards correct? Gigabit copper cards only run
> a max of 50MB/second, so getting 35MB/second over one of them is pretty good.
>
> I am getting 1500cs/second and 6500ints/second at about 8MB/second to
> the network.
>
> I get 3600cs/second and 15000ints/second at about 20MB/second to the
> network.
>
> How fast of IO are you getting on the local machine?
>
> Roger
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This sf.net email is sponsored by: viaVerio will pay you up to
$1,000 for every account that you consolidate with us.
http://ad.doubleclick.net/clk;4749864;7604308;v?http://www.viaverio.com/
consolidator/osdn.cfm
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
After doing some reading I figured out my mistake.
Others had told me that it ran 4 pairs at 125Mhz each, and that it=20
worked just like 100BT (1 bit per clock cycle), apparently they are
sending 2bit per cycle so 2 bits x 125Mhz * 4 channels is=20
1Gbps, if they were only using 1 bit per cycle it would be
only 50MB/second.
What is your underlying setup to get the 93MB/second,
ie disks/controllers/cpus?
Roger
> -----Original Message-----
> From: Eric Whiting [SMTP:[email protected]]
> Sent: Tuesday, October 15, 2002 3:52 PM
> To: Heflin, Roger A.
> Cc: [email protected]
> Subject: Re: [NFS] huge number of intr/s on large nfs server
>=20
> I'm running 93MBytes/s on gigE copper (testing using a 1000M file).
> (jumbo frames enabled)
>=20
> eric
>=20
>=20
>=20
> "Heflin, Roger A." wrote:
> >=20
> > Something else,
> >=20
> > if you are dd a 10mb file on the local machine that is all =
going to be in cache, so
> > the rate will be horribly incorrect.
> >=20
> > Also these are gigabit copper cards correct? Gigabit copper =
cards only run
> > a max of 50MB/second, so getting 35MB/second over one of =
them is pretty good.
> >=20
> > I am getting 1500cs/second and 6500ints/second at about =
8MB/second to
> > the network.
> >=20
> > I get 3600cs/second and 15000ints/second at about =
20MB/second to the
> > network.
> >=20
> > How fast of IO are you getting on the local machine?
> >=20
> > Roger
> >=20
> > -------------------------------------------------------
> > This sf.net email is sponsored by:ThinkGeek
> > Welcome to geek heaven.
> > http://thinkgeek.com/sf
> > _______________________________________________
> > NFS maillist - [email protected]
> > https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This sf.net email is sponsored by: viaVerio will pay you up to
$1,000 for every account that you consolidate with us.
http://ad.doubleclick.net/clk;4749864;7604308;v?
http://www.viaverio.com/consolidator/osdn.cfm
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
"Heflin, Roger A." wrote:
>
> After doing some reading I figured out my mistake.
>
> Others had told me that it ran 4 pairs at 125Mhz each, and that it
> worked just like 100BT (1 bit per clock cycle), apparently they are
> sending 2bit per cycle so 2 bits x 125Mhz * 4 channels is
> 1Gbps, if they were only using 1 bit per cycle it would be
> only 50MB/second.
>
> What is your underlying setup to get the 93MB/second,
> ie disks/controllers/cpus?
Going OT here...
That 93MB/s is with an odd mix of systems:
1. AMD MP 2000+ dual
Tyan Motherboard
3G RAM
Netgear GA620
2. Intel Xeon 2.2G dual
? Motherboard
4G RAM
Motherboard Intel gigE
gigE nics are tied together with a crossover cable.
boxes were not idle, disks are slow, but ftp of a 1G file shows 93MB/s.
I'm sure others could do better -- I think I could do better. Actually
given my disks on those boxes I have to put to /dev/null to see the
93MB/s... Linux ide-raid list shows people with disks running 100+MB/s
-- my disks are not that fast...
eric
-------------------------------------------------------
This sf.net email is sponsored by: viaVerio will pay you up to
$1,000 for every account that you consolidate with us.
http://ad.doubleclick.net/clk;4749864;7604308;v?
http://www.viaverio.com/consolidator/osdn.cfm
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Hi Roger,
> if you are dd a 10mb file on the local machine that is all
> going to be in cache, so
> the rate will be horribly incorrect.
Didn't do that - did a 10mb DD from 30 simultaneous network clients. I also
tried 30 simultaneous local processes of 5GB dd each and got *much* better
local disk performance than NFS performance once again.
> Also these are gigabit copper cards correct? Gigabit
> copper cards only run
> a max of 50MB/second, so getting 35MB/second over one of
> them is pretty good.
They are copper - Syskonnect. Where did you get the idea that they only do
50MB/sec from? If they're gigabit, I expect them to be able to do 250MB/sec
full duplex.
> I am getting 1500cs/second and 6500ints/second at about
> 8MB/second to
> the network.
>
> I get 3600cs/second and 15000ints/second at about 20MB/second to the
> network.
>
> How fast of IO are you getting on the local machine?
If I do a large single DD, I see 140MB/sec local write. If I do multiple
local dd's, I see 105MB/sec write. I didn't measure the intr/s yet but will
do.
Eff
-------------------------------------------------------
This sf.net email is sponsored by: viaVerio will pay you up to
$1,000 for every account that you consolidate with us.
http://ad.doubleclick.net/clk;4749864;7604308;v?
http://www.viaverio.com/consolidator/osdn.cfm
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> I'm running 93MBytes/s on gigE copper (testing using a 1000M file).
> (jumbo frames enabled)
Thanks for the data point Eric. I thought gigE copper was able to do more.
What's your frame size/MTU? Out of curiosity, what card are you using?
Eff
-------------------------------------------------------
This sf.net email is sponsored by: viaVerio will pay you up to
$1,000 for every account that you consolidate with us.
http://ad.doubleclick.net/clk;4749864;7604308;v?
http://www.viaverio.com/consolidator/osdn.cfm
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> This system in question is a SuperMicro P4DL6 MB with dual Xeon 2.4's and
> 2.0GB of memory. One of the PCI-X slots/busses has a 3ware 7850 IDE RAID
> card that is *not* using its on card raid. Two more of the PCI-X
> slots/busses have Syskonnect SK-9821 copper gigabit cards. I'm using x8
IBM
> Deskstar 120GB drives configured via IBM's EVMS 1.2.0 as software RAID5.
> Distro is Debian 3.0 Woody stable. Kernel is 2.4.19 with all of Neil
Brown's
> latest KNFSD patches from his web site. Syskonnect and 3ware driver are
> those found in the 2.4.19 kernel.
Eff, do you happen to know the cpu utilization for the local mulitple dd
test? I am concerned that the evms_raid5d is eating up a lot of cpu, which
is fine for the local dd test, but may be too much when nfsd is running.
Andrew Theurer
-------------------------------------------------------
This sf.net email is sponsored by: viaVerio will pay you up to
$1,000 for every account that you consolidate with us.
http://ad.doubleclick.net/clk;4749864;7604308;v?
http://www.viaverio.com/consolidator/osdn.cfm
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Hi Andrew,
> Eff, do you happen to know the cpu utilization for the local mulitple dd
> test? I am concerned that the evms_raid5d is eating up a lot of
> cpu, which
> is fine for the local dd test, but may be too much when nfsd is running.
That's a really good question and a point I hadn't made. The EVMS Raid 5
process seems to stay in the 12-20% utilization range for both local and NFS
dd's. For local, dd consumes as much of the remaining cpu as it can get
(~80%) with an occasional interruption from kswapd-very steady and even. For
NFS, the nfsd's (16 of them) in total also consume as much of the remaining
cpu as they can get (~80%) BUT they fluctuate from 80% to 0% then back 80%
all the time. I think much of the performance loss is caused by these
"pulsations" because the box is just sitting around idle waiting for the
next pulse of nfsd activity.
I tried everything in this:
http://people.redhat.com/alikins/system_tuning.html
except playing "Crazy Train" by Ozzy Ozbourne and have as of yet not been
able to get the nfsd's to stop "pulsating". I think that would help things a
lot. If I need to play "Crazy Train" I can arrange that. :)
Thanks,
Eff
-------------------------------------------------------
This sf.net email is sponsored by: viaVerio will pay you up to
$1,000 for every account that you consolidate with us.
http://ad.doubleclick.net/clk;4749864;7604308;v?
http://www.viaverio.com/consolidator/osdn.cfm
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs