Hi there,
I'm running two servers, both Dual Opterons with Broadcom Gb NICs. I'm
using the bcm5700 driver and I see no errors whatsoever with 'ifconfig'.
I'm using nfs-utils-1.0.6 and the 2.6.6 kernel, compiled with both NFSv3
and NFSv4 support. The client and servers are identical except for the
server having a scsi-raid under /home and the client being and IDE-box.
Distribution slackware-9.1 on both computers.
/etc/exports;
/home 193.167.32.175(ro,no_root_squash,async)
/etc/fstab;
193.167.32.187:/home /home nfs \
defaults,noauto,rsize=32768,wsize=32768,nfsvers=3 0 0
(as you can see I have experimented with different rsize,wsize without any
noticeable effect, same goes for nfsvers 2 and 3. I even mounted the
filesystem as ext2 on the server and as ext3 with data=journal with no
effect)
The computers are connected through a HP 8-port Gb switch and are on the
same subnet.
rsync over ssh gives me roughly 200Mbps (37 GB dataset)
netcat over tcp with a 2GB file gives me 457 Mbps
netcat over udp with a 2GB file gives me 640 Mbps
But dd'ing the 2GB file over nfs as some NFS-HOWTO suggests takes over 5
minutes. That should equal to something around 49 Mbps, correct?
The netcat takes 25 secs to transfer 2GB file.
Is there some way I can look at the nfs-server what it's doing? Any
suggestions on solutions for this?
And what datarates over NFS could I expect on a setup like this?
Half-wirespeed maybe?
Thanks in advance,
Harald
--
A: Top Posters
Q: What is the most annoying thing on mailing lists?
Harald H Hannelius | harald/a\arcada.fi | GSM +358 50 594 1020
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
[2.6 over loopback nfs]
> real 6m21.086s
> user 0m1.712s
> sys 0m5.454s
[2.4 over loopback nfs]
> real 1m11.875s
> user 0m1.730s
> sys 0m7.730s
This roughly matches some performance issues I've been seeing on a 2.6.5
kernel. Are you using reiser or ext3?
In general, dd to a local file is much different from what NFS does,
because nfsd flushes data to disk.
The NFS client will usually issue a COMMIT call roughly every 1 Meg,
so the following should mimic the behavior of nfsd a little better,
when run on your file system locally
iozone -s 1g -r 1m -o -i 0
Higher numbers indicate better performance.
Could you run this on 2.4 and 2.6, please?
Olaf
--
Olaf Kirch | The Hardware Gods hate me.
[email protected] |
---------------+
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Hello,
I have been watching this thread with interest. I
have a 2.6.6-mm5 kernel NFS server in production and
have noticed some significant slow down to that server
(in comparison to an old crusty 2.4.22 kernel and
NetApp). I too would like to understand the issue
here so I have run iozone as indicated to give another
point of interest. Note, that during one of the tests
(nfsv2,udp,8192, the server crawled to a halt, load
sky rocketed to 30 with zero load (100% available
reported by top), at this point the server refused
access via ssh/rsh and refused to export any volumes,
but ping-able. At terminal, everything behavied
normal.
Prior to the lock up, here are the numbers I was
getting:
Client 2.6.6-m5
2.6.6-mm5:
File size set to 1048576 KB
Record Size 1024 KB
SYNC Mode.
Command line used:
/var/tmp/iozone/src/current/iozone -s 1g -r 1m -o -i 0
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
UDP, NFSv2, sync, r/w 8192
1048576 1024 39889 38745
UDP, NFSv3, sync, r/w 32768
1048576 1024 12226 10749
TCP, NFSv3, sync, r/w 32768
1048576 1024 11909 12208
2.4.22
UDP, NFSv3, sync, r/w 32768
1048576 1024 11259 10761
UDP, NFSv3, sync, r/w 8192
1048576 1024 6136 6266
I cc'ed the linux-kernel mailing list as well.
Cheers,
Phy
--- Olaf Kirch <[email protected]> wrote:
> [2.6 over loopback nfs]
> > real 6m21.086s
> > user 0m1.712s
> > sys 0m5.454s
>
> [2.4 over loopback nfs]
> > real 1m11.875s
> > user 0m1.730s
> > sys 0m7.730s
>
> This roughly matches some performance issues I've
> been seeing on a 2.6.5
> kernel. Are you using reiser or ext3?
>
> In general, dd to a local file is much different
> from what NFS does,
> because nfsd flushes data to disk.
>
> The NFS client will usually issue a COMMIT call
> roughly every 1 Meg,
> so the following should mimic the behavior of nfsd a
> little better,
> when run on your file system locally
>
> iozone -s 1g -r 1m -o -i 0
>
> Higher numbers indicate better performance.
>
> Could you run this on 2.4 and 2.6, please?
>
> Olaf
> --
> Olaf Kirch | The Hardware Gods hate me.
> [email protected] |
> ---------------+
>
>
>
-------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the
> market... Oracle 10g.
> Take an Oracle 10g class now, and we'll give you the
> exam FREE.
>
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
__________________________________
Do you Yahoo!?
Friends. Fun. Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Fri, 28 May 2004, Olaf Kirch wrote:
> [2.6 over loopback nfs]
> > real 6m21.086s
> > user 0m1.712s
> > sys 0m5.454s
>
> [2.4 over loopback nfs]
> > real 1m11.875s
> > user 0m1.730s
> > sys 0m7.730s
>
> This roughly matches some performance issues I've been seeing on a 2.6.5
> kernel. Are you using reiser or ext3?
ext3.
> In general, dd to a local file is much different from what NFS does,
> because nfsd flushes data to disk.
I guess (as a total don't-know) that nfsd reads and writes files in a
different way than local processes do. I read somewhere something about
read-ahead but that was in 2.6.0-pre-something..
> The NFS client will usually issue a COMMIT call roughly every 1 Meg,
> so the following should mimic the behavior of nfsd a little better,
> when run on your file system locally
>
> iozone -s 1g -r 1m -o -i 0
>
> Higher numbers indicate better performance.
>
> Could you run this on 2.4 and 2.6, please?
Oh sorry, the server went into production to day so I can't run the tests
on 2.6 without rebooting first. Not a good idea right now :)
But here are (useless without 2.6 ditto) results with kernel 2.4.26 run
locally on the fileserver;
http://people.arcada.fi/~harald/nfs/iozone-1MBcommit.txt
http://people.arcada.fi/~harald/nfs/iozone-1MBcommit2.txt
--
A: Top Posters
Q: What is the most annoying thing on mailing lists?
Harald H Hannelius | harald/a\arcada.fi | GSM +358 50 594 1020
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
P=E5 fr , 28/05/2004 klokka 14:03, skreiv Harald Hannelius:
> I guess (as a total don't-know) that nfsd reads and writes files in a
> different way than local processes do. I read somewhere something about
> read-ahead but that was in 2.6.0-pre-something..
Hmm... The current readahead code does a load of crap in order to
determine whether or not the user is doing linear or random I/O. It may
be that the nfsd server with all its multi-threading and out-of-sequence
I/O is triggering the random I/O readahead code (which of course aims to
limit readahead).
Cheers
Trond
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, 27 May 2004, Vincent ROQUETA wrote:
> Le Jeudi 27 Mai 2004 12:31, Harald Hannelius a =E9crit :
> > Is there some way I can look at the nfs-server what it's doing? Any
> > suggestions on solutions for this?
> >
> > And what datarates over NFS could I expect on a setup like this?
> > Half-wirespeed maybe?
> on NFSv3, 1 ethernet 1GB I reached 89% of the avalaible brandwith for NF=
Sv3,
> kernel 2.6.3, for file less than the total amount of ram (4Go), and 22% o=
f
> the brandwith for file over 6Go, because of the use of harddisk use in th=
e
> copy process (swap).
>
> What is your memory configuration?
I took a look at "top" while reading the file on the client;
It looks like it's sitting there waiting for io. Something for the
kernel-people maybe?
15:15:10 up 1 day, 19:31, 4 users, load average: 7.55, 4.31, 3.54
93 processes: 92 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 0.0% user 1.2% system 0.0% nice 95.1% iowait 3.0%
idle
CPU1 states: 0.0% user 1.2% system 0.0% nice 90.1% iowait 8.1%
idle
Mem: 1035184k av, 1029660k used, 5524k free, 0k shrd, 18216k
buff
52712k active, 929460k inactive
Swap: 987956k av, 0k used, 987956k free 957788k
cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
4473 root 16 0 4008 1084 3832 R 0.3 0.1 0:00 1 top
8 root 5 -10 0 0 0 SW< 0.1 0.0 0:15 0
kblockd/0
9 root 5 -10 0 0 0 SW< 0.1 0.0 0:14 1
kblockd/1
4382 root 15 0 0 0 0 SW 0.1 0.0 0:01 0 nfsd
4384 root 15 0 0 0 0 SW 0.1 0.0 0:01 0 nfsd
4386 root 15 0 0 0 0 DW 0.1 0.0 0:01 1 nfsd
4387 root 15 0 0 0 0 SW 0.1 0.0 0:01 0 nfsd
4392 root 15 0 0 0 0 DW 0.1 0.0 0:01 0 nfsd
4394 root 15 0 0 0 0 DW 0.1 0.0 0:00 1 nfsd
4395 root 15 0 0 0 0 SW 0.1 0.0 0:00 1 nfsd
4396 root 15 0 0 0 0 DW 0.1 0.0 0:01 0 nfsd
1 root 16 0 488 240 464 S 0.0 0.0 0:05 1 init
--=20
A: Top Posters
Q: What is the most annoying thing on mailing lists?
Harald H Hannelius | harald/a\arcada.fi | GSM +358 50 594 1020
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
How are the GB NICs attached? Onboard? Integrated into Southbridge? PCI?
PCI-X?
If it's plain old PCI, you won't get more, PCI maxes out here.
On Thu, 27 May 2004 13:31:57 +0300 (EEST)
Harald Hannelius <[email protected]> wrote:
>
> Hi there,
>
> I'm running two servers, both Dual Opterons with Broadcom Gb NICs. I'm
> using the bcm5700 driver and I see no errors whatsoever with 'ifconfig'.
>
> I'm using nfs-utils-1.0.6 and the 2.6.6 kernel, compiled with both NFSv3
> and NFSv4 support. The client and servers are identical except for the
> server having a scsi-raid under /home and the client being and IDE-box.
>
> Distribution slackware-9.1 on both computers.
>
> /etc/exports;
>
> /home 193.167.32.175(ro,no_root_squash,async)
>
> /etc/fstab;
>
> 193.167.32.187:/home /home nfs \
> defaults,noauto,rsize=32768,wsize=32768,nfsvers=3 0 0
>
> (as you can see I have experimented with different rsize,wsize without any
> noticeable effect, same goes for nfsvers 2 and 3. I even mounted the
> filesystem as ext2 on the server and as ext3 with data=journal with no
> effect)
>
> The computers are connected through a HP 8-port Gb switch and are on the
> same subnet.
>
> rsync over ssh gives me roughly 200Mbps (37 GB dataset)
> netcat over tcp with a 2GB file gives me 457 Mbps
> netcat over udp with a 2GB file gives me 640 Mbps
>
> But dd'ing the 2GB file over nfs as some NFS-HOWTO suggests takes over 5
> minutes. That should equal to something around 49 Mbps, correct?
>
> The netcat takes 25 secs to transfer 2GB file.
>
> Is there some way I can look at the nfs-server what it's doing? Any
> suggestions on solutions for this?
>
> And what datarates over NFS could I expect on a setup like this?
> Half-wirespeed maybe?
>
> Thanks in advance,
>
> Harald
>
>
> --
> A: Top Posters
> Q: What is the most annoying thing on mailing lists?
>
> Harald H Hannelius | harald/a\arcada.fi | GSM +358 50 594 1020
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
>
--
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C+++(++++) UL+>++++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS++(+) PE(-) Y+ PGP(-) t++(---)@ 5 X+(++) R+(++) tv--(+)@
b+(+++) DI+++ D G++ e* h>++ r%>* y?
------END GEEK CODE BLOCK------
http://www.againsttcpa.com - nothing fights like the opposition
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Le Jeudi 27 Mai 2004 14:17, Harald Hannelius a =E9crit :
> On Thu, 27 May 2004, Vincent ROQUETA wrote:
> > Le Jeudi 27 Mai 2004 12:31, Harald Hannelius a =E9crit :
> > > Is there some way I can look at the nfs-server what it's doing? Any
> > > suggestions on solutions for this?
> > >
> > > And what datarates over NFS could I expect on a setup like this?
> > > Half-wirespeed maybe?
> >
> > on NFSv3, 1 ethernet 1GB I reached 89% of the avalaible brandwith for
> > NFSv3, kernel 2.6.3, for file less than the total amount of ram (4Go),
> > and 22% of the brandwith for file over 6Go, because of the use of
> > harddisk use in the copy process (swap).
> >
> > What is your memory configuration?
>
> I took a look at "top" while reading the file on the client;
>
> It looks like it's sitting there waiting for io. Something for the
> kernel-people maybe?
>
> 15:15:10 up 1 day, 19:31, 4 users, load average: 7.55, 4.31, 3.54
> 93 processes: 92 sleeping, 1 running, 0 zombie, 0 stopped
> CPU0 states: 0.0% user 1.2% system 0.0% nice 95.1% iowait 3.0%
> idle
> CPU1 states: 0.0% user 1.2% system 0.0% nice 90.1% iowait 8.1%
> idle
> Mem: 1035184k av, 1029660k used, 5524k free, 0k shrd, 18216k
> buff
> 52712k active, 929460k inactive
> Swap: 987956k av, 0k used, 987956k free 957788k
> cached
>
> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
> 4473 root 16 0 4008 1084 3832 R 0.3 0.1 0:00 1 top
> 8 root 5 -10 0 0 0 SW< 0.1 0.0 0:15 0
> kblockd/0
> 9 root 5 -10 0 0 0 SW< 0.1 0.0 0:14 1
> kblockd/1
> 4382 root 15 0 0 0 0 SW 0.1 0.0 0:01 0 nfsd
> 4384 root 15 0 0 0 0 SW 0.1 0.0 0:01 0 nfsd
> 4386 root 15 0 0 0 0 DW 0.1 0.0 0:01 1 nfsd
> 4387 root 15 0 0 0 0 SW 0.1 0.0 0:01 0 nfsd
> 4392 root 15 0 0 0 0 DW 0.1 0.0 0:01 0 nfsd
> 4394 root 15 0 0 0 0 DW 0.1 0.0 0:00 1 nfsd
> 4395 root 15 0 0 0 0 SW 0.1 0.0 0:00 1 nfsd
> 4396 root 15 0 0 0 0 DW 0.1 0.0 0:01 0 nfsd
> 1 root 16 0 488 240 464 S 0.0 0.0 0:05 1 init
That's normal : waiting for IO -> probably waiting for reading something fr=
om=20
the network. I am trying to do the same experience.
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, 27 May 2004, Dexter Filmore wrote:
> How are the GB NICs attached? Onboard? Integrated into Southbridge? PCI?
> PCI-X?
It's a Tyan K8S PRo, dual tigon 3 NICs onboard.
eth0: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:100MHz:64-bit)
10/100/1000BaseT Ethernet 00:e0:81:28:cf:89
> If it's plain old PCI, you won't get more, PCI maxes out here.
Oh, and why do netcat transfer the 2GB file in 25 secs? With the same NIC
and network ofcourse.. As stated in my original message.
I tried linux-2.4.26 kernel instead. Now the 'time cat file > /dev/null'
takes 1 min 3s on the client which is more like I wanted it. That should
equal to something around 254Mbps..
Since I don't have any real reason for a 2.6 kernel right now (32bit
distro) I will rest my case here. Just thought someone would like to
know..
Cheers, and many thanks
Harald
--
A: Top Posters
Q: What is the most annoying thing on mailing lists?
Harald H Hannelius | harald/a\arcada.fi | GSM +358 50 594 1020
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Hello,
>
> Distribution slackware-9.1 on both computers.
>
> /etc/exports;
>
> /home 193.167.32.175(ro,no_root_squash,async)
>
> /etc/fstab;
>
> 193.167.32.187:/home /home nfs \
> defaults,noauto,rsize=32768,wsize=32768,nfsvers=3 0 0
>
if you read the list archive, you will see that Trond always suggests to use
tcp in situations like this. Some weeks ago there was a discussion on LKML
about 2.6., udp and nfs, the final statement was, that one always should use
tcp with 2.6. kernel versions (I still don't understand why, but my knowledge
about network protocol design and kernel internal networking is probably too
much limited to understand the reason ;) ).
Hope it helps,
Bernd
P=E5 to , 27/05/2004 klokka 14:07, skreiv Bernd Schubert:
> if you read the list archive, you will see that Trond always suggests to =
use=20
> tcp in situations like this. Some weeks ago there was a discussion on LKM=
L=20
> about 2.6., udp and nfs, the final statement was, that one always should =
use=20
> tcp with 2.6. kernel versions (I still don't understand why, but my knowl=
edge=20
> about network protocol design and kernel internal networking is probably =
too=20
> much limited to understand the reason ;) ).
It has all to do with the network protocols. It's nothing we can fix in
the kernel.
See the discussion in FAQ number B10 on http://nfs.sourceforge.net
see also Casper's discussion on the same issue on
http://www.netsys.com/sunmgr/1997-09/msg00070.html
Cheers,
Trond
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, 27 May 2004, Bernd Schubert wrote:
> if you read the list archive, you will see that Trond always suggests to use
> tcp in situations like this. Some weeks ago there was a discussion on LKML
I did, same results.
--
A: Top Posters
Q: What is the most annoying thing on mailing lists?
Harald H Hannelius | harald/a\arcada.fi | GSM +358 50 594 1020
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
P=E5 to , 27/05/2004 klokka 06:31, skreiv Harald Hannelius:
> But dd'ing the 2GB file over nfs as some NFS-HOWTO suggests takes over 5
> minutes. That should equal to something around 49 Mbps, correct?
So what is the speed when dding the file locally on the server? (that is
*not* the same as rsync)
What if you then do the same thing on the server via a loopback mount on
the server (i.e. mount localhost:/home /mnt)?
Finally, if you do "nfsstat", what sort of values are you getting for
"retrans"?
Cheers,
Trond
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, 27 May 2004, Trond Myklebust wrote:
> P=E5 to , 27/05/2004 klokka 06:31, skreiv Harald Hannelius:
>
> > But dd'ing the 2GB file over nfs as some NFS-HOWTO suggests takes over =
5
> > minutes. That should equal to something around 49 Mbps, correct?
>
> So what is the speed when dding the file locally on the server? (that is
> *not* the same as rsync)
:)
Kernel 2.4.26:
# time dd if=3D/dev/zero of=3Dlarge bs=3D1M count=3D2000
2000+0 records in
2000+0 records out
real 1m11.565s
user 0m0.010s
sys 0m10.520s
# time dd if=3Dlarge of=3D/dev/null
4096000+0 records in
4096000+0 records out
real 0m23.506s
user 0m1.360s
sys 0m6.820s
From=20now on all tests with kernel 2.6.6;
# time dd if=3Dlarge of=3D/dev/null
4096000+0 records in
4096000+0 records out
real 0m21.190s
user 0m1.622s
sys 0m5.192s
> What if you then do the same thing on the server via a loopback mount on
> the server (i.e. mount localhost:/home /mnt)?
While doing the dd;
10:07:25 up 8 min, 2 users, load average: 9.87, 6.16, 2.60
And timings for the dd-command;
# time dd if=3Dlarge of=3D/dev/null
4096000+0 records in
4096000+0 records out
real 6m21.086s
user 0m1.712s
sys 0m5.454s
Just for the heck of it I rebooted into 2.4.26 and did the same test over
nfs on localhost;
# time dd if=3Dlarge of=3D/dev/null
4096000+0 records in
4096000+0 records out
real 1m11.875s
user 0m1.730s
sys 0m7.730s
> Finally, if you do "nfsstat", what sort of values are you getting for
> "retrans"?
(2.6.6)
# nfsstat
Server rpc stats:
calls badcalls badauth badclnt xdrcall
48018 0 0 0 0
Server nfs v2:
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
Server nfs v3:
null getattr setattr lookup access readlink
0 0% 9 0% 0 0% 1 0% 10 0% 0 0%
read write create mkdir symlink mknod
47995 99% 0 0% 0 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 2 0%
fsstat fsinfo pathconf commit
0 0% 1 0% 0 0% 0 0%
Client rpc stats:
calls retrans authrefrsh
47656 609 0
Client nfs v2:
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 9 0% 0 0% 1 0% 10 0% 0 0%
read write create mkdir symlink mknod
47633 99% 0 0% 0 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 2 0%
fsstat fsinfo pathconf commit
0 0% 1 0% 0 0% 0 0%
And here's output from vmstat while doing the dd (2.6.6);
# vmstat 1
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us sy
wa id
1 15 0 5452 1548 1009476 0 0 3350 4 910 1186 0 2
55 42
0 8 0 6452 1556 1008312 0 0 6104 0 2583 4576 1 2
94 3
0 12 0 4876 1552 1009948 0 0 5252 0 2395 3915 0 2
98 1
0 8 0 5388 1536 1009488 0 0 5132 0 2404 4099 0 2
89 10
0 12 0 6564 1536 1008264 0 0 4856 0 2310 3737 0 2
96 3
0 17 0 4556 1544 1010364 0 0 4364 72 2157 3128 0 1
98 0
0 7 0 5772 1552 1009132 0 0 4880 4 2325 3639 0 2
97 1
0 5 0 6220 1536 1008808 0 0 5316 0 2377 4361 1 2
94 4
0 2 0 5708 1528 1009156 0 0 6748 0 2756 4942 0 3
93 4
0 6 0 6388 1536 1008468 0 0 4784 0 2269 3458 0 2
92 7
0 16 0 5196 1504 1009724 0 0 5928 0 2563 4368 1 2
96 2
0 3 0 6284 1508 1008632 0 0 5136 8 2384 4121 0 3
95 2
0 2 0 6452 1488 1008380 0 0 5288 4 2400 4311 1 1
92 6
0 6 0 4812 1492 1010144 0 0 5256 0 2428 3806 0 2
96 2
0 7 0 5260 1472 1009756 0 0 5244 0 2379 4180 0 2
98 1
0 8 0 5580 1456 1009500 0 0 5332 0 2418 3858 0 2
98 1
0 6 0 5004 1464 1009968 0 0 4524 8 2238 3263 0 1
95 3
>
> Cheers,
> Trond
>
--=20
A: Top Posters
Q: What is the most annoying thing on mailing lists?
Harald H Hannelius | harald/a\arcada.fi | GSM +358 50 594 1020
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> > I guess (as a total don't-know) that nfsd reads and writes files in a
> > different way than local processes do. I read somewhere something about
> > read-ahead but that was in 2.6.0-pre-something..
>
> Hmm... The current readahead code does a load of crap in order to
> determine whether or not the user is doing linear or random I/O. It may
> be that the nfsd server with all its multi-threading and out-of-sequence
> I/O is triggering the random I/O readahead code (which of course aims to
> limit readahead).
Any thoughts on this patch posted to lkml
http://marc.theaimsgroup.com/?l=linux-kernel&m=108603045809565&w=2
Regards, /yg
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
P=E5 m=E5 , 31/05/2004 klokka 17:16, skreiv Yusuf Goolamabbas:
> > > I guess (as a total don't-know) that nfsd reads and writes files in a
> > > different way than local processes do. I read somewhere something abo=
ut
> > > read-ahead but that was in 2.6.0-pre-something..
> >=20
> > Hmm... The current readahead code does a load of crap in order to
> > determine whether or not the user is doing linear or random I/O. It may
> > be that the nfsd server with all its multi-threading and out-of-sequenc=
e
> > I/O is triggering the random I/O readahead code (which of course aims t=
o
> > limit readahead).
>=20
> Any thoughts on this patch posted to lkml=20
>=20
> http://marc.theaimsgroup.com/?l=3Dlinux-kernel&m=3D108603045809565&w=3D2
May or may not fix a bug in the NFSd code. Does nothing to change the
basic readahead code in the VFS layer in order to fix the issues I
raised above.
Cheers,
Trond
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
I haven't been following this thread closely, but figured I'd chime in
with my own experience with this. Here's a rather unscientific test,
dd'ing to a file on an NFS-mounted filesystem. Mount options are:
udp,soft,intr,nfsvers=3
(I'm using UDP as I'm trying to set up a HA-NFS server, and the clients
seem to recover much faster when using UDP as a transport).
With a 2.4 kernel server:
% time dd if=/dev/zero of=./testfile bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes transferred in 11.806235 seconds (8881544 bytes/sec)
dd if=/dev/zero of=./testfile bs=100M count=1 0.00s user 0.28s system
2% cpu 12.147 total
With a 2.6 kernel server:
% time dd if=/dev/zero of=./testfile bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes transferred in 66.997572 seconds (1565096 bytes/sec)
dd if=/dev/zero of=./testfile bs=100M count=1 0.00s user 0.29s system
0% cpu 1:07.06 total
The machines are different hardware, but local write performance is
pretty comparable (in fact the 2.6 box is a faster machine, and is
currently less utilized than the 2.4 kernel machine). Both are using
reiserfs as the underlying filesystem.
Write performance in this cursory test was 10x worse! Clearly, there's
some sort of problem with NFS on 2.6. I'll be happy to send in what info
I can. My 2.4 machine is currently a production box, but I can run an
instrumented kernel, etc. on the 2.6 box in the near future if anyone
here can guide me on what I can do to help.
-- Jeff
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
On Wed, Jun 02, 2004 at 08:33:53AM -0400, Jeffrey Layton wrote:
> The machines are different hardware, but local write performance is
> pretty comparable (in fact the 2.6 box is a faster machine, and is
> currently less utilized than the 2.4 kernel machine). Both are using
> reiserfs as the underlying filesystem.
What is the speed on each server for a local dd from /dev/zero to disk?
What actual kernel versions are you running?
> Write performance in this cursory test was 10x worse!
This completely fails to correlate with my experience. I'm using XFS
on 2.6.4 and I see single thread streaming write performance increase
by about 25% going from 2.4 to 2.6. Other metrics show improvements
as good or better, except the scalability of UDP reads tops out lower.
Greg.
--
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
Le Mercredi 02 Juin 2004 14:33, Jeffrey Layton a =C3=A9crit :
> I haven't been following this thread closely, but figured I'd chime in
> with my own experience with this. Here's a rather unscientific test,
> dd'ing to a file on an NFS-mounted filesystem. Mount options are:
>
> udp,soft,intr,nfsvers=3D3
>
> (I'm using UDP as I'm trying to set up a HA-NFS server, and the clients
> seem to recover much faster when using UDP as a transport).
>
> With a 2.4 kernel server:
>
> % time dd if=3D/dev/zero of=3D./testfile bs=3D100M count=3D1
> 1+0 records in
> 1+0 records out
> 104857600 bytes transferred in 11.806235 seconds (8881544 bytes/sec)
> dd if=3D/dev/zero of=3D./testfile bs=3D100M count=3D1 0.00s user 0.28s s=
ystem
> 2% cpu 12.147 total
>
> With a 2.6 kernel server:
>
> % time dd if=3D/dev/zero of=3D./testfile bs=3D100M count=3D1
> 1+0 records in
> 1+0 records out
> 104857600 bytes transferred in 66.997572 seconds (1565096 bytes/sec)
> dd if=3D/dev/zero of=3D./testfile bs=3D100M count=3D1 0.00s user 0.29s s=
ystem
> 0% cpu 1:07.06 total
>
>
> The machines are different hardware, but local write performance is
> pretty comparable (in fact the 2.6 box is a faster machine, and is
> currently less utilized than the 2.4 kernel machine). Both are using
> reiserfs as the underlying filesystem.
>
> Write performance in this cursory test was 10x worse! Clearly, there's
> some sort of problem with NFS on 2.6. I'll be happy to send in what info
> I can. My 2.4 machine is currently a production box, but I can run an
> instrumented kernel, etc. on the 2.6 box in the near future if anyone
> here can guide me on what I can do to help.
>
> -- Jeff
>
What appends with linux 2.6.6?
Is NFSv4 enabled?
Can you use bonnie / IOzone and repport activity?
(-> http://nfsv4.bullopensource.org/tools/sondage.php for performances too=
ls=20
)
=20
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
On Wed, 2004-06-02 at 09:00, Greg Banks wrote:
> On Wed, Jun 02, 2004 at 08:33:53AM -0400, Jeffrey Layton wrote:
> > The machines are different hardware, but local write performance is
> > pretty comparable (in fact the 2.6 box is a faster machine, and is
> > currently less utilized than the 2.4 kernel machine). Both are using
> > reiserfs as the underlying filesystem.
>
> What is the speed on each server for a local dd from /dev/zero to disk?
> What actual kernel versions are you running?
>
I need to amend my statement above. I'm having a problem mounting the
filesystem from the 2.6 kernel as v3. I keep getting a message saying
that v3 isn't supported, even though I can mount using v3 from the 2.4
box without an issue. Another 2.6 box doesn't seem to have this issue,
so I'm unsure of what the deal is. Rebuilding my client's kernel to see
if I can clear up that problem. Once I get that redone, I'll redo this
test and see if the poor performance remains...
In any case...
Less than .2 secs to do a local dd to disk:
laytonj@angstrom:/services/NFS/home% time dd if=/dev/zero of=./testfile
bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes transferred in 0.200756 seconds (522313514 bytes/sec)
dd if=/dev/zero of=./testfile bs=100M count=1 0.00s user 0.20s system
107% cpu 0.186 total
It's a stock 2.6.6 kernel (the 'a' is an internal versioning scheme I
use when I have multiple kernel builds of same version):
laytonj@angstrom:/services/NFS/home% uname -a
Linux angstrom 2.6.6a #1 Mon May 17 14:29:24 EDT 2004 i686 GNU/Linux
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
Ok, it looks like the problem here was that I already had the filesystem
mounted on another directory as a NFSv2 filesystem. When I unmounted
this filesystem, and remounted it as v3 only, I started getting
performance roughly on par with the 2.4 kernels. Sorry for the false
alarm!
Cheers,
-- Jeff
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.