Now that I have 74GB raptors in both of my Linux boxes, I thought I would
compare throughput between FTP and NFS over a gigabit network.
I am using the same kernel versions and same motherboard on both machines
and even the same raptor hdd model.
Here are my results:
NFS, COPY 700MB FILE FROM 1 RAPTOR TO ANOTHER RAPTOR VIA GIGABIT ETHERNET:
$ cp file /remote/dst
0.02user 1.86system 0:38.07elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps
FTP, SAME
lftp> put file
733045488 bytes transferred in 10 seconds (67.38M/s)
What is wrong with NFS?
NFS options used: rw,bg,hard,intr,nfsvers=3
Is it doing some kind of weird caching?
I am using NFSv3 & XFS as the filesystem, any ideas?
I suppose I should try NFS with TCP, yes?
Thanks!
Justin.
On Mon, Jan 16, 2006 at 08:07:02PM -0500, Justin Piszcz wrote:
> I suppose I should try NFS with TCP, yes?
Precisely.
Phil
Also, some people mentioned tuning, I used 8192 as the w/r size it then
took 15 seconds, with 65535 it took 28 seconds.
I wonder how much faster NFS over TCP would be, or if NFS in the kernel is
the problem itself?
Will try later thanks.
FTP seems to be the winner for now:
<--- 226 8.927 seconds (measured here), 78.31 Mbytes per second
733045488 bytes transferred in 9 seconds (77.08M/s)
On Mon, 16 Jan 2006, Phil Oester wrote:
> On Mon, Jan 16, 2006 at 08:07:02PM -0500, Justin Piszcz wrote:
>> I suppose I should try NFS with TCP, yes?
>
> Precisely.
>
> Phil
>
NFS is still twice as slow as FTP, but best with a r/w size of 8192.
DEFAULT, NO OPTIONS
# mount p34:/ /nfs/p34
$ /usr/bin/time cp 700mb.img /p34/x/d
0.01user 1.64system 0:34.23elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps
TCP, NO CACHING
# mount p34:/ /nfs/p34 -o nfsvers=3,tcp,noac
$ /usr/bin/time cp 700mb.img /p34/x/a
0.02user 5.25system 0:58.43elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+197minor)pagefaults 0swaps
UDP, NO CACHING
# mount p34:/ /nfs/p34 -o nfsvers=3,noac
$ /usr/bin/time cp 700mb.img /p34/x/b
0.02user 5.54system 1:00.34elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps
UDP, NO CACHING (w/65535 r/w size)
# mount p34:/ /nfs/p34 -o nfsvers=3,noac,rsize=65535,wsize=65535
$ /usr/bin/time cp 700mb.img /p34/x/c
0.01user 5.75system 0:59.89elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps
# mount p34:/ /nfs/p34 -o nfsvers=3,rsize=8192,wsize=8192
0.04user 1.78system 0:14.16elapsed 12%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (0major+190minor)pagefaults 0swaps
UDP, NFSV3 + (w/8192 r/w size)
$ /usr/bin/time cp 700mb.img /p34/x/g
0.04user 1.78system 0:14.16elapsed 12%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (0major+190minor)pagefaults 0swaps
TCP, NFSV3 + (w/8192 r/w size)
0.03user 1.81system 0:14.98elapsed 12%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (0major+190minor)pagefaults 0swaps
UDP, NFSV3 + (w/16384 r/w size)
# mount p34:/ /nfs/p34 -o nfsvers=3,rsize=16834,wsize=16384
$ /usr/bin/time cp 700mb.img /p34/x/e
0.03user 1.75system 0:20.20elapsed 8%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+192minor)pagefaults 0swaps
UDP, NFSV3 + (w/32768 r/w size)
# mount p34:/ /nfs/p34 -o nfsvers=3,rsize=32768,wsize=32768
/usr/bin/time cp 700mb.img /p34/x/f
0.01user 1.59system 0:32.87elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps
On Mon, 16 Jan 2006, Phil Oester wrote:
> On Mon, Jan 16, 2006 at 08:07:02PM -0500, Justin Piszcz wrote:
>> I suppose I should try NFS with TCP, yes?
>
> Precisely.
>
> Phil
>
> NFS is still twice as slow as FTP, but best with a r/w size of 8192.
Screams for a kftpd ;)
Jan Engelhardt
--
| Alphagate Systems, http://alphagate.hopto.org/
| jengelh's site, http://jengelh.hopto.org/
On Mon, 16 Jan 2006, Justin Piszcz wrote:
> Also, some people mentioned tuning, I used 8192 as the w/r size it then took
> 15 seconds, with 65535 it took 28 seconds.
>
> I wonder how much faster NFS over TCP would be, or if NFS in the kernel is
> the problem itself?
On Linux NFS over TCP is slower than over UDP ~10%.
kloczek
--
-----------------------------------------------------------
*Ludzie nie maj? problem?w, tylko sobie sami je stwarzaj?*
-----------------------------------------------------------
Tomasz K?oczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
On Maw, 2006-01-17 at 18:48 +0100, Tomasz Kłoczko wrote:
> > I wonder how much faster NFS over TCP would be, or if NFS in the kernel is
> > the problem itself?
>
> On Linux NFS over TCP is slower than over UDP ~10%.
For the specific case you measured. Its never quite that simple because
behaviour over different networks and error patterns varies a lot and
TCP can be a big win on loaded networks or under error conditions,
especially packet loss, where fragmentation losses kill throughput on
UDP.
Alan, is it normal for FTP to be 2x as fast as NFS?
With 100mbps, I never seemed to have any issues, but with GIGABIT I
definitely see all sorts of weird issues.
On Tue, 17 Jan 2006, Alan Cox wrote:
> On Maw, 2006-01-17 at 18:48 +0100, Tomasz Kłoczko wrote:
>>> I wonder how much faster NFS over TCP would be, or if NFS in the kernel is
>>> the problem itself?
>>
>> On Linux NFS over TCP is slower than over UDP ~10%.
>
> For the specific case you measured. Its never quite that simple because
> behaviour over different networks and error patterns varies a lot and
> TCP can be a big win on loaded networks or under error conditions,
> especially packet loss, where fragmentation losses kill throughput on
> UDP.
>
>
On Maw, 2006-01-17 at 13:24 -0500, Justin Piszcz wrote:
> Alan, is it normal for FTP to be 2x as fast as NFS?
> With 100mbps, I never seemed to have any issues, but with GIGABIT I
> definitely see all sorts of weird issues.
NFS performance is limited by the fact it is a file system so sees only
what the file system can tell it. It also takes a hit because it has
strict rules on committing data to disk before acknowledging it (so data
is not lost over a crash). That makes NFS a bigger user of CPU resources
and more disk dependant than FTP which simply throws the entire file
down the pipe when in binary mode, does no processing and makes no
guarantee about restarts or what hits disk
On Tue, 2006-01-17 at 13:24 -0500, Justin Piszcz wrote:
> Alan, is it normal for FTP to be 2x as fast as NFS?
> With 100mbps, I never seemed to have any issues, but with GIGABIT I
> definitely see all sorts of weird issues.
Reading or writing?
The readahead algorithm has been borken in 2.6.x for at least the past 6
months. It leads to NFS collapsing down to 4k reads on the wire instead
of doing 32k or 64k.
An effort was made to look at fixing this, but it turns out that nobody
really understands the current messy implementation, and so progress has
been slow.
Cheers,
Trond
Writing from SRC(A) -> DST(B).
I have not tested reading, but as I recall there were similar speed issues
going the other way too, although I have not tested it recently.
Justin.
On Tue, 17 Jan 2006, Trond Myklebust wrote:
> On Tue, 2006-01-17 at 13:24 -0500, Justin Piszcz wrote:
>> Alan, is it normal for FTP to be 2x as fast as NFS?
>> With 100mbps, I never seemed to have any issues, but with GIGABIT I
>> definitely see all sorts of weird issues.
>
> Reading or writing?
>
> The readahead algorithm has been borken in 2.6.x for at least the past 6
> months. It leads to NFS collapsing down to 4k reads on the wire instead
> of doing 32k or 64k.
> An effort was made to look at fixing this, but it turns out that nobody
> really understands the current messy implementation, and so progress has
> been slow.
>
> Cheers,
> Trond
>
On Tue, 2006-01-17 at 13:38 -0500, Justin Piszcz wrote:
> Writing from SRC(A) -> DST(B).
> I have not tested reading, but as I recall there were similar speed issues
> going the other way too, although I have not tested it recently.
How were you testing it? I'm not sure that ftp will actually sync your
file to disk (whereas that is pretty much mandatory for an NFS server),
so unless you are transferring very large files, you would expect to see
a speed difference due to caching of writes by the server.
Cheers,
Trond
Did you get my other e-mail?
$ cp file /nfs/destination
$ lftp> put file
On Tue, 17 Jan 2006, Trond Myklebust wrote:
> On Tue, 2006-01-17 at 13:38 -0500, Justin Piszcz wrote:
>> Writing from SRC(A) -> DST(B).
>> I have not tested reading, but as I recall there were similar speed issues
>> going the other way too, although I have not tested it recently.
>
> How were you testing it? I'm not sure that ftp will actually sync your
> file to disk (whereas that is pretty much mandatory for an NFS server),
> so unless you are transferring very large files, you would expect to see
> a speed difference due to caching of writes by the server.
>
> Cheers,
> Trond
>
On Tue, 2006-01-17 at 13:55 -0500, Justin Piszcz wrote:
> Did you get my other e-mail?
>
> $ cp file /nfs/destination
> $ lftp> put file
Yes, but how big a file is this? Is it significantly larger than the
amount of cache memory on the server? As I said, if ftp is failing to
sync the file to disk, then you may be comparing apples and oranges.
Cheers,
Trond
> On Tue, 17 Jan 2006, Trond Myklebust wrote:
>
> > On Tue, 2006-01-17 at 13:38 -0500, Justin Piszcz wrote:
> >> Writing from SRC(A) -> DST(B).
> >> I have not tested reading, but as I recall there were similar speed issues
> >> going the other way too, although I have not tested it recently.
> >
> > How were you testing it? I'm not sure that ftp will actually sync your
> > file to disk (whereas that is pretty much mandatory for an NFS server),
> > so unless you are transferring very large files, you would expect to see
> > a speed difference due to caching of writes by the server.
> >
> > Cheers,
> > Trond
> >
The file is 700MB.
Machine A (src) has 2GB of RAM / 2GB of swap
Machine B (dst) has 1GB of RAM / 2GB of swap.
Justin.
On Tue, 17 Jan 2006, Trond Myklebust wrote:
> On Tue, 2006-01-17 at 13:55 -0500, Justin Piszcz wrote:
>> Did you get my other e-mail?
>>
>> $ cp file /nfs/destination
>> $ lftp> put file
>
>
> Yes, but how big a file is this? Is it significantly larger than the
> amount of cache memory on the server? As I said, if ftp is failing to
> sync the file to disk, then you may be comparing apples and oranges.
>
> Cheers,
> Trond
>
>> On Tue, 17 Jan 2006, Trond Myklebust wrote:
>>
>>> On Tue, 2006-01-17 at 13:38 -0500, Justin Piszcz wrote:
>>>> Writing from SRC(A) -> DST(B).
>>>> I have not tested reading, but as I recall there were similar speed issues
>>>> going the other way too, although I have not tested it recently.
>>>
>>> How were you testing it? I'm not sure that ftp will actually sync your
>>> file to disk (whereas that is pretty much mandatory for an NFS server),
>>> so unless you are transferring very large files, you would expect to see
>>> a speed difference due to caching of writes by the server.
>>>
>>> Cheers,
>>> Trond
>>>
>
>> Did you get my other e-mail?
>>
>> $ cp file /nfs/destination
>> $ lftp> put file
>
>
>Yes, but how big a file is this? Is it significantly larger than the
>amount of cache memory on the server? As I said, if ftp is failing to
>sync the file to disk, then you may be comparing apples and oranges.
Ok, so what happens if you use NFS with the async option, does it go a
little faster?
Jan Engelhardt
--
auto Can be mounted with the -a option.
defaults
Use default options: rw, suid, dev, exec, auto,
nouser, and async.
The default is async, no?
On Tue, 17 Jan 2006, Jan Engelhardt wrote:
>>> Did you get my other e-mail?
>>>
>>> $ cp file /nfs/destination
>>> $ lftp> put file
>>
>>
>> Yes, but how big a file is this? Is it significantly larger than the
>> amount of cache memory on the server? As I said, if ftp is failing to
>> sync the file to disk, then you may be comparing apples and oranges.
>
>
> Ok, so what happens if you use NFS with the async option, does it go a
> little faster?
>
>
>
> Jan Engelhardt
> --
>
> auto Can be mounted with the -a option.
>
> defaults
> Use default options: rw, suid, dev, exec, auto,
> nouser, and async.
>
> The default is async, no?
The server side also needs to specify async in exports. You even get a
warning if you do not specify sync or async, because the default had
been changed once.
Jan Engelhardt
--
On Tue, 2006-01-17 at 23:07 +0100, Jan Engelhardt wrote:
> > auto Can be mounted with the -a option.
> >
> > defaults
> > Use default options: rw, suid, dev, exec, auto,
> > nouser, and async.
> >
> > The default is async, no?
>
> The server side also needs to specify async in exports. You even get a
> warning if you do not specify sync or async, because the default had
> been changed once.
>
What is the date on the above man page? Looks like the docs need to be
updated.
I believe the default was originally async, which violates the NFS spec
and is dangerous, and changed to sync at some point.
Lee
man mount
On Tue, 17 Jan 2006, Lee Revell wrote:
> On Tue, 2006-01-17 at 23:07 +0100, Jan Engelhardt wrote:
>>> auto Can be mounted with the -a option.
>>>
>>> defaults
>>> Use default options: rw, suid, dev, exec, auto,
>>> nouser, and async.
>>>
>>> The default is async, no?
>>
>> The server side also needs to specify async in exports. You even get a
>> warning if you do not specify sync or async, because the default had
>> been changed once.
>>
>
> What is the date on the above man page? Looks like the docs need to be
> updated.
>
> I believe the default was originally async, which violates the NFS spec
> and is dangerous, and changed to sync at some point.
>
> Lee
>
On Tue, 2006-01-17 at 18:19 -0500, Justin Piszcz wrote:
> man mount
>
async is the default for most filesystems but the NFS standard requires
writes to be synchronous.
Lee
On Tue, 2006-01-17 at 18:39 -0500, Lee Revell wrote:
> On Tue, 2006-01-17 at 18:19 -0500, Justin Piszcz wrote:
> > man mount
> >
>
> async is the default for most filesystems but the NFS standard requires
> writes to be synchronous.
On the server side, note. Not the client side. Justin appears to be
looking at the client, whereas you are referring to an export option on
the server.
The client only guarantees that writes must have been committed to disk
on the server when either fsync() or close() have been called.
Cheers,
Trond