I have a system that I've been running some NFS benchmarks on. I've also
been running the benchmarks on a Netapp F880. I have been using iozone
as the testing tool. I am quite impressed with the linux system in every
regard excepting it's close() / nfs commit performance (iozone -Rac).
The following URL contains some graphs and spreadsheet data.
http://www.wanfear.com/~eroper/nfs/
Both the netapp and the linux system take a significant performance hit
when close() is involved. However the Linux system drops to an almost
unbelievably pathetic level.
The graphs may also be of interest as they compare various block sizes,
transport protos, sync/async options on the same linux server. I also
have data for tcp,sync,32768 block sizes although it hasn't made it up
there yet. All nfs data is v3.
I'm hoping someone can tell me why there is such a huge write
performance drop when close() is involved, and why it is such a more
substantial drop than experienced with the netapp. I've also got data
for two local iozone runs on the filesystem both with and without the
close() option in iozone.
Just in case anyone is interested in the specs (mordor == linux server):
2.4.20-18.7-nfsserv (redhat(7.3) kernel with tcp-nfs built in)
Pentium III 1.0ghz
1gb ram
Intel(R) PRO/1000
3ware Storage Controller (Escalade 7500-8)(raid-5, 7 ata disks)
The filesystem is ext3
Thanks,
Edward
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
ed-
read the FAQ: http://nfs.sourceforge.net/
close(2) on an NFS file means flush all outstanding
writes. benchmarking without the close() means you're
measuring essentially client memory bandwidth.
the filer doesn't need v3 COMMITs because of its NVRAM
facility -- whether the client asks for UNSTABLE or
FILE_SYNC writes, it will always return a FILE_SYNC
result. so writes to a filer are always fast, and
v3 writes to a filer are simpler than a pure disk-based
server.
> -----Original Message-----
> From: Edward Roper [mailto:[email protected]]
> Sent: Monday, June 30, 2003 4:28 PM
> To: [email protected]
> Cc: [email protected]
> Subject: [NFS] close() / nfs commit performance
>=20
>=20
> I have a system that I've been running some NFS benchmarks=20
> on. I've also
> been running the benchmarks on a Netapp F880. I have been using iozone
> as the testing tool. I am quite impressed with the linux=20
> system in every
> regard excepting it's close() / nfs commit performance (iozone -Rac).
> The following URL contains some graphs and spreadsheet data.
>=20
> http://www.wanfear.com/~eroper/nfs/
>=20
> Both the netapp and the linux system take a significant=20
> performance hit
> when close() is involved. However the Linux system drops to an almost
> unbelievably pathetic level.
>=20
> The graphs may also be of interest as they compare various=20
> block sizes,
> transport protos, sync/async options on the same linux server. I also
> have data for tcp,sync,32768 block sizes although it hasn't made it up
> there yet. All nfs data is v3.
>=20
> I'm hoping someone can tell me why there is such a huge write
> performance drop when close() is involved, and why it is such a more
> substantial drop than experienced with the netapp. I've also got data
> for two local iozone runs on the filesystem both with and without the
> close() option in iozone.
>=20
> Just in case anyone is interested in the specs (mordor =3D=3D=20
> linux server):
>=20
> 2.4.20-18.7-nfsserv (redhat(7.3) kernel with tcp-nfs built in)
> Pentium III 1.0ghz
> 1gb ram
> Intel(R) PRO/1000
> 3ware Storage Controller (Escalade 7500-8)(raid-5, 7 ata disks)
> The filesystem is ext3
>=20
> Thanks,
> Edward
>=20
>=20
>=20
>=20
> -------------------------------------------------------
> This SF.Net email sponsored by: Free pre-built ASP.NET sites including
> Data Reports, E-commerce, Portals, and Forums are available now.
> Download today and enter to win an XBOX or Visual Studio .NET.
> http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_06
1203_01/01
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Fair enough, I've read the faq and understand your argument. However I
would really be curious as to why local write benchmarks (on the server
itself) with calling close() and using a test-file of 2GB (twice my ram)
and a record size of 8192 I obtain a write throughput of 24,739KB/Second
whereas over NFS I obtain only 4,924KB/Second on a 500mb file with 8192
record size?
I realize that plopping a pci-nvramish device (http://www.umem.com/) and
setting data=journal with my ext3 journal residing on the nvram would
indeed offer a performance increase. I don't see where it's going to be
a 5x increase however when I don't seem to be bottle necking at
direct-attach-disk-i/o.
Perhaps local close() calls don't actually commit to disk?
Watching the machines leds (albeit a very primitive method) doesn't show
constant harddisk action while I'm supposedly waiting on a physical disk
commit. I get a brief flash of activity once every 30 seconds or so
instead across the array.
I plan on purchasing the nvram but I'm just making sure it's really
going to make the difference that I'm looking for.
On Mon, 2003-06-30 at 15:39, Lever, Charles wrote:
> ed-
>
> read the FAQ: http://nfs.sourceforge.net/
>
> close(2) on an NFS file means flush all outstanding
> writes. benchmarking without the close() means you're
> measuring essentially client memory bandwidth.
>
> the filer doesn't need v3 COMMITs because of its NVRAM
> facility -- whether the client asks for UNSTABLE or
> FILE_SYNC writes, it will always return a FILE_SYNC
> result. so writes to a filer are always fast, and
> v3 writes to a filer are simpler than a pure disk-based
> server.
>
> > -----Original Message-----
> > From: Edward Roper [mailto:[email protected]]
> > Sent: Monday, June 30, 2003 4:28 PM
> > To: [email protected]
> > Cc: [email protected]
> > Subject: [NFS] close() / nfs commit performance
> >
> >
> > I have a system that I've been running some NFS benchmarks
> > on. I've also
> > been running the benchmarks on a Netapp F880. I have been using iozone
> > as the testing tool. I am quite impressed with the linux
> > system in every
> > regard excepting it's close() / nfs commit performance (iozone -Rac).
> > The following URL contains some graphs and spreadsheet data.
> >
> > http://www.wanfear.com/~eroper/nfs/
> >
> > Both the netapp and the linux system take a significant
> > performance hit
> > when close() is involved. However the Linux system drops to an almost
> > unbelievably pathetic level.
> >
> > The graphs may also be of interest as they compare various
> > block sizes,
> > transport protos, sync/async options on the same linux server. I also
> > have data for tcp,sync,32768 block sizes although it hasn't made it up
> > there yet. All nfs data is v3.
> >
> > I'm hoping someone can tell me why there is such a huge write
> > performance drop when close() is involved, and why it is such a more
> > substantial drop than experienced with the netapp. I've also got data
> > for two local iozone runs on the filesystem both with and without the
> > close() option in iozone.
> >
> > Just in case anyone is interested in the specs (mordor ==
> > linux server):
> >
> > 2.4.20-18.7-nfsserv (redhat(7.3) kernel with tcp-nfs built in)
> > Pentium III 1.0ghz
> > 1gb ram
> > Intel(R) PRO/1000
> > 3ware Storage Controller (Escalade 7500-8)(raid-5, 7 ata disks)
> > The filesystem is ext3
> >
> > Thanks,
> > Edward
> >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email sponsored by: Free pre-built ASP.NET sites including
> > Data Reports, E-commerce, Portals, and Forums are available now.
> > Download today and enter to win an XBOX or Visual Studio .NET.
> > http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_06
> 1203_01/01
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> " " == Edward Roper <[email protected]> writes:
> Perhaps local close() calls don't actually commit to disk?
man 2 close
(and read the NOTES)
and then
man 2 fsync
Cheers,
Trond
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
So in theory running nfs async is no more dangerous than using a local
app that calls close() without fsync()?
On Mon, 2003-06-30 at 17:34, Trond Myklebust wrote:
> >>>>> " " == Edward Roper <[email protected]> writes:
>
> > Perhaps local close() calls don't actually commit to disk?
>
> man 2 close
> (and read the NOTES)
>
> and then
>
> man 2 fsync
>
> Cheers,
> Trond
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Alright so rerunning iozone with the flush option which calls fsync() my
new number on the 2GB file is: 16,797kB/second vs. the 4,924kB/second on
the Linux server and the 31,757kB/second on the netapp. I'm still not
understanding where the flush performance hit to the tune of
~12,000kB/second is coming from? I would assume that with nvram I would
have a greater fsync() local performance than the 16,797kB/second?
On Mon, 2003-06-30 at 17:34, Trond Myklebust wrote:
> man 2 fsync
>
> Cheers,
> Trond
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
And one last number: data=journal with a ramdisk (trd) journal device
leads to only 8,782KB/Second. (via tcp nfs 500mb filesize 8192 record
size) vs. 4,924kB/Second. Quite an improvement, but still not
16,797kB/Second.
On Tue, 2003-07-01 at 12:55, Edward Roper wrote:
> Alright so rerunning iozone with the flush option which calls fsync() my
> new number on the 2GB file is: 16,797kB/second vs. the 4,924kB/second on
> the Linux server and the 31,757kB/second on the netapp. I'm still not
> understanding where the flush performance hit to the tune of
> ~12,000kB/second is coming from? I would assume that with nvram I would
> have a greater fsync() local performance than the 16,797kB/second?
>
> On Mon, 2003-06-30 at 17:34, Trond Myklebust wrote:
>
> > man 2 fsync
> >
> > Cheers,
> > Trond
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Yikes! Absolutely not! Its way dangerous, particularly since its not immediately
detectable that you've lost data if the server reboots. Of course, these
problems are
based on an complex if condition - if data has been written across the wire, and
if the
client has received the async response, and if the server reboots while the data is
still in its own caches - but all you need is one time, and you're toast.
Tom
Edward Roper wrote:
> So in theory running nfs async is no more dangerous than using a local
> app that calls close() without fsync()?
>
> On Mon, 2003-06-30 at 17:34, Trond Myklebust wrote:
>
>>>>>>>" " == Edward Roper <[email protected]> writes:
>>>>>>>
>> > Perhaps local close() calls don't actually commit to disk?
>>
>>man 2 close
>> (and read the NOTES)
>>
>>and then
>>
>>man 2 fsync
>>
>>Cheers,
>> Trond
>>
>
>
>
>
> -------------------------------------------------------
> This SF.Net email sponsored by: Free pre-built ASP.NET sites including
> Data Reports, E-commerce, Portals, and Forums are available now.
> Download today and enter to win an XBOX or Visual Studio .NET.
> http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
>
>
--
Tom McNeal
(650)906-0761(cell)
(650)964-8459(fax)
Email: [email protected]
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs