Hello,
We've noticed some interesting behavior regarding mmap file IO and were
wondering if anyone here had some clues as to what might be going on.
We have some applications that mmap() files into memory for the purpose
of updating some potentially scattered word-sized data values. These
apps were originally written on Solaris with Solaris NFS servers assumed
to be the data source; the Sun guys said that mmap would be much faster
than read/write and they were correct. However, now that we have a few
Linux NFS servers, we're seeing the opposite.
It takes, for example, 20-24 hours to update a 2GB file (there are on
the order of 1M words to update) via mmap, whereas reading in the whole
file, updating it in memory, and writing it out again takes about 30
minutes (limited by network speed in our case). This when the file
resides on the Linux server. On Solaris the mmap update runs on the
order of 15-20 minutes. Client in both cases is a Solaris 8 machine.
What we are speculating is that the Solaris NFS server is "cheating" by
keeping file state information between NFS reads and writes. (The
updates do occur in sequence.) Are there any heuristics or
optimizations done by the Linux NFS server that might help the
performance here?
All things, alas, are not equal in this comparison -- the Solaris server
in question is an 8 CPU Enterprise 4500 with 8GB of RAM, and the Linux
server is a single-processor P4 with 512MB of RAM. The Solaris server
is much busier than the Linux server, though (I know it cannot be
caching the whole file in memory). The Linux server is running Red
Hat's 2.4.18-14 kernel.
Anyone have any ideas as to why this is happening and what, if anything,
we can do to speed up the mmap IO when using the Linux server?
--
Matthew Mitchell
Systems Programmer/Administrator [email protected]
Geophysical Development Corporation phone 713 782 1234
1 Riverway Suite 2100, Houston, TX 77056 fax 713 782 1829
-------------------------------------------------------
This sf.net email is sponsored by:
With Great Power, Comes Great Responsibility
Learn to use your power at OSDN's High Performance Computing Channel
http://hpc.devchannel.org/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
I would love to see a packet trace in the absence of any known
problems...
> Hello,
>
> We've noticed some interesting behavior regarding mmap file IO and were
> wondering if anyone here had some clues as to what might be going on.
>
> We have some applications that mmap() files into memory for the purpose
> of updating some potentially scattered word-sized data values. These
> apps were originally written on Solaris with Solaris NFS servers assumed
> to be the data source; the Sun guys said that mmap would be much faster
> than read/write and they were correct. However, now that we have a few
> Linux NFS servers, we're seeing the opposite.
>
> It takes, for example, 20-24 hours to update a 2GB file (there are on
> the order of 1M words to update) via mmap, whereas reading in the whole
> file, updating it in memory, and writing it out again takes about 30
> minutes (limited by network speed in our case). This when the file
> resides on the Linux server. On Solaris the mmap update runs on the
> order of 15-20 minutes. Client in both cases is a Solaris 8 machine.
>
> What we are speculating is that the Solaris NFS server is "cheating" by
> keeping file state information between NFS reads and writes. (The
> updates do occur in sequence.) Are there any heuristics or
> optimizations done by the Linux NFS server that might help the
> performance here?
>
> All things, alas, are not equal in this comparison -- the Solaris server
> in question is an 8 CPU Enterprise 4500 with 8GB of RAM, and the Linux
> server is a single-processor P4 with 512MB of RAM. The Solaris server
> is much busier than the Linux server, though (I know it cannot be
> caching the whole file in memory). The Linux server is running Red
> Hat's 2.4.18-14 kernel.
>
> Anyone have any ideas as to why this is happening and what, if anything,
> we can do to speed up the mmap IO when using the Linux server?
>
> --
> Matthew Mitchell
> Systems Programmer/Administrator [email protected]
> Geophysical Development Corporation phone 713 782 1234
> 1 Riverway Suite 2100, Houston, TX 77056 fax 713 782 1829
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:
> With Great Power, Comes Great Responsibility
> Learn to use your power at OSDN's High Performance Computing Channel
> http://hpc.devchannel.org/
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This sf.net email is sponsored by:
With Great Power, Comes Great Responsibility
Learn to use your power at OSDN's High Performance Computing Channel
http://hpc.devchannel.org/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> " " == Matthew Mitchell <[email protected]> writes:
> values. These apps were originally written on Solaris with
> Solaris NFS servers assumed to be the data source; the Sun guys
> said that mmap would be much faster than read/write and they
> were correct. However, now that we have a few Linux NFS
> servers, we're seeing the opposite.
As long as the clients are still Solaris, then the only difference can
be the network, and the server performance.
Of the 2, the bigger 'generic' troublemaker tends to be the network.
Solaris clients always tend to prefer NFS over TCP since that tends to
be more reliable on poor networks than does UDP. Unfortunately, NFS
over TCP on the server side is a fairly recent addition to Linux: it
only just made it into the stable release 2 weeks ago (when 2.4.20 was
released). To the best of my knowledge, none of the RedHat kernels
support it yet.
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:
With Great Power, Comes Great Responsibility
Learn to use your power at OSDN's High Performance Computing Channel
http://hpc.devchannel.org/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
When we moved some nfs clients from 2.4.18 to 2.4.20, we saw some
performance issues that initially appeared to be an nfs problem.
However, it turned out that the upgrade apparently broke autonegotiation
between the NICs in the nodes and our network switches. This did not
turn up immediately in network testing with tools such as netperf,
because we still had good one-way bandwidth with the nodes set to full
duplex and the switches set to half. However, two-way traffic
immediately caused lots of problems.
I never trust autonegotiation, and turn it off whenever I can.
You may be experiencing a similar problem.
dbr
On Fri, 2002-12-13 at 16:09, Matthew Mitchell wrote:
> Hello,
>
> We've noticed some interesting behavior regarding mmap file IO and were
> wondering if anyone here had some clues as to what might be going on.
--
David B. Ritch
High Performance Technologies, Inc.
-------------------------------------------------------
This sf.net email is sponsored by:
With Great Power, Comes Great Responsibility
Learn to use your power at OSDN's High Performance Computing Channel
http://hpc.devchannel.org/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs