Our tech support person told me her applications running over NFS
were really slow.
We use a copy of each client's data and their installed program
version for testing and support issues. The data can be updated using
a simple zip file from the client.
This is all stored on an NFS server for support's use. Server and
client are both on the same gigabit switch. The LAN she is connected
to has only gigabit connections on the switch. We're using TCP anyway
for everything (including NFS over a PTP T1: the T1 does go through
the same switch).
Just FYI, I tried both UDP and TCP, and it seems that TCP was
slightly faster. However, I was using a system I couldn't shut down,
etc., so I didn't remove caching effects well.
However, what changed things was removing sync from the mount. I
remounted the directory that stores the client systems without async
instead of sync (it is a subdirectory of a directory already mounted
sync).
The test that I tried (which requires user input, so exact times
aren't very useful) went from several minutes to several seconds! I
know I don't have many hard facts here, but the difference was so
dramatic I didn't care to be exact. With async, the experience was
about the same as running it off a copy on the local disk.
This solves my problem. I don't see a problem with using async for
this purpose. If a file gets corrupted, it doesn't really matter,
since it is a test system that can be restored at any time. It might
confuse the support person, but hopefully not. The systems don't crash
very often. And that's part of the environment: to be able to restore
the data after testing to its initial state, so it is easy to deal
with if that starts happening. Time will tell.
The server file system is EXT3 on a couple of mirrored SCSI disks.
hdparm reports about 80 M/sec. I'm not using data=journal: the mounts
uses defaults (except for nocheck). Linux (Gentoo kernel) 2.6.11-r6.
Everything was done with NFS v3.
I'm reporting this because it is very disturbing. I didn't expect
sync to have that huge an effect on write times. Fortunately, most of
the files we export with NFS are rarely written, mostly read. The
change to sync was fairly recent, which explains why noone complained
previously.
Is there a technical reason why sync has to be so incredibly slow?
Is the write really as slow as it appears to be? I do know that when I
did all this the server was not being used heavily at all, because
everyone was at lunch (except for one person to answer the phones,
who doesn't use that server much -- and there are very few users on
that LAN). I find it hard to believe it actually took that much time
to do the writes. The amount of data written was perhaps 100 M.
-------------------------------------------------------
This SF.Net email is sponsored by Yahoo.
Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
Search APIs Find out how you can build Yahoo! directly into your own
Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> We use a copy of each client's data and their installed program
> version for testing and support issues. The data can be updated using
> a simple zip file from the client.
> This is all stored on an NFS server for support's use. Server and
> client are both on the same gigabit switch. The LAN she is connected
> to has only gigabit connections on the switch. We're using TCP anyway
> for everything (including NFS over a PTP T1: the T1 does go through
> the same switch).
> Just FYI, I tried both UDP and TCP, and it seems that TCP was
> slightly faster. However, I was using a system I couldn't shut down,
> etc., so I didn't remove caching effects well.
>=20
> However, what changed things was removing sync from the mount. I
> remounted the directory that stores the client systems without async
> instead of sync (it is a subdirectory of a directory already mounted
> sync).
to clarify, you are using the "sync" mount option on your NFS clients?
why did you initially decide to use "sync"?
without knowing more details about your situation, let me suggest you
might find something useful in the Linux NFS FAQ:
http://nfs.sourceforge.net/
-------------------------------------------------------
This SF.Net email is sponsored by Yahoo.
Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
Search APIs Find out how you can build Yahoo! directly into your own
Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Fri, May 27, 2005 at 07:45:12PM -0700, Lever, Charles wrote:
> to clarify, you are using the "sync" mount option on your NFS clients?
Yes.
> why did you initially decide to use "sync"?
Reading urgings from some of the NFS authors ...
Also, for a while we were using an app that had a shared data base
base on NFS store. Async concerned me in that scenario. That has also
ended.
>
> without knowing more details about your situation, let me suggest you
> might find something useful in the Linux NFS FAQ:
>
> http://nfs.sourceforge.net/
I don't understand. I've resolved the situation, as I said.
But I still don't understand why the sync option caused such a
dramatic slowdown. The actual part of the program that does the
writing went from several minutes to three seconds. If more details
or investigation are required to answer that question, I'd love to
know what they are so they can be provided.
-------------------------------------------------------
This SF.Net email is sponsored by Yahoo.
Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
Search APIs Find out how you can build Yahoo! directly into your own
Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> > why did you initially decide to use "sync"?
> Reading urgings from some of the NFS authors ...
> Also, for a while we were using an app that had a shared data base
> base on NFS store. Async concerned me in that scenario. That has also
> ended.
generally there is no need to use the "sync" mount option unless your
application depends on single system semantics (ie as soon as a write()
system call completes, all other instances of the application must be
able to see the new data). in most cases these days, applications are
smart enough to use "fsync" and "fflush" appropriately so that "sync"
isn't necessary.
> But I still don't understand why the sync option caused such a
> dramatic slowdown. The actual part of the program that does the
> writing went from several minutes to three seconds. If more details
> or investigation are required to answer that question, I'd love to
> know what they are so they can be provided.
the "sync" mount option on Linux restricts the behavior of the NFS
client.
1. like opening a file with O_SYNC, the client no longer caches writes.
it always pushes written data to the server before a write() system call
returns. normally the client allows the write() system call to return
before it has started to commit the data to lessen the write latency
experienced by applications.
2. unlike O_SYNC, however, the "sync" mount option also causes writes
to go over the wire in page-size chunks no matter how large the wsize
is.
3. also unlike O_SYNC, the "sync" mount option forces all application
write requests to be sent on the wire serially (ie one NFS write at a
time) and in byte order. the reason for this behavior is to ensure that
if a large write is interrupted, at least the first portion of the write
request is guaranteed to be on permanent storage. this makes it easier
to detect and recover from an interrupted write.
so normally, an application can make a 1MB write request via write(2)
and the client will return control immediately to the application and
push the written data to the server in the background by sending all NFS
write requests containing "wsize" (probably 32KB) bytes, all at once,
then waiting once for the result of the writes. with the "sync" mount
option, that 1MB write request will go to the server as 256 separate 4KB
write requests, each being sent one at a time.
if the latency of a write request is 3-4msec, the "sync" case will wait
about 1 second for that 1MB write to complete. the normal case will
return immediately to the application with only a small latency.
-------------------------------------------------------
This SF.Net email is sponsored by Yahoo.
Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
Search APIs Find out how you can build Yahoo! directly into your own
Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
lau den 28.05.2005 Klokka 05:53 (-0700) skreiv George Garvey:
> On Fri, May 27, 2005 at 07:45:12PM -0700, Lever, Charles wrote:
> > to clarify, you are using the "sync" mount option on your NFS clients?
> Yes.
> > why did you initially decide to use "sync"?
> Reading urgings from some of the NFS authors ...
Note that there are two "sync" options: one affects the server behaviour
causing it to always make sure that the data reaches the disk if/when
the client requests it (see 'man 5 exports') and the other affects the
client (and is a mount option).
You almost always want to set the "sync" option in /etc/exports (unless
you really don't care about losing data if the server crashes and/or
reboots). Are you sure that the urgings you mentioned above didn't
concern this option?
Conversely, you almost never want to set the sync mount option on the
client. Any application that needs synchronous read/write behaviour and
that doesn't want to use fcntl() or flock() locking will usually want to
be rewritten to use the O_DIRECT mode for uncached file access.
Cheers,
Trond
-------------------------------------------------------
This SF.Net email is sponsored by Yahoo.
Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
Search APIs Find out how you can build Yahoo! directly into your own
Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Sat, May 28, 2005 at 01:51:59PM -0700, Trond Myklebust wrote:
> Note that there are two "sync" options: one affects the server behaviour
> causing it to always make sure that the data reaches the disk if/when
> the client requests it (see 'man 5 exports') and the other affects the
> client (and is a mount option).
Thanks. I'm an idiot. I need to recheck everything based on this
misunderstanding. I may have made some other mistakes.
I believed I saw some people talking about changing the mount
default to sync from async. That may be bad memory, but helped my
misconception ;)
-------------------------------------------------------
This SF.Net email is sponsored by Yahoo.
Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
Search APIs Find out how you can build Yahoo! directly into your own
Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Sat, May 28, 2005 at 09:13:54AM -0700, Lever, Charles wrote:
> generally there is no need to use the "sync" mount option unless your
Thanks. Between your and Trond's responses, I have a better
understanding and a misconception to look at.
-------------------------------------------------------
This SF.Net email is sponsored by Yahoo.
Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
Search APIs Find out how you can build Yahoo! directly into your own
Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs