Hi,
I wonder if someone knows why files open with O_SYNC option on an NFS
mounted drive don't get synchronized? Is it an open issue?
The directory is both exported and mounted using sync option.
Thanks in advance.
Giga
>>>>> " " == Gregory Giguashvili <[email protected]> writes:
> Hi, I wonder if someone knows why files open with O_SYNC option
> on an NFS mounted drive don't get synchronized? Is it an open
> issue? The directory is both exported and mounted using sync
> option.
You'll have to ask RedHat. 2.4.18-3 is *not* a stock Linux kernel.
Cheers,
Trond
>>>>> " " == Gregory Giguashvili <[email protected]> writes:
>> You'll have to ask RedHat. 2.4.18-3 is *not* a stock Linux
>> kernel.
> What about 2.4.7-10 (Default for RH7.2)? I was referring to the
> latest version I have...
Same story. RedHat/Suse/... all patch their kernels with extra stuff
that often is not included in any of Marcelo/Linus' "standard"
kernels. The latter are found on ftp.kernel.org and mirrors...
Cheers,
Trond
>You'll have to ask RedHat. 2.4.18-3 is *not* a stock Linux kernel.
What about 2.4.7-10 (Default for RH7.2)? I was referring to the latest
version I have...
Thanks a lot.
Giga
>The latter are found on ftp.kernel.org and mirrors...
Are you saying that this problem is non-existent in the latest kernel
patches? Can you, please, point to at least one of them?
Thanks in advance,
Giga
>>>>> " " == Gregory Giguashvili <[email protected]> writes:
>> The latter are found on ftp.kernel.org and mirrors...
> Are you saying that this problem is non-existent in the latest
> kernel patches? Can you, please, point to at least one of them?
No. I said that reporting bugs on RedHat-specific kernels should be
done to RedHat rather than to this list.
As for O_SYNC, AFAICS there are no bugs in the standard
kernel. However if you are thinking that O_SYNC means that you can
write simultaneously to the same file from 2 different clients, then
the answer is "the NFS protocol won't allow you to do that".
The *only* method of ensuring cache consistency in such a case is to
use POSIX file locking.
Cheers,
Trond
>>>>> " " == Gregory Giguashvili <[email protected]> writes:
> In what I see, a simple test doesn't work in the expected way,
> which is one client writes to a file opened with O_SYNC on a
> drive mounted with sync option and the other client cannot
> immediatelly see the written data. Are you saying that this is
> the way it should be?
Yes. That is all the NFS protocol allows you to do.
Cheers,
Trond
>However if you are thinking that O_SYNC means that you can
>write simultaneously to the same file from 2 different clients, then
>the answer is "the NFS protocol won't allow you to do that".
>The *only* method of ensuring cache consistency in such a case is to
>use POSIX file locking.
I thought that O_SYNC opens the file for synchronous I/O by blocking writes
till the data is written to the disk. In case of NFS mounted disk, this
should work only if "sync" option is present both in export and mount
options, shouldn't it?
In what I see, a simple test doesn't work in the expected way, which is one
client writes to a file opened with O_SYNC on a drive mounted with sync
option and the other client cannot immediatelly see the written data.
Are you saying that this is the way it should be?
Thanks a lot.
Giga
>
> > In what I see, a simple test doesn't work in the expected way,
> > which is one client writes to a file opened with O_SYNC on a
> > drive mounted with sync option and the other client cannot
> > immediatelly see the written data. Are you saying that this is
> > the way it should be?
>
>Yes. That is all the NFS protocol allows you to do.
Well, this is the way it's been working on all UN*X platforms I know. In
fact, we came across this problem with NFS clients being unable to
synchronize on Linux.
If this is a "feature", it makes it impossible to read/write files via NFS
because of the risk of corruption - or am I missing something here?
Also, integrating Linux into heterogeneous networks becomes impossible
(though, it doesn't work in Linux only network as well).
Best,
Giga
>>>>> " " == Gregory Giguashvili <[email protected]> writes:
> Well, this is the way it's been working on all UN*X platforms I
> know. In fact, we came across this problem with NFS clients
> being unable to synchronize on Linux.
Sorry. AFAICS O_SYNC only affects writes (just like we do).
As for reads: all commercial NFS clients I know of will check data
cache consistency on open(). They will assume that they can cache
attributes and data as per the documentation in 'man 5 nfs' (although
you can turn this off by using the 'noac' mount option).
Furthermore, even with 'noac' they *all* have problems with races in
the sort of scenario you describe because there is no atomic
GETATTR+READ operation.
Bottom line: If you want the sort of data cache consistency you are
describing, you *have* to use file locking.
Cheers,
Trond
Gregory Giguashvili <[email protected]>
> > > In what I see, a simple test doesn't work in the expected way,
> > > which is one client writes to a file opened with O_SYNC on a
> > > drive mounted with sync option and the other client cannot
> > > immediatelly see the written data. Are you saying that this is
> > > the way it should be?
> >
> >Yes. That is all the NFS protocol allows you to do.
>
> Well, this is the way it's been working on all UN*X platforms I know. In
> fact, we came across this problem with NFS clients being unable to
> synchronize on Linux.
> If this is a "feature", it makes it impossible to read/write files via NFS
> because of the risk of corruption - or am I missing something here?
>
> Also, integrating Linux into heterogeneous networks becomes impossible
> (though, it doesn't work in Linux only network as well).
This in neither a "feature" nor an error. It is the way it was designed
to work. There are VERY few distributed file systems, and NFS is NOT
one of them.
It is intended that a file server support one writer/reader to one client
for a file.
A server may support multiple clients, multiple files, BUT the same file
MUST NOT be used by multiple clients at the same time.
If you want examples of an attempt to support this, look at the gyrations
necessary for mail handling via NFS.
Only file level locking is permitted. You can only open the file IF you get
the lock. And be sure to close the file before releasing the lock. Make sure
that all accesses to the file use the SAME lock technique.
Integrating Linux is possible. You just have to follow the rules. I'm working
in a heterogeneous mix now - Solaris (file server, and one workstation), Linux
(several workstations), and SGI. It works. The hardest part was dropping
the NFS mail (couldn't agree on file locking for all systems/applications).
We are also migrating from an SGI based network (different net) to a Linux
net. A mixed environment does have some surprises (uid for mail, application
lock functions). Basic user access to data works fine though. We just don't
mix multiple client access to the same file, unless everybody is only reading
that file.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
>Furthermore, even with 'noac' they *all* have problems with races in
>the sort of scenario you describe because there is no atomic
>GETATTR+READ operation.
>
>Bottom line: If you want the sort of data cache consistency you are
>describing, you *have* to use file locking.
File locking, meaning lockd? There are so many problems with file locking in
heterogeneous environments that we were moving towards dropping its usage.
Instead, we planned to use some home grown TCP based lock server mechanism.
I understand that locking file flushes NFS cache, isn't it? Why can't it be
flushed by O_SYNC and "sync" options presence? This would make the life much
easier for programmers...
This means that we will never be able to drop lockd locking and at the same
time achieve file consistency via NFS?
Best,
Giga
>Only file level locking is permitted. You can only open the
>file IF you get
>the lock. And be sure to close the file before releasing the
>lock. Make sure
>that all accesses to the file use the SAME lock technique.
You need a file descriptor to lock the file, don't you? That's assuming that
you intend to use lockd locking.
>We just don't mix multiple client access to the same file, unless everybody
>is only reading that file.
When the writer closes the file, how do you make the readers see the latest
changes (assuming that you always open/close files per transaction type).
Thanks a lot.
Giga
On Wed, Aug 07, 2002 at 06:24:23PM +0200, Trond Myklebust wrote:
> >>>>> " " == Gregory Giguashvili <[email protected]> writes:
>
> > Hi, I wonder if someone knows why files open with O_SYNC option
> > on an NFS mounted drive don't get synchronized? Is it an open
> > issue? The directory is both exported and mounted using sync
> > option.
>
> You'll have to ask RedHat. 2.4.18-3 is *not* a stock Linux kernel.
Too bad the NFS in -3 was stock.
-ben
--
"You will be reincarnated as a toad; and you will be much happier."
>>>>> " " == Gregory Giguashvili <[email protected]> writes:
> File locking, meaning lockd? There are so many problems with
> file locking in heterogeneous environments that we were moving
> towards dropping its usage. Instead, we planned to use some
> home grown TCP based lock server mechanism.
> I understand that locking file flushes NFS cache, isn't it? Why
> can't it be flushed by O_SYNC and "sync" options presence? This
> would make the life much easier for programmers...
Tough. The above is not part of the O_SYNC spec on any platform and I
have no intention of implementing it.
The sort of thing you would like to do will be possible with O_DIRECT
(which is a design to allow user programs to manage their own
caches). That has yet to be integrated into the standard kernel
though.
Cheers,
Trond
On Wed, 2002-08-07 at 20:24, Gregory Giguashvili wrote:
> I understand that locking file flushes NFS cache, isn't it? Why can't it be
> flushed by O_SYNC and "sync" options presence? This would make the life much
> easier for programmers...
>
> This means that we will never be able to drop lockd locking and at the same
> time achieve file consistency via NFS?
Welcome to NFS. You basically need file locking (lockf/posix - flock is
not portably applicable to NFS) and you may need to turn off attribute
caching to make O_APPEND work portably.
Its a pain, but NFS has no other way to infer your transaction
constraints. You need to lock for read and for write on all clients, a
client reading without locking may get stale data.
Personally I use my own little daemons to manage transactional data,
ones that understand my transaction rules
--------- Received message begins Here ---------
>
>
> >Furthermore, even with 'noac' they *all* have problems with races in
> >the sort of scenario you describe because there is no atomic
> >GETATTR+READ operation.
> >
> >Bottom line: If you want the sort of data cache consistency you are
> >describing, you *have* to use file locking.
> File locking, meaning lockd? There are so many problems with file locking in
> heterogeneous environments that we were moving towards dropping its usage.
> Instead, we planned to use some home grown TCP based lock server mechanism.
>
> I understand that locking file flushes NFS cache, isn't it? Why can't it be
> flushed by O_SYNC and "sync" options presence? This would make the life much
> easier for programmers...
>
> This means that we will never be able to drop lockd locking and at the same
> time achieve file consistency via NFS?
NO. no and no.
It depends on the protocol. NFS v2 does no locking whatsoever. lockd was
created to have some shared locks to allow the APPLICATION to flush buffers,
but I don't think the daemon could force any of the NFS daemons (nfsd/biod)
to flush anything.
NFS v3 allows better traffic formation (using TCP). But I don't remember
locking being included either. I think that is aimed at NFS v4.
None of the versions of NFS are race free. This is due to the requirement
that NFS be layered on top of a real file system. Anything that goes
directly to the underlying file system will always beat out a remote NFS
client.
There never was any such thing as "cache coherency" in NFS. And there won't
be - the overhead is way too high. Think- to lock a section of a file,
you first tell the local daemon. That local daemon must then contact the
remote file server. That file server must then contact EVERY client to verify
that a lock is not in the process of being established. And confirm that the
locked section just hasn't yet been flushed back to the server. Then the server
can tell the client the section is locked.
What happens when one of the clients is down....
How long do you wait to determine a client is down...
What happens to other clients while the client holding the lock is down...
What happens when the server goes down....
What happens when the down client comes back up....
What happens when the server comes back up....
How do you request all clients to re-acquire locks... (and in what order)
And remember... NFS is a stateless protocol. No additional information about
currently open files can be saved by the clients or the server (daemons, not
applications). This makes NFS recovery much faster, but is a complete failure
at lock support. The file handles that are passed out on file open are not
foolproof. A server failure may require the clients to dismount/remount/reopen
the file to get a new/updated file handle. And this is on each block access.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Trond Myklebust wrote:
>>>>>>" " == Gregory Giguashvili <[email protected]> writes:
>>>>>
>
> >> You'll have to ask RedHat. 2.4.18-3 is *not* a stock Linux
> >> kernel.
> > What about 2.4.7-10 (Default for RH7.2)? I was referring to the
> > latest version I have...
>
> Same story. RedHat/Suse/... all patch their kernels with extra stuff
> that often is not included in any of Marcelo/Linus' "standard"
> kernels. The latter are found on ftp.kernel.org and mirrors...
??? You tell us that this list is only made for official kernels?
In this case it would not flood my mailing box, no -ac, no -arca/-aa,
no -dj, no..., no ide-2.4, no experimental patches... NO DEVELOPMENT!
NOO. This list is also for bug report of RH/Debian/SuSE/connectiva... kernels.
The pourpose of lkml is not only the development but also to find the
stablest kernel. The errata of RH (and other) will help Marcelo and Alan
to find the right patches to include in the official kernels.
ciao
cate
On Thu, Aug 08, 2002 at 09:11:24AM +0200, Giacomo Catenazzi wrote:
> ??? You tell us that this list is only made for official kernels?
> In this case it would not flood my mailing box, no -ac, no -arca/-aa,
> no -dj, no..., no ide-2.4, no experimental patches... NO DEVELOPMENT!
at least, the kernel and patches you are talking about are *announced*
here, so that people have a clue about changes between versions.
> NOO. This list is also for bug report of RH/Debian/SuSE/connectiva...
> kernels.
do you remember one of these vendors announcing their kernels with their
changelog here ? I think that could be a good idea anyway, because that
would tell us what set of patches they found to be stable, but unfortunately,
they don't at the moment. So nobody knows what's in without pulling regularly
their new packages from their respective sites.
> The pourpose of lkml is not only the development but also to find the
> stablest kernel. The errata of RH (and other) will help Marcelo and Alan
> to find the right patches to include in the official kernels.
If you want to help, ask these vendors to post their errata here, associated
with the bug reports, so that anyone can benefit from it.
Cheers,
Willy
On 7 Aug 2002, at 15:48, Jesse Pollard wrote:
>
> There never was any such thing as "cache coherency" in NFS. And there won't
> be - the overhead is way too high. Think- to lock a section of a file,
> you first tell the local daemon. That local daemon must then contact the
> remote file server. That file server must then contact EVERY client to verify
> that a lock is not in the process of being established. And confirm that the
> locked section just hasn't yet been flushed back to the server. Then the server
> can tell the client the section is locked.
>
On VMS we call it "Distributed Lock Manager". The overhead is not
so high and it works well.
>
> What happens when one of the clients is down....
> How long do you wait to determine a client is down...
> What happens to other clients while the client holding the lock is down...
> What happens when the server goes down....
> What happens when the down client comes back up....
> What happens when the server comes back up....
> How do you request all clients to re-acquire locks... (and in what order)
>
To solve this problem you need the cluster votes/quorum technique.
> And remember... NFS is a stateless protocol. No additional information about
The only way to do it is the implementation of a real linux cluster
with it's distributed and shared disk access. NFS can never be a
replacement for a clusterd disk access.
Martin Brulisauer
On Thu, 2002-08-08 at 09:06, Willy Tarreau wrote:
> do you remember one of these vendors announcing their kernels with their
> changelog here ? I think that could be a good idea anyway, because that
If there is a general consensus that is a good idea then it can be
arranged.
> would tell us what set of patches they found to be stable, but unfortunately,
> they don't at the moment. So nobody knows what's in without pulling regularly
> their new packages from their respective sites.
The best place is kernelnewbies.org which has them all nicely broken out
> If you want to help, ask these vendors to post their errata here, associated
> with the bug reports, so that anyone can benefit from it.
This list goes to US citizens so posting some of the details raises DMCA
issues
>>>>> " " == Gregory Giguashvili <[email protected]> writes:
> When the writer closes the file, how do you make the readers
> see the latest changes (assuming that you always open/close
> files per transaction type).
There is a convention amongst NFS clients that each client will always
flush *all* changes to the server upon close(), and it will always
check the file attributes upon a call to open() (and if the mtime or
file size have changed, one flushes the page cache).
This suffices to guarantee file cache consistency *provided* that only
one client opens the file at any given time.
If locking is used, all changes are flushed to the server upon
taking/releasing a lock, and the page cache is guaranteed to get
flushed upon taking a lock.
Cheers,
Trond
--------- Received message begins Here ---------
>
> >>>>> " " == Gregory Giguashvili <[email protected]> writes:
>
> > When the writer closes the file, how do you make the readers
> > see the latest changes (assuming that you always open/close
> > files per transaction type).
>
> There is a convention amongst NFS clients that each client will always
> flush *all* changes to the server upon close(), and it will always
> check the file attributes upon a call to open() (and if the mtime or
> file size have changed, one flushes the page cache).
> This suffices to guarantee file cache consistency *provided* that only
> one client opens the file at any given time.
>
> If locking is used, all changes are flushed to the server upon
> taking/releasing a lock, and the page cache is guaranteed to get
> flushed upon taking a lock.
And remember- this depends entirely on the server working the way
lockd assumes. Some systems DON'T do this anyway, since locks are not
necessarily available except on the local system, and not on the server.
Even when they are available, race conditions will/can still void the
assumptions.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Dear All,
I wanted to thank you All for your patience and help. As I have triggered
this discussion, I would like to summarize it by listing my findings, so it
can help others having same/similar questions.
I want to mention that I was looking for a way to make the best out of NFS
with regard to reading/writing files while not breaking the consistency of
data. I wasn't implying to get rid of locking as some of you suggested (I
fully understand the stateless nature of NFS protocol), instead I was
looking for a mix (both application and OS options) that will make Linux
systems behave similarly to other UN*X colleagues. In my opinion, any
standard behavior would greatly affect the strategic decision of large
companies to move to Linux with all the benefits from this to all of us. I'm
sorry if I wasn't clear enough.
Most of the current UN*X systems provide you with the mentioned
functionality by using the strictest NFS mount options as default (sync,
noac, etc.). Linux, on the contrary, strives to achieve maximum performance
on the expense of data consistency, which is not the most expected behavior
(BTW, the performance is great anyway!).
OS options:
===========
- The directories should be exported with "sync" option, which is not
default.
- The directories should be NFS mounted using "sync" and "noac" options,
which are not default.
Application code (per read/write transaction):
==============================================
- lock_file (doesn't have to be a lockd lock)
- open_file (O_SYNC)
- read/write file
- close_file
- unlock_file
The above sequence works OK, but it's still problematic since it doesn't
ensure that the data is coming from the server instead of NFS cache. In one
of my previous discussions on this list, I was told to use the following
technique to flush NFS buffers. It turns out that acquiring write lock using
fcntl (F_SETLK) interface acts as NFS write barrier and flushes all the file
NFS buffers. So, the above sequence would result in the following one:
- lock_file (doesn't have to be a lockd lock)
- open_file (O_SYNC)
- if (read) fcntl (F_SETLK) (doing lock/unlock here to flush NFS buffers)
- read/write file
- if (write) fcntl (F_SETLK) (doing lock/unlock here to flush NFS buffers)
- close_file
- unlock_file
FCNTL Unexpected behavior (Possible BUG?):
==========================================
There is a catch when using fcntl system call. It appears not to flush NFS
buffers when _FILE_OFFSET_BITS=64 is defined. This is because F_SETLK is
redefined for 64 bit version to be equal to 13, which makes fcntl call
behave differently. Passing hard-coded 6 to the call would produce the
expected results (one can see all this in strace).
Summary:
========
Using the fcntl dirty trick I finally achieved the expected behavior. BTW, I
don't understand why open/close calls do not flush NFS buffers - this seems
like a natural thing to do.
Anyway, I'm looking forward to your comments on this. I might have arrived
to some wrong conclusions, which I hope you'll help me to correct.
Best regards,
Giga
Gregory Giguashvili <[email protected]> writes:
>> The above sequence works OK, but it's still problematic since it
>> doesn't ensure that the data is coming from the server instead of
>> NFS cache. In one of my previous discussions on this list, I was
>> told to use the following technique to flush NFS buffers. It turns
>> out that acquiring write lock using fcntl (F_SETLK) interface acts
>> as NFS write barrier and flushes all the file NFS buffers. So, the
>> above sequence would result in the following one:
>>
>> - lock_file (doesn't have to be a lockd lock)
>> - open_file (O_SYNC)
>> - if (read) fcntl (F_SETLK) (doing lock/unlock here to flush NFS buffers)
>> - read/write file
>> - if (write) fcntl (F_SETLK) (doing lock/unlock here to flush NFS buffers)
>> - close_file
>> - unlock_file
I have been working in a mixed Linux/Solaris environment and this is
what works for me. (>40 machines reading/writing/modifying the same
files at the same time for hundreds of iterations with no errors.)
- open()
To get a file descriptor that can be locked
- fcntl(F_SETLK)
To ensure that all other clients have written their data to the file
and to clear the client cache
- fchmod()
File attributes (most importantly file size) were cached when the
file was opened, these may change between that time and when the
lock is granted, a side effect of fchmod() is to refresh the file
attributes from the server
- lseek() / read() / write()
If appending, it is important to lseek(SEEK_END) at this point to
make sure you are at the true end of file
- close()
This will flush any data written and remove the lock
If anyone experiments with this with systems other than Linux/Solaris
I would be interested in knowing if it works there also.
--
+----------------------------------+----------------------------------+
| Daniel K. Forrest | Laboratory for Molecular and |
| [email protected] | Computational Genomics |
| (608)262-9479 | University of Wisconsin, Madison |
+----------------------------------+----------------------------------+