the alignment issues are different depending on which kernel you use.
in 2.6 there should be no alignment issues, you can do O_DIRECT I/O on
NFS files at any offset and alignment.
in 2.4, O_DIRECT is supported on NFS files after 2.4.22 ish. buffer and
offset alignment is restricted. at some point we may have changed the
behavior, but i'm not remembering when and which kernels, because we
backported some of the 2.6 behavior into the Red Hat kernels. i believe
it depends on what your server returns as its on-disk block size (NFSv3
PATHCONF).
-----Original Message-----
From: Edward Hibbert [mailto:[email protected]]
Sent: Monday, May 23, 2005 4:13 PM
To: [email protected]
Subject: RE: [NFS] Disabling the client cache - O_DIRECT?
And then there's O_DIRECT, which might do just what I want. Is
that working over NFS now?
I'm trying to use it but getting EINVAL back - can someone
explain whether O_DIRECT over NFS has the alignment restrictions that
O_DIRECT does in general?
-----Original Message-----
From: [email protected]
[mailto:[email protected]]On Behalf Of Edward Hibbert
Sent: 23 May 2005 15:43
To: [email protected]
Subject: [NFS] Disabling the client cache using file
locks
The NFS FAQ says:
For greater consistency among clients, applications can
use file locking, where a client purges file data when a file is locked,
and flushes changes back to the server before unlocking a file.
My experience with this is that to force a read to go to
the NFS server, I:
- lock an area in the file (I think any area works, it
doesn't need to overlap with what you want to read)
- read the data
- unlock it
When I want a write to go to the NFS server, I do the
same. Obviously if I'm reading and writing the same area, I can save on
the locks.
This produces a lot of lock and unlock operations, and I
find that:
- NFS servers aren't as good at handling a high volume
of these as they are at reads and writes.
- lockd/statd on the client may also be a serialisation
point (I have suggestive but not conclusive evidence).
So I'm looking for ways of reducing the number of locks.
Possibilities:
- Make a patch to the kernel (or NFS utils?) to
_entirely_ disable client-side caching of file contents over NFS. I
have dedicated boxes which have no requirement to do client-side caching
at all so this is a serious option. Any code pointers?
- Use nolock to prevent the locks hitting the NFS server
- but I think that would still hit lockd/statd on the client?
- Not to bother with the locks round the writes, but use
fsync instead. Do folk think that would work?
Any comments or other ideas?
Edward.