offset and alignment of memory buffer and byte offset in the file
shouldn't matter for NFS files. in 2.6.9, there is a 16MB per request
size limit on systems with 4KB pages.
_____
From: Edward Hibbert [mailto:[email protected]]
Sent: Monday, May 23, 2005 4:57 PM
To: Lever, Charles
Cc: [email protected]
Subject: RE: [NFS] Disabling the client cache - O_DIRECT?
I'm most interested in a 2.6 kernel (2.6.9).
You mention the offset and alignment (of the memory buffer, I
presume). Are there any restrictions on the length of data?
I have a simple test program, which seems to show that I need to
use 512-byte alignment on a 2.6.9 kernel, but given what you say I'll
need to look into it more carefully.
Edward.
-----Original Message-----
From: Lever, Charles [mailto:[email protected]]
Sent: 23 May 2005 21:47
To: Edward Hibbert
Cc: [email protected]
Subject: RE: [NFS] Disabling the client cache -
O_DIRECT?
the alignment issues are different depending on which
kernel you use.
in 2.6 there should be no alignment issues, you can do
O_DIRECT I/O on NFS files at any offset and alignment.
in 2.4, O_DIRECT is supported on NFS files after 2.4.22
ish. buffer and offset alignment is restricted. at some point we may
have changed the behavior, but i'm not remembering when and which
kernels, because we backported some of the 2.6 behavior into the Red Hat
kernels. i believe it depends on what your server returns as its
on-disk block size (NFSv3 PATHCONF).
-----Original Message-----
From: Edward Hibbert
[mailto:[email protected]]
Sent: Monday, May 23, 2005 4:13 PM
To: [email protected]
Subject: RE: [NFS] Disabling the client cache -
O_DIRECT?
And then there's O_DIRECT, which might do just
what I want. Is that working over NFS now?
I'm trying to use it but getting EINVAL back -
can someone explain whether O_DIRECT over NFS has the alignment
restrictions that O_DIRECT does in general?
-----Original Message-----
From: [email protected]
[mailto:[email protected]]On Behalf Of Edward Hibbert
Sent: 23 May 2005 15:43
To: [email protected]
Subject: [NFS] Disabling the client
cache using file locks
The NFS FAQ says:
For greater consistency among clients,
applications can use file locking, where a client purges file data when
a file is locked, and flushes changes back to the server before
unlocking a file.
My experience with this is that to force
a read to go to the NFS server, I:
- lock an area in the file (I think any
area works, it doesn't need to overlap with what you want to read)
- read the data
- unlock it
When I want a write to go to the NFS
server, I do the same. Obviously if I'm reading and writing the same
area, I can save on the locks.
This produces a lot of lock and unlock
operations, and I find that:
- NFS servers aren't as good at handling
a high volume of these as they are at reads and writes.
- lockd/statd on the client may also be
a serialisation point (I have suggestive but not conclusive evidence).
So I'm looking for ways of reducing the
number of locks. Possibilities:
- Make a patch to the kernel (or NFS
utils?) to _entirely_ disable client-side caching of file contents over
NFS. I have dedicated boxes which have no requirement to do client-side
caching at all so this is a serious option. Any code pointers?
- Use nolock to prevent the locks
hitting the NFS server - but I think that would still hit lockd/statd on
the client?
- Not to bother with the locks round the
writes, but use fsync instead. Do folk think that would work?
Any comments or other ideas?
Edward.