nfs'rs-
i'm in the middle of implementing a job distribution mechanism that uses an
nfs mounted work queue. obviously clients serialize their access to this
queue using locking. i'm using the double lock technique of lockfile + fcntl
lock on lockfile. the code implements crude transactions using the following
logic:
obtain_lock
read_queue
do_work_with_queue
if changes_were_made
copy queue, backup
err = write queue
if err
copy backup, queue
end
end
release_lock
in otherwords, all or none wholesale replacement of queue. in order for this
sort of thing to work i need to be able to do a few things:
- reliable lock (done)
- reliable read
- reliable write
now, i undertand the word 'reliable' is subjective here when we're talking
about networking but the inode caching of nfs makes the read/write bit more
that little problematic. my questions are:
- what is the 'safest' way read the most recent contents of an nfs mounted
file accounting for attribute caching?
- same question but for writing
i am not concerned with speed since the cost of accessing the queue is grossly
outweighed by the time to execute a job taken from it in almost all cases - i
simply want to make the absolutely most reliable read/write __possible__. i
am open to mount option suggestions but want to make the code reliable even in
the face of the most aggressive attribute caching - if possible.
thanks in advance for any suggestions.
-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===============================================================================
-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
digital self defense, top technical experts, no vendor pitches,
unmatched networking opportunities. Visit http://www.blackhat.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Mon, 28 Jun 2004, Daniel Forrest wrote:
> On Mon, Jun 28, 2004 at 09:13:29AM -0600, Ara.T.Howard wrote:
>>
>> - what is the 'safest' way read the most recent contents of an
>> nfs mounted file accounting for attribute caching?
>>
>> - same question but for writing
>
> We have a queueing system that does the same types of operations (over
> 45 machines all reading and writing files from the same NFS mounted
> filesystems). We use the following technique:
>
> fd = open ("file", O_RDWR);
> lockf (fd, F_TLOCK, 0);
> fchmod (fd, 0644);
> size = lseek (fd, 0, SEEK_END);
>
> The "open" is required to get a file handle for the "lockf" operation.
>
> The "lockf" clears the NFS client cache.
>
> You would hope at this point that things would be okay, but it turns
> out that the file size is still the file size that was cached at the
> time the file was "open"ed. So...
>
> The "fchmod" operation is basically a no-op (we happen to know the
> file already has these permissions), but it forces a round-trip
> request to the server which updates the actual file size.
>
> The "lseek" gives the actual file size to the program.
>
> I don't remember if "fstat" could be used in place of the "lseek" as
> we are usually appending to the end of the file so the "lseek" is
> being called anyway. In the case when we are copying the file we use
> the same algorithm and then "lseek (fd, 0, SEEK_SET)" before reading.
>
> While the "open"/"lockf" itself seems to work almost all of the time,
> we had rare occasions when we lost lines from the end of the file
> before we added the "fchmod" before the "lseek". Also, there are
> probably other operations besides "fchmod" that will update the actual
> file size, but I don't recall which ones they are.
>
> I hope this helps.
very much! i have a similar approach now and wondered if i was crazy - or if
this was just par for the course.
cheers.
-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===============================================================================
-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
digital self defense, top technical experts, no vendor pitches,
unmatched networking opportunities. Visit http://www.blackhat.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Mon, Jun 28, 2004 at 09:13:29AM -0600, Ara.T.Howard wrote:
>
> - what is the 'safest' way read the most recent contents of an
> nfs mounted file accounting for attribute caching?
>
> - same question but for writing
We have a queueing system that does the same types of operations (over
45 machines all reading and writing files from the same NFS mounted
filesystems). We use the following technique:
fd = open ("file", O_RDWR);
lockf (fd, F_TLOCK, 0);
fchmod (fd, 0644);
size = lseek (fd, 0, SEEK_END);
The "open" is required to get a file handle for the "lockf" operation.
The "lockf" clears the NFS client cache.
You would hope at this point that things would be okay, but it turns
out that the file size is still the file size that was cached at the
time the file was "open"ed. So...
The "fchmod" operation is basically a no-op (we happen to know the
file already has these permissions), but it forces a round-trip
request to the server which updates the actual file size.
The "lseek" gives the actual file size to the program.
I don't remember if "fstat" could be used in place of the "lseek" as
we are usually appending to the end of the file so the "lseek" is
being called anyway. In the case when we are copying the file we use
the same algorithm and then "lseek (fd, 0, SEEK_SET)" before reading.
While the "open"/"lockf" itself seems to work almost all of the time,
we had rare occasions when we lost lines from the end of the file
before we added the "fchmod" before the "lseek". Also, there are
probably other operations besides "fchmod" that will update the actual
file size, but I don't recall which ones they are.
I hope this helps.
--
Daniel K. Forrest Laboratory for Molecular and
[email protected] Computational Genomics
University of Wisconsin, Madison
-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
digital self defense, top technical experts, no vendor pitches,
unmatched networking opportunities. Visit http://www.blackhat.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs