Hello,
Does NFSv4 close-to-open consistency on the client work with server
"async" mount open?
We see several build errors here with parallel GNU Makefile update,
where one process writes a file, exists, and the next process doesn't
see all data (linker ar: file too short).
But if you manually look at it the files are OK, and completely written.
What is this? NFSv4 client bug, NFSv4 server bug, admin error (async
breaking close-to-open consistency?
Also, is close-to-open consistency guaranteed between different NFSv4 clients?
Thanks,
Martin
On 16 Nov 2023, at 2:11, Martin Wege wrote:
> Hello,
>
> Does NFSv4 close-to-open consistency on the client work with server
> "async" mount open?
Yes, I believe it should, but I am looking at knfsd code and I think it
won't.
> We see several build errors here with parallel GNU Makefile update,
> where one process writes a file, exists, and the next process doesn't
> see all data (linker ar: file too short).
> But if you manually look at it the files are OK, and completely written.
>
> What is this? NFSv4 client bug, NFSv4 server bug, admin error (async
> breaking close-to-open consistency?
Hard to say, a wire capture would show. My guess is that the server's
"async" disables the write gather in nfsd_vfs_write().
> Also, is close-to-open consistency guaranteed between different NFSv4 clients?
Yes.
Ben
On Thu, Nov 16, 2023 at 1:30 PM Benjamin Coddington <[email protected]> wrote:
>
> On 16 Nov 2023, at 2:11, Martin Wege wrote:
>
> > Hello,
> >
> > Does NFSv4 close-to-open consistency on the client work with server
> > "async" mount open?
>
> Yes, I believe it should, but I am looking at knfsd code and I think it
> won't.
What does that mean? nfsd server bug?
>
> > We see several build errors here with parallel GNU Makefile update,
> > where one process writes a file, exists, and the next process doesn't
> > see all data (linker ar: file too short).
> > But if you manually look at it the files are OK, and completely written.
> >
> > What is this? NFSv4 client bug, NFSv4 server bug, admin error (async
> > breaking close-to-open consistency?
>
> Hard to say, a wire capture would show. My guess is that the server's
> "async" disables the write gather in nfsd_vfs_write().
What is the correct option in Linux /etc/exports to enforce NFSv4
standard conformance?
> > Also, is close-to-open consistency guaranteed between different NFSv4 clients?
>
> Yes.
How does it work in a NFSv4 client implementation?
Thanks,
Martin
On Thu, 2023-11-16 at 07:30 -0500, Benjamin Coddington wrote:
> On 16 Nov 2023, at 2:11, Martin Wege wrote:
>
> > Hello,
> >
> > Does NFSv4 close-to-open consistency on the client work with server
> > "async" mount open?
>
> Yes, I believe it should, but I am looking at knfsd code and I think
> it
> won't.
>
> > We see several build errors here with parallel GNU Makefile update,
> > where one process writes a file, exists, and the next process
> > doesn't
> > see all data (linker ar: file too short).
> > But if you manually look at it the files are OK, and completely
> > written.
If the process doing the checking is running on the same one that wrote
the file, then we should be able to emulate full POSIX semantics, and
your problem is not NFS related.
If the two processes are running on different clients, then the caching
issues come into play. If that is the case, then is the linker opening
the file, or is it just using stat()? If the latter, then it is not
relying on close-to-open cache consistency, but just standard attribute
caching (i.e. polling).
> >
> > What is this? NFSv4 client bug, NFSv4 server bug, admin error
> > (async
> > breaking close-to-open consistency?
>
> Hard to say, a wire capture would show. My guess is that the
> server's
> "async" disables the write gather in nfsd_vfs_write().
The "async" flag in /etc/exports disables COMMIT and metadata commits.
IOW: it is a data persistence threat, but does not change any caching
semantics.
> > Also, is close-to-open consistency guaranteed between different
> > NFSv4 clients?
>
> Yes.
>
> Ben
>
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]