2007-08-06 16:03:54

by Wim Colgate

[permalink] [raw]
Subject: NFS_UNSTABLE vs. FILE and DATA sync.

Hi,



My last question was a bit too nebulous, and didn't really expect an
answer. Learning my lesson, I have a specific question.



If I have a soft mount, and open a file with O_DIRECT and O_SYNC, should
I ever expect a callback (nfs_writeback_done) with a successful
task->tk_status (i.e >= 0) with the committed state
(resp->verf->committed) set to NFS_UNSTABLE?



A secondary question: if the above is expected, does this occur because
someone is caching the write and is there a mechanism to disable this
effect?



Regards,



Wim


Attachments:
(No filename) (315.00 B)
(No filename) (140.00 B)
Download all attachments

2007-08-06 17:33:05

by Peter Staubach

[permalink] [raw]
Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync.

Chuck Lever wrote:
> Wim Colgate wrote:
>> If I have a soft mount, and open a file with O_DIRECT and O_SYNC,
>> should I ever expect a callback (nfs_writeback_done) with a
>> successful task->tk_status (i.e >= 0) with the committed state
>> (resp->verf->committed) set to NFS_UNSTABLE?
>
> Yes, this can happen if the server decides to return NFS_UNSTABLE.
> Rare, but possible.
>
>> A secondary question: if the above is expected, does this occur
>> because someone is caching the write and is there a mechanism to
>> disable this effect?
>
> Servers can return NFS_UNSTABLE to any WRITE request, so I can't think
> of a way this might be disabled.

Actually, it would be a protocol error for a server to return
a commitment level less than was requested by the client. The
server can return a greater commitment level, but not less than.

ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-06 17:42:44

by Wim Colgate

[permalink] [raw]
Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync.

Interesting information.

Specifically I am trying to inject errors by manually (but politely)
bringing the NFS server down then up, then down (rinse and repeat ...)
while doing IO from a linux client. As mentioned the open file is
O_DIRECT and O_SYNC -- which I thought should mean either the data hits
the server's storage or I should get an error; and I'm more than happy
to deal with an IO error.

I'm confident the writes are less than wsize (4096 bytes to be precise).


Is there a 100% guaranteed method to get the behavior I thought O_DIRECT
and O_SYNC was providing?

Thanks,

Wim


-----Original Message-----
From: Peter Staubach [mailto:[email protected]]
Sent: Monday, August 06, 2007 10:33 AM
To: [email protected]
Cc: Wim Colgate; [email protected]
Subject: Re: [NFS] NFS_UNSTABLE vs. FILE and DATA sync.

Chuck Lever wrote:
> Wim Colgate wrote:
>> If I have a soft mount, and open a file with O_DIRECT and O_SYNC,
>> should I ever expect a callback (nfs_writeback_done) with a
>> successful task->tk_status (i.e >= 0) with the committed state
>> (resp->verf->committed) set to NFS_UNSTABLE?
>
> Yes, this can happen if the server decides to return NFS_UNSTABLE.
> Rare, but possible.
>
>> A secondary question: if the above is expected, does this occur
>> because someone is caching the write and is there a mechanism to
>> disable this effect?
>
> Servers can return NFS_UNSTABLE to any WRITE request, so I can't think

> of a way this might be disabled.

Actually, it would be a protocol error for a server to return
a commitment level less than was requested by the client. The
server can return a greater commitment level, but not less than.

ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-06 18:58:39

by Trond Myklebust

[permalink] [raw]
Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync.

On Mon, 2007-08-06 at 13:10 -0400, Chuck Lever wrote:
> Even though an NFS client requests an NFS_FILE_SYNC write, the server
> still has the choice of returning something less, even NFS_UNSTABLE. In
> general that's a rare occurrence, but is something I've seen in practice.

As Peter said, a server that return anything other than FILE_SYNC to a
FILE_SYNC write request would be in clear violation of the description
of WRITE semantics on page 51 of RFC1813:

committed
The server should return an indication of the level of
commitment of the data and metadata via committed. If
the server committed all data and metadata to stable
storage, committed should be set to FILE_SYNC. If the
level of commitment was at least as strong as
DATA_SYNC, then committed should be set to DATA_SYNC.
Otherwise, committed must be returned as UNSTABLE. If
stable was FILE_SYNC, then committed must also be
FILE_SYNC: anything else constitutes a protocol
violation. If stable was DATA_SYNC, then committed may
be FILE_SYNC or DATA_SYNC: anything else constitutes a
protocol violation. If stable was UNSTABLE, then
committed may be either FILE_SYNC, DATA_SYNC, or
UNSTABLE.

I see no reason why we should care about supporting such a server.

Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-06 19:19:54

by Trond Myklebust

[permalink] [raw]
Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync.

On Mon, 2007-08-06 at 15:13 -0400, Chuck Lever wrote:
> Trond Myklebust wrote:
> > On Mon, 2007-08-06 at 13:10 -0400, Chuck Lever wrote:
> >> Even though an NFS client requests an NFS_FILE_SYNC write, the server
> >> still has the choice of returning something less, even NFS_UNSTABLE. In
> >> general that's a rare occurrence, but is something I've seen in practice.
> >
> > As Peter said, a server that return anything other than FILE_SYNC to a
> > FILE_SYNC write request would be in clear violation of the description
> > of WRITE semantics on page 51 of RFC1813:
> >
> > committed
> > The server should return an indication of the level of
> > commitment of the data and metadata via committed. If
> > the server committed all data and metadata to stable
> > storage, committed should be set to FILE_SYNC. If the
> > level of commitment was at least as strong as
> > DATA_SYNC, then committed should be set to DATA_SYNC.
> > Otherwise, committed must be returned as UNSTABLE. If
> > stable was FILE_SYNC, then committed must also be
> > FILE_SYNC: anything else constitutes a protocol
> > violation. If stable was DATA_SYNC, then committed may
> > be FILE_SYNC or DATA_SYNC: anything else constitutes a
> > protocol violation. If stable was UNSTABLE, then
> > committed may be either FILE_SYNC, DATA_SYNC, or
> > UNSTABLE.
> >
> > I see no reason why we should care about supporting such a server.
>
> I said nothing about whether the server should or should not return such
> a value. I just said that it is a possibility, and that I have observed
> the behavior in the field.
>
> The client, if it is a good implementation, needs to check for this
> possibility and throw an error in that case.

We have never supported servers that blatantly violate the protocol, and
I see no reason to be burdening the client with a whole load of checks
for server protocol violations either.

If you want a tool for testing servers, then use something like pynfs.

Trond


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2007-08-06 19:34:25

by Wim Colgate

[permalink] [raw]
Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync.

Hi Chuck,

The linux kernel I was using is 2.6.18-8.

To be fair, I was not trying to force NFS_FILE_SYNC; to make a long
story short, I started with O_DIRECT (please don't cache data). I moved
to add O_SYNC (don't return until my data is written safely). And when I
couldn't explain why I was missing some data (discrepancy between client
and server), I started investigating what was happening under the hood.
I didn't really want to start a controversy -- I just wanted to
understand what was happening.

My understanding of NFS is fairly pedestrian in that I merely get the
big picture. Which is why I posted here.

Thanks,

Wim

-----Original Message-----
From: Chuck Lever [mailto:[email protected]]
Sent: Monday, August 06, 2007 12:16 PM
To: Wim Colgate
Cc: [email protected]
Subject: Re: [NFS] NFS_UNSTABLE vs. FILE and DATA sync.

Wim Colgate wrote:
> Specifically I am trying to inject errors by manually (but politely)
> bringing the NFS server down then up, then down (rinse and repeat ...)
> while doing IO from a linux client. As mentioned the open file is
> O_DIRECT and O_SYNC -- which I thought should mean either the data
hits
> the server's storage or I should get an error; and I'm more than happy
> to deal with an IO error.
>
> I'm confident the writes are less than wsize (4096 bytes to be
precise).
>
>
> Is there a 100% guaranteed method to get the behavior I thought
O_DIRECT
> and O_SYNC was providing?

What behavior did you expect O_DIRECT + O_SYNC to provide? O_DIRECT
means "don't cache data" and O_SYNC means "make sure the data is flushed

to the server's disk before each write() system call returns."
Technically, you don't need NFS_FILE_SYNC writes to do either of those.

Which kernel are you testing? The client's use of NFS_FILE_SYNC writes
changed over time.

> -----Original Message-----
> From: Peter Staubach [mailto:[email protected]]
> Sent: Monday, August 06, 2007 10:33 AM
> To: [email protected]
> Cc: Wim Colgate; [email protected]
> Subject: Re: [NFS] NFS_UNSTABLE vs. FILE and DATA sync.
>
> Chuck Lever wrote:
>> Wim Colgate wrote:
>>> If I have a soft mount, and open a file with O_DIRECT and O_SYNC,
>>> should I ever expect a callback (nfs_writeback_done) with a
>>> successful task->tk_status (i.e >= 0) with the committed state
>>> (resp->verf->committed) set to NFS_UNSTABLE?
>> Yes, this can happen if the server decides to return NFS_UNSTABLE.
>> Rare, but possible.
>>
>>> A secondary question: if the above is expected, does this occur
>>> because someone is caching the write and is there a mechanism to
>>> disable this effect?
>> Servers can return NFS_UNSTABLE to any WRITE request, so I can't
think
>
>> of a way this might be disabled.
>
> Actually, it would be a protocol error for a server to return
> a commitment level less than was requested by the client. The
> server can return a greater commitment level, but not less than.
>
> ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs