Hi,
When testing against the knfsd in a Linux 5.15.1 kernel, I received a
write reply with FILE_SYNC and a writeverf of all 0 bytes.
(Previous write verifiers were not all 0 bytes.)
The server seemed to be functioning normally and had not rebooted.
Is this intended behaviour?
Normally I would not expect the writeverf in a Write operation reply
to change unless the server had rebooted, but I can see there might
be circumstances where the knfsd server wants all non-FILE_SYNC
writes to be redone by the client and would choose to change the
writeverf.
However, changing it to all 0 bytes seems particularly odd?
rick
Hi Rick-
> On Dec 9, 2021, at 6:15 PM, Rick Macklem <[email protected]> wrote:
>
> Hi,
>
> When testing against the knfsd in a Linux 5.15.1 kernel, I received a
> write reply with FILE_SYNC and a writeverf of all 0 bytes.
> (Previous write verifiers were not all 0 bytes.)
>
> The server seemed to be functioning normally and had not rebooted.
>
> Is this intended behaviour?
>
> Normally I would not expect the writeverf in a Write operation reply
> to change unless the server had rebooted, but I can see there might
> be circumstances where the knfsd server wants all non-FILE_SYNC
> writes to be redone by the client and would choose to change the
> writeverf.
> However, changing it to all 0 bytes seems particularly odd?
I don't immediately see a code path for WRITE or COMMIT that would
set the verifier to zeroes. When Linux NFSD resets its write verifier,
it sets it to the current time.
Do you have a reproducer you can share?
--
Chuck Lever
On Mon, 2021-12-13 at 19:42 +0000, Chuck Lever III wrote:
> Hi Rick-
>
> > On Dec 9, 2021, at 6:15 PM, Rick Macklem <[email protected]>
> > wrote:
> >
> > Hi,
> >
> > When testing against the knfsd in a Linux 5.15.1 kernel, I received
> > a
> > write reply with FILE_SYNC and a writeverf of all 0 bytes.
> > (Previous write verifiers were not all 0 bytes.)
> >
> > The server seemed to be functioning normally and had not rebooted.
> >
> > Is this intended behaviour?
> >
> > Normally I would not expect the writeverf in a Write operation
> > reply
> > to change unless the server had rebooted, but I can see there might
> > be circumstances where the knfsd server wants all non-FILE_SYNC
> > writes to be redone by the client and would choose to change the
> > writeverf.
> > However, changing it to all 0 bytes seems particularly odd?
>
> I don't immediately see a code path for WRITE or COMMIT that would
> set the verifier to zeroes. When Linux NFSD resets its write
> verifier,
> it sets it to the current time.
>
> Do you have a reproducer you can share?
>
Rick is using FILE_SYNC, whereas nfsd_vfs_write() only actually sets
the boot verifier if the write is unstable or DATA_SYNC.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
> On Dec 13, 2021, at 2:54 PM, Trond Myklebust <[email protected]> wrote:
>
> On Mon, 2021-12-13 at 19:42 +0000, Chuck Lever III wrote:
>> Hi Rick-
>>
>>> On Dec 9, 2021, at 6:15 PM, Rick Macklem <[email protected]>
>>> wrote:
>>>
>>> Hi,
>>>
>>> When testing against the knfsd in a Linux 5.15.1 kernel, I received
>>> a
>>> write reply with FILE_SYNC and a writeverf of all 0 bytes.
>>> (Previous write verifiers were not all 0 bytes.)
>>>
>>> The server seemed to be functioning normally and had not rebooted.
>>>
>>> Is this intended behaviour?
>>>
>>> Normally I would not expect the writeverf in a Write operation
>>> reply
>>> to change unless the server had rebooted, but I can see there might
>>> be circumstances where the knfsd server wants all non-FILE_SYNC
>>> writes to be redone by the client and would choose to change the
>>> writeverf.
>>> However, changing it to all 0 bytes seems particularly odd?
>>
>> I don't immediately see a code path for WRITE or COMMIT that would
>> set the verifier to zeroes. When Linux NFSD resets its write
>> verifier,
>> it sets it to the current time.
>>
>> Do you have a reproducer you can share?
>>
>
> Rick is using FILE_SYNC, whereas nfsd_vfs_write() only actually sets
> the boot verifier if the write is unstable or DATA_SYNC.
It wasn't clear from Rick's note that the verifier change he
observed was not permanent.
So then the answer to "Is this intended behavior?" is "Yes,
Linux NFSD returns an all-zero verifier for FILE_SYNC writes,
since the client does not use the verifier in this case. The
boot verifier for subsequent non-FILE_SYNC writes is not
altered."
--
Chuck Lever
Chuck Lever III wrote:
>> On Dec 13, 2021, at 2:54 PM, Trond Myklebust <[email protected]> wrote:
>>
>> On Mon, 2021-12-13 at 19:42 +0000, Chuck Lever III wrote:
>>> Hi Rick-
>>>
>>>> On Dec 9, 2021, at 6:15 PM, Rick Macklem <[email protected]>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> When testing against the knfsd in a Linux 5.15.1 kernel, I received
>>>> a
>>>> write reply with FILE_SYNC and a writeverf of all 0 bytes.
>>>> (Previous write verifiers were not all 0 bytes.)
>>>>
>>>> The server seemed to be functioning normally and had not rebooted.
>>>>
>>>> Is this intended behaviour?
>>>>
>>>> Normally I would not expect the writeverf in a Write operation
>>>> reply
>>>> to change unless the server had rebooted, but I can see there might
>>>> be circumstances where the knfsd server wants all non-FILE_SYNC
>>>> writes to be redone by the client and would choose to change the
>>>> writeverf.
>>>> However, changing it to all 0 bytes seems particularly odd?
>>>
>>> I don't immediately see a code path for WRITE or COMMIT that would
>>> set the verifier to zeroes. When Linux NFSD resets its write
>>> verifier,
>>> it sets it to the current time.
>>>
>>> Do you have a reproducer you can share?
>>>
>>
>> Rick is using FILE_SYNC, whereas nfsd_vfs_write() only actually sets
>> the boot verifier if the write is unstable or DATA_SYNC.
>
>It wasn't clear from Rick's note that the verifier change he
>observed was not permanent.
>
>So then the answer to "Is this intended behavior?" is "Yes,
>Linux NFSD returns an all-zero verifier for FILE_SYNC writes,
>since the client does not use the verifier in this case. The
>boot verifier for subsequent non-FILE_SYNC writes is not
>altered."
Yes, that is what I observed.
The good news is that the FreeBSD client can easily ignore
the writeverf reply for FILESYNC writes (ones where the
write request specifies FILESYNC and the reply is expected
to be FILESYNC). I have already patched the client for this case.
Although I agree that a write reply that is FILESYNC does not
need to be redone, no matter what happens to writeverf, I
do not see anything in RFC8881 that suggests it should be
ignored in this case or that the value of all 0 bytes is special.
My interpretation of RFC8881 is that the writeverf is global
for the server and any change would indicate all writes that
did not reply FILESYNC and have not been committed, must
be redone.
--> There is the weird case w.r.t. File Layout pNFS when
"commit through MDS" is indicated, where the writeverf
appears to be "per file" and not "per server". And I'll admit
I don't really understand this, despite having read it.
Does this need to be discussed on [email protected]?
In any case, the FreeBSD client never does FILESYNC writes
by default and is now fixed to ignore the writeverf reply in
this case.
And what about the case where the client does a write
with UNSTABLE, but the server chooses to reply FILESYNC?
--> Is the writeverf ever going to be zero for that case?
Thanks, rick
--
Chuck Lever