Date:   Mon, 3 Jan 2022 11:20:41 -0500
From:   "'bfields@fieldses.org'" <bfields@fieldses.org>
To:     NeilBrown <neilb@suse.de>
Cc:     "inoguchi.yuki@fujitsu.com" <inoguchi.yuki@fujitsu.com>,
        'Matt Benjamin' <mbenjami@redhat.com>,
        'Trond Myklebust' <trondmy@hammerspace.com>,
        "'linux-nfs@vger.kernel.org'" <linux-nfs@vger.kernel.org>
Subject: Re: client caching and locks
Message-ID: <20220103162041.GC21514@fieldses.org>
References: <20200608211945.GB30639@fieldses.org>
 <22b841f7a8979f19009c96f31a7be88dd177a47a.camel@hammerspace.com>
 <20200618200905.GA10313@fieldses.org>
 <20200622135222.GA6075@fieldses.org>
 <20201001214749.GK1496@fieldses.org>
 <CAKOnarndL1-u5jGG2VAENz2bEc9wsERH6rGTbZeYZy+WyAUk-w@mail.gmail.com>
 <20201006172607.GA32640@fieldses.org>
 <164066831190.25899.16641224253864656420@noble.neil.brown.name>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <164066831190.25899.16641224253864656420@noble.neil.brown.name>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk

On Tue, Dec 28, 2021 at 04:11:51PM +1100, NeilBrown wrote:
> This is due to an (arguable) weakness in the NFSv4 protocol.
> In NFSv3 the reply to the WRITE request had "wcc" data which would
> report change information before and after the write and, if present, it
> was required to be atomic.  So, providing timestamps had a high
> resolution, the client0 would see change information from *before* the
> write from client1 completed.  So it would know it needed to refresh
> after that write became visible.
> 
> With NFSv4 there is no atomicity guarantees relating to writes and
> changeid.
> There is provision for atomicity around directory operations, but not
> around data operations.
> 
> This means that if different clients access a file concurrently, then
> their cache can become incorrect.  The only way to ensure uncorrupted
> data is to use locking for ALL reads and writes.  The above 'od -i' does
> not perform a locked read, so can give incorrect data.
> If you got a whole-file lock before reading, then you should get correct
> data. 

You'd also have to get a whole-file write lock on every write, wouldn't
you, to prevent your own write from obscuring the change-attribute
update caused by a concurrent writer?

> You could argue that this requirement (always lock if there is any risk)
> is by design, and so this aspect of the protocl is not a weakness.

The spec discussion of byte-range locking and caching is here:
https://datatracker.ietf.org/doc/html/rfc7530#section-10.3.2

The nfs man page, under ac/noac, says "Using the noac option provides
greater  cache  coherence among  NFS  clients accessing the same files,
but it extracts a significant performance penalty.  As such,  judicious
use of file locking is encouraged instead.  The DATA AND METADATA
COHERENCE section contains a detailed discussion of these trade-offs."

That section does have a "Using file locks with NFS" subsection, but
that subsection doesn't actually discuss the interaction of file locks
with client caching.

It's a confusing and under-documented area.

--b.