Received-SPF: pass (google.com: domain of linux-nfs+bounces-119-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Precedence: bulk
MIME-Version: 1.0
References: <c609e5f9df75438dbfe3810859935d58@epfl.ch> <2d948b43fa625952e50589e4bedf9551df7ee112.camel@hammerspace.com>
 <7d2d17e4d3904d29b75fadcfd916b2a3@epfl.ch> <ZWTFn0/FtJ5WuQGc@infradead.org> <7E2914D2-B9AB-4280-9A44-875DA8B58328@oracle.com>
In-Reply-To: <7E2914D2-B9AB-4280-9A44-875DA8B58328@oracle.com>
From: Rick Macklem <rick.macklem@gmail.com>
Date: Mon, 27 Nov 2023 17:50:49 -0800
Message-ID: <CAM5tNy4qVXXS3sHqx7Y3Ndt2YNnd1hrj32iJdo9KMi+ByMfEuQ@mail.gmail.com>
Subject: Re: Question about O_APPEND | O_DIRECT
To: Chuck Lever III <chuck.lever@oracle.com>
Cc: Christoph Hellwig <hch@infradead.org>, Tao Lyu <tao.lyu@epfl.ch>, 
	Trond Myklebust <trondmy@hammerspace.com>, 
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, Nov 27, 2023 at 8:51=E2=80=AFAM Chuck Lever III <chuck.lever@oracle=
.com> wrote:
>
>
> > On Nov 27, 2023, at 11:36=E2=80=AFAM, Christoph Hellwig <hch@infradead.=
org> wrote:
> >
> > On Mon, Nov 27, 2023 at 03:28:16PM +0000, Tao Lyu wrote:
> >>
> >> O_APPEND | O_DIRECT can be used to bypass the client cache for multipl=
e threads writing data without caring of the orders (e.g., logs).
> >>
> >> Yes, to support O_APPEND | O_DIRECT, NFS must first support APPEND.
> >> But the key point is that looks like NFS has supported O_APPEND alread=
y.
> >> I can successfully open a file with "O_RDWR|O_APPEND".
> >>
> >> My confusion is why NFS supports O_RDWR and O_APPEND individually but =
does not support this combination.
>
> O_DIRECT is supposed to not depend on any cached information,
> including the file size, which the client needs to know to
> form an NFS WRITE with the correct offset to ensure it is an
> appending write.
>
> File sizes are managed on the server, so the server needs to
> know that the client is requesting an appending write so it
> knows where to put the payload.
>
>
> > Well, it does support O_RDWR|O_APPEND, just not with O_DIRECT?
> >
> > Btw, I think an APPEND operation in NFS would be a very good idea, and
> > I'd love to work with interested parties in the IETF on it.
It is not easy to deal with w.r.t. RPC retries.
I suppose a NFSv4.2 extension that either requires (or strongly
recommends) persistent sessions might work?
(Persistent sessions should pretty well guarantee an RPC is not
redone on the server.)

>
> You can write and submit a personal draft that describes it; it
> wouldn't need to be more than a few pages. The hard part of that
> would be accumulating use case descriptions.
>
> I think you could create a proof of concept by including a VERIFY
> operation in front of the WRITE to ensure the WRITE occurs only
> if the offset argument in the WRITE agrees with the file's size
> on the server. If the VERIFY fails, the client grabs the updated
> file size and tries again.
This is what the FreeBSD NFSv4 client does.
Since compounds are not atomic, it is not guaranteed to work and
you might get a lot of "tries again" if multiple clients were doing the
appends on the same file concurrently. (The compound includes a
GETTTR size before the VERIFY, so trying again is pretty straightforward.)

rick

>
>
> > Not that
> > we (Damien to be specific) plan to add support to Linux to also report
> > the actual offset an O_APPEND write wrote to through io_uring as we
> > have varios use cases for out of place write data stores for that.
> > It would be great to also support that programming model over NFS.
>
> --
> Chuck Lever
>
>