MIME-Version: 1.0
References: <2065755c-f888-9c62-f6e5-f143d42c51ee@oracle.com>
 <20190402161116.GA2828@fieldses.org> <2f1f6582-3672-1361-4392-80cb1e62e19c@oracle.com>
 <20190402194148.GA5269@fieldses.org> <58230e155813e866cb057e6543ab7e61f51fedf6.camel@hammerspace.com>
 <20190403002822.GA7667@fieldses.org> <ec8d33473da41b3bde45f5ef1b4a02b98a535ac5.camel@hammerspace.com>
 <20190403020750.GA8272@fieldses.org> <c538dd35-fd1f-bab6-1cd3-6c9639462eab@oracle.com>
 <20190404010559.GA17840@fieldses.org> <CAOhYWNLZ2OmnGo2nEegmuXOrU_ncmAwWgQSAMXxKfn3JP-UMUw@mail.gmail.com>
 <97542732-49F5-4BEA-9903-D9801370A221@oracle.com>
In-Reply-To: <97542732-49F5-4BEA-9903-D9801370A221@oracle.com>
From:   Jeff Layton <jlayton@poochiereds.net>
Date:   Thu, 4 Apr 2019 11:36:25 -0400
Message-ID: <CAOhYWNJ=w2ni2aR703dhSov155UR2D-1HP3MT5D-NSoegv2bJw@mail.gmail.com>
Subject: Re: directory delegations
To:     Chuck Lever <chuck.lever@oracle.com>
Cc:     Bruce Fields <bfields@fieldses.org>,
        "Bradley C. Kuszmaul" <bradley.kuszmaul@oracle.com>,
        Trond Myklebust <trondmy@hammerspace.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On Thu, Apr 4, 2019 at 11:22 AM Chuck Lever <chuck.lever@oracle.com> wrote:
>
>
>
> > On Apr 4, 2019, at 11:09 AM, Jeff Layton <jlayton@poochiereds.net> wrot=
e:
> >
> > On Wed, Apr 3, 2019 at 9:06 PM bfields@fieldses.org
> > <bfields@fieldses.org> wrote:
> >>
> >> On Wed, Apr 03, 2019 at 12:56:24PM -0400, Bradley C. Kuszmaul wrote:
> >>> This proposal does look like it would be helpful.   How does this
> >>> kind of proposal play out in terms of actually seeing the light of
> >>> day in deployed systems?
> >>
> >> We need some people to commit to implementing it.
> >>
> >> We have 2-3 testing events a year, so ideally we'd agree to show up wi=
th
> >> implementations at one of those to test and hash out any issues.
> >>
> >> We revise the draft based on any experience or feedback we get.  If
> >> nothing else, it looks like it needs some updates for v4.2.
> >>
> >> The on-the-wire protocol change seems small, and my feeling is that if
> >> there's running code then documenting the protocol and getting it
> >> through the IETF process shouldn't be a big deal.
> >>
> >> --b.
> >>
> >>> On 4/2/19 10:07 PM, bfields@fieldses.org wrote:
> >>>> On Wed, Apr 03, 2019 at 02:02:54AM +0000, Trond Myklebust wrote:
> >>>>> The create itself needs to be sync, but the attribute delegations m=
ean
> >>>>> that the client, not the server, is authoritative for the timestamp=
s.
> >>>>> So the client now owns the atime and mtime, and just sets them as p=
art
> >>>>> of the (asynchronous) delegreturn some time after you are done writ=
ing.
> >>>>>
> >>>>> Were you perhaps thinking about this earlier proposal?
> >>>>> https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__tools.ietf.o=
rg_html_draft-2Dmyklebust-2Dnfsv4-2Dunstable-2Dfile-2Dcreation-2D01&d=3DDwI=
BAg&c=3DRoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=3DYIKOmJLMLfe5wQR3VJI=
7jGjCnepZlMwumApzvaKItrY&m=3DqlAJ6dZPGjbcTzNIpkTyk-RTii6lWw1CLIjF6jp3P2Y&s=
=3DaTTFNJlRH-dXrQmE4cSYEUd8Kv3ij5cqTJtvgIixMa8&e=3D
> >>>> That's it, thanks!
> >>>>
> >>>> Bradley is concerned about performance of something like untar on a
> >>>> backend filesystem with particularly high-latency metadata operation=
s,
> >>>> so something like your unstable file createion proposal (or actual w=
rite
> >>>> delegations) seems like it should help.
> >>>>
> >>>> --b.
> >
> > The serialized create with something like an untar is a
> > performance-killer though.
> >
> > FWIW, I'm working on something similar right now for Ceph. If a ceph
> > client has adequate caps [1] for a directory and the dentry inode,
> > then we should (in principle) be able to buffer up directory morphing
> > operations and flush them out to the server asynchronously.
> >
> > I'm starting with unlink (mostly because it's simpler), and am mainly
> > just returning early when we do have the right caps -- after issuing
> > the call but before the reply comes in. We should be able to do the
> > same for link, rename and create too. Create will require the Ceph MDS
> > to delegate out a range of inode numbers (and that bit hasn't been
> > implemented yet).
> >
> > My thinking with all of this is that the buffering of directory
> > morphing operations is not as helpful as something like a pagecache
> > write is, as we aren't that interested in merging operations that
> > change the same dentry. However, being able to do them asynchronously
> > should work really well. That should allow us to better parallellize
> > create/link/unlink/rename on different dentries even when they are
> > issued serially by a single task.
>
> What happens if an asynchronous directory change fails (eg. ENOSPC)?
>

We have a well-established expectation with most local filesystems
that directory changes are not necessarily persisted until you issue
fsync on the parent(s). My thinking is that we'd report these sorts of
errors to that fsync.

All of this is _really_ experimental so far, so I don't claim to have
worked out all of the gory details as of yet. :)
--=20
Jeff Layton <jlayton@poochiereds.net>