Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\))
Subject: Re: directory delegations
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <CAOhYWNLZ2OmnGo2nEegmuXOrU_ncmAwWgQSAMXxKfn3JP-UMUw@mail.gmail.com>
Date:   Thu, 4 Apr 2019 11:22:34 -0400
Cc:     Bruce Fields <bfields@fieldses.org>,
        "Bradley C. Kuszmaul" <bradley.kuszmaul@oracle.com>,
        Trond Myklebust <trondmy@hammerspace.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <97542732-49F5-4BEA-9903-D9801370A221@oracle.com>
References: <2065755c-f888-9c62-f6e5-f143d42c51ee@oracle.com>
 <20190402161116.GA2828@fieldses.org>
 <2f1f6582-3672-1361-4392-80cb1e62e19c@oracle.com>
 <20190402194148.GA5269@fieldses.org>
 <58230e155813e866cb057e6543ab7e61f51fedf6.camel@hammerspace.com>
 <20190403002822.GA7667@fieldses.org>
 <ec8d33473da41b3bde45f5ef1b4a02b98a535ac5.camel@hammerspace.com>
 <20190403020750.GA8272@fieldses.org>
 <c538dd35-fd1f-bab6-1cd3-6c9639462eab@oracle.com>
 <20190404010559.GA17840@fieldses.org>
 <CAOhYWNLZ2OmnGo2nEegmuXOrU_ncmAwWgQSAMXxKfn3JP-UMUw@mail.gmail.com>
To:     Jeff Layton <jlayton@poochiereds.net>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Apr 4, 2019, at 11:09 AM, Jeff Layton <jlayton@poochiereds.net> =
wrote:
>=20
> On Wed, Apr 3, 2019 at 9:06 PM bfields@fieldses.org
> <bfields@fieldses.org> wrote:
>>=20
>> On Wed, Apr 03, 2019 at 12:56:24PM -0400, Bradley C. Kuszmaul wrote:
>>> This proposal does look like it would be helpful.   How does this
>>> kind of proposal play out in terms of actually seeing the light of
>>> day in deployed systems?
>>=20
>> We need some people to commit to implementing it.
>>=20
>> We have 2-3 testing events a year, so ideally we'd agree to show up =
with
>> implementations at one of those to test and hash out any issues.
>>=20
>> We revise the draft based on any experience or feedback we get.  If
>> nothing else, it looks like it needs some updates for v4.2.
>>=20
>> The on-the-wire protocol change seems small, and my feeling is that =
if
>> there's running code then documenting the protocol and getting it
>> through the IETF process shouldn't be a big deal.
>>=20
>> --b.
>>=20
>>> On 4/2/19 10:07 PM, bfields@fieldses.org wrote:
>>>> On Wed, Apr 03, 2019 at 02:02:54AM +0000, Trond Myklebust wrote:
>>>>> The create itself needs to be sync, but the attribute delegations =
mean
>>>>> that the client, not the server, is authoritative for the =
timestamps.
>>>>> So the client now owns the atime and mtime, and just sets them as =
part
>>>>> of the (asynchronous) delegreturn some time after you are done =
writing.
>>>>>=20
>>>>> Were you perhaps thinking about this earlier proposal?
>>>>> =
https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__tools.ietf.org_html=
_draft-2Dmyklebust-2Dnfsv4-2Dunstable-2Dfile-2Dcreation-2D01&d=3DDwIBAg&c=3D=
RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=3DYIKOmJLMLfe5wQR3VJI7jGjCne=
pZlMwumApzvaKItrY&m=3DqlAJ6dZPGjbcTzNIpkTyk-RTii6lWw1CLIjF6jp3P2Y&s=3DaTTF=
NJlRH-dXrQmE4cSYEUd8Kv3ij5cqTJtvgIixMa8&e=3D
>>>> That's it, thanks!
>>>>=20
>>>> Bradley is concerned about performance of something like untar on a
>>>> backend filesystem with particularly high-latency metadata =
operations,
>>>> so something like your unstable file createion proposal (or actual =
write
>>>> delegations) seems like it should help.
>>>>=20
>>>> --b.
>=20
> The serialized create with something like an untar is a
> performance-killer though.
>=20
> FWIW, I'm working on something similar right now for Ceph. If a ceph
> client has adequate caps [1] for a directory and the dentry inode,
> then we should (in principle) be able to buffer up directory morphing
> operations and flush them out to the server asynchronously.
>=20
> I'm starting with unlink (mostly because it's simpler), and am mainly
> just returning early when we do have the right caps -- after issuing
> the call but before the reply comes in. We should be able to do the
> same for link, rename and create too. Create will require the Ceph MDS
> to delegate out a range of inode numbers (and that bit hasn't been
> implemented yet).
>=20
> My thinking with all of this is that the buffering of directory
> morphing operations is not as helpful as something like a pagecache
> write is, as we aren't that interested in merging operations that
> change the same dentry. However, being able to do them asynchronously
> should work really well. That should allow us to better parallellize
> create/link/unlink/rename on different dentries even when they are
> issued serially by a single task.

What happens if an asynchronous directory change fails (eg. ENOSPC)?


> RFC5661 doesn't currently provide for writeable directory delegations,
> AFAICT, but they could eventually be implemented in a similar way.
>=20
> [1]: cephfs capabilies (aka caps) are like a delegation for a subset
> of inode metadata
> --
> Jeff Layton <jlayton@poochiereds.net>

--
Chuck Lever