Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
From:   "NeilBrown" <neilb@suse.de>
To:     "Josef Bacik" <josef@toxicpanda.com>
Cc:     "Chris Mason" <clm@fb.com>, "David Sterba" <dsterba@suse.com>,
        linux-fsdevel@vger.kernel.org,
        "Linux NFS list" <linux-nfs@vger.kernel.org>,
        "Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: [PATCH/RFC 0/4] Attempt to make progress with btrfs dev number
 strangeness.
In-reply-to: <e6496956-0df3-6232-eecb-5209b28ca790@toxicpanda.com>
References: <162848123483.25823.15844774651164477866.stgit@noble.brown>,
 <e6496956-0df3-6232-eecb-5209b28ca790@toxicpanda.com>
Date:   Thu, 12 Aug 2021 08:13:23 +1000
Message-id: <162872000356.22261.854151210687377005@noble.neil.brown.name>
Precedence: bulk

On Wed, 11 Aug 2021, Josef Bacik wrote:
>=20
> I think this is a step in the right direction, but I want to figure out a w=
ay to=20
> accomplish this without magical mount points that users must be aware of.

magic mount *options* ???

>=20
> I think the stat() st_dev ship as sailed, we're stuck with that.  However=20
> Christoph does have a valid point where it breaks the various info spit out=
 by=20
> /proc.  You've done a good job with the treeid here, but it still makes it =

> impossible for somebody to map the st_dev back to the correct mount.

The ship might have sailed, but it is not water tight.  And as the world
it round, it can still come back to bite us from behind.
Anything can be transitioned away from, whether it is devfs or 32-bit
time or giving different device numbers to different file-trees.

The linkage between device number and and filesystem is quite strong.
We could modified all of /proc and /sys/ and audit and whatever else to
report the fake device number, but we cannot get the fake device number
into the mount table (without making the mount table unmanageablely
large). =20
And if subtrees aren't in the mount-table for the NFS server, I don't
think they should be in the mount-table of the NFS client.  So we cannot
export them to NFS.

I understand your dislike for mount options.  An alternative with
different costs and benefits would be to introduce a new filesystem type
- btrfs2 or maybe betrfs.  This would provide numdevs=3D1 semantics and do
whatever we decided was best with inode numbers.  How much would you
hate that?

>=20
> I think we aren't going to solve that problem, at least not with stat().  I=
=20
> think with statx() spitting out treeid we have given userspace a way to=20
> differentiate subvolumes, and so we should fix statx() to spit out the the =
super=20
> block device, that way new userspace things can do their appropriate lookup=
 if=20
> they so choose.

I don't think we should normalize having multiple devnums per filesystem
by encoding it in statx().  It *would* make sense to add a btrfs ioctl
which reports the real device number of a file.  Tools that really need
to work with btrfs could use that, but it would always be obvious that
it was an exception.

>=20
> This leaves the problem of nfsd.  Can you just integrate this new treeid in=
to=20
> nfsd, and use that to either change the ino within nfsd itself, or do somet=
hing=20
> similar to what your first patchset did and generate a fsid based on the tr=
eeid?

I would only want nfsd to change the inode number.  I no longer think it
is acceptable for nfsd to report different device number (as I mention
above).
I would want the new inode number to be explicitly provided by the
filesystem.  Whether that is a new export_operation or a new field in
'struct kstat' doesn't really bother me.  I'd *prefer* it to be st_ino,
but I can live without that.

On the topic of inode numbers....  I've recently learned that btrfs
never reuses inode (objectid) numbers (except possibly after an
unmount).  Equally it doesn't re-use subvol numbers.  How much does this
contribute to the 64 bits not being enough for subtree+inode?

It would be nice if we could be comfortable limiting the objectid number
to 40 bits and the root.objectid (filetree) number to 24 bits, and
combine them into a 64bit inode number.

If we added a inode number reuse scheme that was suitably performant,
would that make this possible?  That would remove the need for a treeid,
and allow us to use project-id to identify subtrees.

>=20
> Mount options are messy, and are just going to lead to distro's turning the=
m on=20
> without understanding what's going on and then we have to support them fore=
ver.=20
>   I want to get this fixed in a way that we all hate the least with as litt=
le=20
> opportunity for confused users to make bad decisions.  Thanks,

Hence my question: how much do you hate creating a new filesystem type
to fix the problems?

Thanks,
NeilBrown