Message-ID: <1ea88c8d1e666b85342374ed7c0ddf7d661e0ee1.camel@redhat.com>
Subject: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified
 image filesystem
From:   Alexander Larsson <alexl@redhat.com>
To:     Amir Goldstein <amir73il@gmail.com>
Cc:     linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
        gscrivan@redhat.com, david@fromorbit.com, brauner@kernel.org,
        viro@zeniv.linux.org.uk, Vivek Goyal <vgoyal@redhat.com>,
        Miklos Szeredi <miklos@szeredi.hu>
Date:   Mon, 23 Jan 2023 18:56:19 +0100
In-Reply-To: <CAOQ4uxgGc33_QVBXMbQTnmbpHio4amv=W7ax2vQ1UMet0k_KoA@mail.gmail.com>
References: <cover.1674227308.git.alexl@redhat.com>
         <CAOQ4uxgGc33_QVBXMbQTnmbpHio4amv=W7ax2vQ1UMet0k_KoA@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.2 (3.46.2-1.fc37) 
MIME-Version: 1.0
Precedence: bulk

On Fri, 2023-01-20 at 21:44 +0200, Amir Goldstein wrote:
> On Fri, Jan 20, 2023 at 5:30 PM Alexander Larsson <alexl@redhat.com>
> wrote:
> >=20
> > Giuseppe Scrivano and I have recently been working on a new project
> > we
> > call composefs. This is the first time we propose this publically
> > and
> > we would like some feedback on it.
> >=20
>=20
> Hi Alexander,
>=20
> I must say that I am a little bit puzzled by this v3.
> Gao, Christian and myself asked you questions on v2
> that are not mentioned in v3 at all.

I got lots of good feedback from Dave Chinner on V2 that caused rather
large changes to simplify the format. So I wanted the new version with
those changes out to continue that review. I think also having that
simplified version will be helpful for the general discussion.

> To sum it up, please do not propose composefs without explaining
> what are the barriers for achieving the exact same outcome with
> the use of a read-only overlayfs with two lower layer -
> uppermost with erofs containing the metadata files, which include
> trusted.overlay.metacopy and trusted.overlay.redirect xattrs that
> refer to the lowermost layer containing the content files.

So, to be more precise, and so that everyone is on the same page, lemme
state the two options in full.

For both options, we have a directory "objects" with content-addressed
backing files (i.e. files named by sha256). In this directory all
files have fs-verity enabled. Additionally there is an image file
which you downloaded to the system that somehow references the objects
directory by relative filenames.

Composefs option:

 The image file has fs-verity enabled. To use the image, you mount it
 with options "basedir=3Dobjects,digest=3D$imagedigest".

Overlayfs option:

 The image file is a loopback image of a gpt disk with two partitions,
 one partition contains the dm-verity hashes, and the other contains
 some read-only filesystem.

 The read-only filesystem has regular versions of directories and
 symlinks, but for regular files it has sparse files with the xattrs
 "trusted.overlay.metacopy" and "trusted.overlay.redirect" set, the
 later containing a string like like "/de/adbeef..." referencing a
 backing file in the "objects" directory. In addition, the image also
 contains overlayfs whiteouts to cover any toplevel filenames from the
 objects directory that would otherwise appear if objects is used as
 a lower dir.

 To use this you loopback mount the file, and use dm-verity to set up
 the combined partitions, which you then mount somewhere. Then you
 mount an overlayfs with options:
  "metacopy=3Don,redirect_dir=3Dfollow,lowerdir=3Dveritydev:objects"

I would say both versions of this can work. There are some minor
technical issues with the overlay option:

* To get actual verification of the backing files you would need to
add support to overlayfs for an "trusted.overlay.digest" xattrs, with
behaviour similar to composefs.

* mkfs.erofs doesn't support sparse files (not sure if the kernel code
does), which means it is not a good option for the backing all these
sparse files. Squashfs seems to support this though, so that is an
option.

However, the main issue I have with the overlayfs approach is that it
is sort of clumsy and over-complex. Basically, the composefs approach
is laser focused on read-only images, whereas the overlayfs approach
just chains together technologies that happen to work, but also do a
lot of other stuff. The result is that it is more work to use it, it
uses more kernel objects (mounts, dm devices, loopbacks) and it has
worse performance.

To measure performance I created a largish image (2.6 GB centos9
rootfs) and mounted it via composefs, as well as overlay-over-squashfs,
both backed by the same objects directory (on xfs).

If I clear all caches between each run, a `ls -lR` run on composefs
runs in around 700 msec:

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR cfs-mount"
Benchmark 1: ls -lR cfs-mount
  Time (mean =C2=B1 =CF=83):     701.0 ms =C2=B1  21.9 ms    [User: 153.6 m=
s, System: 373.3 ms]
  Range (min =E2=80=A6 max):   662.3 ms =E2=80=A6 725.3 ms    10 runs

Whereas same with overlayfs takes almost four times as long:

# hyperfine -i -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR ovl-mount"
Benchmark 1: ls -lR ovl-mount
  Time (mean =C2=B1 =CF=83):      2.738 s =C2=B1  0.029 s    [User: 0.176 s=
, System: 1.688 s]
  Range (min =E2=80=A6 max):    2.699 s =E2=80=A6  2.787 s    10 runs

With page cache between runs the difference is smaller, but still
there:

# hyperfine "ls -lR cfs-mnt"
Benchmark 1: ls -lR cfs-mnt
  Time (mean =C2=B1 =CF=83):     390.1 ms =C2=B1   3.7 ms    [User: 140.9 m=
s, System: 247.1 ms]
  Range (min =E2=80=A6 max):   381.5 ms =E2=80=A6 393.9 ms    10 runs

vs

# hyperfine -i "ls -lR ovl-mount"
Benchmark 1: ls -lR ovl-mount
  Time (mean =C2=B1 =CF=83):     431.5 ms =C2=B1   1.2 ms    [User: 124.3 m=
s, System: 296.9 ms]
  Range (min =E2=80=A6 max):   429.4 ms =E2=80=A6 433.3 ms    10 runs

This isn't all that strange, as overlayfs does a lot more work for
each lookup, including multiple name lookups as well as several xattr
lookups, whereas composefs just does a single lookup in a pre-computed
table. But, given that we don't need any of the other features of
overlayfs here, this performance loss seems rather unnecessary.

I understand that there is a cost to adding more code, but efficiently
supporting containers and other forms of read-only images is a pretty
important usecase for Linux these days, and having something tailored
for that seems pretty useful to me, even considering the code
duplication.


I also understand Cristians worry about stacking filesystem, having
looked a bit more at the overlayfs code. But, since composefs doesn't
really expose the metadata or vfs structure of the lower directories it
is much simpler in a fundamental way.

--=20
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-
=3D-=3D-=3D
 Alexander Larsson                                            Red Hat,
Inc=20
       alexl@redhat.com            alexander.larsson@gmail.com=20
He's a fast talking sweet-toothed farmboy who must take medication to=20
keep him sane. She's a wealthy streetsmart magician's assistant who=20
dreams of becoming Elvis. They fight crime!=20