MIME-Version: 1.0
References: <cover.1674227308.git.alexl@redhat.com> <CAOQ4uxgGc33_QVBXMbQTnmbpHio4amv=W7ax2vQ1UMet0k_KoA@mail.gmail.com>
 <1ea88c8d1e666b85342374ed7c0ddf7d661e0ee1.camel@redhat.com>
 <CAOQ4uxinsBB-LpGh4h44m6Afv0VT5yWRveDG7sNvE2uJyEGOkg@mail.gmail.com>
 <5fb32a1297821040edd8c19ce796fc0540101653.camel@redhat.com>
 <CAOQ4uxhGX9NVxwsiBMP0q21ZRot6-UA0nGPp1wGNjgmKBjjBBA@mail.gmail.com>
 <b8601c976d6e5d3eccf6ef489da9768ad72f9571.camel@redhat.com>
 <e840d413-c1a7-d047-1a63-468b42571846@linux.alibaba.com> <2ef122849d6f35712b56ffbcc95805672980e185.camel@redhat.com>
 <8ffa28f5-77f6-6bde-5645-5fb799019bca@linux.alibaba.com> <51d9d1b3-2b2a-9b58-2f7f-f3a56c9e04ac@linux.alibaba.com>
 <071074ad149b189661681aada453995741f75039.camel@redhat.com>
 <0d2ef9d6-3b0e-364d-ec2f-c61b19d638e2@linux.alibaba.com> <de57aefc-30e8-470d-bf61-a1cca6514988@linux.alibaba.com>
In-Reply-To: <de57aefc-30e8-470d-bf61-a1cca6514988@linux.alibaba.com>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Thu, 2 Feb 2023 08:37:37 +0200
Message-ID: <CAOQ4uxgS+-MxydqgO8+NQfOs9N881bHNbov28uJYX9XpthPPiw@mail.gmail.com>
Subject: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified
 image filesystem
To:     Gao Xiang <hsiangkao@linux.alibaba.com>
Cc:     Alexander Larsson <alexl@redhat.com>,
        Jingbo Xu <jefflexu@linux.alibaba.com>, gscrivan@redhat.com,
        brauner@kernel.org, linux-fsdevel@vger.kernel.org,
        linux-kernel@vger.kernel.org, david@fromorbit.com,
        viro@zeniv.linux.org.uk, Vivek Goyal <vgoyal@redhat.com>,
        Miklos Szeredi <miklos@szeredi.hu>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Wed, Feb 1, 2023 at 1:22 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
>
>
>
> On 2023/2/1 18:01, Gao Xiang wrote:
> >
> >
> > On 2023/2/1 17:46, Alexander Larsson wrote:
> >
> > ...
> >
> >>>
> >>>                                    | uncached(ms)| cached(ms)
> >>> ----------------------------------|-------------|-----------
> >>> composefs (with digest)           | 326         | 135
> >>> erofs (w/o -T0)                   | 264         | 172
> >>> erofs (w/o -T0) + overlayfs       | 651         | 238
> >>> squashfs (compressed)                | 538         | 211
> >>> squashfs (compressed) + overlayfs | 968         | 302
> >>
> >>
> >> Clearly erofs with sparse files is the best fs now for the ro-fs +
> >> overlay case. But still, we can see that the additional cost of the
> >> overlayfs layer is not negligible.
> >>
> >> According to amir this could be helped by a special composefs-like mode
> >> in overlayfs, but its unclear what performance that would reach, and
> >> we're then talking net new development that further complicates the
> >> overlayfs codebase. Its not clear to me which alternative is easier to
> >> develop/maintain.
> >>
> >> Also, the difference between cached and uncached here is less than in
> >> my tests. Probably because my test image was larger. With the test
> >> image I use, the results are:
> >>
> >>                                    | uncached(ms)| cached(ms)
> >> ----------------------------------|-------------|-----------
> >> composefs (with digest)           | 681         | 390
> >> erofs (w/o -T0) + overlayfs       | 1788        | 532
> >> squashfs (compressed) + overlayfs | 2547        | 443
> >>
> >>
> >> I gotta say it is weird though that squashfs performed better than
> >> erofs in the cached case. May be worth looking into. The test data I'm
> >> using is available here:
> >
> > As another wild guess, cached performance is a just vfs-stuff.
> >
> > I think the performance difference may be due to ACL (since both
> > composefs and squashfs don't support ACL).  I already asked Jingbo
> > to get more "perf data" to analyze this but he's now busy in another
> > stuff.
> >
> > Again, my overall point is quite simple as always, currently
> > composefs is a read-only filesystem with massive symlink-like files.
> > It behaves as a subset of all generic read-only filesystems just
> > for this specific use cases.
> >
> > In facts there are many options to improve this (much like Amir
> > said before):
> >    1) improve overlayfs, and then it can be used with any local fs;
> >
> >    2) enhance erofs to support this (even without on-disk change);
> >
> >    3) introduce fs/composefs;
> >
> > In addition to option 1), option 2) has many benefits as well, since
> > your manifest files can save real regular files in addition to composefs
> > model.
>
> (add some words..)
>
> My first response at that time (on Slack) was "kindly request
> Giuseppe to ask in the fsdevel mailing list if this new overlay model
> and use cases is feasable", if so, I'm much happy to integrate in to
> EROFS (in a cooperative way) in several ways:
>
>   - just use EROFS symlink layout and open such file in a stacked way;
>
> or (now)
>
>   - just identify overlayfs "trusted.overlay.redirect" in EROFS itself
>     and open file so such image can be both used for EROFS only and
>     EROFS + overlayfs.
>
> If that happened, then I think the overlayfs "metacopy" option can
> also be shown by other fs community people later (since I'm not an
> overlay expert), but I'm not sure why they becomes impossible finally
> and even not mentioned at all.
>
> Or if you guys really don't want to use EROFS for whatever reasons
> (EROFS is completely open-source, used, contributed by many vendors),
> you could improve squashfs, ext4, or other exist local fses with this
> new use cases (since they don't need any on-disk change as well, for
> example, by using some xattr), I don't think it's really hard.
>

Engineering-wise, merging composefs features into EROFS
would be the simplest option and FWIW, my personal preference.

However, you need to be aware that this will bring into EROFS
vfs considerations, such as  s_stack_depth nesting (which AFAICS
is not see incremented composefs?). It's not the end of the world, but this
is no longer plain fs over block game. There's a whole new class of bugs
(that syzbot is very eager to explore) so you need to ask yourself whether
this is a direction you want to lead EROFS towards.

Giuseppe expressed his plans to make use of the composefs method
inside userns one day. It is not a hard dependency, but I believe that
keeping the "RO efficient verifiable image format" functionality (EROFS)
separate from "userns composition of verifiable images" (overlayfs)
may benefit the userns mount goal in the long term.

Thanks,
Amir.