Message-ID: <2ef122849d6f35712b56ffbcc95805672980e185.camel@redhat.com>
Subject: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified
 image filesystem
From:   Alexander Larsson <alexl@redhat.com>
To:     Gao Xiang <hsiangkao@linux.alibaba.com>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
        gscrivan@redhat.com, david@fromorbit.com, brauner@kernel.org,
        viro@zeniv.linux.org.uk, Vivek Goyal <vgoyal@redhat.com>,
        Miklos Szeredi <miklos@szeredi.hu>
Date:   Wed, 25 Jan 2023 11:15:59 +0100
In-Reply-To: <e840d413-c1a7-d047-1a63-468b42571846@linux.alibaba.com>
References: <cover.1674227308.git.alexl@redhat.com>
         <CAOQ4uxgGc33_QVBXMbQTnmbpHio4amv=W7ax2vQ1UMet0k_KoA@mail.gmail.com>
         <1ea88c8d1e666b85342374ed7c0ddf7d661e0ee1.camel@redhat.com>
         <CAOQ4uxinsBB-LpGh4h44m6Afv0VT5yWRveDG7sNvE2uJyEGOkg@mail.gmail.com>
         <5fb32a1297821040edd8c19ce796fc0540101653.camel@redhat.com>
         <CAOQ4uxhGX9NVxwsiBMP0q21ZRot6-UA0nGPp1wGNjgmKBjjBBA@mail.gmail.com>
         <b8601c976d6e5d3eccf6ef489da9768ad72f9571.camel@redhat.com>
         <e840d413-c1a7-d047-1a63-468b42571846@linux.alibaba.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.2 (3.46.2-1.fc37) 
MIME-Version: 1.0
Precedence: bulk

On Wed, 2023-01-25 at 18:05 +0800, Gao Xiang wrote:
>=20
>=20
> On 2023/1/25 17:37, Alexander Larsson wrote:
> > On Tue, 2023-01-24 at 21:06 +0200, Amir Goldstein wrote:
> > > On Tue, Jan 24, 2023 at 3:13 PM Alexander Larsson
> > > <alexl@redhat.com>
>=20
> ...
>=20
> > > >=20
> > > > They are all strictly worse than squashfs in the above testing.
> > > >=20
> > >=20
> > > It's interesting to know why and if an optimized mkfs.erofs
> > > mkfs.ext4 would have done any improvement.
> >=20
> > Even the non-loopback mounted (direct xfs backed) version performed
> > worse than the squashfs one. I'm sure a erofs with sparse files
> > would
> > do better due to a more compact file, but I don't really see how it
> > would perform significantly different than the squashfs code. Yes,
> > squashfs lookup is linear in directory length, while erofs is
> > log(n),
> > but the directories are not so huge that this would dominate the
> > runtime.
> >=20
> > To get an estimate of this I made a broken version of the erofs
> > image,
> > where the metacopy files are actually 0 byte size rather than
> > sparse.
> > This made the erofs file 18M instead, and gained 10% in the cold
> > cache
> > case. This, while good, is not near enough to matter compared to
> > the
> > others.
> >=20
> > I don't think the base performance here is really much dependent on
> > the
> > backing filesystem. An ls -lR workload is just a measurement of the
> > actual (i.e. non-dcache) performance of the filesystem
> > implementation
> > of lookup and iterate, and overlayfs just has more work to do here,
> > especially in terms of the amount of i/o needed.
>=20
> I will form a formal mkfs.erofs version in one or two days since
> we're
> cerebrating Lunar New year now.
>=20
> Since you don't have more I/O traces for analysis, I have to do
> another
> wild guess.
>=20
> Could you help benchmark your v2 too? I'm not sure if such
> performance also exists in v2.=C2=A0 The reason why I guess as this is
> that it seems that you read all dir inode pages when doing the first
> lookup, it can benefit to seq dir access.
>=20
> I'm not sure if EROFS can make a similar number by doing forcing
> readahead on dirs to read all dir data at once as well.
>=20
> Apart from that I don't see significant difference, at least
> personally
> I'd like to know where it could have such huge difference.=C2=A0 I don't
> think that is all because of read-only on-disk format differnce.

I think the performance difference between v2 and v3 would be rather
minor in this case, because I don't think a lot of the directories are
large enough to be split in chunks. I also don't believe erofs and
composefs should fundamentally differ much in performance here, given
that both use a compact binary searchable layout for dirents. However,
the full comparison is "composefs" vs "overlayfs + erofs", and in that
case composefs wins.

--=20
=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D=
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-
=3D-=3D-=3D
 Alexander Larsson                                            Red Hat,
Inc=20
       alexl@redhat.com            alexander.larsson@gmail.com=20
He's an obese Catholic messiah who knows the secret of the alien=20
invasion. She's a provocative Bolivian single mother living on borrowed
time. They fight crime!=20