Message-ID: <0d2ef9d6-3b0e-364d-ec2f-c61b19d638e2@linux.alibaba.com>
Date:   Wed, 1 Feb 2023 18:01:05 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.6.1
Subject: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified
 image filesystem
To:     Alexander Larsson <alexl@redhat.com>,
        Jingbo Xu <jefflexu@linux.alibaba.com>,
        Amir Goldstein <amir73il@gmail.com>, gscrivan@redhat.com,
        brauner@kernel.org
Cc:     linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
        david@fromorbit.com, viro@zeniv.linux.org.uk,
        Vivek Goyal <vgoyal@redhat.com>,
        Miklos Szeredi <miklos@szeredi.hu>
References: <cover.1674227308.git.alexl@redhat.com>
 <CAOQ4uxgGc33_QVBXMbQTnmbpHio4amv=W7ax2vQ1UMet0k_KoA@mail.gmail.com>
 <1ea88c8d1e666b85342374ed7c0ddf7d661e0ee1.camel@redhat.com>
 <CAOQ4uxinsBB-LpGh4h44m6Afv0VT5yWRveDG7sNvE2uJyEGOkg@mail.gmail.com>
 <5fb32a1297821040edd8c19ce796fc0540101653.camel@redhat.com>
 <CAOQ4uxhGX9NVxwsiBMP0q21ZRot6-UA0nGPp1wGNjgmKBjjBBA@mail.gmail.com>
 <b8601c976d6e5d3eccf6ef489da9768ad72f9571.camel@redhat.com>
 <e840d413-c1a7-d047-1a63-468b42571846@linux.alibaba.com>
 <2ef122849d6f35712b56ffbcc95805672980e185.camel@redhat.com>
 <8ffa28f5-77f6-6bde-5645-5fb799019bca@linux.alibaba.com>
 <51d9d1b3-2b2a-9b58-2f7f-f3a56c9e04ac@linux.alibaba.com>
 <071074ad149b189661681aada453995741f75039.camel@redhat.com>
From:   Gao Xiang <hsiangkao@linux.alibaba.com>
In-Reply-To: <071074ad149b189661681aada453995741f75039.camel@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk


On 2023/2/1 17:46, Alexander Larsson wrote:

...

>>
>>                                    | uncached(ms)| cached(ms)
>> ----------------------------------|-------------|-----------
>> composefs (with digest)           | 326         | 135
>> erofs (w/o -T0)                   | 264         | 172
>> erofs (w/o -T0) + overlayfs       | 651         | 238
>> squashfs (compressed)	            | 538         | 211
>> squashfs (compressed) + overlayfs | 968         | 302
> 
> 
> Clearly erofs with sparse files is the best fs now for the ro-fs +
> overlay case. But still, we can see that the additional cost of the
> overlayfs layer is not negligible.
> 
> According to amir this could be helped by a special composefs-like mode
> in overlayfs, but its unclear what performance that would reach, and
> we're then talking net new development that further complicates the
> overlayfs codebase. Its not clear to me which alternative is easier to
> develop/maintain.
> 
> Also, the difference between cached and uncached here is less than in
> my tests. Probably because my test image was larger. With the test
> image I use, the results are:
> 
>                                    | uncached(ms)| cached(ms)
> ----------------------------------|-------------|-----------
> composefs (with digest)           | 681         | 390
> erofs (w/o -T0) + overlayfs       | 1788        | 532
> squashfs (compressed) + overlayfs | 2547        | 443
> 
> 
> I gotta say it is weird though that squashfs performed better than
> erofs in the cached case. May be worth looking into. The test data I'm
> using is available here:

As another wild guess, cached performance is a just vfs-stuff.

I think the performance difference may be due to ACL (since both
composefs and squashfs don't support ACL).  I already asked Jingbo
to get more "perf data" to analyze this but he's now busy in another
stuff.

Again, my overall point is quite simple as always, currently
composefs is a read-only filesystem with massive symlink-like files.
It behaves as a subset of all generic read-only filesystems just
for this specific use cases.

In facts there are many options to improve this (much like Amir
said before):
   1) improve overlayfs, and then it can be used with any local fs;

   2) enhance erofs to support this (even without on-disk change);

   3) introduce fs/composefs;

In addition to option 1), option 2) has many benefits as well, since
your manifest files can save real regular files in addition to composefs
model.

Even if you guys still consider 3), I'm not sure that is all codebase
you will just do bugfix and don't add any new features like what I
said.  So eventually, I still think that is another read-only fs which
is much similar to compressed-part-truncated EROFS.


Thanks,
Gao Xiang


>    
> https://my.owndrive.com/index.php/s/irHJXRpZHtT3a5i
> 
>