Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Andreas Dilger <adilger@dilger.ca>
Message-Id: <A0C999DB-A6D5-4C95-A5B8-92E7002395A7@dilger.ca>
Content-Type: multipart/signed;
 boundary="Apple-Mail=_081B02CA-CF63-4E53-9C27-1E112BD24159";
 protocol="application/pgp-signature"; micalg=pgp-sha256
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: e4defrag seems too optimistic
Date:   Thu, 29 Apr 2021 14:47:36 -0600
In-Reply-To: <YIpFK3or2Creo1qg@vapier>
Cc:     linux-ext4@vger.kernel.org
To:     Mike Frysinger <vapier@gentoo.org>
References: <YIpFK3or2Creo1qg@vapier>
Precedence: bulk


--Apple-Mail=_081B02CA-CF63-4E53-9C27-1E112BD24159
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Apr 28, 2021, at 11:33 PM, Mike Frysinger <vapier@gentoo.org> wrote:
>=20
> i started running e4defrag out of curiosity on some large files that =
i'm
> archiving long term.  its results seem exceedingly optimistic and i =
have
> a hard time agreeing with it.  am i pessimistic ?
>=20
> for example, i have a ~4GB archive:
> $ e4defrag -c ./foo.tar.xz
> <File>                                         now/best       size/ext
> ./foo.tar.xz
>                                             39442/2             93 KB
>=20
> Total/best extents				39442/2
> Average size per extent			93 KB
> Fragmentation score				34
> [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
> This file (./foo.tar.xz) does not need defragmentation.
> Done.
>=20
> i have a real hard time seeing this file as barely "a little bit =
fragmented".
> shouldn't the fragmentation score be higher ?

I would tend to agree.  A 4GB file with 39k 100KB extents is not great.
On an HDD with 125 IOPS (not counting track buffers and such) this would
take about 300s to read at a whopping 13MB/s.  On flash, small writes do
lead to increased wear, but the seeks are free and you may not care.

IMHO, anything below 1MB/extent is sub-optimal in terms of IO =
performance,
and a sign of filesystem fragmentation (or a very poor IO pattern), =
since
mballoc should try to do allocation in 8MB chunks for large writes.

In many respects, if the extents are large enough, the "cost" of a seek
hidden by the device bandwidth (e.g. 250 MB/s / 125 seeks/sec =3D 2MB =
for
a good HDD today, scale linearly for RAID-5/6), so any extent larger =
than
this is not limited by seeks. Should 1024 x 4MB extents in a 4GB file be
considered fragmented or not?  Definitely 108KB/extent should be.

However, the "ideal =3D 2" case is bogus, since extents are max size =
128MB,
so you would need at least 32 for a perfect 4GB file.  In that respect,
e4defrag is at best a "working prototype" but I don't think many people
use it, and has not gotten many improvements since it was first landed.
If you have a better idea for a "fragmentation score" I would be open
to looking at it, doubly so if it comes in the form of a patch.

You could check the actual file layout using "fallocate -v" before/after
running e4defrag to see how the allocation was changed.  This would tell
you if it is actually helping or not.  I've thought for a while that it
would be useful to add the same "fragmentation score" to filefrag, but
that would be contingent on the score actually making sense.

You can also use "e2freefrag" to check the filesystem as a whole to see
whether the free space is badly fragmented (i.e. most free chunks < =
8MB).
In that case, running e4defrag _may_ help you, but it is not "smart" =
like
the old DOS defrag utilities, since it just rewrites each file =
separately
instead of having a "plan" for how to defrag the whole filesystem.

> as a measure of "how fragmented is it really", if i copy the file and =
then
> delete the original, there's a noticeable delay before `rm` finishes.

Yes, that would be totally clear if you ran filefrag on the file first.

Cheers, Andreas


--Apple-Mail=_081B02CA-CF63-4E53-9C27-1E112BD24159
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIzBAEBCAAdFiEEDb73u6ZejP5ZMprvcqXauRfMH+AFAmCLG2kACgkQcqXauRfM
H+DYahAAtYliCbrm7xQGyBDRKOnnvJY/4guIHa2IyzU8dETuGXtnaj8gHuBN9Ven
WDQXyAeCGlj29onP89bY4ujvpPtFMboLcjeImm31TgV4oZJvzyeE0mjoHrO1SQPw
bGgVbPIQJPbps1uMHTiWVWpFdih7us+lJFwL248N5SbHoiL+zYE3/rgyWUQ1FR/L
Kv4UMKAW00mZjeHqpwNbhWt6hfnyl2q+0dOInKhvWYsRHK5Ccc9mqhq8jGZhvsKq
hrs4AeV4VQH8xD45nxFaS3OS0XS6wPrVg5D/HEgB8kjqsvVbZK3kBfk0c0IRa4Qb
6Wr50lbupJQ3bSHI4Ah5ouSQ7twQpbao5uEF+hdoWY9gUVFTHIOEZt9TZVNEUchY
9wO2VfM+EPYiLVV4/RH1NF4jLDzpaB5Isd8dCXv1t1EIdzH1tG4Ernjj3veeaBE6
b7IdQX7Q7Bad+eKIWz8zdwc3X2dvNbrcx1ENLT6eCYgPX7I5RC5e3NEuXRqlyIAk
aeX43jBVt+lgVsgE+g3dS6jHme4n9tNRbPOHPH7ZPulQqge1QBFgxdpT9i8mFAZO
8WXlq31zYBUEyuOwfoqTPva9EF+AHxzIjrBnd6yVGnRCifkRaUPCgymDSYI6/5ib
p8TBXi4v1WTW4zc+KBcqcIz00ZoKVUtqHLzeLYaFGfYlB+IBSCo=
=LyN4
-----END PGP SIGNATURE-----

--Apple-Mail=_081B02CA-CF63-4E53-9C27-1E112BD24159--