Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Andreas Dilger <adilger@dilger.ca>
Message-Id: <4D72360F-7836-4C4F-920D-4D1BC1DE704E@dilger.ca>
Content-Type: multipart/signed;
 boundary="Apple-Mail=_FF3BEE4F-DE16-4E0A-A5B1-FE6312B325F0";
 protocol="application/pgp-signature"; micalg=pgp-sha256
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: libext2fs: mkfs.ext3 really slow on centos 8.2
Date:   Wed, 12 Aug 2020 16:45:00 -0600
In-Reply-To: <CAPQccj7XwunXerNYxPBTpBa0JVX7vzC=7aBoE8m35ttFHYNOPg@mail.gmail.com>
Cc:     linux-ext4@vger.kernel.org
To:     Maciej Jablonski <mafjmafj@gmail.com>
References: <CAPQccj4_Tz-11AfXaSiPj4aRWYU2mX9eJuJyGNR68Mini0PZjw@mail.gmail.com>
 <CAPQccj7XwunXerNYxPBTpBa0JVX7vzC=7aBoE8m35ttFHYNOPg@mail.gmail.com>
Sender: linux-ext4-owner@vger.kernel.org
Precedence: bulk


--Apple-Mail=_FF3BEE4F-DE16-4E0A-A5B1-FE6312B325F0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Aug 10, 2020, at 6:37 AM, Maciej Jablonski <mafjmafj@gmail.com> =
wrote:
>=20
> Hi,
>=20
> On upgrading from centos 7.6 to centos 8.2 mkfs slowed down by orders
> of magnitude.
>=20
> e.g. 35GB partition from under 8s to 4m+ on the same host.
>=20
> Most time is spent on writing the journal to the disk.
>=20
> strace shows the following:
>=20
> We have got strace which shows that each each block is zeroed with
> fallocate and each
> invocation of fallocate takes 10ms, this accumulates of course.

Do you really need to use mkfs.ext3, or can you use mkfs.ext4 and
mount the filesystem as type ext4?  Then you can use the "flexbg"
feature and it will not only speed up mkfs but also many other
normal operations (e.g. mount, e2fsck, allocation, etc).

Cheers, Andreas

>=20
> We have found that using
>=20
> UNIX_IO_NOZEROOUT=3D1 to affect libext2fs
>=20
> Brings the timings back in line down to seconds.
>=20
> If this is not a known bug I can send more details,
>=20
> Looks that calling fallocate for each block is very inefficient on =
some system.
> In our case this is dellr640 (skylake) with a mechanical disk.
>=20
> Kind Regards,
>=20
> Maciej
>=20
>=20
> On Mon, 10 Aug 2020 at 13:35, Maciej Jablonski <mafjmafj@gmail.com> =
wrote:
>>=20
>> Hi,
>>=20
>> On upgrading from centos 7.6 to centos 8.2 mkfs slowed down by orders =
of magnitude.
>>=20
>> e.g. 35GB partition from under 8s to 4m+ on the same host.
>>=20
>> Most time is spent on writing the journal to the disk.
>>=20
>> strace shows the following:
>>=20
>> 16:19:49.827056 prctl(PR_GET_DUMPABLE)  =3D 1 (SUID_DUMP_USER)
>> 16:19:49.827112 fallocate(3, FALLOC_FL_ZERO_RANGE, 3383296, 4096) =3D =
0
>> 16:19:49.835203 pwrite64(3, =
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., =
4096, 3362816) =3D 4096
>> 16:19:49.835321 getuid()                =3D 0
>> 16:19:49.835403 geteuid()               =3D 0
>> 16:19:49.835463 getgid()                =3D 0
>> 16:19:49.835513 getegid()               =3D 0
>> 16:19:49.835582 prctl(PR_GET_DUMPABLE)  =3D 1 (SUID_DUMP_USER)
>> 16:19:49.835657 fallocate(3, FALLOC_FL_ZERO_RANGE, 3387392, 4096) =3D =
0
>> 16:19:49.843471 pwrite64(3, =
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., =
4096, 3366912) =3D 4096
>> 16:19:49.843562 getuid()                =3D 0
>> 16:19:49.843619 geteuid()               =3D 0
>> 16:19:49.843669 getgid()                =3D 0
>> 16:19:49.843715 getegid()               =3D 0
>> 16:19:49.843785 prctl(PR_GET_DUMPABLE)  =3D 1 (SUID_DUMP_USER)
>> 16:19:49.843836 fallocate(3, FALLOC_FL_ZERO_RANGE, 3391488, 4096) =3D =
0
>> 16:19:49.851885 pwrite64(3, =
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., =
4096, 3371008) =3D 4096
>>=20
>>=20
>> Each invocation of fallocate takes 10ms, this accumulates of course.
>> We have found that using
>>=20
>> UNIX_IO_NOZEROOUT=3D1 to affect libext2fs
>>=20
>> Brings the timings back in line down to seconds.
>>=20
>> If this is not a known bug I can send more details,
>>=20
>> Looks that calling fallocate for each block is very inefficient on =
some system.
>> In our case this is dellr640 (skylake) with a mechanical disk.
>>=20
>> Kind Regards,
>>=20
>> Maciej
>>=20
>>=20


Cheers, Andreas


--Apple-Mail=_FF3BEE4F-DE16-4E0A-A5B1-FE6312B325F0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIzBAEBCAAdFiEEDb73u6ZejP5ZMprvcqXauRfMH+AFAl80cOwACgkQcqXauRfM
H+DltQ/+JQCRpXqp15M18K2nn80ABG/3QPxAyZ5grqMkya/WmprMT0zqW3IJxmc5
dU4/+cMY4gMomseTvWQh/iFiidb0Ic6z0nl8sXNKqP8T/GqOrhNwh+zcdyfLwX40
c5XSSbIDp6cGg0J7wzjztJdDRL/ol3EIVwQbLdpX6hy2EDvvh6MmwIK6m7w+fstW
rB0q/oPpBcuBueUeWiZhmqLbxj0TGPPGrRGx4sCj8oi9eo02h3sWU1FfeGqMYRb6
kH89iPSzXQ3WnsV6Rbt6OA9CzdkI9XmEvDXXVMKGlEY/G/wWVKZJAdwndFl7kaRq
/BdvP5ox1ge/EbmCLJ+pQy/82VZPppDt9C3GllDhNYZY8gRLhU3tlprQliamdaRT
WWGb34RdIwJkyZs1CNegFcXUCkCVCCrscb/CfJtUuUof6lfjPUUb+5wzjBCM/iG8
9Tq3UeltvXrZUXCJiXzeUL++OSi1GRVdCaCTL+E6TWxyzOOKaCqOAaRpWHiekn6+
oOZW2zGXJhgsJGW9JL6MKb5bNI1MlxlU80C4MAEl6sQuS0lrttGoSYFhqNcXuX6j
xxaGBWGdUatv6P/ccJKavsY3ifHO1jTvQvtpdUaSPEHvSrU4ttCEaQTGxD1lI3MS
bDEVifU0Chas2CLFLHssoKYxa2WUaLFpbkx6dqOszmoj/t54+4s=
=cSaP
-----END PGP SIGNATURE-----

--Apple-Mail=_FF3BEE4F-DE16-4E0A-A5B1-FE6312B325F0--