Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CAPQccj4_Tz-11AfXaSiPj4aRWYU2mX9eJuJyGNR68Mini0PZjw@mail.gmail.com>
 <CAPQccj7XwunXerNYxPBTpBa0JVX7vzC=7aBoE8m35ttFHYNOPg@mail.gmail.com>
 <4D72360F-7836-4C4F-920D-4D1BC1DE704E@dilger.ca> <3cc33ea5-ab63-344e-7251-daa808b855bb@thelounge.net>
In-Reply-To: <3cc33ea5-ab63-344e-7251-daa808b855bb@thelounge.net>
From:   Maciej Jablonski <mafjmafj@gmail.com>
Date:   Fri, 14 Aug 2020 11:33:12 +0100
Message-ID: <CAPQccj4SGG4JMcDMdJff3w-BVpv5vfQm2H4DuSk2X8Wu4o5j0Q@mail.gmail.com>
Subject: Re: libext2fs: mkfs.ext3 really slow on centos 8.2
To:     Reindl Harald <h.reindl@thelounge.net>
Cc:     Andreas Dilger <adilger@dilger.ca>, linux-ext4@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Sender: linux-ext4-owner@vger.kernel.org
Precedence: bulk

On Thu, 13 Aug 2020 at 00:14, Reindl Harald <h.reindl@thelounge.net> wrote:
>
>
>
> Am 13.08.20 um 00:45 schrieb Andreas Dilger:
> > On Aug 10, 2020, at 6:37 AM, Maciej Jablonski <mafjmafj@gmail.com> wrote:
> >> On upgrading from centos 7.6 to centos 8.2 mkfs slowed down by orders
> >> of magnitude.
> >>
> >> e.g. 35GB partition from under 8s to 4m+ on the same host.
> >>
> >> Most time is spent on writing the journal to the disk.
> >>
> >> strace shows the following:
> >>
> >> We have got strace which shows that each each block is zeroed with
> >> fallocate and each
> >> invocation of fallocate takes 10ms, this accumulates of course.
> >
> > Do you really need to use mkfs.ext3, or can you use mkfs.ext4 and
> > mount the filesystem as type ext4?  Then you can use the "flexbg"
> > feature and it will not only speed up mkfs but also many other
> > normal operations (e.g. mount, e2fsck, allocation, etc)
>
> typo: it's "flex_bg" and enabled by default (Filesystem created: Sun Aug
>  9 13:24:15 2020)
>
> Filesystem features: has_journal ext_attr resize_inode dir_index
> filetype needs_recovery extent 64bit flex_bg sparse_super large_file
> huge_file dir_nlink extra_isize metadata_csum
>
> ext3 is something of the past for a full decade now

Hi Andreas,

Thanks for the insights,

A bit of background in what circumstances and how widely we see the problem

We run stock OS installers of wide range of linux distros - all
supported versions of RHEL, Ubuntu, Debian, SLES on bare metal
machines  probably 60 different models in total of major brands from
some 5 past generations to current.

And we have noticed that installs of recent distro releases on recent
hardware are just significantly slower,
e.g. RHEL8.0 vs RHEL8.2 went up from 40 minutes to 90 (300GB partition
with default ext3 journal entries) on dell r640 and dell r630
machines. We confirmed at least some of the problems with distros to
be down to mkfs as I mentioned. Note on other machines (older ones)
there is seemingly no difference, we only suspect this might be down
to a disk controller.

We have been historically pegged to ext3, however, it now looks worth
to reconsider ext4.

Thanks,
Maciej