Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
From:   Andreas Dilger <adilger@dilger.ca>
Mime-Version: 1.0 (1.0)
Subject: Re: [PATCH 0/2] ext4: Fix performance regression with mballoc
Date:   Sun, 4 Sep 2022 12:32:59 -1000
Message-Id: <46CC2CC6-FB9B-4F6B-AA68-926D28F69592@dilger.ca>
References: <c449eea8-87e4-3f74-5d11-d159eae28c0b@i2se.com>
Cc:     Ojaswin Mujoo <ojaswin@linux.ibm.com>, Jan Kara <jack@suse.cz>,
        Ted Tso <tytso@mit.edu>, linux-ext4@vger.kernel.org,
        Thorsten Leemhuis <regressions@leemhuis.info>,
        Harshad Shirwadkar <harshadshirwadkar@gmail.com>
In-Reply-To: <c449eea8-87e4-3f74-5d11-d159eae28c0b@i2se.com>
To:     Stefan Wahren <stefan.wahren@i2se.com>
Precedence: bulk

On Sep 4, 2022, at 00:01, Stefan Wahren <stefan.wahren@i2se.com> wrote:
>=20
>> Am 27.08.22 um 16:36 schrieb Ojaswin Mujoo:
>>> On Fri, Aug 26, 2022 at 12:15:22PM +0200, Jan Kara wrote:
>>> Hi Stefan,
>>>=20
>>> On Thu 25-08-22 18:57:08, Stefan Wahren wrote:
>>>>> Perhaps if you just download the archive manually, call sync(1), and m=
easure
>>>>> how long it takes to (untar the archive + sync) in mb_optimize_scan=3D=
0/1 we
>>>>> can see whether plain untar is indeed making the difference or there's=

>>>>> something else influencing the result as well (I have checked and
>>>>> rpi-update does a lot of other deleting & copying as the part of the
>>>>> update)? Thanks.
>>>> mb_optimize_scan=3D0 -> almost 5 minutes
>>>>=20
>>>> mb_optimize_scan=3D1 -> almost 18 minutes
>>>>=20
>>>> https://github.com/lategoodbye/mb_optimize_scan_regress/commit/3f3fe8f8=
7881687bb654051942923a6b78f16dec
>>> Thanks! So now the iostat data indeed looks substantially different.
>>>=20
>>>            nooptimize    optimize
>>> Total written        183.6 MB    190.5 MB
>>> Time (recorded)        283 s        1040 s
>>> Avg write request size    79 KB        41 KB
>>>=20
>>> So indeed with mb_optimize_scan=3D1 we do submit substantially smaller
>>> requests on average. So far I'm not sure why that is. Since Ojaswin can
>>> reproduce as well, let's see what he can see from block location info.
>>> Thanks again for help with debugging this and enjoy your vacation!
>>>=20
>> Hi Jan and Stefan,
>>=20
>> Apologies for the delay, I was on leave yesterday and couldn't find time t=
o get to this.
>>=20
>> So I was able to collect the block numbers using the method you suggested=
. I converted the
>> blocks numbers to BG numbers and plotted that data to visualze the alloca=
tion spread. You can
>> find them here:
>>=20
>> mb-opt=3D0, patched kernel: https://github.com/OjaswinM/mbopt-bug/blob/ma=
ster/grpahs/mbopt-0-patched.png
>> mb-opt=3D1, patched kernel: https://github.com/OjaswinM/mbopt-bug/blob/ma=
ster/grpahs/mbopt-1-patched.png
>> mb-opt=3D1, unpatched kernel: https://github.com/OjaswinM/mbopt-bug/blob/=
master/grpahs/mbopt-1-unpatched.png
>>=20
>> Observations:
>> * Before the patched mb_optimize_scan=3D1 allocations were way more sprea=
d out in
>>   40 different BGs.
>> * With the patch, we still allocate in 36 different BGs but majority happ=
en in
>>   just 1 or 2 BGs.
>> * With mb_optimize_scan=3D0, we only allocate in just 7 unique BGs, which=
 could
>>   explain why this is faster.
>=20
> thanks this is very helpful for me to understand. So it seems to me that w=
ith disabled mb_optimized_scan we have a more sequential write behavior and w=
ith enabled mb_optimized_scan a more random write behavior.
>=20
> =46rom my understanding writing small blocks at random addresses of the sd=
 card flash causes a lot of overhead, because the sd card controller need to=
 erase huge blocks (up to 1 MB) before it's able to program the flash pages.=
 This would explain why this series doesn't fix the performance issue, the t=
otal amount of BGs is still much higher.
>=20
> Is this new block allocation pattern a side effect of the optimization or d=
esired?

The goal of the mb_optimized_scan is to avoid a large amount of linear scann=
ing for
very large filesystems when there are many block groups that are full or fra=
gmented.=20

It seems for empty filesystems the new list management is not very ideal. It=
 probably
makes sense to have a hybrid, with some small amount of linear scanning (eg.=
 a meta
block group worth), and then use the new list to find a new group if that do=
esn't find any
group with free space.=20

Cheers, Andreas=