Received-SPF: pass (google.com: domain of linux-ext4+bounces-28-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: bugzilla-daemon@kernel.org
To: linux-ext4@vger.kernel.org
Subject: [Bug 217965] ext4(?) regression since 6.5.0 on sata hdd
Date: Fri, 17 Nov 2023 15:39:32 +0000
Message-ID: <bug-217965-13602-fyCk8yWSBF@https.bugzilla.kernel.org/>
In-Reply-To: <bug-217965-13602@https.bugzilla.kernel.org/>
References: <bug-217965-13602@https.bugzilla.kernel.org/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Auto-Submitted: auto-generated
Precedence: bulk
MIME-Version: 1.0

https://bugzilla.kernel.org/show_bug.cgi?id=3D217965

--- Comment #36 from Ojaswin Mujoo (ojaswin.mujoo@ibm.com) ---
Hey Eyal,

So the trace data has given me an idea of what's going on. Basically in ext=
4 we
maintain a list of FS blocks groups (BGs) where each list will have BGs bas=
ed
on the order of free blocks (BG with 64 free blocks goes in list order 6. 6=
40
free blocks goes in order 9 list etc). In our case, we are trying to alloca=
te
stripe size blocks at a time ie 640 blocks or roughly 2.5 KB and ext4 tries=
 to
look at the order 9 list to find a BG that is fit to satisfy our request.=20

Unfortunately there seems to be a lot of BGs in the order 9 list (> 1000) b=
ut
most of them dont have enough free blocks to satisfy the request so we keep
looping=20
and trying to call ext4_mb_good_group() on each of them to see if anyone is
good enough. Once we do find a good enough BG, due to striping we actually =
try
to look for blocks which are specially aligned to stripe size and once we d=
on't
find it we just start looping in the list again from the beginning (!!).

Although I have a good idea now, I'm not able to point my finger at the exa=
ct
change in 6.5 that might have caused this. We did change the allocator to s=
ome
extent and it might be related to this but we need to dig a bit more deeper=
 to
confirm.

Would it be possible to share the same perf record again but this time I'm
adding a few more probes and removing -g so we can fit more in 5MBlimit and
also the commands for Linux 6.4 so we can compare whats changed:

Linux 6.5+:

Probe adding commands:

sudo perf probe -a "ext4_mb_find_good_group_avg_frag_lists order"
sudo perf probe -a "ext4_mb_find_good_group_avg_frag_lists:18 cr
iter->bb_group"
sudo perf probe -a "ext4_mb_good_group:20 free fragments ac->ac_g_ex.fe_len
ac->ac_2order"
sudo perf probe -a "ext4_mb_scan_aligned:26 i max"

Record command:

perf record -e probe:ext4_mb_find_good_group_avg_frag_lists_L18 -e
probe:ext4_mb_good_group_L20 -e probe:ext4_mb_find_good_group_avg_frag_list=
s -e
probe:ext4_mb_    scan_aligned_L26 -e ext4:ext4_mballoc_alloc -p <pid> slee=
p 20


Linux 6.4.x:

Probe adding commands:

sudo perf probe -a "ext4_mb_choose_next_group_cr1:25 i iter->bb_group"
sudo perf probe -a "ext4_mb_good_group:20 free fragments ac->ac_g_ex.fe_len
ac->ac_2order"
sudo perf probe -a "ext4_mb_scan_aligned:26 i max"

Record command:

sudo perf record -e probe:ext4_mb_choose_next_group_cr1_L25 -e
probe:ext4_mb_good_group_L20 -e probe:ext4_mb_scan_aligned_L26 -e
ext4:ext4_mballoc_alloc -p <pid> sleep 20

Thanks again for all your help on this!

Regards,
ojaswin

--=20
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.=