On Mon 18-03-24 18:09:18, Ojaswin Mujoo wrote:
> On Thu, Mar 14, 2024 at 10:09:01PM +0800, Baokun Li wrote:
> > We can trigger a slab-out-of-bounds with the following commands:
> >
> > mkfs.ext4 -F /dev/$disk 10G
> > mount /dev/$disk /tmp/test
> > echo 2147483647 > /sys/fs/ext4/$disk/mb_group_prealloc
> > echo test > /tmp/test/file && sync
> >
> > ==================================================================
> > BUG: KASAN: slab-out-of-bounds in ext4_mb_find_good_group_avg_frag_lists+0x8a/0x200 [ext4]
> > Read of size 8 at addr ffff888121b9d0f0 by task kworker/u2:0/11
> > CPU: 0 PID: 11 Comm: kworker/u2:0 Tainted: GL 6.7.0-next-20240118 #521
> > Call Trace:
> > dump_stack_lvl+0x2c/0x50
> > kasan_report+0xb6/0xf0
> > ext4_mb_find_good_group_avg_frag_lists+0x8a/0x200 [ext4]
> > ext4_mb_regular_allocator+0x19e9/0x2370 [ext4]
> > ext4_mb_new_blocks+0x88a/0x1370 [ext4]
> > ext4_ext_map_blocks+0x14f7/0x2390 [ext4]
> > ext4_map_blocks+0x569/0xea0 [ext4]
> > ext4_do_writepages+0x10f6/0x1bc0 [ext4]
> > [...]
> > ==================================================================
> >
> > The flow of issue triggering is as follows:
> >
> > // Set s_mb_group_prealloc to 2147483647 via sysfs
> > ext4_mb_new_blocks
> > ext4_mb_normalize_request
> > ext4_mb_normalize_group_request
> > ac->ac_g_ex.fe_len = EXT4_SB(sb)->s_mb_group_prealloc
> > ext4_mb_regular_allocator
> > ext4_mb_choose_next_group
> > ext4_mb_choose_next_group_best_avail
> > mb_avg_fragment_size_order
> > order = fls(len) - 2 = 29
> > ext4_mb_find_good_group_avg_frag_lists
> > frag_list = &sbi->s_mb_avg_fragment_size[order]
> > if (list_empty(frag_list)) // Trigger SOOB!
> >
> > At 4k block size, the length of the s_mb_avg_fragment_size list is 14,
> > but an oversized s_mb_group_prealloc is set, causing slab-out-of-bounds
> > to be triggered by an attempt to access an element at index 29.
> >
> > Add a new attr_id attr_clusters_in_group with values in the range
> > [0, sbi->s_clusters_per_group] and declare mb_group_prealloc as
> > that type to fix the issue. In addition avoid returning an order
> > from mb_avg_fragment_size_order() greater than MB_NUM_ORDERS(sb)
> > and reduce some useless loops.
> >
> > Fixes: 7e170922f06b ("ext4: Add allocation criteria 1.5 (CR1_5)")
> > CC: [email protected]
> > Signed-off-by: Baokun Li <[email protected]>
> > Reviewed-by: Jan Kara <[email protected]>
> > ---
> > fs/ext4/mballoc.c | 4 ++++
> > fs/ext4/sysfs.c | 13 ++++++++++++-
> > 2 files changed, 16 insertions(+), 1 deletion(-)
> >
> > diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
> > index 12b3f196010b..48afe5aa228c 100644
> > --- a/fs/ext4/mballoc.c
> > +++ b/fs/ext4/mballoc.c
> > @@ -831,6 +831,8 @@ static int mb_avg_fragment_size_order(struct super_block *sb, ext4_grpblk_t len)
> > return 0;
> > if (order == MB_NUM_ORDERS(sb))
> > order--;
> > + if (WARN_ON_ONCE(order > MB_NUM_ORDERS(sb)))
> > + order = MB_NUM_ORDERS(sb) - 1;
>
> Hey Baokun,
>
> Thanks for fixing this. This patch looks good to me, feel free to add:
>
> Reviewed-by: Ojaswin Mujoo <[email protected]>
>
> my comments after this are less about the patch and more about some
> thoughts on the working of average fragment lists.
>
> So going through the v2 and this patch got me thinking about what really
> is going to happen when a user tries to allocate 32768 blocks which is also
> the maximum value we could have in say ac->ac_g_ex.fe_len.
>
> When this happens, ext4_mb_regular_allocator() will directly set the
> criteria as CR_GOAL_LEN_FAST. Now, we'll follow:
>
> ext4_mb_choose_next_group_goal_fast()
> for (i=mb_avg_fragment_size_order(); i < MB_NUM_ORDERS; i++) { .. }
>
> Here, mb_avg_fragment_siz_order() will do something like:
>
> order = fls(32768) - 2 = 14
> ...
> if (order == MB_NUM_ORDERS(sb))
> order--;
>
> return order;
>
> And we'll look in the fragment list[13] and since none of the groups
> there would have 32768 blocks free (since we dont track it here) we'll
> unnecessarily traverse the full list before falling to CR_BEST_AVAIL_LEN
> (this will become a noop due to the way order and min_order
> are calculated) and eventually to CR_GOAL_LEN_SLOW where we might get
> something or end up splitting.
Yeah, agreed this looks a bit suboptimal. I'm just not 100% sure whether
we'll ever generate a request to allocate 32768 blocks - that would need
verification with tracing - because I have some vague recollection I once
arrived at conclusion this is not possible.
> I think something more optimal would be to:
>
> 1. Add another entry to average fragment lists for completely empty
> groups. (As a sidenote i think we should use something like MB_NUM_FRAG_ORDER
> instead of MB_NUM_ORDERS in calculating limits related to average
> fragment lists since the NUM_ORDERS seems to be the buddy max order ie
> 8192 blocks only valid for CR_POWER2 and shouldn't really limit the
> fragment size lists)
I guess the thinking was that you can never get larger than
1<<(MB_NUM_ORDERS-1) chunk from mballoc so there's no point to keep
fragment lists of such chunks?
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Mon, Mar 18, 2024 at 04:25:09PM +0100, Jan Kara wrote:
> On Mon 18-03-24 18:09:18, Ojaswin Mujoo wrote:
> > On Thu, Mar 14, 2024 at 10:09:01PM +0800, Baokun Li wrote:
> > > We can trigger a slab-out-of-bounds with the following commands:
> > >
> > > mkfs.ext4 -F /dev/$disk 10G
> > > mount /dev/$disk /tmp/test
> > > echo 2147483647 > /sys/fs/ext4/$disk/mb_group_prealloc
> > > echo test > /tmp/test/file && sync
> > >
> > > ==================================================================
> > > BUG: KASAN: slab-out-of-bounds in ext4_mb_find_good_group_avg_frag_lists+0x8a/0x200 [ext4]
> > > Read of size 8 at addr ffff888121b9d0f0 by task kworker/u2:0/11
> > > CPU: 0 PID: 11 Comm: kworker/u2:0 Tainted: GL 6.7.0-next-20240118 #521
> > > Call Trace:
> > > dump_stack_lvl+0x2c/0x50
> > > kasan_report+0xb6/0xf0
> > > ext4_mb_find_good_group_avg_frag_lists+0x8a/0x200 [ext4]
> > > ext4_mb_regular_allocator+0x19e9/0x2370 [ext4]
> > > ext4_mb_new_blocks+0x88a/0x1370 [ext4]
> > > ext4_ext_map_blocks+0x14f7/0x2390 [ext4]
> > > ext4_map_blocks+0x569/0xea0 [ext4]
> > > ext4_do_writepages+0x10f6/0x1bc0 [ext4]
> > > [...]
> > > ==================================================================
> > >
> > > The flow of issue triggering is as follows:
> > >
> > > // Set s_mb_group_prealloc to 2147483647 via sysfs
> > > ext4_mb_new_blocks
> > > ext4_mb_normalize_request
> > > ext4_mb_normalize_group_request
> > > ac->ac_g_ex.fe_len = EXT4_SB(sb)->s_mb_group_prealloc
> > > ext4_mb_regular_allocator
> > > ext4_mb_choose_next_group
> > > ext4_mb_choose_next_group_best_avail
> > > mb_avg_fragment_size_order
> > > order = fls(len) - 2 = 29
> > > ext4_mb_find_good_group_avg_frag_lists
> > > frag_list = &sbi->s_mb_avg_fragment_size[order]
> > > if (list_empty(frag_list)) // Trigger SOOB!
> > >
> > > At 4k block size, the length of the s_mb_avg_fragment_size list is 14,
> > > but an oversized s_mb_group_prealloc is set, causing slab-out-of-bounds
> > > to be triggered by an attempt to access an element at index 29.
> > >
> > > Add a new attr_id attr_clusters_in_group with values in the range
> > > [0, sbi->s_clusters_per_group] and declare mb_group_prealloc as
> > > that type to fix the issue. In addition avoid returning an order
> > > from mb_avg_fragment_size_order() greater than MB_NUM_ORDERS(sb)
> > > and reduce some useless loops.
> > >
> > > Fixes: 7e170922f06b ("ext4: Add allocation criteria 1.5 (CR1_5)")
> > > CC: [email protected]
> > > Signed-off-by: Baokun Li <[email protected]>
> > > Reviewed-by: Jan Kara <[email protected]>
> > > ---
> > > fs/ext4/mballoc.c | 4 ++++
> > > fs/ext4/sysfs.c | 13 ++++++++++++-
> > > 2 files changed, 16 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
> > > index 12b3f196010b..48afe5aa228c 100644
> > > --- a/fs/ext4/mballoc.c
> > > +++ b/fs/ext4/mballoc.c
> > > @@ -831,6 +831,8 @@ static int mb_avg_fragment_size_order(struct super_block *sb, ext4_grpblk_t len)
> > > return 0;
> > > if (order == MB_NUM_ORDERS(sb))
> > > order--;
> > > + if (WARN_ON_ONCE(order > MB_NUM_ORDERS(sb)))
> > > + order = MB_NUM_ORDERS(sb) - 1;
> >
> > Hey Baokun,
> >
> > Thanks for fixing this. This patch looks good to me, feel free to add:
> >
> > Reviewed-by: Ojaswin Mujoo <[email protected]>
> >
> > my comments after this are less about the patch and more about some
> > thoughts on the working of average fragment lists.
> >
> > So going through the v2 and this patch got me thinking about what really
> > is going to happen when a user tries to allocate 32768 blocks which is also
> > the maximum value we could have in say ac->ac_g_ex.fe_len.
> >
> > When this happens, ext4_mb_regular_allocator() will directly set the
> > criteria as CR_GOAL_LEN_FAST. Now, we'll follow:
> >
> > ext4_mb_choose_next_group_goal_fast()
> > for (i=mb_avg_fragment_size_order(); i < MB_NUM_ORDERS; i++) { .. }
> >
> > Here, mb_avg_fragment_siz_order() will do something like:
> >
> > order = fls(32768) - 2 = 14
> > ...
> > if (order == MB_NUM_ORDERS(sb))
> > order--;
> >
> > return order;
> >
> > And we'll look in the fragment list[13] and since none of the groups
> > there would have 32768 blocks free (since we dont track it here) we'll
> > unnecessarily traverse the full list before falling to CR_BEST_AVAIL_LEN
> > (this will become a noop due to the way order and min_order
> > are calculated) and eventually to CR_GOAL_LEN_SLOW where we might get
> > something or end up splitting.
>
> Yeah, agreed this looks a bit suboptimal. I'm just not 100% sure whether
> we'll ever generate a request to allocate 32768 blocks - that would need
> verification with tracing - because I have some vague recollection I once
> arrived at conclusion this is not possible.
Ahh, right! I see the following line in mpage_add_bh_to_extent():
/* Don't go larger than mballoc is willing to allocate */
if (map->m_len >= MAX_WRITEPAGES_EXTENT_LEN)
return false;
Where MAX_WRITEPAGES_EXTENT_LEN is 2048 ie 8MB on 4k filesystem. As pointed out
by your comment there, it seems to come from the fact that we have some restrictions
in ext4_mb_normalize_range() which don't allow us to go beyond 8MB length. I think
I was also looking at that code sometime back and it really needs some rework, I'll
try to test a few things out.
So yep, in the usual paths we shouldn't be sending a request as big as 32768 blocks
but it's still possible with group prealloc with s_mb_group_prealloc set to 32768.
Anyways, thanks for pointing this out, I'll try to look into the code path more to see
how we can optimize it better and maybe if we can lift the 2048 block restriction.
Regards,
ojaswin
>
> > I think something more optimal would be to:
> >
> > 1. Add another entry to average fragment lists for completely empty
> > groups. (As a sidenote i think we should use something like MB_NUM_FRAG_ORDER
> > instead of MB_NUM_ORDERS in calculating limits related to average
> > fragment lists since the NUM_ORDERS seems to be the buddy max order ie
> > 8192 blocks only valid for CR_POWER2 and shouldn't really limit the
> > fragment size lists)
>
> I guess the thinking was that you can never get larger than
> 1<<(MB_NUM_ORDERS-1) chunk from mballoc so there's no point to keep
> fragment lists of such chunks?
>
> Honza
> --
> Jan Kara <[email protected]>
> SUSE Labs, CR