2015-09-17 22:42:24

by Eric Sandeen

Subject: [PATCH] resize2fs: clear uninit BG if allocating from new group

If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
group, nothing clears the UNINIT flag, so it is skipped when we
go to write out modified bitmaps. This leads to post-resize2fs
e2fsck errors; used blocks in UNINIT groups, not marked in the
block bitmap.

This shamelessly cuts & pastes clear_block_uninit() into
resize2fs.c, and my problem goes away.

Signed-off-by: Eric Sandeen <[email protected]>
---

I've kind of lost the thread on resize2fs lately, so maybe this is
a hack job? At least it highlights the issue, even if it's not
quite right. Passes "make check" here and seems ok to me...

Thanks,
-Eric

diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 07c6a0c..0f202bd 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -1614,12 +1614,27 @@ static blk64_t get_new_block(ext2_resize_t rfs)
}
}

+static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
+{
+ if (!ext2fs_has_group_desc_csum(fs) ||
+ !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
+ return;
+
+ /* uninit block bitmaps are now initialized in read_bitmaps() */
+
+ ext2fs_bg_flags_clear(fs, group, EXT2_BG_BLOCK_UNINIT);
+ ext2fs_group_desc_csum_set(fs, group);
+ ext2fs_mark_super_dirty(fs);
+ ext2fs_mark_bb_dirty(fs);
+}
+
static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
blk64_t goal EXT2FS_ATTR((unused)),
blk64_t *ret)
{
ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
blk64_t blk;
+ int group;

blk = get_new_block(rfs);
if (!blk)
@@ -1632,6 +1647,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,

ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
+
+ group = ext2fs_group_of_blk2(rfs->old_fs, blk);
+ clear_block_uninit(rfs->old_fs, group);
+ group = ext2fs_group_of_blk2(rfs->new_fs, blk);
+ clear_block_uninit(rfs->new_fs, group);
+
*ret = (blk64_t) blk;
return 0;
}

2015-09-17 23:55:29

by Darrick J. Wong

Subject: Re: [PATCH] resize2fs: clear uninit BG if allocating from new group

On Thu, Sep 17, 2015 at 05:42:18PM -0500, Eric Sandeen wrote:
> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
> group, nothing clears the UNINIT flag, so it is skipped when we
> go to write out modified bitmaps. This leads to post-resize2fs
> e2fsck errors; used blocks in UNINIT groups, not marked in the
> block bitmap.
>
> This shamelessly cuts & pastes clear_block_uninit() into
> resize2fs.c, and my problem goes away.

Hmm... which test was it that exhibited this error?

> Signed-off-by: Eric Sandeen <[email protected]>
> ---
>
> I've kind of lost the thread on resize2fs lately, so maybe this is
> a hack job? At least it highlights the issue, even if it's not
> quite right. Passes "make check" here and seems ok to me...
>
> Thanks,
> -Eric
>
> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
> index 07c6a0c..0f202bd 100644
> --- a/resize/resize2fs.c
> +++ b/resize/resize2fs.c
> @@ -1614,12 +1614,27 @@ static blk64_t get_new_block(ext2_resize_t rfs)
> }
> }
>
> +static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
> +{
> + if (!ext2fs_has_group_desc_csum(fs) ||
> + !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
> + return;
> +
> + /* uninit block bitmaps are now initialized in read_bitmaps() */
> +
> + ext2fs_bg_flags_clear(fs, group, EXT2_BG_BLOCK_UNINIT);
> + ext2fs_group_desc_csum_set(fs, group);
> + ext2fs_mark_super_dirty(fs);
> + ext2fs_mark_bb_dirty(fs);
> +}
> +
> static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
> blk64_t goal EXT2FS_ATTR((unused)),
> blk64_t *ret)
> {
> ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
> blk64_t blk;
> + int group;
>
> blk = get_new_block(rfs);
> if (!blk)
> @@ -1632,6 +1647,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>
> ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
> ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
> +
> + group = ext2fs_group_of_blk2(rfs->old_fs, blk);
> + clear_block_uninit(rfs->old_fs, group);

Why does the old fs need to have BLOCK_UNINIT cleared?

--D

> + group = ext2fs_group_of_blk2(rfs->new_fs, blk);
> + clear_block_uninit(rfs->new_fs, group);
> +
> *ret = (blk64_t) blk;
> return 0;
> }
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2015-09-18 00:54:13

by Eric Sandeen

Subject: Re: [PATCH] resize2fs: clear uninit BG if allocating from new group

On Sep 17, 2015, at 6:55 PM, Darrick J. Wong <[email protected]> wrote:
>
>> On Thu, Sep 17, 2015 at 05:42:18PM -0500, Eric Sandeen wrote:
>> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
>> group, nothing clears the UNINIT flag, so it is skipped when we
>> go to write out modified bitmaps. This leads to post-resize2fs
>> e2fsck errors; used blocks in UNINIT groups, not marked in the
>> block bitmap.
>>
>> This shamelessly cuts & pastes clear_block_uninit() into
>> resize2fs.c, and my problem goes away.
>
> Hmm... which test was it that exhibited this error?
>
r_ext4_small_bg

Originally it was claimed that only a gcc change exposed it; I'm still trying to make sense of that. But the pre-resize image exhibits it with any recent resizefs when resized up to 2g as the test does.

>> Signed-off-by: Eric Sandeen <[email protected]>
>> ---
>>
>> I've kind of lost the thread on resize2fs lately, so maybe this is
>> a hack job? At least it highlights the issue, even if it's not
>> quite right. Passes "make check" here and seems ok to me...
>>
>> Thanks,
>> -Eric
>>
>> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
>> index 07c6a0c..0f202bd 100644
>> --- a/resize/resize2fs.c
>> +++ b/resize/resize2fs.c
>> @@ -1614,12 +1614,27 @@ static blk64_t get_new_block(ext2_resize_t rfs)
>> }
>> }
>>
>> +static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
>> +{
>> + if (!ext2fs_has_group_desc_csum(fs) ||
>> + !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
>> + return;
>> +
>> + /* uninit block bitmaps are now initialized in read_bitmaps() */
>> +
>> + ext2fs_bg_flags_clear(fs, group, EXT2_BG_BLOCK_UNINIT);
>> + ext2fs_group_desc_csum_set(fs, group);
>> + ext2fs_mark_super_dirty(fs);
>> + ext2fs_mark_bb_dirty(fs);
>> +}
>> +
>> static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>> blk64_t goal EXT2FS_ATTR((unused)),
>> blk64_t *ret)
>> {
>> ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
>> blk64_t blk;
>> + int group;
>>
>> blk = get_new_block(rfs);
>> if (!blk)
>> @@ -1632,6 +1647,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>>
>> ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
>> ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
>> +
>> + group = ext2fs_group_of_blk2(rfs->old_fs, blk);
>> + clear_block_uninit(rfs->old_fs, group);
>
> Why does the old fs need to have BLOCK_UNINIT cleared?
>
Same reason we mark the block used in the old fs? TBH I didn't understand why we do that either, but sinc we do....

Eric

> --D
>
>> + group = ext2fs_group_of_blk2(rfs->new_fs, blk);
>> + clear_block_uninit(rfs->new_fs, group);
>> +
>> *ret = (blk64_t) blk;
>> return 0;
>> }
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html

2015-09-18 03:19:13

by Eric Sandeen

Subject: Re: [PATCH] resize2fs: clear uninit BG if allocating from new group

On 9/17/15 7:54 PM, Eric Sandeen wrote:
> On Sep 17, 2015, at 6:55 PM, Darrick J. Wong <[email protected]> wrote:
>>
>>> On Thu, Sep 17, 2015 at 05:42:18PM -0500, Eric Sandeen wrote:
>>> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
>>> group, nothing clears the UNINIT flag, so it is skipped when we
>>> go to write out modified bitmaps. This leads to post-resize2fs
>>> e2fsck errors; used blocks in UNINIT groups, not marked in the
>>> block bitmap.
>>>
>>> This shamelessly cuts & pastes clear_block_uninit() into
>>> resize2fs.c, and my problem goes away.
>>
>> Hmm... which test was it that exhibited this error?
>>
> r_ext4_small_bg
>
> Originally it was claimed that only a gcc change exposed it; I'm
> still trying to make sense of that. But the pre-resize image exhibits
> it with any recent resizefs when resized up to 2g as the test does.

I think the only reason the different gcc tweaked the bug is that the
test copies the e2fsck/e2fsck binary into the filesystem under test,
and the size changes depending on the compiler.

I can put up an fs image that exhibits it if you're curious...

-Eric

2016-01-14 14:09:54

by Eric Sandeen

Subject: Re: [PATCH] resize2fs: clear uninit BG if allocating from new group

On 9/17/15 5:42 PM, Eric Sandeen wrote:
> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
> group, nothing clears the UNINIT flag, so it is skipped when we
> go to write out modified bitmaps. This leads to post-resize2fs
> e2fsck errors; used blocks in UNINIT groups, not marked in the
> block bitmap.
>
> This shamelessly cuts & pastes clear_block_uninit() into
> resize2fs.c, and my problem goes away.

Ping on this?

> Signed-off-by: Eric Sandeen <[email protected]>
> ---
>
> I've kind of lost the thread on resize2fs lately, so maybe this is
> a hack job? At least it highlights the issue, even if it's not
> quite right. Passes "make check" here and seems ok to me...
>
> Thanks,
> -Eric
>
> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
> index 07c6a0c..0f202bd 100644
> --- a/resize/resize2fs.c
> +++ b/resize/resize2fs.c
> @@ -1614,12 +1614,27 @@ static blk64_t get_new_block(ext2_resize_t rfs)
> }
> }
>
> +static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
> +{
> + if (!ext2fs_has_group_desc_csum(fs) ||
> + !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
> + return;
> +
> + /* uninit block bitmaps are now initialized in read_bitmaps() */
> +
> + ext2fs_bg_flags_clear(fs, group, EXT2_BG_BLOCK_UNINIT);
> + ext2fs_group_desc_csum_set(fs, group);
> + ext2fs_mark_super_dirty(fs);
> + ext2fs_mark_bb_dirty(fs);
> +}
> +
> static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
> blk64_t goal EXT2FS_ATTR((unused)),
> blk64_t *ret)
> {
> ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
> blk64_t blk;
> + int group;
>
> blk = get_new_block(rfs);
> if (!blk)
> @@ -1632,6 +1647,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>
> ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
> ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
> +
> + group = ext2fs_group_of_blk2(rfs->old_fs, blk);
> + clear_block_uninit(rfs->old_fs, group);
> + group = ext2fs_group_of_blk2(rfs->new_fs, blk);
> + clear_block_uninit(rfs->new_fs, group);
> +
> *ret = (blk64_t) blk;
> return 0;
> }
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

2016-01-14 17:20:27

by Andreas Dilger

Subject: Re: [PATCH] resize2fs: clear uninit BG if allocating from new group

On Jan 14, 2016, at 07:09, Eric Sandeen <[email protected]> wrote:
>
>> On 9/17/15 5:42 PM, Eric Sandeen wrote:
>> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
>> group, nothing clears the UNINIT flag, so it is skipped when we
>> go to write out modified bitmaps. This leads to post-resize2fs
>> e2fsck errors; used blocks in UNINIT groups, not marked in the
>> block bitmap.
>>
>> This shamelessly cuts & pastes clear_block_uninit() into
>> resize2fs.c, and my problem goes away.
>
> Ping on this?

My only minor concern is about the code duplication. Is there a common place this could be moved?

Cheers, Andreas

>> Signed-off-by: Eric Sandeen <[email protected]>
>> ---
>>
>> I've kind of lost the thread on resize2fs lately, so maybe this is
>> a hack job? At least it highlights the issue, even if it's not
>> quite right. Passes "make check" here and seems ok to me...
>>
>> Thanks,
>> -Eric
>>
>> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
>> index 07c6a0c..0f202bd 100644
>> --- a/resize/resize2fs.c
>> +++ b/resize/resize2fs.c
>> @@ -1614,12 +1614,27 @@ static blk64_t get_new_block(ext2_resize_t rfs)
>> }
>> }
>>
>> +static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
>> +{
>> + if (!ext2fs_has_group_desc_csum(fs) ||
>> + !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
>> + return;
>> +
>> + /* uninit block bitmaps are now initialized in read_bitmaps() */
>> +
>> + ext2fs_bg_flags_clear(fs, group, EXT2_BG_BLOCK_UNINIT);
>> + ext2fs_group_desc_csum_set(fs, group);
>> + ext2fs_mark_super_dirty(fs);
>> + ext2fs_mark_bb_dirty(fs);
>> +}
>> +
>> static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>> blk64_t goal EXT2FS_ATTR((unused)),
>> blk64_t *ret)
>> {
>> ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
>> blk64_t blk;
>> + int group;
>>
>> blk = get_new_block(rfs);
>> if (!blk)
>> @@ -1632,6 +1647,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>>
>> ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
>> ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
>> +
>> + group = ext2fs_group_of_blk2(rfs->old_fs, blk);
>> + clear_block_uninit(rfs->old_fs, group);
>> + group = ext2fs_group_of_blk2(rfs->new_fs, blk);
>> + clear_block_uninit(rfs->new_fs, group);
>> +
>> *ret = (blk64_t) blk;
>> return 0;
>> }
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2016-01-14 18:46:05

by Eric Sandeen

Subject: [PATCH V2] resize2fs: clear uninit BG if allocating from new group

If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
group, nothing clears the UNINIT flag, so it is skipped when we
go to write out modified bitmaps. This leads to post-resize2fs
e2fsck errors; used blocks in UNINIT groups, not marked in the
block bitmap. This was seen on r_ext4_small_bg.

This patch uses clear_block_uninit() to clear the flag,
and my problem goes away.

Signed-off-by: Eric Sandeen <[email protected]>
---

V2: don't cut and paste, sorry! :)

Thanks,
-Eric

diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index 86e7f99..9a575fd 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -39,7 +39,7 @@
/*
* Clear the uninit block bitmap flag if necessary
*/
-static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
+void clear_block_uninit(ext2_filsys fs, dgrp_t group)
{
if (!ext2fs_has_group_desc_csum(fs) ||
!(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 86d860f..0dca002 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -647,6 +647,7 @@ static inline int ext2fs_needs_large_file_feature(unsigned long long file_size)
}

/* alloc.c */
+extern void clear_block_uninit(ext2_filsys fs, dgrp_t group);
extern errcode_t ext2fs_new_inode(ext2_filsys fs, ext2_ino_t dir, int mode,
ext2fs_inode_bitmap map, ext2_ino_t *ret);
extern errcode_t ext2fs_new_block(ext2_filsys fs, blk_t goal,
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 07c6a0c..ffe9473 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -1620,6 +1620,7 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
{
ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
blk64_t blk;
+ int group;

blk = get_new_block(rfs);
if (!blk)
@@ -1632,6 +1633,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,

ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
+
+ group = ext2fs_group_of_blk2(rfs->old_fs, blk);
+ clear_block_uninit(rfs->old_fs, group);
+ group = ext2fs_group_of_blk2(rfs->new_fs, blk);
+ clear_block_uninit(rfs->new_fs, group);
+
*ret = (blk64_t) blk;
return 0;
}

2016-01-14 18:50:48

by Darrick J. Wong

Subject: Re: [PATCH V2] resize2fs: clear uninit BG if allocating from new group

On Thu, Jan 14, 2016 at 12:46:03PM -0600, Eric Sandeen wrote:
> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
> group, nothing clears the UNINIT flag, so it is skipped when we
> go to write out modified bitmaps. This leads to post-resize2fs
> e2fsck errors; used blocks in UNINIT groups, not marked in the
> block bitmap. This was seen on r_ext4_small_bg.
>
> This patch uses clear_block_uninit() to clear the flag,
> and my problem goes away.
>
> Signed-off-by: Eric Sandeen <[email protected]>
> ---
>
> V2: don't cut and paste, sorry! :)
>
> Thanks,
> -Eric
>
> diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
> index 86e7f99..9a575fd 100644
> --- a/lib/ext2fs/alloc.c
> +++ b/lib/ext2fs/alloc.c
> @@ -39,7 +39,7 @@
> /*
> * Clear the uninit block bitmap flag if necessary
> */
> -static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
> +void clear_block_uninit(ext2_filsys fs, dgrp_t group)
> {
> if (!ext2fs_has_group_desc_csum(fs) ||
> !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
> diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
> index 86d860f..0dca002 100644
> --- a/lib/ext2fs/ext2fs.h
> +++ b/lib/ext2fs/ext2fs.h
> @@ -647,6 +647,7 @@ static inline int ext2fs_needs_large_file_feature(unsigned long long file_size)
> }
>
> /* alloc.c */
> +extern void clear_block_uninit(ext2_filsys fs, dgrp_t group);

If you're going to expose this as a new API call then this needs to be
ext2fs_clear_block_uninit().

(Do we want a flags arg just in case we ever need one?)

--D

> extern errcode_t ext2fs_new_inode(ext2_filsys fs, ext2_ino_t dir, int mode,
> ext2fs_inode_bitmap map, ext2_ino_t *ret);
> extern errcode_t ext2fs_new_block(ext2_filsys fs, blk_t goal,
> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
> index 07c6a0c..ffe9473 100644
> --- a/resize/resize2fs.c
> +++ b/resize/resize2fs.c
> @@ -1620,6 +1620,7 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
> {
> ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
> blk64_t blk;
> + int group;
>
> blk = get_new_block(rfs);
> if (!blk)
> @@ -1632,6 +1633,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>
> ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
> ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
> +
> + group = ext2fs_group_of_blk2(rfs->old_fs, blk);
> + clear_block_uninit(rfs->old_fs, group);
> + group = ext2fs_group_of_blk2(rfs->new_fs, blk);
> + clear_block_uninit(rfs->new_fs, group);
> +
> *ret = (blk64_t) blk;
> return 0;
> }
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2016-01-14 19:26:23

by Eric Sandeen

Subject: Re: [PATCH V2] resize2fs: clear uninit BG if allocating from new group

On 1/14/16 12:50 PM, Darrick J. Wong wrote:
> On Thu, Jan 14, 2016 at 12:46:03PM -0600, Eric Sandeen wrote:
...

>> diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
>> index 86d860f..0dca002 100644
>> --- a/lib/ext2fs/ext2fs.h
>> +++ b/lib/ext2fs/ext2fs.h
>> @@ -647,6 +647,7 @@ static inline int ext2fs_needs_large_file_feature(unsigned long long file_size)
>> }
>>
>> /* alloc.c */
>> +extern void clear_block_uninit(ext2_filsys fs, dgrp_t group);
>
> If you're going to expose this as a new API call then this needs to be
> ext2fs_clear_block_uninit().

Hm, yes. I tried to work around it but don't see a nice way.

> (Do we want a flags arg just in case we ever need one?)

No. ;)

(EXT2_CLEAR_GROUP_UNINIT_HALFWAY? I can't really think of any reasonable
variation on "clear this group's uninit status")

-Eric

2016-01-14 19:27:27

by Eric Sandeen

Subject: [PATCH V3] resize2fs: clear uninit BG if allocating from new group

If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
group, nothing clears the UNINIT flag, so it is skipped when we
go to write out modified bitmaps. This leads to post-resize2fs
e2fsck errors; used blocks in UNINIT groups, not marked in the
block bitmap. This was seen on r_ext4_small_bg.

This patch uses clear_block_uninit() to clear the flag,
and my problem goes away.

Signed-off-by: Eric Sandeen <[email protected]>
---

V2: don't cut and paste, sorry!
V3: ext2fs_* namespace for the exported function, thanks Darrick

Thanks,
-Eric

diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
index 86e7f99..ce59e85 100644
--- a/lib/ext2fs/alloc.c
+++ b/lib/ext2fs/alloc.c
@@ -39,7 +39,7 @@
/*
* Clear the uninit block bitmap flag if necessary
*/
-static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
+void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group)
{
if (!ext2fs_has_group_desc_csum(fs) ||
!(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
@@ -183,7 +183,7 @@ allocated:
if (retval)
return retval;

- clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
+ ext2fs_clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
*ret = b;
return 0;
}
@@ -455,7 +455,7 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
allocated:
for (b = start; b < end;
b += fs->super->s_blocks_per_group)
- clear_block_uninit(fs,
+ ext2fs_clear_block_uninit(fs,
ext2fs_group_of_blk2(fs, b));
return 0;
}
diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
index 86d860f..d5e9684 100644
--- a/lib/ext2fs/ext2fs.h
+++ b/lib/ext2fs/ext2fs.h
@@ -647,6 +647,7 @@ static inline int ext2fs_needs_large_file_feature(unsigned long long file_size)
}

/* alloc.c */
+extern void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group);
extern errcode_t ext2fs_new_inode(ext2_filsys fs, ext2_ino_t dir, int mode,
ext2fs_inode_bitmap map, ext2_ino_t *ret);
extern errcode_t ext2fs_new_block(ext2_filsys fs, blk_t goal,
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 07c6a0c..42435b2 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -1620,6 +1620,7 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
{
ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
blk64_t blk;
+ int group;

blk = get_new_block(rfs);
if (!blk)
@@ -1632,6 +1633,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,

ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
+
+ group = ext2fs_group_of_blk2(rfs->old_fs, blk);
+ ext2fs_clear_block_uninit(rfs->old_fs, group);
+ group = ext2fs_group_of_blk2(rfs->new_fs, blk);
+ ext2fs_clear_block_uninit(rfs->new_fs, group);
+
*ret = (blk64_t) blk;
return 0;
}

2016-01-14 19:38:57

by Darrick J. Wong

Subject: Re: [PATCH V3] resize2fs: clear uninit BG if allocating from new group

On Thu, Jan 14, 2016 at 01:27:26PM -0600, Eric Sandeen wrote:
> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
> group, nothing clears the UNINIT flag, so it is skipped when we
> go to write out modified bitmaps. This leads to post-resize2fs
> e2fsck errors; used blocks in UNINIT groups, not marked in the
> block bitmap. This was seen on r_ext4_small_bg.
>
> This patch uses clear_block_uninit() to clear the flag,
> and my problem goes away.

Looks reasonable,
Reviewed-by: Darrick J. Wong <[email protected]>

--D

>
> Signed-off-by: Eric Sandeen <[email protected]>
> ---
>
> V2: don't cut and paste, sorry!
> V3: ext2fs_* namespace for the exported function, thanks Darrick
>
> Thanks,
> -Eric
>
> diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
> index 86e7f99..ce59e85 100644
> --- a/lib/ext2fs/alloc.c
> +++ b/lib/ext2fs/alloc.c
> @@ -39,7 +39,7 @@
> /*
> * Clear the uninit block bitmap flag if necessary
> */
> -static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
> +void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group)
> {
> if (!ext2fs_has_group_desc_csum(fs) ||
> !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
> @@ -183,7 +183,7 @@ allocated:
> if (retval)
> return retval;
>
> - clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
> + ext2fs_clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
> *ret = b;
> return 0;
> }
> @@ -455,7 +455,7 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
> allocated:
> for (b = start; b < end;
> b += fs->super->s_blocks_per_group)
> - clear_block_uninit(fs,
> + ext2fs_clear_block_uninit(fs,
> ext2fs_group_of_blk2(fs, b));
> return 0;
> }
> diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
> index 86d860f..d5e9684 100644
> --- a/lib/ext2fs/ext2fs.h
> +++ b/lib/ext2fs/ext2fs.h
> @@ -647,6 +647,7 @@ static inline int ext2fs_needs_large_file_feature(unsigned long long file_size)
> }
>
> /* alloc.c */
> +extern void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group);
> extern errcode_t ext2fs_new_inode(ext2_filsys fs, ext2_ino_t dir, int mode,
> ext2fs_inode_bitmap map, ext2_ino_t *ret);
> extern errcode_t ext2fs_new_block(ext2_filsys fs, blk_t goal,
> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
> index 07c6a0c..42435b2 100644
> --- a/resize/resize2fs.c
> +++ b/resize/resize2fs.c
> @@ -1620,6 +1620,7 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
> {
> ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
> blk64_t blk;
> + int group;
>
> blk = get_new_block(rfs);
> if (!blk)
> @@ -1632,6 +1633,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>
> ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
> ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
> +
> + group = ext2fs_group_of_blk2(rfs->old_fs, blk);
> + ext2fs_clear_block_uninit(rfs->old_fs, group);
> + group = ext2fs_group_of_blk2(rfs->new_fs, blk);
> + ext2fs_clear_block_uninit(rfs->new_fs, group);
> +
> *ret = (blk64_t) blk;
> return 0;
> }
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2016-01-14 19:39:31

by Andreas Dilger

Subject: Re: [PATCH V3] resize2fs: clear uninit BG if allocating from new group

On Jan 14, 2016, at 12:27 PM, Eric Sandeen <[email protected]> wrote:
>
> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
> group, nothing clears the UNINIT flag, so it is skipped when we
> go to write out modified bitmaps. This leads to post-resize2fs
> e2fsck errors; used blocks in UNINIT groups, not marked in the
> block bitmap. This was seen on r_ext4_small_bg.
>
> This patch uses clear_block_uninit() to clear the flag,
> and my problem goes away.
>
> Signed-off-by: Eric Sandeen <[email protected]>

Reviewed-by: Andreas Dilger <[email protected]>

> ---
>
> V2: don't cut and paste, sorry!
> V3: ext2fs_* namespace for the exported function, thanks Darrick
>
> Thanks,
> -Eric
>
> diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
> index 86e7f99..ce59e85 100644
> --- a/lib/ext2fs/alloc.c
> +++ b/lib/ext2fs/alloc.c
> @@ -39,7 +39,7 @@
> /*
> * Clear the uninit block bitmap flag if necessary
> */
> -static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
> +void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group)
> {
> if (!ext2fs_has_group_desc_csum(fs) ||
> !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
> @@ -183,7 +183,7 @@ allocated:
> if (retval)
> return retval;
>
> - clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
> + ext2fs_clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
> *ret = b;
> return 0;
> }
> @@ -455,7 +455,7 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
> allocated:
> for (b = start; b < end;
> b += fs->super->s_blocks_per_group)
> - clear_block_uninit(fs,
> + ext2fs_clear_block_uninit(fs,
> ext2fs_group_of_blk2(fs, b));
> return 0;
> }
> diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
> index 86d860f..d5e9684 100644
> --- a/lib/ext2fs/ext2fs.h
> +++ b/lib/ext2fs/ext2fs.h
> @@ -647,6 +647,7 @@ static inline int ext2fs_needs_large_file_feature(unsigned long long file_size)
> }
>
> /* alloc.c */
> +extern void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group);
> extern errcode_t ext2fs_new_inode(ext2_filsys fs, ext2_ino_t dir, int mode,
> ext2fs_inode_bitmap map, ext2_ino_t *ret);
> extern errcode_t ext2fs_new_block(ext2_filsys fs, blk_t goal,
> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
> index 07c6a0c..42435b2 100644
> --- a/resize/resize2fs.c
> +++ b/resize/resize2fs.c
> @@ -1620,6 +1620,7 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
> {
> ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
> blk64_t blk;
> + int group;
>
> blk = get_new_block(rfs);
> if (!blk)
> @@ -1632,6 +1633,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>
> ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
> ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
> +
> + group = ext2fs_group_of_blk2(rfs->old_fs, blk);
> + ext2fs_clear_block_uninit(rfs->old_fs, group);
> + group = ext2fs_group_of_blk2(rfs->new_fs, blk);
> + ext2fs_clear_block_uninit(rfs->new_fs, group);
> +
> *ret = (blk64_t) blk;
> return 0;
> }
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html


Cheers, Andreas

Attachments:

2016-01-14 19:57:15

by Eric Sandeen

Subject: Re: [PATCH V3] resize2fs: clear uninit BG if allocating from new group

On 1/14/16 1:27 PM, Eric Sandeen wrote:
> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
> group, nothing clears the UNINIT flag, so it is skipped when we
> go to write out modified bitmaps. This leads to post-resize2fs
> e2fsck errors; used blocks in UNINIT groups, not marked in the
> block bitmap. This was seen on r_ext4_small_bg.
>
> This patch uses clear_block_uninit() to clear the flag,
> and my problem goes away.

Ted, if you want to clarify the commit log, I realized that the
issue here is that although we are called via ->get_alloc_block,
and the caller clears uninit after the call, in resize-land we
have *2* filesystems to deal with, hanging off the fs private
pointer. Explicitly clearing uninit in both filesystems
in the resize code seems like the safest path.

> Signed-off-by: Eric Sandeen <[email protected]>
> ---
>
> V2: don't cut and paste, sorry!
> V3: ext2fs_* namespace for the exported function, thanks Darrick
>
> Thanks,
> -Eric
>
> diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
> index 86e7f99..ce59e85 100644
> --- a/lib/ext2fs/alloc.c
> +++ b/lib/ext2fs/alloc.c
> @@ -39,7 +39,7 @@
> /*
> * Clear the uninit block bitmap flag if necessary
> */
> -static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
> +void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group)
> {
> if (!ext2fs_has_group_desc_csum(fs) ||
> !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
> @@ -183,7 +183,7 @@ allocated:
> if (retval)
> return retval;
>
> - clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
> + ext2fs_clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
> *ret = b;
> return 0;
> }
> @@ -455,7 +455,7 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
> allocated:
> for (b = start; b < end;
> b += fs->super->s_blocks_per_group)
> - clear_block_uninit(fs,
> + ext2fs_clear_block_uninit(fs,
> ext2fs_group_of_blk2(fs, b));
> return 0;
> }
> diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
> index 86d860f..d5e9684 100644
> --- a/lib/ext2fs/ext2fs.h
> +++ b/lib/ext2fs/ext2fs.h
> @@ -647,6 +647,7 @@ static inline int ext2fs_needs_large_file_feature(unsigned long long file_size)
> }
>
> /* alloc.c */
> +extern void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group);
> extern errcode_t ext2fs_new_inode(ext2_filsys fs, ext2_ino_t dir, int mode,
> ext2fs_inode_bitmap map, ext2_ino_t *ret);
> extern errcode_t ext2fs_new_block(ext2_filsys fs, blk_t goal,
> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
> index 07c6a0c..42435b2 100644
> --- a/resize/resize2fs.c
> +++ b/resize/resize2fs.c
> @@ -1620,6 +1620,7 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
> {
> ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
> blk64_t blk;
> + int group;
>
> blk = get_new_block(rfs);
> if (!blk)
> @@ -1632,6 +1633,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>
> ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
> ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
> +
> + group = ext2fs_group_of_blk2(rfs->old_fs, blk);
> + ext2fs_clear_block_uninit(rfs->old_fs, group);
> + group = ext2fs_group_of_blk2(rfs->new_fs, blk);
> + ext2fs_clear_block_uninit(rfs->new_fs, group);
> +
> *ret = (blk64_t) blk;
> return 0;
> }
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

2016-02-19 22:21:53

by Eric Sandeen

Subject: Re: [PATCH V3] resize2fs: clear uninit BG if allocating from new group

On 1/14/16 1:27 PM, Eric Sandeen wrote:
> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
> group, nothing clears the UNINIT flag, so it is skipped when we
> go to write out modified bitmaps. This leads to post-resize2fs
> e2fsck errors; used blocks in UNINIT groups, not marked in the
> block bitmap. This was seen on r_ext4_small_bg.
>
> This patch uses clear_block_uninit() to clear the flag,
> and my problem goes away.
>
> Signed-off-by: Eric Sandeen <[email protected]>
> ---
>
> V2: don't cut and paste, sorry!
> V3: ext2fs_* namespace for the exported function, thanks Darrick

Ted, any chance this could get merged? It has 2 reviews now.

Thanks,
-Eric

> Thanks,
> -Eric
>
> diff --git a/lib/ext2fs/alloc.c b/lib/ext2fs/alloc.c
> index 86e7f99..ce59e85 100644
> --- a/lib/ext2fs/alloc.c
> +++ b/lib/ext2fs/alloc.c
> @@ -39,7 +39,7 @@
> /*
> * Clear the uninit block bitmap flag if necessary
> */
> -static void clear_block_uninit(ext2_filsys fs, dgrp_t group)
> +void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group)
> {
> if (!ext2fs_has_group_desc_csum(fs) ||
> !(ext2fs_bg_flags_test(fs, group, EXT2_BG_BLOCK_UNINIT)))
> @@ -183,7 +183,7 @@ allocated:
> if (retval)
> return retval;
>
> - clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
> + ext2fs_clear_block_uninit(fs, ext2fs_group_of_blk2(fs, b));
> *ret = b;
> return 0;
> }
> @@ -455,7 +455,7 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
> allocated:
> for (b = start; b < end;
> b += fs->super->s_blocks_per_group)
> - clear_block_uninit(fs,
> + ext2fs_clear_block_uninit(fs,
> ext2fs_group_of_blk2(fs, b));
> return 0;
> }
> diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h
> index 86d860f..d5e9684 100644
> --- a/lib/ext2fs/ext2fs.h
> +++ b/lib/ext2fs/ext2fs.h
> @@ -647,6 +647,7 @@ static inline int ext2fs_needs_large_file_feature(unsigned long long file_size)
> }
>
> /* alloc.c */
> +extern void ext2fs_clear_block_uninit(ext2_filsys fs, dgrp_t group);
> extern errcode_t ext2fs_new_inode(ext2_filsys fs, ext2_ino_t dir, int mode,
> ext2fs_inode_bitmap map, ext2_ino_t *ret);
> extern errcode_t ext2fs_new_block(ext2_filsys fs, blk_t goal,
> diff --git a/resize/resize2fs.c b/resize/resize2fs.c
> index 07c6a0c..42435b2 100644
> --- a/resize/resize2fs.c
> +++ b/resize/resize2fs.c
> @@ -1620,6 +1620,7 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
> {
> ext2_resize_t rfs = (ext2_resize_t) fs->priv_data;
> blk64_t blk;
> + int group;
>
> blk = get_new_block(rfs);
> if (!blk)
> @@ -1632,6 +1633,12 @@ static errcode_t resize2fs_get_alloc_block(ext2_filsys fs,
>
> ext2fs_mark_block_bitmap2(rfs->old_fs->block_map, blk);
> ext2fs_mark_block_bitmap2(rfs->new_fs->block_map, blk);
> +
> + group = ext2fs_group_of_blk2(rfs->old_fs, blk);
> + ext2fs_clear_block_uninit(rfs->old_fs, group);
> + group = ext2fs_group_of_blk2(rfs->new_fs, blk);
> + ext2fs_clear_block_uninit(rfs->new_fs, group);
> +
> *ret = (blk64_t) blk;
> return 0;
> }
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

2016-03-07 02:51:56

by Theodore Ts'o

Subject: Re: [PATCH V3] resize2fs: clear uninit BG if allocating from new group

On Thu, Jan 14, 2016 at 01:27:26PM -0600, Eric Sandeen wrote:
> If resize2fs_get_alloc_block() allocates from a BLOCK_UNINIT
> group, nothing clears the UNINIT flag, so it is skipped when we
> go to write out modified bitmaps. This leads to post-resize2fs
> e2fsck errors; used blocks in UNINIT groups, not marked in the
> block bitmap. This was seen on r_ext4_small_bg.
>
> This patch uses clear_block_uninit() to clear the flag,
> and my problem goes away.
>
> Signed-off-by: Eric Sandeen <[email protected]>

Thanks, applied. Apologies for the delay.

- Ted