Building e2fsprogs 1.42.8 on ppc, I got this:
r_1024_small_bg: ext2 1024 blocksize with small block groups: failed
Because during the resize step it did this:
Itable move group 1 block 1030->1092 (diff 62)
but during e2fsck it found:
/tmp/e2fsprogs-tmp.uiFhgP: Inode table for group 1 is not in group. (block 1092)
i.e. from dumpe2fs we can see:
Group 1: (Blocks 1025-1110)
Backup superblock at 1025, Group descriptors at 1026-1026
Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
Inode table at 1092-1123 (+67)
^^^^ beyond end of block group
ext2fs_allocate_group_table() currently sends the last block of the
group as an acceptable *starting* point for the inode table allocation.
Because the inode table may be several blocks, and must reside wholly
within the group, move the last acceptable starting block back by this
amount so that the allocated range cannot extend past the end of the
group.
Signed-off-by: Eric Sandeen <[email protected]>
---
p.s. I *think* this fixes it; I've not convinced myself that the failure
is 100% deterministic, so extra eyeballs on the patch are welcome.
Thanks,
-Eric
Index: e2fsprogs-1.42.8/lib/ext2fs/alloc_tables.c
===================================================================
--- e2fsprogs-1.42.8.orig/lib/ext2fs/alloc_tables.c
+++ e2fsprogs-1.42.8/lib/ext2fs/alloc_tables.c
@@ -200,6 +200,9 @@ errcode_t ext2fs_allocate_group_table(ex
}
if (!ext2fs_inode_table_loc(fs, group)) {
+ /* Inode table must fit within group, so adjust last blk */
+ last_blk -= (fs->inode_blocks_per_group - 1);
+
retval = ext2fs_get_free_blocks2(fs, group_blk, last_blk,
fs->inode_blocks_per_group,
bmap, &new_blk);
On 7/2/13 3:14 PM, Eric Sandeen wrote:
> Building e2fsprogs 1.42.8 on ppc, I got this:
>
> r_1024_small_bg: ext2 1024 blocksize with small block groups: failed
You can add a RH FTBFS bug to the commit now if you like,
[Bug 980519] New: FTBFS: self checks failure: Tests failed: r_1024_small_bg
https://bugzilla.redhat.com/show_bug.cgi?id=980519
since Karsten Hopp has now reported it for Rawhide as well.
-Eric
> Because during the resize step it did this:
>
> Itable move group 1 block 1030->1092 (diff 62)
>
> but during e2fsck it found:
>
> /tmp/e2fsprogs-tmp.uiFhgP: Inode table for group 1 is not in group. (block 1092)
>
> i.e. from dumpe2fs we can see:
>
> Group 1: (Blocks 1025-1110)
> Backup superblock at 1025, Group descriptors at 1026-1026
> Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
> Inode table at 1092-1123 (+67)
> ^^^^ beyond end of block group
>
> ext2fs_allocate_group_table() currently sends the last block of the
> group as an acceptable *starting* point for the inode table allocation.
>
> Because the inode table may be several blocks, and must reside wholly
> within the group, move the last acceptable starting block back by this
> amount so that the allocated range cannot extend past the end of the
> group.
>
> Signed-off-by: Eric Sandeen <[email protected]>
> ---
>
> p.s. I *think* this fixes it; I've not convinced myself that the failure
> is 100% deterministic, so extra eyeballs on the patch are welcome.
>
> Thanks,
> -Eric
>
> Index: e2fsprogs-1.42.8/lib/ext2fs/alloc_tables.c
> ===================================================================
> --- e2fsprogs-1.42.8.orig/lib/ext2fs/alloc_tables.c
> +++ e2fsprogs-1.42.8/lib/ext2fs/alloc_tables.c
> @@ -200,6 +200,9 @@ errcode_t ext2fs_allocate_group_table(ex
> }
>
> if (!ext2fs_inode_table_loc(fs, group)) {
> + /* Inode table must fit within group, so adjust last blk */
> + last_blk -= (fs->inode_blocks_per_group - 1);
> +
> retval = ext2fs_get_free_blocks2(fs, group_blk, last_blk,
> fs->inode_blocks_per_group,
> bmap, &new_blk);
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On Tue, Jul 02, 2013 at 03:14:35PM -0400, Eric Sandeen wrote:
> Building e2fsprogs 1.42.8 on ppc, I got this:
>
> r_1024_small_bg: ext2 1024 blocksize with small block groups: failed
>
> Because during the resize step it did this:
>
> Itable move group 1 block 1030->1092 (diff 62)
>
> but during e2fsck it found:
>
> /tmp/e2fsprogs-tmp.uiFhgP: Inode table for group 1 is not in group. (block 1092)
>
> i.e. from dumpe2fs we can see:
>
> Group 1: (Blocks 1025-1110)
> Backup superblock at 1025, Group descriptors at 1026-1026
> Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
> Inode table at 1092-1123 (+67)
> ^^^^ beyond end of block group
There seems to be something wrong here. The test file system was
created like this:
mke2fs -t ext2 -O ^resize_inode -b 1024 -g 1024 -qF /tmp/foo.img 64M
The file system hence should have 64 block groups, and dumpe2fs before
the resize looks like this on an x86 system:
Group 1: (Blocks 1025-2048)
Backup superblock at 1025, Group descriptors at 1026-1027
Block bitmap at 1028 (+3), Inode bitmap at 1029 (+4)
Inode table at 1030-1061 (+5)
... and after:
Group 1: (Blocks 1025-2048)
Backup superblock at 1025, Group descriptors at 1026-1089
Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
Inode table at 1092-1123 (+67)
Note the range of block group #1: 1025-2048, whereas on the PPC,
apparently the range is quite different: Group 1: (Blocks 1025-1110)
So there's something else going really wrong here....
- Ted
On Sun, Jul 07, 2013 at 11:53:16AM -0400, Theodore Ts'o wrote:
> There seems to be something wrong here. The test file system was
> created like this:
>
> mke2fs -t ext2 -O ^resize_inode -b 1024 -g 1024 -qF /tmp/foo.img 64M
>
> The file system hence should have 64 block groups, and dumpe2fs before
> the resize looks like this on an x86 system:
>
> Group 1: (Blocks 1025-2048)
> Backup superblock at 1025, Group descriptors at 1026-1027
> Block bitmap at 1028 (+3), Inode bitmap at 1029 (+4)
> Inode table at 1030-1061 (+5)
>
> ... and after:
>
> Group 1: (Blocks 1025-2048)
> Backup superblock at 1025, Group descriptors at 1026-1089
> Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
> Inode table at 1092-1123 (+67)
>
> Note the range of block group #1: 1025-2048, whereas on the PPC,
> apparently the range is quite different: Group 1: (Blocks 1025-1110)
>
> So there's something else going really wrong here....
I just tried building e2fsprogs 1.42.8 on a powerpc system in a Debian
unstable (sid) chroot, and I'm not able to reproduce this test
failure. I tried building using both gcc 4.6.4 and gcc 4.8, in case
it was a compiler bug.
So again, there's something really, REALLY wrong....
- Ted
On Jul 7, 2013, at 10:53 AM, "Theodore Ts'o" <[email protected]> wrote:
> On Tue, Jul 02, 2013 at 03:14:35PM -0400, Eric Sandeen wrote:
>> Building e2fsprogs 1.42.8 on ppc, I got this:
>>
>> r_1024_small_bg: ext2 1024 blocksize with small block groups: failed
>>
>> Because during the resize step it did this:
>>
>> Itable move group 1 block 1030->1092 (diff 62)
>>
>> but during e2fsck it found:
>>
>> /tmp/e2fsprogs-tmp.uiFhgP: Inode table for group 1 is not in group. (block 1092)
>>
>> i.e. from dumpe2fs we can see:
>>
>> Group 1: (Blocks 1025-1110)
>> Backup superblock at 1025, Group descriptors at 1026-1026
>> Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
>> Inode table at 1092-1123 (+67)
>> ^^^^ beyond end of block group
>
> There seems to be something wrong here. The test file system was
> created like this:
>
> mke2fs -t ext2 -O ^resize_inode -b 1024 -g 1024 -qF /tmp/foo.img 64M
>
> The file system hence should have 64 block groups, and dumpe2fs before
> the resize looks like this on an x86 system:
>
> Group 1: (Blocks 1025-2048)
> Backup superblock at 1025, Group descriptors at 1026-1027
> Block bitmap at 1028 (+3), Inode bitmap at 1029 (+4)
> Inode table at 1030-1061 (+5)
>
> ... and after:
>
> Group 1: (Blocks 1025-2048)
> Backup superblock at 1025, Group descriptors at 1026-1089
> Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
> Inode table at 1092-1123 (+67)
>
> Note the range of block group #1: 1025-2048, whereas on the PPC,
> apparently the range is quite different: Group 1: (Blocks 1025-1110)
>
> So there's something else going really wrong here....
>
> - Ted
Perhaps so, I'll look into it more. But nonetheless isn't the patch correct?
Don't we need to ensure the inode table is wholly within be group, however we may have gotten here?
-Eric
On 7/7/13 10:53 AM, Theodore Ts'o wrote:
> On Tue, Jul 02, 2013 at 03:14:35PM -0400, Eric Sandeen wrote:
>> Building e2fsprogs 1.42.8 on ppc, I got this:
>>
>> r_1024_small_bg: ext2 1024 blocksize with small block groups: failed
>>
>> Because during the resize step it did this:
>>
>> Itable move group 1 block 1030->1092 (diff 62)
>>
>> but during e2fsck it found:
>>
>> /tmp/e2fsprogs-tmp.uiFhgP: Inode table for group 1 is not in group. (block 1092)
>>
>> i.e. from dumpe2fs we can see:
>>
>> Group 1: (Blocks 1025-1110)
>> Backup superblock at 1025, Group descriptors at 1026-1026
>> Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
>> Inode table at 1092-1123 (+67)
>> ^^^^ beyond end of block group
>
> There seems to be something wrong here. The test file system was
> created like this:
>
> mke2fs -t ext2 -O ^resize_inode -b 1024 -g 1024 -qF /tmp/foo.img 64M
>
> The file system hence should have 64 block groups, and dumpe2fs before
> the resize looks like this on an x86 system:
>
> Group 1: (Blocks 1025-2048)
> Backup superblock at 1025, Group descriptors at 1026-1027
> Block bitmap at 1028 (+3), Inode bitmap at 1029 (+4)
> Inode table at 1030-1061 (+5)
>
> ... and after:
>
> Group 1: (Blocks 1025-2048)
> Backup superblock at 1025, Group descriptors at 1026-1089
> Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
> Inode table at 1092-1123 (+67)
>
> Note the range of block group #1: 1025-2048, whereas on the PPC,
> apparently the range is quite different: Group 1: (Blocks 1025-1110)
>
> So there's something else going really wrong here....
I don't think so - that's only after the shrink, right?
However, I understand your confusion. The test is not at all obvious
about what it's doing:
> test_description="ext2 1024 blocksize with small block groups"
> FEATURES="-t ext2 -O ^resize_inode -b 1024 -g 1024"
> SIZE_1=64M
> SIZE_2=2G
> LOG=$test_name.log
> E2FSCK=../e2fsck/e2fsck
>
>
> . $cmd_dir/scripts/resize_test
Um, ok, so we "resize test" with those parameters.
Looking at that script, which sadly has no comments, it invokes
a resize to $SIZE_2 (2G) first, but then does something different:
# grep RESIZE tests/scripts/resize_test | grep -v echo
if ! $RESIZE2FS $RESIZE2FS_OPTS -d 31 $TMPFILE $SIZE_2 >> $LOG 2>&1
if ! $RESIZE2FS $RESIZE2FS_OPTS -d 31 -M $TMPFILE $SIZE_2 >> $LOG 2>&1
if ! $RESIZE2FS $RESIZE2FS_OPTS -d 31 -M $TMPFILE $SIZE_2 >> $LOG 2>&1
if ! $RESIZE2FS $RESIZE2FS_OPTS -d 31 -M $TMPFILE $SIZE_2 >> $LOG 2>&1
so it does 4 resizes; the last 3 are using "-M" which says "go to the
minimum size." Which is a weird thing to do when also specifying
an actual size:
# grep "\-M" tests/r_1024_small_bg.log
../resize/resize2fs -d 31 -M /tmp/e2fsprogs-tmp.LoNby5 2G
../resize/resize2fs -d 31 -M /tmp/e2fsprogs-tmp.LoNby5 2G
../resize/resize2fs -d 31 -M /tmp/e2fsprogs-tmp.LoNby5 2G
But the 2G is ignored.... so it's re-resizing to minimum sizes. On x86:
# grep -w "resize\|is now" tests/r_1024_small_bg.log
../resize/resize2fs -d 31 /tmp/e2fsprogs-tmp.LoNby5 2G
The filesystem on /tmp/e2fsprogs-tmp.LoNby5 is now 2097152 blocks long.
../resize/resize2fs -d 31 -M /tmp/e2fsprogs-tmp.LoNby5 2G
The filesystem on /tmp/e2fsprogs-tmp.LoNby5 is now 1174 blocks long.
../resize/resize2fs -d 31 -M /tmp/e2fsprogs-tmp.LoNby5 2G
The filesystem on /tmp/e2fsprogs-tmp.LoNby5 is now 1002 blocks long.
on the failing PPC build:
# grep -w "resize\|is now" tests/r_1024_small_bg.log
../resize/resize2fs -d 31 /tmp/e2fsprogs-tmp.4KTO7a 2G
The filesystem on /tmp/e2fsprogs-tmp.4KTO7a is now 2097152 blocks long.
../resize/resize2fs -d 31 -M /tmp/e2fsprogs-tmp.4KTO7a 2G
The filesystem on /tmp/e2fsprogs-tmp.4KTO7a is now 1247 blocks long.
../resize/resize2fs -d 31 -M /tmp/e2fsprogs-tmp.4KTO7a 2G
The filesystem on /tmp/e2fsprogs-tmp.4KTO7a is now 1121 blocks long.
so they're both shrinking quite a bit. But I think due to the vagaries
of the initial filesystem population, the test is behaving differently
and just happening to hit this corner case.
On the last failing run, the minimum size happens to be calculated
at 1121 blocks; with debug printfs:
> ../resize/resize2fs -d 31 -M /tmp/e2fsprogs-tmp.gB4G7E 2G
> resize2fs 1.42.8 (20-Jun-2013)
> forcing to min size 1121
> Resizing the filesystem on /tmp/e2fsprogs-tmp.gB4G7E to 1121 (1k) blocks.
But what's weird is that in this resize run it didn't actually
move the inode table; that was done in a prior minimization run,
and in this case it seems to have simply shrunk past the existing
inode table end. Argh. Still looking.
-Eric
Ok, resetting a little on this one.
The actual problem seems to be that the test does successive "-M" minimal resizes, and eventually we resize into the middle of an inode table, leaving the end of the table beyond the fs.
Point "resize2fs -M" at the attached image once or twice w/ fscks in between and you should see it.
It seems like the obvious fix would be to move the inode table if necessary, as with the following patch.
But then of course we find that we've run out of room to move the table, and the resize fails; so we've let resize2fs choose its own minimum size - which which it cannot handle. Presumably something's gone wrong in calculate_minimum_resize_size(), though I can't tell what.
Truth be told, I'm not enjoying resize2fs right now!
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 204b10a..27dc5e6 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -891,18 +891,25 @@ static errcode_t blocks_to_move(ext2_resize_t rfs)
new_size = ext2fs_blocks_count(fs->super);
if (new_size < ext2fs_blocks_count(old_fs->super)) {
for (g = 0; g < fs->group_desc_count; g++) {
+ int realloc = 0;
/*
* ext2fs_allocate_group_table re-allocates bitmaps
* which are set to block 0.
*/
if (ext2fs_block_bitmap_loc(fs, g) >= new_size) {
ext2fs_block_bitmap_loc_set(fs, g, 0);
- retval = ext2fs_allocate_group_table(fs, g, 0);
- if (retval)
- return retval;
+ realloc = 1;
}
if (ext2fs_inode_bitmap_loc(fs, g) >= new_size) {
ext2fs_inode_bitmap_loc_set(fs, g, 0);
+ realloc = 1;
+ }
+ if (ext2fs_inode_table_loc(fs, g) + fs->inode_blocks_per_group > new_size) {
+ ext2fs_inode_table_loc_set(fs, g, 0);
+ realloc = 1;
+ }
+
+ if (realloc) {
retval = ext2fs_allocate_group_table(fs, g, 0);
if (retval)
return retval;
On Mon, Jul 08, 2013 at 04:27:21PM -0500, Eric Sandeen wrote:
>
> The actual problem seems to be that the test does successive "-M" minimal resizes, and eventually we resize into the middle of an inode table, leaving the end of the table beyond the fs.
>
> Point "resize2fs -M" at the attached image once or twice w/ fscks in between and you should see it.
I've been going through my patch backlog, so I finally had a chance to
take a very close look at your test image. I now understand why
things are failing.
1) The test image (which you said was generated on a ppc e2fsprogs?)
was doing something very weird as far as the location of the
allocation bitmaps and inode table:
Filesystem features: ext_attr dir_index filetype sparse_super
Inode count: 512
Block count: 1247
...
Group 0: (Blocks 1-1024)
Primary superblock at 1, Group descriptors at 2-2
Block bitmap at 66 (+65), Inode bitmap at 67 (+66)
Inode table at 68-99 (+67)
Group 1: (Blocks 1025-1246)
Backup superblock at 1025, Group descriptors at 1026-1026
Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
Inode table at 1092-1123 (+67)
Compare and contrast this with what x86 and Debian's ppc mke2fs creates:
Group 0: (Blocks 1-1024)
Primary superblock at 1, Group descriptors at 2-2
Block bitmap at 3 (+2), Inode bitmap at 4 (+3)
Inode table at 5-14 (+4)
Group 1: (Blocks 1025-1246)
Backup superblock at 1025, Group descriptors at 1026-1026
Block bitmap at 1027 (+2), Inode bitmap at 1028 (+3)
Inode table at 1029-1038 (+4)
So I'm not sure why Fedora's ppc mke2fs is creating file systems in
this way, but that's one of the causes of the bug.
2) The second cause of the bug is that
calculate_minimum_resize_size(), when we calculate the number of
blocks for the last group, the code has an implicit assumption that
the metadata blocks are located at the very beginning of the block
group. That's an easy fix.
> It seems like the obvious fix would be to move the inode table if
> necessary, as with the following patch.
Your patch is a good one, but at least in the context of resize2fs -M,
we should be fixing calculate_minimum_resize_size() so we can avoid
needing to move the inode table (since even if it can succeed, it's
not worth the danger).
I'll send out some patches to address this. Thanks for sending the
test image; and my apologies for not having time to get back to this
until now.
- Ted
If the file system is being shrunk, and a block group's inode table
falls beyond the end of the inode table, we need to try to relocate
the inode table blocks.
Signed-off-by: "Theodore Ts'o" <[email protected]>
---
resize/resize2fs.c | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 1e4ac19..ec1594e 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -884,24 +884,34 @@ static errcode_t blocks_to_move(ext2_resize_t rfs)
fs = rfs->new_fs;
/*
- * If we're shrinking the filesystem, we need to move any group's
- * bitmaps which are beyond the end of the new filesystem.
+ * If we're shrinking the filesystem, we need to move any
+ * group's metadata blocks (either allocation bitmaps or the
+ * inode table) which are beyond the end of the new
+ * filesystem.
*/
new_size = ext2fs_blocks_count(fs->super);
if (new_size < ext2fs_blocks_count(old_fs->super)) {
for (g = 0; g < fs->group_desc_count; g++) {
+ int realloc = 0;
/*
- * ext2fs_allocate_group_table re-allocates bitmaps
- * which are set to block 0.
+ * ext2fs_allocate_group_table will re-allocate any
+ * metadata blocks whose location is set to zero.
*/
if (ext2fs_block_bitmap_loc(fs, g) >= new_size) {
ext2fs_block_bitmap_loc_set(fs, g, 0);
- retval = ext2fs_allocate_group_table(fs, g, 0);
- if (retval)
- return retval;
+ realloc = 1;
}
if (ext2fs_inode_bitmap_loc(fs, g) >= new_size) {
ext2fs_inode_bitmap_loc_set(fs, g, 0);
+ realloc = 1;
+ }
+ if ((ext2fs_inode_table_loc(fs, g) +
+ fs->inode_blocks_per_group) > new_size) {
+ ext2fs_inode_table_loc_set(fs, g, 0);
+ realloc = 1;
+ }
+
+ if (realloc) {
retval = ext2fs_allocate_group_table(fs, g, 0);
if (retval)
return retval;
--
1.7.12.rc0.22.gcdd159b
If the file system's inode table blocks in the last block group are
located in the middle or the end of the block group, it's possible for
resize2fs -M to use a size which will require relocating the inode
table blocks in the last block group. This can lead to all sorts of
problems, so solve it by simply guaranteeing that we will never do
that.
Reported-by: Eric Sandeen <[email protected]>
Signed-off-by: "Theodore Ts'o" <[email protected]>
---
resize/resize2fs.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 51b85b8..1e4ac19 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -2224,6 +2224,15 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs, int flags)
blks_needed = (groups-1) * EXT2_BLOCKS_PER_GROUP(fs->super);
blks_needed += overhead;
+ /*
+ * Make sure blks_needed covers the end of the inode table in
+ * the last block group.
+ */
+ overhead = ext2fs_inode_table_loc(fs, groups-1) +
+ fs->inode_blocks_per_group;
+ if (blks_needed < overhead)
+ blks_needed = overhead;
+
#ifdef RESIZE2FS_DEBUG
if (flags & RESIZE_DEBUG_MIN_CALC)
printf("Estimated blocks needed: %llu\n", blks_needed);
--
1.7.12.rc0.22.gcdd159b
Eric Sandeen reported that Fedora's mke2fs when compiled for ppc was
creating a file system which caused problems with resize2fs -M.
Closer examination showed that the problem was file system which
looked like this:
Filesystem features: ext_attr dir_index filetype sparse_super
Inode count: 512
Block count: 1247
...
Group 0: (Blocks 1-1024)
Primary superblock at 1, Group descriptors at 2-2
Block bitmap at 66 (+65), Inode bitmap at 67 (+66)
Inode table at 68-99 (+67)
Group 1: (Blocks 1025-1246)
Backup superblock at 1025, Group descriptors at 1026-1026
Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
Inode table at 1092-1123 (+67)
It's not obvious to me why Fedora's ppc mke2fs is creating file
systems like this (I can't reproduce this on debian ppc systems), but
resize2fs -M should be able to deal with such file systems, which is
what this test is designed to check.
Signed-off-by: "Theodore Ts'o" <[email protected]>
---
tests/r_min_itable/expect | 14 ++++++++++++++
tests/r_min_itable/image.gz | Bin 0 -> 3597 bytes
tests/r_min_itable/name | 1 +
tests/r_min_itable/script | 43 +++++++++++++++++++++++++++++++++++++++++++
4 files changed, 58 insertions(+)
create mode 100644 tests/r_min_itable/expect
create mode 100644 tests/r_min_itable/image.gz
create mode 100644 tests/r_min_itable/name
create mode 100644 tests/r_min_itable/script
diff --git a/tests/r_min_itable/expect b/tests/r_min_itable/expect
new file mode 100644
index 0000000..104688a
--- /dev/null
+++ b/tests/r_min_itable/expect
@@ -0,0 +1,14 @@
+resize2fs test
+resize2fs -M test.img
+Resizing the filesystem on test.img to 1124 (1k) blocks.
+The filesystem on test.img is now 1124 blocks long.
+
+Exit status is 0
+e2fsck 1.42.8 (20-Jun-2013)
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 13/512 files (0.0% non-contiguous), 1120/1124 blocks
+Exit status is 0
diff --git a/tests/r_min_itable/image.gz b/tests/r_min_itable/image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..7c40c5661c86c8fc81e5c73b2adfe1658666df2c
GIT binary patch
literal 3597
zc-rNbTToMX9>?k00#l3Z=q^|opdDM?4im(o(QOo|f}kQ7K_EG~p{`O70$TR~1_COz
zw2Q&*x}pX_E&{oni?;(3xu~onML_HMClWT`WlhRI4Hw~%5)9d$KJ3gs?L()1@$>k7
z`F$Tg^P6u_nWvkKw>9ZE3v8x6f5l&%JMwh>p?J>?i{Ja;{kNmu|Lw;2Lh?PM9$f$C
z&Ad;p-}o<U3nJ6cm@eI*Q+oFLl3H{1tgf?qbhO2BxXJbBzI3;?3@<PD7RAzc8sPz2
zhCWUY{lR95X&bCe8(udWch%c{$vfu4{g3_A+7BMCh#CH)U2j)Nw1%i+=VXqk^T`*>
zGEDMk0t??_{P6C2`Q49vof+$xhOt1ob3ukO>+#!RIi2H3o`Po>3bOEX){({cUfy;I
z9)W3YWL$VY8v07Hvv_DWBwiZEe4KOQrDvJD+s|Sm<l|R;6xHG0W;x-Kj(huf4z~|0
zr!FsV-#63Z7%6m4-n%CEn~b+d><XQ39_kCboymPNZk;yRFZ7J;bG_?#!v3FhXg*BZ
zCtb)KqZ2uzGNz8^b0o2^yk|77LYKcIZ*Jsot{P_!%XKfYbT73Aj1@mlKfm2&%Z<fc
z%Q8oi#tPyEI*Zh%;-}FUNJA>Iuk>)y8*Z;DZNj)U`$NwbeHMCl=XLssT{&9mdKZ>*
zcRN*kHRr6|>?y}SC%#uL$ZS@&n^m7!Eo_-%k5f}5?o^NH$Dy4qb?oj-Db_?7r0AHk
z$xO_m^GJ;kejI&@T=XICpp!^l1>S)Ei6ic1uFxWmG?@7dokWPVjD|)CNhYJDQwgz^
z(a{n@>d$;hZ_g4{GL5t#OCn?{=#(t+5YtSHv!rX8ayr>2y1|gN&?YHhs_8VFc#i3y
zr8cPta}-Mq5S22u7(YM~!+eA72oQHOmoZU*bOrM@mQ*EbV$>K?CCOmUVX0MO3)6~8
zs-(WmacsL-bcSib1Y$`Va~ew#iwz8oiN(@2i~>tGiCP!}6PhG>Ocj=965E+}Olp#P
zGbL<dxacHP&+@}13Ct;WN4R)^xyp*dr7M{-HmO}iF<KV!aulq=zeMvDDm(ZCJMTxY
zPpk*ubE6f4Alw`9xw(quAo8Gf8T-(|SEL0|3Cs#>GCS(nrjRaSzF-F(B6&wgq7+{V
zqPPd;t;^M0tifD?oL){*;CI|T^7fAHBX|yIv94uJ4vD<8C@~o@+_*U5Bd<Z1+w07c
zPHxwO^ocIz<)R&7;6D4@vBi0<DD^6XbA^49L(F-0%CX6Lr6{G79AR>-dF-4c(s|je
zi2xU@E4g`n$y545ZZRiup?)KMfRj|F-$mzhw#Vv|xK}F3J?J1pR#3l3?;z?_G6e+)
zkxFsVK18c0G1P=mddds!GOAXSap)r>zMA5oy+%zlnS~A+$!015?KY}&Nik|L61h}3
zI$+e!kaBdyNX<|S;7y)t5g7>|@bE>{GI)=tDJOTs$2_u}S_1#UQ^%9Zkl_*WR2aO^
z)Ao~x;6Hg(KeZ6v)~Q0sSooa|521qLw>k|;W<sNmBq@J*SEtS<g;1{}vZ=MOU#A@-
z3*fMh8lybmEtx8ijDZhjcp$X`_Q*8VWCk3Rk=2whydzUf$TSFKgnO4q$xiN$uFR-*
zDq3wfsMw=^L^QPuz3)hutL|zpXP$Q)PzZz6C-vQEn<HOdw_b2TlVHd|Uny&r;{$pN
zoMMkVcFEPtHD6nMxm)58sk#ShxrgOMqv|88l~p+Q%C#Nh5%sv<4(HesrKXYGhZ>b+
zV^Fqh!D}t^yxzOi+dWKw5^Yqr7KtL%7xV*Yg_7Q@9n36OUM-Szs-$`f8rVulmXj=!
z7ONukBDTqK$k|{f52M3M%1ph5-a=G9<R<hXg8NV(qCJQvTK^rrjU(&@FX@d0KNx&V
zZzT{dfOIq=$OJ|@o)B7rgWizE_Xl_BEm=q<Fw>i|1VW&v<FbT9V1|yg@z;WWI@X5V
z0AqBFO;7-a=>(f_4$Nbp2Jk(=Eo^fDQVO18Q2~M&@DSS;AnXQ{*v2aU3ebaXtwNf>
zC>C8M$N+;_e3j4wX0Z)ozAw0gZ4o19zzb}XSdaz)7AF=O0E<PM_-jBP7HdLUfDMZ=
z3G#pmOE3xTz{!3Z&i4jg?B;OfBzVq7g$okEBX(Q3Z~#oP8{7FSK`*<t9ic#aRHpRT
zsoCo}=l}XzJ7Ev|Y3|Le$u989Pn#Q`@d$o%+%)U)e?GVUSgqgg(EXoXDGm?xQ{oHj
z^VW@}U#s2Z6Vy8W!4#+C{;sLEqU_M?vLC;yZLkEiUOQdlaz(qmr(Dw$o&6CLfvavW
zJ~HcZKO%kpjVbT5#W!CqFb9RYeu^esx0aUV6}bk^28=`${(a|E{Hl?At-j6i|5JS6
zlJTos=`2#O$7$4tH0cQzbsB3{;|lb-@#1Qt51lgBHRA+oH8wUAHgwKdn~PVW7Gpy$
zVM6W3dh@9LT#H#dn|Tzu<fAc+9J{uegOq`1bQDJr3+~CB7Z!7pd-b2wg+yYoek=Vs
zk)-XIzn=LsPR`)%=(w?IhH#=TUd<xB1U}(iTtsxi30_?}UJt+LHI@_4;g7u9c>EOn
z7q20ncm$_;_5Ju&_ye!0pO}KLbTuJ(8GNR@7((>Im%2I<*FvakBnc~Y=xVd^N@&(K
zWD`ShMpr+E-+*JfrZHj;&dX{7@lyCyb}^9XhLf_oYP<=K${MQ)3!IhJO7JuAg{(nB
j7$n6ny{+}X`X1^FoeTP9Rlg~me@j+7@zkwwiQB&c?_X|7
literal 0
Hc-jL100001
diff --git a/tests/r_min_itable/name b/tests/r_min_itable/name
new file mode 100644
index 0000000..841b043
--- /dev/null
+++ b/tests/r_min_itable/name
@@ -0,0 +1 @@
+resize2fs -M with inode table in middle of last block group
diff --git a/tests/r_min_itable/script b/tests/r_min_itable/script
new file mode 100644
index 0000000..00aaa60
--- /dev/null
+++ b/tests/r_min_itable/script
@@ -0,0 +1,43 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+
+IMAGE=$test_dir/image.gz
+FSCK_OPT=-yf
+OUT=$test_name.log
+EXP=$test_dir/expect
+
+gunzip < $IMAGE > $TMPFILE
+
+echo "resize2fs test" > $OUT
+
+echo "resize2fs -M test.img" >> $OUT
+$RESIZE2FS -M $TMPFILE 2>&1 >> $OUT.new 2>&1
+status=$?
+echo Exit status is $status >> $OUT.new
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
+
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE >> $OUT 2>&1
+echo Exit status is $status >> $OUT
+
+rm $TMPFILE $OUT.new
+
+#
+# Do the verification
+#
+
+cmp -s $OUT $EXP
+status=$?
+
+if [ "$status" = 0 ] ; then
+ echo "$test_name: $test_description: ok"
+ touch $test_name.ok
+else
+ echo "$test_name: $test_description: failed"
+ diff $DIFF_OPTS $EXP $OUT > $test_name.failed
+fi
+
+unset IMAGE FSCK_OPT OUT EXP
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+ echo "$test_name: $test_description: skipped"
+fi
+
--
1.7.12.rc0.22.gcdd159b
Signed-off-by: "Theodore Ts'o" <[email protected]>
---
resize/main.c | 2 +-
resize/resize2fs.8.in | 4 +++-
resize/resize2fs.c | 59 +++++++++++++++++++++++++++++++++++++++++++++------
resize/resize2fs.h | 3 ++-
4 files changed, 59 insertions(+), 9 deletions(-)
diff --git a/resize/main.c b/resize/main.c
index b648a15..1394ae1 100644
--- a/resize/main.c
+++ b/resize/main.c
@@ -328,7 +328,7 @@ int main (int argc, char ** argv)
exit(1);
}
- min_size = calculate_minimum_resize_size(fs);
+ min_size = calculate_minimum_resize_size(fs, flags);
if (print_min_size) {
if (!force && ((fs->super->s_state & EXT2_ERROR_FS) ||
diff --git a/resize/resize2fs.8.in b/resize/resize2fs.8.in
index 735fc91..a1f3099 100644
--- a/resize/resize2fs.8.in
+++ b/resize/resize2fs.8.in
@@ -101,7 +101,9 @@ from the following list:
8 \-\ Debug moving the inode table
.br
16 \-\ Print timing information
-.TP
+.br
+ 32 \-\ Debug minimum filesystem size (\-M) calculation
+.TP
.B \-f
Forces resize2fs to proceed with the filesystem resize operation, overriding
some safety checks which resize2fs normally enforces.
diff --git a/resize/resize2fs.c b/resize/resize2fs.c
index 3b48206..51b85b8 100644
--- a/resize/resize2fs.c
+++ b/resize/resize2fs.c
@@ -2034,10 +2034,11 @@ static int calc_group_overhead(ext2_filsys fs, blk64_t grp,
/*
* calcluate the minimum number of blocks the given fs can be resized to
*/
-blk64_t calculate_minimum_resize_size(ext2_filsys fs)
+blk64_t calculate_minimum_resize_size(ext2_filsys fs, int flags)
{
ext2_ino_t inode_count;
- blk64_t blks_needed, groups, data_blocks;
+ dgrp_t groups;
+ blk64_t blks_needed, data_blocks;
blk64_t grp, data_needed, last_start;
blk64_t overhead = 0;
int old_desc_blocks;
@@ -2055,6 +2056,11 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs)
EXT2_BLOCKS_PER_GROUP(fs->super);
groups = ext2fs_div64_ceil(blks_needed,
EXT2_BLOCKS_PER_GROUP(fs->super));
+#ifdef RESIZE2FS_DEBUG
+ if (flags & RESIZE_DEBUG_MIN_CALC)
+ printf("fs has %d inodes, %d groups required.\n",
+ inode_count, groups);
+#endif
/*
* number of old-style block group descriptor blocks
@@ -2071,6 +2077,10 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs)
for (grp = 0; grp < fs->group_desc_count; grp++)
data_needed -= calc_group_overhead(fs, grp, old_desc_blocks);
+#ifdef RESIZE2FS_DEBUG
+ if (flags & RESIZE_DEBUG_MIN_CALC)
+ printf("fs requires %llu data blocks.\n", data_needed);
+#endif
/*
* For ext4 we need to allow for up to a flex_bg worth of
@@ -2103,6 +2113,11 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs)
data_blocks -= overhead;
}
+#ifdef RESIZE2FS_DEBUG
+ if (flags & RESIZE_DEBUG_MIN_CALC)
+ printf("With %d group(s), we have %llu blocks available.\n",
+ groups, data_blocks);
+#endif
/*
* if we need more group descriptors in order to accomodate our data
@@ -2110,7 +2125,7 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs)
*/
while (data_needed > data_blocks) {
blk64_t remainder = data_needed - data_blocks;
- blk64_t extra_grps;
+ dgrp_t extra_grps;
/* figure out how many more groups we need for the data */
extra_grps = ext2fs_div64_ceil(remainder,
@@ -2153,10 +2168,22 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs)
extra_groups);
extra_groups = groups % flexbg_size;
}
+#ifdef RESIZE2FS_DEBUG
+ if (flags & RESIZE_DEBUG_MIN_CALC)
+ printf("Added %d extra group(s), "
+ "data_needed %llu, data_blocks %llu, "
+ "last_start %llu\n",
+ extra_grps, data_needed, data_blocks,
+ last_start);
+#endif
}
/* now for the fun voodoo */
overhead = calc_group_overhead(fs, groups-1, old_desc_blocks);
+#ifdef RESIZE2FS_DEBUG
+ if (flags & RESIZE_DEBUG_MIN_CALC)
+ printf("Last group's overhead is %llu\n", overhead);
+#endif
/*
* if this is the case then the last group is going to have data in it
@@ -2165,6 +2192,11 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs)
if (last_start < data_needed) {
blk64_t remainder = data_needed - last_start;
+#ifdef RESIZE2FS_DEBUG
+ if (flags & RESIZE_DEBUG_MIN_CALC)
+ printf("Need %llu data blocks in last group\n",
+ remainder);
+#endif
/*
* 50 is a magic number that mkfs/resize uses to see if its
* even worth making/resizing the fs. basically you need to
@@ -2179,6 +2211,10 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs)
overhead += 50;
overhead += fs->super->s_first_data_block;
+#ifdef RESIZE2FS_DEBUG
+ if (flags & RESIZE_DEBUG_MIN_CALC)
+ printf("Final size of last group is %lld\n", overhead);
+#endif
/*
* since our last group doesn't have to be BLOCKS_PER_GROUP large, we
@@ -2188,6 +2224,11 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs)
blks_needed = (groups-1) * EXT2_BLOCKS_PER_GROUP(fs->super);
blks_needed += overhead;
+#ifdef RESIZE2FS_DEBUG
+ if (flags & RESIZE_DEBUG_MIN_CALC)
+ printf("Estimated blocks needed: %llu\n", blks_needed);
+#endif
+
/*
* If at this point we've already added up more "needed" than
* the current size, just return current size as minimum.
@@ -2199,9 +2240,15 @@ blk64_t calculate_minimum_resize_size(ext2_filsys fs)
* enabled, in case we need to grow the extent tree. The more
* we shrink the file system, the more space we need.
*/
- if (fs->super->s_feature_incompat & EXT3_FEATURE_INCOMPAT_EXTENTS)
- blks_needed += (ext2fs_blocks_count(fs->super) -
- blks_needed)/500;
+ if (fs->super->s_feature_incompat & EXT3_FEATURE_INCOMPAT_EXTENTS) {
+ blk64_t safe_margin = (ext2fs_blocks_count(fs->super) -
+ blks_needed)/500;
+#ifdef RESIZE2FS_DEBUG
+ if (flags & RESIZE_DEBUG_MIN_CALC)
+ printf("Extents safety margin: %llu\n", safe_margin);
+#endif
+ blks_needed += safe_margin;
+ }
return blks_needed;
}
diff --git a/resize/resize2fs.h b/resize/resize2fs.h
index d425491..52319b5 100644
--- a/resize/resize2fs.h
+++ b/resize/resize2fs.h
@@ -77,6 +77,7 @@ typedef struct ext2_sim_progress *ext2_sim_progmeter;
#define RESIZE_DEBUG_INODEMAP 0x0004
#define RESIZE_DEBUG_ITABLEMOVE 0x0008
#define RESIZE_DEBUG_RTRACK 0x0010
+#define RESIZE_DEBUG_MIN_CALC 0x0020
#define RESIZE_PERCENT_COMPLETE 0x0100
#define RESIZE_VERBOSE 0x0200
@@ -145,7 +146,7 @@ extern errcode_t resize_fs(ext2_filsys fs, blk64_t *new_size, int flags,
extern errcode_t adjust_fs_info(ext2_filsys fs, ext2_filsys old_fs,
ext2fs_block_bitmap reserve_blocks,
blk64_t new_size);
-extern blk64_t calculate_minimum_resize_size(ext2_filsys fs);
+extern blk64_t calculate_minimum_resize_size(ext2_filsys fs, int flags);
/* extent.c */
--
1.7.12.rc0.22.gcdd159b
On 9/30/13 8:57 PM, Theodore Ts'o wrote:
> On Mon, Jul 08, 2013 at 04:27:21PM -0500, Eric Sandeen wrote:
>>
>> The actual problem seems to be that the test does successive "-M" minimal resizes, and eventually we resize into the middle of an inode table, leaving the end of the table beyond the fs.
>>
>> Point "resize2fs -M" at the attached image once or twice w/ fscks in between and you should see it.
>
> I've been going through my patch backlog, so I finally had a chance to
> take a very close look at your test image. I now understand why
> things are failing.
>
> 1) The test image (which you said was generated on a ppc e2fsprogs?)
> was doing something very weird as far as the location of the
> allocation bitmaps and inode table:
Yes, this was just during a fedora build, during the "make check" phase.
https://bugzilla.redhat.com/show_bug.cgi?id=980519
No idea why things should be coming out differently, that's a bit
alarming in and of itself.
(Fedora isn't carrying any interesting patches to speak of).
-Eric
> Filesystem features: ext_attr dir_index filetype sparse_super
> Inode count: 512
> Block count: 1247
> ...
>
> Group 0: (Blocks 1-1024)
> Primary superblock at 1, Group descriptors at 2-2
> Block bitmap at 66 (+65), Inode bitmap at 67 (+66)
> Inode table at 68-99 (+67)
>
> Group 1: (Blocks 1025-1246)
> Backup superblock at 1025, Group descriptors at 1026-1026
> Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
> Inode table at 1092-1123 (+67)
>
> Compare and contrast this with what x86 and Debian's ppc mke2fs creates:
>
> Group 0: (Blocks 1-1024)
> Primary superblock at 1, Group descriptors at 2-2
> Block bitmap at 3 (+2), Inode bitmap at 4 (+3)
> Inode table at 5-14 (+4)
>
> Group 1: (Blocks 1025-1246)
> Backup superblock at 1025, Group descriptors at 1026-1026
> Block bitmap at 1027 (+2), Inode bitmap at 1028 (+3)
> Inode table at 1029-1038 (+4)
>
> So I'm not sure why Fedora's ppc mke2fs is creating file systems in
> this way, but that's one of the causes of the bug.
>
>
> 2) The second cause of the bug is that
> calculate_minimum_resize_size(), when we calculate the number of
> blocks for the last group, the code has an implicit assumption that
> the metadata blocks are located at the very beginning of the block
> group. That's an easy fix.
>
>> It seems like the obvious fix would be to move the inode table if
>> necessary, as with the following patch.
>
> Your patch is a good one, but at least in the context of resize2fs -M,
> we should be fixing calculate_minimum_resize_size() so we can avoid
> needing to move the inode table (since even if it can succeed, it's
> not worth the danger).
>
> I'll send out some patches to address this. Thanks for sending the
> test image; and my apologies for not having time to get back to this
> until now.
>
> - Ted
>
On 10/1/13 10:26 AM, Eric Sandeen wrote:
> On 9/30/13 8:57 PM, Theodore Ts'o wrote:
>> On Mon, Jul 08, 2013 at 04:27:21PM -0500, Eric Sandeen wrote:
>>>
>>> The actual problem seems to be that the test does successive "-M" minimal resizes, and eventually we resize into the middle of an inode table, leaving the end of the table beyond the fs.
>>>
>>> Point "resize2fs -M" at the attached image once or twice w/ fscks in between and you should see it.
>>
>> I've been going through my patch backlog, so I finally had a chance to
>> take a very close look at your test image. I now understand why
>> things are failing.
>>
>> 1) The test image (which you said was generated on a ppc e2fsprogs?)
>> was doing something very weird as far as the location of the
>> allocation bitmaps and inode table:
>
> Yes, this was just during a fedora build, during the "make check" phase.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=980519
>
> No idea why things should be coming out differently, that's a bit
> alarming in and of itself.
>
> (Fedora isn't carrying any interesting patches to speak of).
But I am doing this:
%check
+# XXX ERS Hack for now; this bug has existed for a while,
+# i.e. it is not a regression in this release, but there
+# is no fix yet, and we need to get this package building.
+# See Bug 987133 - resize2fs tests failing on ppc, s390
+rm -rf tests/r_1024_small_bg*
+rm -rf tests/r_64bit_big_expand*
+rm -rf tests/r_bigalloc_big_expand*
+rm -rf tests/r_ext4_big_expand*
make check
I'll retest w/ your patches, thanks.
-Eric
On 10/1/13 10:35 AM, Eric Sandeen wrote:
> On 10/1/13 10:26 AM, Eric Sandeen wrote:
>> On 9/30/13 8:57 PM, Theodore Ts'o wrote:
>>> On Mon, Jul 08, 2013 at 04:27:21PM -0500, Eric Sandeen wrote:
>>>>
>>>> The actual problem seems to be that the test does successive "-M" minimal resizes, and eventually we resize into the middle of an inode table, leaving the end of the table beyond the fs.
>>>>
>>>> Point "resize2fs -M" at the attached image once or twice w/ fscks in between and you should see it.
>>>
>>> I've been going through my patch backlog, so I finally had a chance to
>>> take a very close look at your test image. I now understand why
>>> things are failing.
>>>
>>> 1) The test image (which you said was generated on a ppc e2fsprogs?)
>>> was doing something very weird as far as the location of the
>>> allocation bitmaps and inode table:
>>
>> Yes, this was just during a fedora build, during the "make check" phase.
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=980519
>>
>> No idea why things should be coming out differently, that's a bit
>> alarming in and of itself.
>>
>> (Fedora isn't carrying any interesting patches to speak of).
>
> But I am doing this:
>
> %check
> +# XXX ERS Hack for now; this bug has existed for a while,
> +# i.e. it is not a regression in this release, but there
> +# is no fix yet, and we need to get this package building.
> +# See Bug 987133 - resize2fs tests failing on ppc, s390
> +rm -rf tests/r_1024_small_bg*
> +rm -rf tests/r_64bit_big_expand*
> +rm -rf tests/r_bigalloc_big_expand*
> +rm -rf tests/r_ext4_big_expand*
> make check
>
> I'll retest w/ your patches, thanks.
Now all are passing on ppc64, last 3 are still failing on s390. :(
-Eric