LinuxLists.cc - EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

On Mon, Jun 04, 2012 at 11:49:41PM +0200, Sander Eikelenboom wrote:
> Hello Ted,
>
> I have enabled some debug options, this is from around when the error occurs (kernel: [ 950.610126] EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1699, 32254 clusters in bitmap, 32258 in gd)

This error is usually caused by a filesystem corruption; specifically,
either a corrupted block allocation bitmap or a corrupted block group
descriptor. This error means that number of bits set in the block
allocation bitmap (32254) doesn't match the number that is reflected
in the summary block group descriptor statistics (32258).

I should probably change this error message so it's clearer what this
means to civilians.

If you run e2fsck on the file system, it will fix the file system
corruption. If you can reliably reproduce the problem after e2fsck is
run, that would be useful to know.

Regards,

- Ted

2012-06-04 23:04:11

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

On Mon, Jun 04, 2012 at 07:20:48PM +0200, Sander Eikelenboom wrote:
> Hello Ted,
>
> I have a problem back that occured , but didn't receive much respons in debugging:
>
> [ 4688.270789] EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd
> [ 4688.279172] Aborting journal on device dm-42-8.
> [ 4688.288634] EXT4-fs (dm-42): Remounting filesystem read-only
> [ 4688.299011] EXT4-fs (dm-42): ext4_da_writepages: jbd2_start: 6144 pages, ino 15597569; err -30

Ah, sorry, I didn't see this message when I responded to your earlier
message (you didn't mail thread it). I also didn't recall your
earlier complaint until I did an search of my mail archives.

The main problem is that we don't have an easy reproduction case.
It's not a problem which has been showing up on any of my testing.
Earlier you had said that this happened after a read-only snapshot, so
I had assumed it was an DM issue.

But you say this time it's not happening without a snapshot. OK, how
frequently does this happen? How easily can you reproduce it? Can
you do it pretty much on demand? And are the numbers *always* the same?

>
> Running: Fsck -D -f -v -p, results in:
>

Can you run this command instead? e2fsck -f /dev/XXXX

And send me the output? The -p overrides the -f option, so it wasn't
doing a full fsck check. It should have done a full check if the file
system was marked as containing an error, regardless of the -p, but
there was a bug that was fixed in 3.5-rc1 which prevented that. I'm
at a loss to explain why you were still seeing problem in 3.5-rc1 ---
was the fsck log from after running a kernel running 3.5-rc1? In any
case, please do a full fsck using "e2fsck -f /dev/XXX" and send me the
output from that command.

Regards,

- Ted

2012-06-05 00:04:05

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

Hi,

On Mon, Jun 04, 2012 at 11:49:41PM +0200, Sander Eikelenboom wrote:
> I have enabled some debug options, this is from around when the error occurs (kernel: [ 950.610126] EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1699, 32254 clusters in bitmap, 32258 in gd)

Today I've finally managed to bisect this to here:
[fd034a84e1ea5c8c8d159cd2089c32e792c269b0] ext4: split out ext4_free_blocks_after_init()

If I revert all the "ext4:" patches in 3.2 up to and including that one,
I get normal behavior again.

-Kees

--
Kees Cook
ChromeOS Security

2012-06-05 06:56:37

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

Tuesday, June 5, 2012, 1:04:04 AM, you wrote:

> On Mon, Jun 04, 2012 at 07:20:48PM +0200, Sander Eikelenboom wrote:
>> Hello Ted,
>>
>> I have a problem back that occured , but didn't receive much respons in debugging:
>>
>> [ 4688.270789] EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd
>> [ 4688.279172] Aborting journal on device dm-42-8.
>> [ 4688.288634] EXT4-fs (dm-42): Remounting filesystem read-only
>> [ 4688.299011] EXT4-fs (dm-42): ext4_da_writepages: jbd2_start: 6144 pages, ino 15597569; err -30

> Ah, sorry, I didn't see this message when I responded to your earlier
> message (you didn't mail thread it). I also didn't recall your
> earlier complaint until I did an search of my mail archives.

> The main problem is that we don't have an easy reproduction case.
> It's not a problem which has been showing up on any of my testing.
> Earlier you had said that this happened after a read-only snapshot, so
> I had assumed it was an DM issue.

Since it doesn't seem to reported much, it is perhaps related to DM, but where the actual cause is I can't say, and the symptoms are in ext4, so that seems where to start.
Especially since it's odd that e2fsck doesn't report and fix anything (apart from the journal).

> But you say this time it's not happening without a snapshot. OK, how
> frequently does this happen? How easily can you reproduce it? Can
> you do it pretty much on demand? And are the numbers *always* the same?

This time it IS happening WITHOUT a snapshot of this particular lvm-partition.
The numbers are currently always the same, so it's completely reproducible, just by copying say about 10-50 Mb of data to the filesystem.

>>
>> Running: Fsck -D -f -v -p, results in:
>>

> Can you run this command instead? e2fsck -f /dev/XXXX
> And send me the output? The -p overrides the -f option, so it wasn't
> doing a full fsck check. It should have done a full check if the file
> system was marked as containing an error, regardless of the -p, but
> there was a bug that was fixed in 3.5-rc1 which prevented that. I'm
> at a loss to explain why you were still seeing problem in 3.5-rc1 ---
> was the fsck log from after running a kernel running 3.5-rc1? In any
> case, please do a full fsck using "e2fsck -f /dev/XXX" and send me the
> output from that command.

serveerstertje:~# e2fsck -f /dev/serveerstertje/media
e2fsck 1.41.12 (17-May-2010)
/dev/serveerstertje/media: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/serveerstertje/media: ***** FILE SYSTEM WAS MODIFIED *****
/dev/serveerstertje/media: 19122/16384000 files (8.1% non-contiguous), 55602577/65536000 blocks

It didn't prompt for any error and yes this is with running 3.5-rc1.
Perhaps fsck gets the correct numbers and matching numbers for the clusters in the bitmap and gd, and it only differs when actually writing something to the partition, due to some bug ?

I also saw a report from Kees Cook, he has slightly different numbers, 32258 clusters in bitmap, 32262 in gd, although the 32258 seems to match.

> Regards,

> - Ted

--
Best regards,
Sander mailto:[email protected]

2012-06-05 20:42:06

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

Tuesday, June 5, 2012, 2:03:56 AM, you wrote:

> Hi,

> On Mon, Jun 04, 2012 at 11:49:41PM +0200, Sander Eikelenboom wrote:
>> I have enabled some debug options, this is from around when the error occurs (kernel: [ 950.610126] EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1699, 32254 clusters in bitmap, 32258 in gd)

> Today I've finally managed to bisect this to here:
> [fd034a84e1ea5c8c8d159cd2089c32e792c269b0] ext4: split out ext4_free_blocks_after_init()

> If I revert all the "ext4:" patches in 3.2 up to and including that one,
> I get normal behavior again.

Hmm that seems consistent with this report which mentions 3.1.5 as last working kernel:
http://comments.gmane.org/gmane.comp.file-systems.ext4/30575

So somehow this only effects ext4 on DM (it's THE common factor in all reports) and fsck seems incapable of seeing the error (and thus repair it.)

--
Sander

> -Kees

2012-06-05 21:08:15

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

On Tue, Jun 05, 2012 at 10:41:54PM +0200, Sander Eikelenboom wrote:
>
> > Today I've finally managed to bisect this to here:
> > [fd034a84e1ea5c8c8d159cd2089c32e792c269b0] ext4: split out ext4_free_blocks_after_init()
>
> > If I revert all the "ext4:" patches in 3.2 up to and including that one,
> > I get normal behavior again.

This is excellent. Thanks! I'll take a deep look at that ocmmit to
see what might be going on.

> So somehow this only effects ext4 on DM (it's THE common factor in all reports) and fsck seems incapable of seeing the error (and thus repair it.)

What's probably happening is it's only the in-memory copy of the
bitmap which is getting corrupted, and because the file system is
getting remounted read-only, the corrupted bitmap is not getting
written back to disk. (This is the whole point of remount-ro when fs
corruptions are detected. :-)

Cheers,

- Ted

2012-06-05 21:35:20

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

Hello Ted,

Tuesday, June 5, 2012, 11:08:06 PM, you wrote:

> On Tue, Jun 05, 2012 at 10:41:54PM +0200, Sander Eikelenboom wrote:
>>
>> > Today I've finally managed to bisect this to here:
>> > [fd034a84e1ea5c8c8d159cd2089c32e792c269b0] ext4: split out ext4_free_blocks_after_init()
>>
>> > If I revert all the "ext4:" patches in 3.2 up to and including that one,
>> > I get normal behavior again.

> This is excellent. Thanks! I'll take a deep look at that ocmmit to
> see what might be going on.

>> So somehow this only effects ext4 on DM (it's THE common factor in all reports) and fsck seems incapable of seeing the error (and thus repair it.)

> What's probably happening is it's only the in-memory copy of the
> bitmap which is getting corrupted, and because the file system is
> getting remounted read-only, the corrupted bitmap is not getting
> written back to disk. (This is the whole point of remount-ro when fs
> corruptions are detected. :-)

The commits Kees pointed too, made me:

- Copy files to the fs and trigger the error.
- "fsck -v -f", no errors where reported, only journal recovered.
- "tune2fs -O ^uninit_bg" on the fs.
- "fsck -v -f", which found some bitmap size errors now, let it fix these.
- Mounted the fs
- Copy the same set of files, no error has occured.

So it seems related to the uninit_bg changes in the commits on 10 september 2011.

--
Sander

> Cheers,

> - Ted

--
Best regards,
Sander mailto:[email protected]

2012-06-05 23:37:41

by Andi Kleen

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

Sander Eikelenboom <[email protected]> writes:
>
> Hmm that seems consistent with this report which mentions 3.1.5 as last working kernel:
> http://comments.gmane.org/gmane.comp.file-systems.ext4/30575
>
> So somehow this only effects ext4 on DM (it's THE common factor in all reports) and fsck seems incapable of seeing the error (and thus repair it.)

I have a system that throws this error all the time and it's not using
DM.

-Andi
--
[email protected] -- Speaking for myself only

2012-06-06 07:01:35

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

Hello Ted,

Another thing that thought of that could be of interest, i think this was a ext3 converted to ext4 partition.
Don't know if that is true for the other reports as well though.

--
Sander

Tuesday, June 5, 2012, 11:08:06 PM, you wrote:

> On Tue, Jun 05, 2012 at 10:41:54PM +0200, Sander Eikelenboom wrote:
>>
>> > Today I've finally managed to bisect this to here:
>> > [fd034a84e1ea5c8c8d159cd2089c32e792c269b0] ext4: split out ext4_free_blocks_after_init()
>>
>> > If I revert all the "ext4:" patches in 3.2 up to and including that one,
>> > I get normal behavior again.

> This is excellent. Thanks! I'll take a deep look at that ocmmit to
> see what might be going on.

>> So somehow this only effects ext4 on DM (it's THE common factor in all reports) and fsck seems incapable of seeing the error (and thus repair it.)

> What's probably happening is it's only the in-memory copy of the
> bitmap which is getting corrupted, and because the file system is
> getting remounted read-only, the corrupted bitmap is not getting
> written back to disk. (This is the whole point of remount-ro when fs
> corruptions are detected. :-)

> Cheers,

> - Ted

--
Best regards,
Sander mailto:[email protected]

2012-06-06 14:32:56

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

On Wed, Jun 6, 2012 at 12:01 AM, Sander Eikelenboom
<[email protected]> wrote:
> Another thing that thought of that could be of interest, i think this was a ext3 converted to ext4 partition.
> Don't know if that is true for the other reports as well though.

FWIW, in my case, I have just built a fresh ext4 and immediately copy
data to it.

-Kees

--
Kees Cook
Chrome OS Security

2012-06-06 14:40:21

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

Wednesday, June 6, 2012, 4:32:52 PM, you wrote:

> On Wed, Jun 6, 2012 at 12:01 AM, Sander Eikelenboom
> <[email protected]> wrote:
>> Another thing that thought of that could be of interest, i think this was a ext3 converted to ext4 partition.
>> Don't know if that is true for the other reports as well though.

> FWIW, in my case, I have just built a fresh ext4 and immediately copy
> data to it.

> -Kees

And it still results in the error ?
And if you use it without the uninit_bg feature by running:

tune2fs -O ^uninit_bg

Then fsck found errors in my case after fixing them, i could copy files normally.
Does that work in your case as well ?
--
Sander

2012-06-07 04:28:08

[permalink] [raw]

Subject: Re: EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

I think this patch should fix things. Could you give it a try (after
re-enabling uninit_bg and then running e2fsck on the file systems)?

Many thanks,

- Ted

commit b6b218940d47f20fc14e4d203930c575be0c65bf
Author: Theodore Ts'o <[email protected]>
Date: Thu Jun 7 00:26:56 2012 -0400

ext4: fix the free blocks calculation for ext3 file systems w/ uninit_bg

Ext3 filesystems that are converted to use as many ext4 file systems
as possible will enable uninit_bg to speed up e2fsck times. File
systems with an native ext3 layout of inode tables and block
allocation bitmaps (as opposed to ext4's flex_bg layout), ext4 would
sometimes incorrectly calculate the number of free blocks in an
uninitialized block group. As a result, when first allocating a block
in block group marked uninitalized, ext4 would incorrectly report that
the file system was corrupt:

EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1699, 32254 clusters in bitmap, 32258 in gd

Fix the calculation in ext4_num_overhead_clusters() to avoid this
problem.

Signed-off-by: "Theodore Ts'o" <[email protected]>

diff --git a/fs/ext4/balloc.c b/fs/ext4/balloc.c
index 99b6324..cee7812 100644
--- a/fs/ext4/balloc.c
+++ b/fs/ext4/balloc.c
@@ -90,8 +90,8 @@ unsigned ext4_num_overhead_clusters(struct super_block *sb,
* unusual file system layouts.
*/
if (ext4_block_in_group(sb, ext4_block_bitmap(sb, gdp), block_group)) {
- block_cluster = EXT4_B2C(sbi, (start -
- ext4_block_bitmap(sb, gdp)));
+ block_cluster = EXT4_B2C(sbi,
+ ext4_block_bitmap(sb, gdp) - start);
if (block_cluster < num_clusters)
block_cluster = -1;
else if (block_cluster == num_clusters) {
@@ -102,7 +102,7 @@ unsigned ext4_num_overhead_clusters(struct super_block *sb,

if (ext4_block_in_group(sb, ext4_inode_bitmap(sb, gdp), block_group)) {
inode_cluster = EXT4_B2C(sbi,
- start - ext4_inode_bitmap(sb, gdp));
+ ext4_inode_bitmap(sb, gdp) - start);
if (inode_cluster < num_clusters)
inode_cluster = -1;
else if (inode_cluster == num_clusters) {
@@ -114,7 +114,7 @@ unsigned ext4_num_overhead_clusters(struct super_block *sb,
itbl_blk = ext4_inode_table(sb, gdp);
for (i = 0; i < sbi->s_itb_per_group; i++) {
if (ext4_block_in_group(sb, itbl_blk + i, block_group)) {
- c = EXT4_B2C(sbi, start - itbl_blk + i);
+ c = EXT4_B2C(sbi, itbl_blk + i - start);
if ((c < num_clusters) || (c == inode_cluster) ||
(c == block_cluster) || (c == itbl_cluster))
continue;

2012-06-07 22:28:55

by djwong

[permalink] [raw]

Subject: Re: [dm-devel] EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

On Thu, Jun 07, 2012 at 12:27:56AM -0400, Ted Ts'o wrote:
> I think this patch should fix things. Could you give it a try (after
> re-enabling uninit_bg and then running e2fsck on the file systems)?
>
> Many thanks,
>
> - Ted
>
> commit b6b218940d47f20fc14e4d203930c575be0c65bf
> Author: Theodore Ts'o <[email protected]>
> Date: Thu Jun 7 00:26:56 2012 -0400
>
> ext4: fix the free blocks calculation for ext3 file systems w/ uninit_bg
>
> Ext3 filesystems that are converted to use as many ext4 file systems
> as possible

Just to clarify the description a bit:

Did you mean "..as many ext4 filesystem _features_ as possible" ?

> will enable uninit_bg to speed up e2fsck times. File
> systems with an native ext3 layout of inode tables and block

Perhaps "On file systems with a native ext3 layout..." ?

--D

> allocation bitmaps (as opposed to ext4's flex_bg layout), ext4 would
> sometimes incorrectly calculate the number of free blocks in an
> uninitialized block group. As a result, when first allocating a block
> in block group marked uninitalized, ext4 would incorrectly report that
> the file system was corrupt:
>
> EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1699, 32254 clusters in bitmap, 32258 in gd
>
> Fix the calculation in ext4_num_overhead_clusters() to avoid this
> problem.
>
> Signed-off-by: "Theodore Ts'o" <[email protected]>
>
> diff --git a/fs/ext4/balloc.c b/fs/ext4/balloc.c
> index 99b6324..cee7812 100644
> --- a/fs/ext4/balloc.c
> +++ b/fs/ext4/balloc.c
> @@ -90,8 +90,8 @@ unsigned ext4_num_overhead_clusters(struct super_block *sb,
> * unusual file system layouts.
> */
> if (ext4_block_in_group(sb, ext4_block_bitmap(sb, gdp), block_group)) {
> - block_cluster = EXT4_B2C(sbi, (start -
> - ext4_block_bitmap(sb, gdp)));
> + block_cluster = EXT4_B2C(sbi,
> + ext4_block_bitmap(sb, gdp) - start);
> if (block_cluster < num_clusters)
> block_cluster = -1;
> else if (block_cluster == num_clusters) {
> @@ -102,7 +102,7 @@ unsigned ext4_num_overhead_clusters(struct super_block *sb,
>
> if (ext4_block_in_group(sb, ext4_inode_bitmap(sb, gdp), block_group)) {
> inode_cluster = EXT4_B2C(sbi,
> - start - ext4_inode_bitmap(sb, gdp));
> + ext4_inode_bitmap(sb, gdp) - start);
> if (inode_cluster < num_clusters)
> inode_cluster = -1;
> else if (inode_cluster == num_clusters) {
> @@ -114,7 +114,7 @@ unsigned ext4_num_overhead_clusters(struct super_block *sb,
> itbl_blk = ext4_inode_table(sb, gdp);
> for (i = 0; i < sbi->s_itb_per_group; i++) {
> if (ext4_block_in_group(sb, itbl_blk + i, block_group)) {
> - c = EXT4_B2C(sbi, start - itbl_blk + i);
> + c = EXT4_B2C(sbi, itbl_blk + i - start);
> if ((c < num_clusters) || (c == inode_cluster) ||
> (c == block_cluster) || (c == itbl_cluster))
> continue;
>
> --
> dm-devel mailing list
> [email protected]
> https://www.redhat.com/mailman/listinfo/dm-devel
>

2012-06-07 22:40:44

[permalink] [raw]

Subject: Re: [dm-devel] EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

On Thu, Jun 7, 2012 at 3:27 PM, djwong <[email protected]> wrote:
> On Thu, Jun 07, 2012 at 12:27:56AM -0400, Ted Ts'o wrote:
>> I think this patch should fix things. ?Could you give it a try (after
>> re-enabling uninit_bg and then running e2fsck on the file systems)?
>>
>> Many thanks,
>>
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? - Ted
>>
>> commit b6b218940d47f20fc14e4d203930c575be0c65bf
>> Author: Theodore Ts'o <[email protected]>
>> Date: ? Thu Jun 7 00:26:56 2012 -0400
>>
>> ? ? ext4: fix the free blocks calculation for ext3 file systems w/ uninit_bg
>>
>> ? ? Ext3 filesystems that are converted to use as many ext4 file systems
>> ? ? as possible
>
> Just to clarify the description a bit:
>
> Did you mean "..as many ext4 filesystem _features_ as possible" ?
>
>> will enable uninit_bg to speed up e2fsck times. ?File
>> ? ? systems with an native ext3 layout of inode tables and block
>
> Perhaps "On file systems with a native ext3 layout..." ?

FWIW, I was building the filesystem that triggered this as ext4:

mkfs.ext4 -T default -m 0 -O ^huge_file,^flex_bg -E
discard,lazy_itable_init /dev/mapper/...

-Kees

--
Kees Cook
Chrome OS Security

2012-06-07 22:54:56

[permalink] [raw]

Subject: Re: [dm-devel] EXT4-fs error (device dm-42): ext4_mb_generate_buddy:741: group 1904, 32254 clusters in bitmap, 32258 in gd

On Thu, Jun 07, 2012 at 03:40:40PM -0700, Kees Cook wrote:
>
> FWIW, I was building the filesystem that triggered this as ext4:
>
> mkfs.ext4 -T default -m 0 -O ^huge_file,^flex_bg -E
> discard,lazy_itable_init /dev/mapper/...

Ouy of curiosity, was there a reason you chose those particular file
system parameters? It's a surprising set, because if you're starting
with a fresh file system, enabling flex_bg produces a more optimal
file system layout.

The reason why I wrote the commit description way I did was because it
seemed to be the most likely way that someone would trip across this
bug. We didn't notice this for as long as we did since most of the
users of ext4 (especially those who use community distributions) have
been using freshly mkfs'ed ext4 file systems.

- Ted

2012-06-07 23:38:35