Hi!
I have a dirty ext4 volume. The system just hangs at startup. After removing that volume from fstab the system starts.
Mount and e2fsck just flood with "Invalid checksum recovering block 1152 in log" messages.
(Mounting with ro,noload let me access most files.)
I also tried the e2fsck from current git.
debugfs just fails with "Block bitmap checksum does not match bitmap while reading block bitmap". (catastrophic mode does work)
Three points:
- e2fsck should not get into an endless loop, blocking the system.
- mount should not get into an endless loop, blocking the system and flooding the system log.
- At least e2fsck should fix the filesystem.
Any help or hints?
Thanks,
Markus
PS: Original lkml-mail:
https://lkml.org/lkml/2014/4/1/467
On Fri, Apr 04, 2014 at 12:35:45PM +0200, Markus wrote:
> Hi!
>
> I have a dirty ext4 volume. The system just hangs at startup. After removing that volume from fstab the system starts.
>
> Mount and e2fsck just flood with "Invalid checksum recovering block 1152 in log" messages.
>
> (Mounting with ro,noload let me access most files.)
>
> I also tried the e2fsck from current git.
>
> debugfs just fails with "Block bitmap checksum does not match bitmap while reading block bitmap". (catastrophic mode does work)
>
> Three points:
> - e2fsck should not get into an endless loop, blocking the system.
> - mount should not get into an endless loop, blocking the system and flooding the system log.
> - At least e2fsck should fix the filesystem.
>
>
> Any help or hints?
Hmm, that's probably a bug in the journal replay code. :(
Can you send me the output of "e2image -r /dev/sdXX - | bzip2 > hd.e2i.bz2" if
it's not too huge? The exact error messages (if you can capture/photograph
them) would also be useful.
I'm guessing you have metadata_csum enabled...
PS: lkml.org is dead; I'm assuming the URL referenced the discussion "Ext4
Recovery: Invalid checksum recovering block # in log" but it's hard to tell
since there was no subject line provided with that URL.
--D
>
>
> Thanks,
> Markus
>
>
> PS: Original lkml-mail:
> https://lkml.org/lkml/2014/4/1/467
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi!
Its a md-raid6 of about 10 TiB.
The disks have no bad sectors, long smart-test completetd without errors and
raid check does not have any mismatched blocks.
The e2image is still running. Dont know how it big it will grow.
The e2fsck messages:
> e2fsck 1.43-WIP (4-Feb-2014)
> /dev/md5: recovering journal
> JBD: Invalid checksum recovering block 1152 in log
> JBD: Invalid checksum recovering block 1152 in log
> …
> JBD: Invalid checksum recovering block 1152 in log
> Killed
(needed to kill -9 the process)
=> e2fsck/recovery.c:594
> debugfs 1.43-WIP (4-Feb-2014)
> /dev/md5: Block bitmap checksum does not match bitmap while reading block
> bitmap
The filesystem was not loaded. Catastrophic mode does work.
Or what "exact error messages" do you mean?
metadata_csum is enabled, correct.
Thanks,
Markus
PS: Yes that is the correct lkml message.
Darrick J. Wong wroteat 04.04.2014:
> On Fri, Apr 04, 2014 at 12:35:45PM +0200, Markus wrote:
> > Hi!
> >
> > I have a dirty ext4 volume. The system just hangs at startup. After
removing that volume from fstab the system starts.
> >
> > Mount and e2fsck just flood with "Invalid checksum recovering block 1152
in log" messages.
> >
> > (Mounting with ro,noload let me access most files.)
> >
> > I also tried the e2fsck from current git.
> >
> > debugfs just fails with "Block bitmap checksum does not match bitmap while
reading block bitmap". (catastrophic mode does work)
> >
> > Three points:
> > - e2fsck should not get into an endless loop, blocking the system.
> > - mount should not get into an endless loop, blocking the system and
flooding the system log.
> > - At least e2fsck should fix the filesystem.
> >
> >
> > Any help or hints?
>
> Hmm, that's probably a bug in the journal replay code. :(
>
> Can you send me the output of "e2image -r /dev/sdXX - | bzip2 > hd.e2i.bz2"
if
> it's not too huge? The exact error messages (if you can capture/photograph
> them) would also be useful.
>
> I'm guessing you have metadata_csum enabled...
>
> PS: lkml.org is dead; I'm assuming the URL referenced the discussion "Ext4
> Recovery: Invalid checksum recovering block # in log" but it's hard to tell
> since there was no subject line provided with that URL.
>
> --D
> >
> >
> > Thanks,
> > Markus
> >
> >
> > PS: Original lkml-mail:
> > https://lkml.org/lkml/2014/4/1/467
Hi!
Finally e2image finished successfully. But the produced file is way too big for a mail.
Any other possibility?
(e2image does dump everything except file data and free space. But the problem seems to be just in the bitmap and/or journal.)
Actually, when I look at the code around e2fsck/recovery.c:594
The error is detected and continue is called.
But tagp/tag is never changed, but the checksum is always compared to the one from tag. Intended?
Thanks,
Markus
Markus wrote on 05.04.2014:
> Hi!
>
> Its a md-raid6 of about 10 TiB.
> The disks have no bad sectors, long smart-test completetd without errors and
> raid check does not have any mismatched blocks.
>
> The e2image is still running. Dont know how it big it will grow.
>
> The e2fsck messages:
> > e2fsck 1.43-WIP (4-Feb-2014)
> > /dev/md5: recovering journal
> > JBD: Invalid checksum recovering block 1152 in log
> > JBD: Invalid checksum recovering block 1152 in log
> > …
> > JBD: Invalid checksum recovering block 1152 in log
> > Killed
>
> (needed to kill -9 the process)
> => e2fsck/recovery.c:594
>
> > debugfs 1.43-WIP (4-Feb-2014)
> > /dev/md5: Block bitmap checksum does not match bitmap while reading block
> > bitmap
>
> The filesystem was not loaded. Catastrophic mode does work.
>
> Or what "exact error messages" do you mean?
>
> metadata_csum is enabled, correct.
>
>
> Thanks,
> Markus
>
>
> PS: Yes that is the correct lkml message.
>
>
> Darrick J. Wong wroteat 04.04.2014:
> > On Fri, Apr 04, 2014 at 12:35:45PM +0200, Markus wrote:
> > > Hi!
> > >
> > > I have a dirty ext4 volume. The system just hangs at startup. After
> removing that volume from fstab the system starts.
> > >
> > > Mount and e2fsck just flood with "Invalid checksum recovering block 1152
> in log" messages.
> > >
> > > (Mounting with ro,noload let me access most files.)
> > >
> > > I also tried the e2fsck from current git.
> > >
> > > debugfs just fails with "Block bitmap checksum does not match bitmap while
> reading block bitmap". (catastrophic mode does work)
> > >
> > > Three points:
> > > - e2fsck should not get into an endless loop, blocking the system.
> > > - mount should not get into an endless loop, blocking the system and
> flooding the system log.
> > > - At least e2fsck should fix the filesystem.
> > >
> > >
> > > Any help or hints?
> >
> > Hmm, that's probably a bug in the journal replay code. :(
> >
> > Can you send me the output of "e2image -r /dev/sdXX - | bzip2 > hd.e2i.bz2"
> if
> > it's not too huge? The exact error messages (if you can capture/photograph
> > them) would also be useful.
> >
> > I'm guessing you have metadata_csum enabled...
> >
> > PS: lkml.org is dead; I'm assuming the URL referenced the discussion "Ext4
> > Recovery: Invalid checksum recovering block # in log" but it's hard to tell
> > since there was no subject line provided with that URL.
> >
> > --D
> > >
> > >
> > > Thanks,
> > > Markus
> > >
> > >
> > > PS: Original lkml-mail:
> > > https://lkml.org/lkml/2014/4/1/467
On Mon, Apr 07, 2014 at 12:58:40PM +0200, Markus wrote:
>
> Finally e2image finished successfully. But the produced file is way too big for a mail.
>
> Any other possibility?
> (e2image does dump everything except file data and free space. But the problem seems to be just in the bitmap and/or journal.)
>
> Actually, when I look at the code around e2fsck/recovery.c:594
> The error is detected and continue is called.
> But tagp/tag is never changed, but the checksum is always compared to the one from tag. Intended?
What mount options are you using? It appears that you have journal
checksums enabled, which isn't on by default, and unfortunately,
there's a good reason for that. The original code assumed that the
most common case for journal corruption would be caused by an
incomplete journal transaction getting written out if one were using
journal_async_commit. This feature has not been enabled by default
because the qeustion of what to do when the journal gets corrupted in
other cases is not an easy one.
If some part of a transaction which is not the very last transaction
in the journal gets corrupted, replaying it could do severe damage to
the file system. Unfortunately, simply deleting the journal and then
recreating it could also do more damage as well. Most of the time, a
bad checksum happens because the last transaction hasn't fully made it
out to disk (especially if you use the journal_async_commit option,
which is a bit of a misnomer and has its own caveats[1]). But if the
checksum violation happens in a journal transaction that is not the
last transaction in the journal, right now the recovery code aborts,
because we don't have good automated logic to handle this case.
I suspect if you need to get your file system back on its feet, the
best thing to do is to create a patched e2fsck that doesn't abort when
it finds a checksum error, but instead continues. Then run it to
replay the journal, and then force a full file system check and hope
for the best.
What has been on my todo list to implement, but has been relatively
low priority because this is not a feature that we've documented or
encouraged peple to use, is to have e2fsck skip the transaction has a
bad checksum (i.e., not replay it at all), and then force a full file
system check. This is a bit safer, but if you make e2fsck ignore the
checksum, it's no worse than if journal checksums weren't enabled in
the first place.
The long term thing that we need to add before we can really support
journal checksums is to checksum each individual data block, instead
of just each transaction. Then when we have a bad checksum, we can
skip just the one bad data block, and then force a full fsck.
I'm sorry you ran into this. What I should do is to disable these
mount options for now, since users who stumble across them, as
apparently you have, might be tempted to use them, and then get into
trouble.
- Ted
[1] The issue with journal_async_commit is that it's possible (fairly
unlikely, but still possible) that the guarantees of data=ordered will
be violated. If the data blocks that were written out while we are
resolving a delayed allocation writeback haven't made it all the way
down to the platter, it's possible for all of the journal writes and
the commit block to be reordered ahead of the data blocks. In that
case, the checksum for the commit block would be valid, but some of
the data blocks might not have been written back to disk.
Theodore Ts'o wrote on 07.04.2014:
> On Mon, Apr 07, 2014 at 12:58:40PM +0200, Markus wrote:
> >
> > Finally e2image finished successfully. But the produced file is way too
big for a mail.
> >
> > Any other possibility?
> > (e2image does dump everything except file data and free space. But the
problem seems to be just in the bitmap and/or journal.)
> >
> > Actually, when I look at the code around e2fsck/recovery.c:594
> > The error is detected and continue is called.
> > But tagp/tag is never changed, but the checksum is always compared to the
one from tag. Intended?
>
> What mount options are you using? It appears that you have journal
> checksums enabled, which isn't on by default, and unfortunately,
> there's a good reason for that. The original code assumed that the
> most common case for journal corruption would be caused by an
> incomplete journal transaction getting written out if one were using
> journal_async_commit. This feature has not been enabled by default
> because the qeustion of what to do when the journal gets corrupted in
> other cases is not an easy one.
Normally just "noatime,journal_checksum", but with the corrupted journal I use
"ro,noload".
The "man mount" reads well about that "journal_checksum" option ;)
> If some part of a transaction which is not the very last transaction
> in the journal gets corrupted, replaying it could do severe damage to
> the file system. Unfortunately, simply deleting the journal and then
> recreating it could also do more damage as well. Most of the time, a
> bad checksum happens because the last transaction hasn't fully made it
> out to disk (especially if you use the journal_async_commit option,
> which is a bit of a misnomer and has its own caveats[1]). But if the
> checksum violation happens in a journal transaction that is not the
> last transaction in the journal, right now the recovery code aborts,
> because we don't have good automated logic to handle this case.
The recovery does not seem to abort. It calles continue and is caught in an
endless loop.
> I suspect if you need to get your file system back on its feet, the
> best thing to do is to create a patched e2fsck that doesn't abort when
> it finds a checksum error, but instead continues. Then run it to
> replay the journal, and then force a full file system check and hope
> for the best.
The code calls "continue". ;)
So I just remove the whole if clause:
/* Look for block corruption */
if (!jbd2_block_tag_csum_verify(
journal, tag, obh->b_data,
be32_to_cpu(tmp->h_sequence))) {
- brelse(obh);
- success = -EIO;
printk(KERN_ERR "JBD: Invalid "
"checksum recovering "
"block %lld in log\n",
blocknr);
- continue;
}
It would then ignore the checksum and just issue a message. Right?
> What has been on my todo list to implement, but has been relatively
> low priority because this is not a feature that we've documented or
> encouraged peple to use, is to have e2fsck skip the transaction has a
> bad checksum (i.e., not replay it at all), and then force a full file
> system check. This is a bit safer, but if you make e2fsck ignore the
> checksum, it's no worse than if journal checksums weren't enabled in
> the first place.
>
> The long term thing that we need to add before we can really support
> journal checksums is to checksum each individual data block, instead
> of just each transaction. Then when we have a bad checksum, we can
> skip just the one bad data block, and then force a full fsck.
>
> I'm sorry you ran into this. What I should do is to disable these
> mount options for now, since users who stumble across them, as
> apparently you have, might be tempted to use them, and then get into
> trouble.
>
> - Ted
>
> [1] The issue with journal_async_commit is that it's possible (fairly
> unlikely, but still possible) that the guarantees of data=ordered will
> be violated. If the data blocks that were written out while we are
> resolving a delayed allocation writeback haven't made it all the way
> down to the platter, it's possible for all of the journal writes and
> the commit block to be reordered ahead of the data blocks. In that
> case, the checksum for the commit block would be valid, but some of
> the data blocks might not have been written back to disk.
Thanks so far,
Markus
I patched e2fsck as mentionied below.
./e2fsck /dev/md5
e2fsck 1.43-WIP (4-Feb-2014)
/dev/md5: recovering journal
JBD: Invalid checksum recovering block 1152 in log
JBD: Invalid checksum recovering block 1156 in log
Setting free inodes count to 366227296 (was 366241761)
Setting free blocks count to 652527218 (was 730998757)
/dev/md5: clean, 41120/366268416 files, 2277606286/2930133504 blocks
So two blocks were bad. But the the recovery worked and the last few files all were intact.
A full check did not find any errors:
./e2fsck -f -n /dev/md5
e2fsck 1.43-WIP (4-Feb-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md5: 41120/366268416 files (4.5% non-contiguous), 2277606286/2930133504 blocks
So I think that fs is now fine again.
But still, e2fsck should not be trapped in an endless loop.
Thanks,
Markus
Markus wrote on 07.04.2014:
> Theodore Ts'o wrote on 07.04.2014:
> > On Mon, Apr 07, 2014 at 12:58:40PM +0200, Markus wrote:
> > >
> > > Finally e2image finished successfully. But the produced file is way too
> big for a mail.
> > >
> > > Any other possibility?
> > > (e2image does dump everything except file data and free space. But the
> problem seems to be just in the bitmap and/or journal.)
> > >
> > > Actually, when I look at the code around e2fsck/recovery.c:594
> > > The error is detected and continue is called.
> > > But tagp/tag is never changed, but the checksum is always compared to the
> one from tag. Intended?
> >
> > What mount options are you using? It appears that you have journal
> > checksums enabled, which isn't on by default, and unfortunately,
> > there's a good reason for that. The original code assumed that the
> > most common case for journal corruption would be caused by an
> > incomplete journal transaction getting written out if one were using
> > journal_async_commit. This feature has not been enabled by default
> > because the qeustion of what to do when the journal gets corrupted in
> > other cases is not an easy one.
>
> Normally just "noatime,journal_checksum", but with the corrupted journal I use
> "ro,noload".
>
> The "man mount" reads well about that "journal_checksum" option ;)
>
>
> > If some part of a transaction which is not the very last transaction
> > in the journal gets corrupted, replaying it could do severe damage to
> > the file system. Unfortunately, simply deleting the journal and then
> > recreating it could also do more damage as well. Most of the time, a
> > bad checksum happens because the last transaction hasn't fully made it
> > out to disk (especially if you use the journal_async_commit option,
> > which is a bit of a misnomer and has its own caveats[1]). But if the
> > checksum violation happens in a journal transaction that is not the
> > last transaction in the journal, right now the recovery code aborts,
> > because we don't have good automated logic to handle this case.
>
> The recovery does not seem to abort. It calles continue and is caught in an
> endless loop.
>
>
> > I suspect if you need to get your file system back on its feet, the
> > best thing to do is to create a patched e2fsck that doesn't abort when
> > it finds a checksum error, but instead continues. Then run it to
> > replay the journal, and then force a full file system check and hope
> > for the best.
>
> The code calls "continue". ;)
> So I just remove the whole if clause:
> /* Look for block corruption */
> if (!jbd2_block_tag_csum_verify(
> journal, tag, obh->b_data,
> be32_to_cpu(tmp->h_sequence))) {
> - brelse(obh);
> - success = -EIO;
> printk(KERN_ERR "JBD: Invalid "
> "checksum recovering "
> "block %lld in log\n",
> blocknr);
> - continue;
> }
>
> It would then ignore the checksum and just issue a message. Right?
>
>
> > What has been on my todo list to implement, but has been relatively
> > low priority because this is not a feature that we've documented or
> > encouraged peple to use, is to have e2fsck skip the transaction has a
> > bad checksum (i.e., not replay it at all), and then force a full file
> > system check. This is a bit safer, but if you make e2fsck ignore the
> > checksum, it's no worse than if journal checksums weren't enabled in
> > the first place.
> >
> > The long term thing that we need to add before we can really support
> > journal checksums is to checksum each individual data block, instead
> > of just each transaction. Then when we have a bad checksum, we can
> > skip just the one bad data block, and then force a full fsck.
> >
> > I'm sorry you ran into this. What I should do is to disable these
> > mount options for now, since users who stumble across them, as
> > apparently you have, might be tempted to use them, and then get into
> > trouble.
> >
> > - Ted
> >
> > [1] The issue with journal_async_commit is that it's possible (fairly
> > unlikely, but still possible) that the guarantees of data=ordered will
> > be violated. If the data blocks that were written out while we are
> > resolving a delayed allocation writeback haven't made it all the way
> > down to the platter, it's possible for all of the journal writes and
> > the commit block to be reordered ahead of the data blocks. In that
> > case, the checksum for the commit block would be valid, but some of
> > the data blocks might not have been written back to disk.
>
> Thanks so far,
> Markus
On Tue, Apr 08, 2014 at 04:25:08PM +0200, Markus wrote:
>
> So I think that fs is now fine again.
Great, I'm glad you were able to recover your data.
>
> But still, e2fsck should not be trapped in an endless loop.
I agree, it's been on my todo list to fix, one way or another.
- Ted
On Mon, Apr 07, 2014 at 04:06:50PM +0200, Markus wrote:
> Theodore Ts'o wrote on 07.04.2014:
> > On Mon, Apr 07, 2014 at 12:58:40PM +0200, Markus wrote:
> > >
> > > Finally e2image finished successfully. But the produced file is way too
> big for a mail.
:(
> > >
> > > Any other possibility?
> > > (e2image does dump everything except file data and free space. But the
> problem seems to be just in the bitmap and/or journal.)
Yes, it might be less work if I just turn on data=journal + metadata_csum +
journal_checksum and see if I can easily reproduce it myself. Or, I suppose it
wouldn't be too hard just to format a fresh FS and tweak the journal to
"replay" into arbitrary empty blocks, and then corrupt the journal checksums to
see what happens.
> > >
> > > Actually, when I look at the code around e2fsck/recovery.c:594
> > > The error is detected and continue is called.
> > > But tagp/tag is never changed, but the checksum is always compared to the
> one from tag. Intended?
I think you're right, but that function makes my eyes bleed. :(
> > What mount options are you using? It appears that you have journal
> > checksums enabled, which isn't on by default, and unfortunately,
> > there's a good reason for that. The original code assumed that the
> > most common case for journal corruption would be caused by an
> > incomplete journal transaction getting written out if one were using
> > journal_async_commit. This feature has not been enabled by default
> > because the qeustion of what to do when the journal gets corrupted in
> > other cases is not an easy one.
>
> Normally just "noatime,journal_checksum", but with the corrupted journal I use
> "ro,noload".
>
> The "man mount" reads well about that "journal_checksum" option ;)
>
>
> > If some part of a transaction which is not the very last transaction
> > in the journal gets corrupted, replaying it could do severe damage to
> > the file system. Unfortunately, simply deleting the journal and then
> > recreating it could also do more damage as well. Most of the time, a
> > bad checksum happens because the last transaction hasn't fully made it
> > out to disk (especially if you use the journal_async_commit option,
> > which is a bit of a misnomer and has its own caveats[1]). But if the
> > checksum violation happens in a journal transaction that is not the
> > last transaction in the journal, right now the recovery code aborts,
> > because we don't have good automated logic to handle this case.
>
> The recovery does not seem to abort. It calles continue and is caught in an
> endless loop.
>
>
> > I suspect if you need to get your file system back on its feet, the
> > best thing to do is to create a patched e2fsck that doesn't abort when
> > it finds a checksum error, but instead continues. Then run it to
> > replay the journal, and then force a full file system check and hope
> > for the best.
>
> The code calls "continue". ;)
> So I just remove the whole if clause:
> /* Look for block corruption */
> if (!jbd2_block_tag_csum_verify(
> journal, tag, obh->b_data,
> be32_to_cpu(tmp->h_sequence))) {
> - brelse(obh);
> - success = -EIO;
> printk(KERN_ERR "JBD: Invalid "
> "checksum recovering "
> "block %lld in log\n",
> blocknr);
> - continue;
> }
>
> It would then ignore the checksum and just issue a message. Right?
Umm... I think you just made it replay the corrupt block too. Granted, it
looks as though fsck made everything right anyway, so in this case nothing bad
happened.
> > What has been on my todo list to implement, but has been relatively
> > low priority because this is not a feature that we've documented or
> > encouraged peple to use, is to have e2fsck skip the transaction has a
> > bad checksum (i.e., not replay it at all), and then force a full file
> > system check. This is a bit safer, but if you make e2fsck ignore the
> > checksum, it's no worse than if journal checksums weren't enabled in
> > the first place.
> >
> > The long term thing that we need to add before we can really support
> > journal checksums is to checksum each individual data block, instead
> > of just each transaction. Then when we have a bad checksum, we can
> > skip just the one bad data block, and then force a full fsck.
I think the metadata-csum patchset added per-block checksums, but now that
we've brought it up, I think (IBM) pulled me off ext4 before I could get to
implementing a more sane strategy for replaying with bad checksums. I can git
blame that particular hunk on myself. :/
Ugh, not documented in the on disk format wiki page either. Well, I guess I'll
go update the wiki while I reread the code to figure out just what's going on
here. Sorry about that. Apparently the poweroff testing I did didn't catch
it. <groan>
--D
> > I'm sorry you ran into this. What I should do is to disable these
> > mount options for now, since users who stumble across them, as
> > apparently you have, might be tempted to use them, and then get into
> > trouble.
> >
> > - Ted
> >
> > [1] The issue with journal_async_commit is that it's possible (fairly
> > unlikely, but still possible) that the guarantees of data=ordered will
> > be violated. If the data blocks that were written out while we are
> > resolving a delayed allocation writeback haven't made it all the way
> > down to the platter, it's possible for all of the journal writes and
> > the commit block to be reordered ahead of the data blocks. In that
> > case, the checksum for the commit block would be valid, but some of
> > the data blocks might not have been written back to disk.
>
> Thanks so far,
> Markus
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html