I have yet to bisect, but have had trouble with recent, post 5.1.0 kernels built from Linus' git head on both i386 (Pentium 4 pc) and amd64 (Athlon II X4 640).
The easiest way to trigger the problem is:
git gc
on the kernel source tree, although the problem can occur without doing a git gc.
The filesystem with the kernel source tree is the root file system, ext3, mounted as:
/dev/sdb7 on / type ext3 (rw,relatime,errors=remount-ro)
After the "Compressing objects" stage, the following appears in dmesg:
[ 848.968550] EXT4-fs error (device sdb7): ext4_get_branch:171: inode #8: block 30343695: comm jbd2/sdb7-8: invalid block
[ 849.077426] Aborting journal on device sdb7-8.
[ 849.100963] EXT4-fs (sdb7): Remounting filesystem read-only
[ 849.100976] jbd2_journal_bmap: journal block not found at offset 989 on sdb7-8
fsck -yv
then reports:
# fsck -yv
fsck from util-linux 2.33.1
e2fsck 1.45.0 (6-Mar-2019)
/dev/sdb7: recovering journal
/dev/sdb7 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (4619656, counted=4619444).
Fix? yes
Free inodes count wrong (15884075, counted=15884058).
Fix? yes
/dev/sdb7: ***** FILE SYSTEM WAS MODIFIED *****
Other times, I have gotten:
"Inodes that were part of a corrupted orphan linked list found."
"Block bitmap differences:"
"Free blocks sound wrong for group"
No problems have been observed with the 5.1.0 release kernel.
Any suggestions for narrowing down the issue welcome.
Arthur.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
On 12 May 2019 7:36:59 am ACST, Theodore Ts'o <[email protected]> wrote:
>On Sat, May 11, 2019 at 02:43:16PM +0200, Richard Weinberger wrote:
>> [CC'in linux-ext4]
>>
>> On Sat, May 11, 2019 at 1:47 PM Arthur Marsh
>> <[email protected]> wrote:
>> >
>> >
>> > The filesystem with the kernel source tree is the root file system,
>ext3, mounted as:
>> >
>> > /dev/sdb7 on / type ext3 (rw,relatime,errors=remount-ro)
>> >
>> > After the "Compressing objects" stage, the following appears in
>dmesg:
>> >
>> > [ 848.968550] EXT4-fs error (device sdb7): ext4_get_branch:171:
>inode #8: block 30343695: comm jbd2/sdb7-8: invalid block
>> > [ 849.077426] Aborting journal on device sdb7-8.
>> > [ 849.100963] EXT4-fs (sdb7): Remounting filesystem read-only
>> > [ 849.100976] jbd2_journal_bmap: journal block not found at offset
>989 on sdb7-8
>
>This indicates that the extent tree blocks for the journal was found
>to be corrupt; so the journal couldn't be found.
>
>> > # fsck -yv
>> > fsck from util-linux 2.33.1
>> > e2fsck 1.45.0 (6-Mar-2019)
>> > /dev/sdb7: recovering journal
>> > /dev/sdb7 contains a file system with errors, check forced.
>
>But e2fsck had no problem finding the journal.
>
>> > Pass 1: Checking inodes, blocks, and sizes
>> > Pass 2: Checking directory structure
>> > Pass 3: Checking directory connectivity
>> > Pass 4: Checking reference counts
>> > Pass 5: Checking group summary information
>> > Free blocks count wrong (4619656, counted=4619444).
>> > Fix? yes
>> >
>> > Free inodes count wrong (15884075, counted=15884058).
>> > Fix? yes
>
>And no other significant problems were found. (Ext4 never updates or
>relies on the summary number of free blocks and free inodes, since
>updating it is a scalability bottleneck and these values can be
>calculated from the per block group free block/inodes count. So the
>fact that e2fsck needed to update them is not an issue.)
>
>So that implies that we got one set of values when we read the journal
>inode when attempting to mount the file system, and a *different* set
>of values when e2fsck was run. Which makes means that we need
>consider the possibility that the problem is below the file system
>layer (e.g., the block layer, device drivers, etc.).
>
>
>> > /dev/sdb7: ***** FILE SYSTEM WAS MODIFIED *****
>> >
>> > Other times, I have gotten:
>> >
>> > "Inodes that were part of a corrupted orphan linked list found."
>> > "Block bitmap differences:"
>> > "Free blocks sound wrong for group"
>> >
>
>This variety of issues also implies that the issue may be in the data
>read by the file system, as opposed to an issue in the file system.
>
>Arthur, can you give us the full details of your hardware
>configuration and your kernel config file? Also, what kernel git
>commit ID were you testing?
>
> - Ted
Please see the attachments, this machine has a Pentium-D and PATA drive for the root filesystem and 32 bit kernel but I also experienced the same problem on an Athlon II X4 640 machine with SATA drive for the root filesystem and 64 bit kernel.
Arthur.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Tuesday 14 May 2019, Arthur Marsh wrote:
> Apologies, I had forgotten to
>
> got bisect - - hard origin/master
>
> I am still seeing the corruption leading to the invalid block error on 5.1.0+ kernels on both my machines.
>
> Arthur.
I've been probably hit by the same bug. ext3 filesystem on my test machine was corrupted twice with 5.1.0+. Only the corruption was heavier. Some files that were open (e.g. logs) became cros-linked with files that were not used for ages.
--
Ondrej Zary
On Mon, Jul 01, 2019 at 02:43:14PM +0200, Geert Uytterhoeven wrote:
> Hi Ted,
>
> Despite this fix having been applied upstream, the kernel prints from
> time to time:
>
> EXT4-fs (sda1): error count since last fsck: 5
> EXT4-fs (sda1): initial error at time 1557931133:
> ext4_get_branch:171: inode 1980: block 27550
> EXT4-fs (sda1): last error at time 1558114349:
> ext4_get_branch:171: inode 1980: block 27550
>
> This happens even after a manual run of "e2fsck -f" (while it's mounted
> RO), which reports a clean file system.
What's happening is this. When the kernel detects a corruption, newer
kernels will set these superblock fields:
__le32 s_error_count; /* number of fs errors */
__le32 s_first_error_time; /* first time an error happened */
__le32 s_first_error_ino; /* inode involved in first error */
__le64 s_first_error_block; /* block involved of first error */
__u8 s_first_error_func[32] __nonstring; /* function where the error happened */
__le32 s_first_error_line; /* line number where error happened */
__le32 s_last_error_time; /* most recent time of an error */
__le32 s_last_error_ino; /* inode involved in last error */
__le32 s_last_error_line; /* line number where error happened */
__le64 s_last_error_block; /* block involved of last error */
__u8 s_last_error_func[32] __nonstring; /* function where the error happened */
When newer versions of e2fsck *fix* the corruption, it will clear
these fields. It's basically a safety check because *way* too many
ext4 users run with errors=continue (aka, "don't worry, be happy"
mode), and so this is a poke in the system logs that the file system
is corrupted, and they, really, *REALLY* should fix it before they
lose (more) data.
> The inode and block numbers match the numbers printed due to the
> previous bug.
You can also see when the last file system error was detected via:
% date -d @1558114349
Fri 17 May 2019 01:32:29 PM EDT
> Do you have an idea what's wrong?
> Note that I run a very old version of e2fsck (from a decade ago).
... and that's the problem. If you're going to be using newer
versions of the kernel, you really should be using newer versions of
e2fsprogs.
There have been a lot of bug fixes in the last 10 years, and some of
them can be data corruption bugs....
- Ted