Hello Ext4 Developers,
I hope this email finds you well. We are reaching out to report a
persistent issue that we have been facing on Windows Subsystem for
Linux (WSL)[1] with various kernel versions. We have encountered the
problem on kernel versions v5.15, v6.1, v6.6 stable kernels, and also
the current upstream kernel. While the issue takes longer to reproduce
on v5.15, it is consistently observable across these versions.
Issue Description:
Intermittent segfault with memory corruption. The time of segfault and
output can vary, though one of the notable failures manifests as a
segfault with the following error message:
EXT4-fs error (device sdc): ext4_find_dest_de:2092: inode #32168:
block 2334198: comm dpkg: bad entry in directory: rec_len is smaller
than minimal - offset=0, inode=0, rec_len=0, size=4084 fake=0
and
EXT4-fs warning (device sdc): dx_probe:890: inode #27771: comm dpkg:
dx entry: limit 0 != root limit 508
EXT4-fs warning (device sdc): dx_probe:964: inode #27771: comm dpkg:
Corrupt directory, running e2fsck is recommended
EXT4-fs error (device sdc): ext4_empty_dir:3098: inode #27753: block
133944722: comm dpkg: bad entry in directory: rec_len is smaller than
minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
or we see a segfault message where the source can change depending on
which command we're testing with (dpkg, apt, gcc..):
dpkg[135]: segfault at 0 ip 00007f9209eb6a19 sp 00007ffd8a6a0b08 error
4 in libc-2.31.so[7f9209d6e000+159000] likely on CPU 1 (core 0, socket
0)
Reproduction Steps:
The steps to reproduce the issue are seemingly straightforward: Run an
install or upgrade. The larger the change the better.
Installing Gimp has been a go to for testing, though we have
reproduced the failure with:
- apt upgrade
- apt install
- dpkg install
- gcc building source files
Observations:
The issue occurs consistently across multiple kernel versions.
Reproduction is faster on more recent kernels.
Longer intervals are required for v5.15.
When adding more debugging options that increases processing time,
segfault seems to be harder to hit.
When DX_DEBUG is enabled, during the installation process(dpkg
install), we observed instances where both rlen and de->name_len
values become 0.
We wanted to bring this to your attention and seek guidance on how we
could proceed with debugging and resolving this issue. Your insights
and assistance would be greatly appreciated.
Thank you for your time and consideration.
[1] What is Windows Subsystem for Linux:
https://learn.microsoft.com/en-us/windows/wsl/about
--
- Allen
On Thu, Jan 11, 2024 at 07:26:06AM -0800, Allen wrote:
>
> I hope this email finds you well. We are reaching out to report a
> persistent issue that we have been facing on Windows Subsystem for
> Linux (WSL)[1] with various kernel versions. We have encountered the
> problem on kernel versions v5.15, v6.1, v6.6 stable kernels, and also
> the current upstream kernel. While the issue takes longer to reproduce
> on v5.15, it is consistently observable across these versions.
You've tried reproducing (successfully) the problem across multiple
kernel versions. Have you tried reproducing this on multiple
different hardware platforms? e.g., with different desktops and/or
servers, and with different storage devices?
The symptoms you are reporting are very highly correlated with
hardware problems, or in the case where you are running under
virtualization, with bugs in the VMM and/or the host OS's storage
stack.
In particular, these errors:
> EXT4-fs error (device sdc): ext4_find_dest_de:2092: inode #32168:
> block 2334198: comm dpkg: bad entry in directory: rec_len is smaller
> than minimal - offset=0, inode=0, rec_len=0, size=4084 fake=0
>
> and
>
> EXT4-fs warning (device sdc): dx_probe:890: inode #27771: comm dpkg:
> dx entry: limit 0 != root limit 508
> EXT4-fs warning (device sdc): dx_probe:964: inode #27771: comm dpkg:
> Corrupt directory, running e2fsck is recommended
> EXT4-fs error (device sdc): ext4_empty_dir:3098: inode #27753: block
> 133944722: comm dpkg: bad entry in directory: rec_len is smaller than
> minimal - offset=0, inode=0, rec_len=0, size=4096 fake=0
... sesem to hint that ext4 has read a directory block where all or
part of its contents have been replaced with all zeros (hence the
record length, or the hash tree index, is zero). That typically is
caused by a hardware and/or VMM problem.
> or we see a segfault message where the source can change depending on
> which command we're testing with (dpkg, apt, gcc..):
>
> dpkg[135]: segfault at 0 ip 00007f9209eb6a19 sp 00007ffd8a6a0b08 error
> 4 in libc-2.31.so[7f9209d6e000+159000] likely on CPU 1 (core 0, socket
> 0)
And this could very well be because a data block has been replaced
with garbage, or the wrong data block, or all zeroes.
It might also be load related --- that is, the problem only shows up
the system is more heavily loaded, which might explain when enabling
debugging causes the problem to be harder to reproduce.
I am very doubtful that the problem is in the ext4 code proper,
especially since no one else has reported this problem, and at $WORK,
we are running continuous testing where we are running fstests runs on
ext4 against a wide range of hardware (e.g., HDD's, SSD's, iscsi,
etc.) and hardware platforms (arm64 and x86). And that's just for our
data center kernels which are based on various LTS kernels. For
Google's Compute Optimized OS, which is used in both 1st party and 3rd
party VM's in Google Cloud VM's, we are doing similar testing using
gce-xfstests[1] on a continuous basis, and we haven't seen the kind of
bugs that you are reporting.
[1] https://thunk.org/gce-xfstests.
For that matter, I am regularly running gce-xfstests for ext4's
upstream development, and other ext4 developers run fstests using
kvm-xfstests and fstests on a varriety of different hardware devices
and virtualization environments. So that tends to suggest that the
problem is either in the hardware or virtualization environment (WSL)
that you are using.
So to that end, you might want to consider running some lower-level
tests --- for example using fio with data verification enabled. We
also get a huge amount of mileage using fstests to detect problems
lower in the file system stack. This is why we use fstests/xfstests
on ext4 for essentially every single storage device (such as iSCSI,
HDD, Flash, etc.) So setting up fstesets on a variety of file systems
and storage devices is not a bad idea.
It shouldn't be difficult to take the test appliance in
kvm-xfstests[2][3] and getting it to work under WSL. (For example,
over the holidays, I've gotten fstests running on MacOS on a Macbook
Air M2 15" using the hvf framework.) However, I suggest that you
focus on lower-level block and memory stress testing before worrying
about how to run fstests under WSL.
[2] https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md
[3] https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-xfstests.md
Cheers,
- Ted