Hi,
I originally observed this issue when creating ext4 disk images on a 9p
filesystem which differed from the images I created on a tmpfs. I observed that
the difference also exists when the underlying file system is fat32, so I'm
using this as an example here. For what it's worth, the ext4 filesystem images
created on a tmpfs are identical to those created on an ext4 fs. To demonstrate
the issue, please see the script at the end of this mail (it requires sudo to
mount and unmount the fat32 disk image). As you can see from the printed
hashes, the disk images produced outside the fat32 disk are always identical as
expected. The diff between the reproducible images and those stored on fat32 is
also very short but I don't know what data is stored at those points:
@@ -85,7 +85,7 @@
00000540: 0000 0000 0000 0000 0000 0000 0000 1000 ................
00000550: 0000 0000 0000 0000 0000 0000 2000 2000 ............ . .
00000560: 0200 0000 0000 0000 0000 0000 0000 0000 ................
-00000570: 0000 0000 0401 0000 8c04 0000 0000 0000 ................
+00000570: 0000 0000 0401 0000 4900 0000 0000 0000 ........I.......
00000580: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000590: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000005a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
@@ -125,9 +125,9 @@
000007c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000007d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000007e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
-000007f0: 0000 0000 0000 0000 0000 0000 264c 0251 ............&L.Q
+000007f0: 0000 0000 0000 0000 0000 0000 64ca bba5 ............d...
00000800: 1200 0000 2200 0000 3200 0000 9d03 7300 ...."...2.....s.
-00000810: 0200 0000 0000 0000 babb 8a41 7300 2004 ...........As. .
+00000810: 0200 0400 0000 0000 babb 8a41 7300 69f5 ...........As.i.
00000820: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000830: 0000 0000 0000 0000 bc7a 6e31 0000 0000 .........zn1....
00000840: 0000 0000 0000 0000 0000 0000 0000 0000 ................
Any idea what is going on? Is there a better way to diff two ext4 disk images
than diffing the xxd output? If I try diffing the dumpe2fs output I get these
differences:
@@ -32,7 +32,7 @@
Maximum mount count: -1
Last checked: Fri May 3 16:14:49 2024
Check interval: 0 (<none>)
-Lifetime writes: 1164 kB
+Lifetime writes: 73 kB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
@@ -44,7 +44,7 @@
Directory Hash Seed: 0b7f9cfd-0113-486c-a453-4f5483bd486b
Journal backup: inode blocks
Checksum type: crc32c
-Checksum: 0x51024c26
+Checksum: 0xa5bbca64
Checksum seed: 0xf81d767d
Orphan file inode: 12
Journal features: (none)
@@ -56,7 +56,7 @@
Journal start: 0
-Group 0: (Blocks 1-2047) csum 0x0420
+Group 0: (Blocks 1-2047) csum 0xf569 [ITABLE_ZEROED]
Primary superblock at 1, Group descriptors at 2-2
Reserved GDT blocks at 3-17
Block bitmap at 18 (+17), csum 0x7abcbbba
Why would these bits differ depending on the filesystem on which the disk image
is stored? Is there a way to equalize this information so that the disk image
looks the same independent on the underlying filesystem?
Thanks!
cheers, josch
#!/bin/sh
set -eu
mkfs() {
imgpath="$1"
rm -f "$imgpath"
dd if=/dev/zero of="$imgpath" bs=1024 count=2048 2>/dev/null
echo H4sIAAAAAAAAA+3OQQrCMBCF4Vl7ihwho9PkPKVEtJgU2rjo7a240JXSRSnC/20ew5vFy/P5ekulzUk24xfB7JkaG/+ZL3oUtaCnYE2IUZZbTcX57Sa93afajs5JP0zd5cvfr/5P5bkbSk2lHvZeAgAAAAAAAAAAAAAAAABY4wEWZDwwACgAAA== \
| base64 -d \
| env LC_ALL=C.UTF-8 SOURCE_DATE_EPOCH=1714745689 /sbin/mke2fs -d - \
-q -F -o Linux -T ext4 -O metadata_csum,64bit \
-U 0b7f9cfd-0113-486c-a453-4f5483bd486b \
-E hash_seed=0b7f9cfd-0113-486c-a453-4f5483bd486b \
-b 1024 "$imgpath"
md5sum "$imgpath"
}
mkfs "/dev/shm/disk.ext4"
mkfs disk.ext4
rm -f fat32.img
mkdir -p mnt
dd if=/dev/zero of=fat32.img bs=1024 count=65536 2>/dev/null
/sbin/mkfs.vfat -F 32 fat32.img
sudo mount -o rw,umask=0000 fat32.img mnt
mkfs mnt/disk.ext4
bash -c 'diff -u <(xxd mnt/disk.ext4) <(xxd disk.ext4) || true'
bash -c 'diff -u <(/sbin/dumpe2fs mnt/disk.ext4) <(/sbin/dumpe2fs disk.ext4) || true'
sudo umount mnt
mkfs disk.ext4
mkfs "/dev/shm/disk.ext4"
rm "/dev/shm/disk.ext4" disk.ext4 fat32.img
rmdir mnt
Quoting Johannes Schauer Marin Rodrigues (2024-05-04 16:32:50)
> I originally observed this issue when creating ext4 disk images on a 9p
> filesystem which differed from the images I created on a tmpfs. I observed
> that the difference also exists when the underlying file system is fat32, so
> I'm using this as an example here. For what it's worth, the ext4 filesystem
> images created on a tmpfs are identical to those created on an ext4 fs. To
> demonstrate the issue, please see the script at the end of this mail (it
> requires sudo to mount and unmount the fat32 disk image). As you can see from
> the printed hashes, the disk images produced outside the fat32 disk are
> always identical as expected. The diff between the reproducible images and
> those stored on fat32 is also very short but I don't know what data is stored
> at those points:
>
> @@ -85,7 +85,7 @@
> 00000540: 0000 0000 0000 0000 0000 0000 0000 1000 ................
> 00000550: 0000 0000 0000 0000 0000 0000 2000 2000 ............ . .
> 00000560: 0200 0000 0000 0000 0000 0000 0000 0000 ................
> -00000570: 0000 0000 0401 0000 8c04 0000 0000 0000 ................
> +00000570: 0000 0000 0401 0000 4900 0000 0000 0000 ........I.......
> 00000580: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000590: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 000005a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> @@ -125,9 +125,9 @@
> 000007c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 000007d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 000007e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> -000007f0: 0000 0000 0000 0000 0000 0000 264c 0251 ............&L.Q
> +000007f0: 0000 0000 0000 0000 0000 0000 64ca bba5 ............d...
> 00000800: 1200 0000 2200 0000 3200 0000 9d03 7300 ...."...2.....s.
> -00000810: 0200 0000 0000 0000 babb 8a41 7300 2004 ...........As. .
> +00000810: 0200 0400 0000 0000 babb 8a41 7300 69f5 ...........As.i.
> 00000820: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000830: 0000 0000 0000 0000 bc7a 6e31 0000 0000 .........zn1....
> 00000840: 0000 0000 0000 0000 0000 0000 0000 0000 ................
>
> Any idea what is going on? Is there a better way to diff two ext4 disk images
> than diffing the xxd output? If I try diffing the dumpe2fs output I get these
> differences:
>
> @@ -32,7 +32,7 @@
> Maximum mount count: -1
> Last checked: Fri May 3 16:14:49 2024
> Check interval: 0 (<none>)
> -Lifetime writes: 1164 kB
> +Lifetime writes: 73 kB
> Reserved blocks uid: 0 (user root)
> Reserved blocks gid: 0 (group root)
> First inode: 11
> @@ -44,7 +44,7 @@
> Directory Hash Seed: 0b7f9cfd-0113-486c-a453-4f5483bd486b
> Journal backup: inode blocks
> Checksum type: crc32c
> -Checksum: 0x51024c26
> +Checksum: 0xa5bbca64
> Checksum seed: 0xf81d767d
> Orphan file inode: 12
> Journal features: (none)
> @@ -56,7 +56,7 @@
> Journal start: 0
>
>
> -Group 0: (Blocks 1-2047) csum 0x0420
> +Group 0: (Blocks 1-2047) csum 0xf569 [ITABLE_ZEROED]
> Primary superblock at 1, Group descriptors at 2-2
> Reserved GDT blocks at 3-17
> Block bitmap at 18 (+17), csum 0x7abcbbba
>
> Why would these bits differ depending on the filesystem on which the disk image
> is stored? Is there a way to equalize this information so that the disk image
> looks the same independent on the underlying filesystem?
The diff becomes a bit smaller when using
-E lazy_itable_init=0,assume_storage_prezeroed=0,nodiscard
@@ -85,7 +85,7 @@
00000540: 0000 0000 0000 0000 0000 0000 0000 1000 ................
00000550: 0000 0000 0000 0000 0000 0000 2000 2000 ............ . .
00000560: 0200 0000 0000 0000 0000 0000 0000 0000 ................
-00000570: 0000 0000 0401 0000 ac04 0000 0000 0000 ................
+00000570: 0000 0000 0401 0000 4900 0000 0000 0000 ........I.......
00000580: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000590: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000005a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
@@ -125,7 +125,7 @@
000007c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000007d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000007e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
-000007f0: 0000 0000 0000 0000 0000 0000 fb8d a90f ................
+000007f0: 0000 0000 0000 0000 0000 0000 64ca bba5 ............d...
00000800: 1200 0000 2200 0000 3200 0000 9d03 7300 ...."...2.....s.
00000810: 0200 0400 0000 0000 babb 8a41 7300 69f5 ...........As.i.
00000820: 0000 0000 0000 0000 0000 0000 0000 0000 ................
@@ -32,7 +32,7 @@
Maximum mount count: -1
Last checked: Fri May 3 16:14:49 2024
Check interval: 0 (<none>)
-Lifetime writes: 1196 kB
+Lifetime writes: 73 kB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
@@ -44,7 +44,7 @@
Directory Hash Seed: 0b7f9cfd-0113-486c-a453-4f5483bd486b
Journal backup: inode blocks
Checksum type: crc32c
-Checksum: 0x0fa98dfb
+Checksum: 0xa5bbca64
Checksum seed: 0xf81d767d
Orphan file inode: 12
Journal features: (none)
The "Lifetime writes" being much higher on fat32 suggests that despite
"nodiscard", less zeroes were written out when ext4 or tmpfs are the underlying
FS?
Thanks!
cheers, josch
On Sat, May 04, 2024 at 07:53:29PM +0200, Johannes Schauer Marin Rodrigues wrote:
> >
> > Any idea what is going on?
The fundamental issue has to do with how ext2fs_zero_blocks() in
lib/ext2fs/mkjournal.c is implemented.
> The "Lifetime writes" being much higher on fat32 suggests that despite
> "nodiscard", less zeroes were written out when ext4 or tmpfs are the underlying
> FS?
Yes, that's exactly right.
The ext2fs_zero_blocks() function will attempt to call the io
channel's zeroout function --- for Unix systems, that's
lib/ext2fs/unix_io.c's __unix_zeroout() function. This will attempt
to use fallocate's FALLOC_FL_ZERO_RANGE or FALLOCATE_FL_PUNCH_HOLE to
zero a range of blocks. Now, exactly how ZERO_RANGE and PUNCH_HOLE is
implemented depends on whether the "storage device" being accessed via
unix_io is a block device or a file, and if it is a file, whether the
underlying file system supports ZERO_RANGE or PUNCH_HOLE.
Depending on how the underlying file system supports ZERO_RANGE and/or
PUNCH_HOLE, it may simply manipulate metadata blocks (e.g., ext4's
extent tree) so that the relevant file offsets will return zero --- or
if the file system doesn't support unitialized extent range, and/or
doesn't support sparse files, the file system MAY write all zeros, or
the file system MAY simply return an EOPNOTSUPP error, or the file
system MAY issue a SCSI WRITE SAME or moral equivalent for UFS, NVMe,
etc., if the block device supports it (and this might turn into a
SSD-level discard, so long as it is a reliable discard). And of
course, if unix_io is accessing a block device, depending on the
capabilities of the storage device and its connection bus, this might
also turn into a SCSI WRITE SAME, or some other zeroout command.
Now, the zeroout command doesn't actually increment the lifetime
writes counter. Whether or not it should is an interesting
philosophical question, since it might actually result in writes to
the device, or it might just simply involve metadata updates, either
on the underlying file (if the file system supports it), or
potentially in the metadata for the SSD's Flash Translation Layer. At
the userspace level, we simply don't know how FALLOC_FL_ZERO_RANGE and
FALLOC_FL_PUNCH_HOLE will be implemented.
In the case of FAT32, the file system doesn't support sparse files,
and it also doesn't support unitialized extents. So
FALLOC_FL_ZERO_RANGE and FALLOC_FL_PUNCH_HOLE will fail on a fat32
file system. As a result, ext2fs_zero_blocks() will fall back to
explicitly writing zeros using io_channel_write_blk64(), and this
*does* increment the lifetime writes counter.
If you enhance the script by adding "ls -ls "$imgpath" and "filefrag
-v "$imgpath" || /bin/true", you can see that the disk space consumed
by the image file varies, and it varies even more if you use the
original version of the script that doesn't disable lazy_itable_init,
discard, et.al.
Unfortunately tmpfs and fat don't support filefrag -v, but you could
see the difference if you write a debugging program which used lseek's
SEEK_HOLE and SEEK_DATA to see which parts of the file are sparse
(although it won't show which parts of the file are marked
unitialized, assuming the file system supported it).
If your goal is to create completely reproducible image files, one
question is whether keeping the checksums identical is enough, or do
you care about whether the underlying file is being more efficiently
stored by using sparse files or extents marked unitialized?
Depending on how much you care about reproducibility versus file
storage efficiency, I could imagine adding some kind of option which
disables the zeroout function, and forces e2fsprogs to always write
zeros, even if that increases the write wearout rate of the underlying
flash file system, and increasing the size of the image file. Or I
could imageine some kind of extended option which hacks mke2fs to zero
out the lifetime writes counter.;
Cheers,
- Ted