Received-SPF: pass (google.com: best guess record for domain of linux-ext4-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: vmlinux ELF header sometimes corrupt
From:   Rasmus Villemoes <linux@rasmusvillemoes.dk>
Cc:     linux-ext4@vger.kernel.org
References: <71aa76d0-a3b8-b4f3-a7c3-766cfb75412f@rasmusvillemoes.dk>
 <5997e9b6-95fd-405b-05f8-16f9e34d9d87@rasmusvillemoes.dk>
Message-ID: <70be00f8-1559-b74d-c9cf-2e5423f625d4@rasmusvillemoes.dk>
Date:   Thu, 27 Feb 2020 16:57:45 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.4.1
MIME-Version: 1.0
In-Reply-To: <5997e9b6-95fd-405b-05f8-16f9e34d9d87@rasmusvillemoes.dk>
Content-Type: text/plain; charset=windows-1252
Content-Language: en-US
Content-Transfer-Encoding: 7bit
To:     unlisted-recipients:; (no To-header on input)
Sender: linux-ext4-owner@vger.kernel.org
Precedence: bulk

On 28/01/2020 09.12, Rasmus Villemoes wrote:

> I've done a long overdue kernel update, and there are quite a few
> fs/ext4/ -stable patches in there, so now I'll see if it still happens.

OK, I still see the vmlinux corruption.

To recap: My laptop uses a Ubuntu 4.15 kernel. I'm building a ppc32 kernel

export ARCH=powerpc
export CROSS_COMPILE=powerpc-linux-gnu-
make 83xx/mpc8315_rdb_defconfig
make -j8

Then "file vmlinux" says all is good: "ELF 32-bit MSB executable,
PowerPC or ..."

Time passes, I/O in other parts of the system etc., now suddenly "file
vmlinux" says "data". Inspecting the file shows that the first 52 bytes
(aka exactly the ELF header) are 0. Instrumenting the kernel build [1]
to save some intermediate stages tells me that the vmlinux file is fine
beyond 52 bytes. I.e.,

$ cmp -i 52 vmlinux vmlinux.after_sort

is fine. Now comparing stat(1) on vmlinux immediately after sortextable
to stat(1) on the corrupted vmlinux shows something a bit odd in the
Blocks: field, but ino, mtime are unchanged (as one would expect when
nothing should have touched that file).

$ cat vmlinux.stat.after_sort
  File: vmlinux
  Size: 8055408         Blocks: 15624      IO Block: 4096   regular file
Device: 811h/2065d      Inode: 85736362    Links: 1
Access: (0775/-rwxrwxr-x)  Uid: ( 1000/    ravi)   Gid: ( 1000/    ravi)
Access: 2020-02-27 13:15:10.989950361 +0100
Modify: 2020-02-27 13:15:11.033950470 +0100
Change: 2020-02-27 13:15:11.033950470 +0100
 Birth: -

$ stat vmlinux
  File: vmlinux
  Size: 8055408         Blocks: 15616      IO Block: 4096   regular file
Device: 811h/2065d      Inode: 85736362    Links: 1
Access: (0775/-rwxrwxr-x)  Uid: ( 1000/    ravi)   Gid: ( 1000/    ravi)
Access: 2020-02-27 15:46:55.354409876 +0100
Modify: 2020-02-27 13:15:11.033950470 +0100
Change: 2020-02-27 13:15:11.033950470 +0100
 Birth: -

Oddly enough, I never seem to have problems with any other files, so I'm
thinking there's some particular I/O pattern involved when ld creates
the vmlinux file. It does

lseek(6, 0, SEEK_SET)                   = 0
read(6, "", 52)                         = 0
lseek(6, 52, SEEK_CUR)                  = 52

initially, and then much later actually writes the header

lseek(6, 0, SEEK_SET)                   = 0
write(6,
"\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\2\0\24\0\0\0\1\300\0\0\0\0\0\0004"...,
52) = 52

Then sortextable does some poking around, but that's just via mmap(), so
not much to see in strace.

I also don't seem to be able to reproduce it when I build an x86_64
kernel instead of ppc32 - perhaps the bigger ELF header matters? I'll
try some other 32 bit platforms.

I have tried creating an ext4 image in a 50 GB file with the exact same
file system options [2] and did a "make O=..." build to see if I could
make an image file containing the corrupted state, but so far no luck.
fsck on /dev/sdb1 says my filesystem is just fine.

This is starting to drive me a bit crazy, so I really hope someone goes
"yeah, you should run at least v5.2, that has commit abc123def which
fixes that".

Thanks,
Rasmus

[1] I've applied this on top of v4.19.82:
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index c8cf45362bd6..6a6d8ac79607 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -94,8 +94,13 @@ vmlinux_link()
                        ${KBUILD_VMLINUX_LIBS}                  \
                        --end-group                             \
                        ${1}"
+                if [ "${2}" = vmlinux ] ; then
+                    strace="strace -o ${objtree}/vmlinux.strace"
+                else
+                    strace=""
+                fi

-               ${LD} ${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux} -o ${2}      \
+               $strace ${LD} ${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux} -o
${2}      \
                        -T ${lds} ${objects}
        else
                objects="-Wl,--whole-archive                    \
@@ -151,7 +156,7 @@ mksysmap()

 sortextable()
 {
-       ${objtree}/scripts/sortextable ${1}
+       strace -o ${objtree}/sortextable.strace
${objtree}/scripts/sortextable ${1}
 }

 # Delete output files in case of error
@@ -283,7 +288,13 @@ vmlinux_link "${kallsymso}" vmlinux

 if [ -n "${CONFIG_BUILDTIME_EXTABLE_SORT}" ]; then
        info SORTEX vmlinux
+       stat vmlinux > vmlinux.stat.before_sort
+       cp --preserve=all vmlinux vmlinux.before_sort
+       file vmlinux >&2
        sortextable vmlinux
+       stat vmlinux > vmlinux.stat.after_sort
+       cp --preserve=all vmlinux vmlinux.after_sort
+       file vmlinux >&2
 fi

 info SYSMAP System.map

[2] $ sudo dumpe2fs -h /dev/sdb1
[sudo] password for ravi:
dumpe2fs 1.44.1 (24-Mar-2018)
Filesystem volume name:   <none>
Last mounted on:          /mnt/ext4
Filesystem UUID:          2b70b7ba-fdd7-4616-8431-2b04ea31d803
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr dir_index filetype
needs_recovery extent 64bit flex_bg metadata_csum_seed inline_data
sparse_super large_file extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              117440512
Block count:              469762048
Reserved block count:     23488102
Free blocks:              223934526
Free inodes:              88111686
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   1024
Flex block group size:    16
Filesystem created:       Thu Jun 13 08:04:11 2019
Last mount time:          Wed Jan 29 21:01:18 2020
Last write time:          Wed Jan 29 21:01:18 2020
Mount count:              1
Maximum mount count:      -1
Last checked:             Wed Jan 29 20:49:04 2020
Check interval:           0 (<none>)
Lifetime writes:          1563 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               512
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
First orphan inode:       87165617
Default directory hash:   half_md4
Directory Hash Seed:      d1ac8c45-70f3-4e6e-9710-b7e434bb59ba
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x950a844d
Checksum seed:            0x143063d3
Journal features:         journal_incompat_revoke journal_64bit
journal_checksum_v3
Journal size:             1024M
Journal length:           262144
Journal sequence:         0x0003218a
Journal start:            219662
Journal checksum type:    crc32c
Journal checksum:         0xea20aea2