Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 7 Apr 2020 15:05:59 +0200
From:   Corentin Labbe <clabbe.montjoie@gmail.com>
To:     Russell King - ARM Linux admin <linux@armlinux.org.uk>
Cc:     kexec@lists.infradead.org, linux-kernel@vger.kernel.org,
        mripard@kernel.org, wens@csie.org, ebiederm@xmission.com,
        linux-arm-kernel@lists.infradead.org
Subject: Re: Trying to kexec on Allwinner A80
Message-ID: <20200407130559.GB457@Red>
References: <20200406082720.GA31279@Red>
 <20200407100203.GK25745@shell.armlinux.org.uk>
 <20200407101912.GL25745@shell.armlinux.org.uk>
 <20200407113454.GA457@Red>
 <20200407122243.GN25745@shell.armlinux.org.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20200407122243.GN25745@shell.armlinux.org.uk>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Apr 07, 2020 at 01:22:43PM +0100, Russell King - ARM Linux admin wrote:
> On Tue, Apr 07, 2020 at 01:34:54PM +0200, Corentin Labbe wrote:
> > On Tue, Apr 07, 2020 at 11:19:12AM +0100, Russell King - ARM Linux admin wrote:
> > > On Tue, Apr 07, 2020 at 11:02:03AM +0100, Russell King - ARM Linux admin wrote:
> > > > On Mon, Apr 06, 2020 at 10:27:20AM +0200, Corentin Labbe wrote:
> > > > > Hello
> > > > > 
> > > > > I am trying to add the last missing Allwinner Soc in kernelci: the A80.
> > > > > But this SoC does not have any way to be used in kernelci, no USB nor network in uboot.
> > > > > So I have tried to fake uboot since the kernel has network support and run the new kernel via kexec.
> > > > > 
> > > > > But kexec 2.0.18 fail to work:
> > > > > kexec --force /tmp/kernel --initrd /tmp/ramdisk --dtb /tmp/dtb --command-line='console=ttyS0,115200n8 root=/dev/ram0 earlycon=uart,mmio32,0x7000000 ip=dhcp'
> > > > 
> > > > As I mentioned in my other reply, this apparent "kexec" command line
> > > > does not seem to reflect the arguments you actually used to invoke
> > > > the kexec output below.
> > > > 
> > > > > syscall kexec_file_load not available.
> > > > 
> > > > This message is produced by do_kexec_file_load().  This function is only
> > > > invoked if the do_kexec_file_syscall flag in main() is set.  This flag
> > > > is only set if one of:
> > > > 
> > > > 	--kexec-file-syscall
> > > > 	--kexec-syscall
> > > > 	--kexec-syscall-auto
> > > > 	-s
> > > > 	-c
> > > > 	-a
> > > > 
> > > > are provided on the kexec command line.  Your command line above does
> > > > not contain any of those arguments, so either the command line is not
> > > > what you used, or you are using a patched kexec, or your compiler is
> > > > grossly miscompiling kexec.
> > > > 
> > > > > Try gzip decompression.
> > > > > kernel: 0xb6535008 kernel_size: 0x853200
> > > > > MEMORY RANGES
> > > > > 0000000020000000-000000009fffffff (0)
> > > > 
> > > > Then there's the debug output, which is only produced if the
> > > > kexec_debug global is set, which in turn is only set if --debug or -d
> > > > is supplied on the kexec command line - which again, your kexec
> > > > command line does not contain this.
> > > > 
> > > > > zImage header: 0x016f2818 0x00000000 0x00853200
> > > > > zImage size 0x853200, file size 0x853200
> > > > > zImage requires 0x00864200 bytes
> > > > >   offset 0x0000bae4 tag 0x5a534c4b size 8
> > > > > Decompressed kernel sizes:
> > > > >  text+data 0x0158b3a0 bss 0x000632f0 total 0x015ee690
> > > > > Resulting kernel space: 0x01def5a0
> > > > > Kernel: address=0x20008000 size=0x01def5a0
> > > > > Initrd: address=0x21df8000 size=0x0090b6fa
> > > > > DT    : address=0x22704000 size=0x00005c09
> > > > > kexec_load: entry = 0x20008000 flags = 0x280000
> > > > > nr_segments = 3
> > > > > segment[0].buf   = 0xb6535008
> > > > > segment[0].bufsz = 0x853204
> > > > > segment[0].mem   = 0x20008000
> > > > > segment[0].memsz = 0x854000
> > > > > segment[1].buf   = 0xb5c29008
> > > > > segment[1].bufsz = 0x90b6fa
> > > > > segment[1].mem   = 0x21df8000
> > > > > segment[1].memsz = 0x90c000
> > > > > segment[2].buf   = 0x4db50
> > > > > segment[2].bufsz = 0x5c09
> > > > > segm[   71.039126] kexec_core: Starting new kernel
> > > > > ent[2].mem   = 0[   71.044712] Disabling non-boot CPUs ...
> > > > > x22704000
> > > > > segment[2].memsz = 0x6000
> > > > > [   71.489070] Bye!
> > > > > 
> > > > > 
> > > > > I have tried also kexec-2.0.20
> > > > > Try gzip decompression.
> > > > > zImage header: 0x00000000 0x000019b4 0x00001000
> > > > > zImage requires 0x008641c0 bytes
> > > > > Could not find a free area of memory of 0x86c1c0 bytes...
> > > > > Cannot load /tmp/kernel
> > > > 
> > > > kexec 2.0.20 doesn't appear to have changed anything to do with how
> > > > allocations are done.  The above output looks even more strange and
> > > > confusing.  "zImage header" is produced by debugging prints, which
> > > > imply that kexec_debug was set.
> > > > 
> > > > However, the "MEMORY RANGES" output is missing - this has not gone
> > > > away with kexec 2.0.20, it's still there, and works for me (I've
> > > > just built and tested kexec 2.0.20).
> > > > 
> > > > Also, the values on the "zImage header" line are completely messed
> > > > up; the first should be the zImage magic value and it is not - that
> > > > suggests that the file being loaded is not a zImage file, or is
> > > > corrupted.
> > > 
> > > Under a VM (the kernel doesn't have kexec support - but that's not a
> > > problem, because the initial loading stages are the relevant parts
> > > which all happen in userspace):
> > > 
> > > # build/sbin/kexec --version
> > > kexec-tools 2.0.20
> > > # build/sbin/kexec --debug --load virt-vmlinuz-5.4.0+
> > > Try gzip decompression.
> > > kernel: 0xb6a6c008 kernel_size: 0x407358
> > > MEMORY RANGES
> > > 0000000040000000-000000007fffffff (0)
> > > zImage header: 0x016f2818 0x00000000 0x00407358
> > > zImage size 0x407358, file size 0x407358
> > > zImage requires 0x00418358 bytes
> > >   offset 0x00007178 tag 0x5a534c4b size 12
> > > Decompressed kernel sizes:
> > >  text+data 0x00c2ed24 bss 0x000319ec total 0x00c60710
> > > Resulting kernel space: 0x0104707c
> > > Kernel: address=0x40008000 size=0x0104707c
> > > DT    : address=0x41051000 size=0x00100000
> > > kexec_load: entry = 0x40008000 flags = 0x280000
> > > nr_segments = 2
> > > segment[0].buf   = 0xb6a6c008
> > > segment[0].bufsz = 0x40735c
> > > segment[0].mem   = 0x40008000
> > > segment[0].memsz = 0x408000
> > > segment[1].buf   = 0xb696b008
> > > segment[1].bufsz = 0x100000
> > > segment[1].mem   = 0x41051000
> > > segment[1].memsz = 0x100000
> > > kexec_load failed: Function not implemented
> > > entry       = 0x40008000 flags = 0x280000
> > > nr_segments = 2
> > > segment[0].buf   = 0xb6aa0008
> > > segment[0].bufsz = 0x40735c
> > > segment[0].mem   = 0x40008000
> > > segment[0].memsz = 0x408000
> > > segment[1].buf   = 0xb699f008
> > > segment[1].bufsz = 0x100000
> > > segment[1].mem   = 0x41051000
> > > segment[1].memsz = 0x100000
> > > #
> > > 
> > > On a SolidRun Hummingboard2 (iMX6 based, which has kexec support, same
> > > kexec binary, first without an initrd, and then with):
> > > 
> > > # build/sbin/kexec --version
> > > kexec-tools 2.0.20
> > > # build/sbin/kexec --debug --load multi-vmlinuz-5.6.0+
> > > Try gzip decompression.
> > > kernel: 0xb6763008 kernel_size: 0x7273a8
> > > MEMORY RANGES
> > > 0000000010000000-000000004fffffff (0)
> > > zImage header: 0x016f2818 0x00000000 0x007273a8
> > > zImage size 0x7273a8, file size 0x7273a8
> > > zImage requires 0x007383a8 bytes
> > >   offset 0x00004da8 tag 0x5a534c4b size 12
> > > Decompressed kernel sizes:
> > >  text+data 0x00dbedb8 bss 0x007489f8 total 0x015077b0
> > > Resulting kernel space: 0x015077b0
> > > Kernel: address=0x10008000 size=0x015077b0
> > > DT    : address=0x11511000 size=0x0000b000
> > > kexec_load: entry = 0x10008000 flags = 0x280000
> > > nr_segments = 2
> > > segment[0].buf   = 0xb6763008
> > > segment[0].bufsz = 0x7273ac
> > > segment[0].mem   = 0x10008000
> > > segment[0].memsz = 0x728000
> > > segment[1].buf   = 0x1207cb0
> > > segment[1].bufsz = 0xb000
> > > segment[1].mem   = 0x11511000
> > > segment[1].memsz = 0xb000
> > > # build/sbin/kexec --unload
> > > # build/sbin/kexec --debug --load multi-vmlinuz-5.6.0+ --initrd /boot/initrd.img-5.4.0+
> > > Try gzip decompression.
> > > kernel: 0xb65d8008 kernel_size: 0x7273a8
> > > MEMORY RANGES
> > > 0000000010000000-000000004fffffff (0)
> > > zImage header: 0x016f2818 0x00000000 0x007273a8
> > > zImage size 0x7273a8, file size 0x7273a8
> > > zImage requires 0x007383a8 bytes
> > >   offset 0x00004da8 tag 0x5a534c4b size 12
> > > Decompressed kernel sizes:
> > >  text+data 0x00dbedb8 bss 0x007489f8 total 0x015077b0
> > > Resulting kernel space: 0x015077b0
> > > Kernel: address=0x10008000 size=0x015077b0
> > > Initrd: address=0x11510000 size=0x0053f46a
> > > DT    : address=0x11a50000 size=0x0000b044
> > > kexec_load: entry = 0x10008000 flags = 0x280000
> > > nr_segments = 3
> > > segment[0].buf   = 0xb65d8008
> > > segment[0].bufsz = 0x7273ac
> > > segment[0].mem   = 0x10008000
> > > segment[0].memsz = 0x728000
> > > segment[1].buf   = 0xb6098008
> > > segment[1].bufsz = 0x53f46a
> > > segment[1].mem   = 0x11510000
> > > segment[1].memsz = 0x540000
> > > segment[2].buf   = 0x993cf0
> > > segment[2].bufsz = 0xb044
> > > segment[2].mem   = 0x11a50000
> > > segment[2].memsz = 0xc000
> > > 
> > > On clearfog (Armada 388):
> > > 
> > > # build/sbin/kexec --debug --load multi-vmlinuz-5.6.0+
> > > Try gzip decompression.
> > > kernel: 0xb6745008 kernel_size: 0x7273a8
> > > MEMORY RANGES
> > > 0000000000000000-000000003fffffff (0)
> > > zImage header: 0x016f2818 0x00000000 0x007273a8
> > > zImage size 0x7273a8, file size 0x7273a8
> > > zImage requires 0x007383a8 bytes
> > >   offset 0x00004da8 tag 0x5a534c4b size 12
> > > Decompressed kernel sizes:
> > >  text+data 0x00dbedb8 bss 0x007489f8 total 0x015077b0
> > > Resulting kernel space: 0x015077b0
> > > Kernel: address=0x00008000 size=0x015077b0
> > > DT    : address=0x01511000 size=0x00007be3
> > > kexec_load: entry = 0x8000 flags = 0x280000
> > > nr_segments = 2
> > > segment[0].buf   = 0xb6745008
> > > segment[0].bufsz = 0x7273ac
> > > segment[0].mem   = 0x8000
> > > segment[0].memsz = 0x728000
> > > segment[1].buf   = 0x1be7f68
> > > segment[1].bufsz = 0x7be3
> > > segment[1].mem   = 0x1511000
> > > segment[1].memsz = 0x8000
> > > 
> > > All appears to work fine.
> > > 
> > 
> > I have found a part of my problem, kexec-tool seems to always use the OPT_KEXEC_FILE_SYSCALL.
> > Even if I set --kexec-syscall.
> 
> You mention below that you're using buildroot.  Does buildroot have
> patches on top of the reference kexec-tools codebase as found at
> git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git ?
> 
> Looking at the 2.0.20 code, there's no way for that to be happening;
> the only way that _could_ happen is if getopt_long() always returns
> 's' when we get to the last argument, which is highly unlikely.
> 
> > On my early tries I got "syscall kexec_file_load not available."
> > When I did a full rebuild of my buildroot to go back to kexec-tool 2.0.18, this syscall become availlable.
> 
> 32-bit kernels have no support for kexec_file_load(), although the
> syscall is reserved.
> 
> Irrespective of that, presence of the syscall number allocation is
> not sufficient to tell whether the syscall is implemented by the
> kernel; that is controlled by the kernel's CONFIG_KEXEC_FILE symbol.
> This symbol is never offered for 32-bit ARM kernels.
> 
> > The get_memory_ranges seems to be called only from my_load() which is executed when !do_kexec_file_syscall.
> > 
> > Since kexec-tool always set do_kexec_file_syscall, it is never called.
> > I have added a print for each syscal option, and it seems that OPT_KEXEC_SYSCALL_AUTO is called after the handling of OPT_KEXEC_SYSCALL.
> > 
> > So I have hack to always set do_kexec_file_syscall=0
> 
> It sounds like buildroot is patching kexec-tools in ways that break
> 32-bit ARM, and it sounds like this needs to be reported as a bug to
> buildroot.
> 
> That said, I think there's a weakness in the ARM support in
> kexec-tools, which this patch should address for 2.0.20.  Things have
> changed in the merged patches since 2.0.20, so the return needs to be
> EFALLBACK not ENOSYS, so please pay attention to exactly which version
> of kexec-tools you have (whether it's the 2.0.20 tagged version or has
> further patches from kexec-tools on top.)
> 
> diff --git a/kexec/arch/arm/kexec-zImage-arm.c b/kexec/arch/arm/kexec-zImage-arm.c
> index 9a67b8a4db98..ca4e35382ea1 100644
> --- a/kexec/arch/arm/kexec-zImage-arm.c
> +++ b/kexec/arch/arm/kexec-zImage-arm.c
> @@ -421,6 +421,12 @@ int zImage_arm_load(int argc, char **argv, const char *buf, off_t len,
>  	};
>  	static const char short_options[] = KEXEC_ARCH_OPT_STR "a:r:";
>  
> +	/* We do not support file mode */
> +	if (info->file_mode) {
> +		fprintf(stderr, "syscall kexec_file_load not implemented.\n");
> +		return -ENOSYS;
> +	}
> +
>  	/*
>  	 * Parse the command line arguments
>  	 */
> 
> > So now my test go further, but the final kernel crash.
> > DEBUG: bootz: run kexec with /tmp/kernel --debug --kexec-syscall --force --initrd /tmp/ramdisk --dtb /tmp/dtb --command-line='console=ttyS0,115200n8 root=/dev/ram0 earlycon=uart,mmio32,0x7000000 ip=dhcp'
> > Set DEBUG!
> > main:1417 OPT_KEXEC_SYSCALL
> > main:1422 OPT_KEXEC_SYSCALL_AUTO
> > arch_process_options:119
> > main:1500
> > main:1517 res=0 do_load=1
> > main:1519 res=0 do_kexec_file_syscall=0
> > my_load:713
> > Try gzip decompression.
> > kernel: 0xb693b008 kernel_size: 0x443ac0
> > get_memory_ranges:36
> > MEMORY RANGES
> > 0000000020000000-000000009fffffff (0)
> > DEBUG: my_load:737 memory_ranges=1
> > zImage_arm_load:423
> > zImage header: 0x016f2818 0x00000000 0x00443ac0
> > zImage size 0x443ac0, file size 0x443ac0
> > zImage requires 0x00454ac0 bytes
> >   offset 0x00006678 tag 0x5a534c4b size 8
> > Decompressed kernel sizes:
> >  text+data 0x00b78080 bss 0x0003d3c0 total 0x00bb5440
> > Resulting kernel space: 0x00fccb40
> > DEBUG: locate_hole:237 memory_ranges=1
> > Check 0/1 536870912 0 type=-1610612737
> > Check 0/1 536870912 0
> > Kernel: address=0x20008000 size=0x00fccb40
> > DEBUG: locate_hole:237 memory_ranges=1
> > Check 0/1 536870912 0 type=-1610612737
> > Check 0/1 536870912 0
> > Initrd: address=0x20fd5000 size=0x0167213d
> > DEBUG: locate_hole:237 memory_ranges=1
> > Check 0/1 536870912 0 type=-1610612737
> > Check 0/2 536870912 0
> > Check 1/2 577011712 0
> > DT    : address=0x22648000 size=0x00006043
> > kexec_load: entry = 0x20008000 flags = 0x280000
> > nr_segments = 3
> > segment[0].buf   = 0xb693b008
> > segment[0].bufsz = 0x443ac4
> > segment[0].mem   = 0x20008000
> > segment[0].memsz = 0x444000
> > segment[1].buf   = 0xb52c8008
> > segment[1].bufsz = 0x167213d
> > segment[1].mem   = 0x20fd5000
> > segment[1].memsz = 0x1673000
> > segment[2].buf   = 0x4ef88
> > segment[2].bufsz = 0x6043
> > segment[2].mem   = 0x2[   31.265096] sun7i-dwmac 830000.ethernet eth0: Link is Down
> > 2648000
> > segment[2].memsz = 0x7000
> > main:1568 res=0
> > main:1582 res=0 do_exec=1
> > [   31.277297] kexec_core: Starting new kernel
> > [   31.282700] Disabling non-boot CPUs ...
> > [   31.692085] Bye!
> > C:0x200080C0-0x2044BAC0->0x20B80B00-0x20FC4500
> 
> This line shows that we entered the new kernel, which was located
> (correctly) at 0x200080C0-0x2044BAC0, and it has decided to relocate
> itself to 0x20B80B00-0x20FC4500.
> 
> The lack of further output suggests that the decompressor itself
> wasn't able to run after being relocated higher in memory.
> 
> It may be worth booting the same kernel image and see what the C:
> line comes out as there.
> 

I have checked and some CONFIG was missing, now I get one line further:
[   32.903765] kexec_core: Starting new kernel
[   32.907958] Disabling non-boot CPUs ...
[   33.392056] Bye!
C:0x200080C0-0x2044AE80->0x20B80300-0x20FC30C0
Uncompressing Linux... done, booting the kernel.

So the uncompressor seems ok.

As you said, I have started the same kernel, but I dont see any C:
Starting kernel ...
Uncompressing Linux... done, booting the kernel.
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.6.0-10890-gc27180b13770-dirty (compile@Red) (gcc version 8.3.0 (Gentoo 8.3.0-r1 p1.1)) #78 SMP Tue Apr 7 14:55:53 CEST 2020
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code