2018-07-01 21:33:49

by Benjamin Gilbert

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> config for reference, and am happy to test patches, provide sample QCOW
> images, etc.

Adding linux-x86_64, LKML.

--Benjamin Gilbert


Attachments:
(No filename) (591.00 B)
config.gz (29.10 kB)
Download all attachments

2018-07-02 09:51:33

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Sun, Jul 01, 2018 at 09:32:43PM +0000, Benjamin Gilbert wrote:
> On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> > up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> > QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> > Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> > paging boot if kernel is above 4G") fixes it. I've attached our kernel
> > config for reference, and am happy to test patches, provide sample QCOW
> > images, etc.
>
> Adding linux-x86_64, LKML.

Thank you for the report.

I'm not able to trigger it in my environment. Tried v4.17.3 and current
Linus' tree.

I had to change config slightly to get kernel build in my setup. Diff is
below.

I run KVM this way:

qemu-system-x86_64 -kernel ./arch/x86/boot/bzImage -nographic -append "console=ttyS0"

Could you check if you can trigger the issue with my changes to config and
the way I run KVM?

If not, what the key difference in our setups that make the issue visible?

How do you run KVM? Do you have EFI enable? Config difference?
Running kernel directly, without GRUB makes a difference?

--- config 2018-07-02 11:59:23.662685761 +0300
+++ .config 2018-07-02 12:09:51.822677398 +0300
@@ -1,6 +1,6 @@
#
# Automatically generated file; DO NOT EDIT.
-# Linux/x86 4.17.3-coreos Kernel Configuration
+# Linux/x86 4.17.3 Kernel Configuration
#
CONFIG_64BIT=y
CONFIG_X86_64=y
@@ -187,16 +187,13 @@
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="bootengine.cpio"
-CONFIG_INITRAMFS_ROOT_UID=0
-CONFIG_INITRAMFS_ROOT_GID=0
+CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
-CONFIG_INITRAMFS_COMPRESSION=".gz"
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
@@ -1564,8 +1561,7 @@
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
-CONFIG_EXTRA_FIRMWARE="intel-ucode/0f-04-04 intel-ucode/06-55-03 intel-ucode/06-1d-01 intel-ucode/06-9e-09 intel-ucode/06-05-03 intel-ucode/0f-02-09 intel-ucode/06-56-02 intel-ucode/0f-04-01 intel-ucode/06-03-02 intel-ucode/06-3e-06 intel-ucode/06-0e-0c intel-ucode/0f-04-0a intel-ucode/06-25-05 intel-ucode/06-56-03 intel-ucode/06-0f-0a intel-ucode/06-0f-0d intel-ucode/0f-03-04 intel-ucode/06-2d-07 intel-ucode/06-3d-04 intel-ucode/06-0f-02 intel-ucode/06-05-00 intel-ucode/0f-04-03 intel-ucode/06-47-01 intel-ucode/.keep_sys-firmware_intel-microcode-0 intel-ucode/06-07-02 intel-ucode/06-0a-00 intel-ucode/06-9e-0a intel-ucode/06-56-05 intel-ucode/06-1c-0a intel-ucode/06-05-02 intel-ucode/06-2d-06 intel-ucode/06-08-01 intel-ucode/06-06-0d intel-ucode/06-46-01 intel-ucode/06-0f-06 intel-ucode/06-0b-01 intel-ucode/06-45-01 intel-ucode/0f-04-09 intel-ucode/0f-06-02 intel-ucode/0f-04-07 intel-ucode/06-16-01 intel-ucode/06-5e-03 intel-ucode/06-06-05 intel-ucode/06-08-06 intel-ucode/0f-00-07 intel-ucode/06-07-01 intel-ucode/06-3e-07 intel-ucode/06-1c-02 intel-ucode/06-08-0a intel-ucode/0f-03-02 intel-ucode/06-07-03 intel-ucode/0f-02-07 intel-ucode/06-56-04 intel-ucode/06-2a-07 intel-ucode/06-17-06 intel-ucode/06-3e-04 intel-ucode/0f-02-05 intel-ucode/06-06-0a intel-ucode/06-3f-04 intel-ucode/06-25-02 intel-ucode/0f-01-02 intel-ucode/06-0b-04 intel-ucode/0f-04-08 intel-ucode/06-09-05 intel-ucode/06-2f-02 intel-ucode/06-4f-01 intel-ucode/06-3f-02 intel-ucode/06-3c-03 intel-ucode/06-7a-01 intel-ucode/0f-02-06 intel-ucode/06-1a-05 intel-ucode/06-26-01 intel-ucode/06-0f-07 intel-ucode/06-0d-06 intel-ucode/0f-03-03 intel-ucode/06-9e-0b intel-ucode/06-05-01 intel-ucode/06-0a-01 intel-ucode/06-4e-03 intel-ucode/06-8e-09 intel-ucode/06-1a-04 intel-ucode/06-5c-09 intel-ucode/0f-00-0a intel-ucode/0f-06-05 intel-ucode/0f-06-08 intel-ucode/06-08-03 intel-ucode/06-55-04 intel-ucode/06-0f-0b intel-ucode/06-1e-05 intel-ucode/06-0e-08 intel-ucode/06-3a-09 intel-ucode/06-17-07 intel-ucode/06-06-00 intel-ucode/06-8e-0a intel-ucode/0f-02-04 intel-ucode/06-17-0a intel-ucode/0f-06-04 amd-ucode/microcode_amd.bin amd-ucode/microcode_amd_fam17h.bin amd-ucode/microcode_amd_fam16h.bin amd-ucode/microcode_amd_fam15h.bin "
-CONFIG_EXTRA_FIRMWARE_DIR="/build/amd64-usr/lib/firmware"
+CONFIG_EXTRA_FIRMWARE=""
CONFIG_FW_LOADER_USER_HELPER=y
# CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set
CONFIG_ALLOW_DEV_COREDUMP=y
@@ -4742,7 +4738,7 @@
#
# Certificates for signature checking
#
-CONFIG_MODULE_SIG_KEY="certs/modules.pem"
+CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"
CONFIG_SYSTEM_TRUSTED_KEYRING=y
CONFIG_SYSTEM_TRUSTED_KEYS=""
# CONFIG_SYSTEM_EXTRA_CERTIFICATE is not set
--
Kirill A. Shutemov

2018-07-02 19:02:39

by Benjamin Gilbert

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Mon, Jul 02, 2018 at 12:34:50PM +0300, Kirill A. Shutemov wrote:
> Could you check if you can trigger the issue with my changes to config and
> the way I run KVM?

Yes, the issue still triggers in that case. I've also verified that the
kernel boots normally with your qemu command if the commit is reverted.

--Benjamin Gilbert

2018-07-03 08:31:51

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Mon, Jul 02, 2018 at 07:01:28PM +0000, Benjamin Gilbert wrote:
> On Mon, Jul 02, 2018 at 12:34:50PM +0300, Kirill A. Shutemov wrote:
> > Could you check if you can trigger the issue with my changes to config and
> > the way I run KVM?
>
> Yes, the issue still triggers in that case. I've also verified that the
> kernel boots normally with your qemu command if the commit is reverted.

Hm. What toolchain do you use?

--
Kirill A. Shutemov

2018-07-03 09:00:55

by Thomas Gleixner

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, 3 Jul 2018, Kirill A. Shutemov wrote:

> On Mon, Jul 02, 2018 at 07:01:28PM +0000, Benjamin Gilbert wrote:
> > On Mon, Jul 02, 2018 at 12:34:50PM +0300, Kirill A. Shutemov wrote:
> > > Could you check if you can trigger the issue with my changes to config and
> > > the way I run KVM?
> >
> > Yes, the issue still triggers in that case. I've also verified that the
> > kernel boots normally with your qemu command if the commit is reverted.
>
> Hm. What toolchain do you use?

On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,

2018-07-03 11:03:10

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, Jul 03, 2018 at 10:59:48AM +0200, Thomas Gleixner wrote:
> On Tue, 3 Jul 2018, Kirill A. Shutemov wrote:
>
> > On Mon, Jul 02, 2018 at 07:01:28PM +0000, Benjamin Gilbert wrote:
> > > On Mon, Jul 02, 2018 at 12:34:50PM +0300, Kirill A. Shutemov wrote:
> > > > Could you check if you can trigger the issue with my changes to config and
> > > > the way I run KVM?
> > >
> > > Yes, the issue still triggers in that case. I've also verified that the
> > > kernel boots normally with your qemu command if the commit is reverted.
> >
> > Hm. What toolchain do you use?
>
> On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,

I've built the kernel with toolchain from CoreOS alpha (gcc-7.3.0,
binutils-2.29.1). Still cannot trigger the problem.

Benjamin, could you share the kernel image?

--
Kirill A. Shutemov

2018-07-03 11:26:26

by Gabriel C

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
>> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
>> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
>> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
>> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
>> paging boot if kernel is above 4G") fixes it. I've attached our kernel
>> config for reference, and am happy to test patches, provide sample QCOW
>> images, etc.
>

Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,

0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
too with the same symptoms

BR

2018-07-03 12:45:14

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
> 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> >> config for reference, and am happy to test patches, provide sample QCOW
> >> images, etc.
> >
>
> Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
>
> 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> too with the same symptoms

I tracked it down to -flto in LDFLAGS. I'll look more into this.

--
Kirill A. Shutemov

2018-07-03 14:03:41

by Thomas Gleixner

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, 3 Jul 2018, Kirill A. Shutemov wrote:
> On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
> > 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> > > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> > >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> > >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> > >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> > >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> > >> config for reference, and am happy to test patches, provide sample QCOW
> > >> images, etc.
> > >
> >
> > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
> >
> > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> > too with the same symptoms
>
> I tracked it down to -flto in LDFLAGS. I'll look more into this.

And what sets -flto in LDFLAGS? I can't find anything in the kernel
build/Makefiles.

Thanks,

tglx

2018-07-03 14:23:29

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, Jul 03, 2018 at 03:44:03PM +0300, Kirill A. Shutemov wrote:
> On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
> > 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> > > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> > >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> > >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> > >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> > >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> > >> config for reference, and am happy to test patches, provide sample QCOW
> > >> images, etc.
> > >
> >
> > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
> >
> > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> > too with the same symptoms
>
> I tracked it down to -flto in LDFLAGS. I'll look more into this.

-flto in LDFLAGS screws up this part of paging_prepare():

/* Copy trampoline code in place */
memcpy(trampoline_32bit + TRAMPOLINE_32BIT_CODE_OFFSET / sizeof(unsigned long),
&trampoline_32bit_src, TRAMPOLINE_32BIT_CODE_SIZE);

In particular, relocation for trampoline_32bit_src solved in the wrong
way. Without -flto, we have rip-realtive address load:

982d30: 48 8d 35 09 cc ff ff lea -0x33f7(%rip),%rsi # 97f940 <trampoline_32bit_src>

With -flto we have immediate load:

982cf0: 48 c7 c6 f0 f8 97 00 mov $0x97f8f0,%rsi

It only would be okay if bootloader loads kernel at the address we compile
it for. But it's not usually the case.

As result we copy garbage into trampoline and crash when trying to execute
it.

I don't know how to solve it. As far as I know we don't support compiling
kernel with LTO in mainline.

Any suggestions?

Benjamin, do you change LDFLAGS or CFLAGS when compiling the kernel?

--
Kirill A. Shutemov

2018-07-03 14:28:43

by Thomas Gleixner

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, 3 Jul 2018, Kirill A. Shutemov wrote:
> On Tue, Jul 03, 2018 at 03:44:03PM +0300, Kirill A. Shutemov wrote:
> > On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
> > > 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> > > > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > > >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> > > >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> > > >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> > > >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> > > >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> > > >> config for reference, and am happy to test patches, provide sample QCOW
> > > >> images, etc.
> > > >
> > >
> > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
> > >
> > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> > > too with the same symptoms
> >
> > I tracked it down to -flto in LDFLAGS. I'll look more into this.
>
> -flto in LDFLAGS screws up this part of paging_prepare():
>
> /* Copy trampoline code in place */
> memcpy(trampoline_32bit + TRAMPOLINE_32BIT_CODE_OFFSET / sizeof(unsigned long),
> &trampoline_32bit_src, TRAMPOLINE_32BIT_CODE_SIZE);
>
> In particular, relocation for trampoline_32bit_src solved in the wrong
> way. Without -flto, we have rip-realtive address load:
>
> 982d30: 48 8d 35 09 cc ff ff lea -0x33f7(%rip),%rsi # 97f940 <trampoline_32bit_src>
>
> With -flto we have immediate load:
>
> 982cf0: 48 c7 c6 f0 f8 97 00 mov $0x97f8f0,%rsi
>
> It only would be okay if bootloader loads kernel at the address we compile
> it for. But it's not usually the case.
>
> As result we copy garbage into trampoline and crash when trying to execute
> it.
>
> I don't know how to solve it. As far as I know we don't support compiling
> kernel with LTO in mainline.
>
> Any suggestions?

LTO is broken. The boot code is compiled as position independent, so this
'optimization' is pure garbage.

Thanks,

tglx


2018-07-03 18:04:15

by Andi Kleen

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, Jul 03, 2018 at 05:21:50PM +0300, Kirill A. Shutemov wrote:
> On Tue, Jul 03, 2018 at 03:44:03PM +0300, Kirill A. Shutemov wrote:
> > On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
> > > 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> > > > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > > >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> > > >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> > > >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> > > >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> > > >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> > > >> config for reference, and am happy to test patches, provide sample QCOW
> > > >> images, etc.
> > > >
> > >
> > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
> > >
> > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> > > too with the same symptoms
> >
> > I tracked it down to -flto in LDFLAGS. I'll look more into this.
>
> -flto in LDFLAGS screws up this part of paging_prepare():

Where is that coming from? The LTO patches are not upstream.

And I don't see any LTO usage in the main line.

>
> /* Copy trampoline code in place */
> memcpy(trampoline_32bit + TRAMPOLINE_32BIT_CODE_OFFSET / sizeof(unsigned long),
> &trampoline_32bit_src, TRAMPOLINE_32BIT_CODE_SIZE);


> In particular, relocation for trampoline_32bit_src solved in the wrong
> way. Without -flto, we have rip-realtive address load:
>
> 982d30: 48 8d 35 09 cc ff ff lea -0x33f7(%rip),%rsi # 97f940 <trampoline_32bit_src>
>
> With -flto we have immediate load:
>
> 982cf0: 48 c7 c6 f0 f8 97 00 mov $0x97f8f0,%rsi

Strange.

Can you add some RELOC_HIDE()s and see if that helps?

> It only would be okay if bootloader loads kernel at the address we compile
> it for. But it's not usually the case.
>
> As result we copy garbage into trampoline and crash when trying to execute
> it.
>
> I don't know how to solve it. As far as I know we don't support compiling
> kernel with LTO in mainline.

Right.


-Andi

2018-07-03 20:27:17

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, Jul 03, 2018 at 11:03:07AM -0700, Andi Kleen wrote:
> On Tue, Jul 03, 2018 at 05:21:50PM +0300, Kirill A. Shutemov wrote:
> > On Tue, Jul 03, 2018 at 03:44:03PM +0300, Kirill A. Shutemov wrote:
> > > On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
> > > > 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> > > > > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > > > >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> > > > >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> > > > >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> > > > >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> > > > >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> > > > >> config for reference, and am happy to test patches, provide sample QCOW
> > > > >> images, etc.
> > > > >
> > > >
> > > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
> > > >
> > > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> > > > too with the same symptoms
> > >
> > > I tracked it down to -flto in LDFLAGS. I'll look more into this.
> >
> > -flto in LDFLAGS screws up this part of paging_prepare():
>
> Where is that coming from? The LTO patches are not upstream.
>
> And I don't see any LTO usage in the main line.

Apparently some distros try to hack it around:

https://bugzilla.kernel.org/show_bug.cgi?id=200385

I'm amazed that it kinda worked for them.


> > /* Copy trampoline code in place */
> > memcpy(trampoline_32bit + TRAMPOLINE_32BIT_CODE_OFFSET / sizeof(unsigned long),
> > &trampoline_32bit_src, TRAMPOLINE_32BIT_CODE_SIZE);
>
>
> > In particular, relocation for trampoline_32bit_src solved in the wrong
> > way. Without -flto, we have rip-realtive address load:
> >
> > 982d30: 48 8d 35 09 cc ff ff lea -0x33f7(%rip),%rsi # 97f940 <trampoline_32bit_src>
> >
> > With -flto we have immediate load:
> >
> > 982cf0: 48 c7 c6 f0 f8 97 00 mov $0x97f8f0,%rsi
>
> Strange.
>
> Can you add some RELOC_HIDE()s and see if that helps?

Nope. No difference in generated code.

--
Kirill A. Shutemov

2018-07-03 21:02:57

by Andi Kleen

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, Jul 03, 2018 at 11:26:09PM +0300, Kirill A. Shutemov wrote:
> On Tue, Jul 03, 2018 at 11:03:07AM -0700, Andi Kleen wrote:
> > On Tue, Jul 03, 2018 at 05:21:50PM +0300, Kirill A. Shutemov wrote:
> > > On Tue, Jul 03, 2018 at 03:44:03PM +0300, Kirill A. Shutemov wrote:
> > > > On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
> > > > > 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> > > > > > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > > > > >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> > > > > >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> > > > > >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> > > > > >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> > > > > >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> > > > > >> config for reference, and am happy to test patches, provide sample QCOW
> > > > > >> images, etc.
> > > > > >
> > > > >
> > > > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
> > > > >
> > > > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> > > > > too with the same symptoms
> > > >
> > > > I tracked it down to -flto in LDFLAGS. I'll look more into this.
> > >
> > > -flto in LDFLAGS screws up this part of paging_prepare():
> >
> > Where is that coming from? The LTO patches are not upstream.
> >
> > And I don't see any LTO usage in the main line.
>
> Apparently some distros try to hack it around:
>
> https://bugzilla.kernel.org/show_bug.cgi?id=200385
>
> I'm amazed that it kinda worked for them.

I think it only works on older gccs that don't default to
thin LTO, but always generate a fallback non LTO object.
The kernel directly uses ld in the link step (without my patches), so LTO
shouldn't be able to ever generate code.

The early boot code may be an exception of this, and it's likely
the only code that actually uses LTO in such a set up.

The -fPIC is actually scarier than the -flto. The generated code
must create quite a mess and I'm not sure why you even would want that
because the kernel can be relocatable without it.

BTW I hope to eventually resend the full LTO patches.
They seem to get more and more users recently, mainly for smaller
code size.

>
>
> > > /* Copy trampoline code in place */
> > > memcpy(trampoline_32bit + TRAMPOLINE_32BIT_CODE_OFFSET / sizeof(unsigned long),
> > > &trampoline_32bit_src, TRAMPOLINE_32BIT_CODE_SIZE);
> >
> >
> > > In particular, relocation for trampoline_32bit_src solved in the wrong
> > > way. Without -flto, we have rip-realtive address load:
> > >
> > > 982d30: 48 8d 35 09 cc ff ff lea -0x33f7(%rip),%rsi # 97f940 <trampoline_32bit_src>
> > >
> > > With -flto we have immediate load:
> > >
> > > 982cf0: 48 c7 c6 f0 f8 97 00 mov $0x97f8f0,%rsi
> >
> > Strange.
> >
> > Can you add some RELOC_HIDE()s and see if that helps?
>
> Nope. No difference in generated code.

Ok will need to put together some self contained test case for the compiler people.
I'll try to take a look.

-Andi

2018-07-04 03:11:37

by Benjamin Gilbert

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, Jul 03, 2018 at 05:21:50PM +0300, Kirill A. Shutemov wrote:
> I don't know how to solve it. As far as I know we don't support compiling
> kernel with LTO in mainline.
>
> Any suggestions?
>
> Benjamin, do you change LDFLAGS or CFLAGS when compiling the kernel?

We're using the standard build flags as far as I can tell. In particular,
we don't enable LTO, and I've verified that -flto isn't in the build logs.

Here's a sample image:

https://users.developer.core-os.net/bgilbert/4.17/vmlinuz-4.17.3-coreos
https://users.developer.core-os.net/bgilbert/4.17/vmlinux-4.17.3-coreos
https://users.developer.core-os.net/bgilbert/4.17/System.map

--Benjamin Gilbert

2018-07-04 13:24:20

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, Jul 03, 2018 at 11:10:27PM -0400, Benjamin Gilbert wrote:
> On Tue, Jul 03, 2018 at 05:21:50PM +0300, Kirill A. Shutemov wrote:
> > I don't know how to solve it. As far as I know we don't support compiling
> > kernel with LTO in mainline.
> >
> > Any suggestions?
> >
> > Benjamin, do you change LDFLAGS or CFLAGS when compiling the kernel?
>
> We're using the standard build flags as far as I can tell. In particular,
> we don't enable LTO, and I've verified that -flto isn't in the build logs.
>
> Here's a sample image:
>
> https://users.developer.core-os.net/bgilbert/4.17/vmlinuz-4.17.3-coreos
> https://users.developer.core-os.net/bgilbert/4.17/vmlinux-4.17.3-coreos
> https://users.developer.core-os.net/bgilbert/4.17/System.map

It's basically the same issue. We have immidiate load instead of
RIP-relative address load.

You can make the vmlinuz bootable with this binary patch:

echo -en "\x8d\x05\xa9\xa9\xff\xff" | dd of=vmlinuz-4.17.3-coreos seek=$((0x005d1fc1)) bs=1 conv=notrunc

Now we need to find out how linker gets it wrong.

Please, *after* complete build of the kernel with your toolchain do this:

touch arch/x86/boot/compressed/pgtable_64.c
make V=1

And share your build log.

--
Kirill A. Shutemov

2018-07-04 15:38:34

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Tue, Jul 03, 2018 at 05:21:50PM +0300, Kirill A. Shutemov wrote:
> On Tue, Jul 03, 2018 at 03:44:03PM +0300, Kirill A. Shutemov wrote:
> > On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
> > > 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> > > > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > > >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> > > >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> > > >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> > > >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> > > >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> > > >> config for reference, and am happy to test patches, provide sample QCOW
> > > >> images, etc.
> > > >
> > >
> > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
> > >
> > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> > > too with the same symptoms
> >
> > I tracked it down to -flto in LDFLAGS. I'll look more into this.
>
> -flto in LDFLAGS screws up this part of paging_prepare():

+Masahiro, Michal.

I've got it wrong. *Any* LDFLAGS option passed to make this way:

make LDFLAGS="..."

would cause a issue. Even empty.

It overrides all assignments to the variable in the makefile.
As result the image is built without -pie and linker doesn't generate
position independed code.

Looks like the patch below helps, but my make-fu is poor.
I don't see many override directives in kernel makefiles.
It makes me think that there's a better way to fix this.

Hm?

diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index fa42f895fdde..4f24baa8cdeb 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -42,16 +42,16 @@ KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__
GCOV_PROFILE := n
UBSAN_SANITIZE :=n

-LDFLAGS := -m elf_$(UTS_MACHINE)
+override LDFLAGS := -m elf_$(UTS_MACHINE)
# Compressed kernel should be built as PIE since it may be loaded at any
# address by the bootloader.
ifeq ($(CONFIG_X86_32),y)
-LDFLAGS += $(call ld-option, -pie) $(call ld-option, --no-dynamic-linker)
+override LDFLAGS += $(call ld-option, -pie) $(call ld-option, --no-dynamic-linker)
else
# To build 64-bit compressed kernel as PIE, we disable relocation
# overflow check to avoid relocation overflow error with a new linker
# command-line option, -z noreloc-overflow.
-LDFLAGS += $(shell $(LD) --help 2>&1 | grep -q "\-z noreloc-overflow" \
+override LDFLAGS += $(shell $(LD) --help 2>&1 | grep -q "\-z noreloc-overflow" \
&& echo "-z noreloc-overflow -pie --no-dynamic-linker")
endif
LDFLAGS_vmlinux := -T
--
Kirill A. Shutemov

2018-07-04 20:43:47

by Benjamin Gilbert

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Wed, Jul 04, 2018 at 06:08:57PM +0300, Kirill A. Shutemov wrote:
> On Tue, Jul 03, 2018 at 05:21:50PM +0300, Kirill A. Shutemov wrote:
> > On Tue, Jul 03, 2018 at 03:44:03PM +0300, Kirill A. Shutemov wrote:
> > > On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
> > > > 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
> > > > > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
> > > > >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
> > > > >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
> > > > >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
> > > > >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
> > > > >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
> > > > >> config for reference, and am happy to test patches, provide sample QCOW
> > > > >> images, etc.
> > > > >
> > > >
> > > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
> > > >
> > > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> > > > too with the same symptoms
> > >
> > > I tracked it down to -flto in LDFLAGS. I'll look more into this.
> >
> > -flto in LDFLAGS screws up this part of paging_prepare():
>
> I've got it wrong. *Any* LDFLAGS option passed to make this way:
>
> make LDFLAGS="..."
>
> would cause a issue. Even empty.
>
> It overrides all assignments to the variable in the makefile.
> As result the image is built without -pie and linker doesn't generate
> position independed code.
>
> Looks like the patch below helps, but my make-fu is poor.

Sure enough, we're passing LDFLAGS="" to make. Your patch fixes the boot
failure for me.

--Benjamin Gilbert

2018-07-06 06:41:05

by Masahiro Yamada

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

Hi.


2018-07-05 0:08 GMT+09:00 Kirill A. Shutemov <[email protected]>:
> On Tue, Jul 03, 2018 at 05:21:50PM +0300, Kirill A. Shutemov wrote:
>> On Tue, Jul 03, 2018 at 03:44:03PM +0300, Kirill A. Shutemov wrote:
>> > On Tue, Jul 03, 2018 at 01:24:49PM +0200, Gabriel C wrote:
>> > > 2018-07-01 23:32 GMT+02:00 Benjamin Gilbert <[email protected]>:
>> > > > On Sun, Jul 01, 2018 at 05:15:59PM -0400, Benjamin Gilbert wrote:
>> > > >> 4.17 kernels built with the CoreOS Container Linux toolchain and kconfig,
>> > > >> up to and including 4.17.3, fail to boot on AMD64 running in (at least)
>> > > >> QEMU/KVM. No messages are shown post-GRUB; the VM instantly reboots.
>> > > >> Reverting commit 194a9749c73d ("x86/boot/compressed/64: Handle 5-level
>> > > >> paging boot if kernel is above 4G") fixes it. I've attached our kernel
>> > > >> config for reference, and am happy to test patches, provide sample QCOW
>> > > >> images, etc.
>> > > >
>> > >
>> > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
>> > >
>> > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
>> > > too with the same symptoms
>> >
>> > I tracked it down to -flto in LDFLAGS. I'll look more into this.
>>
>> -flto in LDFLAGS screws up this part of paging_prepare():
>
> +Masahiro, Michal.
>
> I've got it wrong. *Any* LDFLAGS option passed to make this way:
>
> make LDFLAGS="..."
>
> would cause a issue. Even empty.
>
> It overrides all assignments to the variable in the makefile.
> As result the image is built without -pie and linker doesn't generate
> position independed code.
>
> Looks like the patch below helps, but my make-fu is poor.
> I don't see many override directives in kernel makefiles.
> It makes me think that there's a better way to fix this.
>
> Hm?


LDFLAGS is for internal-use.
Please do not override it from the command line.


You want to pass your own linker flags
for building vmlinux and modules,
but do not want to pass them to
the decompressor (arch/x86/boot/compressed).

Correct?



Kbuild provides a way for users
to pass additional linker flags to modules.
(LDFLAGS_MODULE)


But, there is no way to do that for vmlinux.

It is easy to support it, though.

https://patchwork.kernel.org/patch/10510833/

If this is the one you want, I can merge this.


make LDFLAGS_KERNEL=... LDFLAGS_MODULE=...
will allow you to append linker flags.





> diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
> index fa42f895fdde..4f24baa8cdeb 100644
> --- a/arch/x86/boot/compressed/Makefile
> +++ b/arch/x86/boot/compressed/Makefile
> @@ -42,16 +42,16 @@ KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__
> GCOV_PROFILE := n
> UBSAN_SANITIZE :=n
>
> -LDFLAGS := -m elf_$(UTS_MACHINE)
> +override LDFLAGS := -m elf_$(UTS_MACHINE)
> # Compressed kernel should be built as PIE since it may be loaded at any
> # address by the bootloader.
> ifeq ($(CONFIG_X86_32),y)
> -LDFLAGS += $(call ld-option, -pie) $(call ld-option, --no-dynamic-linker)
> +override LDFLAGS += $(call ld-option, -pie) $(call ld-option, --no-dynamic-linker)
> else
> # To build 64-bit compressed kernel as PIE, we disable relocation
> # overflow check to avoid relocation overflow error with a new linker
> # command-line option, -z noreloc-overflow.
> -LDFLAGS += $(shell $(LD) --help 2>&1 | grep -q "\-z noreloc-overflow" \
> +override LDFLAGS += $(shell $(LD) --help 2>&1 | grep -q "\-z noreloc-overflow" \
> && echo "-z noreloc-overflow -pie --no-dynamic-linker")
> endif
> LDFLAGS_vmlinux := -T
> --
> Kirill A. Shutemov



--
Best Regards
Masahiro Yamada

2018-07-06 10:42:54

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Fri, Jul 06, 2018 at 03:37:58PM +0900, Masahiro Yamada wrote:
> >> > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
> >> > >
> >> > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
> >> > > too with the same symptoms
> >> >
> >> > I tracked it down to -flto in LDFLAGS. I'll look more into this.
> >>
> >> -flto in LDFLAGS screws up this part of paging_prepare():
> >
> > +Masahiro, Michal.
> >
> > I've got it wrong. *Any* LDFLAGS option passed to make this way:
> >
> > make LDFLAGS="..."
> >
> > would cause a issue. Even empty.
> >
> > It overrides all assignments to the variable in the makefile.
> > As result the image is built without -pie and linker doesn't generate
> > position independed code.
> >
> > Looks like the patch below helps, but my make-fu is poor.
> > I don't see many override directives in kernel makefiles.
> > It makes me think that there's a better way to fix this.
> >
> > Hm?
>
>
> LDFLAGS is for internal-use.
> Please do not override it from the command line.

Can we generate a build error if a user try to override LDFLAGS, CFLAGS or
other critical internal-use-only variables?

This breakage was rather hard to debug. We need to have some kind of
fail-safe for the future.

> You want to pass your own linker flags
> for building vmlinux and modules,
> but do not want to pass them to
> the decompressor (arch/x86/boot/compressed).
>
> Correct?

I personally don't think that changing compiler/linker options for kernel
build is good idea in general.

> Kbuild provides a way for users
> to pass additional linker flags to modules.
> (LDFLAGS_MODULE)
>
>
> But, there is no way to do that for vmlinux.
>
> It is easy to support it, though.
>
> https://patchwork.kernel.org/patch/10510833/
>
> If this is the one you want, I can merge this.
>
>
> make LDFLAGS_KERNEL=... LDFLAGS_MODULE=...
> will allow you to append linker flags.

Okay. It makes me wounder if we should taint kernel in such cases?
Custom compiler/linker flags are risky and can lead to weird bugs.

--
Kirill A. Shutemov

2018-07-06 14:15:36

by Masahiro Yamada

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

Hi.

2018-07-06 19:41 GMT+09:00 Kirill A. Shutemov <[email protected]>:
> On Fri, Jul 06, 2018 at 03:37:58PM +0900, Masahiro Yamada wrote:
>> >> > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
>> >> > >
>> >> > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
>> >> > > too with the same symptoms
>> >> >
>> >> > I tracked it down to -flto in LDFLAGS. I'll look more into this.
>> >>
>> >> -flto in LDFLAGS screws up this part of paging_prepare():
>> >
>> > +Masahiro, Michal.
>> >
>> > I've got it wrong. *Any* LDFLAGS option passed to make this way:
>> >
>> > make LDFLAGS="..."
>> >
>> > would cause a issue. Even empty.
>> >
>> > It overrides all assignments to the variable in the makefile.
>> > As result the image is built without -pie and linker doesn't generate
>> > position independed code.
>> >
>> > Looks like the patch below helps, but my make-fu is poor.
>> > I don't see many override directives in kernel makefiles.
>> > It makes me think that there's a better way to fix this.
>> >
>> > Hm?
>>
>>
>> LDFLAGS is for internal-use.
>> Please do not override it from the command line.
>
> Can we generate a build error if a user try to override LDFLAGS, CFLAGS or
> other critical internal-use-only variables?

Yes, Make can check where variables came from.


> This breakage was rather hard to debug. We need to have some kind of
> fail-safe for the future.
>
>> You want to pass your own linker flags
>> for building vmlinux and modules,
>> but do not want to pass them to
>> the decompressor (arch/x86/boot/compressed).
>>
>> Correct?
>
> I personally don't think that changing compiler/linker options for kernel
> build is good idea in general.
>
>> Kbuild provides a way for users
>> to pass additional linker flags to modules.
>> (LDFLAGS_MODULE)
>>
>>
>> But, there is no way to do that for vmlinux.
>>
>> It is easy to support it, though.
>>
>> https://patchwork.kernel.org/patch/10510833/
>>
>> If this is the one you want, I can merge this.
>>
>>
>> make LDFLAGS_KERNEL=... LDFLAGS_MODULE=...
>> will allow you to append linker flags.
>
> Okay. It makes me wounder if we should taint kernel in such cases?
> Custom compiler/linker flags are risky and can lead to weird bugs.

OK.
So, what problem are we discussing?


> I've got it wrong. *Any* LDFLAGS option passed to make this way:
>
> make LDFLAGS="..."

In your previous mail, I thought you were asking me how to pass
custom linker flags.

If not, we do not need to think about that case.
Just say "Do not do that".



--
Best Regards
Masahiro Yamada

2018-07-06 14:41:12

by Gabriel C

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

2018-07-06 16:13 GMT+02:00 Masahiro Yamada <[email protected]>:
> Hi.
>
> 2018-07-06 19:41 GMT+09:00 Kirill A. Shutemov <[email protected]>:
>> On Fri, Jul 06, 2018 at 03:37:58PM +0900, Masahiro Yamada wrote:
>>> >> > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
>>> >> > >
>>> >> > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
>>> >> > > too with the same symptoms
>>> >> >
>>> >> > I tracked it down to -flto in LDFLAGS. I'll look more into this.
>>> >>
>>> >> -flto in LDFLAGS screws up this part of paging_prepare():
>>> >
>>> > +Masahiro, Michal.
>>> >
>>> > I've got it wrong. *Any* LDFLAGS option passed to make this way:
>>> >
>>> > make LDFLAGS="..."
>>> >
>>> > would cause a issue. Even empty.
>>> >
>>> > It overrides all assignments to the variable in the makefile.
>>> > As result the image is built without -pie and linker doesn't generate
>>> > position independed code.
>>> >
>>> > Looks like the patch below helps, but my make-fu is poor.
>>> > I don't see many override directives in kernel makefiles.
>>> > It makes me think that there's a better way to fix this.
>>> >
>>> > Hm?
>>>
>>>
>>> LDFLAGS is for internal-use.
>>> Please do not override it from the command line.
>>
>> Can we generate a build error if a user try to override LDFLAGS, CFLAGS or
>> other critical internal-use-only variables?
>
> Yes, Make can check where variables came from.
>
>
>> This breakage was rather hard to debug. We need to have some kind of
>> fail-safe for the future.
>>
>>> You want to pass your own linker flags
>>> for building vmlinux and modules,
>>> but do not want to pass them to
>>> the decompressor (arch/x86/boot/compressed).
>>>
>>> Correct?
>>
>> I personally don't think that changing compiler/linker options for kernel
>> build is good idea in general.
>>
>>> Kbuild provides a way for users
>>> to pass additional linker flags to modules.
>>> (LDFLAGS_MODULE)
>>>
>>>
>>> But, there is no way to do that for vmlinux.
>>>
>>> It is easy to support it, though.
>>>
>>> https://patchwork.kernel.org/patch/10510833/
>>>
>>> If this is the one you want, I can merge this.
>>>
>>>
>>> make LDFLAGS_KERNEL=... LDFLAGS_MODULE=...
>>> will allow you to append linker flags.
>>
>> Okay. It makes me wounder if we should taint kernel in such cases?
>> Custom compiler/linker flags are risky and can lead to weird bugs.
>
> OK.
> So, what problem are we discussing?
>
>
>> I've got it wrong. *Any* LDFLAGS option passed to make this way:
>>
>> make LDFLAGS="..."
>
> In your previous mail, I thought you were asking me how to pass
> custom linker flags.
>
> If not, we do not need to think about that case.
> Just say "Do not do that".

I am sorry but I have a hard time to get your logic here.

You are saying : the *env* variable LDFLAGS as well passing
LDFLAGS to make , which your build allows should not be use
because is for 'internal usage' .. ?

Well that logic you have here is wrong and wrong for any project
not just for the kernel,

If you know 'parts' need have particular flags then 'you' have to
ensure nothing
overrides these or nothing at all can chage these.

So swap your logic and apped LDFLAGS to your private
'call_it_whatever_you_wish_KERNEL_NEED_BE_THERE_ANY_KIND_FLAGS'
and don't allow these to be changed at all , when you
*know* they have be there.


Teling users to not use LD/C/CXX flags is not really going to work right ?


BR

2018-07-06 16:30:39

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Fri, Jul 06, 2018 at 11:13:02PM +0900, Masahiro Yamada wrote:
> >> LDFLAGS is for internal-use.
> >> Please do not override it from the command line.
> >
> > Can we generate a build error if a user try to override LDFLAGS, CFLAGS or
> > other critical internal-use-only variables?
>
> Yes, Make can check where variables came from.

I think we should do this.

> >> make LDFLAGS_KERNEL=... LDFLAGS_MODULE=...
> >> will allow you to append linker flags.
> >
> > Okay. It makes me wounder if we should taint kernel in such cases?
> > Custom compiler/linker flags are risky and can lead to weird bugs.
>
> OK.
> So, what problem are we discussing?

Users set custom LDFLAGS/CFLAGS and break kernel. Then report bug that
hard to debug. See

https://bugzilla.kernel.org/show_bug.cgi?id=200385

and start of this thread:

https://lore.kernel.org/lkml/[email protected]/

It took me a while to track down the issue. I blamed linker for a while.

> > I've got it wrong. *Any* LDFLAGS option passed to make this way:
> >
> > make LDFLAGS="..."
>
> In your previous mail, I thought you were asking me how to pass
> custom linker flags.
>
> If not, we do not need to think about that case.
> Just say "Do not do that".

At least we need to make user aware about risk of setting custom flags.

--
Kirill A. Shutemov

2018-07-06 16:34:18

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Fri, Jul 06, 2018 at 04:39:28PM +0200, Gabriel C wrote:
> > If not, we do not need to think about that case.
> > Just say "Do not do that".
>
> I am sorry but I have a hard time to get your logic here.
>
> You are saying : the *env* variable LDFLAGS as well passing
> LDFLAGS to make , which your build allows should not be use
> because is for 'internal usage' .. ?

Environment variables do not override make variables. Only passing varible
assignment as make argument will do this.

--
Kirill A. Shutemov

2018-07-06 17:33:07

by Gabriel C

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

2018-07-06 18:33 GMT+02:00 Kirill A. Shutemov <[email protected]>:
> On Fri, Jul 06, 2018 at 04:39:28PM +0200, Gabriel C wrote:
>> > If not, we do not need to think about that case.
>> > Just say "Do not do that".
>>
>> I am sorry but I have a hard time to get your logic here.
>>
>> You are saying : the *env* variable LDFLAGS as well passing
>> LDFLAGS to make , which your build allows should not be use
>> because is for 'internal usage' .. ?
>
> Environment variables do not override make variables. Only passing varible
> assignment as make argument will do this.
>

Still .. When the build system allows to do 'make FOO=bar' and you know
when using that things will break , the build system should be fixed.

2018-07-06 18:12:50

by Andi Kleen

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

> At least we need to make user aware about risk of setting custom flags.

There are valid use cases to override the flags. I use it sometimes too,
and know some other people do to.

But you need to know what you're doing.

Perhaps a warning during build would be reasonable. So if you ask
for a build log you would see it.

-Andi

2018-07-06 19:36:28

by Benjamin Gilbert

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Fri, Jul 06, 2018 at 11:11:10AM -0700, Andi Kleen wrote:
> There are valid use cases to override the flags. I use it sometimes too,
> and know some other people do to.
>
> But you need to know what you're doing.
>
> Perhaps a warning during build would be reasonable. So if you ask
> for a build log you would see it.

In our case, the package is presumably passing LDFLAGS="" to override the
LDFLAGS environment variable already set by the packaging system. This has
worked for years without a problem.

--Benjamin Gilbert

2018-07-07 00:58:43

by Masahiro Yamada

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

2018-07-06 23:39 GMT+09:00 Gabriel C <[email protected]>:
> 2018-07-06 16:13 GMT+02:00 Masahiro Yamada <[email protected]>:
>> Hi.
>>
>> 2018-07-06 19:41 GMT+09:00 Kirill A. Shutemov <[email protected]>:
>>> On Fri, Jul 06, 2018 at 03:37:58PM +0900, Masahiro Yamada wrote:
>>>> >> > > Also see https://bugzilla.kernel.org/show_bug.cgi?id=200385 ,
>>>> >> > >
>>>> >> > > 0a1756bd2897951c03c1cb671bdfd40729ac2177 is acting up
>>>> >> > > too with the same symptoms
>>>> >> >
>>>> >> > I tracked it down to -flto in LDFLAGS. I'll look more into this.
>>>> >>
>>>> >> -flto in LDFLAGS screws up this part of paging_prepare():
>>>> >
>>>> > +Masahiro, Michal.
>>>> >
>>>> > I've got it wrong. *Any* LDFLAGS option passed to make this way:
>>>> >
>>>> > make LDFLAGS="..."
>>>> >
>>>> > would cause a issue. Even empty.
>>>> >
>>>> > It overrides all assignments to the variable in the makefile.
>>>> > As result the image is built without -pie and linker doesn't generate
>>>> > position independed code.
>>>> >
>>>> > Looks like the patch below helps, but my make-fu is poor.
>>>> > I don't see many override directives in kernel makefiles.
>>>> > It makes me think that there's a better way to fix this.
>>>> >
>>>> > Hm?
>>>>
>>>>
>>>> LDFLAGS is for internal-use.
>>>> Please do not override it from the command line.
>>>
>>> Can we generate a build error if a user try to override LDFLAGS, CFLAGS or
>>> other critical internal-use-only variables?
>>
>> Yes, Make can check where variables came from.
>>
>>
>>> This breakage was rather hard to debug. We need to have some kind of
>>> fail-safe for the future.
>>>
>>>> You want to pass your own linker flags
>>>> for building vmlinux and modules,
>>>> but do not want to pass them to
>>>> the decompressor (arch/x86/boot/compressed).
>>>>
>>>> Correct?
>>>
>>> I personally don't think that changing compiler/linker options for kernel
>>> build is good idea in general.
>>>
>>>> Kbuild provides a way for users
>>>> to pass additional linker flags to modules.
>>>> (LDFLAGS_MODULE)
>>>>
>>>>
>>>> But, there is no way to do that for vmlinux.
>>>>
>>>> It is easy to support it, though.
>>>>
>>>> https://patchwork.kernel.org/patch/10510833/
>>>>
>>>> If this is the one you want, I can merge this.
>>>>
>>>>
>>>> make LDFLAGS_KERNEL=... LDFLAGS_MODULE=...
>>>> will allow you to append linker flags.
>>>
>>> Okay. It makes me wounder if we should taint kernel in such cases?
>>> Custom compiler/linker flags are risky and can lead to weird bugs.
>>
>> OK.
>> So, what problem are we discussing?
>>
>>
>>> I've got it wrong. *Any* LDFLAGS option passed to make this way:
>>>
>>> make LDFLAGS="..."
>>
>> In your previous mail, I thought you were asking me how to pass
>> custom linker flags.
>>
>> If not, we do not need to think about that case.
>> Just say "Do not do that".
>
> I am sorry but I have a hard time to get your logic here.
>
> You are saying : the *env* variable LDFLAGS as well passing
> LDFLAGS to make , which your build allows should not be use
> because is for 'internal usage' .. ?
>
> Well that logic you have here is wrong and wrong for any project
> not just for the kernel,


Why 'my logic'?

LDFLAGS has been long used internally since the old days,
before I ever worked on the kernel.


I shared my knowledge about the kernel build system.

The current situation is not nice,
but why are you blaming me for the code I did not add ?


Note:
I have never said 'the *env* variable LDFLAGS'


> If you know 'parts' need have particular flags then 'you' have to
> ensure nothing
> overrides these or nothing at all can chage these.
>
> So swap your logic and apped LDFLAGS to your private
> 'call_it_whatever_you_wish_KERNEL_NEED_BE_THERE_ANY_KIND_FLAGS'
> and don't allow these to be changed at all , when you
> *know* they have be there.
>
>
> Teling users to not use LD/C/CXX flags is not really going to work right ?
>
>
> BR



--
Best Regards
Masahiro Yamada

2018-07-07 01:23:48

by Masahiro Yamada

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

2018-07-07 1:29 GMT+09:00 Kirill A. Shutemov <[email protected]>:
> On Fri, Jul 06, 2018 at 11:13:02PM +0900, Masahiro Yamada wrote:
>> >> LDFLAGS is for internal-use.
>> >> Please do not override it from the command line.
>> >
>> > Can we generate a build error if a user try to override LDFLAGS, CFLAGS or
>> > other critical internal-use-only variables?
>>
>> Yes, Make can check where variables came from.
>
> I think we should do this.
>
>> >> make LDFLAGS_KERNEL=... LDFLAGS_MODULE=...
>> >> will allow you to append linker flags.
>> >
>> > Okay. It makes me wounder if we should taint kernel in such cases?
>> > Custom compiler/linker flags are risky and can lead to weird bugs.
>>
>> OK.
>> So, what problem are we discussing?
>
> Users set custom LDFLAGS/CFLAGS and break kernel. Then report bug that
> hard to debug. See
>
> https://bugzilla.kernel.org/show_bug.cgi?id=200385


CFLAGS is only used under tools/.
Passing CFLAGS is probably no effect to the kernel.

In Linux makefiles,
KBUILD_ prefixed variables are used internally.

KBUILD_CFLAGS, KBUILD_CPPFLAGS, KBUILD_AFLAGS, etc.


LDFLAGS is an exception. I do not know why.
Renaming LDFLAGS to KBUILD_LDFLAGS
will make the code consistent.

At least, it will avoid overriding flags by accident.

Of course, users still can change KBUILD_LDFLAGS
if they really want.

The build system could add belt and braces checks for that,
but it is arguable since
there are lots of lots of internal variables.




> and start of this thread:
>
> https://lore.kernel.org/lkml/[email protected]/
>
> It took me a while to track down the issue. I blamed linker for a while.
>
>> > I've got it wrong. *Any* LDFLAGS option passed to make this way:
>> >
>> > make LDFLAGS="..."
>>
>> In your previous mail, I thought you were asking me how to pass
>> custom linker flags.
>>
>> If not, we do not need to think about that case.
>> Just say "Do not do that".
>
> At least we need to make user aware about risk of setting custom flags.
>
> --
> Kirill A. Shutemov



--
Best Regards
Masahiro Yamada

2018-07-09 10:12:40

by Kirill A. Shutemov

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

On Sat, Jul 07, 2018 at 10:21:47AM +0900, Masahiro Yamada wrote:
> 2018-07-07 1:29 GMT+09:00 Kirill A. Shutemov <[email protected]>:
> > On Fri, Jul 06, 2018 at 11:13:02PM +0900, Masahiro Yamada wrote:
> >> >> LDFLAGS is for internal-use.
> >> >> Please do not override it from the command line.
> >> >
> >> > Can we generate a build error if a user try to override LDFLAGS, CFLAGS or
> >> > other critical internal-use-only variables?
> >>
> >> Yes, Make can check where variables came from.
> >
> > I think we should do this.
> >
> >> >> make LDFLAGS_KERNEL=... LDFLAGS_MODULE=...
> >> >> will allow you to append linker flags.
> >> >
> >> > Okay. It makes me wounder if we should taint kernel in such cases?
> >> > Custom compiler/linker flags are risky and can lead to weird bugs.
> >>
> >> OK.
> >> So, what problem are we discussing?
> >
> > Users set custom LDFLAGS/CFLAGS and break kernel. Then report bug that
> > hard to debug. See
> >
> > https://bugzilla.kernel.org/show_bug.cgi?id=200385
>
>
> CFLAGS is only used under tools/.
> Passing CFLAGS is probably no effect to the kernel.
>
> In Linux makefiles,
> KBUILD_ prefixed variables are used internally.
>
> KBUILD_CFLAGS, KBUILD_CPPFLAGS, KBUILD_AFLAGS, etc.
>
>
> LDFLAGS is an exception. I do not know why.
> Renaming LDFLAGS to KBUILD_LDFLAGS
> will make the code consistent.
>
> At least, it will avoid overriding flags by accident.
>
> Of course, users still can change KBUILD_LDFLAGS
> if they really want.
>
> The build system could add belt and braces checks for that,
> but it is arguable since
> there are lots of lots of internal variables.

I think renaming LDFLAGS to KBUILD_LDFLAGS is good idea.
Would you prepare patch?

--
Kirill A. Shutemov

2018-07-09 10:39:54

by Masahiro Yamada

[permalink] [raw]
Subject: Re: 4.17.x won't boot due to "x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G"

2018-07-09 19:10 GMT+09:00 Kirill A. Shutemov <[email protected]>:
> On Sat, Jul 07, 2018 at 10:21:47AM +0900, Masahiro Yamada wrote:
>> 2018-07-07 1:29 GMT+09:00 Kirill A. Shutemov <[email protected]>:
>> > On Fri, Jul 06, 2018 at 11:13:02PM +0900, Masahiro Yamada wrote:
>> >> >> LDFLAGS is for internal-use.
>> >> >> Please do not override it from the command line.
>> >> >
>> >> > Can we generate a build error if a user try to override LDFLAGS, CFLAGS or
>> >> > other critical internal-use-only variables?
>> >>
>> >> Yes, Make can check where variables came from.
>> >
>> > I think we should do this.
>> >
>> >> >> make LDFLAGS_KERNEL=... LDFLAGS_MODULE=...
>> >> >> will allow you to append linker flags.
>> >> >
>> >> > Okay. It makes me wounder if we should taint kernel in such cases?
>> >> > Custom compiler/linker flags are risky and can lead to weird bugs.
>> >>
>> >> OK.
>> >> So, what problem are we discussing?
>> >
>> > Users set custom LDFLAGS/CFLAGS and break kernel. Then report bug that
>> > hard to debug. See
>> >
>> > https://bugzilla.kernel.org/show_bug.cgi?id=200385
>>
>>
>> CFLAGS is only used under tools/.
>> Passing CFLAGS is probably no effect to the kernel.
>>
>> In Linux makefiles,
>> KBUILD_ prefixed variables are used internally.
>>
>> KBUILD_CFLAGS, KBUILD_CPPFLAGS, KBUILD_AFLAGS, etc.
>>
>>
>> LDFLAGS is an exception. I do not know why.
>> Renaming LDFLAGS to KBUILD_LDFLAGS
>> will make the code consistent.
>>
>> At least, it will avoid overriding flags by accident.
>>
>> Of course, users still can change KBUILD_LDFLAGS
>> if they really want.
>>
>> The build system could add belt and braces checks for that,
>> but it is arguable since
>> there are lots of lots of internal variables.
>
> I think renaming LDFLAGS to KBUILD_LDFLAGS is good idea.
> Would you prepare patch?


Yes, targeting for 4.19-rc1.


--
Best Regards
Masahiro Yamada