Hi,
With this set of patches, it becomes possible to build a LTO'd kernel
with LLVM/Clang on MIPS.
This was tested on a Ingenic JZ4770 based system. It requires the
linking step of the vmlinuz.bin to be done with binutils' LD instead of
LLVM's ld.lld [1], but the vmlinuz ELF itself can be completely built
with LLVM.
Strangely, the LTO'd kernel is bigger in size (3.6 MiB vs. 3.1 MiB
without LTO), which might be completely normal and I just have wrong
expectations.
Cheers,
-Paul
[1]: https://github.com/ClangBuiltLinux/linux/issues/1333
Paul Cercueil (3):
MIPS: boot/compressed: Disable abicalls
MIPS: boot/compressed: Build without LTO
MIPS: Add support for LTO
arch/mips/Kconfig | 1 +
arch/mips/boot/compressed/Makefile | 7 +++++--
2 files changed, 6 insertions(+), 2 deletions(-)
--
2.33.0
Avoid complaints from Clang/LLVM by building the decompress program with
-mno-abicalls.
Signed-off-by: Paul Cercueil <[email protected]>
---
arch/mips/boot/compressed/Makefile | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
index f27cf31b4140..4c9ecfbb0ef4 100644
--- a/arch/mips/boot/compressed/Makefile
+++ b/arch/mips/boot/compressed/Makefile
@@ -27,10 +27,10 @@ ifdef CONFIG_CPU_LOONGSON64
KBUILD_CFLAGS := $(filter-out -march=loongson3a, $(KBUILD_CFLAGS)) -march=mips64r2
endif
-KBUILD_CFLAGS := $(KBUILD_CFLAGS) -D__KERNEL__ -D__DISABLE_EXPORTS \
+KBUILD_CFLAGS := $(KBUILD_CFLAGS) -mno-abicalls -D__KERNEL__ -D__DISABLE_EXPORTS \
-DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) -D"VMLINUX_LOAD_ADDRESS_ULL=$(VMLINUX_LOAD_ADDRESS)ull"
-KBUILD_AFLAGS := $(KBUILD_AFLAGS) -D__ASSEMBLY__ \
+KBUILD_AFLAGS := $(KBUILD_AFLAGS) -mno-abicalls -D__ASSEMBLY__ \
-DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) \
-DKERNEL_ENTRY=$(VMLINUX_ENTRY_ADDRESS)
--
2.33.0
We need a valid ELF object for dummy.o, so that objcopy can insert the
vmlinuz payload.
Therefore, we must build the decompresser program without LTO, otherwise
dummy.o will be LLVM bytecode instead of a ELF object file.
Building the decompresser with LTO wouldn't make much sense anyway,
unlike building the vmlinuz itself with LTO.
Signed-off-by: Paul Cercueil <[email protected]>
---
arch/mips/boot/compressed/Makefile | 3 +++
1 file changed, 3 insertions(+)
diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
index 4c9ecfbb0ef4..2d01c50fb0b1 100644
--- a/arch/mips/boot/compressed/Makefile
+++ b/arch/mips/boot/compressed/Makefile
@@ -27,6 +27,9 @@ ifdef CONFIG_CPU_LOONGSON64
KBUILD_CFLAGS := $(filter-out -march=loongson3a, $(KBUILD_CFLAGS)) -march=mips64r2
endif
+# Disable LTO
+KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS))
+
KBUILD_CFLAGS := $(KBUILD_CFLAGS) -mno-abicalls -D__KERNEL__ -D__DISABLE_EXPORTS \
-DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) -D"VMLINUX_LOAD_ADDRESS_ULL=$(VMLINUX_LOAD_ADDRESS)ull"
--
2.33.0
Allow CONFIG_LTO_CLANG to be enabled. The ThinLTO variant is not yet
supported.
While this option allows to build a LTO'd kernel, the result kernel file
ends up being *bigger* than the non-LTO variant (about 3.6 MiB with LTO
vs. 3.1 MiB without with a ZSTD-compressed kernel).
Signed-off-by: Paul Cercueil <[email protected]>
---
arch/mips/Kconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 0215dc1529e9..6987db8d5f64 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -22,6 +22,7 @@ config MIPS
select ARCH_USE_QUEUED_RWLOCKS
select ARCH_USE_QUEUED_SPINLOCKS
select ARCH_SUPPORTS_HUGETLBFS if CPU_SUPPORTS_HUGEPAGES
+ select ARCH_SUPPORTS_LTO_CLANG if CPU_LITTLE_ENDIAN
select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
select ARCH_WANT_IPC_PARSE_VERSION
select ARCH_WANT_LD_ORPHAN_WARN
--
2.33.0
On Mon, Dec 13, 2021 at 10:49:12PM +0000, Paul Cercueil wrote:
> Avoid complaints from Clang/LLVM by building the decompress program with
> -mno-abicalls.
What is the message?
> Signed-off-by: Paul Cercueil <[email protected]>
> ---
> arch/mips/boot/compressed/Makefile | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
> index f27cf31b4140..4c9ecfbb0ef4 100644
> --- a/arch/mips/boot/compressed/Makefile
> +++ b/arch/mips/boot/compressed/Makefile
> @@ -27,10 +27,10 @@ ifdef CONFIG_CPU_LOONGSON64
> KBUILD_CFLAGS := $(filter-out -march=loongson3a, $(KBUILD_CFLAGS)) -march=mips64r2
> endif
>
> -KBUILD_CFLAGS := $(KBUILD_CFLAGS) -D__KERNEL__ -D__DISABLE_EXPORTS \
> +KBUILD_CFLAGS := $(KBUILD_CFLAGS) -mno-abicalls -D__KERNEL__ -D__DISABLE_EXPORTS \
> -DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) -D"VMLINUX_LOAD_ADDRESS_ULL=$(VMLINUX_LOAD_ADDRESS)ull"
>
> -KBUILD_AFLAGS := $(KBUILD_AFLAGS) -D__ASSEMBLY__ \
> +KBUILD_AFLAGS := $(KBUILD_AFLAGS) -mno-abicalls -D__ASSEMBLY__ \
> -DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) \
> -DKERNEL_ENTRY=$(VMLINUX_ENTRY_ADDRESS)
>
> --
> 2.33.0
>
>
On Mon, Dec 13, 2021 at 10:49:13PM +0000, Paul Cercueil wrote:
> We need a valid ELF object for dummy.o, so that objcopy can insert the
> vmlinuz payload.
>
> Therefore, we must build the decompresser program without LTO, otherwise
> dummy.o will be LLVM bytecode instead of a ELF object file.
>
> Building the decompresser with LTO wouldn't make much sense anyway,
> unlike building the vmlinuz itself with LTO.
>
> Signed-off-by: Paul Cercueil <[email protected]>
Seems reasonable.
Reviewed-by: Nathan Chancellor <[email protected]>
> ---
> arch/mips/boot/compressed/Makefile | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
> index 4c9ecfbb0ef4..2d01c50fb0b1 100644
> --- a/arch/mips/boot/compressed/Makefile
> +++ b/arch/mips/boot/compressed/Makefile
> @@ -27,6 +27,9 @@ ifdef CONFIG_CPU_LOONGSON64
> KBUILD_CFLAGS := $(filter-out -march=loongson3a, $(KBUILD_CFLAGS)) -march=mips64r2
> endif
>
> +# Disable LTO
> +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS))
> +
> KBUILD_CFLAGS := $(KBUILD_CFLAGS) -mno-abicalls -D__KERNEL__ -D__DISABLE_EXPORTS \
> -DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) -D"VMLINUX_LOAD_ADDRESS_ULL=$(VMLINUX_LOAD_ADDRESS)ull"
>
> --
> 2.33.0
>
>
On Mon, Dec 13, 2021 at 10:49:14PM +0000, Paul Cercueil wrote:
> Allow CONFIG_LTO_CLANG to be enabled. The ThinLTO variant is not yet
> supported.
>
> While this option allows to build a LTO'd kernel, the result kernel file
> ends up being *bigger* than the non-LTO variant (about 3.6 MiB with LTO
> vs. 3.1 MiB without with a ZSTD-compressed kernel).
I believe this happens because the linker is able to inline more, which
should make the code faster but makes it larger. I believe we limit the
inlining threshold so that it is not too bad but maybe that could be
further tuned.
> Signed-off-by: Paul Cercueil <[email protected]>
Reviewed-by: Nathan Chancellor <[email protected]>
> ---
> arch/mips/Kconfig | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
> index 0215dc1529e9..6987db8d5f64 100644
> --- a/arch/mips/Kconfig
> +++ b/arch/mips/Kconfig
> @@ -22,6 +22,7 @@ config MIPS
> select ARCH_USE_QUEUED_RWLOCKS
> select ARCH_USE_QUEUED_SPINLOCKS
> select ARCH_SUPPORTS_HUGETLBFS if CPU_SUPPORTS_HUGEPAGES
> + select ARCH_SUPPORTS_LTO_CLANG if CPU_LITTLE_ENDIAN
> select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
> select ARCH_WANT_IPC_PARSE_VERSION
> select ARCH_WANT_LD_ORPHAN_WARN
> --
> 2.33.0
>
>
On Mon, Dec 13, 2021 at 10:49:11PM +0000, Paul Cercueil wrote:
> Hi,
>
> With this set of patches, it becomes possible to build a LTO'd kernel
> with LLVM/Clang on MIPS.
>
> This was tested on a Ingenic JZ4770 based system. It requires the
> linking step of the vmlinuz.bin to be done with binutils' LD instead of
> LLVM's ld.lld [1], but the vmlinuz ELF itself can be completely built
> with LLVM.
The patches seem fine to me but it might be nice to have this fixed up
before the series is merged so that everything just works when this
configuration is selected. I have added some thoughts on the GitHub
issue.
> Strangely, the LTO'd kernel is bigger in size (3.6 MiB vs. 3.1 MiB
> without LTO), which might be completely normal and I just have wrong
> expectations.
>
> Cheers,
> -Paul
>
> [1]: https://github.com/ClangBuiltLinux/linux/issues/1333
>
> Paul Cercueil (3):
> MIPS: boot/compressed: Disable abicalls
> MIPS: boot/compressed: Build without LTO
> MIPS: Add support for LTO
>
> arch/mips/Kconfig | 1 +
> arch/mips/boot/compressed/Makefile | 7 +++++--
> 2 files changed, 6 insertions(+), 2 deletions(-)
>
> --
> 2.33.0
>
>
Hi Nathan,
Le mar., d?c. 14 2021 at 11:06:57 -0700, Nathan Chancellor
<[email protected]> a ?crit :
> On Mon, Dec 13, 2021 at 10:49:12PM +0000, Paul Cercueil wrote:
>> Avoid complaints from Clang/LLVM by building the decompress program
>> with
>> -mno-abicalls.
>
> What is the message?
ld.lld: warning: lto.tmp: linking abicalls code with non-abicalls code
arch/mips/kernel/head.o
Cheers,
-Paul
>
>> Signed-off-by: Paul Cercueil <[email protected]>
>> ---
>> arch/mips/boot/compressed/Makefile | 4 ++--
>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/mips/boot/compressed/Makefile
>> b/arch/mips/boot/compressed/Makefile
>> index f27cf31b4140..4c9ecfbb0ef4 100644
>> --- a/arch/mips/boot/compressed/Makefile
>> +++ b/arch/mips/boot/compressed/Makefile
>> @@ -27,10 +27,10 @@ ifdef CONFIG_CPU_LOONGSON64
>> KBUILD_CFLAGS := $(filter-out -march=loongson3a, $(KBUILD_CFLAGS))
>> -march=mips64r2
>> endif
>>
>> -KBUILD_CFLAGS := $(KBUILD_CFLAGS) -D__KERNEL__ -D__DISABLE_EXPORTS
>> \
>> +KBUILD_CFLAGS := $(KBUILD_CFLAGS) -mno-abicalls -D__KERNEL__
>> -D__DISABLE_EXPORTS \
>> -DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE)
>> -D"VMLINUX_LOAD_ADDRESS_ULL=$(VMLINUX_LOAD_ADDRESS)ull"
>>
>> -KBUILD_AFLAGS := $(KBUILD_AFLAGS) -D__ASSEMBLY__ \
>> +KBUILD_AFLAGS := $(KBUILD_AFLAGS) -mno-abicalls -D__ASSEMBLY__ \
>> -DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) \
>> -DKERNEL_ENTRY=$(VMLINUX_ENTRY_ADDRESS)
>>
>> --
>> 2.33.0
>>
>>
On Mon, Dec 13, 2021 at 2:49 PM Paul Cercueil <[email protected]> wrote:
>
> Hi,
>
> With this set of patches, it becomes possible to build a LTO'd kernel
> with LLVM/Clang on MIPS.
Paul,
Overall, nice work pursuing this and working through all the related
issues; thanks for the series!
We should clarify whether this is LTO_CLANG_THIN or LTO_CLANG_FULL.
Getting support for either would be nice, but for each architecture
has had its own distinct challenges.
>
> This was tested on a Ingenic JZ4770 based system. It requires the
> linking step of the vmlinuz.bin to be done with binutils' LD instead of
> LLVM's ld.lld [1], but the vmlinuz ELF itself can be completely built
> with LLVM.
eh, this patch set sounds incomplete then; Can we wait to resolve
https://github.com/ClangBuiltLinux/linux/issues/1333 BEFORE this
series? Hacking up arch/mips/boot/compressed/Makefile to force a
different linker is not something we should encourage.
>
> Strangely, the LTO'd kernel is bigger in size (3.6 MiB vs. 3.1 MiB
> without LTO), which might be completely normal and I just have wrong
> expectations.
Are you building with CC_OPTIMIZE_FOR_SIZE or CC_OPTIMIZE_FOR_SIZE?
LTO != LD_DEAD_CODE_DATA_ELIMINATION
>
> Cheers,
> -Paul
>
> [1]: https://github.com/ClangBuiltLinux/linux/issues/1333
>
> Paul Cercueil (3):
> MIPS: boot/compressed: Disable abicalls
> MIPS: boot/compressed: Build without LTO
> MIPS: Add support for LTO
>
> arch/mips/Kconfig | 1 +
> arch/mips/boot/compressed/Makefile | 7 +++++--
> 2 files changed, 6 insertions(+), 2 deletions(-)
>
> --
> 2.33.0
>
--
Thanks,
~Nick Desaulniers
On Tue, Dec 14, 2021 at 11:09 AM Paul Cercueil <[email protected]> wrote:
>
> Hi Nathan,
>
> Le mar., déc. 14 2021 at 11:06:57 -0700, Nathan Chancellor
> <[email protected]> a écrit :
> > On Mon, Dec 13, 2021 at 10:49:12PM +0000, Paul Cercueil wrote:
> >> Avoid complaints from Clang/LLVM by building the decompress program
> >> with
> >> -mno-abicalls.
> >
> > What is the message?
>
> ld.lld: warning: lto.tmp: linking abicalls code with non-abicalls code
> arch/mips/kernel/head.o
That might be good to put in the commit message, as well as a link to
the corresponding issue in our issue tracker.
Though, I'm unfamiliar with that flag; removing it may allow us to
link, but it may cause problems at runtime. It sounds like clang
supports the flag, so I have a sinking feeling this is a flag that LTO
is dropping by accident, which we've seen in the past. Please file a
bug in the issue tracker and let's follow up there?
>
> Cheers,
> -Paul
>
> >
> >> Signed-off-by: Paul Cercueil <[email protected]>
> >> ---
> >> arch/mips/boot/compressed/Makefile | 4 ++--
> >> 1 file changed, 2 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/arch/mips/boot/compressed/Makefile
> >> b/arch/mips/boot/compressed/Makefile
> >> index f27cf31b4140..4c9ecfbb0ef4 100644
> >> --- a/arch/mips/boot/compressed/Makefile
> >> +++ b/arch/mips/boot/compressed/Makefile
> >> @@ -27,10 +27,10 @@ ifdef CONFIG_CPU_LOONGSON64
> >> KBUILD_CFLAGS := $(filter-out -march=loongson3a, $(KBUILD_CFLAGS))
> >> -march=mips64r2
> >> endif
> >>
> >> -KBUILD_CFLAGS := $(KBUILD_CFLAGS) -D__KERNEL__ -D__DISABLE_EXPORTS
> >> \
> >> +KBUILD_CFLAGS := $(KBUILD_CFLAGS) -mno-abicalls -D__KERNEL__
> >> -D__DISABLE_EXPORTS \
> >> -DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE)
> >> -D"VMLINUX_LOAD_ADDRESS_ULL=$(VMLINUX_LOAD_ADDRESS)ull"
> >>
> >> -KBUILD_AFLAGS := $(KBUILD_AFLAGS) -D__ASSEMBLY__ \
> >> +KBUILD_AFLAGS := $(KBUILD_AFLAGS) -mno-abicalls -D__ASSEMBLY__ \
> >> -DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) \
> >> -DKERNEL_ENTRY=$(VMLINUX_ENTRY_ADDRESS)
> >>
> >> --
> >> 2.33.0
> >>
> >>
>
>
--
Thanks,
~Nick Desaulniers
On Mon, Dec 13, 2021 at 2:49 PM Paul Cercueil <[email protected]> wrote:
>
> We need a valid ELF object for dummy.o, so that objcopy can insert the
> vmlinuz payload.
>
> Therefore, we must build the decompresser program without LTO, otherwise
> dummy.o will be LLVM bytecode instead of a ELF object file.
While the below may work and follows the pattern observed in
drivers/firmware/efi/libstub/Makefile
43:KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS))
I can't help but wonder whether we should instead be doing something like:
arch/x86/power/Makefile
9:CFLAGS_REMOVE_cpu.o := $(CC_FLAGS_LTO)
scripts/mod/Makefile
3:CFLAGS_REMOVE_empty.o += $(CC_FLAGS_LTO)
drivers/misc/lkdtm/Makefile
19:CFLAGS_REMOVE_rodata.o += $(CC_FLAGS_LTO)
but for dummy.o? ie.
CFLAGS_REMOVE_dummy.o += $(CC_FLAGS_LTO)
Can you please try that rather than disabling LTO for a large subset of files?
>
> Building the decompresser with LTO wouldn't make much sense anyway,
> unlike building the vmlinuz itself with LTO.
Why not? We do it for x86 and arm64 AFAICT.
>
> Signed-off-by: Paul Cercueil <[email protected]>
> ---
> arch/mips/boot/compressed/Makefile | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
> index 4c9ecfbb0ef4..2d01c50fb0b1 100644
> --- a/arch/mips/boot/compressed/Makefile
> +++ b/arch/mips/boot/compressed/Makefile
> @@ -27,6 +27,9 @@ ifdef CONFIG_CPU_LOONGSON64
> KBUILD_CFLAGS := $(filter-out -march=loongson3a, $(KBUILD_CFLAGS)) -march=mips64r2
> endif
>
> +# Disable LTO
> +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS))
> +
> KBUILD_CFLAGS := $(KBUILD_CFLAGS) -mno-abicalls -D__KERNEL__ -D__DISABLE_EXPORTS \
> -DBOOT_HEAP_SIZE=$(BOOT_HEAP_SIZE) -D"VMLINUX_LOAD_ADDRESS_ULL=$(VMLINUX_LOAD_ADDRESS)ull"
>
> --
> 2.33.0
>
--
Thanks,
~Nick Desaulniers
On Mon, Dec 13, 2021 at 2:49 PM Paul Cercueil <[email protected]> wrote:
>
> Allow CONFIG_LTO_CLANG to be enabled. The ThinLTO variant is not yet
> supported.
>
> While this option allows to build a LTO'd kernel, the result kernel file
> ends up being *bigger* than the non-LTO variant (about 3.6 MiB with LTO
> vs. 3.1 MiB without with a ZSTD-compressed kernel).
LGTM but I think there's more work to be done first based on my
comments on the earlier patches.
In particular, we should be able to boot the result and not have to
modify any Makefiles outside of the series to get it to link.
Good to know we're close though! Don't give up!
>
> Signed-off-by: Paul Cercueil <[email protected]>
> ---
> arch/mips/Kconfig | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
> index 0215dc1529e9..6987db8d5f64 100644
> --- a/arch/mips/Kconfig
> +++ b/arch/mips/Kconfig
> @@ -22,6 +22,7 @@ config MIPS
> select ARCH_USE_QUEUED_RWLOCKS
> select ARCH_USE_QUEUED_SPINLOCKS
> select ARCH_SUPPORTS_HUGETLBFS if CPU_SUPPORTS_HUGEPAGES
> + select ARCH_SUPPORTS_LTO_CLANG if CPU_LITTLE_ENDIAN
> select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
> select ARCH_WANT_IPC_PARSE_VERSION
> select ARCH_WANT_LD_ORPHAN_WARN
> --
> 2.33.0
>
--
Thanks,
~Nick Desaulniers