I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
to gcc-8 all object files now end up having .eh_frame section. For
vmlinux, that's not a problem, because they all get discarded in
arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
modules, which doesn't seem to be useful - given that everything worked
just fine with gcc-7, and I don't see anything in the module loader that
handles .eh_frame.
The reason I care is that my target has a rather tight rootfs budget,
and the .eh_frame section seem to occupy 10-30% of the file size
(obviously very depending on the particular module).
Comparing the .foo.o.cmd files, I don't see change in options that might
explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
is apparently no longer supported in gcc-8). Both before and after, there's
-fno-dwarf2-cfi-asm
about which gcc's documentation says
'-fno-dwarf2-cfi-asm'
Emit DWARF unwind info as compiler generated '.eh_frame' section
instead of using GAS '.cfi_*' directives.
Looking into where that comes from got me even more confused, because
both arm and unicore32 say
# Never generate .eh_frame
KBUILD_CFLAGS += $(call cc-option,-fno-dwarf2-cfi-asm)
while the ppc32 case at hand says
# FIXME: the module load should be taught about the additional relocs
# generated by this.
# revert to pre-gcc-4.4 behaviour of .eh_frame
but prior to gcc-8, .eh_frame didn't seem to get generated anyway.
Can .eh_frame sections be discarded for modules (on ppc32 at least), or
is there some magic that makes them necessary when building with gcc-8?
Rasmus
On Mon, Mar 02, 2020 at 11:56:05AM +0100, Rasmus Villemoes wrote:
> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
> to gcc-8 all object files now end up having .eh_frame section.
Since GCC 8, we enable -fasynchronous-unwind-tables by default for
PowerPC. See https://gcc.gnu.org/r259298 .
> For
> vmlinux, that's not a problem, because they all get discarded in
> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
> modules, which doesn't seem to be useful - given that everything worked
> just fine with gcc-7, and I don't see anything in the module loader that
> handles .eh_frame.
It is useful for debugging. Not many people debug the kernel like this,
of course.
Segher
Rasmus Villemoes wrote:
> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
> to gcc-8 all object files now end up having .eh_frame section. For
> vmlinux, that's not a problem, because they all get discarded in
> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
> modules, which doesn't seem to be useful - given that everything worked
> just fine with gcc-7, and I don't see anything in the module loader that
> handles .eh_frame.
>
> The reason I care is that my target has a rather tight rootfs budget,
> and the .eh_frame section seem to occupy 10-30% of the file size
> (obviously very depending on the particular module).
>
> Comparing the .foo.o.cmd files, I don't see change in options that might
> explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
> is apparently no longer supported in gcc-8). Both before and after, there's
>
> -fno-dwarf2-cfi-asm
>
> about which gcc's documentation says
>
> '-fno-dwarf2-cfi-asm'
> Emit DWARF unwind info as compiler generated '.eh_frame' section
> instead of using GAS '.cfi_*' directives.
>
> Looking into where that comes from got me even more confused, because
> both arm and unicore32 say
>
> # Never generate .eh_frame
> KBUILD_CFLAGS += $(call cc-option,-fno-dwarf2-cfi-asm)
>
> while the ppc32 case at hand says
>
> # FIXME: the module load should be taught about the additional relocs
> # generated by this.
> # revert to pre-gcc-4.4 behaviour of .eh_frame
Michael opened a task to look into this recently and I had spent some
time last week on this. The original commit/discussion adding
-fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being
handled by our module loader:
http://lkml.kernel.org/r/[email protected]
However, that is now handled thanks to commit 9f751b82b491d:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d
I did a test build and a simple module loaded fine, so I think
-fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen
some breakages with it. Michael?
>
> but prior to gcc-8, .eh_frame didn't seem to get generated anyway.
>
> Can .eh_frame sections be discarded for modules (on ppc32 at least), or
> is there some magic that makes them necessary when building with gcc-8?
As Segher points out, it looks like we need to add
-fno-asynchronous-unwind-tables. Most other architectures seem to use
that too.
- Naveen
Segher Boessenkool wrote:
> On Mon, Mar 02, 2020 at 11:56:05AM +0100, Rasmus Villemoes wrote:
>> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
>> to gcc-8 all object files now end up having .eh_frame section.
>
> Since GCC 8, we enable -fasynchronous-unwind-tables by default for
> PowerPC. See https://gcc.gnu.org/r259298 .
>
>> For
>> vmlinux, that's not a problem, because they all get discarded in
>> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
>> modules, which doesn't seem to be useful - given that everything worked
>> just fine with gcc-7, and I don't see anything in the module loader that
>> handles .eh_frame.
>
> It is useful for debugging. Not many people debug the kernel like this,
> of course.
I'm trying to understand if we need that. Other architectures seems to
pass -fasynchronous-unwind-tables only for the vdso, but disable it for
the kernel build. I suppose we can do the same.
If using -fno-asynchronous-unwind-tables, would crash/perf have
problems?
- Naveen
Naveen N. Rao wrote:
> Rasmus Villemoes wrote:
>> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
>> to gcc-8 all object files now end up having .eh_frame section. For
>> vmlinux, that's not a problem, because they all get discarded in
>> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
>> modules, which doesn't seem to be useful - given that everything worked
>> just fine with gcc-7, and I don't see anything in the module loader that
>> handles .eh_frame.
>>
>> The reason I care is that my target has a rather tight rootfs budget,
>> and the .eh_frame section seem to occupy 10-30% of the file size
>> (obviously very depending on the particular module).
>>
>> Comparing the .foo.o.cmd files, I don't see change in options that might
>> explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
>> is apparently no longer supported in gcc-8). Both before and after, there's
>>
>> -fno-dwarf2-cfi-asm
>>
>> about which gcc's documentation says
>>
>> '-fno-dwarf2-cfi-asm'
>> Emit DWARF unwind info as compiler generated '.eh_frame' section
>> instead of using GAS '.cfi_*' directives.
>>
>> Looking into where that comes from got me even more confused, because
>> both arm and unicore32 say
>>
>> # Never generate .eh_frame
>> KBUILD_CFLAGS += $(call cc-option,-fno-dwarf2-cfi-asm)
>>
>> while the ppc32 case at hand says
>>
>> # FIXME: the module load should be taught about the additional relocs
>> # generated by this.
>> # revert to pre-gcc-4.4 behaviour of .eh_frame
>
> Michael opened a task to look into this recently and I had spent some
> time last week on this. The original commit/discussion adding
> -fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being
> handled by our module loader:
> http://lkml.kernel.org/r/[email protected]
>
> However, that is now handled thanks to commit 9f751b82b491d:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d
>
> I did a test build and a simple module loaded fine, so I think
> -fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen
> some breakages with it. Michael?
>
>>
>> but prior to gcc-8, .eh_frame didn't seem to get generated anyway.
>>
>> Can .eh_frame sections be discarded for modules (on ppc32 at least), or
>> is there some magic that makes them necessary when building with gcc-8?
>
> As Segher points out, it looks like we need to add
> -fno-asynchronous-unwind-tables. Most other architectures seem to use
> that too.
Can you check if the below patch works? I am yet to test this in more
detail, but would be good to know the implications for ppc32.
- Naveen
---
diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile
index f35730548e42..5b5bf98b8217 100644
--- a/arch/powerpc/Makefile
+++ b/arch/powerpc/Makefile
@@ -239,10 +239,7 @@ KBUILD_CFLAGS += $(call cc-option,-mno-vsx)
KBUILD_CFLAGS += $(call cc-option,-mno-spe)
KBUILD_CFLAGS += $(call cc-option,-mspe=no)
-# FIXME: the module load should be taught about the additional relocs
-# generated by this.
-# revert to pre-gcc-4.4 behaviour of .eh_frame
-KBUILD_CFLAGS += $(call cc-option,-fno-dwarf2-cfi-asm)
+KBUILD_CFLAGS += $(call cc-option,-fno-asynchronous-unwind-tables)
# Never use string load/store instructions as they are
# often slow when they are implemented at all
diff --git a/arch/powerpc/kernel/vdso32/Makefile b/arch/powerpc/kernel/vdso32/Makefile
index e147bbdc12cd..d43b0b18137c 100644
--- a/arch/powerpc/kernel/vdso32/Makefile
+++ b/arch/powerpc/kernel/vdso32/Makefile
@@ -25,6 +25,7 @@ KCOV_INSTRUMENT := n
UBSAN_SANITIZE := n
ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
+ -fasynchronous-unwind-tables \
-Wl,-soname=linux-vdso32.so.1 -Wl,--hash-style=both
asflags-y := -D__VDSO32__ -s
diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
index 32ebb3522ea1..b2cbb5c49bad 100644
--- a/arch/powerpc/kernel/vdso64/Makefile
+++ b/arch/powerpc/kernel/vdso64/Makefile
@@ -13,6 +13,7 @@ KCOV_INSTRUMENT := n
UBSAN_SANITIZE := n
ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
+ -fasynchronous-unwind-tables \
-Wl,-soname=linux-vdso64.so.1 -Wl,--hash-style=both
asflags-y := -D__VDSO64__ -s
On 02/03/2020 18.32, Naveen N. Rao wrote:
> Naveen N. Rao wrote:
>> Michael opened a task to look into this recently and I had spent some
>> time last week on this. The original commit/discussion adding
>> -fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being
>> handled by our module loader:
>> http://lkml.kernel.org/r/[email protected]
>>
>> However, that is now handled thanks to commit 9f751b82b491d:
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d
>>
>>
>> I did a test build and a simple module loaded fine, so I think
>> -fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen
>> some breakages with it. Michael?
>>
>>>
>>> but prior to gcc-8, .eh_frame didn't seem to get generated anyway.
>>>
>>> Can .eh_frame sections be discarded for modules (on ppc32 at least), or
>>> is there some magic that makes them necessary when building with gcc-8?
>>
>> As Segher points out, it looks like we need to add
>> -fno-asynchronous-unwind-tables. Most other architectures seem to use
>> that too.
Yes. Thanks, Segher, that explains that part.
> Can you check if the below patch works? I am yet to test this in more
> detail, but would be good to know the implications for ppc32.
I'll see if that produces a bootable kernel, but I think I'd prefer a
more piecemeal approach.
One patch to add -fno-asynchronous-unwind-tables (given that other
arches do it unconditionally I don't think cc-option is needed), with a
commit log saying something like "no-op for gcc < 8, prevents .eh_frame
sections that are discarded anyway for vmlinux and waste disk space for
modules". Then another patch can get rid of -fno-dwarf2-cfi-asm if
that's no longer required.
Rasmus
"Naveen N. Rao" <[email protected]> writes:
> Rasmus Villemoes wrote:
>> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
>> to gcc-8 all object files now end up having .eh_frame section. For
>> vmlinux, that's not a problem, because they all get discarded in
>> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
>> modules, which doesn't seem to be useful - given that everything worked
>> just fine with gcc-7, and I don't see anything in the module loader that
>> handles .eh_frame.
>>
>> The reason I care is that my target has a rather tight rootfs budget,
>> and the .eh_frame section seem to occupy 10-30% of the file size
>> (obviously very depending on the particular module).
>>
>> Comparing the .foo.o.cmd files, I don't see change in options that might
>> explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
>> is apparently no longer supported in gcc-8). Both before and after, there's
>>
>> -fno-dwarf2-cfi-asm
>>
>> about which gcc's documentation says
>>
>> '-fno-dwarf2-cfi-asm'
>> Emit DWARF unwind info as compiler generated '.eh_frame' section
>> instead of using GAS '.cfi_*' directives.
>>
>> Looking into where that comes from got me even more confused, because
>> both arm and unicore32 say
>>
>> # Never generate .eh_frame
>> KBUILD_CFLAGS += $(call cc-option,-fno-dwarf2-cfi-asm)
>>
>> while the ppc32 case at hand says
>>
>> # FIXME: the module load should be taught about the additional relocs
>> # generated by this.
>> # revert to pre-gcc-4.4 behaviour of .eh_frame
>
> Michael opened a task to look into this recently and I had spent some
> time last week on this. The original commit/discussion adding
> -fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being
> handled by our module loader:
> http://lkml.kernel.org/r/[email protected]
I opened that issue purely based on noticing the wart in the Makefile,
not because I'd actually tested it.
> However, that is now handled thanks to commit 9f751b82b491d:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d
Haha, written by me, what an idiot.
So the Makefile hack can presumably be dropped, because the module
loader can handle the relocations.
And then maybe we also want to turn off the unwind tables, but that
would be a separate patch.
> I did a test build and a simple module loaded fine, so I think
> -fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen
> some breakages with it. Michael?
No, as I said above it was just reading the Makefile.
cheers
Michael Ellerman wrote:
> "Naveen N. Rao" <[email protected]> writes:
>> Rasmus Villemoes wrote:
>>> I'm building a ppc32 kernel, and noticed that after upgrading from gcc-7
>>> to gcc-8 all object files now end up having .eh_frame section. For
>>> vmlinux, that's not a problem, because they all get discarded in
>>> arch/powerpc/kernel/vmlinux.lds.S . However, they stick around in
>>> modules, which doesn't seem to be useful - given that everything worked
>>> just fine with gcc-7, and I don't see anything in the module loader that
>>> handles .eh_frame.
>>>
>>> The reason I care is that my target has a rather tight rootfs budget,
>>> and the .eh_frame section seem to occupy 10-30% of the file size
>>> (obviously very depending on the particular module).
>>>
>>> Comparing the .foo.o.cmd files, I don't see change in options that might
>>> explain this (there's a bunch of new -Wno-*, and the -mspe=no spelling
>>> is apparently no longer supported in gcc-8). Both before and after, there's
>>>
>>> -fno-dwarf2-cfi-asm
>>>
>>> about which gcc's documentation says
>>>
>>> '-fno-dwarf2-cfi-asm'
>>> Emit DWARF unwind info as compiler generated '.eh_frame' section
>>> instead of using GAS '.cfi_*' directives.
>>>
>>> Looking into where that comes from got me even more confused, because
>>> both arm and unicore32 say
>>>
>>> # Never generate .eh_frame
>>> KBUILD_CFLAGS += $(call cc-option,-fno-dwarf2-cfi-asm)
>>>
>>> while the ppc32 case at hand says
>>>
>>> # FIXME: the module load should be taught about the additional relocs
>>> # generated by this.
>>> # revert to pre-gcc-4.4 behaviour of .eh_frame
>>
>> Michael opened a task to look into this recently and I had spent some
>> time last week on this. The original commit/discussion adding
>> -fno-dwarf2-cfi-asm refers to R_PPC64_REL32 relocations not being
>> handled by our module loader:
>> http://lkml.kernel.org/r/[email protected]
>
> I opened that issue purely based on noticing the wart in the Makefile,
> not because I'd actually tested it.
>
>> However, that is now handled thanks to commit 9f751b82b491d:
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f751b82b491d
>
> Haha, written by me, what an idiot.
>
> So the Makefile hack can presumably be dropped, because the module
> loader can handle the relocations.
>
> And then maybe we also want to turn off the unwind tables, but that
> would be a separate patch.
>
>> I did a test build and a simple module loaded fine, so I think
>> -fno-dwarf2-cfi-asm is not required anymore, unless Michael has seen
>> some breakages with it. Michael?
>
> No, as I said above it was just reading the Makefile.
Ok, thanks for clarifying. To test, I did 'allmodconfig' builds across
three environments:
- gcc (Ubuntu 9.2.1-9ubuntu2) 9.2.1 20191008 -- ppc64le
- gcc (SUSE Linux) 7.5.0 -- ppc64le
- gcc (GCC) 8.2.1 20181215 (Red Hat 8.2.1-6) -- ppc64 (BE)
Then, used the below command to list all relocations in the modules:
$ find . -name '*.ko' | xargs -n 1 readelf -Wr | grep -v "Relocation " | grep -v "Offset " | cut -d' ' -f4 | sort | uniq
R_PPC64_ADDR32
R_PPC64_ADDR64
R_PPC64_ENTRY
R_PPC64_REL24
R_PPC64_REL32
R_PPC64_REL64
R_PPC64_TOC
R_PPC64_TOC16_HA
R_PPC64_TOC16_LO
R_PPC64_TOC16_LO_DS
All three environments show up similar set of relocations, all of which
we handle in the module loader today.
If Rasmus/Christophe can confirm that this is true for ppc32 as well,
then we should be fine.
- Naveen
Naveen N. Rao wrote:
> Naveen N. Rao wrote:
>> Rasmus Villemoes wrote:
<snip>
> Can you check if the below patch works? I am yet to test this in more
> detail, but would be good to know the implications for ppc32.
>
> - Naveen
>
>
> ---
> diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile
> index f35730548e42..5b5bf98b8217 100644
> --- a/arch/powerpc/Makefile
> +++ b/arch/powerpc/Makefile
> @@ -239,10 +239,7 @@ KBUILD_CFLAGS += $(call cc-option,-mno-vsx)
> KBUILD_CFLAGS += $(call cc-option,-mno-spe)
> KBUILD_CFLAGS += $(call cc-option,-mspe=no)
>
> -# FIXME: the module load should be taught about the additional relocs
> -# generated by this.
> -# revert to pre-gcc-4.4 behaviour of .eh_frame
> -KBUILD_CFLAGS += $(call cc-option,-fno-dwarf2-cfi-asm)
> +KBUILD_CFLAGS += $(call cc-option,-fno-asynchronous-unwind-tables)
In terms of the CFI information, the primary difference I see with
-fno-dwarf2-cfi-asm is that when dumping the debug frames, CIE indicates
version 3, while otherwise (i.e., without -fno-dwarf2-cfi-asm and
with/without -fasynchronous-unwind-tables), it is version 1, regardless
of -gdwarf-2/-gdwarf-4. There are few more minor changes, but none of
these looked significant to me.
>
> # Never use string load/store instructions as they are
> # often slow when they are implemented at all
> diff --git a/arch/powerpc/kernel/vdso32/Makefile b/arch/powerpc/kernel/vdso32/Makefile
> index e147bbdc12cd..d43b0b18137c 100644
> --- a/arch/powerpc/kernel/vdso32/Makefile
> +++ b/arch/powerpc/kernel/vdso32/Makefile
> @@ -25,6 +25,7 @@ KCOV_INSTRUMENT := n
> UBSAN_SANITIZE := n
>
> ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
> + -fasynchronous-unwind-tables \
> -Wl,-soname=linux-vdso32.so.1 -Wl,--hash-style=both
> asflags-y := -D__VDSO32__ -s
>
> diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
> index 32ebb3522ea1..b2cbb5c49bad 100644
> --- a/arch/powerpc/kernel/vdso64/Makefile
> +++ b/arch/powerpc/kernel/vdso64/Makefile
> @@ -13,6 +13,7 @@ KCOV_INSTRUMENT := n
> UBSAN_SANITIZE := n
>
> ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
> + -fasynchronous-unwind-tables \
> -Wl,-soname=linux-vdso64.so.1 -Wl,--hash-style=both
> asflags-y := -D__VDSO64__ -s
The above vdso hunks can be dropped since all our VDSO are assembly, so
the above have no impact.
- Naveen