2022-09-25 00:27:03

by Mikhail Gavrilov

[permalink] [raw]
Subject: Unable bisect issue because kernel not building from old commits

Hi!
I want to make bisect between commits fdaf9a5840ac and babf0bb978e3,
because after commit babf0bb978e3 Wi-Fi (mt7921e) upload speed became
limited to 170Mbps. On commit fdaf9a5840ac upload speed was 784Mbps.

But I ran into a problem that the gcc-12.2.1 compiler on my
distribution (Fedora) can't build the kernel from old commits. The
build fails with the following error:
ld: warning: arch/x86/power/hibernate_asm_64.o: missing
.note.GNU-stack section implies executable stack
ld: NOTE: This behaviour is deprecated and will be removed in a future
version of the linker
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo
GEN modules.builtin
CC .vmlinux.export.o
LD .tmp_vmlinux.btf
ld: warning: arch/x86/power/hibernate_asm_64.o: missing
.note.GNU-stack section implies executable stack
ld: NOTE: This behaviour is deprecated and will be removed in a future
version of the linker
ld: warning: .tmp_vmlinux.btf has a LOAD segment with RWX permissions
BTF .btf.vmlinux.bin.o
LD .tmp_vmlinux.kallsyms1
ld: warning: .btf.vmlinux.bin.o: missing .note.GNU-stack section
implies executable stack
ld: NOTE: This behaviour is deprecated and will be removed in a future
version of the linker
ld: warning: .tmp_vmlinux.kallsyms1 has a LOAD segment with RWX permissions
KSYMS .tmp_vmlinux.kallsyms1.S
AS .tmp_vmlinux.kallsyms1.S
LD .tmp_vmlinux.kallsyms2
ld: warning: .btf.vmlinux.bin.o: missing .note.GNU-stack section
implies executable stack
ld: NOTE: This behaviour is deprecated and will be removed in a future
version of the linker
ld: warning: .tmp_vmlinux.kallsyms2 has a LOAD segment with RWX permissions
KSYMS .tmp_vmlinux.kallsyms2.S
AS .tmp_vmlinux.kallsyms2.S
LD vmlinux
ld: warning: .btf.vmlinux.bin.o: missing .note.GNU-stack section
implies executable stack
ld: NOTE: This behaviour is deprecated and will be removed in a future
version of the linker
ld: warning: vmlinux has a LOAD segment with RWX permissions
BTFIDS vmlinux
FAILED: load BTF from vmlinux: Invalid argument
make: *** [Makefile:1160: vmlinux] Error 255
make: *** Deleting file 'vmlinux'


I found a commit that fixes the build:
dffbbdc2d9889670c30e07d05fc0dd712e8ad430 is the first bad commit
commit dffbbdc2d9889670c30e07d05fc0dd712e8ad430
Author: Yonghong Song <[email protected]>
Date: Mon Jun 6 23:26:21 2022 -0700

libbpf: Add enum64 parsing and new enum64 public API

Add enum64 parsing support and two new enum64 public APIs:
btf__add_enum64
btf__add_enum64_value

Also add support of signedness for BTF_KIND_ENUM. The
BTF_KIND_ENUM API signatures are not changed. The signedness
will be changed from unsigned to signed if btf__add_enum_value()
finds any negative values.

Acked-by: Andrii Nakryiko <[email protected]>
Signed-off-by: Yonghong Song <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Alexei Starovoitov <[email protected]>

tools/lib/bpf/btf.c | 103 +++++++++++++++++++++++++++++++++++++++++++++++
tools/lib/bpf/btf.h | 12 ++++++
tools/lib/bpf/libbpf.map | 2 +
3 files changed, 117 insertions(+)


But I can't cherry pick it because there are conflicts:
$ git bisect start
status: waiting for both good and bad commits
$ git bisect good fdaf9a5840ac
status: waiting for bad commit, 1 good commit known
$ git bisect bad babf0bb978e3
Bisecting: 2687 revisions left to test after this (roughly 11 steps)
[86c87bea6b42100c67418af690919c44de6ede6e] Merge tag
'devicetree-for-5.19' of
git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux
$ git cherry-pick -n dffbbdc2d9889670c30e07d05fc0dd712e8ad430
Auto-merging tools/lib/bpf/btf.c
CONFLICT (content): Merge conflict in tools/lib/bpf/btf.c
Auto-merging tools/lib/bpf/libbpf.map
CONFLICT (content): Merge conflict in tools/lib/bpf/libbpf.map
error: could not apply dffbbdc2d988... libbpf: Add enum64 parsing and
new enum64 public API
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'

Are there any other options to fix the kernel build?

Thanks.


--
Best Regards,
Mike Gavrilov.


2022-09-25 04:57:15

by Willy Tarreau

[permalink] [raw]
Subject: Re: Unable bisect issue because kernel not building from old commits

On Sun, Sep 25, 2022 at 03:22:10AM +0500, Mikhail Gavrilov wrote:
> Hi!
> I want to make bisect between commits fdaf9a5840ac and babf0bb978e3,
> because after commit babf0bb978e3 Wi-Fi (mt7921e) upload speed became
> limited to 170Mbps. On commit fdaf9a5840ac upload speed was 784Mbps.
>
> But I ran into a problem that the gcc-12.2.1 compiler on my
> distribution (Fedora) can't build the kernel from old commits. The
> build fails with the following error:
> ld: warning: arch/x86/power/hibernate_asm_64.o: missing
> .note.GNU-stack section implies executable stack
(...)

Note that this looks more related to binutils here. Regardless, there
are pre-built toolchains including compiler+binutils for various gcc
versions from 4.9 and up here, for all supported architectures:

https://mirrors.edge.kernel.org/pub/tools/crosstool/

These ones are sufficient to build a kernel and are likely easier to
deal with than trying to port a patch over a bisect session. For example
you could have luck with 7.5 which has been supported for a very long
time and still is supported.

Hoping this helps,
Willy

2022-09-25 14:32:24

by Mikhail Gavrilov

[permalink] [raw]
Subject: Re: Unable bisect issue because kernel not building from old commits

>
> Note that this looks more related to binutils here. Regardless, there
> are pre-built toolchains including compiler+binutils for various gcc
> versions from 4.9 and up here, for all supported architectures:
>
> https://mirrors.edge.kernel.org/pub/tools/crosstool/
>
> These ones are sufficient to build a kernel and are likely easier to
> deal with than trying to port a patch over a bisect session. For example
> you could have luck with 7.5 which has been supported for a very long
> time and still is supported.

I don't quite understand how to switch to the downloaded binutils
without breaking the distribution.
I am building the kernel with the following command:
$ make clean && make -j32 bzImage && make -j32 modules

Thanks.

--
Best Regards,
Mike Gavrilov.

2022-09-25 15:33:46

by Willy Tarreau

[permalink] [raw]
Subject: Re: Unable bisect issue because kernel not building from old commits

On Sun, Sep 25, 2022 at 07:02:39PM +0500, Mikhail Gavrilov wrote:
> >
> > Note that this looks more related to binutils here. Regardless, there
> > are pre-built toolchains including compiler+binutils for various gcc
> > versions from 4.9 and up here, for all supported architectures:
> >
> > https://mirrors.edge.kernel.org/pub/tools/crosstool/
> >
> > These ones are sufficient to build a kernel and are likely easier to
> > deal with than trying to port a patch over a bisect session. For example
> > you could have luck with 7.5 which has been supported for a very long
> > time and still is supported.
>
> I don't quite understand how to switch to the downloaded binutils
> without breaking the distribution.
> I am building the kernel with the following command:
> $ make clean && make -j32 bzImage && make -j32 modules

Just add:

CROSS_COMPILE=/path/to/toolchain/bin/prefix-

to your make command line and it will be fine.

The makefile will append {gcc,ld,...} to this CROSS_COMPILE prefix to
construct the full pathname to the binaries.

Willy

2022-09-25 16:40:53

by Mikhail Gavrilov

[permalink] [raw]
Subject: Re: Unable bisect issue because kernel not building from old commits

On Sun, Sep 25, 2022 at 8:21 PM Willy Tarreau <[email protected]> wrote:
> Just add:
>
> CROSS_COMPILE=/path/to/toolchain/bin/prefix-
>
> to your make command line and it will be fine.
>
> The makefile will append {gcc,ld,...} to this CROSS_COMPILE prefix to
> construct the full pathname to the binaries.

It works!
But I hit another build issue.

$ make clean &&
CROSS_COMPILE=~/Downloads/x86_64-gcc-7.5.0-nolibc-x86_64-linux/gcc-7.5.0-nolibc/x86_64-linux/bin/x86_64-linux-
make -j32 bzImage
***
GEN modules.builtin
LD .tmp_vmlinux.btf
BTF .btf.vmlinux.bin.o
LD .tmp_vmlinux.kallsyms1
KSYMS .tmp_vmlinux.kallsyms1.S
AS .tmp_vmlinux.kallsyms1.S
LD .tmp_vmlinux.kallsyms2
KSYMS .tmp_vmlinux.kallsyms2.S
AS .tmp_vmlinux.kallsyms2.S
LD vmlinux
BTFIDS vmlinux
FAILED: load BTF from vmlinux: Invalid argument
make: *** [Makefile:1159: vmlinux] Error 255
make: *** Deleting file 'vmlinux'

Full built output is here: https://pastebin.com/icF3gVgN
Thanks.

--
Best Regards,
Mike Gavrilov.

2022-09-25 19:25:27

by Willy Tarreau

[permalink] [raw]
Subject: Re: Unable bisect issue because kernel not building from old commits

On Sun, Sep 25, 2022 at 09:13:13PM +0500, Mikhail Gavrilov wrote:
> On Sun, Sep 25, 2022 at 8:21 PM Willy Tarreau <[email protected]> wrote:
> > Just add:
> >
> > CROSS_COMPILE=/path/to/toolchain/bin/prefix-
> >
> > to your make command line and it will be fine.
> >
> > The makefile will append {gcc,ld,...} to this CROSS_COMPILE prefix to
> > construct the full pathname to the binaries.
>
> It works!
> But I hit another build issue.
>
> $ make clean &&
> CROSS_COMPILE=~/Downloads/x86_64-gcc-7.5.0-nolibc-x86_64-linux/gcc-7.5.0-nolibc/x86_64-linux/bin/x86_64-linux-
> make -j32 bzImage
> ***
> GEN modules.builtin
> LD .tmp_vmlinux.btf
> BTF .btf.vmlinux.bin.o
> LD .tmp_vmlinux.kallsyms1
> KSYMS .tmp_vmlinux.kallsyms1.S
> AS .tmp_vmlinux.kallsyms1.S
> LD .tmp_vmlinux.kallsyms2
> KSYMS .tmp_vmlinux.kallsyms2.S
> AS .tmp_vmlinux.kallsyms2.S
> LD vmlinux
> BTFIDS vmlinux
> FAILED: load BTF from vmlinux: Invalid argument
> make: *** [Makefile:1159: vmlinux] Error 255
> make: *** Deleting file 'vmlinux'

This was reported and addressed a month ago:

https://lore.kernel.org/bpf/[email protected]/t/#m24a9de4b8cc11eb9c3bbed022487d6de0d42ea4b

In short if you don't need XDP you could possibly disable
CONFIG_DEBUG_INFO_BTF, otherwise you may have to rebuild the latest
pahole utility that seems to be involved in this. But I never faced
this issue so I can only recommend hints about things to look at,
and could be wrong.

Willy