Until the link-vmlinux.sh split (cf. the commit below), the linker was
run with jobserver set in MAKEFLAGS. After the split, the command in
Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
is lost.
Restore it as linkers working in parallel (esp. the LTO ones) make a use
of it.
Cc: Sedat Dilek <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: Michal Marek <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
Signed-off-by: Jiri Slaby <[email protected]>
---
scripts/Makefile.vmlinux_o | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
index 3c97a1564947..d032f625b576 100644
--- a/scripts/Makefile.vmlinux_o
+++ b/scripts/Makefile.vmlinux_o
@@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
endef
vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
- $(call if_changed_rule,ld_vmlinux.o)
+ +$(call if_changed_rule,ld_vmlinux.o)
targets += vmlinux.o
--
2.36.1
On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
>
> Until the link-vmlinux.sh split (cf. the commit below), the linker was
> run with jobserver set in MAKEFLAGS. After the split, the command in
> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> is lost.
>
> Restore it as linkers working in parallel (esp. the LTO ones) make a use
> of it.
>
> Cc: Sedat Dilek <[email protected]>
> Cc: Masahiro Yamada <[email protected]>
> Cc: Michal Marek <[email protected]>
> Cc: Nick Desaulniers <[email protected]>
> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> Signed-off-by: Jiri Slaby <[email protected]>
Nice catch.
( I have seen some slowdown in my build-time. Will test and report. )
Reviewed-by: Sedat Dilek <[email protected]>
-Sedat-
> ---
> scripts/Makefile.vmlinux_o | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
> index 3c97a1564947..d032f625b576 100644
> --- a/scripts/Makefile.vmlinux_o
> +++ b/scripts/Makefile.vmlinux_o
> @@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
> endef
>
> vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
> - $(call if_changed_rule,ld_vmlinux.o)
> + +$(call if_changed_rule,ld_vmlinux.o)
>
> targets += vmlinux.o
>
> --
> 2.36.1
>
On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
>
> On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> >
> > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > run with jobserver set in MAKEFLAGS. After the split, the command in
> > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > is lost.
> >
> > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > of it.
> >
> > Cc: Sedat Dilek <[email protected]>
> > Cc: Masahiro Yamada <[email protected]>
> > Cc: Michal Marek <[email protected]>
> > Cc: Nick Desaulniers <[email protected]>
> > Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> > Signed-off-by: Jiri Slaby <[email protected]>
>
> Nice catch.
> ( I have seen some slowdown in my build-time. Will test and report. )
>
> Reviewed-by: Sedat Dilek <[email protected]>
>
No measurable difference in build-time.
Tested-by: Sedat Dilek <[email protected]> # LLVM-14 (x86-64)
-Sedat-
>
> > ---
> > scripts/Makefile.vmlinux_o | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
> > index 3c97a1564947..d032f625b576 100644
> > --- a/scripts/Makefile.vmlinux_o
> > +++ b/scripts/Makefile.vmlinux_o
> > @@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
> > endef
> >
> > vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
> > - $(call if_changed_rule,ld_vmlinux.o)
> > + +$(call if_changed_rule,ld_vmlinux.o)
> >
> > targets += vmlinux.o
> >
> > --
> > 2.36.1
> >
On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
>
> On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
> >
> > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> > >
> > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > is lost.
> > >
> > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > of it.
> > >
> > > Cc: Sedat Dilek <[email protected]>
> > > Cc: Masahiro Yamada <[email protected]>
> > > Cc: Michal Marek <[email protected]>
> > > Cc: Nick Desaulniers <[email protected]>
> > > Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> > > Signed-off-by: Jiri Slaby <[email protected]>
> >
> > Nice catch.
> > ( I have seen some slowdown in my build-time. Will test and report. )
> >
> > Reviewed-by: Sedat Dilek <[email protected]>
> >
>
> No measurable difference in build-time.
>
> Tested-by: Sedat Dilek <[email protected]> # LLVM-14 (x86-64)
>
[ CC Nathan and Sami ]
I re-checked as my build-time takes approx. 15mins more than usual
with Linux v5.17.
[ Perf - seconds time elapsed ]
v5.17: approx. 12500
v5.19: approx. 13500
+1.000 secs = +16,67mins
I am still investigating.
I use LLVM v14.0.4 and CONFIG_LTO_CLANG_THIN=y.
When I check for...
$ egrep 'Makefile.vmlinux_o|jobserver-exec|link-vmlinux.sh'
build-log_5.19.0-rc2-2-amd64-clang14-lto.txt
61414: sh scripts/link-vmlinux.sh "ld.lld" "-m elf_x86_64
--thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5"
"--emit-relocs --discard-none -z max-page-size=0x200000
--build-id=sha1 -X --orphan-handling=warn"; true
61426:+ make -f ./scripts/Makefile.vmlinux_o
61427: python3 ./scripts/jobserver-exec perl
scripts/generate_initcall_order.pl arch/x86/kernel/head_64.o
arch/x86/kernel/head64.o arch/x86/kernel/ebda.o
arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a
arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a
fs/built-in.a ipc/built-in.a security/built-i
n.a crypto/built-in.a block/built-in.a lib/built-in.a
arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a
drivers/built-in.a sound/built-in.a net/built-in.a virt/buil
t-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a
arch/x86/video/built-in.a > .tmp_initcalls.lds
61693:+ echo vmlinux: scripts/link-vmlinux.sh
Plus vmlinux.o
61428: ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm
-import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds
--whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o
arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o
init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a
certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a
security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a
arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a
drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a
arch/x86/pci/built-in.a arch/x86/power/built-in.a
arch/x86/video/built-in.a --no-whole-archive --start-group
--end-group ; ./tools/objtool/objtool --hacks=jump_label
--hacks=noinstr --mcount --orc --retpoline --static-call
--uaccess --link --
link vmlinux.o
Pooh, truncated via Gmail? Attached as a file.
Checking via ps...
$ ps -ef | grep l[l]d > lld-jobserver.txt
...and top util shows me only one lld link-job is running (file attached).
Unsure, if in my setup lld linker benefits from jobserver feature or not.
Am I missing something?
My kernel-config is attached.
Regards,
-Sedat-
On Fri, Jun 17, 2022 at 12:35 PM Sedat Dilek <[email protected]> wrote:
>
> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
> >
> > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
> > >
> > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> > > >
> > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > is lost.
> > > >
> > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > of it.
> > > >
> > > > Cc: Sedat Dilek <[email protected]>
> > > > Cc: Masahiro Yamada <[email protected]>
> > > > Cc: Michal Marek <[email protected]>
> > > > Cc: Nick Desaulniers <[email protected]>
> > > > Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> > > > Signed-off-by: Jiri Slaby <[email protected]>
> > >
> > > Nice catch.
> > > ( I have seen some slowdown in my build-time. Will test and report. )
> > >
> > > Reviewed-by: Sedat Dilek <[email protected]>
> > >
> >
> > No measurable difference in build-time.
> >
> > Tested-by: Sedat Dilek <[email protected]> # LLVM-14 (x86-64)
> >
>
> [ CC Nathan and Sami ]
>
> I re-checked as my build-time takes approx. 15mins more than usual
> with Linux v5.17.
>
> [ Perf - seconds time elapsed ]
> v5.17: approx. 12500
> v5.19: approx. 13500
>
> +1.000 secs = +16,67mins
>
> I am still investigating.
>
> I use LLVM v14.0.4 and CONFIG_LTO_CLANG_THIN=y.
>
> When I check for...
>
> $ egrep 'Makefile.vmlinux_o|jobserver-exec|link-vmlinux.sh'
> build-log_5.19.0-rc2-2-amd64-clang14-lto.txt
> 61414: sh scripts/link-vmlinux.sh "ld.lld" "-m elf_x86_64
> --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5"
> "--emit-relocs --discard-none -z max-page-size=0x200000
> --build-id=sha1 -X --orphan-handling=warn"; true
> 61426:+ make -f ./scripts/Makefile.vmlinux_o
> 61427: python3 ./scripts/jobserver-exec perl
> scripts/generate_initcall_order.pl arch/x86/kernel/head_64.o
> arch/x86/kernel/head64.o arch/x86/kernel/ebda.o
> arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a
> arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a
> fs/built-in.a ipc/built-in.a security/built-i
> n.a crypto/built-in.a block/built-in.a lib/built-in.a
> arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a
> drivers/built-in.a sound/built-in.a net/built-in.a virt/buil
> t-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a
> arch/x86/video/built-in.a > .tmp_initcalls.lds
> 61693:+ echo vmlinux: scripts/link-vmlinux.sh
>
> Plus vmlinux.o
>
> 61428: ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm
> -import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds
> --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o
> arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o
> init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a
> certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a
> security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a
> arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a
> drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a
> arch/x86/pci/built-in.a arch/x86/power/built-in.a
> arch/x86/video/built-in.a --no-whole-archive --start-group
> --end-group ; ./tools/objtool/objtool --hacks=jump_label
> --hacks=noinstr --mcount --orc --retpoline --static-call
> --uaccess --link --
> link vmlinux.o
>
> Pooh, truncated via Gmail? Attached as a file.
>
> Checking via ps...
>
> $ ps -ef | grep l[l]d > lld-jobserver.txt
>
> ...and top util shows me only one lld link-job is running (file attached).
>
> Unsure, if in my setup lld linker benefits from jobserver feature or not.
>
> Am I missing something?
>
> My kernel-config is attached.
>
Found this...
$ ld.lld-14 --help
--thinlto-jobs=<value> Number of ThinLTO jobs. Default to --threads=
-Sedat-
(+LLVM list, Fangrui Song)
On Fri, Jun 17, 2022 at 7:41 PM Sedat Dilek <[email protected]> wrote:
>
> On Fri, Jun 17, 2022 at 12:35 PM Sedat Dilek <[email protected]> wrote:
> >
> > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
> > >
> > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
> > > >
> > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> > > > >
> > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > > is lost.
> > > > >
> > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > > of i
Hi Jiri,
Please let me clarify first.
Here, is it OK to assume you are talking about Clang LTO
instead of GCC LTO because the latter is not upstreamed ?
I tested this patch but I did not see any performance change for Clang LTO.
[1] CONFIG_CLANG_LTO_FULL
lld always runs sequential.
It never runs in parallel even if you pass -j option to Make
[2] CONFIG_CLANG_LTO_THIN
lld always runs in parallel even if you do not pass -j option
In my machine, lld always allocated 12 threads.
This is irrespective of the Make parallelisms.
One more thing, if a program wants to participate in
Make's jobserver, it must parse MAKEFLAGS, and extract
file descriptors to be used to communicate to the jobserver.
As a code example in the kernel tree,
scripts/jobserver-exec parses "MAKEFLAGS" and "--jobserver".
I grepped the lld source code, but it does not contain
"MAKEFLAGS" or "jobserver".
masahiro@oscar:~/ref/lld$ git remote show origin
* remote origin
Fetch URL: https://github.com/llvm-mirror/lld.git
Push URL: https://github.com/llvm-mirror/lld.git
HEAD branch: master
Remote branches:
master tracked
release_36 tracked
release_37 tracked
release_38 tracked
release_39 tracked
release_40 tracked
release_50 tracked
release_60 tracked
release_70 tracked
release_80 tracked
release_90 tracked
Local branch configured for 'git pull':
master merges with remote master
Local ref configured for 'git push':
master pushes to master (up to date)
masahiro@oscar:~/ref/lld$ git grep MAKEFLAGS
masahiro@oscar:~/ref/lld$ git grep jobserver
So, in my research, LLD does not seem to support the jobserver.
If you are talking about GCC LTO, yes, the code
tries to parse "--jobserver-auth=" from the MAKEFLAGS
environment variable. [1]
[1]: https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/lto-wrapper.cc#L1341
But, as you may know, GCC LTO works in a different way,
at least, we cannot do it before modpost.
--
Best Regards
Masahiro Yamada
On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <[email protected]> wrote:
>
> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
> >
> > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
> > >
> > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> > > >
> > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > is lost.
> > > >
> > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > of it.
TBH, I agree with Masahiro. I didn't understand this comment "esp. the
LTO ones." Jiri, can you clarify what you mean here?
> [ CC Nathan and Sami ]
>
> I re-checked as my build-time takes approx. 15mins more than usual
> with Linux v5.17.
>
> [ Perf - seconds time elapsed ]
> v5.17: approx. 12500
> v5.19: approx. 13500
>
> +1.000 secs = +16,67mins
Does this patch make a difference?
https://lore.kernel.org/linux-kbuild/[email protected]/
--
Thanks,
~Nick Desaulniers
On Fri, Jun 17, 2022 at 6:32 PM Nick Desaulniers
<[email protected]> wrote:
>
> On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <[email protected]> wrote:
> >
> > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
> > >
> > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
> > > >
> > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> > > > >
> > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > > is lost.
> > > > >
> > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > > of it.
>
> TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> LTO ones." Jiri, can you clarify what you mean here?
>
> > [ CC Nathan and Sami ]
> >
> > I re-checked as my build-time takes approx. 15mins more than usual
> > with Linux v5.17.
> >
> > [ Perf - seconds time elapsed ]
> > v5.17: approx. 12500
> > v5.19: approx. 13500
> >
> > +1.000 secs = +16,67mins
>
> Does this patch make a difference?
> https://lore.kernel.org/linux-kbuild/[email protected]/
>
Running a new build...
-sed@-
On Fri, Jun 17, 2022 at 07:50:58PM +0200, Sedat Dilek wrote:
> On Fri, Jun 17, 2022 at 6:32 PM Nick Desaulniers
> <[email protected]> wrote:
> >
> > On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <[email protected]> wrote:
> > >
> > > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
> > > >
> > > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
> > > > >
> > > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> > > > > >
> > > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > > > is lost.
> > > > > >
> > > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > > > of it.
> >
> > TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> > LTO ones." Jiri, can you clarify what you mean here?
> >
> > > [ CC Nathan and Sami ]
> > >
> > > I re-checked as my build-time takes approx. 15mins more than usual
> > > with Linux v5.17.
> > >
> > > [ Perf - seconds time elapsed ]
> > > v5.17: approx. 12500
> > > v5.19: approx. 13500
> > >
> > > +1.000 secs = +16,67mins
So roughly 7.5% increase if I did my math correctly?
> > Does this patch make a difference?
> > https://lore.kernel.org/linux-kbuild/[email protected]/
> >
>
> Running a new build...
I did a quick benchmark with Fedora's clang 14.0.0 and I didn't see an
increase of that proportion with your configuration (only about 1%,
which I would expect would be within the noise for new drivers and
such):
Benchmark 1: 5.17
Time (abs ≡): 287.744 s [User: 23497.818 s, System: 4444.602 s]
Benchmark 2: 5.19-rc2
Time (abs ≡): 290.027 s [User: 24567.290 s, System: 4834.458 s]
Summary
'5.17' ran
1.01 times faster than '5.19-rc2'
Not to say that there is not an issue but it might be more visible with
slower hardware.
Cheers,
Nathan
On 2022-06-18, Masahiro Yamada wrote:
>(+LLVM list, Fangrui Song)
Thanks for tagging me. I'll clarify some stuff.
>On Fri, Jun 17, 2022 at 7:41 PM Sedat Dilek <[email protected]> wrote:
>>
>> On Fri, Jun 17, 2022 at 12:35 PM Sedat Dilek <[email protected]> wrote:
>> >
>> > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
>> > >
>> > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
>> > > >
>> > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
>> > > > >
>> > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
>> > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
>> > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
>> > > > > is lost.
>> > > > >
>> > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
>> > > > > of i
>
>Hi Jiri,
>
>Please let me clarify first.
>
>Here, is it OK to assume you are talking about Clang LTO
>instead of GCC LTO because the latter is not upstreamed ?
>
>
>
>
>
>I tested this patch but I did not see any performance change for Clang LTO.
>
>
>[1] CONFIG_CLANG_LTO_FULL
>
> lld always runs sequential.
> It never runs in parallel even if you pass -j option to Make
"lld always runs sequential" is not accurate. There are a number of
parallel linker passes. ld.lld --threads= defaults to
llvm::hardware_concurrency (similar to
https://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency,
but uses sched_getaffinity to compute the number of available cores).
"lld always runs sequential" is only correct only when --threads=1 is
specified or the system only provides one thread to the lld process.
I think people may be more interested in LTO parallelism here. Regular
LTO (sometimes called full LTO when there is mixed-thin-and-regular LTO)
supports limited parallelism which applies to code generation, but not
IR-level optimization. (IR-level optimization has many interprocedural
optimizations passes. Splitting will make LTO less effective. Code
generation is per function, so parallelism does not regress
optimization.)
>
>[2] CONFIG_CLANG_LTO_THIN
>
> lld always runs in parallel even if you do not pass -j option
>
> In my machine, lld always allocated 12 threads.
> This is irrespective of the Make parallelisms.
>
>
>
>
>One more thing, if a program wants to participate in
>Make's jobserver, it must parse MAKEFLAGS, and extract
>file descriptors to be used to communicate to the jobserver.
>
>As a code example in the kernel tree,
>scripts/jobserver-exec parses "MAKEFLAGS" and "--jobserver".
>
>
>I grepped the lld source code, but it does not contain
>"MAKEFLAGS" or "jobserver".
>masahiro@oscar:~/ref/lld$ git remote show origin
>* remote origin
> Fetch URL: https://github.com/llvm-mirror/lld.git
> Push URL: https://github.com/llvm-mirror/lld.git
> HEAD branch: master
> Remote branches:
> master tracked
> release_36 tracked
> release_37 tracked
> release_38 tracked
> release_39 tracked
> release_40 tracked
> release_50 tracked
> release_60 tracked
> release_70 tracked
> release_80 tracked
> release_90 tracked
> Local branch configured for 'git pull':
> master merges with remote master
> Local ref configured for 'git push':
> master pushes to master (up to date)
>masahiro@oscar:~/ref/lld$ git grep MAKEFLAGS
>masahiro@oscar:~/ref/lld$ git grep jobserver
>
>
>So, in my research, LLD does not seem to support the jobserver.
Correct. lld does not support GNU make's jobserver. On the other hand,
I don't think the jobserver implementation supports flexible "give this
target N hardware concurrency". A heavy link target does not necessarily
get more resources than a quick target.
If a make target knows how many hardware concurrency it gets, we can
pass --threads= to lld. LTO easily takes 95+% link time, so LTO
parallelism may needs a dedicated setting. lld has --thinlto-jobs=.
>
>
>
>If you are talking about GCC LTO, yes, the code
>tries to parse "--jobserver-auth=" from the MAKEFLAGS
>environment variable. [1]
>
>[1]: https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/lto-wrapper.cc#L1341
>
>
>But, as you may know, GCC LTO works in a different way,
>at least, we cannot do it before modpost.
>
>
>--
>Best Regards
>Masahiro Yamada
>
On Fri, Jun 17, 2022 at 9:34 PM Nathan Chancellor <[email protected]> wrote:
>
> On Fri, Jun 17, 2022 at 07:50:58PM +0200, Sedat Dilek wrote:
> > On Fri, Jun 17, 2022 at 6:32 PM Nick Desaulniers
> > <[email protected]> wrote:
> > >
> > > On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <[email protected]> wrote:
> > > >
> > > > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
> > > > >
> > > > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
> > > > > >
> > > > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> > > > > > >
> > > > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > > > > is lost.
> > > > > > >
> > > > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > > > > of it.
> > >
> > > TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> > > LTO ones." Jiri, can you clarify what you mean here?
> > >
> > > > [ CC Nathan and Sami ]
> > > >
> > > > I re-checked as my build-time takes approx. 15mins more than usual
> > > > with Linux v5.17.
> > > >
> > > > [ Perf - seconds time elapsed ]
> > > > v5.17: approx. 12500
> > > > v5.19: approx. 13500
> > > >
> > > > +1.000 secs = +16,67mins
>
> So roughly 7.5% increase if I did my math correctly?
>
> > > Does this patch make a difference?
> > > https://lore.kernel.org/linux-kbuild/[email protected]/
> > >
> >
> > Running a new build...
>
> I did a quick benchmark with Fedora's clang 14.0.0 and I didn't see an
> increase of that proportion with your configuration (only about 1%,
> which I would expect would be within the noise for new drivers and
> such):
>
> Benchmark 1: 5.17
> Time (abs ≡): 287.744 s [User: 23497.818 s, System: 4444.602 s]
>
> Benchmark 2: 5.19-rc2
> Time (abs ≡): 290.027 s [User: 24567.290 s, System: 4834.458 s]
>
> Summary
> '5.17' ran
> 1.01 times faster than '5.19-rc2'
>
> Not to say that there is not an issue but it might be more visible with
> slower hardware.
>
@Nick Desaulniers
I applied the patch.
Seen from build-time not really measurable.
Noticed some binary size reduction.
@Nathan Chancellor
Yeah, that I was thinking, too.
This is due to the number of driver (and modules) increased.
And of course the changes to kbuild-system.
It's hard to compare.
Surely, ancient CPU (10 years) plus HDD etc.
-Sedat-
4
On Fri, Jun 17, 2022 at 10:05 PM Fangrui Song <[email protected]> wrote:
>
> On 2022-06-18, Masahiro Yamada wrote:
> >(+LLVM list, Fangrui Song)
>
> Thanks for tagging me. I'll clarify some stuff.
>
> >On Fri, Jun 17, 2022 at 7:41 PM Sedat Dilek <[email protected]> wrote:
> >>
> >> On Fri, Jun 17, 2022 at 12:35 PM Sedat Dilek <[email protected]> wrote:
> >> >
> >> > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
> >> > >
> >> > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
> >> > > >
> >> > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> >> > > > >
> >> > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> >> > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> >> > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> >> > > > > is lost.
> >> > > > >
> >> > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> >> > > > > of i
> >
> >Hi Jiri,
> >
> >Please let me clarify first.
> >
> >Here, is it OK to assume you are talking about Clang LTO
> >instead of GCC LTO because the latter is not upstreamed ?
> >
> >
> >
> >
> >
> >I tested this patch but I did not see any performance change for Clang LTO.
> >
> >
> >[1] CONFIG_CLANG_LTO_FULL
> >
> > lld always runs sequential.
> > It never runs in parallel even if you pass -j option to Make
>
> "lld always runs sequential" is not accurate. There are a number of
> parallel linker passes. ld.lld --threads= defaults to
> llvm::hardware_concurrency (similar to
> https://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency,
> but uses sched_getaffinity to compute the number of available cores).
>
> "lld always runs sequential" is only correct only when --threads=1 is
> specified or the system only provides one thread to the lld process.
>
> I think people may be more interested in LTO parallelism here. Regular
> LTO (sometimes called full LTO when there is mixed-thin-and-regular LTO)
> supports limited parallelism which applies to code generation, but not
> IR-level optimization. (IR-level optimization has many interprocedural
> optimizations passes. Splitting will make LTO less effective. Code
> generation is per function, so parallelism does not regress
> optimization.)
>
> >
> >[2] CONFIG_CLANG_LTO_THIN
> >
> > lld always runs in parallel even if you do not pass -j option
> >
> > In my machine, lld always allocated 12 threads.
> > This is irrespective of the Make parallelisms.
> >
> >
> >
> >
> >One more thing, if a program wants to participate in
> >Make's jobserver, it must parse MAKEFLAGS, and extract
> >file descriptors to be used to communicate to the jobserver.
> >
> >As a code example in the kernel tree,
> >scripts/jobserver-exec parses "MAKEFLAGS" and "--jobserver".
> >
> >
> >I grepped the lld source code, but it does not contain
> >"MAKEFLAGS" or "jobserver".
>
> >masahiro@oscar:~/ref/lld$ git remote show origin
> >* remote origin
> > Fetch URL: https://github.com/llvm-mirror/lld.git
> > Push URL: https://github.com/llvm-mirror/lld.git
> > HEAD branch: master
> > Remote branches:
> > master tracked
> > release_36 tracked
> > release_37 tracked
> > release_38 tracked
> > release_39 tracked
> > release_40 tracked
> > release_50 tracked
> > release_60 tracked
> > release_70 tracked
> > release_80 tracked
> > release_90 tracked
> > Local branch configured for 'git pull':
> > master merges with remote master
> > Local ref configured for 'git push':
> > master pushes to master (up to date)
> >masahiro@oscar:~/ref/lld$ git grep MAKEFLAGS
> >masahiro@oscar:~/ref/lld$ git grep jobserver
> >
> >
> >So, in my research, LLD does not seem to support the jobserver.
>
>
> Correct. lld does not support GNU make's jobserver. On the other hand,
> I don't think the jobserver implementation supports flexible "give this
> target N hardware concurrency". A heavy link target does not necessarily
> get more resources than a quick target.
>
> If a make target knows how many hardware concurrency it gets, we can
> pass --threads= to lld. LTO easily takes 95+% link time, so LTO
> parallelism may needs a dedicated setting. lld has --thinlto-jobs=.
>
Hey Fangrui,
I played a bit with --thinlto-jobs=4 yesterday.
$ cat 0001-vmlinux-clang-thinlto-Add-thinlto-jobs-4-to-KBUILD_L.patch
From f548c34abd49e01407de26c81f29ef89b3cae213 Mon Sep 17 00:00:00 2001
From: Sedat Dilek <[email protected]>
Date: Fri, 17 Jun 2022 13:24:50 +0200
Subject: [PATCH] vmlinux: clang: thinlto: Add --thinlto-jobs=4 to
KBUILD_LDFLAGS
---
scripts/Makefile.vmlinux_o | 2 +-
scripts/link-vmlinux.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
index 3c97a1564947..4c1991c91417 100644
--- a/scripts/Makefile.vmlinux_o
+++ b/scripts/Makefile.vmlinux_o
@@ -53,7 +53,7 @@ objtool_args := \
quiet_cmd_ld_vmlinux.o = LD $@
cmd_ld_vmlinux.o = \
- $(LD) ${KBUILD_LDFLAGS} -r -o $@ \
+ $(LD) ${KBUILD_LDFLAGS} --thinlto-jobs=4 -r -o $@ \
$(addprefix -T , $(initcalls-lds)) \
--whole-archive $(KBUILD_VMLINUX_OBJS) --no-whole-archive \
--start-group $(KBUILD_VMLINUX_LIBS) --end-group \
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index eecc1863e556..1624da57807b 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -83,7 +83,7 @@ vmlinux_link()
else
wl=
ld="${LD}"
- ldflags="${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux}"
+ ldflags="${KBUILD_LDFLAGS} --thinlto-jobs=4 ${LDFLAGS_vmlinux}"
ldlibs=
fi
--
2.36.1
Hmm, not a significant performance gain - not measurable here.
Unsure, if passing --thinlto-jobs=4 to KBUILD_LDFLAGS in top-level
Makefile is too invasive.
( UNTESTED... )
$ cat 0001-clang-thinlto-Add-thinlto-jobs-4-to-KBUILD_LDFLAGS.patch
From 0970e92867d11d12214ca198578364a17ef17bea Mon Sep 17 00:00:00 2001
From: Sedat Dilek <[email protected]>
Date: Fri, 17 Jun 2022 13:34:49 +0200
Subject: [PATCH] clang: thinlto: Add --thinlto-jobs=4 to KBUILD_LDFLAGS
---
Makefile | 1 +
1 file changed, 1 insertion(+)
diff --git a/Makefile b/Makefile
index 1a6678d817bd..a7a7f12e2349 100644
--- a/Makefile
+++ b/Makefile
@@ -896,6 +896,7 @@ ifdef CONFIG_LTO_CLANG
ifdef CONFIG_LTO_CLANG_THIN
CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit
KBUILD_LDFLAGS += --thinlto-cache-dir=$(extmod_prefix).thinlto-cache
+KBUILD_LDFLAGS += --thinlto-jobs=4
else
CC_FLAGS_LTO := -flto
endif
--
2.36.1
In the case of building my LLVM toolchain I have a very conservative
setting for link-jobs when building a ThinLTO + PGO (x86_64-kernel
defconfig) optimized toolchain.
$ cd /path/to/tc-build.git
$ python3 ./build-llvm.py --no-update --build-type Release -p
clang;lld -t X86;BPF --clang-vendor dileks -B
/home/dileks/src/llvm-toolchain/build -I /opt/llvm-toolchain
--check-targets clang lld --lto thin --pgo kernel-defconfig -L
/home/dileks/src/linux-kernel/git -D LLVM_PARALLEL_LINK_JOBS=1
--show-build-commands
See: -D LLVM_PARALLEL_LINK_JOBS=1
So, I guess the above patch might be counterproductive?
My kernel make-line looks like this:
/usr/bin/perf stat make V=1 -j4 LLVM=1 LLVM_IAS=1
PAHOLE=/opt/pahole/bin/pahole LOCALVERSION=-4-amd64-clang14-lto K
BUILD_BUILD_HOST=iniza [email protected]
KBUILD_BUILD_TIMESTAMP=2022-06-17 bindeb-pkg
KDEB_PKGVERSION=5.19.0~rc2-4~bookworm+dileks1
Attaching 2 patches in case Gmail truncates the formatting and my
latest kernel-config.
Thanks.
Regards,
-Sedat-
> >
> >
> >
> >If you are talking about GCC LTO, yes, the code
> >tries to parse "--jobserver-auth=" from the MAKEFLAGS
> >environment variable. [1]
> >
> >[1]: https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/lto-wrapper.cc#L1341
> >
> >
> >But, as you may know, GCC LTO works in a different way,
> >at least, we cannot do it before modpost.
> >
> >
> >--
> >Best Regards
> >Masahiro Yamada
> >
On 17. 06. 22, 18:32, Nick Desaulniers wrote:
> On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <[email protected]> wrote:
>>
>> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
>>>
>>> On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
>>>>
>>>> On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
>>>>>
>>>>> Until the link-vmlinux.sh split (cf. the commit below), the linker was
>>>>> run with jobserver set in MAKEFLAGS. After the split, the command in
>>>>> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
>>>>> is lost.
>>>>>
>>>>> Restore it as linkers working in parallel (esp. the LTO ones) make a use
>>>>> of it.
>
> TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> LTO ones." Jiri, can you clarify what you mean here?
Sure. clang is not the only compiler/linker that can handle kernel ;).
So while rebasing gcc-LTO-kernel-patches to 5.19-rc (to upstream them
later), this broke. See:
https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc;h=26e06e77be4e0afb2bc3e913062a9c51cab5d205;hb=HEAD#l1336
gcc really parses MAKEFLAGS and looks for "--jobserver-auth=" there, if
one passes specifies -flto=jobserver.
thanks,
--
js
suse labs
On Tue, Jun 21, 2022 at 4:30 PM Jiri Slaby <[email protected]> wrote:
>
> On 17. 06. 22, 18:32, Nick Desaulniers wrote:
> > On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <[email protected]> wrote:
> >>
> >> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
> >>>
> >>> On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
> >>>>
> >>>> On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
> >>>>>
> >>>>> Until the link-vmlinux.sh split (cf. the commit below), the linker was
> >>>>> run with jobserver set in MAKEFLAGS. After the split, the command in
> >>>>> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> >>>>> is lost.
> >>>>>
> >>>>> Restore it as linkers working in parallel (esp. the LTO ones) make a use
> >>>>> of it.
> >
> > TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> > LTO ones." Jiri, can you clarify what you mean here?
>
> Sure. clang is not the only compiler/linker that can handle kernel ;).
> So while rebasing gcc-LTO-kernel-patches to 5.19-rc (to upstream them
> later), this broke. See:
> https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc;h=26e06e77be4e0afb2bc3e913062a9c51cab5d205;hb=HEAD#l1336
>
> gcc really parses MAKEFLAGS and looks for "--jobserver-auth=" there, if
> one passes specifies -flto=jobserver.
>
> thanks,
> --
> js
> suse labs
Then, this patch should go together (or after) the GCC LTO upstreaming.
I still do not understand why you are patching the pre-modpost linking, though.
In my understanding, Clang LTO and GCC LTO are implemented
in different layers.
Clang LTO is implemented in the linker (i.e. lld).
In contrast, GCC LTO is implemented in the compiler (i.e. gcc itself)
The linker (i.e. BFD linker or Gold from GNU binutils project)
is agnostic about LTO.
Your are patching the relocatable link ($(LD) -r),
but I could not find the code parsing jobserver-auth
in the binutils-gdb repository.
--
Best Regards
Masahiro Yamada
On 21. 06. 22, 10:41, Masahiro Yamada wrote:
> On Tue, Jun 21, 2022 at 4:30 PM Jiri Slaby <[email protected]> wrote:
>>
>> On 17. 06. 22, 18:32, Nick Desaulniers wrote:
>>> On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <[email protected]> wrote:
>>>>
>>>> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <[email protected]> wrote:
>>>>>
>>>>> On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <[email protected]> wrote:
>>>>>>
>>>>>> On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <[email protected]> wrote:
>>>>>>>
>>>>>>> Until the link-vmlinux.sh split (cf. the commit below), the linker was
>>>>>>> run with jobserver set in MAKEFLAGS. After the split, the command in
>>>>>>> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
>>>>>>> is lost.
>>>>>>>
>>>>>>> Restore it as linkers working in parallel (esp. the LTO ones) make a use
>>>>>>> of it.
>>>
>>> TBH, I agree with Masahiro. I didn't understand this comment "esp. the
>>> LTO ones." Jiri, can you clarify what you mean here?
>>
>> Sure. clang is not the only compiler/linker that can handle kernel ;).
>> So while rebasing gcc-LTO-kernel-patches to 5.19-rc (to upstream them
>> later), this broke. See:
>> https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc;h=26e06e77be4e0afb2bc3e913062a9c51cab5d205;hb=HEAD#l1336
>>
>> gcc really parses MAKEFLAGS and looks for "--jobserver-auth=" there, if
>> one passes specifies -flto=jobserver.
>>
>> thanks,
>> --
>> js
>> suse labs
>
>
>
> Then, this patch should go together (or after) the GCC LTO upstreaming.
Hi,
OK, I will make it part of the gcc-lto series.
> I still do not understand why you are patching the pre-modpost linking, though.
>
> In my understanding, Clang LTO and GCC LTO are implemented
> in different layers.
>
> Clang LTO is implemented in the linker (i.e. lld).
>
> In contrast, GCC LTO is implemented in the compiler (i.e. gcc itself)
> The linker (i.e. BFD linker or Gold from GNU binutils project)
> is agnostic about LTO.
Right. First, gcc compiles the sources to an intermediate language. Then
gcc-ld (LTO-)links the objects into a binary object. BFD ld/gold is
invoked only from within gcc to actually link the binary.
> Your are patching the relocatable link ($(LD) -r),
> but I could not find the code parsing jobserver-auth
> in the binutils-gdb repository.
With gcc-lto patchset, gcc-ld is invoked in cmd_ld_vmlinux.o.
So you are right, making it part of the gcc-lto patchset makes perfect
sense. First, to see the context. Second, upstream doesn't need the change.
thanks,
--
js
suse labs