2022-07-19 08:49:02

by Jiri Slaby

[permalink] [raw]
Subject: [PATCH v2 1/2] kbuild: pass jobserver to cmd_ld_vmlinux.o

Until the link-vmlinux.sh split (cf. the commit below), the linker was
run with jobserver set in MAKEFLAGS. After the split, the command in
Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
is lost.

Restore it as linkers working in parallel (esp. the LTO ones) make a use
of it. In this case, it is gcc producing this warning otherwise:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

Cc: Sedat Dilek <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: Michal Marek <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Signed-off-by: Martin Liska <[email protected]>
Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
Signed-off-by: Jiri Slaby <[email protected]>
---

Notes:
[v2] only commit message updated

scripts/Makefile.vmlinux_o | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
index 84019814f33f..886db0659d9c 100644
--- a/scripts/Makefile.vmlinux_o
+++ b/scripts/Makefile.vmlinux_o
@@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
endef

vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
- $(call if_changed_rule,ld_vmlinux.o)
+ +$(call if_changed_rule,ld_vmlinux.o)

targets += vmlinux.o

--
2.37.1


2022-07-19 08:49:03

by Jiri Slaby

[permalink] [raw]
Subject: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

From: Martin Liska <[email protected]>

Prefix command in makefile run in order to preserve access to jobserver.
This is needed for gcc at least.

Fixes this warning:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

Cc: Sedat Dilek <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: Michal Marek <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
Signed-off-by: Martin Liska <[email protected]>
Signed-off-by: Jiri Slaby <[email protected]>
---

Notes:
[v2] this is new in v2

scripts/Makefile.build | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index bb63374fb055..493f3c4e8461 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -247,8 +247,8 @@ endef

# Built-in and composite module parts
$(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
- $(call if_changed_rule,cc_o_c)
- $(call cmd,force_checksrc)
+ +$(call if_changed_rule,cc_o_c)
+ +$(call cmd,force_checksrc)

# To make this rule robust against "Argument list too long" error,
# ensure to add $(obj)/ prefix by a shell command.
@@ -457,7 +457,7 @@ endef
$(multi-obj-m): objtool-enabled := $(delay-objtool)
$(multi-obj-m): part-of-module := y
$(multi-obj-m): %.o: %.mod FORCE
- $(call if_changed_rule,ld_multi_m)
+ +$(call if_changed_rule,ld_multi_m)
$(call multi_depend, $(multi-obj-m), .o, -objs -y -m)

targets := $(filter-out $(PHONY), $(targets))
--
2.37.1

2022-07-19 09:19:49

by Masahiro Yamada

[permalink] [raw]
Subject: Re: [PATCH v2 1/2] kbuild: pass jobserver to cmd_ld_vmlinux.o

On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <[email protected]> wrote:
>
> Until the link-vmlinux.sh split (cf. the commit below), the linker was
> run with jobserver set in MAKEFLAGS. After the split, the command in
> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> is lost.
>
> Restore it as linkers working in parallel (esp. the LTO ones) make a use
> of it. In this case, it is gcc producing this warning otherwise:
> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
>
> Cc: Sedat Dilek <[email protected]>
> Cc: Masahiro Yamada <[email protected]>
> Cc: Michal Marek <[email protected]>
> Cc: Nick Desaulniers <[email protected]>
> Cc: Signed-off-by: Martin Liska <[email protected]>
> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> Signed-off-by: Jiri Slaby <[email protected]>
> ---
>
> Notes:
> [v2] only commit message updated


Did you read my previous email?

https://lore.kernel.org/linux-kbuild/CAK7LNAS46hrVh3FNiHHL8_Hqp85VgWnsHyjQ4t-TFH_H344NOg@mail.gmail.com/






>
> scripts/Makefile.vmlinux_o | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
> index 84019814f33f..886db0659d9c 100644
> --- a/scripts/Makefile.vmlinux_o
> +++ b/scripts/Makefile.vmlinux_o
> @@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
> endef
>
> vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
> - $(call if_changed_rule,ld_vmlinux.o)
> + +$(call if_changed_rule,ld_vmlinux.o)
>
> targets += vmlinux.o
>
> --
> 2.37.1
>


--
Best Regards
Masahiro Yamada

2022-07-19 09:31:59

by Masahiro Yamada

[permalink] [raw]
Subject: Re: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <[email protected]> wrote:
>
> From: Martin Liska <[email protected]>
>
> Prefix command in makefile run in order to preserve access to jobserver.
> This is needed for gcc at least.
>
> Fixes this warning:
> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’


What is lto-wrapper ?

>
> Cc: Sedat Dilek <[email protected]>
> Cc: Masahiro Yamada <[email protected]>
> Cc: Michal Marek <[email protected]>
> Cc: Nick Desaulniers <[email protected]>
> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> Signed-off-by: Martin Liska <[email protected]>
> Signed-off-by: Jiri Slaby <[email protected]>
> ---
>
> Notes:
> [v2] this is new in v2
>
> scripts/Makefile.build | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index bb63374fb055..493f3c4e8461 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -247,8 +247,8 @@ endef
>
> # Built-in and composite module parts
> $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
> - $(call if_changed_rule,cc_o_c)
> - $(call cmd,force_checksrc)
> + +$(call if_changed_rule,cc_o_c)
> + +$(call cmd,force_checksrc)


cmd_force_checksrc invokes sparse.
How is it related to GCC?




>
> # To make this rule robust against "Argument list too long" error,
> # ensure to add $(obj)/ prefix by a shell command.
> @@ -457,7 +457,7 @@ endef
> $(multi-obj-m): objtool-enabled := $(delay-objtool)
> $(multi-obj-m): part-of-module := y
> $(multi-obj-m): %.o: %.mod FORCE
> - $(call if_changed_rule,ld_multi_m)
> + +$(call if_changed_rule,ld_multi_m)
> $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)


cmd_ld_multi_m invokes $(LD) directly.
How is it related to gcc?






>
> targets := $(filter-out $(PHONY), $(targets))
> --
> 2.37.1
>


--
Best Regards
Masahiro Yamada

2022-09-20 07:15:52

by Jiri Slaby

[permalink] [raw]
Subject: Re: [PATCH v2 1/2] kbuild: pass jobserver to cmd_ld_vmlinux.o

On 19. 07. 22, 11:15, Masahiro Yamada wrote:
> On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <[email protected]> wrote:
>>
>> Until the link-vmlinux.sh split (cf. the commit below), the linker was
>> run with jobserver set in MAKEFLAGS. After the split, the command in
>> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
>> is lost.
>>
>> Restore it as linkers working in parallel (esp. the LTO ones) make a use
>> of it. In this case, it is gcc producing this warning otherwise:
>> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
>>
>> Cc: Sedat Dilek <[email protected]>
>> Cc: Masahiro Yamada <[email protected]>
>> Cc: Michal Marek <[email protected]>
>> Cc: Nick Desaulniers <[email protected]>
>> Cc: Signed-off-by: Martin Liska <[email protected]>
>> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
>> Signed-off-by: Jiri Slaby <[email protected]>
>> ---
>>
>> Notes:
>> [v2] only commit message updated
>
>
> Did you read my previous email?
>
> https://lore.kernel.org/linux-kbuild/CAK7LNAS46hrVh3FNiHHL8_Hqp85VgWnsHyjQ4t-TFH_H344NOg@mail.gmail.com/

Huh, I neither received this email nor the linked one to my inbox.
@suse.cz somehow filters mails from you :/. Let me figure out (and use
kernel.org account for now). (And get back to your comments too.)

>> scripts/Makefile.vmlinux_o | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
>> index 84019814f33f..886db0659d9c 100644
>> --- a/scripts/Makefile.vmlinux_o
>> +++ b/scripts/Makefile.vmlinux_o
>> @@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
>> endef
>>
>> vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
>> - $(call if_changed_rule,ld_vmlinux.o)
>> + +$(call if_changed_rule,ld_vmlinux.o)
>>
>> targets += vmlinux.o
>>
>> --
>> 2.37.1
>>
>
>

--
js

2022-09-20 08:32:23

by Jiri Slaby

[permalink] [raw]
Subject: Re: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

On 19. 07. 22, 11:15, Masahiro Yamada wrote:
> On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <[email protected]> wrote:
>>
>> From: Martin Liska <[email protected]>
>>
>> Prefix command in makefile run in order to preserve access to jobserver.
>> This is needed for gcc at least.
>>
>> Fixes this warning:
>> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
>
>
> What is lto-wrapper ?

lto-wrapper is a part of gcc:
https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc

And is invoked during the link of LTO objects (also to link in parallel
if jobserver is available).

I am leaving the unanswered questions below for Martin to answer.

>> Cc: Sedat Dilek <[email protected]>
>> Cc: Masahiro Yamada <[email protected]>
>> Cc: Michal Marek <[email protected]>
>> Cc: Nick Desaulniers <[email protected]>
>> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
>> Signed-off-by: Martin Liska <[email protected]>
>> Signed-off-by: Jiri Slaby <[email protected]>
>> ---
>>
>> Notes:
>> [v2] this is new in v2
>>
>> scripts/Makefile.build | 6 +++---
>> 1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
>> index bb63374fb055..493f3c4e8461 100644
>> --- a/scripts/Makefile.build
>> +++ b/scripts/Makefile.build
>> @@ -247,8 +247,8 @@ endef
>>
>> # Built-in and composite module parts
>> $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
>> - $(call if_changed_rule,cc_o_c)
>> - $(call cmd,force_checksrc)
>> + +$(call if_changed_rule,cc_o_c)
>> + +$(call cmd,force_checksrc)
>
>
> cmd_force_checksrc invokes sparse.
> How is it related to GCC?
>
>
>
>
>>
>> # To make this rule robust against "Argument list too long" error,
>> # ensure to add $(obj)/ prefix by a shell command.
>> @@ -457,7 +457,7 @@ endef
>> $(multi-obj-m): objtool-enabled := $(delay-objtool)
>> $(multi-obj-m): part-of-module := y
>> $(multi-obj-m): %.o: %.mod FORCE
>> - $(call if_changed_rule,ld_multi_m)
>> + +$(call if_changed_rule,ld_multi_m)
>> $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)
>
>
> cmd_ld_multi_m invokes $(LD) directly.
> How is it related to gcc?



--
js
suse labs

2022-09-20 12:41:20

by Martin Liška

[permalink] [raw]
Subject: Re: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

On 9/20/22 10:06, Jiri Slaby wrote:
> On 19. 07. 22, 11:15, Masahiro Yamada wrote:
>> On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <[email protected]> wrote:
>>>
>>> From: Martin Liska <[email protected]>
>>>
>>> Prefix command in makefile run in order to preserve access to jobserver.
>>> This is needed for gcc at least.
>>>
>>> Fixes this warning:
>>> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
>>
>>
>> What is lto-wrapper ?
>
> lto-wrapper is a part of gcc:
> https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc
>
> And is invoked during the link of LTO objects (also to link in parallel if jobserver is available).

Yes, it's a driver that is invoked by Linker when it decides to use LTO mode in GCC compiler.
Later on the lto-wrapper connects to make's jobserver and runs in parallel. Note a recent [1]
make implemented named pipes when it comes to communication with jobserver. And thus the '+'
won't be needed in the future, but we need it for now.

Cheers,
Martin

[1] https://git.savannah.gnu.org/cgit/make.git/commit/?id=7ad2593b2d2bb5b9332f4444d8bf93ac6f958bc6

>
> I am leaving the unanswered questions below for Martin to answer.
>
>>> Cc: Sedat Dilek <[email protected]>
>>> Cc: Masahiro Yamada <[email protected]>
>>> Cc: Michal Marek <[email protected]>
>>> Cc: Nick Desaulniers <[email protected]>
>>> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
>>> Signed-off-by: Martin Liska <[email protected]>
>>> Signed-off-by: Jiri Slaby <[email protected]>
>>> ---
>>>
>>> Notes:
>>>      [v2] this is new in v2
>>>
>>>   scripts/Makefile.build | 6 +++---
>>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
>>> index bb63374fb055..493f3c4e8461 100644
>>> --- a/scripts/Makefile.build
>>> +++ b/scripts/Makefile.build
>>> @@ -247,8 +247,8 @@ endef
>>>
>>>   # Built-in and composite module parts
>>>   $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
>>> -       $(call if_changed_rule,cc_o_c)
>>> -       $(call cmd,force_checksrc)
>>> +       +$(call if_changed_rule,cc_o_c)
>>> +       +$(call cmd,force_checksrc)
>>
>>
>> cmd_force_checksrc invokes sparse.
>> How is it related to GCC?
>>
>>
>>
>>
>>>
>>>   # To make this rule robust against "Argument list too long" error,
>>>   # ensure to add $(obj)/ prefix by a shell command.
>>> @@ -457,7 +457,7 @@ endef
>>>   $(multi-obj-m): objtool-enabled := $(delay-objtool)
>>>   $(multi-obj-m): part-of-module := y
>>>   $(multi-obj-m): %.o: %.mod FORCE
>>> -       $(call if_changed_rule,ld_multi_m)
>>> +       +$(call if_changed_rule,ld_multi_m)
>>>   $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)
>>
>>
>> cmd_ld_multi_m invokes $(LD) directly.
>> How is it related to gcc?
>
>
>