Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 28 Apr 2020 17:44:25 -0700
From:   Nathan Chancellor <natechancellor@gmail.com>
To:     Fangrui Song <maskray@google.com>
Cc:     Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
        Masahiro Yamada <masahiroy@kernel.org>,
        linux-mips@vger.kernel.org, linux-kernel@vger.kernel.org,
        clang-built-linux@googlegroups.com, linux-kbuild@vger.kernel.org,
        Nick Desaulniers <ndesaulniers@google.com>,
        Sami Tolvanen <samitolvanen@google.com>,
        Dmitry Golovin <dima@golovin.in>,
        Sedat Dilek <sedat.dilek@gmail.com>
Subject: Re: [PATCH v4 4/5] MIPS: VDSO: Use $(LD) instead of $(CC) to link
 VDSO
Message-ID: <20200429004425.GA566019@ubuntu-s3-xlarge-x86>
References: <20200423171807.29713-1-natechancellor@gmail.com>
 <20200428221419.2530697-1-natechancellor@gmail.com>
 <20200428221419.2530697-5-natechancellor@gmail.com>
 <20200428225401.7yrld7u2xr67t4xf@google.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20200428225401.7yrld7u2xr67t4xf@google.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Apr 28, 2020 at 03:54:01PM -0700, Fangrui Song wrote:
> 
> On 2020-04-28, Nathan Chancellor wrote:
> > Currently, the VDSO is being linked through $(CC). This does not match
> > how the rest of the kernel links objects, which is through the $(LD)
> > variable.
> > 
> > When clang is built in a default configuration, it first attempts to use
> > the target triple's default linker then the system's default linker,
> > unless told otherwise through -fuse-ld=... We do not use -fuse-ld=
> > because it can be brittle and we have support for invoking $(LD)
> > directly. See commit fe00e50b2db8c ("ARM: 8858/1: vdso: use $(LD)
> > instead of $(CC) to link VDSO") and commit 691efbedc60d2 ("arm64: vdso:
> > use $(LD) instead of $(CC) to link VDSO") for examples of doing this in
> > the VDSO.
> > 
> > Do the same thing here. Replace the custom linking logic with $(cmd_ld)
> > and ldflags-y so that $(LD) is respected. We need to explicitly add two
> > flags to the linker that were implicitly passed by the compiler:
> > -G 0 (which comes from ccflags-vdso) and --eh-frame-hdr.
> > 
> > Before this patch (generated by adding '-v' to VDSO_LDFLAGS):
> > 
> > <gcc_prefix>/libexec/gcc/mips64-linux/9.3.0/collect2 \
> > -plugin <gcc_prefix>/libexec/gcc/mips64-linux/9.3.0/liblto_plugin.so \
> > -plugin-opt=<gcc_prefix>/libexec/gcc/mips64-linux/9.3.0/lto-wrapper \
> > -plugin-opt=-fresolution=/tmp/ccGEi5Ka.res \
> > --eh-frame-hdr \
> > -G 0 \
> > -EB \
> > -mips64r2 \
> > -shared \
> > -melf64btsmip \
> > -o arch/mips/vdso/vdso.so.dbg.raw \
> > -L<gcc_prefix>/lib/gcc/mips64-linux/9.3.0/64 \
> > -L<gcc_prefix>/lib/gcc/mips64-linux/9.3.0 \
> > -L<gcc_prefix>/lib/gcc/mips64-linux/9.3.0/../../../../mips64-linux/lib \
> > -Bsymbolic \
> > --no-undefined \
> > -soname=linux-vdso.so.1 \
> > -EB \
> > --hash-style=sysv \
> > --build-id \
> > -T arch/mips/vdso/vdso.lds \
> > arch/mips/vdso/elf.o \
> > arch/mips/vdso/vgettimeofday.o \
> > arch/mips/vdso/sigreturn.o
> > 
> > After this patch:
> > 
> > <gcc_prefix>/bin/mips64-linux-ld \
> > -m elf64btsmip \
> > -Bsymbolic \
> > --no-undefined \
> > -soname=linux-vdso.so.1 \
> > -EB \
> > -nostdlib \
> > -shared \
> > -G 0 \
> > --eh-frame-hdr \
> > --hash-style=sysv \
> > --build-id \
> > -T  arch/mips/vdso/vdso.lds \
> > arch/mips/vdso/elf.o \
> > arch/mips/vdso/vgettimeofday.o
> > arch/mips/vdso/sigreturn.o \
> > -o arch/mips/vdso/vdso.so.dbg.raw
> > 
> > Note that we leave behind -mips64r2. Turns out that ld ignores it (see
> > get_emulation in ld/ldmain.c). This is true of current trunk and 2.23,
> > which is the minimum supported version for the kernel:
> > 
> > https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=ld/ldmain.c;hb=aa4209e7b679afd74a3860ce25659e71cc4847d5#l593
> > https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=ld/ldmain.c;hb=a55e30b51bc6227d8d41f707654d0a5620978dcf#l641
> > 
> > Before this patch, LD=ld.lld did nothing:
> > 
> > $ llvm-readelf -p.comment arch/mips/vdso/vdso.so.dbg | sed 's/(.*//'
> > String dump of section '.comment':
> > [     0] ClangBuiltLinux clang version 11.0.0
> > 
> > After this patch, it does:
> > 
> > $ llvm-readelf -p.comment arch/mips/vdso/vdso.so.dbg | sed 's/(.*//'
> > String dump of section '.comment':
> > [     0] Linker: LLD 11.0.0
> > [    62] ClangBuiltLinux clang version 11.0.0
> > 
> > Link: https://github.com/ClangBuiltLinux/linux/issues/785
> > Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
> > ---
> > 
> > v3 -> v4:
> > 
> > * Improve commit message to show that ld command is effectively the
> >  same as the one generated by GCC.
> > 
> > * Add '-G 0' and '--eh-frame-hdr' because they were added by GCC.
> 
> My understanding is that we start to use more -fasynchronous-unwind-tables to eliminate .eh_frame in object files.
> Without .eh_frame, LD --eh-frame-hdr is really not useful.

Ah, I was not paying attention; I figured that this was necessary
because the x86 VDSO broke without it:

cd01544a268ad ("x86/vdso: Pass --eh-frame-hdr to the linker")

However, they explicitly add -fasynchronous-unwind-tables so it seems
like this indeed can be removed. Kind of odd that GCC passes it along
even with -fno-asynchronous-unwind-tables. I will do that in v5 once I
get some feedback on whether or not anything else breaks.

Cheers,
Nathan

> Sigh...  -G 0. This is an option ignored by LLD. GCC devs probably should
> have used the long option --gpsize rather than take the short option -G.
> Even better, -z gpsize= or similar if this option is specific to ELF.
> > v2 -> v3:
> > 
> > * New patch.
> > 
> > arch/mips/vdso/Makefile | 13 ++++---------
> > 1 file changed, 4 insertions(+), 9 deletions(-)
> > 
> > diff --git a/arch/mips/vdso/Makefile b/arch/mips/vdso/Makefile
> > index 92b53d1df42c3..2e64c7600eead 100644
> > --- a/arch/mips/vdso/Makefile
> > +++ b/arch/mips/vdso/Makefile
> > @@ -60,10 +60,9 @@ ifdef CONFIG_MIPS_DISABLE_VDSO
> > endif
> > 
> > # VDSO linker flags.
> > -VDSO_LDFLAGS := \
> > -	-Wl,-Bsymbolic -Wl,--no-undefined -Wl,-soname=linux-vdso.so.1 \
> > -	$(addprefix -Wl$(comma),$(filter -E%,$(KBUILD_CFLAGS))) \
> > -	-nostdlib -shared -Wl,--hash-style=sysv -Wl,--build-id
> > +ldflags-y := -Bsymbolic --no-undefined -soname=linux-vdso.so.1 \
> > +	$(filter -E%,$(KBUILD_CFLAGS)) -nostdlib -shared \
> > +	-G 0 --eh-frame-hdr --hash-style=sysv --build-id -T
> > 
> > CFLAGS_REMOVE_vdso.o = -pg
> > 
> > @@ -82,11 +81,7 @@ quiet_cmd_vdso_mips_check = VDSOCHK $@
> > #
> > 
> > quiet_cmd_vdsold_and_vdso_check = LD      $@
> > -      cmd_vdsold_and_vdso_check = $(cmd_vdsold); $(cmd_vdso_check); $(cmd_vdso_mips_check)
> > -
> > -quiet_cmd_vdsold = VDSO    $@
> > -      cmd_vdsold = $(CC) $(c_flags) $(VDSO_LDFLAGS) \
> > -                   -Wl,-T $(filter %.lds,$^) $(filter %.o,$^) -o $@
> > +      cmd_vdsold_and_vdso_check = $(cmd_ld); $(cmd_vdso_check); $(cmd_vdso_mips_check)
> > 
> > quiet_cmd_vdsoas_o_S = AS      $@
> >       cmd_vdsoas_o_S = $(CC) $(a_flags) -c -o $@ $<
> > -- 
> > 2.26.2
> >