Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 22 Sep 2022 11:37:55 +0100
Message-ID: <87h70zk83g.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Denis Nikitin <denik@chromium.org>
Cc:     Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        James Morse <james.morse@arm.com>,
        Alexandru Elisei <alexandru.elisei@arm.com>,
        Nick Desaulniers <ndesaulniers@google.com>,
        Manoj Gupta <manojgupta@google.com>,
        David Brazdil <dbrazdil@google.com>,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] KVM: arm64: nvhe: Fix build with profile optimization
In-Reply-To: <20220922053145.944786-1-denik@chromium.org>
References: <20220920082005.2459826-1-denik@chromium.org>
        <20220922053145.944786-1-denik@chromium.org>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Thu, 22 Sep 2022 06:31:45 +0100,
Denis Nikitin <denik@chromium.org> wrote:
> 
> Kernel build with clang and KCFLAGS=-fprofile-sample-use fails with:
> 
> error: arch/arm64/kvm/hyp/nvhe/kvm_nvhe.tmp.o: Unexpected SHT_REL
> section ".rel.llvm.call-graph-profile"
> 
> Starting from 13.0.0 llvm can generate SHT_REL section, see
> https://reviews.llvm.org/rGca3bdb57fa1ac98b711a735de048c12b5fdd8086.
> gen-hyprel does not support SHT_REL relocation section.
> 
> Remove ".llvm.call-graph-profile" SHT_REL relocation from kvm_nvhe
> to fix the build.
> 
> Signed-off-by: Denis Nikitin <denik@chromium.org>
> ---
> V1 -> V2: Remove the relocation instead of disabling the profile-use flags.
> ---
>  arch/arm64/kvm/hyp/nvhe/Makefile | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
> index b5c5119c7396..49ec950ac57b 100644
> --- a/arch/arm64/kvm/hyp/nvhe/Makefile
> +++ b/arch/arm64/kvm/hyp/nvhe/Makefile
> @@ -78,8 +78,10 @@ $(obj)/kvm_nvhe.o: $(obj)/kvm_nvhe.rel.o FORCE
>  
>  # The HYPREL command calls `gen-hyprel` to generate an assembly file with
>  # a list of relocations targeting hyp code/data.
> +# Starting from 13.0.0 llvm emits SHT_REL section '.llvm.call-graph-profile'
> +# when profile optimization is applied. gen-hyprel does not support SHT_REL.
>  quiet_cmd_hyprel = HYPREL  $@
> -      cmd_hyprel = $(obj)/gen-hyprel $< > $@
> +	cmd_hyprel = $(OBJCOPY) -R .llvm.call-graph-profile $<; $(obj)/gen-hyprel $< > $@

I was really hoping that you'd just drop the flags from the CFLAGS
instead of removing the generated section. Something like:

diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index b5c5119c7396..e5b2d43925b4 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -88,7 +88,7 @@ quiet_cmd_hypcopy = HYPCOPY $@
 
 # Remove ftrace, Shadow Call Stack, and CFI CFLAGS.
 # This is equivalent to the 'notrace', '__noscs', and '__nocfi' annotations.
-KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_CFI), $(KBUILD_CFLAGS))
+KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_CFI) -fprofile-sample-use, $(KBUILD_CFLAGS))
 
 # KVM nVHE code is run at a different exception code with a different map, so
 # compiler instrumentation that inserts callbacks or checks into the code may

However, I even failed to reproduce your problem using LLVM 14 as
packaged by Debian (if that matters, I'm using an arm64 build
machine). I build the kernel with:

$ make LLVM=1 KCFLAGS=-fprofile-sample-use -j8 vmlinux

and the offending object only contains the following sections:

arch/arm64/kvm/hyp/nvhe/kvm_nvhe.tmp.o:     file format elf64-littleaarch64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .hyp.idmap.text 00000ae4  0000000000000000  0000000000000000  00000800  2**11
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .hyp.text     0000e988  0000000000000000  0000000000000000  00001800  2**11
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  2 .hyp.data..ro_after_init 00000820  0000000000000000  0000000000000000  00010188  2**3
                  CONTENTS, ALLOC, LOAD, DATA
  3 .hyp.rodata   00002e70  0000000000000000  0000000000000000  000109a8  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
  4 .hyp.data..percpu 00001ee0  0000000000000000  0000000000000000  00013820  2**4
                  CONTENTS, ALLOC, LOAD, DATA
  5 .hyp.bss      00001158  0000000000000000  0000000000000000  00015700  2**3
                  ALLOC
  6 .comment      0000001f  0000000000000000  0000000000000000  00017830  2**0
                  CONTENTS, READONLY
  7 .llvm_addrsig 000000b8  0000000000000000  0000000000000000  0001784f  2**0
                  CONTENTS, READONLY, EXCLUDE
  8 .altinstructions 00001284  0000000000000000  0000000000000000  00015700  2**0
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
  9 __jump_table  00000960  0000000000000000  0000000000000000  00016988  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, DATA
 10 __bug_table   0000051c  0000000000000000  0000000000000000  000172e8  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, DATA
 11 __kvm_ex_table 00000028  0000000000000000  0000000000000000  00017808  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
 12 .note.GNU-stack 00000000  0000000000000000  0000000000000000  00027370  2**0
                  CONTENTS, READONLY

So what am I missing to trigger this issue? Does it rely on something
like PGO, which is not upstream yet? A bit of handholding would be
much appreciated.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.