Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220920082005.2459826-1-denik@chromium.org> <877d1yl797.wl-maz@kernel.org>
In-Reply-To: <877d1yl797.wl-maz@kernel.org>
From:   Denis Nikitin <denik@google.com>
Date:   Tue, 20 Sep 2022 17:08:09 -0700
Message-ID: <CAOYpmdHMbDdssEJJwXktEj1SDLncHTeL7x7aKxVe6j1vzSMgtw@mail.gmail.com>
Subject: Re: [PATCH] KVM: arm64: nvhe: Disable profile optimization
To:     Marc Zyngier <maz@kernel.org>
Cc:     Denis Nikitin <denik@chromium.org>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        James Morse <james.morse@arm.com>,
        Alexandru Elisei <alexandru.elisei@arm.com>,
        Nick Desaulniers <ndesaulniers@google.com>,
        Manoj Gupta <manojgupta@google.com>,
        David Brazdil <dbrazdil@google.com>,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu,
        linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Hi Mark,

Thank you for a quick response.

On Tue, Sep 20, 2022 at 2:34 AM Marc Zyngier <maz@kernel.org> wrote:
>
> Hi Denis,
>
> On Tue, 20 Sep 2022 09:20:05 +0100,
> Denis Nikitin <denik@chromium.org> wrote:
> >
> > Kernel build with -fprofile-sample-use raises the following failure:
> >
> > error: arch/arm64/kvm/hyp/nvhe/kvm_nvhe.tmp.o: Unexpected SHT_REL
> > section ".rel.llvm.call-graph-profile"
>
> How is this flag provided? I don't see any occurrence of it in the
> kernel so far.

On ChromeOS we build the kernel with sample profiles by adding
-fprofile-sample-use=/path/to/gcov.profile to KCFLAGS.

>
> >
> > SHT_REL is generated by the latest lld, see
> > https://reviews.llvm.org/rGca3bdb57fa1ac98b711a735de048c12b5fdd8086.
>
> Is this part of a released toolchain? If so, can you spell out the
> first version where this occurs?

Yes, it was added in llvm-13. I will update the patch.

>
> > Disable profile optimization in kvm/nvhe to fix the build with
> > AutoFDO.
>
> It'd be good to at least mention how AutoFDO and -fprofile-sample-use
> relate to each other.

Good point. AutoFDO is an example of sample profiles.
It's not actually relevant for the bug. I will better remove it.

>
> >
> > Signed-off-by: Denis Nikitin <denik@chromium.org>
> > ---
> >  arch/arm64/kvm/hyp/nvhe/Makefile | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
> > index b5c5119c7396..6a6188374a52 100644
> > --- a/arch/arm64/kvm/hyp/nvhe/Makefile
> > +++ b/arch/arm64/kvm/hyp/nvhe/Makefile
> > @@ -89,6 +89,9 @@ quiet_cmd_hypcopy = HYPCOPY $@
> >  # Remove ftrace, Shadow Call Stack, and CFI CFLAGS.
> >  # This is equivalent to the 'notrace', '__noscs', and '__nocfi' annotations.
> >  KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_CFI), $(KBUILD_CFLAGS))
> > +# Profile optimization creates SHT_REL section '.llvm.call-graph-profile' for
> > +# the hot code. SHT_REL is currently not supported by the KVM tools.
>
> 'KVM tools' seems vague. Maybe call out the actual helper that
> processes the relocations?

Agreed.

>
> > +KBUILD_CFLAGS += $(call cc-option,-fno-profile-sample-use,-fno-profile-use)
>
> Why adding these options instead of filtering out the offending option
> as it is done just above?

That was actually the alternative solution and it worked as well.
Let me double check if profile optimization doesn't mess up with other
sections and if it doesn't I will remove the '.llvm.call-graph-profile'
section instead.

>
> Also, is this the only place the kernel fails to compile? The EFI stub
> does similar things AFAIR, and could potentially fail the same way.

This was the only place in 5.15 where we tested it.
Let me see if EFI has this section.

>
> Thanks,
>
>         M.
>
> --
> Without deviation from the norm, progress is not possible.

Thanks,
Denis