Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 15 Aug 2022 10:05:50 -0700
From:   Nathan Chancellor <nathan@kernel.org>
To:     Vitaly Kuznetsov <vkuznets@redhat.com>
Cc:     kernel test robot <lkp@intel.com>, kvm@vger.kernel.org,
        llvm@lists.linux.dev, kbuild-all@lists.01.org,
        linux-kernel@vger.kernel.org,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Paolo Bonzini <pbonzini@redhat.com>
Subject: Re: [stable:linux-5.15.y 5373/9027] arch/x86/kvm/hyperv.c:2185:5:
 warning: stack frame size (1036) exceeds limit (1024) in 'kvm_hv_hypercall'
Message-ID: <Yvp87jlVWg0e376v@dev-arch.thelio-3990X>
References: <202208142025.NHKErAjq-lkp@intel.com>
 <87zgg6sza8.fsf@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87zgg6sza8.fsf@redhat.com>
Precedence: bulk

On Mon, Aug 15, 2022 at 10:10:07AM +0200, Vitaly Kuznetsov wrote:
> kernel test robot <lkp@intel.com> writes:
> 
> > Hi Vitaly,
> >
> > FYI, the error/warning still remains.
> >
> 
> Yes, this is expected as the patch which is supposed to 'fix' this is
> still pendind. The latest version is here:
> 
> https://lore.kernel.org/kvm/20220803134540.399220-1-vkuznets@redhat.com/
> 
> ...
> 
> > tree:   https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux-5.15.y
> > head:   7217df81279835a7aee62a07aabb7b8fb8c766f2
> > commit: cb188e07105f2216f5efbefac95df4b6ce266906 [5373/9027] KVM: x86: hyper-v: HVCALL_SEND_IPI_EX is an XMM fast hypercall
> > config: i386-allmodconfig (https://download.01.org/0day-ci/archive/20220814/202208142025.NHKErAjq-lkp@intel.com/config)
> > compiler: clang version 16.0.0 (https://github.com/llvm/llvm-project 52cd00cabf479aa7eb6dbb063b7ba41ea57bce9e)
> > reproduce (this is a W=1 build):
> >         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
> >         chmod +x ~/bin/make.cross
> >         # https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=cb188e07105f2216f5efbefac95df4b6ce266906
> >         git remote add stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> >         git fetch --no-tags stable linux-5.15.y
> >         git checkout cb188e07105f2216f5efbefac95df4b6ce266906
> >         # save the config file
> >         mkdir build_dir && cp config build_dir/.config
> >         COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=i386 SHELL=/bin/bash arch/x86/kvm/
> >
> > If you fix the issue, kindly add following tag where applicable
> > Reported-by: kernel test robot <lkp@intel.com>
> >
> > All warnings (new ones prefixed by >>):
> >
> >>> arch/x86/kvm/hyperv.c:2185:5: warning: stack frame size (1036) exceeds limit (1024) in 'kvm_hv_hypercall' [-Wframe-larger-than]
> >    int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
> >        ^
> >    1 warning generated.
> >
> >
> > vim +/kvm_hv_hypercall +2185 arch/x86/kvm/hyperv.c
> >
> > 4ad81a91119df7 Vitaly Kuznetsov         2021-05-21  2184  
> > e83d58874ba1de Andrey Smetanin          2015-07-03 @2185  int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
> > e83d58874ba1de Andrey Smetanin          2015-07-03  2186  {
> > 4e62aa96d6e55c Vitaly Kuznetsov         2021-07-30  2187  	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
> > bd38b32053eb1c Siddharth Chandrasekaran 2021-05-26  2188  	struct kvm_hv_hcall hc;
> > bd38b32053eb1c Siddharth Chandrasekaran 2021-05-26  2189  	u64 ret = HV_STATUS_SUCCESS;
> 
> ... but let me repeat myself: (see my previous reply here:
> https://lore.kernel.org/kvm/874jyw2v5n.fsf@redhat.com/)
> The source of the problem seems to be that Clang probably inlines
> kvm_hv_send_ipi() as on-stack variables in kvm_hv_hypercall() can not
> exceed 1024 bytes limit (struct kvm_hv_hcall is 144 bytes, the rest is
> negligible). The patch I mention above will likely fix the issue as it
> significantly reduces on-stack allocations in kvm_hv_send_ipi() but in
> this situation it shouldn't be inlined in the first place.

I seem to recall Nick mentioning at some point that LLVM gives a massive
inlining discount to functions that are only called once so I guess that
would explain why kvm_hv_send_ipi() gets inlined into
kvm_hv_hypercall(). I think there are some inlining cost flags we could
experiment with to see if we can avoid inlining functions with high
stack usage into other functions. Additionally, the configurations that
reported this warning are allmodconfig and allyesconfig, which enable
KASAN, which is known to use more stack usage with clang:

https://github.com/ClangBuiltLinux/linux/issues/39

I am hoping we'll get to addressing that during our meet up right before
Plumbers but we'll see.

> (I still hope that I'm wrong finger pointing at the compiler here and
> someone smart will come to correct me :-)

Other folks more familiar with the LLVM side of things might correct me
:)

Cheers,
Nathan