Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 11 Apr 2023 09:02:52 -0700
In-Reply-To: <ecd3d8de-859b-e5dd-c3bf-ea9c3c0aac60@linux.microsoft.com>
Mime-Version: 1.0
References: <20230227171751.1211786-1-jpiotrowski@linux.microsoft.com>
 <ZAd2MRNLw1JAXmOf@google.com> <959c5bce-beb5-b463-7158-33fc4a4f910c@linux.microsoft.com>
 <ZDSa9Bbqvh0btgQo@google.com> <ecd3d8de-859b-e5dd-c3bf-ea9c3c0aac60@linux.microsoft.com>
Message-ID: <ZDWEgXM/UILjPGiG@google.com>
Subject: Re: [PATCH] KVM: SVM: Disable TDP MMU when running on Hyper-V
From:   Sean Christopherson <seanjc@google.com>
To:     Jeremi Piotrowski <jpiotrowski@linux.microsoft.com>
Cc:     linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Tianyu Lan <ltykernel@gmail.com>,
        Michael Kelley <mikelley@microsoft.com>
Content-Type: text/plain; charset="us-ascii"
Precedence: bulk

On Tue, Apr 11, 2023, Jeremi Piotrowski wrote:
> On 4/11/2023 1:25 AM, Sean Christopherson wrote:
> > On Wed, Apr 05, 2023, Jeremi Piotrowski wrote:
> >> On 3/7/2023 6:36 PM, Sean Christopherson wrote:
> >>> Thinking about this more, I would rather revert commit 1e0c7d40758b ("KVM: SVM:
> >>> hyper-v: Remote TLB flush for SVM") or fix the thing properly straitaway.  KVM
> >>> doesn't magically handle the flushes correctly for the shadow/legacy MMU, KVM just
> >>> happens to get lucky and not run afoul of the underlying bugs.  The revert appears
> >>> to be reasonably straightforward (see bottom).
> >>
> >> Hi Sean,
> >>
> >> I'm back, and I don't have good news. The fix for the missing hyperv TLB flushes has
> >> landed in Linus' tree and I now had the chance to test things outside Azure, in WSL on my
> >> AMD laptop.
> >>
> >> There is some seriously weird interaction going on between TDP MMU and Hyper-V, with
> >> or without enlightened TLB. My laptop has 16 vCPUs, so the WSL VM also has 16 vCPUs.
> >> I have hardcoded the kernel to disable enlightened TLB (so we know that is not interfering).
> >> I'm running a Flatcar Linux VM inside the WSL VM using legacy BIOS, a single CPU
> >> and 4GB of RAM.
> >>
> >> If I run with `kvm.tdp_mmu=0`, I can boot and shutdown my VM consistently in 20 seconds.
> >>
> >> If I run with TDP MMU, the VM boot stalls for seconds at a time in various spots
> >> (loading grub, decompressing kernel, during kernel boot), the boot output feels like
> >> it's happening in slow motion. The fastest I see it finish the same cycle is 2 minutes,
> >> I have also seen it take 4 minutes, sometimes even not finish at all. Same everything,
> >> the only difference is the value of `kvm.tdp_mmu`.
> > 
> > When a stall occurs, can you tell where the time is lost?  E.g. is the CPU stuck
> > in L0, L1, or L2?  L2 being a single vCPU rules out quite a few scenarios, e.g.
> > lock contention and whatnot.
> 
> It shows up as around 90% L2 time, 10% L1 time.

Are those numbers coming from /proc/<pid>/stat?  Something else?

> I don't have great visibility into L0 time right now, I'm trying to find
> someone who might be able to help with that.
> 
> > 
> > If you can run perf in WSL, that might be the easiest way to suss out what's going
> > on.
> 
> I can run perf, what trace would help?

Good question.  I'm not exactly a perf expert and almost never do anything beyond
`perf top`.  That's probably sufficient for now, I really just want to confirm that
L1 doesn't appear to be stuck, e.g. in KVM's page fault handler.

> The results are the same for both branches, and it does look like this affects AMD and
> Intel equally.

Nice.  I have Intel hardware at home that I'll try to repro on, though it will
be several weeks until I can dive into this.

> So seeing as this will likely take a while to figure out (and I know I won't be able to
> spend too many cycles on this in the next few weeks), what do you think of a patch to
> disable tdp_mmu in this configuration (for the time being?).

I don't particularly love the idea of disabling the TDP MMU without having the
slightest clue what's going wrong, but I'm not totally opposed to it.

Paolo, any thoughts?  You have far more experience with supporting downstream
consumers of KVM.

> Something else I've been wondering: in a KVM-on-KVM setup, is tdp_mmu used in both L0
> and L1 hypervisors right now?

By default, yes.  I double checked that L2 has similar boot times for KVM-on-KVM
with and without the TDP MMU.  Certainly nothing remotely close to 2 minutes.