Date:   Fri, 10 Feb 2023 18:45:28 +0000
From:   Sean Christopherson <seanjc@google.com>
To:     Jeremi Piotrowski <jpiotrowski@linux.microsoft.com>
Cc:     Paolo Bonzini <pbonzini@redhat.com>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, Tianyu Lan <ltykernel@gmail.com>,
        "Michael Kelley (LINUX)" <mikelley@microsoft.com>
Subject: Re: "KVM: x86/mmu: Overhaul TDP MMU zapping and flushing" breaks SVM
 on Hyper-V
Message-ID: <Y+aQyFJt9Tn2PJnC@google.com>
References: <43980946-7bbf-dcef-7e40-af904c456250@linux.microsoft.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <43980946-7bbf-dcef-7e40-af904c456250@linux.microsoft.com>
Precedence: bulk

On Fri, Feb 10, 2023, Jeremi Piotrowski wrote:
> Hi Paolo/Sean,
> 
> We've noticed that changes introduced in "KVM: x86/mmu: Overhaul TDP MMU zapping and flushing"
> conflict with a nested Hyper-V enlightenment that is always enabled on AMD CPUs 
> (HV_X64_NESTED_ENLIGHTENED_TLB). The scenario that is affected is L0 Hyper-V + L1 KVM on AMD,
> 
> L2 VMs fail to boot due to to stale data being seen on L1/L2 side, it looks
> like the NPT is not in sync with L0. I can reproduce this on any kernel >=5.18,
> the easiest way is by launching qemu in a loop with debug OVMF, you can observe
> various #GP faults, assert failures, or the guest just suddenly dies. You can try it
> for yourself in Azure by launching an Ubuntu 22.10 image on an AMD SKU with nested
> virtualization (Da_v5).
> 
> In investigating I found that 3 things allow L2 guests to boot again:
> * force tdp_mmu=N when loading kvm
> * recompile L1 kernel to force disable HV_X64_NESTED_ENLIGHTENED_TLB
> * revert both of these commits (found through bisecting):
> bb95dfb9e2dfbe6b3f5eb5e8a20e0259dadbe906 "KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow pages"
> efd995dae5eba57c5d28d6886a85298b390a4f07 "KVM: x86/mmu: Zap defunct roots via asynchronous worker"
> 
> I'll paste our understanding of what is happening (thanks Tianyu):
> """
> Hyper-V provides HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE
> and HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST hvcalls for l1
> hypervisor to notify Hyper-V after L1 hypervisor changes L2 GPA <-> L1 GPA address
> translation tables(Intel calls EPT and AMD calls NPT). This may help not to
> mask whole address translation tables of L1 hypervisor to be write-protected in Hyper-V
> and avoid vmexits triggered by changing address translation table in L1 hypervisor. 
> 
> The following commits defers to call these two hvcalls when there are changes in the L1
> hypervisor address translation table. Hyper-V can't sync/shadow L1 address space
> table at the first time due to the delay and this may cause mismatch between shadow page table
> in the Hyper-V and L1 address translation table. IIRC, KVM side always uses write-protected
> translation table to shadow and so doesn't meet such issue with the commit.
> """
> 
> Let me know if either of you have any ideas on how to approach fixing this.
> I'm not familiar enough with TDP MMU code to be able to contribute a fix directly
> but I'm happy to help in any way I can.

As a hopefully quick-and-easy first step, can you try running KVM built from:

  https://github.com/kvm-x86/linux/tree/mmu

specifically to get the fixes for KVM's usage of range-based TLB flushes:

  https://lore.kernel.org/all/cover.1665214747.git.houwenlong.hwl@antgroup.com