Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <41b1e63ad6e45be019bbedc93bd18cfcb9475b06.camel@redhat.com>
Subject: Re: [PATCH v2 11/12] KVM: SVM: Do not inhibit APICv when x2APIC is
 present
From:   Maxim Levitsky <mlevitsk@redhat.com>
To:     Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org
Cc:     pbonzini@redhat.com, seanjc@google.com, joro@8bytes.org,
        jon.grimm@amd.com, wei.huang2@amd.com, terry.bowman@amd.com
Date:   Tue, 26 Apr 2022 12:56:56 +0300
In-Reply-To: <b9ee5f62e904a690d7e2d8837ade8ece7e24a359.camel@redhat.com>
References: <20220412115822.14351-1-suravee.suthikulpanit@amd.com>
         <20220412115822.14351-12-suravee.suthikulpanit@amd.com>
         <3fd0aabb6288a5703760da854fd6b09a485a2d69.camel@redhat.com>
         <01460b72-1189-fef1-9718-816f2f658d42@amd.com>
         <b9ee5f62e904a690d7e2d8837ade8ece7e24a359.camel@redhat.com>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) 
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Tue, 2022-04-26 at 10:06 +0300, Maxim Levitsky wrote:
> On Tue, 2022-04-26 at 09:25 +0700, Suravee Suthikulpanit wrote:
> > Hi Maim,
> > 
> > On 4/19/22 8:29 PM, Maxim Levitsky wrote:
> > > On Tue, 2022-04-12 at 06:58 -0500, Suravee Suthikulpanit wrote:
> > > 
> > > Hi!
> > > 
> > > 
> > > I just got an idea, while writing a kvm selftest that would use AVIC,
> > > and finding out that selftest code uploads the '-host' cpuid right away
> > > which has x2apic enabled and that inhibits AVIC, and later clearing x2apic
> > > in the cpuid doesn't un-inhibit it.
> > >   
> > > That can be fixed in few ways but that got me thinking:
> > >   
> > > Why do we inhibit AVIC when the guest uses x2apic, even without X2AVIC?
> > > I think that if we didn't it would just work, and even work faster than
> > > pure software x2apic.
> > >   
> > > My thinking is:
> > >   
> > > - when a vcpu itself uses its x2apic, even if its avic is not inhibited,
> > > the guest will write x2apic msrs which kvm intercepts and will correctly emulate a proper x2apic.
> > >   
> > > - vcpu peers will also use x2apic msrs and again it will work correctly
> > > (even when there are more than 256 vcpus).
> > >   
> > > - and the host + iommu will still be able to use AVIC's doorbell to send interrupts to the guest
> > > and that doesn't need apic ids or anything, it should work just fine.
> > > 
> > > Also AVIC should have no issues scanning IRR and injecting interrupts on VM entry,
> > > x2apic mode doesn't matter for that.
> > >   
> > > AVIC mmio can still be though discovered by the guest which is technically against x86 spec
> > > (in x2apic mode, mmio supposed to not work) but that can be fixed easily by disabing
> > > the AVIC memslot if any of the vCPUs are in x2apic mode, or this can be ignored since
> > > it should not cause any issues.
> > > We seem to have a quirk for that KVM_X86_QUIRK_LAPIC_MMIO_HOLE.
> > >   
> > > On top of all this, removing this inhibit will also allow to test AVIC with guest
> > > which does have x2apic in the CPUID but doesn't use it (e.g kvm unit test, or
> > > linux booted with nox2apic, which is also nice IMHO)
> > >   
> > > What do you think?
> > 
> > This is actually a good idea!!! Let's call it hybrid-x2AVIC :)
> > 
> > I am working on prototype and test out the support for this, which will be introduced in V3.
> 
> Thanks! 
> 
> Best regards,
> 	Maxim Levitsky
> 
> > Regards,
> > Suravee
> > 

BTW, can I ask you to check something on the AMD side of things of AVIC?

I noticed that AMD's manual states that:

"Multiprocessor VM requirements. When running a VM which has multiple virtual CPUs, and the
VMM runs a virtual CPU on a core which had last run a different virtual CPU from the same VM,
regardless of the respective ASID values, care must be taken to flush the TLB on the VMRUN using a
TLB_CONTROL value of 3h. Failure to do so may result in stale mappings misdirecting virtual APIC
accesses to the previous virtual CPU's APIC backing page."

It it relevant to KVM? I don't fully understand why it was mentioned that ASID doesn't matter,
what makes it special about 'virtual CPU from the same VM' if ASID doesn't matter? 

Also, is this still relevant on modern AMD cpus, or was a workaround for some old CPU bug?

Best regards,
	Maxim Levitsky