Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 31 Aug 2022 18:19:01 +0000
From:   Sean Christopherson <seanjc@google.com>
To:     Maxim Levitsky <mlevitsk@redhat.com>
Cc:     Paolo Bonzini <pbonzini@redhat.com>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        Li RongQing <lirongqing@baidu.com>
Subject: Re: [PATCH 17/19] KVM: SVM: Handle multiple logical targets in AVIC
 kick fastpath
Message-ID: <Yw+mFbuih3rBjMV8@google.com>
References: <20220831003506.4117148-1-seanjc@google.com>
 <20220831003506.4117148-18-seanjc@google.com>
 <ca3be5f88268f1547e6f02b01a472186566066c5.camel@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ca3be5f88268f1547e6f02b01a472186566066c5.camel@redhat.com>
Precedence: bulk

On Wed, Aug 31, 2022, Maxim Levitsky wrote:
> On Wed, 2022-08-31 at 00:35 +0000, Sean Christopherson wrote:
> > +static void avic_kick_vcpu_by_logical_id(struct kvm *kvm, u32 *avic_logical_id_table,
> > +					 u32 logid_index, u32 icrl)
> > +{
> > +	u32 physical_id;
> > +
> > +	if (!avic_logical_id_table) {
> ^ Typo, the '!' shoudn't be there.

Ouch.  I suspect the tests pass because this just ends up routing events through
the slow path.  I try to concoct a testcase to expose this bug.

> > +static bool is_optimized_logical_map_enabled(struct kvm *kvm)
> > +{
> > +	struct kvm_apic_map *map;
> > +	bool enabled;
> > +
> > +	rcu_read_lock();
> > +	map = rcu_dereference(kvm->arch.apic_map);
> > +	enabled = map && map->logical_mode != KVM_APIC_MODE_MAP_DISABLED;
> > +	rcu_read_unlock();
> > +	return enabled;
> > +}
> 
> This function doesn't belong to avic, it should be in common KVM code.

I'll move it.  I'm not expecting any additional users, but I agree it belongs
elsewhere.  Actually, might be a moot point (see below).

> > @@ -394,50 +449,27 @@ static int avic_kick_target_vcpus_fast(struct kvm *kvm, struct kvm_lapic *source
> >  		if (unlikely(!bitmap))
> >  			return 0;
> >  
> > -		if (!is_power_of_2(bitmap))
> > -			/* multiple logical destinations, use slow path */
> > +		/*
> > +		 * Use the slow path if more than one bit is set in the bitmap
> > +		 * and KVM's optimized logical map is disabled to avoid kicking
> > +		 * a vCPU multiple times.  If the optimized map is disabled, a
> > +		 * vCPU _may_ have multiple bits set in its logical ID, i.e.
> > +		 * may have multiple entries in the logical table.
> > +		 */
> > +		if (!is_power_of_2(bitmap) &&
> > +		    !is_optimized_logical_map_enabled(kvm))
> >  			return -EINVAL;
> 
> 
> I hate to say it but there is another issue here, which I know about for a while
> but haven't gotten yet to fix.
> 
> The issue is that AVIC's logical to physical map can't cover all the corner cases
> that you discovered - it only supports the sane subset: for each cluster, and for each bit
> in the mask, it has a physical apic id - so things like logical ids with multiple bits,
> having same logical id for multiple vcpus and so on can't work.
> 
> In this case we need to either inhibit AVIC (I support this 100%),

I like the idea of inhibiting.

>  or clear its logical ID map, so all logicical IPIs VM exit, and then they
>  can be emulated.
> 
> I haven't studied it formally but the code which rebuilds the AVIC's logical ID map
> starts at 'avic_handle_ldr_update'.

I suspected there are issues here, but the new tests passed (somewhat surprisingly)
so I stopped trying to decipher the AVIC LDR handling.

Eww.  And the VM-Exit trap logic is broken too.  If the guest updates and disables
its LDR, SVM returns immediately and doesn't call into common APIC code, i.e. doesn't
recalc the optimized map.  E.g. if the guest clears its LDR, the optimized map will
be left as is and the vCPU will receive interrupts using its old LDR.

	case APIC_LDR:
		if (avic_handle_ldr_update(vcpu))
			return 0;
		break;

Rather than handling this purely in AVIC code, what if we a key off of
the optimized map being enabled?  E.g. drop the return from avic_handle_ldr_update()
and in the kvm_recalculate_apic_map() do:

diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 3b6ef36b3963..6e188010b614 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -364,6 +364,11 @@ void kvm_recalculate_apic_map(struct kvm *kvm)
                cluster[ldr] = apic;
        }
 out:
+       if (!new || new->logical_mode == KVM_APIC_MODE_MAP_DISABLED)
+               kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_LOGICAL_MAP_DISABLED);
+       else
+               kvm_clear_apicv_inhibit(kvm, APICV_INHIBIT_REASON_LOGICAL_MAP_DISABLED);
+
        old = rcu_dereference_protected(kvm->arch.apic_map,
                        lockdep_is_held(&kvm->arch.apic_map_lock));
        rcu_assign_pointer(kvm->arch.apic_map, new);