Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <1582624061-5814-1-git-send-email-wanpengli@tencent.com> <0af6b96a-16ac-5054-7754-6ab4a239a2d4@redhat.com>
In-Reply-To: <0af6b96a-16ac-5054-7754-6ab4a239a2d4@redhat.com>
From:   Wanpeng Li <kernellwp@gmail.com>
Date:   Tue, 25 Feb 2020 22:26:19 +0800
Message-ID: <CANRm+Cz7mKjm7_9H4O4y2XYEv8VnErTtu=dr-8fN0RCCjf0wvA@mail.gmail.com>
Subject: Re: [PATCH v2] KVM: LAPIC: Recalculate apic map in batch
To:     Paolo Bonzini <pbonzini@redhat.com>
Cc:     LKML <linux-kernel@vger.kernel.org>, kvm <kvm@vger.kernel.org>,
        Sean Christopherson <sean.j.christopherson@intel.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Jim Mattson <jmattson@google.com>,
        Joerg Roedel <joro@8bytes.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, 25 Feb 2020 at 22:20, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 25/02/20 10:47, Wanpeng Li wrote:
> > From: Wanpeng Li <wanpengli@tencent.com>
> >
> > In the vCPU reset and set APIC_BASE MSR path, the apic map will be recalculated
> > several times, each time it will consume 10+ us observed by ftrace in my
> > non-overcommit environment since the expensive memory allocate/mutex/rcu etc
> > operations. This patch optimizes it by recaluating apic map in batch, I hope
> > this can benefit the serverless scenario which can frequently create/destroy
> > VMs.
> >
> > Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
> > ---
> > v1 -> v2:
> >  * add apic_map_dirty to kvm_lapic
> >  * error condition in kvm_apic_set_state, do recalcuate  unconditionally
> >
> >  arch/x86/kvm/lapic.c | 29 +++++++++++++++++++----------
> >  arch/x86/kvm/lapic.h |  2 ++
> >  arch/x86/kvm/x86.c   |  2 ++
> >  3 files changed, 23 insertions(+), 10 deletions(-)
> >
> > diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> > index afcd30d..3476dbc 100644
> > --- a/arch/x86/kvm/lapic.c
> > +++ b/arch/x86/kvm/lapic.c
> > @@ -164,7 +164,7 @@ static void kvm_apic_map_free(struct rcu_head *rcu)
> >       kvfree(map);
> >  }
> >
> > -static void recalculate_apic_map(struct kvm *kvm)
> > +void kvm_recalculate_apic_map(struct kvm *kvm)
> >  {
>
> It's better to add an "if" here rather than in every caller.  It should
> be like:
>
>         if (!apic->apic_map_dirty) {
>                 /*
>                  * Read apic->apic_map_dirty before
>                  * kvm->arch.apic_map.
>                  */
>                 smp_rmb();
>                 return;
>         }
>
>         mutex_lock(&kvm->arch.apic_map_lock);
>         if (!apic->apic_map_dirty) {
>                 /* Someone else has updated the map.  */
>                 mutex_unlock(&kvm->arch.apic_map_lock);
>                 return;
>         }
>         ...
> out:
>         old = rcu_dereference_protected(kvm->arch.apic_map,
>                         lockdep_is_held(&kvm->arch.apic_map_lock));
>         rcu_assign_pointer(kvm->arch.apic_map, new);
>         /*
>          * Write kvm->arch.apic_map before
>          * clearing apic->apic_map_dirty.
>          */
>         smp_wmb();
>         apic->apic_map_dirty = false;
>         mutex_unlock(&kvm->arch.apic_map_lock);
>         ...
>
> But actually it seems to me that, given we're going through all this
> pain, it's better to put the "dirty" flag in kvm->arch, next to the
> mutex and the map itself.  This should also reduce the number of calls
> to kvm_recalculate_apic_map that recompute the map.  A lot of them will
> just wait on the mutex and exit.

Good point, will do in next version.

    Wanpeng