Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <1544083089-13000-1-git-send-email-wanpengli@tencent.com> <20181220144345.GB19579@flask>
In-Reply-To: <20181220144345.GB19579@flask>
From:   Wanpeng Li <kernellwp@gmail.com>
Date:   Fri, 21 Dec 2018 08:46:32 +0800
Message-ID: <CANRm+Cx4HKfXDpCUtkScjQ-wzCBCpx91twBodkqDRD7U1rZBHg@mail.gmail.com>
Subject: Re: [PATCH] KVM: MMU: Introduce single thread to zap collapsible sptes
To:     =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= <rkrcmar@redhat.com>
Cc:     LKML <linux-kernel@vger.kernel.org>, kvm <kvm@vger.kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, 20 Dec 2018 at 22:43, Radim Kr=C4=8Dm=C3=A1=C5=99 <rkrcmar@redhat.c=
om> wrote:
>
> 2018-12-06 15:58+0800, Wanpeng Li:
> > From: Wanpeng Li <wanpengli@tencent.com>
> >
> > Last year guys from huawei reported that the call of memory_global_dirt=
y_log_start/stop()
> > takes 13s for 4T memory and cause guest freeze too long which increases=
 the unacceptable
> > migration downtime. [1] [2]
> >
> > Guangrong pointed out:
> >
> > | collapsible_sptes zaps 4k mappings to make memory-read happy, it is n=
ot
> > | required by the semanteme of KVM_SET_USER_MEMORY_REGION and it is not
> > | urgent for vCPU's running, it could be done in a separate thread and =
use
> > | lock-break technology.
> >
> > [1] https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg05249.html
> > [2] https://www.mail-archive.com/qemu-devel@nongnu.org/msg449994.html
> >
> > Several TB memory guest is common now after NVDIMM is deployed in cloud=
 environment.
> > This patch utilizes worker thread to zap collapsible sptes in order to =
lazy collapse
> > small sptes into large sptes during roll-back after live migration fail=
s.
> >
> > Cc: Paolo Bonzini <pbonzini@redhat.com>
> > Cc: Radim Kr=C4=8Dm=C3=A1=C5=99 <rkrcmar@redhat.com>
> > Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
> > ---
> > @@ -5679,14 +5679,41 @@ static bool kvm_mmu_zap_collapsible_spte(struct=
 kvm *kvm,
> >       return need_tlb_flush;
> >  }
> >
> > +void zap_collapsible_sptes_fn(struct work_struct *work)
> > +{
> > +     struct kvm_memory_slot *memslot;
> > +     struct kvm_memslots *slots;
> > +     struct delayed_work *dwork =3D to_delayed_work(work);
> > +     struct kvm_arch *ka =3D container_of(dwork, struct kvm_arch,
> > +                                        kvm_mmu_zap_collapsible_sptes_=
work);
> > +     struct kvm *kvm =3D container_of(ka, struct kvm, arch);
> > +     int i;
> > +
> > +     mutex_lock(&kvm->slots_lock);
> > +     for (i =3D 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
> > +             spin_lock(&kvm->mmu_lock);
> > +             slots =3D __kvm_memslots(kvm, i);
> > +             kvm_for_each_memslot(memslot, slots) {
> > +                     slot_handle_leaf(kvm, (struct kvm_memory_slot *)m=
emslot,
> > +                             kvm_mmu_zap_collapsible_spte, true);
> > +                     if (need_resched() || spin_needbreak(&kvm->mmu_lo=
ck))
> > +                             cond_resched_lock(&kvm->mmu_lock);
>
> I think we shouldn't zap all memslots when kvm_mmu_zap_collapsible_sptes
> only wanted to zap a specific one.
> Please add a list of memslots to be zapped; delete from the list here
> and add in kvm_mmu_zap_collapsible_sptes().

Yeah, that's my original plan, however, i observe a lot of races here,
the memslot can disappear/modify underneath before the worker thread
start to zap even if i introduce lock to protect the list. This patch
delays the worker thread by 60s(to assume memory_global_dirty_log_stop
can absolutely complete) to coalesce all the zap requirements after
live migration fails.

Regards,
Wanpeng Li

>
> > +             }
> > +             spin_unlock(&kvm->mmu_lock);
> > +     }
> > +     kvm->arch.zap_in_progress =3D false;
> > +     mutex_unlock(&kvm->slots_lock);
> > +}
> > +
> > +#define KVM_MMU_ZAP_DELAYED (60 * HZ)
> >  void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm,
> >                                  const struct kvm_memory_slot *memslot)
> >  {
> > -     /* FIXME: const-ify all uses of struct kvm_memory_slot.  */
> > -     spin_lock(&kvm->mmu_lock);
> > -     slot_handle_leaf(kvm, (struct kvm_memory_slot *)memslot,
> > -                      kvm_mmu_zap_collapsible_spte, true);
> > -     spin_unlock(&kvm->mmu_lock);
> > +     if (!kvm->arch.zap_in_progress) {
>
> The list can also serve in place of zap_in_progress -- if there were any
> elements in it, then there is no need to schedule the work again.
>
> Thanks.