Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
In-Reply-To: <20171113104634.62cmjalnj73scfnl@hirez.programming.kicks-ass.net>
References: <1510533206-9821-1-git-send-email-wanpeng.li@hotmail.com>
 <1510533206-9821-3-git-send-email-wanpeng.li@hotmail.com> <20171113080403.ici7nt5r4b5fpxa6@hirez.programming.kicks-ass.net>
 <CANRm+Cw0x4dFUrvCYFeeCHRibd--0ZVmEcr-4aoKiMiOh9TKUA@mail.gmail.com> <20171113104634.62cmjalnj73scfnl@hirez.programming.kicks-ass.net>
From:   Wanpeng Li <kernellwp@gmail.com>
Date:   Tue, 14 Nov 2017 14:28:56 +0800
Message-ID: <CANRm+Cyn4uv6JqWWMc+aGAPUyB4sj4NiHkBM_6BbFMnpoC+6yg@mail.gmail.com>
Subject: Re: [PATCH v4 2/4] KVM: X86: Add paravirt remote TLB flush
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        kvm <kvm@vger.kernel.org>, Paolo Bonzini <pbonzini@redhat.com>,
        "Radim Kr??m????" <rkrcmar@redhat.com>,
        Wanpeng Li <wanpeng.li@hotmail.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

2017-11-13 18:46 GMT+08:00 Peter Zijlstra <peterz@infradead.org>:
> On Mon, Nov 13, 2017 at 04:26:57PM +0800, Wanpeng Li wrote:
>> 2017-11-13 16:04 GMT+08:00 Peter Zijlstra <peterz@infradead.org>:
>
>> > So if at this point a vCPU gets preempted we'll still spin-wait for it,
>> > which is sub-optimal.
>> >
>> > I think we can come up with something to get around that 'problem' if
>> > indeed it is a problem. But we can easily do that as follow up patches.
>> > Just let me know if you think its worth spending more time on.
>>
>> You can post your idea, it is always smart. :) Then we can evaluate
>> the complexity and gains.
>
> I'm not sure I have a fully baked idea just yet, but the general idea
> would be something like:
>
>  - switch (back) to a dedicated TLB invalidate IPI
>
>  - introduce KVM_VCPU_IPI_PENDING
>
>  - change flush_tlb_others() into something like:
>
>    for_each_cpu(cpu, flushmask) {
>          src = &per_cpu(steal_time, cpu);
>          state = READ_ONCE(src->preempted);
>          do {
>                  if (state & KVM_VCPU_PREEMPTED) {
>                          if (try_cmpxchg(&src->preempted, &state,
>                                                  state | KVM_VCPU_SHOULD_FLUSH)) {
>                                  __cpumask_clear_cpu(cpu, flushmask);
>                                  break;
>                          }
>                  }
>          } while (!try_cmpxchg(&src->preempted, &state,
>                                  state | KVM_VCPU_IPI_PENDING));
>    }
>
>    apic->send_IPI_mask(flushmask, CALL_TLB_VECTOR);
>
>    for_each_cpu(cpu, flushmask) {
>          src = &per_cpu(steal_time, cpu);
>          smp_cond_load_acquire(&src->preempted, !(VAL & KVM_VCPU_IPI_PENDING);
>    }
>
>
>  - have the TLB invalidate handler do something like:
>
>    state = READ_ONCE(src->preempted);
>    if (!(state & KVM_VCPU_IPI_PENDING))
>            return;
>
>    local_flush_tlb();
>
>    do {
>    } while (!try_cmpxchg(&src->preempted, &state,
>                          state & ~KVM_VCPU_IPI_PENDING));

There are a lot of cases handled by flush_tlb_func_remote() ->
flush_tlb_function_common(), so I'm afraid to have hole.

Regards,
Wanpeng Li

>
>  - then at VMEXIT time do something like:
>
>    state = READ_ONCE(src->preempted);
>    do {
>         if (!(state & KVM_VCPU_IPI_PENDING))
>                 break;
>    } while (!try_cmpxchg(&src->preempted, state,
>                          (state & ~KVM_VCPU_IPI_PENDING) |
>                          KVM_VCPU_SHOULD_FLUSH));
>
>    and clear any possible pending TLB_VECTOR in the guest state to avoid
>    raising that IPI spuriously on enter again.
>
>
> This way the preemption will clear the IPI_PENDING and the
> flush_others() wait loop will terminate.

From 1584020853795060032@xxx Tue Nov 14 06:13:24 +0000 2017
X-GM-THRID: 1583909069046568739
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread