Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH] KVM: Boost vCPU candidiate in user mode which is
 delivering interrupt
To:     Wanpeng Li <kernellwp@gmail.com>
Cc:     Sean Christopherson <seanjc@google.com>,
        LKML <linux-kernel@vger.kernel.org>, kvm <kvm@vger.kernel.org>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Jim Mattson <jmattson@google.com>,
        Joerg Roedel <joro@8bytes.org>
References: <1618542490-14756-1-git-send-email-wanpengli@tencent.com>
 <9c49c6ff-d896-e6a5-c051-b6707f6ec58a@redhat.com>
 <CANRm+Cy-xmDRQoUfOYm+GGvWiS+qC_sBjyZmcLykbKqTF2YDxQ@mail.gmail.com>
 <YH2wnl05UBqVhcHr@google.com>
 <c1909fa3-61f3-de6b-1aa1-8bc36285e1e4@redhat.com>
 <CANRm+CwQ266j6wTxqFZtGhp_HfQZ7Y_e843hzROqNUxf9BcaFA@mail.gmail.com>
 <CANRm+CyHX-_vQLck1a9wpCv8a-YnnemEWm+zVv4eWYby5gdAeg@mail.gmail.com>
 <b2fca9a5-9b2b-b8f2-0d1e-fc8b9d9b5659@redhat.com>
 <CANRm+Czysw6z1u+fbsRF3JUyiJc0jErVATusar_Vj8CcSBy5LQ@mail.gmail.com>
From:   Paolo Bonzini <pbonzini@redhat.com>
Message-ID: <e1d07b55-1539-ed33-911c-713403d776b3@redhat.com>
Date:   Tue, 20 Apr 2021 12:23:44 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.7.0
MIME-Version: 1.0
In-Reply-To: <CANRm+Czysw6z1u+fbsRF3JUyiJc0jErVATusar_Vj8CcSBy5LQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 20/04/21 10:48, Wanpeng Li wrote:
>> I was thinking of something simpler:
>>
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index 9b8e30dd5b9b..455c648f9adc 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -3198,10 +3198,9 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)
>>    {
>>          struct kvm *kvm = me->kvm;
>>          struct kvm_vcpu *vcpu;
>> -       int last_boosted_vcpu = me->kvm->last_boosted_vcpu;
>>          int yielded = 0;
>>          int try = 3;
>> -       int pass;
>> +       int pass, num_passes = 1;
>>          int i;
>>
>>          kvm_vcpu_set_in_spin_loop(me, true);
>> @@ -3212,13 +3211,14 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)
>>           * VCPU is holding the lock that we need and will release it.
>>           * We approximate round-robin by starting at the last boosted VCPU.
>>           */
>> -       for (pass = 0; pass < 2 && !yielded && try; pass++) {
>> -               kvm_for_each_vcpu(i, vcpu, kvm) {
>> -                       if (!pass && i <= last_boosted_vcpu) {
>> -                               i = last_boosted_vcpu;
>> -                               continue;
>> -                       } else if (pass && i > last_boosted_vcpu)
>> -                               break;
>> +       for (pass = 0; pass < num_passes; pass++) {
>> +               int idx = me->kvm->last_boosted_vcpu;
>> +               int n = atomic_read(&kvm->online_vcpus);
>> +               for (i = 0; i < n; i++, idx++) {
>> +                       if (idx == n)
>> +                               idx = 0;
>> +
>> +                       vcpu = kvm_get_vcpu(kvm, idx);
>>                          if (!READ_ONCE(vcpu->ready))
>>                                  continue;
>>                          if (vcpu == me)
>> @@ -3226,23 +3226,36 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)
>>                          if (rcuwait_active(&vcpu->wait) &&
>>                              !vcpu_dy_runnable(vcpu))
>>                                  continue;
>> -                       if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode &&
>> -                               !kvm_arch_vcpu_in_kernel(vcpu))
>> -                               continue;
>>                          if (!kvm_vcpu_eligible_for_directed_yield(vcpu))
>>                                  continue;
>>
>> +                       if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode &&
>> +                           !kvm_arch_vcpu_in_kernel(vcpu)) {
>> +                           /*
>> +                            * A vCPU running in userspace can get to kernel mode via
>> +                            * an interrupt.  That's a worse choice than a CPU already
>> +                            * in kernel mode so only do it on a second pass.
>> +                            */
>> +                           if (!vcpu_dy_runnable(vcpu))
>> +                                   continue;
>> +                           if (pass == 0) {
>> +                                   num_passes = 2;
>> +                                   continue;
>> +                           }
>> +                       }
>> +
>>                          yielded = kvm_vcpu_yield_to(vcpu);
>>                          if (yielded > 0) {
>>                                  kvm->last_boosted_vcpu = i;
>> -                               break;
>> +                               goto done;
>>                          } else if (yielded < 0) {
>>                                  try--;
>>                                  if (!try)
>> -                                       break;
>> +                                       goto done;
>>                          }
>>                  }
>>          }
>> +done:
> 
> We just tested the above post against 96 vCPUs VM in an over-subscribe
> scenario, the score of pbzip2 fluctuated drastically. Sometimes it is
> worse than vanilla, but the average improvement is around 2.2%. The
> new version of my post is around 9.3%，the origial posted patch is
> around 10% which is totally as expected since now both IPI receivers
> in user-mode and lock-waiters are second class citizens.

Fair enough.  Of the two patches you posted I prefer the original, so 
I'll go with that one.

Paolo