From: Wanpeng Li <[email protected]>
After commit d73eb57b80b (KVM: Boost vCPUs that are delivering interrupts), a
five years ago bug is exposed. Running ebizzy benchmark in three 80 vCPUs VMs
on one 80 pCPUs Skylake server, a lot of rcu_sched stall warning splatting
in the VMs after stress testing:
INFO: rcu_sched detected stalls on CPUs/tasks: { 4 41 57 62 77} (detected by 15, t=60004 jiffies, g=899, c=898, q=15073)
Call Trace:
flush_tlb_mm_range+0x68/0x140
tlb_flush_mmu.part.75+0x37/0xe0
tlb_finish_mmu+0x55/0x60
zap_page_range+0x142/0x190
SyS_madvise+0x3cd/0x9c0
system_call_fastpath+0x1c/0x21
swait_active() sustains to be true before finish_swait() is called in
kvm_vcpu_block(), voluntarily preempted vCPUs are taken into account
by kvm_vcpu_on_spin() loop greatly increases the probability condition
kvm_arch_vcpu_runnable(vcpu) is checked and can be true, when APICv
is enabled the yield-candidate vCPU's VMCS RVI field leaks(by
vmx_sync_pir_to_irr()) into spinning-on-a-taken-lock vCPU's current
VMCS.
This patch fixes it by reverting the kvm_arch_vcpu_runnable() condition
in kvm_vcpu_on_spin() loop.
Cc: Paolo Bonzini <[email protected]>
Cc: Radim Krčmář <[email protected]>
Fixes: 98f4a1467 (KVM: add kvm_arch_vcpu_runnable() test to kvm_vcpu_on_spin() loop)
Cc: [email protected]
Signed-off-by: Wanpeng Li <[email protected]>
---
virt/kvm/kvm_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index ed061d8..12f2c91 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2506,7 +2506,7 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)
continue;
if (vcpu == me)
continue;
- if (swait_active(&vcpu->wq) && !kvm_arch_vcpu_runnable(vcpu))
+ if (swait_active(&vcpu->wq))
continue;
if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode &&
!kvm_arch_vcpu_in_kernel(vcpu))
--
2.7.4
From: Wanpeng Li <[email protected]>
After commit d73eb57b80b (KVM: Boost vCPUs that are delivering interrupts), a
five years ago bug is exposed. Running ebizzy benchmark in three 80 vCPUs VMs
on one 80 pCPUs Skylake server, a lot of rcu_sched stall warning splatting
in the VMs after stress testing:
INFO: rcu_sched detected stalls on CPUs/tasks: { 4 41 57 62 77} (detected by 15, t=60004 jiffies, g=899, c=898, q=15073)
Call Trace:
flush_tlb_mm_range+0x68/0x140
tlb_flush_mmu.part.75+0x37/0xe0
tlb_finish_mmu+0x55/0x60
zap_page_range+0x142/0x190
SyS_madvise+0x3cd/0x9c0
system_call_fastpath+0x1c/0x21
swait_active() sustains to be true before finish_swait() is called in
kvm_vcpu_block(), voluntarily preempted vCPUs are taken into account
by kvm_vcpu_on_spin() loop greatly increases the probability condition
kvm_arch_vcpu_runnable(vcpu) is checked and can be true, when APICv
is enabled the yield-candidate vCPU's VMCS RVI field leaks(by
vmx_sync_pir_to_irr()) into spinning-on-a-taken-lock vCPU's current
VMCS.
This patch fixes it by reverting the kvm_arch_vcpu_runnable() condition
in kvm_vcpu_on_spin() loop and checking swait_active(&vcpu->wq) for
involuntary preemption.
Cc: Paolo Bonzini <[email protected]>
Cc: Radim Krčmář <[email protected]>
Fixes: 98f4a1467 (KVM: add kvm_arch_vcpu_runnable() test to kvm_vcpu_on_spin() loop)
Cc: [email protected]
Signed-off-by: Wanpeng Li <[email protected]>
---
v1 -> v2:
* checking swait_active(&vcpu->wq) for involuntary preemption
virt/kvm/kvm_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index ed061d8..12f2c91 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2506,7 +2506,7 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)
continue;
if (vcpu == me)
continue;
- if (swait_active(&vcpu->wq) && !kvm_arch_vcpu_runnable(vcpu))
+ if (READ_ONCE(vcpu->preempted) && swait_active(&vcpu->wq))
continue;
if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode &&
!kvm_arch_vcpu_in_kernel(vcpu))
--
2.7.4
On 31/07/19 13:39, Wanpeng Li wrote:
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index ed061d8..12f2c91 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -2506,7 +2506,7 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)
> continue;
> if (vcpu == me)
> continue;
> - if (swait_active(&vcpu->wq) && !kvm_arch_vcpu_runnable(vcpu))
> + if (READ_ONCE(vcpu->preempted) && swait_active(&vcpu->wq))
> continue;
> if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode &&
> !kvm_arch_vcpu_in_kernel(vcpu))
>
This cannot work. swait_active means you are waiting, so you cannot be
involuntarily preempted.
The problem here is simply that kvm_vcpu_has_events is being called
without holding the lock. So kvm_arch_vcpu_runnable is okay, it's the
implementation that's wrong.
Just rename the existing function to just vcpu_runnable and make a new
arch callback kvm_arch_dy_runnable. kvm_arch_dy_runnable can be
conservative and only returns true for a subset of events, in particular
for x86 it can check:
- vcpu->arch.pv.pv_unhalted
- KVM_REQ_NMI or KVM_REQ_SMI or KVM_REQ_EVENT
- PIR.ON if APICv is set
Ultimately, all variables accessed in kvm_arch_dy_runnable should be
accessed with READ_ONCE or atomic_read.
And for all architectures, kvm_vcpu_on_spin should check
list_empty_careful(&vcpu->async_pf.done)
It's okay if your patch renames the function in non-x86 architectures,
leaving the fix to maintainers. So, let's CC Marc and Christian since
ARM and s390 have pretty complex kvm_arch_vcpu_runnable as well.
Paolo
On Wed, 31 Jul 2019 at 20:55, Paolo Bonzini <[email protected]> wrote:
>
> On 31/07/19 13:39, Wanpeng Li wrote:
> > diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> > index ed061d8..12f2c91 100644
> > --- a/virt/kvm/kvm_main.c
> > +++ b/virt/kvm/kvm_main.c
> > @@ -2506,7 +2506,7 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)
> > continue;
> > if (vcpu == me)
> > continue;
> > - if (swait_active(&vcpu->wq) && !kvm_arch_vcpu_runnable(vcpu))
> > + if (READ_ONCE(vcpu->preempted) && swait_active(&vcpu->wq))
> > continue;
> > if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode &&
> > !kvm_arch_vcpu_in_kernel(vcpu))
> >
>
> This cannot work. swait_active means you are waiting, so you cannot be
> involuntarily preempted.
>
> The problem here is simply that kvm_vcpu_has_events is being called
> without holding the lock. So kvm_arch_vcpu_runnable is okay, it's the
> implementation that's wrong.
>
> Just rename the existing function to just vcpu_runnable and make a new
> arch callback kvm_arch_dy_runnable. kvm_arch_dy_runnable can be
> conservative and only returns true for a subset of events, in particular
> for x86 it can check:
>
> - vcpu->arch.pv.pv_unhalted
>
> - KVM_REQ_NMI or KVM_REQ_SMI or KVM_REQ_EVENT
>
> - PIR.ON if APICv is set
>
> Ultimately, all variables accessed in kvm_arch_dy_runnable should be
> accessed with READ_ONCE or atomic_read.
>
> And for all architectures, kvm_vcpu_on_spin should check
> list_empty_careful(&vcpu->async_pf.done)
>
> It's okay if your patch renames the function in non-x86 architectures,
> leaving the fix to maintainers. So, let's CC Marc and Christian since
> ARM and s390 have pretty complex kvm_arch_vcpu_runnable as well.
Ok, just sent patch to do this.
Regards,
Wanpeng Li