The idea is from Xen, when sending a call-function IPI-many to vCPUs,
yield if any of the IPI target vCPUs was preempted. 17% performance
increasement of ebizzy benchmark can be observed in an over-subscribe
environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
IPI-many since call-function is not easy to be trigged by userspace
workload).
v2 -> v3:
* add bounds-check on dest_id
v1 -> v2:
* check map is not NULL
* check map->phys_map[dest_id] is not NULL
* make kvm_sched_yield static
* change dest_id to unsinged long
Wanpeng Li (3):
KVM: X86: Yield to IPI target if necessary
KVM: X86: Implement PV sched yield hypercall
KVM: X86: Expose PV_SCHED_YIELD CPUID feature bit to guest
Documentation/virtual/kvm/cpuid.txt | 4 ++++
Documentation/virtual/kvm/hypercalls.txt | 11 +++++++++++
arch/x86/include/uapi/asm/kvm_para.h | 1 +
arch/x86/kernel/kvm.c | 21 +++++++++++++++++++++
arch/x86/kvm/cpuid.c | 3 ++-
arch/x86/kvm/x86.c | 26 ++++++++++++++++++++++++++
include/uapi/linux/kvm_para.h | 1 +
7 files changed, 66 insertions(+), 1 deletion(-)
--
2.7.4
From: Wanpeng Li <[email protected]>
Expose PV_SCHED_YIELD feature bit to guest, the guest can check this
feature bit before using paravirtualized sched yield.
Cc: Paolo Bonzini <[email protected]>
Cc: Radim Krčmář <[email protected]>
Cc: Liran Alon <[email protected]>
Signed-off-by: Wanpeng Li <[email protected]>
---
Documentation/virtual/kvm/cpuid.txt | 4 ++++
arch/x86/kvm/cpuid.c | 3 ++-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/Documentation/virtual/kvm/cpuid.txt b/Documentation/virtual/kvm/cpuid.txt
index 97ca194..1c39683 100644
--- a/Documentation/virtual/kvm/cpuid.txt
+++ b/Documentation/virtual/kvm/cpuid.txt
@@ -66,6 +66,10 @@ KVM_FEATURE_PV_SEND_IPI || 11 || guest checks this feature bit
|| || before using paravirtualized
|| || send IPIs.
------------------------------------------------------------------------------
+KVM_FEATURE_PV_SHED_YIELD || 12 || guest checks this feature bit
+ || || before using paravirtualized
+ || || sched yield.
+------------------------------------------------------------------------------
KVM_FEATURE_CLOCKSOURCE_STABLE_BIT || 24 || host will warn if no guest-side
|| || per-cpu warps are expected in
|| || kvmclock.
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index e18a9f9..c018fc8 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -643,7 +643,8 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
(1 << KVM_FEATURE_PV_UNHALT) |
(1 << KVM_FEATURE_PV_TLB_FLUSH) |
(1 << KVM_FEATURE_ASYNC_PF_VMEXIT) |
- (1 << KVM_FEATURE_PV_SEND_IPI);
+ (1 << KVM_FEATURE_PV_SEND_IPI) |
+ (1 << KVM_FEATURE_PV_SCHED_YIELD);
if (sched_info_on())
entry->eax |= (1 << KVM_FEATURE_STEAL_TIME);
--
2.7.4
ping, :)
On Thu, 30 May 2019 at 09:05, Wanpeng Li <[email protected]> wrote:
>
> The idea is from Xen, when sending a call-function IPI-many to vCPUs,
> yield if any of the IPI target vCPUs was preempted. 17% performance
> increasement of ebizzy benchmark can be observed in an over-subscribe
> environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
> IPI-many since call-function is not easy to be trigged by userspace
> workload).
>
> v2 -> v3:
> * add bounds-check on dest_id
>
> v1 -> v2:
> * check map is not NULL
> * check map->phys_map[dest_id] is not NULL
> * make kvm_sched_yield static
> * change dest_id to unsinged long
>
> Wanpeng Li (3):
> KVM: X86: Yield to IPI target if necessary
> KVM: X86: Implement PV sched yield hypercall
> KVM: X86: Expose PV_SCHED_YIELD CPUID feature bit to guest
>
> Documentation/virtual/kvm/cpuid.txt | 4 ++++
> Documentation/virtual/kvm/hypercalls.txt | 11 +++++++++++
> arch/x86/include/uapi/asm/kvm_para.h | 1 +
> arch/x86/kernel/kvm.c | 21 +++++++++++++++++++++
> arch/x86/kvm/cpuid.c | 3 ++-
> arch/x86/kvm/x86.c | 26 ++++++++++++++++++++++++++
> include/uapi/linux/kvm_para.h | 1 +
> 7 files changed, 66 insertions(+), 1 deletion(-)
>
> --
> 2.7.4
>
2019-05-30 09:05+0800, Wanpeng Li:
> The idea is from Xen, when sending a call-function IPI-many to vCPUs,
> yield if any of the IPI target vCPUs was preempted. 17% performance
> increasement of ebizzy benchmark can be observed in an over-subscribe
> environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
> IPI-many since call-function is not easy to be trigged by userspace
> workload).
Have you checked if we could gain performance by having the yield as an
extension to our PV IPI call?
It would allow us to skip the VM entry/exit overhead on the caller.
(The benefit of that might be negligible and it also poses a
complication when splitting the target mask into several PV IPI
hypercalls.)
Thanks.
On Mon, Jun 10, 2019 at 04:34:20PM +0200, Radim Krčmář wrote:
> 2019-05-30 09:05+0800, Wanpeng Li:
> > The idea is from Xen, when sending a call-function IPI-many to vCPUs,
> > yield if any of the IPI target vCPUs was preempted. 17% performance
> > increasement of ebizzy benchmark can be observed in an over-subscribe
> > environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
> > IPI-many since call-function is not easy to be trigged by userspace
> > workload).
>
> Have you checked if we could gain performance by having the yield as an
> extension to our PV IPI call?
>
> It would allow us to skip the VM entry/exit overhead on the caller.
> (The benefit of that might be negligible and it also poses a
> complication when splitting the target mask into several PV IPI
> hypercalls.)
Tangetially related to splitting PV IPI hypercalls, are there any major
hurdles to supporting shorthand? Not having to generate the mask for
->send_IPI_allbutself and ->kvm_send_ipi_all seems like an easy to way
shave cycles for affected flows.
On Tue, 11 Jun 2019 at 09:11, Sean Christopherson
<[email protected]> wrote:
>
> On Mon, Jun 10, 2019 at 04:34:20PM +0200, Radim Krčmář wrote:
> > 2019-05-30 09:05+0800, Wanpeng Li:
> > > The idea is from Xen, when sending a call-function IPI-many to vCPUs,
> > > yield if any of the IPI target vCPUs was preempted. 17% performance
> > > increasement of ebizzy benchmark can be observed in an over-subscribe
> > > environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
> > > IPI-many since call-function is not easy to be trigged by userspace
> > > workload).
> >
> > Have you checked if we could gain performance by having the yield as an
> > extension to our PV IPI call?
> >
> > It would allow us to skip the VM entry/exit overhead on the caller.
> > (The benefit of that might be negligible and it also poses a
> > complication when splitting the target mask into several PV IPI
> > hypercalls.)
>
> Tangetially related to splitting PV IPI hypercalls, are there any major
> hurdles to supporting shorthand? Not having to generate the mask for
> ->send_IPI_allbutself and ->kvm_send_ipi_all seems like an easy to way
> shave cycles for affected flows.
Not sure why shorthand is not used for native x2apic mode.
Regards,
Wanpeng Li
> On Jun 10, 2019, at 6:45 PM, Wanpeng Li <[email protected]> wrote:
>
> On Tue, 11 Jun 2019 at 09:11, Sean Christopherson
> <[email protected]> wrote:
>> On Mon, Jun 10, 2019 at 04:34:20PM +0200, Radim Krčmář wrote:
>>> 2019-05-30 09:05+0800, Wanpeng Li:
>>>> The idea is from Xen, when sending a call-function IPI-many to vCPUs,
>>>> yield if any of the IPI target vCPUs was preempted. 17% performance
>>>> increasement of ebizzy benchmark can be observed in an over-subscribe
>>>> environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
>>>> IPI-many since call-function is not easy to be trigged by userspace
>>>> workload).
>>>
>>> Have you checked if we could gain performance by having the yield as an
>>> extension to our PV IPI call?
>>>
>>> It would allow us to skip the VM entry/exit overhead on the caller.
>>> (The benefit of that might be negligible and it also poses a
>>> complication when splitting the target mask into several PV IPI
>>> hypercalls.)
>>
>> Tangetially related to splitting PV IPI hypercalls, are there any major
>> hurdles to supporting shorthand? Not having to generate the mask for
>> ->send_IPI_allbutself and ->kvm_send_ipi_all seems like an easy to way
>> shave cycles for affected flows.
>
> Not sure why shorthand is not used for native x2apic mode.
Why do you say so? native_send_call_func_ipi() checks if allbutself
shorthand should be used and does so (even though the check can be more
efficient - I’m looking at that code right now…)
On Tue, 11 Jun 2019 at 09:48, Nadav Amit <[email protected]> wrote:
>
> > On Jun 10, 2019, at 6:45 PM, Wanpeng Li <[email protected]> wrote:
> >
> > On Tue, 11 Jun 2019 at 09:11, Sean Christopherson
> > <[email protected]> wrote:
> >> On Mon, Jun 10, 2019 at 04:34:20PM +0200, Radim Krčmář wrote:
> >>> 2019-05-30 09:05+0800, Wanpeng Li:
> >>>> The idea is from Xen, when sending a call-function IPI-many to vCPUs,
> >>>> yield if any of the IPI target vCPUs was preempted. 17% performance
> >>>> increasement of ebizzy benchmark can be observed in an over-subscribe
> >>>> environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
> >>>> IPI-many since call-function is not easy to be trigged by userspace
> >>>> workload).
> >>>
> >>> Have you checked if we could gain performance by having the yield as an
> >>> extension to our PV IPI call?
> >>>
> >>> It would allow us to skip the VM entry/exit overhead on the caller.
> >>> (The benefit of that might be negligible and it also poses a
> >>> complication when splitting the target mask into several PV IPI
> >>> hypercalls.)
> >>
> >> Tangetially related to splitting PV IPI hypercalls, are there any major
> >> hurdles to supporting shorthand? Not having to generate the mask for
> >> ->send_IPI_allbutself and ->kvm_send_ipi_all seems like an easy to way
> >> shave cycles for affected flows.
> >
> > Not sure why shorthand is not used for native x2apic mode.
>
> Why do you say so? native_send_call_func_ipi() checks if allbutself
> shorthand should be used and does so (even though the check can be more
> efficient - I’m looking at that code right now…)
Please continue to follow the apic/x2apic driver. Just apic_flat set
APIC_DEST_ALLBUT/APIC_DEST_ALLINC to ICR.
Regards,
Wanpeng Li
On Mon, 10 Jun 2019 at 22:34, Radim Krčmář <[email protected]> wrote:
>
> 2019-05-30 09:05+0800, Wanpeng Li:
> > The idea is from Xen, when sending a call-function IPI-many to vCPUs,
> > yield if any of the IPI target vCPUs was preempted. 17% performance
> > increasement of ebizzy benchmark can be observed in an over-subscribe
> > environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
> > IPI-many since call-function is not easy to be trigged by userspace
> > workload).
>
> Have you checked if we could gain performance by having the yield as an
> extension to our PV IPI call?
It will extend irq disabled time in __send_ipi_mask(). In addition,
sched yield can be used to optimize other synchronization primitives
in guest I think.
Regards,
Wanpeng Li
> On Jun 11, 2019, at 3:02 AM, Wanpeng Li <[email protected]> wrote:
>
> On Tue, 11 Jun 2019 at 09:48, Nadav Amit <[email protected]> wrote:
>>> On Jun 10, 2019, at 6:45 PM, Wanpeng Li <[email protected]> wrote:
>>>
>>> On Tue, 11 Jun 2019 at 09:11, Sean Christopherson
>>> <[email protected]> wrote:
>>>> On Mon, Jun 10, 2019 at 04:34:20PM +0200, Radim Krčmář wrote:
>>>>> 2019-05-30 09:05+0800, Wanpeng Li:
>>>>>> The idea is from Xen, when sending a call-function IPI-many to vCPUs,
>>>>>> yield if any of the IPI target vCPUs was preempted. 17% performance
>>>>>> increasement of ebizzy benchmark can be observed in an over-subscribe
>>>>>> environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
>>>>>> IPI-many since call-function is not easy to be trigged by userspace
>>>>>> workload).
>>>>>
>>>>> Have you checked if we could gain performance by having the yield as an
>>>>> extension to our PV IPI call?
>>>>>
>>>>> It would allow us to skip the VM entry/exit overhead on the caller.
>>>>> (The benefit of that might be negligible and it also poses a
>>>>> complication when splitting the target mask into several PV IPI
>>>>> hypercalls.)
>>>>
>>>> Tangetially related to splitting PV IPI hypercalls, are there any major
>>>> hurdles to supporting shorthand? Not having to generate the mask for
>>>> ->send_IPI_allbutself and ->kvm_send_ipi_all seems like an easy to way
>>>> shave cycles for affected flows.
>>>
>>> Not sure why shorthand is not used for native x2apic mode.
>>
>> Why do you say so? native_send_call_func_ipi() checks if allbutself
>> shorthand should be used and does so (even though the check can be more
>> efficient - I’m looking at that code right now…)
>
> Please continue to follow the apic/x2apic driver. Just apic_flat set
> APIC_DEST_ALLBUT/APIC_DEST_ALLINC to ICR.
Indeed - I was sure by the name that it does it correctly. That’s stupid.
I’ll add it to the patch-set I am working on (TLB shootdown improvements),
if you don’t mind.
On Wed, 12 Jun 2019 at 00:57, Nadav Amit <[email protected]> wrote:
>
> > On Jun 11, 2019, at 3:02 AM, Wanpeng Li <[email protected]> wrote:
> >
> > On Tue, 11 Jun 2019 at 09:48, Nadav Amit <[email protected]> wrote:
> >>> On Jun 10, 2019, at 6:45 PM, Wanpeng Li <[email protected]> wrote:
> >>>
> >>> On Tue, 11 Jun 2019 at 09:11, Sean Christopherson
> >>> <[email protected]> wrote:
> >>>> On Mon, Jun 10, 2019 at 04:34:20PM +0200, Radim Krčmář wrote:
> >>>>> 2019-05-30 09:05+0800, Wanpeng Li:
> >>>>>> The idea is from Xen, when sending a call-function IPI-many to vCPUs,
> >>>>>> yield if any of the IPI target vCPUs was preempted. 17% performance
> >>>>>> increasement of ebizzy benchmark can be observed in an over-subscribe
> >>>>>> environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
> >>>>>> IPI-many since call-function is not easy to be trigged by userspace
> >>>>>> workload).
> >>>>>
> >>>>> Have you checked if we could gain performance by having the yield as an
> >>>>> extension to our PV IPI call?
> >>>>>
> >>>>> It would allow us to skip the VM entry/exit overhead on the caller.
> >>>>> (The benefit of that might be negligible and it also poses a
> >>>>> complication when splitting the target mask into several PV IPI
> >>>>> hypercalls.)
> >>>>
> >>>> Tangetially related to splitting PV IPI hypercalls, are there any major
> >>>> hurdles to supporting shorthand? Not having to generate the mask for
> >>>> ->send_IPI_allbutself and ->kvm_send_ipi_all seems like an easy to way
> >>>> shave cycles for affected flows.
> >>>
> >>> Not sure why shorthand is not used for native x2apic mode.
> >>
> >> Why do you say so? native_send_call_func_ipi() checks if allbutself
> >> shorthand should be used and does so (even though the check can be more
> >> efficient - I’m looking at that code right now…)
> >
> > Please continue to follow the apic/x2apic driver. Just apic_flat set
> > APIC_DEST_ALLBUT/APIC_DEST_ALLINC to ICR.
>
> Indeed - I was sure by the name that it does it correctly. That’s stupid.
>
> I’ll add it to the patch-set I am working on (TLB shootdown improvements),
> if you don’t mind.
Original for hotplug cpu safe.
https://lwn.net/Articles/138365/
https://lwn.net/Articles/138368/
Not sure shortcut native support is acceptable, I will play my
kvm_send_ipi_allbutself and kvm_send_ipi_all. :)
Regards,
Wanpeng Li
> On Jun 11, 2019, at 6:18 PM, Wanpeng Li <[email protected]> wrote:
>
> On Wed, 12 Jun 2019 at 00:57, Nadav Amit <[email protected]> wrote:
>>> On Jun 11, 2019, at 3:02 AM, Wanpeng Li <[email protected]> wrote:
>>>
>>> On Tue, 11 Jun 2019 at 09:48, Nadav Amit <[email protected]> wrote:
>>>>> On Jun 10, 2019, at 6:45 PM, Wanpeng Li <[email protected]> wrote:
>>>>>
>>>>> On Tue, 11 Jun 2019 at 09:11, Sean Christopherson
>>>>> <[email protected]> wrote:
>>>>>> On Mon, Jun 10, 2019 at 04:34:20PM +0200, Radim Krčmář wrote:
>>>>>>> 2019-05-30 09:05+0800, Wanpeng Li:
>>>>>>>> The idea is from Xen, when sending a call-function IPI-many to vCPUs,
>>>>>>>> yield if any of the IPI target vCPUs was preempted. 17% performance
>>>>>>>> increasement of ebizzy benchmark can be observed in an over-subscribe
>>>>>>>> environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
>>>>>>>> IPI-many since call-function is not easy to be trigged by userspace
>>>>>>>> workload).
>>>>>>>
>>>>>>> Have you checked if we could gain performance by having the yield as an
>>>>>>> extension to our PV IPI call?
>>>>>>>
>>>>>>> It would allow us to skip the VM entry/exit overhead on the caller.
>>>>>>> (The benefit of that might be negligible and it also poses a
>>>>>>> complication when splitting the target mask into several PV IPI
>>>>>>> hypercalls.)
>>>>>>
>>>>>> Tangetially related to splitting PV IPI hypercalls, are there any major
>>>>>> hurdles to supporting shorthand? Not having to generate the mask for
>>>>>> ->send_IPI_allbutself and ->kvm_send_ipi_all seems like an easy to way
>>>>>> shave cycles for affected flows.
>>>>>
>>>>> Not sure why shorthand is not used for native x2apic mode.
>>>>
>>>> Why do you say so? native_send_call_func_ipi() checks if allbutself
>>>> shorthand should be used and does so (even though the check can be more
>>>> efficient - I’m looking at that code right now…)
>>>
>>> Please continue to follow the apic/x2apic driver. Just apic_flat set
>>> APIC_DEST_ALLBUT/APIC_DEST_ALLINC to ICR.
>>
>> Indeed - I was sure by the name that it does it correctly. That’s stupid.
>>
>> I’ll add it to the patch-set I am working on (TLB shootdown improvements),
>> if you don’t mind.
>
> Original for hotplug cpu safe.
> https://lwn.net/Articles/138365/
> https://lwn.net/Articles/138368/
> Not sure shortcut native support is acceptable, I will play my
> kvm_send_ipi_allbutself and kvm_send_ipi_all. :)
Yes, I saw these threads before. But I think the test in
native_send_call_func_ipi() should take care of it.
I’ll recheck.
On Wed, 12 Jun 2019 at 09:37, Nadav Amit <[email protected]> wrote:
>
> > On Jun 11, 2019, at 6:18 PM, Wanpeng Li <[email protected]> wrote:
> >
> > On Wed, 12 Jun 2019 at 00:57, Nadav Amit <[email protected]> wrote:
> >>> On Jun 11, 2019, at 3:02 AM, Wanpeng Li <[email protected]> wrote:
> >>>
> >>> On Tue, 11 Jun 2019 at 09:48, Nadav Amit <[email protected]> wrote:
> >>>>> On Jun 10, 2019, at 6:45 PM, Wanpeng Li <[email protected]> wrote:
> >>>>>
> >>>>> On Tue, 11 Jun 2019 at 09:11, Sean Christopherson
> >>>>> <[email protected]> wrote:
> >>>>>> On Mon, Jun 10, 2019 at 04:34:20PM +0200, Radim Krčmář wrote:
> >>>>>>> 2019-05-30 09:05+0800, Wanpeng Li:
> >>>>>>>> The idea is from Xen, when sending a call-function IPI-many to vCPUs,
> >>>>>>>> yield if any of the IPI target vCPUs was preempted. 17% performance
> >>>>>>>> increasement of ebizzy benchmark can be observed in an over-subscribe
> >>>>>>>> environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
> >>>>>>>> IPI-many since call-function is not easy to be trigged by userspace
> >>>>>>>> workload).
> >>>>>>>
> >>>>>>> Have you checked if we could gain performance by having the yield as an
> >>>>>>> extension to our PV IPI call?
> >>>>>>>
> >>>>>>> It would allow us to skip the VM entry/exit overhead on the caller.
> >>>>>>> (The benefit of that might be negligible and it also poses a
> >>>>>>> complication when splitting the target mask into several PV IPI
> >>>>>>> hypercalls.)
> >>>>>>
> >>>>>> Tangetially related to splitting PV IPI hypercalls, are there any major
> >>>>>> hurdles to supporting shorthand? Not having to generate the mask for
> >>>>>> ->send_IPI_allbutself and ->kvm_send_ipi_all seems like an easy to way
> >>>>>> shave cycles for affected flows.
> >>>>>
> >>>>> Not sure why shorthand is not used for native x2apic mode.
> >>>>
> >>>> Why do you say so? native_send_call_func_ipi() checks if allbutself
> >>>> shorthand should be used and does so (even though the check can be more
> >>>> efficient - I’m looking at that code right now…)
> >>>
> >>> Please continue to follow the apic/x2apic driver. Just apic_flat set
> >>> APIC_DEST_ALLBUT/APIC_DEST_ALLINC to ICR.
> >>
> >> Indeed - I was sure by the name that it does it correctly. That’s stupid.
> >>
> >> I’ll add it to the patch-set I am working on (TLB shootdown improvements),
> >> if you don’t mind.
> >
> > Original for hotplug cpu safe.
> > https://lwn.net/Articles/138365/
> > https://lwn.net/Articles/138368/
> > Not sure shortcut native support is acceptable, I will play my
> > kvm_send_ipi_allbutself and kvm_send_ipi_all. :)
>
> Yes, I saw these threads before. But I think the test in
> native_send_call_func_ipi() should take care of it.
Good news, https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=WIP.x86/ipi
Thomas who also is the hotplug state machine author introduces
shorthands support to native kernel now, I will add the support to
kvm_send_ipi_allbutself() and kvm_send_ipi_all() after his work
complete.
Regards,
Wanpeng Li
On Fri, 28 Jun 2019 at 17:12, Wanpeng Li <[email protected]> wrote:
>
> On Wed, 12 Jun 2019 at 09:37, Nadav Amit <[email protected]> wrote:
> >
> > > On Jun 11, 2019, at 6:18 PM, Wanpeng Li <[email protected]> wrote:
> > >
> > > On Wed, 12 Jun 2019 at 00:57, Nadav Amit <[email protected]> wrote:
> > >>> On Jun 11, 2019, at 3:02 AM, Wanpeng Li <[email protected]> wrote:
> > >>>
> > >>> On Tue, 11 Jun 2019 at 09:48, Nadav Amit <[email protected]> wrote:
> > >>>>> On Jun 10, 2019, at 6:45 PM, Wanpeng Li <[email protected]> wrote:
> > >>>>>
> > >>>>> On Tue, 11 Jun 2019 at 09:11, Sean Christopherson
> > >>>>> <[email protected]> wrote:
> > >>>>>> On Mon, Jun 10, 2019 at 04:34:20PM +0200, Radim Krčmář wrote:
> > >>>>>>> 2019-05-30 09:05+0800, Wanpeng Li:
> > >>>>>>>> The idea is from Xen, when sending a call-function IPI-many to vCPUs,
> > >>>>>>>> yield if any of the IPI target vCPUs was preempted. 17% performance
> > >>>>>>>> increasement of ebizzy benchmark can be observed in an over-subscribe
> > >>>>>>>> environment. (w/ kvm-pv-tlb disabled, testing TLB flush call-function
> > >>>>>>>> IPI-many since call-function is not easy to be trigged by userspace
> > >>>>>>>> workload).
> > >>>>>>>
> > >>>>>>> Have you checked if we could gain performance by having the yield as an
> > >>>>>>> extension to our PV IPI call?
> > >>>>>>>
> > >>>>>>> It would allow us to skip the VM entry/exit overhead on the caller.
> > >>>>>>> (The benefit of that might be negligible and it also poses a
> > >>>>>>> complication when splitting the target mask into several PV IPI
> > >>>>>>> hypercalls.)
> > >>>>>>
> > >>>>>> Tangetially related to splitting PV IPI hypercalls, are there any major
> > >>>>>> hurdles to supporting shorthand? Not having to generate the mask for
> > >>>>>> ->send_IPI_allbutself and ->kvm_send_ipi_all seems like an easy to way
> > >>>>>> shave cycles for affected flows.
> > >>>>>
> > >>>>> Not sure why shorthand is not used for native x2apic mode.
> > >>>>
> > >>>> Why do you say so? native_send_call_func_ipi() checks if allbutself
> > >>>> shorthand should be used and does so (even though the check can be more
> > >>>> efficient - I’m looking at that code right now…)
> > >>>
> > >>> Please continue to follow the apic/x2apic driver. Just apic_flat set
> > >>> APIC_DEST_ALLBUT/APIC_DEST_ALLINC to ICR.
> > >>
> > >> Indeed - I was sure by the name that it does it correctly. That’s stupid.
> > >>
> > >> I’ll add it to the patch-set I am working on (TLB shootdown improvements),
> > >> if you don’t mind.
> > >
> > > Original for hotplug cpu safe.
> > > https://lwn.net/Articles/138365/
> > > https://lwn.net/Articles/138368/
> > > Not sure shortcut native support is acceptable, I will play my
> > > kvm_send_ipi_allbutself and kvm_send_ipi_all. :)
> >
> > Yes, I saw these threads before. But I think the test in
> > native_send_call_func_ipi() should take care of it.
>
> Good news, https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=WIP.x86/ipi
> Thomas who also is the hotplug state machine author introduces
> shorthands support to native kernel now, I will add the support to
> kvm_send_ipi_allbutself() and kvm_send_ipi_all() after his work
> complete.
Hmm, should fallback to native shorthands when support.
Regards,
Wanpeng Li