Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <1574306232-872-1-git-send-email-wanpengli@tencent.com>
In-Reply-To: <1574306232-872-1-git-send-email-wanpengli@tencent.com>
From:   Wanpeng Li <kernellwp@gmail.com>
Date:   Wed, 26 Feb 2020 11:32:53 +0800
Message-ID: <CANRm+Cz0RnF=roCkJf-X8pEyVY5wH4ZgQKWv8o0Whu59t8_A2w@mail.gmail.com>
Subject: Re: [PATCH v4 1/2] KVM: VMX: FIXED+PHYSICAL mode single target IPI fastpath
To:     LKML <linux-kernel@vger.kernel.org>, kvm <kvm@vger.kernel.org>
Cc:     Paolo Bonzini <pbonzini@redhat.com>,
        =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= <rkrcmar@redhat.com>,
        Sean Christopherson <sean.j.christopherson@intel.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Jim Mattson <jmattson@google.com>,
        Joerg Roedel <joro@8bytes.org>,
        Liran Alon <liran.alon@oracle.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, 21 Nov 2019 at 11:17, Wanpeng Li <kernellwp@gmail.com> wrote:
>
> From: Wanpeng Li <wanpengli@tencent.com>
>
> ICR and TSCDEADLINE MSRs write cause the main MSRs write vmexits in our
> product observation, multicast IPIs are not as common as unicast IPI like
> RESCHEDULE_VECTOR and CALL_FUNCTION_SINGLE_VECTOR etc.
>
> This patch introduce a mechanism to handle certain performance-critical
> WRMSRs in a very early stage of KVM VMExit handler.
>
> This mechanism is specifically used for accelerating writes to x2APIC ICR
> that attempt to send a virtual IPI with physical destination-mode, fixed
> delivery-mode and single target. Which was found as one of the main cause=
s
> of VMExits for Linux workloads.
>
> The reason this mechanism significantly reduce the latency of such virtua=
l
> IPIs is by sending the physical IPI to the target vCPU in a very early st=
age
> of KVM VMExit handler, before host interrupts are enabled and before expe=
nsive
> operations such as reacquiring KVM=E2=80=99s SRCU lock.
> Latency is reduced even more when KVM is able to use APICv posted-interru=
pt
> mechanism (which allows to deliver the virtual IPI directly to target vCP=
U
> without the need to kick it to host).
>
> Testing on Xeon Skylake server:
>
> The virtual IPI latency from sender send to receiver receive reduces
> more than 200+ cpu cycles.

Testing by IPI microbenchmark(https://lkml.org/lkml/2017/12/19/141):

Normal IPI:           Improved 3%
Broadcast IPI:      Improved 5%

w/ --overcommit cpu-pm=3Don

Normal IPI:           Improved 14%
Broadcast IPI:      Improved 3.6%

    Wanpeng