Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20211020120726.4022086-1-pizhenwei@bytedance.com>
 <CANRm+CxAVA-L0wjm72eohXXWvh9fS7wVFzfKHuEjrsiRFuk9fg@mail.gmail.com>
 <YXB4FHfzh99707EH@google.com> <08757159-1673-5c7b-3efc-e5b54e82d6c3@bytedance.com>
In-Reply-To: <08757159-1673-5c7b-3efc-e5b54e82d6c3@bytedance.com>
From:   Wanpeng Li <kernellwp@gmail.com>
Date:   Thu, 21 Oct 2021 13:03:46 +0800
Message-ID: <CANRm+CzcTUWYJeaj3eWKH84YZYgeMZz3kbpn13c8i97iYGGHFQ@mail.gmail.com>
Subject: Re: Re: [PATCH] x86/kvm: Introduce boot parameter no-kvm-pvipi
To:     zhenwei pi <pizhenwei@bytedance.com>
Cc:     Sean Christopherson <seanjc@google.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Jonathan Corbet <corbet@lwn.net>,
        Wanpeng Li <wanpengli@tencent.com>,
        LKML <linux-kernel@vger.kernel.org>, linux-doc@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Thu, 21 Oct 2021 at 11:05, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
>
> On 10/21/21 4:12 AM, Sean Christopherson wrote:
> > On Wed, Oct 20, 2021, Wanpeng Li wrote:
> >> On Wed, 20 Oct 2021 at 20:08, zhenwei pi <pizhenwei@bytedance.com> wrote:
> >>>
> >>> Although host side exposes KVM PV SEND IPI feature to guest side,
> >>> guest should still have a chance to disable it.
> >>>
> >>> A typicall case of this parameter:
> >>> If the host AMD server enables AVIC feature, the flat mode of APIC
> >>> get better performance in the guest.
> >>
> >> Hmm, I didn't find enough valuable information in your posting. We
> >> observe AMD a lot before.
> >> https://lore.kernel.org/all/CANRm+Cx597FNRUCyVz1D=B6Vs2GX3Sw57X7Muk+yMpi_hb+v1w@mail.gmail.com/T/#u
> >
> > I too would like to see numbers.  I suspect the answer is going to be that
> > AVIC performs poorly in CPU overcommit scenarios because of the cost of managing
> > the tables and handling "failed delivery" exits, but that AVIC does quite well
> > when vCPUs are pinned 1:1 and IPIs rarely require an exit to the host.
> >
>
> Test env:
> CPU: AMD EPYC 7642 48-Core Processor
>
> Kmod args(enable avic and disable nested):
> modprobe kvm-amd nested=0 avic=1 npt=1
>
> QEMU args(disable x2apic):
> ... -cpu host,x2apic=off ...
>
> Benchmark tool:
> https://github.com/bytedance/kvm-utils/tree/master/microbenchmark/apic-ipi
>
> ~# insmod apic_ipi.ko options=5 && dmesg -c
>
>   apic_ipi: 1 NUMA node(s)
>   apic_ipi: apic [flat]
>   apic_ipi: apic->send_IPI[default_send_IPI_single+0x0/0x40]
>   apic_ipi: apic->send_IPI_mask[kvm_send_ipi_mask+0x0/0x10]
>   apic_ipi:     IPI[kvm_send_ipi_mask] from CPU[0] to CPU[1]
>   apic_ipi:             total cycles 375671259, avg 3756
>   apic_ipi:     IPI[flat_send_IPI_mask] from CPU[0] to CPU[1]
>   apic_ipi:             total cycles 221961822, avg 2219
>
>
> apic->send_IPI_mask[kvm_send_ipi_mask+0x0/0x10]
>    -> This line show current send_IPI_mask is kvm_send_ipi_mask(because
> of PV SEND IPI FEATURE)
>
> apic_ipi:       IPI[kvm_send_ipi_mask] from CPU[0] to CPU[1]
> apic_ipi:               total cycles 375671259, avg 3756
>    -->These lines show the average cycles of each kvm_send_ipi_mask: 3756
>
> apic_ipi:       IPI[flat_send_IPI_mask] from CPU[0] to CPU[1]
> apic_ipi:               total cycles 221961822, avg 2219
>    -->These lines show the average cycles of each flat_send_IPI_mask: 2219

Just single target IPI is not eough.

    Wanpeng