Received-SPF: pass (google.com: domain of linux-kernel+bounces-89384-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Subject: Re: [PATCH v6 0/7] LoongArch: Add pv ipi support on LoongArch VM
From: maobibo <maobibo@loongson.cn>
To: Huacai Chen <chenhuacai@kernel.org>,
 Tianrui Zhao <zhaotianrui@loongson.cn>, Juergen Gross <jgross@suse.com>,
 Paolo Bonzini <pbonzini@redhat.com>, Jonathan Corbet <corbet@lwn.net>
Cc: loongarch@lists.linux.dev, linux-kernel@vger.kernel.org,
 virtualization@lists.linux.dev, kvm@vger.kernel.org,
 linux-doc@vger.kernel.org
References: <20240302075120.1414999-1-maobibo@loongson.cn>
Message-ID: <b2084dbd-3ea6-736a-293e-2309e828a960@loongson.cn>
Date: Sat, 2 Mar 2024 16:52:12 +0800
User-Agent: Mozilla/5.0 (X11; Linux loongarch64; rv:68.0) Gecko/20100101
 Thunderbird/68.7.0
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <20240302075120.1414999-1-maobibo@loongson.cn>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit

Sorry for the noise. It seems that there is some problem with my mail
client when batch method.

Please ignore this series, will send one by one manually.

Regards
Bibo Mao

On 2024/3/2 下午3:51, Bibo Mao wrote:
> On physical machine, ipi HW uses IOCSR registers, however there is trap
> into hypervisor when vcpu accesses IOCSR registers if system is in VM
> mode. SWI is a interrupt mechanism like SGI on ARM, software can send
> interrupt to CPU, only that on LoongArch SWI can only be sent to local CPU
> now. So SWI can not used for IPI on real HW system, however it can be used
> on VM when combined with hypercall method. IPI can be sent with hypercall
> method and SWI interrupt is injected to vcpu, vcpu can treat SWI
> interrupt as IPI.
> 
> With PV IPI supported, there is one trap with IPI sending, however with IPI
> receiving there is no trap. with IOCSR HW ipi method, there will be one
> trap with IPI sending and two trap with ipi receiving.
> 
> Also IPI multicast support is added for VM, the idea comes from x86 PV ipi.
> IPI can be sent to 128 vcpus in one time. With IPI multicast support, trap
> will be reduced greatly.
> 
> Here is the microbenchmarck data with "perf bench futex wake" testcase on
> 3C5000 single-way machine, there are 16 cpus on 3C5000 single-way machine,
> VM has 16 vcpus also. The benchmark data is ms time unit to wakeup 16
> threads, the performance is better if data is smaller.
> 
> physical machine                     0.0176 ms
> VM original                          0.1140 ms
> VM with pv ipi patch                 0.0481 ms
> 
> It passes to boot with 128/256 vcpus, runltp command in package ltp-20230516
> passes to run with 16 cores.
> 
> ---
> Change in V6:
>    1. Add privilege checking when emulating cpucfg at index 0x4000000 --
> 0x400000FF, return 0 if not executed at kernel mode.
>    2. Add document about LoongArch pv ipi with new creatly directory
> Documentation/virt/kvm/loongarch/
>    3. Fix pv ipi handling in kvm backend function kvm_pv_send_ipi(),
> where min should plus BITS_PER_LONG with second bitmap, otherwise
> VM with more than 64 vpus fails to boot.
>    4. Adjust patch order and code refine with review comments.
>   
> Change in V5:
>    1. Refresh function/macro name from review comments.
> 
> Change in V4:
>    1. Modfiy pv ipi hook function name call_func_ipi() and
> call_func_single_ipi() with send_ipi_mask()/send_ipi_single(), since pv
> ipi is used for both remote function call and reschedule notification.
>    2. Refresh changelog.
> 
> Change in V3:
>    1. Add 128 vcpu ipi multicast support like x86
>    2. Change cpucfg base address from 0x10000000 to 0x40000000, in order
> to avoid confliction with future hw usage
>    3. Adjust patch order in this patchset, move patch
> Refine-ipi-ops-on-LoongArch-platform to the first one.
> 
> Change in V2:
>    1. Add hw cpuid map support since ipi routing uses hw cpuid
>    2. Refine changelog description
>    3. Add hypercall statistic support for vcpu
>    4. Set percpu pv ipi message buffer aligned with cacheline
>    5. Refine pv ipi send logic, do not send ipi message with if there is
> pending ipi message.
> ---
> Bibo Mao (7):
>    LoongArch/smp: Refine some ipi functions on LoongArch platform
>    LoongArch: KVM: Add hypercall instruction emulation support
>    LoongArch: KVM: Add cpucfg area for kvm hypervisor
>    LoongArch: KVM: Add vcpu search support from physical cpuid
>    LoongArch: KVM: Add pv ipi support on kvm side
>    LoongArch: Add pv ipi support on guest kernel side
>    Documentation: KVM: Add hypercall for LoongArch
> 
>   Documentation/virt/kvm/index.rst              |   1 +
>   .../virt/kvm/loongarch/hypercalls.rst         |  79 +++++++++
>   Documentation/virt/kvm/loongarch/index.rst    |  10 ++
>   arch/loongarch/Kconfig                        |   9 +
>   arch/loongarch/include/asm/Kbuild             |   1 -
>   arch/loongarch/include/asm/hardirq.h          |   5 +
>   arch/loongarch/include/asm/inst.h             |   1 +
>   arch/loongarch/include/asm/irq.h              |  10 +-
>   arch/loongarch/include/asm/kvm_host.h         |  27 +++
>   arch/loongarch/include/asm/kvm_para.h         | 156 ++++++++++++++++++
>   arch/loongarch/include/asm/kvm_vcpu.h         |   1 +
>   arch/loongarch/include/asm/loongarch.h        |  11 ++
>   arch/loongarch/include/asm/paravirt.h         |  27 +++
>   .../include/asm/paravirt_api_clock.h          |   1 +
>   arch/loongarch/include/asm/smp.h              |  31 ++--
>   arch/loongarch/include/uapi/asm/Kbuild        |   2 -
>   arch/loongarch/kernel/Makefile                |   1 +
>   arch/loongarch/kernel/irq.c                   |  24 +--
>   arch/loongarch/kernel/paravirt.c              | 151 +++++++++++++++++
>   arch/loongarch/kernel/perf_event.c            |  14 +-
>   arch/loongarch/kernel/smp.c                   |  62 ++++---
>   arch/loongarch/kernel/time.c                  |  12 +-
>   arch/loongarch/kvm/exit.c                     | 141 ++++++++++++++--
>   arch/loongarch/kvm/vcpu.c                     |  94 ++++++++++-
>   arch/loongarch/kvm/vm.c                       |  11 ++
>   25 files changed, 780 insertions(+), 102 deletions(-)
>   create mode 100644 Documentation/virt/kvm/loongarch/hypercalls.rst
>   create mode 100644 Documentation/virt/kvm/loongarch/index.rst
>   create mode 100644 arch/loongarch/include/asm/kvm_para.h
>   create mode 100644 arch/loongarch/include/asm/paravirt.h
>   create mode 100644 arch/loongarch/include/asm/paravirt_api_clock.h
>   delete mode 100644 arch/loongarch/include/uapi/asm/Kbuild
>   create mode 100644 arch/loongarch/kernel/paravirt.c
> 
> 
> base-commit: 87adedeba51a822533649b143232418b9e26d08b
>