Received-SPF: pass (google.com: domain of linux-kernel+bounces-169237-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Precedence: bulk
MIME-Version: 1.0
References: <20240428100518.1642324-1-maobibo@loongson.cn>
In-Reply-To: <20240428100518.1642324-1-maobibo@loongson.cn>
From: Huacai Chen <chenhuacai@kernel.org>
Date: Mon, 6 May 2024 09:45:10 +0800
Message-ID: <CAAhV-H6c2Wym4w5WchP=K1fHv8uVFDp59CATYKcH3mGYDxnKmA@mail.gmail.com>
Subject: Re: [PATCH v8 0/6] LoongArch: Add pv ipi support on LoongArch VM
To: Bibo Mao <maobibo@loongson.cn>
Cc: Tianrui Zhao <zhaotianrui@loongson.cn>, Juergen Gross <jgross@suse.com>, 
	Paolo Bonzini <pbonzini@redhat.com>, Jonathan Corbet <corbet@lwn.net>, loongarch@lists.linux.dev, 
	linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, 
	kvm@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi, Bibo,

I have done an off-list discussion with some KVM experts, and they
think user-space have its right to know PV features, so cpucfg
solution is acceptable.

And I applied this series with some modifications at
https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson.g=
it/log/?h=3Dloongarch-kvm
You can test it now. But it seems the upstream qemu cannot enable PV IPI no=
w.

I will reply to other patches about my modifications.

Huacai

On Sun, Apr 28, 2024 at 6:05=E2=80=AFPM Bibo Mao <maobibo@loongson.cn> wrot=
e:
>
> On physical machine, ipi HW uses IOCSR registers, however there is trap
> into hypervisor when vcpu accesses IOCSR registers if system is in VM
> mode. SWI is a interrupt mechanism like SGI on ARM, software can send
> interrupt to CPU, only that on LoongArch SWI can only be sent to local CP=
U
> now. So SWI can not used for IPI on real HW system, however it can be use=
d
> on VM when combined with hypercall method. IPI can be sent with hypercall
> method and SWI interrupt is injected to vcpu, vcpu can treat SWI
> interrupt as IPI.
>
> With PV IPI supported, there is one trap with IPI sending, however with I=
PI
> receiving there is no trap. with IOCSR HW ipi method, there will be one
> trap with IPI sending and two trap with ipi receiving.
>
> Also IPI multicast support is added for VM, the idea comes from x86 PV ip=
i.
> IPI can be sent to 128 vcpus in one time. With IPI multicast support, tra=
p
> will be reduced greatly.
>
> Here is the microbenchmarck data with "perf bench futex wake" testcase on
> 3C5000 single-way machine, there are 16 cpus on 3C5000 single-way machine=
,
> VM has 16 vcpus also. The benchmark data is ms time unit to wakeup 16
> threads, the performance is better if data is smaller.
>
> physical machine                     0.0176 ms
> VM original                          0.1140 ms
> VM with pv ipi patch                 0.0481 ms
>
> It passes to boot with 128/256 vcpus, and passes to run runltp command
> with package ltp-20230516.
>
> ---
> v7 --- v8:
>  1. Remove kernel PLV mode checking with cpucfg emulation for hypervisor
> feature inquiry.
>  2. Remove document about loongarch hypercall ABI per request of huacai,
> will add English/Chinese doc at the same time in later.
>
> v6 --- v7:
>   1. Refine LoongArch virt document by review comments.
>   2. Add function kvm_read_reg()/kvm_write_reg() in hypercall emulation,
> and later it can be used for other trap emulations.
>
> v5 --- v6:
>   1. Add privilege checking when emulating cpucfg at index 0x4000000 --
> 0x400000FF, return 0 if not executed at kernel mode.
>   2. Add document about LoongArch pv ipi with new creatly directory
> Documentation/virt/kvm/loongarch/
>   3. Fix pv ipi handling in kvm backend function kvm_pv_send_ipi(),
> where min should plus BITS_PER_LONG with second bitmap, otherwise
> VM with more than 64 vpus fails to boot.
>   4. Adjust patch order and code refine with review comments.
>
> v4 --- v5:
>   1. Refresh function/macro name from review comments.
>
> v3 --- v4:
>   1. Modfiy pv ipi hook function name call_func_ipi() and
> call_func_single_ipi() with send_ipi_mask()/send_ipi_single(), since pv
> ipi is used for both remote function call and reschedule notification.
>   2. Refresh changelog.
>
> v2 --- v3:
>   1. Add 128 vcpu ipi multicast support like x86
>   2. Change cpucfg base address from 0x10000000 to 0x40000000, in order
> to avoid confliction with future hw usage
>   3. Adjust patch order in this patchset, move patch
> Refine-ipi-ops-on-LoongArch-platform to the first one.
>
> v1 --- v2:
>   1. Add hw cpuid map support since ipi routing uses hw cpuid
>   2. Refine changelog description
>   3. Add hypercall statistic support for vcpu
>   4. Set percpu pv ipi message buffer aligned with cacheline
>   5. Refine pv ipi send logic, do not send ipi message with if there is
> pending ipi message.
> ---
> Bibo Mao (6):
>   LoongArch/smp: Refine some ipi functions on LoongArch platform
>   LoongArch: KVM: Add hypercall instruction emulation support
>   LoongArch: KVM: Add cpucfg area for kvm hypervisor
>   LoongArch: KVM: Add vcpu search support from physical cpuid
>   LoongArch: KVM: Add pv ipi support on kvm side
>   LoongArch: Add pv ipi support on guest kernel side
>
>  arch/loongarch/Kconfig                        |   9 +
>  arch/loongarch/include/asm/Kbuild             |   1 -
>  arch/loongarch/include/asm/hardirq.h          |   5 +
>  arch/loongarch/include/asm/inst.h             |   1 +
>  arch/loongarch/include/asm/irq.h              |  10 +-
>  arch/loongarch/include/asm/kvm_host.h         |  27 +++
>  arch/loongarch/include/asm/kvm_para.h         | 155 ++++++++++++++++++
>  arch/loongarch/include/asm/kvm_vcpu.h         |  11 ++
>  arch/loongarch/include/asm/loongarch.h        |  11 ++
>  arch/loongarch/include/asm/paravirt.h         |  27 +++
>  .../include/asm/paravirt_api_clock.h          |   1 +
>  arch/loongarch/include/asm/smp.h              |  31 ++--
>  arch/loongarch/include/uapi/asm/Kbuild        |   2 -
>  arch/loongarch/kernel/Makefile                |   1 +
>  arch/loongarch/kernel/irq.c                   |  24 +--
>  arch/loongarch/kernel/paravirt.c              | 151 +++++++++++++++++
>  arch/loongarch/kernel/perf_event.c            |  14 +-
>  arch/loongarch/kernel/smp.c                   |  62 ++++---
>  arch/loongarch/kernel/time.c                  |  12 +-
>  arch/loongarch/kvm/exit.c                     | 132 +++++++++++++--
>  arch/loongarch/kvm/vcpu.c                     |  94 ++++++++++-
>  arch/loongarch/kvm/vm.c                       |  11 ++
>  22 files changed, 690 insertions(+), 102 deletions(-)
>  create mode 100644 arch/loongarch/include/asm/kvm_para.h
>  create mode 100644 arch/loongarch/include/asm/paravirt.h
>  create mode 100644 arch/loongarch/include/asm/paravirt_api_clock.h
>  delete mode 100644 arch/loongarch/include/uapi/asm/Kbuild
>  create mode 100644 arch/loongarch/kernel/paravirt.c
>
>
> base-commit: 5eb4573ea63d0c83bf58fb7c243fc2c2b6966c02
> --
> 2.39.3
>
>