Received-SPF: pass (google.com: domain of linux-kernel+bounces-27298-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Tue, 16 Jan 2024 11:13:15 +0000
Message-ID: <86v87t8ras.wl-maz@kernel.org>
From: Marc Zyngier <maz@kernel.org>
To: "sundongxu (A)" <sundongxu3@huawei.com>
Cc: <oliver.upton@linux.dev>,
	<yuzenghui@huawei.com>,
	<james.morse@arm.com>,
	<suzuki.poulose@arm.com>,
	<will@kernel.org>,
	<catalin.marinas@arm.com>,
	<wanghaibin.wang@huawei.com>,
	<kvmarm@lists.linux.dev>,
	<linux-kernel@vger.kernel.org>,
	<linux-arm-kernel@lists.infradead.org>
Subject: Re: [bug report] GICv4.1: VM performance degradation due to not trapping vCPU WFI
In-Reply-To: <a481ef04-ddd2-dfc1-41b1-d2ec45c6a3b5@huawei.com>
References: <a481ef04-ddd2-dfc1-41b1-d2ec45c6a3b5@huawei.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.1
 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO)
Precedence: bulk
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII

On Tue, 16 Jan 2024 03:26:08 +0000,
"sundongxu (A)" <sundongxu3@huawei.com> wrote:
> 
> Hi Guys,
> 
> We found a problem about GICv4/4.1, for example:
> We use QEMU to start a VM (4 vCPUs and 8G memory), VM disk was
> configured with virtio, and the network is configured with vhost-net,
> the CPU affinity of the vCPU and emulator is as follows, in VM xml:
> 
>   <cputune>
>     <vcpupin vcpu='0' cpuset='4'/>
>     <vcpupin vcpu='1' cpuset='5'/>
>     <vcpupin vcpu='2' cpuset='6'/>
>     <vcpupin vcpu='3' cpuset='7'/>
>     <emulatorpin cpuset='4,5,6,7'/>
>   </cputune>
> 
> Running Mysql in the VM, and sysbench (Mysql benchmark) on the host,
> the performance index is tps, the higher the better.
> If the host only enabled GICv3, the tps will be around 1400.
> If the host enabled GICv4.1, other configurations remain unchanged, the
> tps will be around 40.
> 
> We found that when the host enabled GICv4.1, because vSGI is directly
> injected to VM, and most time vCPU exclusively occupy the pCPU, vCPU
> will not trap when executing the WFI instruction. Then from the host
> view, the CPU usage of vCPU0~vCPU3 is almost 100%. When running mysql
> service in VM, the vhost-net and qemu processes also need to obtain
> enough CPU time, but unfortunately these processes cannot get that much
> time (for example, only GICv3 enabled, the cpu usage of vhost-net is
> about 43%, but with GICv4.1 enabled, it becomes 0~2%). During the test,
> it was found that vhost-net sleeps and wakes up very frequently. When
> vhost-net wakes up, it often cannot obtain CPU in time (because of
> wake-up preemption check). After waking up, vhost-net will usually run
> for a short period of time before going to sleep again.

Can you elaborate on this preemption check issue?

> 
> If the host enabled GICv4.1, and force vCPU to trap when executing WFI,
> the tps will be around 1400.
> 
> On the other hand, when vCPU executes WFI instruction without trapping,
> the vcpu wake-up delay will be significantly improved. For example, the
> result of running cyclictest in VM:
> WFI trap           6us
> WFI no trap        2us
> 
> Currently, I add a KVM module parameter to control whether the vCPU
> traps (by set or clear HCR_TWI) when executing the WFI instruction with
> host enabled GICv4/4.1, and by default, vCPU traps are set.
> 
> Or, it there a better way?

As you foudn out, KVM has an adaptive way of dealing with HCR_TWI,
turning it off when the vcpu is alone in the run queue. Which means it
doesn't compete with any other thread. How comes the other threads
don't register as being runnable?

Effectively, we apply the same principle to vSGIs as to vLPIs, and it
was found that this heuristic was pretty beneficial to vLPIs. I'm a
bit surprised that vSGIs are so different in their usage pattern.

Does it help if you move your "emulatorpin" to some other physical
CPUs?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.