Received-SPF: pass (google.com: domain of linux-kernel+bounces-26939-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <a481ef04-ddd2-dfc1-41b1-d2ec45c6a3b5@huawei.com>
Date: Tue, 16 Jan 2024 11:26:08 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Content-Language: en-US
From: "sundongxu (A)" <sundongxu3@huawei.com>
To: <maz@kernel.org>, <oliver.upton@linux.dev>, <yuzenghui@huawei.com>,
	<james.morse@arm.com>, <suzuki.poulose@arm.com>, <will@kernel.org>,
	<catalin.marinas@arm.com>
CC: <wanghaibin.wang@huawei.com>, <kvmarm@lists.linux.dev>,
	<linux-kernel@vger.kernel.org>, <linux-arm-kernel@lists.infradead.org>
Subject: [bug report] GICv4.1: VM performance degradation due to not trapping
 vCPU WFI
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit

Hi Guys,

We found a problem about GICv4/4.1, for example:
We use QEMU to start a VM (4 vCPUs and 8G memory), VM disk was
configured with virtio, and the network is configured with vhost-net,
the CPU affinity of the vCPU and emulator is as follows, in VM xml:

  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <emulatorpin cpuset='4,5,6,7'/>
  </cputune>

Running Mysql in the VM, and sysbench (Mysql benchmark) on the host,
the performance index is tps, the higher the better.
If the host only enabled GICv3, the tps will be around 1400.
If the host enabled GICv4.1, other configurations remain unchanged, the
tps will be around 40.

We found that when the host enabled GICv4.1, because vSGI is directly
injected to VM, and most time vCPU exclusively occupy the pCPU, vCPU
will not trap when executing the WFI instruction. Then from the host
view, the CPU usage of vCPU0~vCPU3 is almost 100%. When running mysql
service in VM, the vhost-net and qemu processes also need to obtain
enough CPU time, but unfortunately these processes cannot get that much
time (for example, only GICv3 enabled, the cpu usage of vhost-net is
about 43%, but with GICv4.1 enabled, it becomes 0~2%). During the test,
it was found that vhost-net sleeps and wakes up very frequently. When
vhost-net wakes up, it often cannot obtain CPU in time (because of
wake-up preemption check). After waking up, vhost-net will usually run
for a short period of time before going to sleep again.

If the host enabled GICv4.1, and force vCPU to trap when executing WFI,
the tps will be around 1400.

On the other hand, when vCPU executes WFI instruction without trapping,
the vcpu wake-up delay will be significantly improved. For example, the
result of running cyclictest in VM:
WFI trap           6us
WFI no trap        2us

Currently, I add a KVM module parameter to control whether the vCPU
traps (by set or clear HCR_TWI) when executing the WFI instruction with
host enabled GICv4/4.1, and by default, vCPU traps are set.

Or, it there a better way?