Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
MIME-Version: 1.0
References: <CANi1PHhzk80HvwQbBM46gpJ6_AA_P6+m5Jo0Nuy_MAdA4C2BhQ@mail.gmail.com>
 <87ttql5aq7.wl-maz@kernel.org> <CANi1PHieGooO0DK=6BPwq0UknHzsn9QM3rFQkh3HLMfWxDseUQ@mail.gmail.com>
 <86cyx250w9.wl-maz@kernel.org>
In-Reply-To: <86cyx250w9.wl-maz@kernel.org>
From:   Jan Henrik Weinstock <jan@mwa.re>
Date:   Fri, 27 Oct 2023 19:41:44 +0200
Message-ID: <CANi1PHjAwLWAq9EW7r5Yh_xbvPiJMsq8342JwAGafz1d1NUhSA@mail.gmail.com>
Subject: Re: KVM exit to userspace on WFI
To:     Marc Zyngier <maz@kernel.org>
Cc:     oliver.upton@linux.dev, james.morse@arm.com,
        suzuki.poulose@arm.com, yuzenghui@huawei.com,
        catalin.marinas@arm.com, will@kernel.org,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev,
        linux-kernel@vger.kernel.org,
        =?UTF-8?Q?Lukas_J=C3=BCnger?= <lukas@mwa.re>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Hi Marc,

the basic idea behind this is to have a (single-threaded) execution loop,
something like this:

vcpu-thread:    vcpu-run | process-io-devices | vcpu-run | process-io...
                         ^
                  WFX or timeout

We switch to simulating IO devices whenever the vcpu is idle (wfi) or excee=
ds
a certain budget of instructions (counted via pmu). Our fallback currently =
is
to kick the vcpu out of its execution using a signal (via a timeout/alarm).=
 But
of course, if the cpu is stuck at a wfi, we are wasting a lot of time.

I understand that the proposed behavior is not desirable for most use cases=
,
which is why I suggest locking it behind a flag, e.g.
KVM_ARCH_FLAG_WFX_EXIT_TO_USER.


Am Mi., 25. Okt. 2023 um 14:42 Uhr schrieb Marc Zyngier <maz@kernel.org>:
>
> On Wed, 25 Oct 2023 13:12:14 +0100,
> Jan Henrik Weinstock <jan@mwa.re> wrote:
> >
> > Hi Marc,
> >
> > Thanks for your feedback. I understand that request_interrupt_window
> > is not to be used. I assume a setting a flag is a better way,
> > something similar to KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER, e.g.
> > KVM_ARCH_FLAG_WFX_EXIT_TO_USER.
> >
> > I will also check that WFx traps are always enabled while this mode is
> > active to make sure userspace does not get blocked/scheduled out.
>
> Why would that be an acceptable behaviour?
>
> > The reason for this is that we cannot have the thread that executes
> > KVM_RUN to be blocked or scheduled out whenever it hits a WFI.
>
> Why? If that's not acceptable, how do you even cope with the basic
> preemption?
>
> > Nop-WFIs are not a problem, since the PE will just continue executing
> > instructions, which is fine. We are currently using a timeout signal
> > that kicks KVM_RUN back into userspace, but we are seeing a lot of
> > time wasted because our KVM thread hangs in WFI/WFEs. It would be
> > better if we could just return from KVM_RUN immediately if the thread
> > would otherwise be blocked.
>
> On the face of it, this makes little sense:
>
> - While in userspace, no interrupt source that normally delivered
>   without any userpsace intervention will be blocked (timers,
>   VLPIs...). I cannot how this can be a good idea.
>
> - Trapping WFE is an important scheduling hint, and returning to
>   userspace defeats it. Contended spinlocks, for example, will be even
>   slower to acquire.
>
> I'm sure you have a particular use case for such a degraded behaviour,
> but since you are not describing it, I'm not at all inclined to
> actively break KVM's performance and scalability.
>
> Thanks,
>
>         M.
>
> --
> Without deviation from the norm, progress is not possible.


--=20
Dr.-Ing. Jan Henrik Weinstock
Managing Director

MachineWare GmbH | www.machineware.de
H=C3=BChnermarkt 19, 52062 Aachen, Germany
Amtsgericht Aachen HRB25734

Gesch=C3=A4ftsf=C3=BChrung
Lukas J=C3=BCnger
Dr.-Ing. Jan Henrik Weinstock