Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
MIME-Version: 1.0
References: <CANi1PHhzk80HvwQbBM46gpJ6_AA_P6+m5Jo0Nuy_MAdA4C2BhQ@mail.gmail.com>
 <87ttql5aq7.wl-maz@kernel.org>
In-Reply-To: <87ttql5aq7.wl-maz@kernel.org>
From:   Jan Henrik Weinstock <jan@mwa.re>
Date:   Wed, 25 Oct 2023 14:12:14 +0200
Message-ID: <CANi1PHieGooO0DK=6BPwq0UknHzsn9QM3rFQkh3HLMfWxDseUQ@mail.gmail.com>
Subject: Re: KVM exit to userspace on WFI
To:     Marc Zyngier <maz@kernel.org>
Cc:     oliver.upton@linux.dev, james.morse@arm.com,
        suzuki.poulose@arm.com, yuzenghui@huawei.com,
        catalin.marinas@arm.com, will@kernel.org,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev,
        linux-kernel@vger.kernel.org,
        =?UTF-8?Q?Lukas_J=C3=BCnger?= <lukas@mwa.re>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Hi Marc,

Thanks for your feedback. I understand that request_interrupt_window
is not to be used. I assume a setting a flag is a better way,
something similar to KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER, e.g.
KVM_ARCH_FLAG_WFX_EXIT_TO_USER.

I will also check that WFx traps are always enabled while this mode is
active to make sure userspace does not get blocked/scheduled out.

The reason for this is that we cannot have the thread that executes
KVM_RUN to be blocked or scheduled out whenever it hits a WFI.
Nop-WFIs are not a problem, since the PE will just continue executing
instructions, which is fine. We are currently using a timeout signal
that kicks KVM_RUN back into userspace, but we are seeing a lot of
time wasted because our KVM thread hangs in WFI/WFEs. It would be
better if we could just return from KVM_RUN immediately if the thread
would otherwise be blocked.

Thanks
Jan

Am Fr., 20. Okt. 2023 um 21:56 Uhr schrieb Marc Zyngier <maz@kernel.org>:
>
> Hi Jan,
>
> On Fri, 20 Oct 2023 19:45:05 +0100,
> Jan Henrik Weinstock <jan@mwa.re> wrote:
> >
> > Hi all,
> >
> > I am looking for a way to have KVM_RUN exit back to userspace once the
> > vcpu encounters a WFI. It seems the kvm_run->request_interrupt_window
> > flag is currently ignored by arm64.
>
> Well, that's consistent with arm64 not being an x86 implementation. We
> can inject interrupts any time, and there is no notion of "interrupt
> window".
>
> > So my solution thus far is to
> > patch kvm_handle_wfx in arch/arm64/kvm/handle_exit.c and return to
> > userspace with KVM_EXIT_IRQ_WINDOW_OPEN - working example attached.
> > Any chance to get this (or something similar) mainline?
>
> Certainly not as such. For start, this won't hit all WFIs, but only
> those that actively trap. And we don't even *try* to trap WFx in a
> number of cases (vcpu alone in its run queue and/or direct injection).
> There isn't even any guarantee that WFx is anything other than a NOP
> (it is architecturally only a hint), in which case no trap applies.
>
> So your "working example" really isn't one, as the architecture
> doesn't give you a way to do what you're asking for. If you want to
> cause an exit, writing to 'immediate_exit' and delivering a signal is
> the way.
>
> Thanks,
>
>         M.
>
> --
> Without deviation from the norm, progress is not possible.


--=20
Dr.-Ing. Jan Henrik Weinstock
Managing Director

MachineWare GmbH | www.machineware.de
H=C3=BChnermarkt 19, 52062 Aachen, Germany
Amtsgericht Aachen HRB25734

Gesch=C3=A4ftsf=C3=BChrung
Lukas J=C3=BCnger
Dr.-Ing. Jan Henrik Weinstock