Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
MIME-Version: 1.0
References: <1b52b557beb6606007f7ec5672eab0adf1606a34.camel@infradead.org>
 <CABgObfZgYXaXqP=6s53=+mYWvOnbgYJiCRct-0ob444sK9SvGw@mail.gmail.com>
 <faec494b6df5ebee5644017c9415e747bd34952b.camel@infradead.org>
 <3dc66987-49c7-abda-eb70-1898181ef3fe@redhat.com> <d3e0c3e9-4994-4808-a8df-3d23487ff9c4@amazon.de>
In-Reply-To: <d3e0c3e9-4994-4808-a8df-3d23487ff9c4@amazon.de>
From:   Paolo Bonzini <pbonzini@redhat.com>
Date:   Tue, 26 Sep 2023 19:20:24 +0200
Message-ID: <CABgObfZb4CvzpnSJxz9saw8PJeo1Y2=0uB9y4_K+Cu9P9FpF6g@mail.gmail.com>
Subject: Re: [RFC] KVM: x86: Allow userspace exit on HLT and MWAIT, else yield
 on MWAIT
To:     Alexander Graf <graf@amazon.de>
Cc:     David Woodhouse <dwmw2@infradead.org>, kvm@vger.kernel.org,
        Peter Zijlstra <peterz@infradead.org>,
        Sean Christopherson <seanjc@google.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>, linux-kernel@vger.kernel.org,
        Nicolas Saenz Julienne <nsaenz@amazon.es>,
        "Griffoul, Fred" <fgriffo@amazon.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Sat, Sep 23, 2023 at 6:44=E2=80=AFPM Alexander Graf <graf@amazon.de> wro=
te:
> On 23.09.23 11:24, Paolo Bonzini wrote:
> > Why do you need it?  You can just use KVM_RUN to go to sleep, and if yo=
u
> > get another job you kick out the vCPU with pthread_kill.  (I also didn'=
t
> > get the VSM reference).
>
> With the original VSM patches, we used to make a vCPU aware of the fact
> that it can morph into one of many VTLs. That approach turned out to be
> insanely intrusive and fragile and so we're currently reimplementing
> everything as VTLs as vCPUs. That allows us to move the majority of VSM
> functionality to user space. Everything we've seen so far looks as if
> there is no real performance loss with that approach.

Yes, that was also what I remember, sharing the FPU somehow while
having separate vCPU file descriptors.

> One small problem with that is that now user space is responsible for
> switching between VTLs: It determines which VTL is currently running and
> leaves all others (read: all other vCPUs) as stopped. That means if you
> are running happily in KVM_RUN in VTL0 and VTL1 gets an interrupt, user
> space needs to stop VTL0 and unpause VTL1 until it triggers VTL_RETURN
> at which point VTL1 stops execution and VTL0 runs again.

That's with IPIs in VTL1, right? I understand now. My idea was, since
we need a link from VTL1 to VTL0 for the FPU, to use the same link to
trigger a vmexit to userspace if source VTL > destination VTL. I am
not sure how you would handle the case where the destination vCPU is
not running; probably by detecting the IPI when VTL0 restarts on the
destination vCPU?

In any case, making vCPUs poll()-able is sensible.

Paolo

> Nicolas built a patch that exposes "interrupt on vCPU is pending" as an
> ioeventfd user space can request. That way, user space can know whenever
> a currently paused vCPU has a pending interrupt and can act accordingly.
> You could use the same mechanism if you wanted to implement HLT in user
> space, but still use an in-kernel LAPIC.