Received-SPF: pass (google.com: domain of linux-kernel+bounces-148887-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Wed, 17 Apr 2024 09:11:28 -0700
In-Reply-To: <adb07a02b3923eeb49f425d38509b340f4837e17.camel@cyberus-technology.de>
Precedence: bulk
Mime-Version: 1.0
References: <20240416123558.212040-1-julian.stecklina@cyberus-technology.de>
 <Zh6MmgOqvFPuWzD9@google.com> <ecb314c53c76bc6d2233a8b4d783a15297198ef8.camel@cyberus-technology.de>
 <Zh6WlOB8CS-By3DQ@google.com> <c2ca06e2d8d7ef66800f012953b8ea4be0147c92.camel@cyberus-technology.de>
 <Zh6-e9hy7U6DD2QM@google.com> <adb07a02b3923eeb49f425d38509b340f4837e17.camel@cyberus-technology.de>
Message-ID: <Zh_0sJPPoHKce5Ky@google.com>
Subject: Re: [PATCH 1/2] KVM: nVMX: fix CR4_READ_SHADOW when L0 updates CR4
 during a signal
From: Sean Christopherson <seanjc@google.com>
To: Thomas Prescher <thomas.prescher@cyberus-technology.de>
Cc: "pbonzini@redhat.com" <pbonzini@redhat.com>, "x86@kernel.org" <x86@kernel.org>, 
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>, "hpa@zytor.com" <hpa@zytor.com>, 
	Julian Stecklina <julian.stecklina@cyberus-technology.de>, 
	"tglx@linutronix.de" <tglx@linutronix.de>, "bp@alien8.de" <bp@alien8.de>, 
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>, 
	"mingo@redhat.com" <mingo@redhat.com>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On Wed, Apr 17, 2024, Thomas Prescher wrote:
> On Tue, 2024-04-16 at 11:07 -0700, Sean Christopherson wrote:
> > Hur dur, I forgot that KVM provides a "guest_mode" stat.=C2=A0 Userspac=
e can do
> > KVM_GET_STATS_FD on the vCPU FD to get a file handle to the binary stat=
s,
> > and then you wouldn't need to call back into KVM just to query guest_mo=
de.
> >=20
> > Ah, and I also forgot that we have kvm_run.flags, so adding
> > KVM_RUN_X86_GUEST_MODE would also be trivial (I almost suggested it
> > earlier, but didn't want to add a new field to kvm_run without a very g=
ood
> > reason).
>=20
> Thanks for the pointers. This is really helpful.
>=20
> I tried the "guest_mode" stat as you suggested and it solves the
> immediate issue we have with VirtualBox/KVM.

Note,=20

> What I don't understand is that we do not get the effective CR4 value
> of the L2 guest in kvm_run.s.regs.sregs.cr4.

Because what you're asking for is *not* the effective CR4 value of L2.

E.g. if L1 is using legacy shadowing paging to run L2, L1 is likely going t=
o run
L2 with GUEST_CR0.PG=3D1, GUEST_CR4.PAE=3D1, and GUEST_CR4.PSE=3D0 (though =
PSE is largely
irrelevant), i.e. will either use PAE paging or 64-bit paging to shadow L2.

But L2 itself could be unpaged (CR0.PG=3D0, CR4.PAE=3Dx, CR4.PSE=3Dx), usin=
g 32-bit
paging (CR0.PG=3D1, CR4.PAE=3D0, CR4.PSE=3D0), or using 32-bit paging with =
4MiB hugepages
(CR0.PG=3D1, CR4.PAE=3D0, CR4.PSE=3D1).  In all of those cases, the effecti=
ve CR0 and CR4
values consumed by hardware are CR0.PG=3D1, CR4.PAE=3D1, and CR4.PSE.

Or to convolute things even further, if L0 is running L1 with shadowing pag=
ing,
and L1 is running L2 with shadow paging but doing something weird and using=
 PSE
paging, then it would be possible to end up with:

  vmcs12->guest_cr4:
     .pae =3D 0
     .pse =3D 1

  vmcs12->cr4_read_shadow:
     .pae =3D 0
     .pse =3D 0

  vmcs02->guest_cr4:
     .pae =3D 1
     .pse =3D 0

> Instead, userland sees the contents of Vmcs::GUEST_CR4.=C2=A0Shouldn't th=
is be the
> combination of GUEST_CR4, GUEST_CR4_MASK and CR4_READ_SHADOW, i.e. what L=
2
> actually sees as CR4 value?

Because KVM_{G,S}ET_SREGS (and all other uAPIs in that vein) are defined to=
 operate
on actual vCPU state, and having them do something different if the vCPU is=
 in guest
mode would confusing/odd, and nonsensical to differences between VMX and SV=
M.

SVM doesn't have per-bit CR0/CR4 controls, i.e. CR4 loads and stores need t=
o be
intercepted, and so having KVM_{G,S}ET_SREGS operate on CR4_READ_SHADOW for=
 VMX
would yield different ABI for VMX versus SVM.

Note, what L2 *sees* is not a combination of the above; what L2 sees is pur=
ely
CR4_READ_SHADOW.  The other fields are consulted only if L2 attempts to loa=
d CR4.

> If this is expected, can you please explain the reasoning behind this
> interface decision? For me, it does not make sense that writing back
> the same value we receive at exit time causes a change in what L2 sees
> for CR4.

I doubt there was ever a concious decision, rather it never came up and thu=
s the
code is the result of doing nothing when nested VMX support was added.

That said, KVM's behavior is probably the least awful choice.  The changelo=
g of
the proposed patch is wrong when it says:

  If the signal is meant to be delivered to the L0 VMM, and L0 updates CR4 =
for L1

because the update isn't for L1, it's for the active vCPU state, which is L=
2.

At first glance, skipping the vmcs02.CR4_READ_SHADOW seems to make sense, b=
ut it
would create a bizarre inconsistency as KVM_SET_SREGS would effectively ove=
rride
vmcs12->guest_cr4, but not vmcs12->cr4_read_shadow.  KVM doesn't know the i=
ntent
of userspace, i.e. KVM can't know if userspace wants to change just the eff=
ective
value for CR4, or if userspace wants to change the effective *and* observab=
le
value for CR4.

In your case, where writing CR4 is spurious, preserving the read shadow wor=
ks,
but if there were some use case where userspace actually wanted to change L=
2's
CR4, leaving the read shadow set to vmcs12 would be wrong.

The whole situation is rather nonsensical, because if userspace actually di=
d change
CR4, the changes would be lost on the next nested VM-Exit =3D> VM-Entry.  T=
hat could
be solved by writing to vmcs12, but that creates a headache of its own beca=
use then
userspace changes to L2 become visible to L1, without userspace explicitly =
requesting
that.

Unfortunately, simply disallowing state save/restore when L2 is active does=
n't
work either, because userspace needs to be able to save/restore state that =
_isn't_
context switched by hardware, i.e. isn't in the VMCS or VMCB.

In short, yes, it's goofy and annoying, but there's no great solution and t=
he
issue really does need to be solved/avoided in userspace

> Another question is: when we want to save the VM state during a
> savevm/loadvm cycle, we kick all vCPUs via a singal and save their
> state. If any vCPU runs in L2 at the time of the exit, we somehow need
> to let it continue to run until we get an exit with the L1 state. Is
> there a mechanism to help us here?=20

Hmm, no?  What is it you're trying to do, i.e. why are you doing save/load?=
  If
you really want to save/load _all_ state, the right thing to do is to also =
save
and load nested state.