Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\))
Subject: Re: [PATCH 1/2] KVM: nVMX: Always write vmcs02.GUEST_CR3 during
 nested VM-Enter
From:   Liran Alon <liran.alon@oracle.com>
In-Reply-To: <20190927142725.GC24889@linux.intel.com>
Date:   Fri, 27 Sep 2019 17:44:53 +0300
Cc:     Paolo Bonzini <pbonzini@redhat.com>,
        =?utf-8?B?UmFkaW0gS3LEjW3DocWZ?= <rkrcmar@redhat.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Jim Mattson <jmattson@google.com>,
        Joerg Roedel <joro@8bytes.org>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, Reto Buerki <reet@codelabs.ch>
Content-Transfer-Encoding: quoted-printable
Message-Id: <EF5C03E7-E3C2-4372-955C-06FB416EB164@oracle.com>
References: <20190926214302.21990-1-sean.j.christopherson@intel.com>
 <20190926214302.21990-2-sean.j.christopherson@intel.com>
 <68340081-0094-4A74-9B33-3431F39659AA@oracle.com>
 <20190927142725.GC24889@linux.intel.com>
To:     Sean Christopherson <sean.j.christopherson@intel.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On 27 Sep 2019, at 17:27, Sean Christopherson =
<sean.j.christopherson@intel.com> wrote:
>=20
> On Fri, Sep 27, 2019 at 03:06:02AM +0300, Liran Alon wrote:
>>=20
>>=20
>>> On 27 Sep 2019, at 0:43, Sean Christopherson =
<sean.j.christopherson@intel.com> wrote:
>>>=20
>>> Write the desired L2 CR3 into vmcs02.GUEST_CR3 during nested =
VM-Enter
>>> isntead of deferring the VMWRITE until vmx_set_cr3().  If the =
VMWRITE
>>> is deferred, then KVM can consume a stale vmcs02.GUEST_CR3 when it
>>> refreshes vmcs12->guest_cr3 during nested_vmx_vmexit() if the =
emulated
>>> VM-Exit occurs without actually entering L2, e.g. if the nested run
>>> is squashed because L2 is being put into HLT.
>>=20
>> I would rephrase to =E2=80=9CIf an emulated VMEntry is squashed =
because L1 sets
>> vmcs12->guest_activity_state to HLT=E2=80=9D.  I think it=E2=80=99s a =
bit more explicit.
>>=20
>>>=20
>>> In an ideal world where EPT *requires* unrestricted guest (and vice
>>> versa), VMX could handle CR3 similar to how it handles RSP and RIP,
>>> e.g. mark CR3 dirty and conditionally load it at vmx_vcpu_run().  =
But
>>> the unrestricted guest silliness complicates the dirty tracking =
logic
>>> to the point that explicitly handling vmcs02.GUEST_CR3 during nested
>>> VM-Enter is a simpler overall implementation.
>>>=20
>>> Cc: stable@vger.kernel.org
>>> Reported-by: Reto Buerki <reet@codelabs.ch>
>>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
>>> ---
>>> arch/x86/kvm/vmx/nested.c | 8 ++++++++
>>> arch/x86/kvm/vmx/vmx.c    | 9 ++++++---
>>> 2 files changed, 14 insertions(+), 3 deletions(-)
>>>=20
>>> diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
>>> index 41abc62c9a8a..971a24134081 100644
>>> --- a/arch/x86/kvm/vmx/nested.c
>>> +++ b/arch/x86/kvm/vmx/nested.c
>>> @@ -2418,6 +2418,14 @@ static int prepare_vmcs02(struct kvm_vcpu =
*vcpu, struct vmcs12 *vmcs12,
>>> 				entry_failure_code))
>>> 		return -EINVAL;
>>>=20
>>> +	/*
>>> +	 * Immediately write vmcs02.GUEST_CR3.  It will be propagated to =
vmcs12
>>> +	 * on nested VM-Exit, which can occur without actually running =
L2, e.g.
>>> +	 * if L2 is entering HLT state, and thus without hitting =
vmx_set_cr3().
>>> +	 */
>>=20
>> If I understand correctly, it=E2=80=99s not exactly if L2 is entering =
HLT state in
>> general.  (E.g. issue doesn=E2=80=99t occur if L2 runs HLT directly =
which is not
>> configured to be intercepted by vmcs12).  It=E2=80=99s specifically =
when L1 enters L2
>> with a HLT guest-activity-state. I suggest rephrasing comment.
>=20
> I deliberately worded the comment so that it remains valid if there =
are
> more conditions in the future that cause KVM to skip running L2.  What =
if
> I split the difference and make the changelog more explicit, but leave =
the
> comment as is?

I think what is confusing in comment is that it seems to also refer to =
the case
where L2 directly enters HLT state without L1 intercept. Which isn=E2=80=99=
t related.
So I would explicitly mention it=E2=80=99s when L1 enters L2 but don=E2=80=
=99t physically enter guest
with vmcs02 because L2 is in HLT state.

-Liran

>=20
>>> +	if (enable_ept)
>>> +		vmcs_writel(GUEST_CR3, vmcs12->guest_cr3);
>>> +
>>> 	/* Late preparation of GUEST_PDPTRs now that EFER and CRs are =
set. */
>>> 	if (load_guest_pdptrs_vmcs12 && nested_cpu_has_ept(vmcs12) &&
>>> 	    is_pae_paging(vcpu)) {
>>> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
>>> index d4575ffb3cec..b530950a9c2b 100644
>>> --- a/arch/x86/kvm/vmx/vmx.c
>>> +++ b/arch/x86/kvm/vmx/vmx.c
>>> @@ -2985,6 +2985,7 @@ void vmx_set_cr3(struct kvm_vcpu *vcpu, =
unsigned long cr3)
>>> {
>>> 	struct kvm *kvm =3D vcpu->kvm;
>>> 	unsigned long guest_cr3;
>>> +	bool skip_cr3 =3D false;
>>> 	u64 eptp;
>>>=20
>>> 	guest_cr3 =3D cr3;
>>> @@ -3000,15 +3001,17 @@ void vmx_set_cr3(struct kvm_vcpu *vcpu, =
unsigned long cr3)
>>> 			spin_unlock(&to_kvm_vmx(kvm)->ept_pointer_lock);
>>> 		}
>>>=20
>>> -		if (enable_unrestricted_guest || is_paging(vcpu) ||
>>> -		    is_guest_mode(vcpu))
>>> +		if (is_guest_mode(vcpu))
>>> +			skip_cr3 =3D true;
>>> +		else if (enable_unrestricted_guest || is_paging(vcpu))
>>> 			guest_cr3 =3D kvm_read_cr3(vcpu);
>>> 		else
>>> 			guest_cr3 =3D =
to_kvm_vmx(kvm)->ept_identity_map_addr;
>>> 		ept_load_pdptrs(vcpu);
>>> 	}
>>>=20
>>> -	vmcs_writel(GUEST_CR3, guest_cr3);
>>> +	if (!skip_cr3)
>>=20
>> Nit: It=E2=80=99s a matter of taste, but I prefer positive =
conditions. i.e. =E2=80=9Cbool
>> write_guest_cr3=E2=80=9D.
>>=20
>> Anyway, code seems valid to me. Nice catch.
>> Reviewed-by: Liran Alon <liran.alon@oracle.com>
>>=20
>> -Liran
>>=20
>>> +		vmcs_writel(GUEST_CR3, guest_cr3);
>>> }
>>>=20
>>> int vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
>>> --=20
>>> 2.22.0
>>>=20
>>=20