Received-SPF: pass (google.com: domain of linux-kernel+bounces-139603-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Wed, 10 Apr 2024 18:03:52 -0700
From: Isaku Yamahata <isaku.yamahata@intel.com>
To: "Huang, Kai" <kai.huang@intel.com>
Cc: "seanjc@google.com" <seanjc@google.com>,
	"Yamahata, Isaku" <isaku.yamahata@intel.com>,
	"davidskidmore@google.com" <davidskidmore@google.com>,
	"Edgecombe, Rick P" <rick.p.edgecombe@intel.com>,
	"Li, Xiaoyao" <xiaoyao.li@intel.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"srutherford@google.com" <srutherford@google.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"pankaj.gupta@amd.com" <pankaj.gupta@amd.com>,
	"Wang, Wei W" <wei.w.wang@intel.com>,
	"isaku.yamahata@linux.intel.com" <isaku.yamahata@linux.intel.com>
Subject: Re: [ANNOUNCE] PUCK Notes - 2024.04.03 - TDX Upstreaming Strategy
Message-ID: <20240411010352.GB3039520@ls.amr.corp.intel.com>
References: <20240405165844.1018872-1-seanjc@google.com>
 <73b40363-1063-4cb3-b744-9c90bae900b5@intel.com>
 <ZhQZYzkDPMxXe2RN@google.com>
 <a17c6f2a3b3fc6953eb64a0c181b947e28bb1de9.camel@intel.com>
 <ZhQ8UCf40UeGyfE_@google.com>
 <20240410011240.GA3039520@ls.amr.corp.intel.com>
 <1628a8053e01d84bcc7a480947ca882028dbe5b9.camel@intel.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <1628a8053e01d84bcc7a480947ca882028dbe5b9.camel@intel.com>

On Wed, Apr 10, 2024 at 02:03:26PM +0000,
"Huang, Kai" <kai.huang@intel.com> wrote:

> On Tue, 2024-04-09 at 18:12 -0700, Isaku Yamahata wrote:
> > On Mon, Apr 08, 2024 at 06:51:40PM +0000,
> > Sean Christopherson <seanjc@google.com> wrote:
> > 
> > > On Mon, Apr 08, 2024, Edgecombe, Rick P wrote:
> > > > On Mon, 2024-04-08 at 09:20 -0700, Sean Christopherson wrote:
> > > > > > Another option is that, KVM doesn't allow userspace to configure
> > > > > > CPUID(0x8000_0008).EAX[7:0]. Instead, it provides a gpaw field in struct
> > > > > > kvm_tdx_init_vm for userspace to configure directly.
> > > > > > 
> > > > > > What do you prefer?
> > > > > 
> > > > > Hmm, neither.  I think the best approach is to build on Gerd's series to have KVM
> > > > > select 4-level vs. 5-level based on the enumerated guest.MAXPHYADDR, not on
> > > > > host.MAXPHYADDR.
> > > > 
> > > > So then GPAW would be coded to basically best fit the supported guest.MAXPHYADDR within KVM. QEMU
> > > > could look at the supported guest.MAXPHYADDR and use matching logic to determine GPAW.
> > > 
> > > Off topic, any chance I can bribe/convince you to wrap your email replies closer
> > > to 80 chars, not 100?  Yeah, checkpath no longer complains when code exceeds 80
> > > chars, but my brain is so well trained for 80 that it actually slows me down a
> > > bit when reading mails that are wrapped at 100 chars.
> > > 
> > > > Or are you suggesting that KVM should look at the value of CPUID(0X8000_0008).eax[23:16] passed from
> > > > userspace?
> > > 
> > > This.  Note, my pseudo-patch incorrectly looked at bits 15:8, that was just me
> > > trying to go off memory.
> > > 
> > > > I'm not following the code examples involving struct kvm_vcpu. Since TDX
> > > > configures these at a VM level, there isn't a vcpu.
> > > 
> > > Ah, I take it GPAW is a VM-scope knob?  I forget where we ended up with the ordering
> > > of TDX commands vs. creating vCPUs.  Does KVM allow creating vCPU structures in
> > > advance of the TDX INIT call?  If so, the least awful solution might be to use
> > > vCPU0's CPUID.
> > 
> > The current order is, KVM vm creation (KVM_CREATE_VM),
> > KVM vcpu creation(KVM_CREATE_VCPU), TDX VM initialization (KVM_TDX_INIT_VM).
> > and TDX VCPU initialization(KVM_TDX_INIT_VCPU).
> > We can call KVM_SET_CPUID2 before KVM_TDX_INIT_VM.  We can remove cpuid part
> > from struct kvm_tdx_init_vm by vcpu0 cpuid.
> 
> What's the reason to call KVM_TDX_INIT_VM after KVM_CREATE_VCPU?

The KVM_TDX_INIT_VM (it requires cpuids) doesn't requires any order between two,
KVM_TDX_INIT_VM and KVM_CREATE_VCPU.  We can call KVM_TDX_INIT_VM before or
after KVM_CREATE_VCPU because there is no limitation between two.

The v5 TDX QEMU happens to call KVM_CREATE_VCPU and then KVM_TDX_INIT_VM
because it creates CPUIDs for KVM_TDX_INIT_VM from qemu vCPU structures after
KVM_GET_CPUID2.  Which is after KVM_CREATE_VCPU.


> I guess I have been away for this for too long time, but I had believed
> KVM_TDX_INIT_VM is called before creating any vcpu, which turns out to be wrong.
> 
> I am not against to make KVM_TDX_INIT_VM must be called after creating (at least
> one?) vcpu if there's good reason, but it seems if the purpose is just to pass
> CPUID(0x8000_0008).EAX[23:16] to KVM so KVM can determine GPAW for TDX guest,
> then we can also make KVM_TDX_INIT_VM to pass that.
> 
> KVM just need to manually handle CPUID(0x8000_0008) in KVM_TDX_INIT_VM, but
> that's the thing KVM needs to do anyway even if we use vcpu0's CPUID.
> 
> Am I missing anything?

Userspace VMM needs to create CPUID list somehow for KVM_TDX_INIT_VM or
KVM_SET_CPUID2 whichever is first.  It's effortless to reuse the CPUID list
for KVM_SET_CPUID2.
-- 
Isaku Yamahata <isaku.yamahata@intel.com>