Received-SPF: pass (google.com: domain of linux-kernel+bounces-122304-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <4d925a79-d3cf-4555-9c00-209be445310d@intel.com>
Date: Thu, 28 Mar 2024 14:49:56 +1300
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v19 038/130] KVM: TDX: create/destroy VM structure
To: Isaku Yamahata <isaku.yamahata@intel.com>
CC: "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"isaku.yamahata@gmail.com" <isaku.yamahata@gmail.com>, Paolo Bonzini
	<pbonzini@redhat.com>, "Aktas, Erdem" <erdemaktas@google.com>, "Sean
 Christopherson" <seanjc@google.com>, Sagi Shahar <sagis@google.com>, "Chen,
 Bo2" <chen.bo@intel.com>, "Yuan, Hang" <hang.yuan@intel.com>, "Zhang, Tina"
	<tina.zhang@intel.com>, Sean Christopherson
	<sean.j.christopherson@intel.com>, <isaku.yamahata@linux.intel.com>
References: <cover.1708933498.git.isaku.yamahata@intel.com>
 <7a508f88e8c8b5199da85b7a9959882ddf390796.1708933498.git.isaku.yamahata@intel.com>
 <a0627c0f-5c2d-4403-807f-fc800b43fd3b@intel.com>
 <20240327225337.GF2444378@ls.amr.corp.intel.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
In-Reply-To: <20240327225337.GF2444378@ls.amr.corp.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WDZKYStRZFAzU3YyU01qUmpkc3BTM3BkS2twZGV4SEhsQVBwYWI5MTc4RjdQ?=
 =?utf-8?B?RzNVN2FCQXd3V284MnB6NmJwSXNlY3ZzbFlBWW9tOWZNM2taKytaMGkzSGpN?=
 =?utf-8?B?d0Y2azdyQXlqRHY3L1k4emIrWFBtRm5nVnZlc3lwS1ZwMlpKRDdqUlFIUWUw?=
 =?utf-8?B?d2swQlNFUTd6dTB3MWpzalp1elJJODNhVzVxS3crR2M2VW84WnduWlhFaG5N?=
 =?utf-8?B?MDBtWjdqUXROOWhzd09VY1pLMyt2aU1yTDFhNVBWWVN2MXdRUnNLTW5qTEdY?=
 =?utf-8?B?S0FZbmRscDh3YnFkVlZyTjlFMks4VEtXS1FnOTNLUWc4U1U0bEo5QW11QSs2?=
 =?utf-8?B?RWpuRzltVVU3NmlVOGQvVExDdVZiTzcvelR4eE9DSjQ4SmpGWWtuMVBwSjM5?=
 =?utf-8?B?RFRpekozM0lpZzJxSG1EanBDTkFaWC9BQUU0UElNYUR6LzU5dDF0L25LSjNT?=
 =?utf-8?B?Nkl0dHNZUjNFT014NURjeFBrQ2Y0SVlreUhoaVVuajRva1Jsb2hCMTJqTWhF?=
 =?utf-8?B?Zm01UXpQVzlEY0hGMUNxdUhqUWFwbHZnYVMwa01aeGJLRWszeC9MZEJoQkZF?=
 =?utf-8?B?MjZaNnRkMUhOeFVQR1ZKQWhTVVU1QTkyejdoelViVmFoOXN2WkFnbWRxZXdo?=
 =?utf-8?B?SG00ellZWFdYTXVoZjc3QnFsMzlpUW1QL2dmSXF6Ly93RmZaUlBZMXZ0akZi?=
 =?utf-8?B?UDM1Yjk1WXVVaHUzWFRyc3VPSCs4L3BtUkh4MThiNUhTZGRDa2xTNVNIMlNL?=
 =?utf-8?B?WGhEMzBUWWJiOG1SbkZ5QUYrNzJZUGZWTFVuWWsyRndZMGVLOGM5STJyT3pE?=
 =?utf-8?B?b3ZsckY5c2NURFVMWnlSS0NzcmNRU2pZQ1JZK052WUhtVTJlUmNBODkxTXpH?=
 =?utf-8?B?QS9FZFQxS2t4Zm92V1ViZFA2OS9nT09IdC8xOHpXek02N2RQbjF0UjlqcXNF?=
 =?utf-8?B?bS9KdURXem5YZVJpNGZUTjIxTzFUWkticzdieFN6ZGpjVzdlN1JUaStWaVdI?=
 =?utf-8?B?Q3p1M2MrNDN4NjhXWHpCZDQvKytPdkdkSHJMRFdXTTFnMUhXaHY2WXpRMTNV?=
 =?utf-8?B?MzJZNU9GbGFLS1FwdXp6R21GODQ4MU91b1E4WUN4NWg4emN2UzNLK0pHTmda?=
 =?utf-8?B?eW9GRjYyS2xpbisyZkhpN3NjUnhsZEhlZnBlaVhWSXhZNWVrQzZZcWczSDZk?=
 =?utf-8?B?S2VEUTZVaXF1cUZaMzJFV0I5RkpjcEtSREdlWEFaSG5paGNOVWR5V09IUEU1?=
 =?utf-8?B?NUI0REZIUW9TSUJiTmxTVWJ2cy9Sd0dSb241TVNLWG9JZ2FuMXB4anF5bEM5?=
 =?utf-8?B?Tkl4YVNoR2lJRXQ3K1ZmTjBQb2ZneW1zTFhlL1ZqQzA2WjR2ZnlJaWh0ZXZD?=
 =?utf-8?B?K2lzMHk3Zm83MFNUR2RHVjR2SXg0bkpxN1FUTkJzOUtpR21GWEZ0VzhDSS9W?=
 =?utf-8?B?RDc5RnAwVXNUVVlCWlIvSUkxNURpREFSOVhhL2Nnc3NyWGkyY0d6MnJnUVlj?=
 =?utf-8?B?RWI5cEdaMzJUaVF5VkdGR2JZK01oRG1HdlNIZ0hmVjVZbjhnQ3FZczFOSjFt?=
 =?utf-8?B?Vyt5ejlMamFoSzFLNTk1MWRhdkM0TGVoZStIa0dBRExxeXZUeC9VQ0NuM3pp?=
 =?utf-8?B?eGRWbXdsSklLbVgvUHJxcXN0K1NXbGVwaUppL2NNM1RvVm9oK2E0d05PSEF2?=
 =?utf-8?B?Vys2b1FLL0pTaXppR2hoVXoyQVZuZnZhQW44WDVVU1hkQVZ2STRib0pJK2U0?=
 =?utf-8?B?QTE3SUd3dENxdWtMYzJPR2hlNk41dXRYay85OHNFaFducjQvM0V3OVZNTitG?=
 =?utf-8?B?VURYanVBZ3N3QmxZaTEzT3h0YnQzcVhHaXhFRmJBY3hTalZIUTA4WjRLQ1NU?=
 =?utf-8?B?cmwxQjVmQmZ4UVlZT3JIRjlWK0dQRmtZM01KWlNiSUVCa09vNisrK1FXSk5P?=
 =?utf-8?B?TVJLVkVvYzdzelhMckpMVzJpc0NuZWttWHo2QUp6bGNuaVZBdHJvTUNVL0NE?=
 =?utf-8?B?aXVLdVRDKzNoSGlqK1RZR2RjT0lKODI0TC8rRjF6Mkdvd2Z3Qk9pQ3FXTDM1?=
 =?utf-8?B?WVJiREQ4SHl4NU54YWMvNkpPY3ZJNTRTUGx4WDVuQkVhVEhrOGswK2EwTGRi?=
 =?utf-8?Q?QxPMttZ8FzzwfDYc7aNGJqkDF?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 35b84e5d-6dcc-45b2-098b-08dc4ec96524
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Mar 2024 01:50:07.3841
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: dgrp03ZH0sqeXR/AdFXw6SzxLx++vIev+LHYeJiWpXQ9PEN1lgMk4TEnoBdyrtcMMF/5hVmMCU/Tt0pB+6z+tA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5032
X-OriginatorOrg: intel.com


On 28/03/2024 11:53 am, Isaku Yamahata wrote:
> On Tue, Mar 26, 2024 at 02:43:54PM +1300,
> "Huang, Kai" <kai.huang@intel.com> wrote:
> 
>> ... continue the previous review ...
>>
>>> +
>>> +static void tdx_reclaim_control_page(unsigned long td_page_pa)
>>> +{
>>> +	WARN_ON_ONCE(!td_page_pa);
>>
>>  From the name 'td_page_pa' we cannot tell whether it is a control page, but
>> this function is only intended for control page AFAICT, so perhaps a more
>> specific name.
>>
>>> +
>>> +	/*
>>> +	 * TDCX are being reclaimed.  TDX module maps TDCX with HKID
>>
>> "are" -> "is".
>>
>> Are you sure it is TDCX, but not TDCS?
>>
>> AFAICT TDCX is the control structure for 'vcpu', but here you are handling
>> the control structure for the VM.
> 
> TDCS, TDVPR, and TDCX.  Will update the comment.

But TDCX, TDVPR are vcpu-scoped.  Do you want to mention them _here_?

Otherwise you will have to explain them.

[...]

>>> +
>>> +void tdx_mmu_release_hkid(struct kvm *kvm)
>>> +{
>>> +	bool packages_allocated, targets_allocated;
>>> +	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
>>> +	cpumask_var_t packages, targets;
>>> +	u64 err;
>>> +	int i;
>>> +
>>> +	if (!is_hkid_assigned(kvm_tdx))
>>> +		return;
>>> +
>>> +	if (!is_td_created(kvm_tdx)) {
>>> +		tdx_hkid_free(kvm_tdx);
>>> +		return;
>>> +	}
>>
>> I lost tracking what does "td_created()" mean.
>>
>> I guess it means: KeyID has been allocated to the TDX guest, but not yet
>> programmed/configured.
>>
>> Perhaps add a comment to remind the reviewer?
> 
> As Chao suggested, will introduce state machine for vm and vcpu.
> 
> https://lore.kernel.org/kvm/ZfvI8t7SlfIsxbmT@chao-email/

Could you elaborate what will the state machine look like?

I need to understand it.


[...]


> 
> How about this?
> 
> /*
>   * We need three SEAMCALLs, TDH.MNG.VPFLUSHDONE(), TDH.PHYMEM.CACHE.WB(), and
>   * TDH.MNG.KEY.FREEID() to free the HKID.
>   * Other threads can remove pages from TD.  When the HKID is assigned, we need
>   * to use TDH.MEM.SEPT.REMOVE() or TDH.MEM.PAGE.REMOVE().
>   * TDH.PHYMEM.PAGE.RECLAIM() is needed when the HKID is free.  Get lock to not
>   * present transient state of HKID.
>   */

Could you elaborate why it is still possible to have other thread 
removing pages from TD?

I am probably missing something, but the thing I don't understand is why 
this function is triggered by MMU release?  All the things done in this 
function don't seem to be related to MMU at all.

IIUC, by reaching here, you must already have done VPFLUSHDONE, which 
should be called when you free vcpu?  Freeing vcpus is done in 
kvm_arch_destroy_vm(), which is _after_ mmu_notifier->release(), in 
which this tdx_mmu_release_keyid() is called?

But here we are depending vcpus to be freed before tdx_mmu_release_hkid()?

>>> +	/*
>>> +	 * In the case of error in tdx_do_tdh_phymem_cache_wb(), the following
>>> +	 * tdh_mng_key_freeid() will fail.
>>> +	 */
>>> +	err = tdh_mng_key_freeid(kvm_tdx->tdr_pa);
>>> +	if (WARN_ON_ONCE(err)) {
>>
>> I see KVM_BUG_ON() is normally used for SEAMCALL error.  Why this uses
>> WARN_ON_ONCE() here?
> 
> Because vm_free() hook is (one of) the final steps to free struct kvm.  No one
> else touches this kvm.  Because it doesn't harm to use KVM_BUG_ON() here,
> I'll change it for consistency.

I am fine with either.  You can use KVM_BUG_ON() for SEAMCALLs at 
runtime, but use WARN_ON_ONCE() for those involved during VM creation.

[...]

>>
>>> +	err = tdh_phymem_page_wbinvd(set_hkid_to_hpa(kvm_tdx->tdr_pa,
>>> +						     tdx_global_keyid));
>>> +	if (WARN_ON_ONCE(err)) {
>>
>> Again, KVM_BUG_ON()?
>>
>> Should't matter, though.
> 
> Ok, let's use KVM_BUG_ON() consistently.

Ditto.