Received-SPF: pass (google.com: domain of linux-kernel+bounces-180420-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <eab9201e-702e-46bc-9782-d6dfe3da2127@intel.com>
Date: Thu, 16 May 2024 10:17:50 +1200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 08/16] KVM: x86/mmu: Bug the VM if kvm_zap_gfn_range() is
 called for TDX
To: Isaku Yamahata <isaku.yamahata@intel.com>, Sean Christopherson
	<seanjc@google.com>
CC: Rick Edgecombe <rick.p.edgecombe@intel.com>, <pbonzini@redhat.com>,
	<kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
	<isaku.yamahata@gmail.com>, <erdemaktas@google.com>, <sagis@google.com>,
	<yan.y.zhao@intel.com>, <dmatlack@google.com>,
	<isaku.yamahata@linux.intel.com>
References: <20240515005952.3410568-1-rick.p.edgecombe@intel.com>
 <20240515005952.3410568-9-rick.p.edgecombe@intel.com>
 <ZkTWDfuYD-ThdYe6@google.com> <20240515162240.GC168153@ls.amr.corp.intel.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
In-Reply-To: <20240515162240.GC168153@ls.amr.corp.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZUlSTDBZQ2VKYVNiRHpVajBWcFM4cTg5elhxenp5V0RXQTVFcFA4dlUvek5o?=
 =?utf-8?B?UTA1dmlwOElKUE9oNTdCN2Z4citVaHRXNm8vSVRlMG1mVUZkRVFkbW1zdWwy?=
 =?utf-8?B?RUVvb3RlWEx0RlloaFh3UEFYYlN2NUZScExMQlI0Yi9kLzF5Y3RyanJKeXk3?=
 =?utf-8?B?ajlRT2diWTFyRjRMVzkrR0JTSjluaWMxb0NVVUNDOWFia0xqKzRzbGRMZUwx?=
 =?utf-8?B?RDVFa0MrK21nOEhOazBoR2J5TklpZitCODl5R0dQSzBqbU9WSWt2UVhBUDdh?=
 =?utf-8?B?Rm9QWEtCNjFQLzBpVFV0L3hPZG1NdU0zNks2QWUya1FoUjdYcFN5c3JkSVI0?=
 =?utf-8?B?bklyQktRWEtRN0duc1hndUM4K2twZFpOU2tjeURlMThMMzBETE1yckhZaXIw?=
 =?utf-8?B?c1d2dkJucWRMYndwZFl1L1c2Q3d5TTN0OFN5UEJ2NFVteGpudyt6TnQvMSt2?=
 =?utf-8?B?QU5iZ1VDMWZpdXZDOEU2TCtSQ2h3bnV0bCtGaDdQcUh4Uks3OER6WERVMjNP?=
 =?utf-8?B?czhlaXlCbmJRblVqWml4VlRFWEpCQk0waHNHenB4aTJHbDFrdEtzaVJIc2Rm?=
 =?utf-8?B?TEY0dVhCMjZWT0JVMkR1QXBFN3dQL1pwRjN6YXUvWEdVWGoxYWN3UklwQ3NZ?=
 =?utf-8?B?VlQyZ2w2RW15dlFBVTh3aHdodHBmRmtSaUs3Y1g2QTlFanN2N0JwTUt6VTFi?=
 =?utf-8?B?dXcyQ1E3bHU1RlhHYUVPRVFPT0h4OWduMHBUTVhUWEpHK0F4NWlWQlIyTzR3?=
 =?utf-8?B?V1oxYnRscWl5VHhLY2dNeTB1QkJIVjZlWk01Zk8vT0d2dVlIQUMzK21WRzhV?=
 =?utf-8?B?NXBqeEkzOWVEV1ZaMitwdWk4MHBnME5wTUdJZnlBT0FqOHppUU96cXFPOU9u?=
 =?utf-8?B?RTNzNVVPbFZuTXNpT0c5V05xNmd5SXoyY1pSZjlka2tTbm1kak1NekkxZ05k?=
 =?utf-8?B?Q3NIS2cvSjhYa291cDV1QUVmYmF3eGtnV3lkSmUzMi9lREV4Szl3aUNpeE9J?=
 =?utf-8?B?OHllUVgzTmhDUVFDNmFaYmVvT1E2enNmbEZQTnNLb3l5bUJ5MDZXSkJPUkNM?=
 =?utf-8?B?aGE1OE40QTlxQ2JOeXpubjFkNitwNkF4TFNISzA0Mll4alY3Ukl0bnYybGFB?=
 =?utf-8?B?VUExT0VjL1JQTFAvaE9WRVhwK2Z0b2VRSnozcU4yZlp1MDRaZS9HVGxLNzNF?=
 =?utf-8?B?VGNJUVJUSzNsOWV3d0dSTjZKeTJXc2FpMUNZd1ZielNMUkUrdjFKNWh1WUV3?=
 =?utf-8?B?ZFhTckJtMnFxd3BRVGNXanQwWmFBT0syYlJjVGVYeHVJNnA3NTNvTDZzTDNT?=
 =?utf-8?B?WS9BTDhTTUFVR3V2WlNiS1FXcDZNbmFqZ3VaWUVGM250anZkQTdGbWhFMGYz?=
 =?utf-8?B?eG5mOFVnYm5lbXdCLzhqYTNkdnNEYXBPZ21uTmx2K00rK0hoN2l0dWtBNW04?=
 =?utf-8?B?b0ZYTFF4L2EwTVlBOFBBTGNPTHBuQUVRNWgzK25EVUxsQlU3bjJrY2tnQ3Mz?=
 =?utf-8?B?TWZCNXVPQlNQajY3alEwS2JnUW42ZlNZbzdFdjc3TkMveE14dlQyTk9mYVYr?=
 =?utf-8?B?OWVsOVU4VnRvUWVvdndyLzd1bFdKTVYvQ2N3RkJ3YXp3bm9sbUg2dnZEMzc2?=
 =?utf-8?B?blVzWnlucndRdzZwdkV3c3ptOXZxL0FDU2dlRnV0RUVmWjdKTTRxa2NpYTZt?=
 =?utf-8?B?bHFwNG1qWk5FWC9nemJ0eUgyQ05lUGJJVDRIYW1SdG16ejdNL2ZFeU1zckQv?=
 =?utf-8?B?cmdxSXoybDgrSk8rMDRXVmppQ24xVWZiMnVmcm1NVnE3Y3hqdjF3SnMzRHg2?=
 =?utf-8?B?UVpyZ1RXeFJMTG9hYzNGcW1QbHFTeDNKSEZxYTNwRDFZc0JYdFQybS9Za3Vy?=
 =?utf-8?B?M1hHRUwrZjh1QVpWYm5ZVWFNR1c0TE9kY2RwTzlQd2ljMGE5by9QclVpZCtL?=
 =?utf-8?B?d2p4b2Q4cmlUZWdUbHJFZ2JmaXZTb3EvbjRFWUFWWXhPL2dSNk8zaGNDd291?=
 =?utf-8?B?dVZ1YzhGZzZNSHR3QXlHUkZUbnlRVUk1UjdEZDUxZC9jblFBQjRNNVd1Rytn?=
 =?utf-8?B?RjVTZ1pqZkdITEFvK1pUczRLQnBpVzA1d2J4NFlDU2JIbC81T3dmYUVrWTFR?=
 =?utf-8?Q?jftZNOzgKIP2LJL1E7In/qHrv?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f82411b0-fd71-45b1-85aa-08dc752ce11c
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 May 2024 22:17:59.7335
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: a5/XKHZzwmZ3auoZ8EWl+VGmX1l7snX/nANPEPT5hZYRkliG7U169KTGUE0CJUTX2W7lXZW2EWFxmhDDuTyvhA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB5794
X-OriginatorOrg: intel.com


On 16/05/2024 4:22 am, Isaku Yamahata wrote:
> On Wed, May 15, 2024 at 08:34:37AM -0700,
> Sean Christopherson <seanjc@google.com> wrote:
> 
>>> diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
>>> index d5cf5b15a10e..808805b3478d 100644
>>> --- a/arch/x86/kvm/mmu/mmu.c
>>> +++ b/arch/x86/kvm/mmu/mmu.c
>>> @@ -6528,8 +6528,17 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end)
>>>   
>>>   	flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end);
>>>   
>>> -	if (tdp_mmu_enabled)
>>> +	if (tdp_mmu_enabled) {
>>> +		/*
>>> +		 * kvm_zap_gfn_range() is used when MTRR or PAT memory
>>> +		 * type was changed.  TDX can't handle zapping the private
>>> +		 * mapping, but it's ok because KVM doesn't support either of
>>> +		 * those features for TDX. In case a new caller appears, BUG
>>> +		 * the VM if it's called for solutions with private aliases.
>>> +		 */
>>> +		KVM_BUG_ON(kvm_gfn_shared_mask(kvm), kvm);
>>
>> Please stop using kvm_gfn_shared_mask() as a proxy for "is this TDX".  Using a
>> generic name quite obviously doesn't prevent TDX details for bleeding into common
>> code, and dancing around things just makes it all unnecessarily confusing.
>>
>> If we can't avoid bleeding TDX details into common code, my vote is to bite the
>> bullet and simply check vm_type.
> 
> TDX has several aspects related to the TDP MMU.
> 1) Based on the faulting GPA, determine which KVM page table to walk.
>     (private-vs-shared)
> 2) Need to call TDX SEAMCALL to operate on Secure-EPT instead of direct memory
>     load/store.  TDP MMU needs hooks for it.
> 3) The tables must be zapped from the leaf. not the root or the middle.
> 
> For 1) and 2), what about something like this?  TDX backend code will set
> kvm->arch.has_mirrored_pt = true; I think we will use kvm_gfn_shared_mask() only
> for address conversion (shared<->private).
> 
> For 1), maybe we can add struct kvm_page_fault.walk_mirrored_pt
>          (or whatever preferable name)?
> 
> For 3), flag of memslot handles it.
> 
> ---
>   arch/x86/include/asm/kvm_host.h | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index aabf1648a56a..218b575d24bd 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -1289,6 +1289,7 @@ struct kvm_arch {
>   	u8 vm_type;
>   	bool has_private_mem;
>   	bool has_protected_state;
> +	bool has_mirrored_pt;
>   	struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES];
>   	struct list_head active_mmu_pages;
>   	struct list_head zapped_obsolete_pages;
> @@ -2171,8 +2172,10 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level,
>   
>   #ifdef CONFIG_KVM_PRIVATE_MEM
>   #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
> +#define kvm_arch_has_mirrored_pt(kvm) ((kvm)->arch.has_mirrored_pt)
>   #else
>   #define kvm_arch_has_private_mem(kvm) false
> +#define kvm_arch_has_mirrored_pt(kvm) false
>   #endif
>   
>   static inline u16 kvm_read_ldt(void)

I think this 'has_mirrored_pt' (or a better name) is better, because it 
clearly conveys it is for the "page table", but not the actual page that 
any page table entry maps to.

AFAICT we need to split the concept of "private page table itself" and 
the "memory type of the actual GFN".

E.g., both SEV-SNP and TDX has concept of "private memory" (obviously), 
but I was told only TDX uses a dedicated private page table which isn't 
directly accessible for KVV.  SEV-SNP on the other hand just uses normal 
page table + additional HW managed table to make sure the security.

In other words, I think we should decide whether to invoke TDP MMU 
callback for private mapping (the page table itself may just be normal 
one) depending on the fault->is_private, but not whether the page table 
is private:

	if (fault->is_private && kvm_x86_ops->set_private_spte)
		kvm_x86_set_private_spte(...);
	else
		tdp_mmu_set_spte_atomic(...);

And the 'has_mirrored_pt' should be only used to select the root of the 
page table that we want to operate on.

This also gives a chance that if there's anything special needs to be 
done for page allocated for the "non-leaf" middle page table for 
SEV-SNP, it can just fit.

Of course, if 'has_mirrored_pt' is true, we can assume there's a special 
way to operate it, i.e., kvm_x86_ops->set_private_spte etc must be valid.