Received-SPF: pass (google.com: domain of linux-kernel+bounces-34929-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <dea1ddfa-3f70-49ac-99b9-1077b031451c@intel.com>
Date: Tue, 23 Jan 2024 16:43:57 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v18 060/121] KVM: TDX: TDP MMU TDX support
To: <isaku.yamahata@intel.com>, <kvm@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>
CC: <isaku.yamahata@gmail.com>, Paolo Bonzini <pbonzini@redhat.com>,
	<erdemaktas@google.com>, Sean Christopherson <seanjc@google.com>, Sagi Shahar
	<sagis@google.com>, Kai Huang <kai.huang@intel.com>, <chen.bo@intel.com>,
	<hang.yuan@intel.com>, <tina.zhang@intel.com>
References: <cover.1705965634.git.isaku.yamahata@intel.com>
 <a47c5a9442130f45fc09c1d4ae0e4352054be636.1705965635.git.isaku.yamahata@intel.com>
Content-Language: en-US
From: Chenyi Qiang <chenyi.qiang@intel.com>
In-Reply-To: <a47c5a9442130f45fc09c1d4ae0e4352054be636.1705965635.git.isaku.yamahata@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q05MaGYwbGJTSmhmQjZWcHVueGFFYlZvdnpFSVUzb3pINndlcVF5UFdyTmxH?=
 =?utf-8?B?VE9tSHNJWGZta043Z0lJNHRySWNZT0VtTjdabnpRaC9ibGZzU3RTa0Jud1Bs?=
 =?utf-8?B?M3RyYjNsTHJpc0Y3aTBIQVczc0pXeFd1VGt1L3F0c0o0UDZMeEVCR3dwMDFt?=
 =?utf-8?B?T2pnSjg4YmdNb0tocFF5a2ZnL1NsSnVsTUkrV0Voa3RZbnVZWnQ1WE9wVWZU?=
 =?utf-8?B?SXZ6aEZSd0tNNitNbG15WE1OU3ZDVHZnYTdSSWF5UXdiZmdmeVEzV0VBOUVj?=
 =?utf-8?B?OWtIcWZLSW4zOVhkWkZBSDlnV0pBTFJtVkxaK1ZJVmJIRDNpeFRJbEEvaEVt?=
 =?utf-8?B?TEtjZ1FrS0J1MnFqaGdWSDk5allEOFFlck9jaUpnNVJzT0RLcGUwd1IyY1ZU?=
 =?utf-8?B?R25GNFlKY2lhTXd4YitkMXoxd3NkRm5OTUlPS0QvcjV2YXdPc0J4eEI2Rmcx?=
 =?utf-8?B?a0Fud0ZrcGdYSk1MeEJhcFBSVUhocEZqeXA5VFUyTnJwY01FcXFtZWxlNWtF?=
 =?utf-8?B?SFFqQU1wZmkrUjlJUTJzMXNPUE9IdEZ2dXJTYi9BVVRodmVEQklYUjRlQkdF?=
 =?utf-8?B?eFFYWU80VnZIb3hOZ2JwV2hndDFIeG1ldEJsYnlwTnRkK29KdFBCdzFsRWQy?=
 =?utf-8?B?RWwwclRsNGs0ektoVE9kUDlCbGc5anE0RGtxSTNXakpwT0J3bHVhUzNUanZH?=
 =?utf-8?B?VytoMERBVmRkcTRUSHpiUHVnNkdJR3NkRWdSemU3cVpXazU2ZU1ndjdycVpW?=
 =?utf-8?B?Nk9LVk5OaEVBaGFQNGROMlEwemtkVnJHMUQwbXRRNmV1Y2RySm1WbGVxSkJ4?=
 =?utf-8?B?VkQ4ZXZVVGFXSVUxclk4bWJDd0FPR015WWdKSWtMendtT1YzVWYyNnZUWmMy?=
 =?utf-8?B?WmRWY0lnaTNMSDVrWTJpTTd4UjcrVEhya0RYNkdibGVGd05hN0IwbEYvN0hF?=
 =?utf-8?B?YUljNXUwWjNHUUFyYlNzdVhLZlRPTmJkUlR1UVV2eXA2OVdjT3BKak9KOG1D?=
 =?utf-8?B?ZHVESW1XMTBQaFc2UDd4OVhOV2Z3Y3dOcDdWWkNuRUJmbExXNy9ZRFREY0JI?=
 =?utf-8?B?bU5XZTlCbGVBaGRHZEVJemIwWW1qT0xQaTJGMDB1TVlWQ3FqRHl4QkE2UjNs?=
 =?utf-8?B?SDVxU0REaUQrYkEwZVNSYU9SRVQ5bVNLYmZSNkN6QjkwN09sMGthaENMcEs3?=
 =?utf-8?B?RlVTcjBOajc1ZmpTc0szWFpvUDhGSEtlcW95TVNrZm9aVms4ZWtZUThiQitt?=
 =?utf-8?B?VCtMZEtiQk1Bd3pTYXdMeHU5Q2k3ZzdBdXFHRDl0b0dqeUkrTE1wdlBYa3Zo?=
 =?utf-8?B?R2ZweGdkWld5dTN5QVh4NUFrRnppM0FBTnowdTB0TDNzUWUwTG80ZE82Rjdw?=
 =?utf-8?B?QkdaOVFxOG9IaXU0ZGtRQzZOWjgrNS9uN3l0M25EZTN2WUtib3FzM2pQQnhy?=
 =?utf-8?B?V3pBQVBjZDJ3ZVkxTWFmeDMzVnlHWUhsRVhRbjdLaFRHaE10b3A4WjlWaEpK?=
 =?utf-8?B?WlFHYkZZNU91U1k4WGNCd0RJYjltaE96b0hUZkdvUTBnYjhZbFp5bXRRTmUw?=
 =?utf-8?B?NGJPa0lIdVN2cWE4VWFEMDc4Uk4wZ0IwcythbWdkNDd6cVl2RWpKTU95UWpT?=
 =?utf-8?B?dnA3ZXlGR1JNVzRkMk9pbkF1TllseElCODN3QWU1TGVXaGZjSlQyak1iNVR1?=
 =?utf-8?B?bytzQTVEZ2h6emhLbEJwNFNxa3FkUWNiU1dNN2V1OGRlZ3dROFYxb09nWXFF?=
 =?utf-8?B?SlFpZ0UxU3ZHbXBuNlVPZmhQZU93MWRaMGh6VzYrUmJSU3B2ajV2Skc2UGY1?=
 =?utf-8?B?QzNSU25rK1R4T0ZjT2U3TGdtb1JCclBZeFVuYnJ6TTJQSGtLSjUvcFQweUV3?=
 =?utf-8?B?SHdyS0JIOG0vYjhEY3VmcE5FSFhtWGZOK0srb25kWjdGeE0wVmhjMDNTTHYr?=
 =?utf-8?B?MjFOaUF4OFp4clNNaEZkc1Y5UGZUU1FOMU5MYjZDRHl5bGNpTzVzUEh3U2or?=
 =?utf-8?B?eTN6bGFVNkhBZmRPeEVUakZVNEtjMWdJUGNIVFVUanJwN21NU1hzVjNCc2xk?=
 =?utf-8?B?Mk9Hek1QZUFGWXQzWUNqQ1MzZlB5eFVaSzB0S3NtRU05NkV6eWNjWHdHQXc0?=
 =?utf-8?B?bWxHNnNPeXA4U1ZLanR0MFRQdCtreGY4ZzZURitjb2U3cGtoeVBpSHlwT2xZ?=
 =?utf-8?B?WFE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ebb0296b-6f47-45f8-ac95-08dc1bef74ad
X-MS-Exchange-CrossTenant-AuthSource: DM3PR11MB8735.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jan 2024 08:44:05.1297
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: lC8gS1rZZDOlLCWPIaN3oIpZIqPQ9zcj9bQPBHntIOTS3sK0UhE93EJliUwFetn1xTOUrcKGMROHNkgIZH+z8g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7558
X-OriginatorOrg: intel.com


On 1/23/2024 7:53 AM, isaku.yamahata@intel.com wrote:
> From: Isaku Yamahata <isaku.yamahata@intel.com>
> 
> Implement hooks of TDP MMU for TDX backend.  TLB flush, TLB shootdown,
> propagating the change private EPT entry to Secure EPT and freeing Secure
> EPT page. TLB flush handles both shared EPT and private EPT.  It flushes
> shared EPT same as VMX.  It also waits for the TDX TLB shootdown.  For the
> hook to free Secure EPT page, unlinks the Secure EPT page from the Secure
> EPT so that the page can be freed to OS.
> 
> Propagate the entry change to Secure EPT.  The possible entry changes are
> present -> non-present(zapping) and non-present -> present(population).  On
> population just link the Secure EPT page or the private guest page to the
> Secure EPT by TDX SEAMCALL. Because TDP MMU allows concurrent
> zapping/population, zapping requires synchronous TLB shoot down with the
> frozen EPT entry.  It zaps the secure entry, increments TLB counter, sends
> IPI to remote vcpus to trigger TLB flush, and then unlinks the private
> guest page from the Secure EPT. For simplicity, batched zapping with
> exclude lock is handled as concurrent zapping.  Although it's inefficient,
> it can be optimized in the future.
> 
> For MMIO SPTE, the spte value changes as follows.
> initial value (suppress VE bit is set)
> -> Guest issues MMIO and triggers EPT violation
> -> KVM updates SPTE value to MMIO value (suppress VE bit is cleared)
> -> Guest MMIO resumes.  It triggers VE exception in guest TD
> -> Guest VE handler issues TDG.VP.VMCALL<MMIO>
> -> KVM handles MMIO
> -> Guest VE handler resumes its execution after MMIO instruction
> 
> Signed-off-by: Isaku Yamahata <isaku.yamahata@intel.com>
> 
> ---
> v18:
> - rename tdx_sept_page_aug() -> tdx_mem_page_aug()
> - checkpatch: space => tab
> 
> v15 -> v16:
> - Add the handling of TD_ATTR_SEPT_VE_DISABLE case.
> 
> v14 -> v15:
> - Implemented tdx_flush_tlb_current()
> - Removed unnecessary invept in tdx_flush_tlb().  It was carry over
>   from the very old code base.
> ---
>  arch/x86/kvm/mmu/spte.c    |   3 +-
>  arch/x86/kvm/vmx/main.c    |  71 +++++++-
>  arch/x86/kvm/vmx/tdx.c     | 342 +++++++++++++++++++++++++++++++++++++
>  arch/x86/kvm/vmx/tdx.h     |   2 +-
>  arch/x86/kvm/vmx/tdx_ops.h |   6 +
>  arch/x86/kvm/vmx/x86_ops.h |   6 +
>  6 files changed, 424 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c
> index 318135daf685..83926a35ea47 100644
> --- a/arch/x86/kvm/mmu/spte.c
> +++ b/arch/x86/kvm/mmu/spte.c
> @@ -74,7 +74,8 @@ u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)
>  	u64 spte = generation_mmio_spte_mask(gen);
>  	u64 gpa = gfn << PAGE_SHIFT;
>  
> -	WARN_ON_ONCE(!vcpu->kvm->arch.shadow_mmio_value);
> +	WARN_ON_ONCE(!vcpu->kvm->arch.shadow_mmio_value &&
> +		     !kvm_gfn_shared_mask(vcpu->kvm));
>  
>  	access &= shadow_mmio_access_mask;
>  	spte |= vcpu->kvm->arch.shadow_mmio_value | access;
> diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c
> index e77c045dca84..569f2f67094c 100644
> --- a/arch/x86/kvm/vmx/main.c
> +++ b/arch/x86/kvm/vmx/main.c
> @@ -28,6 +28,7 @@ static int vt_max_vcpus(struct kvm *kvm)
>  
>  	return kvm->max_vcpus;
>  }
> +static int vt_flush_remote_tlbs(struct kvm *kvm);
>  
>  static int vt_hardware_enable(void)
>  {
> @@ -74,8 +75,22 @@ static __init int vt_hardware_setup(void)
>  		pr_warn_ratelimited("TDX requires mmio caching.  Please enable mmio caching for TDX.\n");
>  	}
>  
> +	/*
> +	 * TDX KVM overrides flush_remote_tlbs method and assumes
> +	 * flush_remote_tlbs_range = NULL that falls back to
> +	 * flush_remote_tlbs.  Disable TDX if there are conflicts.
> +	 */
> +	if (vt_x86_ops.flush_remote_tlbs ||
> +	    vt_x86_ops.flush_remote_tlbs_range) {
> +		enable_tdx = false;
> +		pr_warn_ratelimited("TDX requires baremetal. Not Supported on VMM guest.\n");
> +	}
> +
>  	enable_tdx = enable_tdx && !tdx_hardware_setup(&vt_x86_ops);
>  
> +	if (enable_tdx)
> +		vt_x86_ops.flush_remote_tlbs = vt_flush_remote_tlbs;
> +
>  	return 0;
>  }
>  
I hit some build issues when CONFIG_HYPERV=n:

error: ‘struct kvm_x86_ops’ has no member named ‘flush_remote_tlbs’
error: ‘struct kvm_x86_ops’ has no member named ‘flush_remote_tlbs_range’

I think it should be related to the commit
https://lore.kernel.org/all/20231018192325.1893896-1-seanjc@google.com/