Received: by 2002:a05:6358:3188:b0:123:57c1:9b43 with SMTP id q8csp2686321rwd; Sun, 28 May 2023 22:12:12 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6PiAc7D3aJilPU35MO7Nt6kjo3Fgntawe2XzdlERF7XdtQuRjwFOCI1UZpiMGsefY4rrnK X-Received: by 2002:a17:902:ba87:b0:1b0:3224:e53a with SMTP id k7-20020a170902ba8700b001b03224e53amr4333060pls.20.1685337132329; Sun, 28 May 2023 22:12:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685337132; cv=none; d=google.com; s=arc-20160816; b=ixZwRwVrkTKPv2A+B/ziF4blEKW22wZM6OjTcw9rQ2XDni3W4Ra5JJioxZkev/P7+Z Wft+GP+Iheip5yEMegZu7KEJplY+zBkh2RZzUWoVq4nmf38u+QnnVMzimVo4AyOGUiJF RKVysJnBd7MeEf3EySE+Xl0iJ2Qt5wYPU4f3YcGjPUXONqxfoE+OUji5CgI+j5Ewml5u mpw3dV8GBBsFZCsOzlyzKHLqVj9eoeaVH0QpJYxURuImBsGd9FS3CygzXEs3GR/cudNX SFLsiIhVUiMFxMskKxnaZOURH9n+bfc9AMJXdE923nVbMKgoVNBcIgdCWdIP/4aXwFnb y7FQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PxZ/c1MAG3DAdVIj2dbJFtCKFC+sj+zIb3b0/xkTWKo=; b=tI/pyxvq6V7tJowEkk9O8BIj25U6j0Ij6t5xRVEmLrXnHjgNXu9TvaeRN420NVieqt tDYioPWoPmZSaIdK9qvJpFCYO58dGl3bnSkfqt764XXRB2l//TJBVUZchRcDz42XLZXC ng+6ChmHHeuzZ3bLNV+n99U6i2sCwr6b5Tlpdek5jzupuUWC4iZLfbjm1Z+KRh15uPdH 6mmrgmS56cBrx4/ktI7KNDBWMED2IosAhoCVbIihtBZZ78E2UB1ObcdBlQmnpca96Vfo DV1hn++zyeb1yGAVQe2sz3FhvQ4TcVo+a6w7OVDmKzx9F7G+tVQg7gBRyPRmyGktLG+/ W0Lw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=AFnWPk4C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e7-20020a170902744700b001ac939babd7si3349771plt.507.2023.05.28.22.11.59; Sun, 28 May 2023 22:12:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=AFnWPk4C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231726AbjE2EZk (ORCPT + 99 others); Mon, 29 May 2023 00:25:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231844AbjE2EZH (ORCPT ); Mon, 29 May 2023 00:25:07 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FAB51BC0; Sun, 28 May 2023 21:22:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1685334168; x=1716870168; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=42shLDQ/vrZQ+Mcb3ZAvhu4ZGV2Pj2QhDuJvoUhpJW8=; b=AFnWPk4CrpxPQlPdkEiKtpUXDcKBUQ52mvotSQh7Y5V/IIy12ViusSjR +R9WfqZ+FGS+QAJZxdz2RYJERgoU/FxSF7l4BfV8PLrOFuJrDJb0uamQs qljd6oSc9iqVbA/PktKkXkiklm4zzpEbMu7NXy5uFyMZWt3BNc1kwIS8R PZYRwYZQATW1EFJz5MPzzIRa0ELw7g5rU44M9JVqErCHeHwNG2SDppj0m PfLC4FNc1qtBU4vTb0ONO3b8B2kndIMmqK8DMG11f6LeGTlvumhNrVAwA +7ZHnGflK0YMl7n50zzhC0WcppYL6cNQEsiGpXzY6fuTkFSBmfE0W3I4Y A==; X-IronPort-AV: E=McAfee;i="6600,9927,10724"; a="334965965" X-IronPort-AV: E=Sophos;i="6.00,200,1681196400"; d="scan'208";a="334965965" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2023 21:21:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10724"; a="775784289" X-IronPort-AV: E=Sophos;i="6.00,200,1681196400"; d="scan'208";a="775784289" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2023 21:21:13 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com Subject: [PATCH v14 042/113] KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page Date: Sun, 28 May 2023 21:19:24 -0700 Message-Id: <851bcc228f2a99bd4eff10912d09e3aad55a929f.1685333727.git.isaku.yamahata@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-4.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Isaku Yamahata For private GPA, CPU refers a private page table whose contents are encrypted. The dedicated APIs to operate on it (e.g. updating/reading its PTE entry) are used and their cost is expensive. When KVM resolves KVM page fault, it walks the page tables. To reuse the existing KVM MMU code and mitigate the heavy cost to directly walk private page table, allocate one more page to copy the dummy page table for KVM MMU code to directly walk. Resolve KVM page fault with the existing code, and do additional operations necessary for the private page table. To distinguish such cases, the existing KVM page table is called a shared page table (i.e. not associated with private page table), and the page table with private page table is called a private page table. The relationship is depicted below. Add a private pointer to struct kvm_mmu_page for private page table and add helper functions to allocate/initialize/free a private page table page. KVM page fault | | | V | -------------+---------- | | | | V V | shared GPA private GPA | | | | V V | shared PT root dummy PT root | private PT root | | | | V V | V shared PT dummy PT ----propagate----> private PT | | | | | \-----------------+------\ | | | | | V | V V shared guest page | private guest page | non-encrypted memory | encrypted memory | PT: page table - Shared PT is visible to KVM and it is used by CPU. - Private PT is used by CPU but it is invisible to KVM. - Dummy PT is visible to KVM but not used by CPU. It is used to propagate PT change to the actual private PT which is used by CPU. Signed-off-by: Isaku Yamahata --- arch/x86/include/asm/kvm_host.h | 5 ++ arch/x86/kvm/mmu/mmu.c | 7 +++ arch/x86/kvm/mmu/mmu_internal.h | 83 +++++++++++++++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 1 + 4 files changed, 92 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index e8484ec90c61..2b6b713d553d 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -815,6 +815,11 @@ struct kvm_vcpu_arch { struct kvm_mmu_memory_cache mmu_shadow_page_cache; struct kvm_mmu_memory_cache mmu_shadowed_info_cache; struct kvm_mmu_memory_cache mmu_page_header_cache; + /* + * This cache is to allocate private page table. E.g. Secure-EPT used + * by the TDX module. + */ + struct kvm_mmu_memory_cache mmu_private_spt_cache; /* * QEMU userspace and the guest each have their own FPU state. diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a7bf0cecabc2..7caf915a5623 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -694,6 +694,12 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect) 1 + PT64_ROOT_MAX_LEVEL + PTE_PREFETCH_NUM); if (r) return r; + if (kvm_gfn_shared_mask(vcpu->kvm)) { + r = kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_private_spt_cache, + PT64_ROOT_MAX_LEVEL); + if (r) + return r; + } r = kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache, PT64_ROOT_MAX_LEVEL); if (r) @@ -713,6 +719,7 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu) kvm_mmu_free_memory_cache(&vcpu->arch.mmu_pte_list_desc_cache); kvm_mmu_free_memory_cache(&vcpu->arch.mmu_shadow_page_cache); kvm_mmu_free_memory_cache(&vcpu->arch.mmu_shadowed_info_cache); + kvm_mmu_free_memory_cache(&vcpu->arch.mmu_private_spt_cache); kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_header_cache); } diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index 45912ab65075..183b351b21be 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -93,7 +93,23 @@ struct kvm_mmu_page { int root_count; refcount_t tdp_mmu_root_count; }; - unsigned int unsync_children; + union { + struct { + unsigned int unsync_children; + /* + * Number of writes since the last time traversal + * visited this page. + */ + atomic_t write_flooding_count; + }; +#ifdef CONFIG_KVM_MMU_PRIVATE + /* + * Associated private shadow page table, e.g. Secure-EPT page + * passed to the TDX module. + */ + void *private_spt; +#endif + }; union { struct kvm_rmap_head parent_ptes; /* rmap pointers to parent sptes */ tdp_ptep_t ptep; @@ -122,9 +138,6 @@ struct kvm_mmu_page { int clear_spte_count; #endif - /* Number of writes since the last time traversal visited this page. */ - atomic_t write_flooding_count; - #ifdef CONFIG_X86_64 /* Used for freeing the page asynchronously if it is a TDP MMU page. */ struct rcu_head rcu_head; @@ -148,6 +161,68 @@ static inline bool is_private_sp(const struct kvm_mmu_page *sp) return kvm_mmu_page_role_is_private(sp->role); } +#ifdef CONFIG_KVM_MMU_PRIVATE +static inline void *kvm_mmu_private_spt(struct kvm_mmu_page *sp) +{ + return sp->private_spt; +} + +static inline void kvm_mmu_init_private_spt(struct kvm_mmu_page *sp, void *private_spt) +{ + sp->private_spt = private_spt; +} + +static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) +{ + bool is_root = vcpu->arch.root_mmu.root_role.level == sp->role.level; + + KVM_BUG_ON(!kvm_mmu_page_role_is_private(sp->role), vcpu->kvm); + if (is_root) + /* + * Because TDX module assigns root Secure-EPT page and set it to + * Secure-EPTP when TD vcpu is created, secure page table for + * root isn't needed. + */ + sp->private_spt = NULL; + else { + /* + * Because the TDX module doesn't trust VMM and initializes + * the pages itself, KVM doesn't initialize them. Allocate + * pages with garbage and give them to the TDX module. + */ + sp->private_spt = kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_private_spt_cache); + /* + * Because mmu_private_spt_cache is topped up before staring kvm + * page fault resolving, the allocation above shouldn't fail. + */ + WARN_ON_ONCE(!sp->private_spt); + } +} + +static inline void kvm_mmu_free_private_spt(struct kvm_mmu_page *sp) +{ + if (sp->private_spt) + free_page((unsigned long)sp->private_spt); +} +#else +static inline void *kvm_mmu_private_spt(struct kvm_mmu_page *sp) +{ + return NULL; +} + +static inline void kvm_mmu_init_private_spt(struct kvm_mmu_page *sp, void *private_spt) +{ +} + +static inline void kvm_mmu_alloc_private_spt(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) +{ +} + +static inline void kvm_mmu_free_private_spt(struct kvm_mmu_page *sp) +{ +} +#endif + static inline bool kvm_mmu_page_ad_need_write_protect(struct kvm_mmu_page *sp) { /* diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 2a4f38f752e8..cfe8a24b62f8 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -66,6 +66,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) static void tdp_mmu_free_sp(struct kvm_mmu_page *sp) { + kvm_mmu_free_private_spt(sp); free_page((unsigned long)sp->spt); kmem_cache_free(mmu_page_header_cache, sp); } -- 2.25.1