Received: by 2002:a05:6358:3188:b0:123:57c1:9b43 with SMTP id q8csp2684203rwd; Sun, 28 May 2023 22:09:28 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4XX+lzNbnB3kjBXdH3mWAgLhYZNPMfXe3d5L3v+lsIzlBz+YfrqNINVTD+vmJ9L6bgEqpQ X-Received: by 2002:a05:6a00:248c:b0:63b:8423:9e31 with SMTP id c12-20020a056a00248c00b0063b84239e31mr12144555pfv.11.1685336967872; Sun, 28 May 2023 22:09:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685336967; cv=none; d=google.com; s=arc-20160816; b=mdH03HxOHNSh7AGk+uSD/xvwVrmC8rb1mXpsL2kepw2PgQiQ7mGWA04OZfWNrrCV/U HRCCKBKdLiVkYCI1011l4Od4mm9DPP/GF2KAd59aWY6ApBsIXwh0BzEobEZuQNRDDx1B v0JGExwua3QVS7r42Gi90wX7rGHG2FCSzDoXseQlW26ap/Udn/BHPe6vPH/yhqu+uML9 /j5nON6kCgzR4EFIrOzQ/2KShMLiZB0p/GLlNYiCp7hGjk6sL9HIQy26oxuOeES7bKC4 u64MY+WgIVIsGBU5HOP+aqtfY1pfLeXHFRhV7W7zDXXkwPbERcz2ra2BFQ0WEdWuZ2pB MA5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zTurV9glGytOg8qtctJQMEQt/Oe8Vmo+UOITYu/gE4k=; b=kE6vQ3lG8G3UqU0X7bX+TM35aLfR+kxQTcFtQTBSxg/820Dre1wxcAtK8UsO9Ln+Cx 5/RgUPO3m+0ehkHQwTqkDbhnX9WISzfS5J6G/k1MzT+WAEvwW67XiaNgHNb+nz87LRqg pPXc+Nz0MJWqPA9eE2yflZexpmnS+7z5MW2csKQHTPnnhbd4ixflwYGEt8f9Ws3eLU8R LxMlOXkZjeV+KFQvJIi0Eg54RHrmnf4y3ZTxqmiRaC4R+15FxkL8C0WKnsHTHbmX8TwQ JiwMscDEVlbYc4ccwvpK9L2y4be8rZOul4C8TZ7eKzrN6pig+jYgLZL+p9hbT0pyHynV HVUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ZjZlb0E3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d27-20020a63735b000000b0053f2601e5a4si4234919pgn.737.2023.05.28.22.09.15; Sun, 28 May 2023 22:09:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=ZjZlb0E3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232088AbjE2E2K (ORCPT + 99 others); Mon, 29 May 2023 00:28:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232029AbjE2E13 (ORCPT ); Mon, 29 May 2023 00:27:29 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1DA94F5; Sun, 28 May 2023 21:24:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1685334276; x=1716870276; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TvWDA352Dzupin4B8o1xuBVn1lldiJvLOGZpCh+JhMo=; b=ZjZlb0E3GyefSWPGqA3s8xRMsAydxxyjaQ1Uem67orxAaicrevGvseMb ofw6QKAZ1uTh5STLyqi+koAtrWr2PDlAabn9r68QP/BN4BRzM2+4XZE+Q wozJ0qJkZf8hKPk4JO2CVsd0cC+g8Lyp8jBHp4kLBAjecpVLOrJo52H04 bWH+xg3+mfENjge1lZRCbkJmglhRJWTJreJcbsfpU/ucRjxX5CDIQZSYN gnWV8h27ZOqshI2lfoN62SrFmCZMqcM1iVXrbyy/LfkTddcqH5iRpj/bs QhyOpefu/GVvrmQ4gkBzdhWnJLac3RvN32B6UMPKYNHEt3Bzn57DkCBKA Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10724"; a="334966109" X-IronPort-AV: E=Sophos;i="6.00,200,1681196400"; d="scan'208";a="334966109" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2023 21:21:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10724"; a="775784395" X-IronPort-AV: E=Sophos;i="6.00,200,1681196400"; d="scan'208";a="775784395" Received: from ls.sc.intel.com (HELO localhost) ([172.25.112.31]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 May 2023 21:21:23 -0700 From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini , erdemaktas@google.com, Sean Christopherson , Sagi Shahar , David Matlack , Kai Huang , Zhi Wang , chen.bo@intel.com Subject: [PATCH v14 063/113] KVM: TDX: Implement TDX vcpu enter/exit path Date: Sun, 28 May 2023 21:19:45 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-4.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Isaku Yamahata This patch implements running TDX vcpu. Once vcpu runs on the logical processor (LP), the TDX vcpu is associated with it. When the TDX vcpu moves to another LP, the TDX vcpu needs to flush its status on the LP. When destroying TDX vcpu, it needs to complete flush and flush cpu memory cache. Track which LP the TDX vcpu run and flush it as necessary. Do nothing on sched_in event as TDX doesn't support pause loop. TDX vcpu execution requires restoring PMU debug store after returning back to KVM because the TDX module unconditionally resets the value. To reuse the existing code, export perf_restore_debug_store. Signed-off-by: Isaku Yamahata --- arch/x86/kvm/vmx/main.c | 21 +++++++++++++++++++-- arch/x86/kvm/vmx/tdx.c | 34 ++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/tdx.h | 33 +++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/x86_ops.h | 2 ++ arch/x86/kvm/x86.c | 1 + 5 files changed, 89 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 6a06b74bf448..80e4638a6de8 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -169,6 +169,23 @@ static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) vmx_vcpu_reset(vcpu, init_event); } +static int vt_vcpu_pre_run(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + /* Unconditionally continue to vcpu_run(). */ + return 1; + + return vmx_vcpu_pre_run(vcpu); +} + +static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) + return tdx_vcpu_run(vcpu); + + return vmx_vcpu_run(vcpu); +} + static void vt_flush_tlb_all(struct kvm_vcpu *vcpu) { if (is_td_vcpu(vcpu)) { @@ -322,8 +339,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .flush_tlb_gva = vt_flush_tlb_gva, .flush_tlb_guest = vt_flush_tlb_guest, - .vcpu_pre_run = vmx_vcpu_pre_run, - .vcpu_run = vmx_vcpu_run, + .vcpu_pre_run = vt_vcpu_pre_run, + .vcpu_run = vt_vcpu_run, .handle_exit = vmx_handle_exit, .skip_emulated_instruction = vmx_skip_emulated_instruction, .update_emulated_instruction = vmx_update_emulated_instruction, diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 00a7e400c1e0..094281389662 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -10,6 +10,9 @@ #include "x86.h" #include "mmu.h" +#include +#include "trace.h" + #undef pr_fmt #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt @@ -441,6 +444,37 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) */ } +u64 __tdx_vcpu_run(hpa_t tdvpr, void *regs, u32 regs_mask); + +static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu, + struct vcpu_tdx *tdx) +{ + guest_state_enter_irqoff(); + tdx->exit_reason.full = __tdx_vcpu_run(tdx->tdvpr_pa, vcpu->arch.regs, 0); + guest_state_exit_irqoff(); +} + +fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) +{ + struct vcpu_tdx *tdx = to_tdx(vcpu); + + if (unlikely(!tdx->initialized)) + return -EINVAL; + if (unlikely(vcpu->kvm->vm_bugged)) { + tdx->exit_reason.full = TDX_NON_RECOVERABLE_VCPU; + return EXIT_FASTPATH_NONE; + } + + trace_kvm_entry(vcpu); + + tdx_vcpu_enter_exit(vcpu, tdx); + + vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET; + trace_kvm_exit(vcpu, KVM_ISA_VMX); + + return EXIT_FASTPATH_NONE; +} + void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) { td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa & PAGE_MASK); diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h index 9d8445324841..af29e1d89657 100644 --- a/arch/x86/kvm/vmx/tdx.h +++ b/arch/x86/kvm/vmx/tdx.h @@ -25,12 +25,45 @@ struct kvm_tdx { u64 tsc_offset; }; +union tdx_exit_reason { + struct { + /* 31:0 mirror the VMX Exit Reason format */ + u64 basic : 16; + u64 reserved16 : 1; + u64 reserved17 : 1; + u64 reserved18 : 1; + u64 reserved19 : 1; + u64 reserved20 : 1; + u64 reserved21 : 1; + u64 reserved22 : 1; + u64 reserved23 : 1; + u64 reserved24 : 1; + u64 reserved25 : 1; + u64 bus_lock_detected : 1; + u64 enclave_mode : 1; + u64 smi_pending_mtf : 1; + u64 smi_from_vmx_root : 1; + u64 reserved30 : 1; + u64 failed_vmentry : 1; + + /* 63:32 are TDX specific */ + u64 details_l1 : 8; + u64 class : 8; + u64 reserved61_48 : 14; + u64 non_recoverable : 1; + u64 error : 1; + }; + u64 full; +}; + struct vcpu_tdx { struct kvm_vcpu vcpu; unsigned long tdvpr_pa; unsigned long *tdvpx_pa; + union tdx_exit_reason exit_reason; + bool initialized; /* diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index dad23ea98052..02094c963ed8 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -151,6 +151,7 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp); int tdx_vcpu_create(struct kvm_vcpu *vcpu); void tdx_vcpu_free(struct kvm_vcpu *vcpu); void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); +fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu); u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio); int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); @@ -174,6 +175,7 @@ static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOP static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTSUPP; } static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {} static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {} +static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) { return EXIT_FASTPATH_NONE; } static inline u8 tdx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio) { return 0; } static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) { return -EOPNOTSUPP; } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index e87b280b64d3..f8b3dda26753 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -311,6 +311,7 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { }; u64 __read_mostly host_xcr0; +EXPORT_SYMBOL_GPL(host_xcr0); static struct kmem_cache *x86_emulator_cache; -- 2.25.1