Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Date:   Wed, 1 Nov 2023 10:09:42 +0800
From:   Chao Gao <chao.gao@intel.com>
To:     Yang Weijiang <weijiang.yang@intel.com>
CC:     <seanjc@google.com>, <pbonzini@redhat.com>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <dave.hansen@intel.com>,
        <peterz@infradead.org>, <rick.p.edgecombe@intel.com>,
        <john.allen@amd.com>
Subject: Re: [PATCH v6 25/25] KVM: nVMX: Enable CET support for nested guest
Message-ID: <ZUGzZiF0Jn8GVcr+@chao-email>
References: <20230914063325.85503-1-weijiang.yang@intel.com>
 <20230914063325.85503-26-weijiang.yang@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230914063325.85503-26-weijiang.yang@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?zpRERBeftl/UsWUOviUy4Xuxi7HJSVF7wfOJAqnjvVLI9H2jZbdkdf/1telQ?=
 =?us-ascii?Q?AMI0ujmtI8dSndX7GP06qsNbjhtSGBNDZAlSQAAGVxlXRFFQEfesLMVgpHZR?=
 =?us-ascii?Q?tC9osjnWbBH9/tlPA1D8XrY76QeSc8/LzezMn61HB87h6wn13UUy6NsZUkHn?=
 =?us-ascii?Q?vUOoCjqRsGtIdmIa/1uCD2bjU3saAGd1JadimnFawZdEr8dlC+r3lib0MONa?=
 =?us-ascii?Q?tVztsk8L90RGaroZiwZC5cr8fRtrCgw1lMh8gHrmptO0M7d3tqGWhgYjnLe9?=
 =?us-ascii?Q?Q+IfVRiN/bHqL8G+JMy34OPqyBEiLy5K6JhmUQ0sse0EgKtfeXQQQBTsvp+q?=
 =?us-ascii?Q?nf+tnsz1gg64t5CAP+wFm6L1oRVv1eI7xZE27tf4VauZfexPelcplW8pEypO?=
 =?us-ascii?Q?3oiJ7mWXZJn2rV7GAYvdtGj7+7Nn787T64DRGPCxVo4e3KDCcfr7xJ8sBjsi?=
 =?us-ascii?Q?NzTqsd3I75aYRHKIttP7pZ/hz72I46UsCUcTIjIVZJ9KC6mTpKgyU+FKzqMI?=
 =?us-ascii?Q?ajuv/ICYVyvWUf+UQ5k7GXVzLY0qz1G++VjGBRLz8XIGDS9y0PIJ6NgIHkYZ?=
 =?us-ascii?Q?8K0637ZCfy7MzzPkyC26KShVnl0RrJ/FozY2cRAI+7OM/jr0yTxB9bHx7dcE?=
 =?us-ascii?Q?i8zqSlADkrr0e+Mn5lQmFbPS863BFQugsorYMLD/FFR9dL6lWE2BAJVGyjNV?=
 =?us-ascii?Q?KuyplcGLqHcXJSUxsOKr+6omFtp7q0dxFOHuyEaQLyX5vgimJ57It61oVAzj?=
 =?us-ascii?Q?V8kKsDt92HsNvH+vJ9xOFzBi+ByLKHAbBMKYnBtigX6w/E0L87ICaDoFy5Ca?=
 =?us-ascii?Q?Y5x0uNBYkcvXoEOQ95mdWcylWKMg7+iQpr6pIlIKzxaJt137rOP1uh0ZEI8W?=
 =?us-ascii?Q?+0KnoUa+ziUI6tF9yaEvr77UGEvwFti0lL8qbRcN/TPhEQq56UI3ZhPxrZ2B?=
 =?us-ascii?Q?C7wHsIcGPmjcXB9pXuXYv0jUuQLOZlNwFiG2XB6TyJ8fgcLtinpnBMsmbaIA?=
 =?us-ascii?Q?wZpxIWzo/w9zl7rAvlwN/4xikNlogBYFX+pNcGenayP8dQwl/z9xWd422btF?=
 =?us-ascii?Q?F4F+0yAIWLb9KYiRfEDsjJ10P7M5IarM29LQt9UOeD1lVpA7tCBXmnVu4YHE?=
 =?us-ascii?Q?igEKyjSLrUSNRSWpOS1ooum3neSd8tRDNBV1O1bwuYwDp8AmWoJgxUVRgiRj?=
 =?us-ascii?Q?3rsITYEOfiTxdlLwVkfGMqv+mzjslO+s0JxbEEoVjuurJGK2f2ZOR3k4QOM0?=
 =?us-ascii?Q?ePKZy05Li0BrpJ0gHsa8BzqZ7052OLe/W8KTT1XC3NlxrwMu6ztzkvn9WVUl?=
 =?us-ascii?Q?/vH34sL7nivF3C6SFPRAvjrvpXa62DDtbfHt9yqaRUFWvuJdbky+uxkmsSTR?=
 =?us-ascii?Q?AfYzmooB0DpvsfAsLSUii+USvtCK99/09Fa5GebJm/j+a4SC0YTihfd4NXxE?=
 =?us-ascii?Q?YastcKQH0g7faGI7KYMHZ8XslSZzO3mT0dtefO/1Jsmb8N5ahrM0PGhPg16M?=
 =?us-ascii?Q?oQAyX/DYb9tM27Gmjc1phTPuHXZLM5PqvMmQ1F4zqkAVsMKESsgmxM/2JTyq?=
 =?us-ascii?Q?Qy3/sq/++ohHLo1HTNPEMkQy8SaxpUc8diCp0t/f?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 78a0b968-9fc7-4c41-0868-08dbda7fa1a4
X-MS-Exchange-CrossTenant-AuthSource: CH3PR11MB8660.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2023 02:09:51.7437
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: e0rtZ+6iAcbu18H+9gkx52ugQrs0g+QlU3eNIFI6mZ3LiDS8I44rawcJF9R+gS2ZoahgNCaQi1QnpvMfwQF2CQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB5800
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 31 Oct 2023 19:10:11 -0700 (PDT)

On Thu, Sep 14, 2023 at 02:33:25AM -0400, Yang Weijiang wrote:
>Set up CET MSRs, related VM_ENTRY/EXIT control bits and fixed CR4 setting
>to enable CET for nested VM.
>
>Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
>---
> arch/x86/kvm/vmx/nested.c | 27 +++++++++++++++++++++++++--
> arch/x86/kvm/vmx/vmcs12.c |  6 ++++++
> arch/x86/kvm/vmx/vmcs12.h | 14 +++++++++++++-
> arch/x86/kvm/vmx/vmx.c    |  2 ++
> 4 files changed, 46 insertions(+), 3 deletions(-)
>
>diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
>index 78a3be394d00..2c4ff13fddb0 100644
>--- a/arch/x86/kvm/vmx/nested.c
>+++ b/arch/x86/kvm/vmx/nested.c
>@@ -660,6 +660,28 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,
> 	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
> 					 MSR_IA32_FLUSH_CMD, MSR_TYPE_W);
> 
>+	/* Pass CET MSRs to nested VM if L0 and L1 are set to pass-through. */
>+	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>+					 MSR_IA32_U_CET, MSR_TYPE_RW);
>+
>+	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>+					 MSR_IA32_S_CET, MSR_TYPE_RW);
>+
>+	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>+					 MSR_IA32_PL0_SSP, MSR_TYPE_RW);
>+
>+	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>+					 MSR_IA32_PL1_SSP, MSR_TYPE_RW);
>+
>+	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>+					 MSR_IA32_PL2_SSP, MSR_TYPE_RW);
>+
>+	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>+					 MSR_IA32_PL3_SSP, MSR_TYPE_RW);
>+
>+	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>+					 MSR_IA32_INT_SSP_TAB, MSR_TYPE_RW);
>+
> 	kvm_vcpu_unmap(vcpu, &vmx->nested.msr_bitmap_map, false);
> 
> 	vmx->nested.force_msr_bitmap_recalc = false;
>@@ -6794,7 +6816,7 @@ static void nested_vmx_setup_exit_ctls(struct vmcs_config *vmcs_conf,
> 		VM_EXIT_HOST_ADDR_SPACE_SIZE |
> #endif
> 		VM_EXIT_LOAD_IA32_PAT | VM_EXIT_SAVE_IA32_PAT |
>-		VM_EXIT_CLEAR_BNDCFGS;
>+		VM_EXIT_CLEAR_BNDCFGS | VM_EXIT_LOAD_CET_STATE;
> 	msrs->exit_ctls_high |=
> 		VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR |
> 		VM_EXIT_LOAD_IA32_EFER | VM_EXIT_SAVE_IA32_EFER |
>@@ -6816,7 +6838,8 @@ static void nested_vmx_setup_entry_ctls(struct vmcs_config *vmcs_conf,
> #ifdef CONFIG_X86_64
> 		VM_ENTRY_IA32E_MODE |
> #endif
>-		VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS;
>+		VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS |
>+		VM_ENTRY_LOAD_CET_STATE;
> 	msrs->entry_ctls_high |=
> 		(VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR | VM_ENTRY_LOAD_IA32_EFER |
> 		 VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL);
>diff --git a/arch/x86/kvm/vmx/vmcs12.c b/arch/x86/kvm/vmx/vmcs12.c
>index 106a72c923ca..4233b5ca9461 100644
>--- a/arch/x86/kvm/vmx/vmcs12.c
>+++ b/arch/x86/kvm/vmx/vmcs12.c
>@@ -139,6 +139,9 @@ const unsigned short vmcs12_field_offsets[] = {
> 	FIELD(GUEST_PENDING_DBG_EXCEPTIONS, guest_pending_dbg_exceptions),
> 	FIELD(GUEST_SYSENTER_ESP, guest_sysenter_esp),
> 	FIELD(GUEST_SYSENTER_EIP, guest_sysenter_eip),
>+	FIELD(GUEST_S_CET, guest_s_cet),
>+	FIELD(GUEST_SSP, guest_ssp),
>+	FIELD(GUEST_INTR_SSP_TABLE, guest_ssp_tbl),

I think we need to sync guest states, e.g., guest_s_cet/guest_ssp/guest_ssp_tbl,
between vmcs02 and vmcs12 on nested VM entry/exit, probably in
sync_vmcs02_to_vmcs12() and prepare_vmcs12() or "_rare" variants of them.