Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Message-ID: <9a0ac0f9-bdb4-4b05-b583-f1b3ae0edda8@intel.com>
Date:   Wed, 15 Nov 2023 16:56:27 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v6 25/25] KVM: nVMX: Enable CET support for nested guest
Content-Language: en-US
To:     Maxim Levitsky <mlevitsk@redhat.com>, Chao Gao <chao.gao@intel.com>
CC:     <seanjc@google.com>, <pbonzini@redhat.com>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <dave.hansen@intel.com>,
        <peterz@infradead.org>, <rick.p.edgecombe@intel.com>,
        <john.allen@amd.com>
References: <20230914063325.85503-1-weijiang.yang@intel.com>
 <20230914063325.85503-26-weijiang.yang@intel.com>
 <ZUGzZiF0Jn8GVcr+@chao-email>
 <e29c2c8c18f989b83ea2d696ae93590fe5c0ff53.camel@redhat.com>
From:   "Yang, Weijiang" <weijiang.yang@intel.com>
In-Reply-To: <e29c2c8c18f989b83ea2d696ae93590fe5c0ff53.camel@redhat.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?b3JCNFB2aTZDbGxVN3IrZ0NWdDNWQk1qTmljc2hIa2xUMXF5ZWh0MU83UFFj?=
 =?utf-8?B?QW93dGIzUG9KUXg5aEJqUkc0YVE1Ym1VNGNpRWpjZHpyU1UrcDE5dkV2QS84?=
 =?utf-8?B?OEFDQ1c4allrVTVURWhrOHl1Ryt5STBuZzFjaWtJbGFobjZrWEFCbU9hMzVv?=
 =?utf-8?B?Vk1UMzJmNEdJc2J3dFppNjFDdHZXdGxUQUxCVFFHYVB2RndOTmxCR2NybmZU?=
 =?utf-8?B?MnRlc3JFaElXS2RONmtuYlJtcTRNcU9EYnlVT25ZSTdORXBOdkZoalFsczhC?=
 =?utf-8?B?STZlVTBRN2J5STlENURSZ2drN0JVQVJEbUQ1b255MFBDMVFQSm4xL0lNS1Bt?=
 =?utf-8?B?KzVDckdaU2pwYldoN0UyMlBRT1dVa1NHMDFSQ3B4aXkxd1FJdGhVbWtya2R0?=
 =?utf-8?B?TG5YOUNIMWdtUGxPSEY0OUFETHMxanJZQnhnTnI2eHBUK3ZrU0oxS2NOZXhs?=
 =?utf-8?B?V3V3LzRWODVndXhSdWt5TlhKMEMvSUlCNTVsRWRhcjhBeThhS3FYenZIT09S?=
 =?utf-8?B?dkVFZ2N6Y2NTS1J2L09KNnNlbm1vZkF3QzF3cVhldFlZazR5d1N0U21ib1VB?=
 =?utf-8?B?TU52anVZVTQwNDlRK0RzZ2oxVkFiTlJDcXJPY2hFaGxoZmtPazNBU0RzRUZK?=
 =?utf-8?B?a3pnZnR3ajVqbVd5R0l2WEFyN3N0Zi9naGVtcVdGajlhV0RUZnZVVVYyS0dk?=
 =?utf-8?B?Q1V5aDRnMXRTUmEyNXhlVlVjdzRSM09iWHNPbWhLTTh1Z2JYVGFYZjVJMlNX?=
 =?utf-8?B?R0FjTG9LYXJYd2VkcTFqeXA0WHZ4R1MvQThKMExhbyt0N3dnQUJ2YmU2cmYw?=
 =?utf-8?B?YllsVmtCR1JLNjE3OXN0STBRa05OQzZ3Tzg2Z2ZMR2tEcmlhNlFmbjNlUHR1?=
 =?utf-8?B?T0llQzAwS1Jzc1BvTEpaeUlvTnVaWEJYTWhPS1ByeU1JbkQxdFI5c1Z0b0hp?=
 =?utf-8?B?eEEyODRjM0ZjclpyRFo5bW0zQjFnc0VNQnZNenhEMDhTWFNnNS9QR2gxNjhH?=
 =?utf-8?B?ajY0S010MGM4ckVUOFovalBoRjBxU2JqOTAwVFRkY0ZBVEdtRzFyeHgzdlFa?=
 =?utf-8?B?a2dTWUJXVGsrMUVEajhaQzBKV05BaElvN1ZTUncyOXJLbEd2Z1k5MW42a0pa?=
 =?utf-8?B?UzcxVStlbUV5aTBibllmRVl4MlYvY1M1a2h4NjRzZmVZZ1ZJT3VSUGxFdXM0?=
 =?utf-8?B?N25SdEc2SFo2aFZDZFUyZmJyM0daS1pVT05uNDg0RGMzcVh0Mk1EUWtURFNS?=
 =?utf-8?B?UnZrbEJlZHJJNXVidytaTUlrTzlYbmw2Y3VvaGVqZWY1RUJudGJ0eUdGbENF?=
 =?utf-8?B?Z09EMG1INzlBbFc1d3JIQzZORmQ3RDg3bFhJK0p4VUpTK0ltTnZQdWM2WllZ?=
 =?utf-8?B?YWVCeE1xWVdMTXRmTEk0S1RKR29lemRHaVBqRSthUVBNRUdMTTViZXdYQ2xQ?=
 =?utf-8?B?MmtHSUYwS1AyemlFRjhJdCt5cktsdTVIekRrOTF3U1FGMERETCtBbm5TaEIr?=
 =?utf-8?B?UDh0YVBjdWtwbmZ1SXk1d2pRdXJQR091T1VSM1BzOWRQd05HeVBCTXBzV1A1?=
 =?utf-8?B?RVVBZjUrM0ZyTVdZK0VnK1NLUi9QVXMvNnRrYmJRUGJNUklXUE1CTk9Ga0J1?=
 =?utf-8?B?S2wwRUJoTnVQVmJkT3VWRVBzMXVQc2JnY01OOS8zMmcrNGVqTGdRY0RwVzJT?=
 =?utf-8?B?Z2FxRjZMaEdjaU5UaU9KdGFQWWNSa1JoV0dyb3QwemZTekJxYmRKeFlrQlZR?=
 =?utf-8?B?VEZDNHdiUEtxNlFUQ0N2dzJsVm1UUmZ3cFVLdlBUKzNNWW1YL21JZVloYUpK?=
 =?utf-8?B?VGh3aGNuNm9DZGlDcE9lNnZpUFFtSGpuRUh6dUdXczNtM2pBdkxRN0lvN3k5?=
 =?utf-8?B?eHg0L09mUldOT3YvWGJkYkpqYllQS3A3TGRHT2lTZnROL3V1d0lpWHZ5ODlE?=
 =?utf-8?B?QVlZZG54K0pFMHZ4cllURVU0OEpUS3h1TUVLYlNtYVorQlhsaSs0OFhqSVRG?=
 =?utf-8?B?SUxsYUhQRklPcXlqcmVlejcrUWFJTHNXcHZOdFRCOThERVNCVmtEdGJDK1ky?=
 =?utf-8?B?eFRYRW5INU54cUs4eFYwWjg5VmdzVytvMVpUV2hCSWF6OFhIOW1nRDVTUzdw?=
 =?utf-8?B?VVIyZVZCZ1B2VUpCRTVMZHdsYThUekM5Ti83YWxjNWp0UnNKcHU3aDBJSUF6?=
 =?utf-8?B?WlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 274e741e-f7c6-41ce-d7a0-08dbe5b8c67f
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Nov 2023 08:56:37.6773
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: VPdaErqrp+x67PhlaZ47PyAIbifmDfvgp+Vly36IHVI7tgsgOhEWCcgqiCqJSbBGd6QOQSIfTbpTbrQXV2p4JQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR11MB7528
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 00:56:57 -0800 (PST)

On 11/1/2023 5:54 PM, Maxim Levitsky wrote:
> On Wed, 2023-11-01 at 10:09 +0800, Chao Gao wrote:
>> On Thu, Sep 14, 2023 at 02:33:25AM -0400, Yang Weijiang wrote:
>>> Set up CET MSRs, related VM_ENTRY/EXIT control bits and fixed CR4 setting
>>> to enable CET for nested VM.
>>>
>>> Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
>>> ---
>>> arch/x86/kvm/vmx/nested.c | 27 +++++++++++++++++++++++++--
>>> arch/x86/kvm/vmx/vmcs12.c |  6 ++++++
>>> arch/x86/kvm/vmx/vmcs12.h | 14 +++++++++++++-
>>> arch/x86/kvm/vmx/vmx.c    |  2 ++
>>> 4 files changed, 46 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
>>> index 78a3be394d00..2c4ff13fddb0 100644
>>> --- a/arch/x86/kvm/vmx/nested.c
>>> +++ b/arch/x86/kvm/vmx/nested.c
>>> @@ -660,6 +660,28 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,
>>> 	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>>> 					 MSR_IA32_FLUSH_CMD, MSR_TYPE_W);
>>>
>>> +	/* Pass CET MSRs to nested VM if L0 and L1 are set to pass-through. */
>>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>>> +					 MSR_IA32_U_CET, MSR_TYPE_RW);
>>> +
>>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>>> +					 MSR_IA32_S_CET, MSR_TYPE_RW);
>>> +
>>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>>> +					 MSR_IA32_PL0_SSP, MSR_TYPE_RW);
>>> +
>>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>>> +					 MSR_IA32_PL1_SSP, MSR_TYPE_RW);
>>> +
>>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>>> +					 MSR_IA32_PL2_SSP, MSR_TYPE_RW);
>>> +
>>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>>> +					 MSR_IA32_PL3_SSP, MSR_TYPE_RW);
>>> +
>>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>>> +					 MSR_IA32_INT_SSP_TAB, MSR_TYPE_RW);
>>> +
>>> 	kvm_vcpu_unmap(vcpu, &vmx->nested.msr_bitmap_map, false);
>>>
>>> 	vmx->nested.force_msr_bitmap_recalc = false;
>>> @@ -6794,7 +6816,7 @@ static void nested_vmx_setup_exit_ctls(struct vmcs_config *vmcs_conf,
>>> 		VM_EXIT_HOST_ADDR_SPACE_SIZE |
>>> #endif
>>> 		VM_EXIT_LOAD_IA32_PAT | VM_EXIT_SAVE_IA32_PAT |
>>> -		VM_EXIT_CLEAR_BNDCFGS;
>>> +		VM_EXIT_CLEAR_BNDCFGS | VM_EXIT_LOAD_CET_STATE;
>>> 	msrs->exit_ctls_high |=
>>> 		VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR |
>>> 		VM_EXIT_LOAD_IA32_EFER | VM_EXIT_SAVE_IA32_EFER |
>>> @@ -6816,7 +6838,8 @@ static void nested_vmx_setup_entry_ctls(struct vmcs_config *vmcs_conf,
>>> #ifdef CONFIG_X86_64
>>> 		VM_ENTRY_IA32E_MODE |
>>> #endif
>>> -		VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS;
>>> +		VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS |
>>> +		VM_ENTRY_LOAD_CET_STATE;
>>> 	msrs->entry_ctls_high |=
>>> 		(VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR | VM_ENTRY_LOAD_IA32_EFER |
>>> 		 VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL);
>>> diff --git a/arch/x86/kvm/vmx/vmcs12.c b/arch/x86/kvm/vmx/vmcs12.c
>>> index 106a72c923ca..4233b5ca9461 100644
>>> --- a/arch/x86/kvm/vmx/vmcs12.c
>>> +++ b/arch/x86/kvm/vmx/vmcs12.c
>>> @@ -139,6 +139,9 @@ const unsigned short vmcs12_field_offsets[] = {
>>> 	FIELD(GUEST_PENDING_DBG_EXCEPTIONS, guest_pending_dbg_exceptions),
>>> 	FIELD(GUEST_SYSENTER_ESP, guest_sysenter_esp),
>>> 	FIELD(GUEST_SYSENTER_EIP, guest_sysenter_eip),
>>> +	FIELD(GUEST_S_CET, guest_s_cet),
>>> +	FIELD(GUEST_SSP, guest_ssp),
>>> +	FIELD(GUEST_INTR_SSP_TABLE, guest_ssp_tbl),
>> I think we need to sync guest states, e.g., guest_s_cet/guest_ssp/guest_ssp_tbl,
>> between vmcs02 and vmcs12 on nested VM entry/exit, probably in
>> sync_vmcs02_to_vmcs12() and prepare_vmcs12() or "_rare" variants of them.
>>
> Aha, this is why I suspected that nested support is incomplete,
> 100% agree.
>
> In particular, looking at Intel's SDM I see that:
>
> HOST_S_CET, HOST_SSP, HOST_INTR_SSP_TABLE needs to be copied from vmcb12 to vmcb02 but not vise versa
> because the CPU doesn't touch them.
>
> GUEST_S_CET, GUEST_SSP, GUEST_INTR_SSP_TABLE should be copied bi-directionally.

Yes, I'll make this part of code complete in next version, thanks!

> This of course depends on the corresponding vm entry and vm exit controls being set.
> That means that it is legal in theory to do VM entry/exit with CET enabled but not use
> VM_ENTRY_LOAD_CET_STATE and/or VM_EXIT_LOAD_CET_STATE,
> because for example nested hypervisor in theory can opt to save/load these itself.
>
> I think that this is all, but I also can't be 100% sure. This thing has to be tested well before
> we can be sure that it works.
>
> Best regards,
> 	Maxim Levitsky
>