Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Message-ID: <b9fac2d4-61f3-b704-fb43-d3e012239d72@intel.com>
Date:   Wed, 1 Nov 2023 17:22:30 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [PATCH v6 25/25] KVM: nVMX: Enable CET support for nested guest
Content-Language: en-US
To:     Chao Gao <chao.gao@intel.com>
CC:     <seanjc@google.com>, <pbonzini@redhat.com>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <dave.hansen@intel.com>,
        <peterz@infradead.org>, <rick.p.edgecombe@intel.com>,
        <john.allen@amd.com>
References: <20230914063325.85503-1-weijiang.yang@intel.com>
 <20230914063325.85503-26-weijiang.yang@intel.com>
 <ZUGzZiF0Jn8GVcr+@chao-email>
From:   "Yang, Weijiang" <weijiang.yang@intel.com>
In-Reply-To: <ZUGzZiF0Jn8GVcr+@chao-email>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NUZLdHJYM3FhbW1Nc3BoNjN3QWdFMFlxK1d6eEJhaGdHNG9qVkE1MkJ0VW9F?=
 =?utf-8?B?Mzk1NWhMd3NuRVJJU3AxNDRYT2lwbVBLRlBRQjZkQWpCTDhZbXpQc0RPQm5C?=
 =?utf-8?B?a1oxY0RpcEtPMDAyTm9GTXM5MGRaZ29wUUJHWFhuTm1pYzNPUGUwZFdzblNa?=
 =?utf-8?B?UzhnVzVXUzNiUkdBWFd4VHRXTWNPY0dJVXIvWUwrT3IraVM0d3Z0NHFEdGNQ?=
 =?utf-8?B?TGR4d1BCQTJsOGlPUkc4M2FzSnU3c0JobWZRbnZESWFHRzJNeWtsLzJDWUJs?=
 =?utf-8?B?UFVKczNFeVVwRGNnSGNvRUtob3Z6TGZIWkpzMStHeGVreHg4dUd5OHFhSE93?=
 =?utf-8?B?TzZXbkprM3E3aDVtWUxzRTdJSXJLWUorR2M4REdnV29zajM3bUZ6OGN6YUEr?=
 =?utf-8?B?N2ZtUGtSRmxGK0NldjJuaytja1lpUGg3aHQ4RmNhem9YTkl4a1cwRkFyYmlF?=
 =?utf-8?B?NDk1RDdBa0srQlQrL215bktPYnFHL25hK0FSMXZvUmg2Yk11ZjI0aERWZG5T?=
 =?utf-8?B?SU0vc2dwb2UrajdFdi9VUG5KbHl0WWZ4WWlMNFdHbnFRSWJGRUJyR2lSQkc1?=
 =?utf-8?B?UjRNei9LL1dzWTE3RWJtSGFnM2VOSDF5Y2JNM3Y2N3JtL3N3UGI3aTFsd1ho?=
 =?utf-8?B?S1dTbGZVSnp3MzUrRGlRV2ZZZWNWYVQzWUlEa3hQbVF0T2ZIV0x1Nm52WTI1?=
 =?utf-8?B?b01GUzJhNldWVmIrU3l0bG45NHJJMG9ucEdVTFdnT3ZraDZuTUE2amxtNmJ0?=
 =?utf-8?B?WTdySWhnaXZSRkNucloyeG01TllxZE9ROHZGWm5JWk5rQmszeXBHR0s1dkFV?=
 =?utf-8?B?bmJzbGtFOGJjcEd6aU1USnhVY2RFWnlsRWwrZUE5ZTV4UEFjYmlVbnhjK1Nt?=
 =?utf-8?B?WmM2WUFWWCtEN0VWZjBqQ004QjBlWFozdzE1RlIxcXFtbEhlSzlUVlQ5MlFq?=
 =?utf-8?B?eHpscWFwdXhMSy9Nb0VvZ3MxOFdsU2FCMm5HbVk3UUNSbnREajlnWWpsTFdT?=
 =?utf-8?B?OHUveHdGQkl6QVNHMm4zU1A1M3ArdHkxWXBRbDhReEIwckExQjFpWVgvVVNa?=
 =?utf-8?B?Zm54LzE0SWk3MVlaMEc4cEY1WWtWYzl0ditUTUE5WklMNTRKazdGeUM5WG1h?=
 =?utf-8?B?VUkxWDhQUWxUc3o4M25hSzEwaFBaSWdiZXJ5MHgxc2hXS1hIYUtTeUp2WDMw?=
 =?utf-8?B?WlozT1FTRVBEcDc1Z2puK1kvMlkzYlpIT1VlUTFnZVFaMy9UK09CcGFnS2ov?=
 =?utf-8?B?MHhoZjVjMW5ScnpETUNTSE9vV0J6Ym93ampJeHdMVzlrbVBzZ3RsSCtvdmts?=
 =?utf-8?B?MDMrNTJXV0NBbjJodjFwbFdiczhLZWNnTkRPSG5VRVp1ZTRJRlFXRUhTU1Uz?=
 =?utf-8?B?NXVmUnF6UHU5VXFhSWFCQ2FpdWI5QlBuVG1xTFdEbUV2VFFERURrWVdhRU5B?=
 =?utf-8?B?MHk1d2lhQnh6c3dQQjVydmJzeFB0UDVmdDNTMzJERDZhWEcrZGF0S0NoOC9v?=
 =?utf-8?B?clhvNmNQNUdoemlOZDY0b2ZiK1ZGQ083UU0rb3RNemdUN3dEa3NBdUJ5SVRC?=
 =?utf-8?B?T3F3eFRoaEJVU1pyKzF5ZUN2d1JJWndIbUZpTkdXTGtYVS8wSzYxZHhFY1Fh?=
 =?utf-8?B?djVJOTg5cDFLZnRzYmxnU0N2aFRQR21pVFRoQ2I5YWlpMFBmaWFxc0ZGN1Z5?=
 =?utf-8?B?YnovSEdtRlNUY1JTRzQwU1M0MFJJK2pSaTRId3M3SmxYNUpVNzdYZ2VOTnha?=
 =?utf-8?B?RmU2WG03NlEzdjhKaDJjL2FZM1FRUytBb1BjV1lDeE5sQ2lsSVg5Q0tQSTJC?=
 =?utf-8?B?Snc2OU1IK2FXbE1rNWpYSUNVZ1lPRGd0TXlJM1VBUlRFcHBFVHJrdWMwTVc2?=
 =?utf-8?B?N2pyNEsxTXR4NWRMaFp2MDEyUWdQY3p6VTA2ZTk5czBmeFlxQ3ZHMTJUWWox?=
 =?utf-8?B?d1lCVVU4dUNabEpjWU13VE9WS0JrQXVibVNIaGlnNTR2emp3SkMweEliMGhi?=
 =?utf-8?B?NVg1WXNIQzRoWUU1U2ZSeG1STERZc2RnbVN0K2V3M092endWSGFzRjJpSGdL?=
 =?utf-8?B?ZXhlZEN4WkZlMEZKbjM1SEZLYndQMTBKSkM3L2Jzdjc0MStsZ2hoTXZYM3FH?=
 =?utf-8?B?YnJkWiszenFObUtXZjdkTDZlbi9xRFV2bkRhcy8rUzAwMzNOZ3hzdlV4ZHJq?=
 =?utf-8?B?M2c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9e55dfdf-cbb3-4984-c864-08dbdabc1778
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2023 09:22:38.8352
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: AKbpGsg9IyfCDcwN1sG4ywEsAiVVZXosZq4c2TfrtjX4idc06L5QCm46nCMqUBo0OIeRMnpGHhy1jByybHmbHg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4612
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.0 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,
	T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 01 Nov 2023 02:23:11 -0700 (PDT)

On 11/1/2023 10:09 AM, Chao Gao wrote:
> On Thu, Sep 14, 2023 at 02:33:25AM -0400, Yang Weijiang wrote:
>> Set up CET MSRs, related VM_ENTRY/EXIT control bits and fixed CR4 setting
>> to enable CET for nested VM.
>>
>> Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
>> ---
>> arch/x86/kvm/vmx/nested.c | 27 +++++++++++++++++++++++++--
>> arch/x86/kvm/vmx/vmcs12.c |  6 ++++++
>> arch/x86/kvm/vmx/vmcs12.h | 14 +++++++++++++-
>> arch/x86/kvm/vmx/vmx.c    |  2 ++
>> 4 files changed, 46 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
>> index 78a3be394d00..2c4ff13fddb0 100644
>> --- a/arch/x86/kvm/vmx/nested.c
>> +++ b/arch/x86/kvm/vmx/nested.c
>> @@ -660,6 +660,28 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,
>> 	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> 					 MSR_IA32_FLUSH_CMD, MSR_TYPE_W);
>>
>> +	/* Pass CET MSRs to nested VM if L0 and L1 are set to pass-through. */
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_U_CET, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_S_CET, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_PL0_SSP, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_PL1_SSP, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_PL2_SSP, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_PL3_SSP, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_INT_SSP_TAB, MSR_TYPE_RW);
>> +
>> 	kvm_vcpu_unmap(vcpu, &vmx->nested.msr_bitmap_map, false);
>>
>> 	vmx->nested.force_msr_bitmap_recalc = false;
>> @@ -6794,7 +6816,7 @@ static void nested_vmx_setup_exit_ctls(struct vmcs_config *vmcs_conf,
>> 		VM_EXIT_HOST_ADDR_SPACE_SIZE |
>> #endif
>> 		VM_EXIT_LOAD_IA32_PAT | VM_EXIT_SAVE_IA32_PAT |
>> -		VM_EXIT_CLEAR_BNDCFGS;
>> +		VM_EXIT_CLEAR_BNDCFGS | VM_EXIT_LOAD_CET_STATE;
>> 	msrs->exit_ctls_high |=
>> 		VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR |
>> 		VM_EXIT_LOAD_IA32_EFER | VM_EXIT_SAVE_IA32_EFER |
>> @@ -6816,7 +6838,8 @@ static void nested_vmx_setup_entry_ctls(struct vmcs_config *vmcs_conf,
>> #ifdef CONFIG_X86_64
>> 		VM_ENTRY_IA32E_MODE |
>> #endif
>> -		VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS;
>> +		VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS |
>> +		VM_ENTRY_LOAD_CET_STATE;
>> 	msrs->entry_ctls_high |=
>> 		(VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR | VM_ENTRY_LOAD_IA32_EFER |
>> 		 VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL);
>> diff --git a/arch/x86/kvm/vmx/vmcs12.c b/arch/x86/kvm/vmx/vmcs12.c
>> index 106a72c923ca..4233b5ca9461 100644
>> --- a/arch/x86/kvm/vmx/vmcs12.c
>> +++ b/arch/x86/kvm/vmx/vmcs12.c
>> @@ -139,6 +139,9 @@ const unsigned short vmcs12_field_offsets[] = {
>> 	FIELD(GUEST_PENDING_DBG_EXCEPTIONS, guest_pending_dbg_exceptions),
>> 	FIELD(GUEST_SYSENTER_ESP, guest_sysenter_esp),
>> 	FIELD(GUEST_SYSENTER_EIP, guest_sysenter_eip),
>> +	FIELD(GUEST_S_CET, guest_s_cet),
>> +	FIELD(GUEST_SSP, guest_ssp),
>> +	FIELD(GUEST_INTR_SSP_TABLE, guest_ssp_tbl),
> I think we need to sync guest states, e.g., guest_s_cet/guest_ssp/guest_ssp_tbl,
> between vmcs02 and vmcs12 on nested VM entry/exit, probably in
> sync_vmcs02_to_vmcs12() and prepare_vmcs12() or "_rare" variants of them.

Thanks Chao!
Let me double check the nested code part and reply.