Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8;
Message-ID: <33b452ec-c377-481f-b621-e1410ed28ff5@intel.com>
Date:   Wed, 15 Nov 2023 16:23:47 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v6 25/25] KVM: nVMX: Enable CET support for nested guest
To:     Chao Gao <chao.gao@intel.com>
CC:     <seanjc@google.com>, <pbonzini@redhat.com>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <dave.hansen@intel.com>,
        <peterz@infradead.org>, <rick.p.edgecombe@intel.com>,
        <john.allen@amd.com>
References: <20230914063325.85503-1-weijiang.yang@intel.com>
 <20230914063325.85503-26-weijiang.yang@intel.com>
 <ZUGzZiF0Jn8GVcr+@chao-email>
Content-Language: en-US
From:   "Yang, Weijiang" <weijiang.yang@intel.com>
In-Reply-To: <ZUGzZiF0Jn8GVcr+@chao-email>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?djRUSmN5R0lMZkVhWDhBRnMrZWF1UE5WQ2MySEJhMC9wRVZWUlU0NUYwQzlp?=
 =?utf-8?B?Q2RlWTVseVg0bDBUWnF4S21IaERqR3NEOEtQczhqNElTV0JaemJCRVNmZ28r?=
 =?utf-8?B?Ris2WUJrM3c0N01MOTFRSG5XWVN2UmthT2gzTktCRmpUWE5sN0xwQXdaWmh5?=
 =?utf-8?B?amsvNUhaamFDSEdRT05TbkRieHNnRzJNbnNxd3o1bmp0VFd2RFBpMUYvYjhw?=
 =?utf-8?B?bDVTVTZjWUE5YndoYlJEbHhSbDBlbHBIT1VieVNra2pNVlFHL2JYbGxqK1hX?=
 =?utf-8?B?eklzOXViYnJzajVUWFBQeUM4elFiTmVKV1RQM2w1eVhvcXdQbm9RWTRMVWUw?=
 =?utf-8?B?U0c3SURUbU4ybXdoOVk3VFZZNWZLdm12MkNlUzF6c2dlUkhxV2JMQzJHeUph?=
 =?utf-8?B?ZXdIejIyMGFxMVZsaE9adjloS3kxdUtydjBpbG1ZUzdKMVVSanNrenZDSFBH?=
 =?utf-8?B?SHd1OFdSRUN6bytwaEpQSFBsVml3c3VqNE5pYXV5WFo4SUVHNHdqZ0ZRRzd0?=
 =?utf-8?B?VDBaaHVJSTBsbGF2UXdiRHBLOVEwMHNSaWRjUlhEM0RHbFJDWU84TWU4dnl3?=
 =?utf-8?B?WmYyWmYxdjhjNDV2a3kxb1hwMGJQak1ldFYyZmNORkxLSXEzWEt1a1pmUkRa?=
 =?utf-8?B?TXFiN0wzVE16bTJpSngyblpQTkx2QnJJQUJEVWs3clV5allPdWZHUElyRndC?=
 =?utf-8?B?SnRyZXNMUzJwMWV1YVJrOXZ4QW5zSVFHQTgxbTVqMWl0Q0E1UjhGMEx4YXBY?=
 =?utf-8?B?elpQWUNkZXZubXZ5ODZxT2FmS2RrVUc5QXhZQWYzNzdUUVI1ZXlybmExV3NU?=
 =?utf-8?B?TS9QNE5SUG9TRGN5ais5aFppenM4VEZGMGdkOWdwMGxubVprdVRBVEtvdHFN?=
 =?utf-8?B?T3NwRDU1Y2dleGxONHQ1c1ZzbDkyQkNPVDlMMnJ0UXI2MVNZa0xLQ0Z3aStW?=
 =?utf-8?B?c3YzM3haQkQ0dmdmYWVBWEFyaThjQXI1Q0FaQVFodzE5aDR0THcreTZnNXBl?=
 =?utf-8?B?b1JTRmZpZENTWWxNRFZ3VjJKZzNQK2JZeW5rZ2JnS2JqY291elZ6TVVWSHFh?=
 =?utf-8?B?WWZMV2lQeVhpSmRzZE5hdjN6bGwxZGJXWmpQRGg2dGZyRytjYlF4MG5vU0ZX?=
 =?utf-8?B?cGoweFI2YWtYeHJLcHJzMmt4UXNBY01zbnU3YmczelFqR2tmbEhlVW9BTnhE?=
 =?utf-8?B?Ly9uOG9mUVc5RkR1RTdjeUFxcHZRM082ejYzbVdGNThVRlg0QTNXcjQ1dzdj?=
 =?utf-8?B?R0hyamlLeVFKMm44cC9KeHJEQWZIdk9tSENyUXpHZUwyTUxKS0l4S1hXTVNv?=
 =?utf-8?B?S3ZodmJVVndTVkpBMzhQclRiaGJtb3MwMU9Ra3lVU1dFN09SR1NmRHc5MGxW?=
 =?utf-8?B?cXY1QkgzOFdIYjFmdmRseGcrcE1OdmhJTXE2aFg2RGJGcGhYald1R210Snhz?=
 =?utf-8?B?TjBiTld6cjhVWE5GdGljV0NFa2tuVEJzYmZVb3dKcGRvZU12TEY3a2ptWjFF?=
 =?utf-8?B?Z2JoNW5vOG1FQ054SmtEOU5XUHNxYlJEdWY4bXBqSnJKQmZFTE0raEswcFor?=
 =?utf-8?B?azdEcVZiOE43d09uSVpINk05K2wrVXBFclZmMnVUUVBvU3FVMkh0NjNTZ01B?=
 =?utf-8?B?aTVObTJVdllxUHRVdjVZVzV6QkRhN3dPa25xOFRQQTdnbjFvMFJBbU5ES0U3?=
 =?utf-8?B?V1hzMjFzZ0lLR0IwYzhQWU1FbWljaEg1WVBVNWRQQTduajF2QUpLT2hSZ0pW?=
 =?utf-8?B?WEVhR2xMNUlYSGVJUG1vMTZMNzN0TDNTWC9mWDkvc2JlN0VselhNWUJobllE?=
 =?utf-8?B?cnpNZEdveFBoSmE5R0Q5ZkNlQlp3Nmd0UnZvS0VXZURFVHZDdkpHVGU0d1dy?=
 =?utf-8?B?QS9uSGM3QUdEUlpaNFBkZDNBbG9idXZ5YmJ6R2FTNnF2RURYQjVLZE40TGc4?=
 =?utf-8?B?NEJ6WTl6ei9xSUJLWEVKejQ1UUJhY0V0TVl1ZVRIVnRtY1p0NnlkNVZDR1E1?=
 =?utf-8?B?bklCUDRZK1lMZ1I3K29aN0NHajFhRnF3OEFBZjZ1cDgvMXE4Uk1WUHl5Q205?=
 =?utf-8?B?MitCVVJOUXk5Q1RmRWtuQ1NXNllhdEpObUUwbGZMU051aHhZbExveEVscU1Z?=
 =?utf-8?B?dUlWQ3BLZ0UwalN4UXRxbi8wSnM0ZGJUZTRkM2lXMHFrOTAwcnVYU3FoNkIx?=
 =?utf-8?B?MFE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e69965ac-d78a-4fa6-adb1-08dbe5b4365e
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Nov 2023 08:23:57.7070
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: G2FC7S5VrvLnlCDIRJxtkEhMHrdNI+8D7m/4q/Lc46njdiT6H9WFw9ty8Pz8pSYHjOnNN/hbmOX0x1PEFFLYlA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB7771
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 15 Nov 2023 00:24:18 -0800 (PST)

On 11/1/2023 10:09 AM, Chao Gao wrote:
> On Thu, Sep 14, 2023 at 02:33:25AM -0400, Yang Weijiang wrote:
>> Set up CET MSRs, related VM_ENTRY/EXIT control bits and fixed CR4 setting
>> to enable CET for nested VM.
>>
>> Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
>> ---
>> arch/x86/kvm/vmx/nested.c | 27 +++++++++++++++++++++++++--
>> arch/x86/kvm/vmx/vmcs12.c |  6 ++++++
>> arch/x86/kvm/vmx/vmcs12.h | 14 +++++++++++++-
>> arch/x86/kvm/vmx/vmx.c    |  2 ++
>> 4 files changed, 46 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
>> index 78a3be394d00..2c4ff13fddb0 100644
>> --- a/arch/x86/kvm/vmx/nested.c
>> +++ b/arch/x86/kvm/vmx/nested.c
>> @@ -660,6 +660,28 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,
>> 	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> 					 MSR_IA32_FLUSH_CMD, MSR_TYPE_W);
>>
>> +	/* Pass CET MSRs to nested VM if L0 and L1 are set to pass-through. */
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_U_CET, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_S_CET, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_PL0_SSP, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_PL1_SSP, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_PL2_SSP, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_PL3_SSP, MSR_TYPE_RW);
>> +
>> +	nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
>> +					 MSR_IA32_INT_SSP_TAB, MSR_TYPE_RW);
>> +
>> 	kvm_vcpu_unmap(vcpu, &vmx->nested.msr_bitmap_map, false);
>>
>> 	vmx->nested.force_msr_bitmap_recalc = false;
>> @@ -6794,7 +6816,7 @@ static void nested_vmx_setup_exit_ctls(struct vmcs_config *vmcs_conf,
>> 		VM_EXIT_HOST_ADDR_SPACE_SIZE |
>> #endif
>> 		VM_EXIT_LOAD_IA32_PAT | VM_EXIT_SAVE_IA32_PAT |
>> -		VM_EXIT_CLEAR_BNDCFGS;
>> +		VM_EXIT_CLEAR_BNDCFGS | VM_EXIT_LOAD_CET_STATE;
>> 	msrs->exit_ctls_high |=
>> 		VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR |
>> 		VM_EXIT_LOAD_IA32_EFER | VM_EXIT_SAVE_IA32_EFER |
>> @@ -6816,7 +6838,8 @@ static void nested_vmx_setup_entry_ctls(struct vmcs_config *vmcs_conf,
>> #ifdef CONFIG_X86_64
>> 		VM_ENTRY_IA32E_MODE |
>> #endif
>> -		VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS;
>> +		VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS |
>> +		VM_ENTRY_LOAD_CET_STATE;
>> 	msrs->entry_ctls_high |=
>> 		(VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR | VM_ENTRY_LOAD_IA32_EFER |
>> 		 VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL);
>> diff --git a/arch/x86/kvm/vmx/vmcs12.c b/arch/x86/kvm/vmx/vmcs12.c
>> index 106a72c923ca..4233b5ca9461 100644
>> --- a/arch/x86/kvm/vmx/vmcs12.c
>> +++ b/arch/x86/kvm/vmx/vmcs12.c
>> @@ -139,6 +139,9 @@ const unsigned short vmcs12_field_offsets[] = {
>> 	FIELD(GUEST_PENDING_DBG_EXCEPTIONS, guest_pending_dbg_exceptions),
>> 	FIELD(GUEST_SYSENTER_ESP, guest_sysenter_esp),
>> 	FIELD(GUEST_SYSENTER_EIP, guest_sysenter_eip),
>> +	FIELD(GUEST_S_CET, guest_s_cet),
>> +	FIELD(GUEST_SSP, guest_ssp),
>> +	FIELD(GUEST_INTR_SSP_TABLE, guest_ssp_tbl),
> I think we need to sync guest states, e.g., guest_s_cet/guest_ssp/guest_ssp_tbl,
> between vmcs02 and vmcs12 on nested VM entry/exit, probably in
> sync_vmcs02_to_vmcs12() and prepare_vmcs12() or "_rare" variants of them.

After checked around the code, it's necessary to sync related fields from vmcs02 to vmcs12
at nested VM exit so that L1 or userspace can access correct values.
I'll add this part, thanks!