Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <52ea8386-8652-dd91-23de-9d35781cb131@amd.com>
Date:   Mon, 26 Jun 2023 10:57:12 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Subject: Re: [RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss
Content-Language: en-US
To:     Sean Christopherson <seanjc@google.com>,
        Rick P Edgecombe <rick.p.edgecombe@intel.com>
Cc:     "john.allen@amd.com" <john.allen@amd.com>,
        Weijiang Yang <weijiang.yang@intel.com>,
        "bp@alien8.de" <bp@alien8.de>, "x86@kernel.org" <x86@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "andrew.cooper3@citrix.com" <andrew.cooper3@citrix.com>
References: <20230524155339.415820-1-john.allen@amd.com>
 <20230524155339.415820-7-john.allen@amd.com>
 <161174d013dff42ddfd2950fe33a8054f45c223e.camel@intel.com>
 <ZINGaJnNJ54+klsD@johallen-workstation.lan>
 <9ef2faeaa38e667bd4daa8ee338d4cade452c76c.camel@intel.com>
 <ZJYaNSzup+yuYxNy@google.com>
From:   Tom Lendacky <thomas.lendacky@amd.com>
In-Reply-To: <ZJYaNSzup+yuYxNy@google.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a2FCMHp6UjBQc2xKaUlBVDVjK3U4NjBYbFRaY0xRRVRKZWlrMGU1cDNTOGlS?=
 =?utf-8?B?aDRQdmVVQnVKNVB0cFVwZWRldEVhZ1VZeU52TmJ1MVo5ZmdOQ05CQzI0YlAw?=
 =?utf-8?B?S2pXMTZXRlhxckFPMnFxUzFmRFNuRkk5VzhHWmJlQllFalMzOWRhVFdHQnhC?=
 =?utf-8?B?Y0hEOS95SmsxTHJUWEEyVWpsWjJXc2czZmEvUFdqM0xRWHZHSXhuUDJlQjZG?=
 =?utf-8?B?aHBKT3lCWjJPTmJtMnVYM0RkMnRnL1c3dzBYRnQzaHhuME94eEFtc2R5cGJm?=
 =?utf-8?B?bm9XZmI2N2k5a2hGSyt4NXVMcmJDQWVVRElucUtVSUgycUFWN3pQRE5FQzVQ?=
 =?utf-8?B?SDZ4T3ZWM2FuaVBNRzNrdkJkVjlZVmZnUGVhUHBwN0lTN2k0NVBMRlZYeEdT?=
 =?utf-8?B?RTVodi9yenp5bEpOaVFQUHoyZTVJT21neHdOZ1JLYzN1ekZaYWEwR0xqV2tn?=
 =?utf-8?B?SnJwUEFHZk5LM3lvSVprYU92cDJHNWVqbFdQVEN1cFFKRXB1SXpzeHBpUlhR?=
 =?utf-8?B?T3ZxTjBkaXpjUEMwcytCZm9Vb0lUQktQZkZqeWUzVDdDK3E0SFNnWjYxV1dP?=
 =?utf-8?B?KzRPUDNwVFI3bjlCYUpqODFxVVM2MWRXd3NqdU1EZWlTWmZRekY5RzNsY0Zo?=
 =?utf-8?B?VXUycFZnWnpKQzJKaFB4SDhRamRaZ2wyM3BqakludFd0UzgweDFCVGpkNDdt?=
 =?utf-8?B?NzhMRUF2UG11b0JKamNsbVZONXNmbXlGcVVVNTRTMHFqdWkwMmhVbFVkRmRj?=
 =?utf-8?B?TUVRaTRvWWc1SjFFRjcydWtVZVZHQkVjOFBKV0NSakcrbUFkMjkxWTFKNWlG?=
 =?utf-8?B?TFJKK3RlV0RKQkdyRC9Wd3l5T2lOQUZISlJUSXRPaSt6VGlLdzllMm1aRURW?=
 =?utf-8?B?bnB4V2pFNEhFbEFiMWZDaUpBRTUzN1V0Q0hzTVBiOHRuanJpek1rZ1p2RnVU?=
 =?utf-8?B?cldjenV2akEzSjY1OVM1akIxK2JVNmhpUGRzL2JNaTFWL01Ta0RML29Jc3V6?=
 =?utf-8?B?SUl5NnNsd0I1S0JVV2c1N3MzZ0lWQlZmT3hJdGxSN1NsdkNwN2hVWWtzUTBa?=
 =?utf-8?B?N0MvOWlGZHIzeEpKY2srRzU5c0dJN0RSellQcHVxTmh5QzNRd2gzUFpBWUZi?=
 =?utf-8?B?eG9rTUw1TmJjRFl4YWVmeG1jcGJOOHNPK2ZCK2pSa2luSEVnTDRMZFhjRWNK?=
 =?utf-8?B?UEJuWU5ERDZhTFpJRDBMdlJjQkVPNVVObkUrWlBXSFdUUkpuSFN6cGF5UGpB?=
 =?utf-8?B?TnQ3U1BQSmJCQlY5elIzZWhRN0xEazQxTUhWQXZXZ3QwUDRQYlNZVGJsUXc5?=
 =?utf-8?B?ZTBiV0tNdURFMWdWSzhTaE12YjRMYWwvTm5wTmpZcDhkbVZmL0pNT2Y2U0Jy?=
 =?utf-8?B?YWRaZXU5OVZTS0lSZnYzckVDWTJsWWFXWldwMGJxKzFoeGVtM25yWlZBdmRF?=
 =?utf-8?B?bVlmclM1UDlmVnZZRGhmTFhpZ3lEeU9wNlkvZERJRFdaVUFjWk9USzJhU0RX?=
 =?utf-8?B?blRCbmVrZ2pSU3JZMkcvWXJtZDNDTy9DenZNTDZMdkdHd1VMVFZTcVVNQjZM?=
 =?utf-8?B?SWVUdHBQUFFrd3JjeHd2MElJVk03b0d0Q2ttUllWUlJRLzhFbStva2Q5THJE?=
 =?utf-8?B?Ty9qL1FBZG4wa3JUbHNEQjkxSkoyWklNbTA2am9qaUVmZWZ2MzUvRWY0Mkto?=
 =?utf-8?B?dE9wR1BFQWQ2VjlBbDRmb1FsMHVOU1lycEVGM1dnU1dkcWw5bTVIaVRwWjFh?=
 =?utf-8?B?anN3b0NXWUFCS2VSQ0RBN0crTmRpTnpML0kzdDNHaEY4bGxYbjdMbklETlFu?=
 =?utf-8?B?dy9TK3hhWUdMR1dpSnJscG9yQ0FqSFhCYklmVm1lUDRZUXlCY2tVb2toOTJX?=
 =?utf-8?B?YkluSkY3VU90ZjliTUZscktiRjRKcWhzbzlmMXFUYUhldENVWWVTYUVHVjA0?=
 =?utf-8?B?YTZLY0VXNGJ0ODVmZ0VoNVJNdmU3WXIyWVVTYXZIVEFLK3RXY3Q0dHRObC9U?=
 =?utf-8?B?UDFYdXVGWi9xVStFbnhVNmh1aGNzTnZYOTA5SXFHQTVNQjlXdWR5WDZwcG82?=
 =?utf-8?B?YmxPeElIK1Q0K0lmSEhrV0k5TGFHU3l5eStKNm5nMDdvV3ZPOEpuY3Rtd2Fr?=
 =?utf-8?Q?x8GGJ0cSYemrV4XiwLQ8xQv07?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c89fa4bb-9d86-41ed-b3a4-08db765e0257
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5229.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2023 15:57:14.2703
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fhnmA4iuKNM02cZnGYrMfvskzdZxRRpeDAzkm6g2q7Rx6OAZutoUgpWijzopXlGp7FGwAGg6zaDHgx/Jm4V+QQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7938
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 6/23/23 17:18, Sean Christopherson wrote:
> On Fri, Jun 09, 2023, Rick P Edgecombe wrote:
>> On Fri, 2023-06-09 at 10:34 -0500, John Allen wrote:
>>>> Is setting XFEATURE_MASK_CET_KERNEL here ok? The host kernel will not
>>>> support XFEATURE_MASK_CET_KERNEL. I guess after this there is a small
>>>> window of time where host IA32_XSS could have non-host supported
>>>> supervisor state.
>>>>
>>>> Sort of separately, how does SVM work with respect to saving and
>>>> restoring guest supervisor CET state (I mean the CET_S stuff)?
>>>
>>> Apart from a minor exception involving SEV-ES, we are piggybacking on the
>>> state saving/restoring in Yang Weijiang's x86/VMX series. So by inspection,
>>> it looks like guest supervisor support is broken as the supervisor XSAVES
>>> state and MSRs are not included in that series. I currently don't have a
>>> way to test this case, but I think there are operating systems that support
>>> it. I'll work on getting a guest set up that can actually test this and
>>> hopefully have working guest supervisor support in the next version of the
>>> series.
>>
>> Hmm, interesting. VMX has some separate non-xsaves thing to save and
>> restore the guests supervisor CET state, so Weijiang's series doesn't
>> use the xsaves supervisor CET support.
> 
> Heh, that and Weijiang's series is a wee bit incomplete.
> 
>> Also, since the host might have CR4.CET set for its own reasons, if the host
>> handled an exit with the the guests MSR_IA32_S_CET set it could suddenly be
>> subjected to CET enforcement that it doesn't expect. Waiting to restore it
>> until returning to the guest is too late.
>>
>> At least that's the reasoning on the VMX side as I understand it
> 
> The APM doesn't come right out and say it, but I assume/hope that S_CET is saved
> on VMRUN and loaded on #VMEXIT, i.e. is the same as VMX for all intents and
> purposes.
> 
> The host save state definitely has a field for S_CET, and VMRUN documents that the
> guest values are loaded, I just can't find anything in the APM that explicitly states
> how host S_CET and friends are handled.  E.g. in theory, they could have been
> shoved into VMSAVE+VMLOAD, though I very much doubt that's the case.

Yes, the host value is saved/restored on VMRUN/#VMEXIT. Anything that is 
in the VMCB Save Area (the non-SEV-ES save area) is fully virtualized 
(unless noted otherwise) and doesn't require special processing to 
save/restore the host values.

S_CET is list in the SVM/SEV VMCB save area. Similarly, for 
SEV-ES/SEV-SNP, S_CET is swap type A and is saved/restored on VMRUN/#VMEXIT.

Thanks,
Tom

> 
> John?