Received-SPF: pass (google.com: domain of linux-kernel+bounces-19685-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <06fdd362-cb7f-47df-9d1a-9b85d2ed05b5@intel.com>
Date: Mon, 8 Jan 2024 22:17:51 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v8 00/26] Enable CET Virtualization
To: Sean Christopherson <seanjc@google.com>
CC: Rick P Edgecombe <rick.p.edgecombe@intel.com>, Chao Gao
	<chao.gao@intel.com>, Dave Hansen <dave.hansen@intel.com>,
	"peterz@infradead.org" <peterz@infradead.org>, "john.allen@amd.com"
	<john.allen@amd.com>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "pbonzini@redhat.com" <pbonzini@redhat.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, "mlevitsk@redhat.com"
	<mlevitsk@redhat.com>
References: <20231221140239.4349-1-weijiang.yang@intel.com>
 <93f118670137933980e9ed263d01afdb532010ed.camel@intel.com>
 <5f57ce03-9568-4739-b02d-e9fac6ed381a@intel.com>
 <6179ddcb25c683bd178e74e7e2455cee63ba74de.camel@intel.com>
 <ZZdLG5W5u19PsnTo@google.com>
 <a2344e2143ef2b9eca0d153c86091e58e596709d.camel@intel.com>
 <ZZdSSzCqvd-3sdBL@google.com>
 <8f070910-2b2e-425d-995e-dfa03a7695de@intel.com>
 <ZZgsipXoXTKyvCZT@google.com>
Content-Language: en-US
From: "Yang, Weijiang" <weijiang.yang@intel.com>
In-Reply-To: <ZZgsipXoXTKyvCZT@google.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?L0t3ZW9oMjlZTmhKVGRmaFhEaTA1OXhNTUg0amdVUnpzNVFKdjdxREthTEZQ?=
 =?utf-8?B?QVV1cEJDNjJCYmJjMklNbVAwUW1jWGZJeTZxc0dBYm5yTUdyeTc2Q0dpM0Zu?=
 =?utf-8?B?YzFqcHptcDVYNDJRQTVNd3ZlRWp4NXNZaktJK1c3OVpjWXBXQVQvNXZ6NFM0?=
 =?utf-8?B?UnQrdGRhSEFCK2hYdnJuY0dkU3VsT2JwYlBVWUs4Y1lOb2swSTVnZGV6T3dv?=
 =?utf-8?B?MEQyTHF5S0lpQkRyRHdYOVFycDcwWEhuUzNvMklSUmZtbTA3ZC9KdW0vK1lZ?=
 =?utf-8?B?UHBEYXFPM0pyZldjb1RUdXpyTTE2YkhxMExUY3Q5UHdLTWFsUHZrang4aTY5?=
 =?utf-8?B?QjhnVzZ2MU13bGo2UkRwN3RySGtQYTZ2YmJlQUU1VWs1aHozQjlONkRrRUln?=
 =?utf-8?B?OSt6S3IwNXgxei9ZajNGU1dyMElLdkllQzlZQlV4ai9LWGJQcW5IczhBaDhM?=
 =?utf-8?B?amwxSGwxWllvbGJMeFlQK0RHdUdYU012Sms1VTI5VGZhNXlFL3ZWbENWNmhP?=
 =?utf-8?B?amtmRDk1Um12SUMwcmh5SkRVY3R2dGg5azBOVVc5VlFZSEJ1aHlPU29yT1Ru?=
 =?utf-8?B?dFhCUzJ6KzlxMlFBL2loWTZMZnB5bFpJUkEzTWt2eU1GaHVpVC8xK0cxYzFr?=
 =?utf-8?B?ZjRmZlVQaDJLQThYcm1ZaHQrYUU0Y3I3V0lzRHBpN2VOOFU3VUlwajhQNTFC?=
 =?utf-8?B?TTNlQTBNYzBIL08xVHllQ29aSy8vRVYxMXpJQzkrT2NQeGptRGpLN0pDdHly?=
 =?utf-8?B?MnlTSVVVMHZ0T1NyME9pc1NJcGpYcHpQZ2o1MnRvcjhLOEFjd1VrSTUvUFB4?=
 =?utf-8?B?RWhOZkhWSFRMbGhibURWZnVqNDgzOFFtYmxRY0U2ZEpINWZBbjBwTElVWWE0?=
 =?utf-8?B?dWpKWmJDQjl5MVlSSEs2Wll4UlhIdFNYZDhNaFFYZktqT1A2L0FQcjQ4VjVj?=
 =?utf-8?B?aTdxT3UrZ3pkUnlVZm1idHJLVHpDM1l0YTFpZFI5MlRaUlBmRDlwNVg4SmJi?=
 =?utf-8?B?TG4rbUdxUm13YkZxOEdVZFJWdkRicExLWWtPYmZ6OWs5Q01HcFp0dkFZbThN?=
 =?utf-8?B?c0FEaGQwa1VQUmJLSW1oUHQxT3pSWEcrcmVzSmxkenQ3TE5hOVZncFZHYm4x?=
 =?utf-8?B?V0xWS3Uzay9RenRRL3pWbDRUZnVqN00wcjF6UWZFK1NLdTcwbGdLbEU5bXlJ?=
 =?utf-8?B?WWsveWRGbFpURTZyamNTbWNNcXNkVWRmZjJkUzhVZjhkWHJ3NXU3bUczeFhW?=
 =?utf-8?B?dGMyUEZkZ2J0VEVKdTRwdVovbmNoVm1nTktRS1VIWWU4OXpCSDduVDNXeDdB?=
 =?utf-8?B?UzJXTTZKcXd0MHBXeHoxRTBWMW1ad0JoOW9ET1Uwa2piMXJqWFY2aTE5ZGRF?=
 =?utf-8?B?aHhxZmVTOWFYWnVlU1NPM3ZZcWloY1dkUmdRY09FaXV6aGJQcFducFVxQzBk?=
 =?utf-8?B?SmFES1pwamZuSWhra2ROemNFb0lwMnMvWk9RaVduUGJ3Zk1pTmVXTWkwSFRJ?=
 =?utf-8?B?eHdyaU9BTTMreW1EVXNkaVFaYWhNWmsxSyt1S3hGVHFSZ0E0QTRscFUzRVpx?=
 =?utf-8?B?S2VpZ0l0QVB3d2k5N3FORzlmM2FWUUYvdi9yTW44c0FtTzFmM3pyUXNxcDN0?=
 =?utf-8?B?dkVyS3FMT3lhblRpZUV4SGloTFBkU0Z5cHMweW9rcVR6RThCcmFjVW9ieVdx?=
 =?utf-8?B?Q0tEN3hUVVdBOFNjRFZibHI3OWtuYk1FcXZqRGxzbFh3aldReUowVnBnVkVi?=
 =?utf-8?B?ZHFFMDlEc0d3aWlYdFNWTHp2RVlJSlhUSXQ3Vk1kNTBxSVhPRlZ4cjZmT2hS?=
 =?utf-8?B?NERqdmJEU0ZJOWY3ZExSZ3UzdHFFOUl4NW1xUnN1MmtDdFR6aVdUTUcxcUs0?=
 =?utf-8?B?NFk3SmcwMnMzNXVDQWNpUkRBR254cnNZWHVYc0xDYkRacTNRQXlJZVdpbDNt?=
 =?utf-8?B?Um9pYmNlV01hZjZJRWVTcEt0ekNpSklvNjBPcXpaSUR2WE9GdUlWZnFBLzJt?=
 =?utf-8?B?YzF2VEMrRDNBdmJqR0doQmk5Wjk4UXFjdGpRVm80eUZRb1dPZW9adDVMeTVu?=
 =?utf-8?B?NGJqQTJWNmNNVmRMazNMdUtndXJjS1BPVWwraG1adXVtNVBVWmsxU2x3eWM3?=
 =?utf-8?B?M3ZZQVpLMFZlQVJSK0o1a2ZPRjM5N0htRXVZb0JuVGFUd1M4RnB2bWFuUnJR?=
 =?utf-8?B?aVE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e479c59b-65f8-4a52-c78b-08dc10549fdf
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jan 2024 14:18:02.8723
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nrpRKMCtZDyigFk1vIXV1k1wXctBpoqnqYFjtDv1sFznk4M+cwNXutxTwvyzN8tHb3OG0xIBk5qRs2US2i70cA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7511
X-OriginatorOrg: intel.com

On 1/6/2024 12:21 AM, Sean Christopherson wrote:
> On Fri, Jan 05, 2024, Weijiang Yang wrote:
>> On 1/5/2024 8:54 AM, Sean Christopherson wrote:
>>> On Fri, Jan 05, 2024, Rick P Edgecombe wrote:
>>>>> For CALL/RET (and presumably any branch instructions with IBT?) other
>>>>> instructions that are directly affected by CET, the simplest thing would
>>>>> probably be to disable those in KVM's emulator if shadow stacks and/or IBT
>>>>> are enabled, and let KVM's failure paths take it from there.
>>>> Right, that is what I was wondering might be the normal solution for
>>>> situations like this.
>>> If KVM can't emulate something, it either retries the instruction (with some
>>> decent logic to guard against infinite retries) or punts to userspace.
>> What kind of error is proper if KVM has to punt to userspace?
> KVM_INTERNAL_ERROR_EMULATION.  See prepare_emulation_failure_exit().
>
>> Or just inject #UD into guest on detecting this case?
> No, do not inject #UD or do anything else that deviates from architecturally
> defined behavior.

Thanks!
But based on current KVM implementation and patch 24, seems that if CET is exposed
to guest, the emulation code or shadow paging mode couldn't be activated at the same time:

In vmx.c,
hardware_setup(void):
if (!cpu_has_vmx_unrestricted_guest() || !enable_ept)
         enable_unrestricted_guest = 0;

in vmx_set_cr0():
[...]
         if (enable_unrestricted_guest)
                 hw_cr0 |= KVM_VM_CR0_ALWAYS_ON_UNRESTRICTED_GUEST;
         else {
                 hw_cr0 |= KVM_VM_CR0_ALWAYS_ON;
                 if (!enable_ept)
                         hw_cr0 |= X86_CR0_WP;

                 if (vmx->rmode.vm86_active && (cr0 & X86_CR0_PE))
                         enter_pmode(vcpu);

                 if (!vmx->rmode.vm86_active && !(cr0 & X86_CR0_PE))
                         enter_rmode(vcpu);
         }
[...]

And in patch 24:

+   if (!cpu_has_load_cet_ctrl() || !enable_unrestricted_guest ||
+       !cpu_has_vmx_basic_no_hw_errcode()) {
+       kvm_cpu_cap_clear(X86_FEATURE_SHSTK);
+       kvm_cpu_cap_clear(X86_FEATURE_IBT);
+   }

Not sure if I missed anything.