Received-SPF: pass (google.com: domain of linux-kernel+bounces-17659-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <8f070910-2b2e-425d-995e-dfa03a7695de@intel.com>
Date: Fri, 5 Jan 2024 17:28:27 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v8 00/26] Enable CET Virtualization
Content-Language: en-US
To: Sean Christopherson <seanjc@google.com>, Rick P Edgecombe
	<rick.p.edgecombe@intel.com>
CC: Chao Gao <chao.gao@intel.com>, Dave Hansen <dave.hansen@intel.com>,
	"peterz@infradead.org" <peterz@infradead.org>, "john.allen@amd.com"
	<john.allen@amd.com>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "pbonzini@redhat.com" <pbonzini@redhat.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, "mlevitsk@redhat.com"
	<mlevitsk@redhat.com>
References: <20231221140239.4349-1-weijiang.yang@intel.com>
 <93f118670137933980e9ed263d01afdb532010ed.camel@intel.com>
 <5f57ce03-9568-4739-b02d-e9fac6ed381a@intel.com>
 <6179ddcb25c683bd178e74e7e2455cee63ba74de.camel@intel.com>
 <ZZdLG5W5u19PsnTo@google.com>
 <a2344e2143ef2b9eca0d153c86091e58e596709d.camel@intel.com>
 <ZZdSSzCqvd-3sdBL@google.com>
From: "Yang, Weijiang" <weijiang.yang@intel.com>
In-Reply-To: <ZZdSSzCqvd-3sdBL@google.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eXhjdk8zMm9IV0VCQ2VBS2JxaDhodnhSVVNHdW1aZjgrTVp1R1NXOVhMOW9l?=
 =?utf-8?B?ZU5za3BHb2ZOSWN3QkpGWksyWk5sQldSWFVRSmdISTJrMkxNZm5GSFFCZHl1?=
 =?utf-8?B?bFVWdm5yaEVWd1NBR1VQWG1vSDY1aWZQVVNwdExRWFVRNG1JdVlYcGNUVFdk?=
 =?utf-8?B?L0tndENCNFVSNFRuZzhPODNlZzNMTXMvdDdhK2V2Z0xobmp1YUhkdnRGL2RZ?=
 =?utf-8?B?ZTI4TDBsVHZpUEJJNWE2djl5eWh2a1VWZ3NSU3FvZko1OWlvRWM1bTJSejNX?=
 =?utf-8?B?eFJBM1B4WldDaHN4L1hJRmpNMnVDUGxSQ095MVNqdUxkNmt2cVZRWWEyMms0?=
 =?utf-8?B?NTAyUTlzSjJHZGthb2ZWWXNYK2E2enNiTXVnUEdqN3RBYjhBMnlWeG9HcU40?=
 =?utf-8?B?OXZQU3FubnhqUTNMSVhBa09uMUxwVzRBUWJTR3ppSnFZdnNoMVgzNXFMM1JQ?=
 =?utf-8?B?cWczaVAya2dWSisxZ2cwU25udlBaQ3RoRUdCVXFqdGUrb0lQYWJJV094V0NT?=
 =?utf-8?B?NE14QURIWnI5Z1N6T1BnTk9GMW9yS0Z5bmdMSnBJR0RSbW1vTmxscU16eHlQ?=
 =?utf-8?B?OEgyMzRnWmk5OUxJRGZORG5TUitwUzhQa0x0OCtFcWdCNEdjanl4a1FWWW9p?=
 =?utf-8?B?RjRZTHBsWktySndEU3gwTlBnTFpZMGFJNG1wRlQxbXorSlNlMDRxeTJSRVJ1?=
 =?utf-8?B?dDFnalJFVW1zUC9BMzlwbVFETkJieWxzVXMzRXNidnp2YzZhNGFld3NZQnlm?=
 =?utf-8?B?VlhqTXoxWXpsbTdESjBleVdHblI1ZnkyZzZQT1NncGl5VXdvTHpIK2d5QlJ4?=
 =?utf-8?B?T25yc0pzT1hXZ2RrelpEYVlOYUZxSWNMVWdhd3VJZlEzc2ZqVEl5OWFuMCtt?=
 =?utf-8?B?eWJpckgwYjE5cWZXL0EvM2kvdkM3NVVQK3U5L0tiVktIN3hYbXBzbUNGWGpZ?=
 =?utf-8?B?Yk9qYnREck13UksyT0xYU3A1a1hXeXFLQ2Rad1dWNDY3ME5lV0YyZU9RUE9E?=
 =?utf-8?B?L0VQamFtZ2psbW1JSjZHZFFhSjd4UUI1dCs1c2JlQU03cFdNT0VNNDI2bGFw?=
 =?utf-8?B?Vkl3Y0VzRklZQjlCR1pqc3hJQU9sSzVXYzJBY0tBWFhTK0VEcXd3UitjckJB?=
 =?utf-8?B?U0s0THlQeDhWL0hGRHN3dXEvTEp5bFB1OUdyOGNnOFBtNWVKbndnNWwwNXhp?=
 =?utf-8?B?OE9KZzc0Z2s2UmpVMjlmSks0NVVjaXFsSWRTR3hZcEc1Wko0KzJRMUtBYU1r?=
 =?utf-8?B?bWhjanhBWC9GMTkzWWxDZEhYQSsvY0hUVy9SbTBRcEQrSG8zVFA4REorNWR6?=
 =?utf-8?B?dVp5SzdTdmNOcjJGMVhEanFsUkVITGtxVUJTTHoxcyttcHVlOWRwZy9Tb2RH?=
 =?utf-8?B?cUJ0YS96akJiOVNGWGNSc2FOZm1QU1ZNSXFqdmhLcFRKbHg1cGh0VktleDVk?=
 =?utf-8?B?Q2kyMFVMSDhTNHd5dGQzVGdraFc4SDdIMWRGTDh4QVRxQkR1ek1XZFMzc2J0?=
 =?utf-8?B?RlY5V21lR1FVejl0bSs1QTkxbTZTbzVpdjFSeUVsYWozQzVnYWhadjZXampU?=
 =?utf-8?B?cUkrT2ZPVEVtWUwrZ2Z4NmJZUnJ2Y2h4dEN6a2tsR1dMYnBLWkR5Yjh2bENH?=
 =?utf-8?B?TUFYWnRlMm1HVWFNTmpOL25lemJYd2NXWXFEZ3dpZ3NEY21xcWI4bThZcWdY?=
 =?utf-8?B?RE1YRVFONnRkSVNXZXRTV29uU0ZtMzcvSUhHNFlXQVZ1bThqNkl2RnVscnJz?=
 =?utf-8?B?dmd4bm9mWEJTT3hxSTUrTVBZcUh2Y3hURFNSa0J4OWVDS042cWI5UEw2b1Jh?=
 =?utf-8?B?VHY2Mms2bG0zdWQzRzJ4enlQMzY0Q0NCd3RCNStTWEJvTm1Wa0QzdkNYaVRV?=
 =?utf-8?B?OFhTL2VCMEFNby9UNUNGazR3c3NGc2dwNXo4bUhvODAvWUprVkc5bU5ZRVFH?=
 =?utf-8?B?WVFiVjJjRlNHSWFvUmlDUmp5Y3oyc1B3ajNWblNPYTI2RXo5ekt2ODFGQ2hw?=
 =?utf-8?B?QU1ENkNwMTBtbTZNZ2pxWHN2QVRBUTdFOUxEdGZTR2pRbzgrTDBQTjBCUjNX?=
 =?utf-8?B?MnFpODJPVU5nVFNxeVVPODJLT0RET0VUWjVjNUgzMFVWemkxN2l5MGY1ZWZM?=
 =?utf-8?B?TFptbUpTUHNtVzVQRXhNaldCQ2dkdE9KZ0h2UEFxNjNBSVVtTXgzb1crTzA4?=
 =?utf-8?B?ZWc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: bc956235-3ab2-47ac-015d-08dc0dd0b386
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2024 09:28:40.0062
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9JRKxDEU1ukzmUvgzTW1X/UU2B4s7BggVqpOi6HPbotN5hAhmpaS6KnLO45JiEagbeXBQntD8AOIfV0fGYM+JQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR11MB8379
X-OriginatorOrg: intel.com

On 1/5/2024 8:54 AM, Sean Christopherson wrote:
> On Fri, Jan 05, 2024, Rick P Edgecombe wrote:
>> On Thu, 2024-01-04 at 16:22 -0800, Sean Christopherson wrote:
>>> No, the days of KVM making shit up from are done.  IIUC, you're advocating
>>> that it's ok for KVM to induce a #CP that architecturally should not
>>> happen.  That is not acceptable, full stop.
>> Nope, not advocating that at all.
> Heh, wrong "you".  That "you" was directed at Weijiang, who I *think* is saying
> that clobbering the shadow stack by emulating CALL+RET and thus inducing a bogus
> #CP in the guest is ok.

My fault, I just thought of the normal execution instead of the subverting cases :-)

>
>> I'm noticing that in this series KVM has special emulator behavior that
>> doesn't match the HW when CET is enabled. That it *skips* emitting #CPs (and
>> other CET behaviors SW depends on), and wondering if it is a problem.
> Yes, it's a problem.  But IIUC, as is KVM would also induce bogus #CPs (which is
> probably less of a problem in practice, but still not acceptable).

I'd choose to stop emulating the CET sensitive instructions while CET is enabled in guest
as re-enter guest after emulation would raise some kind of risk, but I don't know how to
stop the emulation decently.

>> I'm worried that there is some way attackers will induce the host to
>> emulate an instruction and skip CET enforcement that the HW would
>> normally do.
> Yep.  The best behavior for this is likely KVM's existing behavior, i.e. retry
> the instruction in the guest, and if that doesn't work, kick out to userspace and
> let userspace try to sort things out.
>
>>> For CALL/RET (and presumably any branch instructions with IBT?) other
>>> instructions that are directly affected by CET, the simplest thing would
>>> probably be to disable those in KVM's emulator if shadow stacks and/or IBT
>>> are enabled, and let KVM's failure paths take it from there.
>> Right, that is what I was wondering might be the normal solution for
>> situations like this.
> If KVM can't emulate something, it either retries the instruction (with some
> decent logic to guard against infinite retries) or punts to userspace.

What kind of error is proper if KVM has to punt to userspace? Or just inject #UD into guest
on detecting this case?

>
> Or if the platform owner likes to play with fire and doesn't enable
> KVM_CAP_EXIT_ON_EMULATION_FAILURE, KVM will inject a #UD (and still exit to
> userspace if the emulation happened at CPL0).  And yes, that #UD is 100% KVM
> making shit up, and yes, it has caused problems and confusion. :-)
>   
>>> Then, *if* a use case comes along where the guest is utilizing CET and
>>> "needs" KVM to emulate affected instructions, we can add the necessary
>>> support the emulator.
>>>
>>> Alternatively, if teaching KVM's emulator to play nice with shadow stacks
>>> and IBT is easy-ish, just do that.
>> I think it will not be very easy.
> Yeah.  As Jim alluded to, I think it's probably time to admit that emulating
> instructions for modern CPUs is a fools errand and KVM should simply stop trying.
>