Received-SPF: pass (google.com: domain of linux-kernel+bounces-16302-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <5f57ce03-9568-4739-b02d-e9fac6ed381a@intel.com>
Date: Thu, 4 Jan 2024 15:11:43 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v8 00/26] Enable CET Virtualization
Content-Language: en-US
To: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>
CC: "pbonzini@redhat.com" <pbonzini@redhat.com>, "seanjc@google.com"
	<seanjc@google.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>, "Hansen,
 Dave" <dave.hansen@intel.com>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "john.allen@amd.com" <john.allen@amd.com>,
	"peterz@infradead.org" <peterz@infradead.org>, "Gao, Chao"
	<chao.gao@intel.com>, "mlevitsk@redhat.com" <mlevitsk@redhat.com>
References: <20231221140239.4349-1-weijiang.yang@intel.com>
 <93f118670137933980e9ed263d01afdb532010ed.camel@intel.com>
From: "Yang, Weijiang" <weijiang.yang@intel.com>
In-Reply-To: <93f118670137933980e9ed263d01afdb532010ed.camel@intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?c040MHBNUC9WbFFSQzZBR2I0L1drTmdjeHN4NkVqTEhyenBjc2JFdDBjajlm?=
 =?utf-8?B?dlZwUUlyTjBYczFUYzR2S2V0dmNhOTNxSzlrcjVrK1MwQ0p1MHV1bGpCaWhY?=
 =?utf-8?B?MnFDbVRPUS9XKy8ycEo0L1RNQzlhUWMwWE0xUXh1eUpncDU3R1pBVitHL1A4?=
 =?utf-8?B?M1hEb3kxTzFGZnpuQWQrbG5FZ0tJd01WQ3RZcnUzMC9SVHdlcmJSSkZTV3Yw?=
 =?utf-8?B?MWdKL3BXVDU3NWVidE81ME52THplNkU2OVVEZnhvOFhEdUFBUWVaRnpoS0Z0?=
 =?utf-8?B?UzBSWWdINEZpcmMrSGRldDAxWnFxVlArM1Y3OTZOVlRKVVFDb1BQM3dUWXpW?=
 =?utf-8?B?OHRiMC9HQUJISTVjbDJqL1NLcURJOFRLSE1tYmZjSVFmRFd3TCtTYXFiRzBG?=
 =?utf-8?B?UFprdnFGUnd4bllGTmlleWhsemp0WmJGS2c0QzcxMEEzaHVUR1VjbmNTTEs5?=
 =?utf-8?B?QkUvcWNDZnlaUU1UL2Jqc2hrcGwyUzhSQzlZemd6YTJoOHIyZ0hnN0ozUHkx?=
 =?utf-8?B?bjdpTGkraUZCVjBwNkJvbjVLZVVEZTdoMXlTVDhqQkVQVis4TVlFWFViZnlv?=
 =?utf-8?B?S2xRa1BTbjMyOVN1cHBjeWp4VWlmRlc3WjZISW5tMTRENmpFbDZvQ3hLSDkz?=
 =?utf-8?B?d0JYR3J4U09CTTZlWkw2bUJSRWZ6eHB4clFRYTFCY3dmVjhobFM0enFydEFs?=
 =?utf-8?B?WENqdi9XOEc5d3hFZTlZRXJaekx0V2I2Y1NnZC9ITnl0ZUlBOGhLT0t1cFlp?=
 =?utf-8?B?Q2Q1d1lxK2hzUkIraERUbGRvUS9sV21GaVA3U0N2TU5Qd0lHeFBmclJCdnYx?=
 =?utf-8?B?SW02UkNrelRKa0k2L2FVcENRYTdvV1lHUUk4UGhkN1VySHhhNTRqTUFWVXNK?=
 =?utf-8?B?K1VTSFdrR1JsT3V6SHE2NFVZWklrSFRiMHpLUlh0MmlCM3ZHQ3Z1K3BtNDVS?=
 =?utf-8?B?c3R0OXQ1SFFzclVmQ1RDOXlsa0tDLzhrUkVZTWFCVlJxcGRIRWN6QmhlTFZS?=
 =?utf-8?B?c2dZbEVjNWtnV09xS1NmZCtOVnJqd0EydlpNalNCSTdPS2l5ZlhmL0ZTMFZo?=
 =?utf-8?B?K21PZTVaUGhudXAxK0ZoUTFIbDJrU3d4OEFlbUptekpPTnB5SHBZcWowQytD?=
 =?utf-8?B?U1kvUzNYZGErQjNSb0lJVlRzeUtkNGY1dlQ1M3hVQUgwQjJvRmQycENseU8x?=
 =?utf-8?B?OEhJemx6TGRoVE81eGZYbXNFMGVETjc3ejJ3RlA2bXlqWHJXRGkrZzhHVFpo?=
 =?utf-8?B?QmxITFR2MEtpdWJZYnNiUnQ5TjBjVjRIekhwQ1NUSStabkhJb1ZmdTN1b0Rv?=
 =?utf-8?B?TU5uSEVReXpPVU1zUWhUWm1mUFZzYitHbTFYVE1jRVdOVzJQNm0rYndwang5?=
 =?utf-8?B?TEdicG5xams5VjJWc3hBd1dhb0Q3WDdISTVRUG1FQUNnUmtuU0pjUDZNVkZJ?=
 =?utf-8?B?Y20xZTIyYUcrUnNIWHN0REp3L1dOU3ZmcjJvcS9JV0Fld0RkTzVUQlh3VHgw?=
 =?utf-8?B?V1MwSDB1RlV2MjR6K3dHQ05wOXcwUklIRnR1SDUyek4xcXRIcHU2Vm9wQ2N6?=
 =?utf-8?B?dkFrVUpCWnc3NEZaVE9jb0Zici9MbU5jRXVvYjljOGpWVUpuRHNCTnJoaHpM?=
 =?utf-8?B?QmtoaXdnRlQ3aE92Z3JzZU5HSHJiUjNhUlpWaVMvaThOM2lxT09wWkFhQU5O?=
 =?utf-8?B?Q3dwcExqR1BXVDNFODhmVmV1NUIweEpTdDZRV29ZYWV6ajBlQnFJUzY3OWwv?=
 =?utf-8?B?ZDFIa1dDa3M0d0tkWldoU2JJZWFrVW5UYytybTVhRkhUZlNycE5td0lPV1lC?=
 =?utf-8?B?YjlaSC9QcmdBa3BFWG9hbXJIMDV5TUNSQ3VKK29XSFIzWXh0SkpieExHdGgz?=
 =?utf-8?B?ckdTaHFwMUFvaUFVZ2RHMEZMTlRMOUZOSWhHNCtxcUFiM2JCNGRONmQ2MU9S?=
 =?utf-8?B?b2cxQ0F2UHc1dnNtdG5vUGRkdW1jL0ZZaUVjMjl3dkxIOXpIQkdTUkZUdWoz?=
 =?utf-8?B?MkNHRjc2MTE4Zkx6eWVUcURtMEtpV3phc05ZdnJFb3NtYkFIZDRvNGtPTHBz?=
 =?utf-8?B?bkg1dzRiT0RJQUlEUEc5NHRVdXBIRE9vbHY1MkZjcllkQWFRaVNCZ2FzR2Nn?=
 =?utf-8?B?RlhMejQ2dGhSdVlIKzNjSUYxbldWSjdGZktCRjUzWjZoQTAyYk5hVTNPZ01m?=
 =?utf-8?B?ZlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2da064db-fb74-4e5b-1cc2-08dc0cf46ec9
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jan 2024 07:11:55.3476
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LCx/ITAEKisdcFZ7ysLdzD9VvXcW+KL5TzDn9BBuDXZ7+htaCI8dWcqIFWC1vNcXA+hhVHlv5JoMVzZEj8I4uA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6193
X-OriginatorOrg: intel.com

On 1/4/2024 2:50 AM, Edgecombe, Rick P wrote:
> On Thu, 2023-12-21 at 09:02 -0500, Yang Weijiang wrote:
>> Control-flow Enforcement Technology (CET) is a kind of CPU feature
>> used
>> to prevent Return/CALL/Jump-Oriented Programming (ROP/COP/JOP)
>> attacks.
>> It provides two sub-features(SHSTK,IBT) to defend against ROP/COP/JOP
>> style control-flow subversion attacks.
>>
>> Shadow Stack (SHSTK):
>>    A shadow stack is a second stack used exclusively for control
>> transfer
>>    operations. The shadow stack is separate from the data/normal stack
>> and
>>    can be enabled individually in user and kernel mode. When shadow
>> stack
>>    is enabled, CALL pushes the return address on both the data and
>> shadow
>>    stack. RET pops the return address from both stacks and compares
>> them.
>>    If the return addresses from the two stacks do not match, the
>> processor
>>    generates a #CP.
>>
>> Indirect Branch Tracking (IBT):
>>    IBT introduces new instruction(ENDBRANCH)to mark valid target
>> addresses of
>>    indirect branches (CALL, JMP etc...). If an indirect branch is
>> executed
>>    and the next instruction is _not_ an ENDBRANCH, the processor
>> generates a
>>    #CP. These instruction behaves as a NOP on platforms that doesn't
>> support
>>    CET.
> What is the design around CET and the KVM emulator?

KVM doesn't emulate CET HW behavior for guest CET, instead it leaves CET related
checks and handling in guest kernel. E.g., if emulated JMP/CALL in emulator triggers
mismatch of data stack and shadow stack contents, #CP is generated in non-root
mode instead of being injected by KVM.  KVM only emulates basic x86 HW behaviors,
e.g., call/jmp/ret/in/out etc.

> My understanding is that the KVM emulator kind of does what it has to
> keep things running, and isn't expected to emulate every possible
> instruction. With CET though, it is changing the behavior of existing
> supported instructions. I could imagine a guest could skip over CET
> enforcement by causing an MMIO exit and racing to overwrite the exit-
> causing instruction from a different vcpu to be an indirect CALL/RET,
> etc.

Can you elaborate the case? I cannot figure out how it works.

> With reasonable assumptions around the threat model in use by the
> guest this is probably not a huge problem. And I guess also reasonable
> assumptions about functional expectations, as a misshandled CALL or RET
> by the emulator would corrupt the shadow stack.

KVM emulates general x86 HW behaviors, if something wrong happens after emulation
then it can happen even on bare metal, i.e., guest SW most likely gets wrong somewhere
and it's expected to trigger CET exceptions in guest kernel.

> But, another thing to do could be to just return X86EMUL_UNHANDLEABLE
> or X86EMUL_RETRY_INSTR when CET is active and RET or CALL are emulated.

IMHO, translating the CET induced exceptions into X86EMUL_UNHANDLEABLE or X86EMUL_RETRY_INSTR would confuse guest kernel or even VMM, I prefer letting guest kernel handle #CP directly.
> And I guess also for all instructions if the TRACKER bit is set. It
> might tie up that loose end without too much trouble.
>
> Anyway, was there a conscious decision to just punt on CET enforcement
> in the emulator?

I don't remember we ever discussed it in community, but since KVM maintainers reviewed
the CET virtualization series for a long time, I assume we're moving on the right way :-)