Received-SPF: pass (google.com: domain of linux-kernel+bounces-57689-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <f0e37b73-1d8d-4717-9dcd-2596384d0d19@intel.com>
Date: Thu, 8 Feb 2024 10:04:52 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2] KVM: x86: nSVM/nVMX: Fix handling triple fault on RSM
 instruction
To: <seanjc@google.com>, <pbonzini@redhat.com>, <mlevitsk@redhat.com>
CC: <tglx@linutronix.de>, <mingo@redhat.com>, <bp@alien8.de>,
	<dave.hansen@linux.intel.com>, <x86@kernel.org>, <hpa@zytor.com>,
	<kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <dedekind1@gmail.com>,
	<yuan.yao@intel.com>, Zheyu Ma <zheyuma97@gmail.com>
References: <20240123001555.4168188-1-michal.wilczynski@intel.com>
Content-Language: en-US
From: "Wilczynski, Michal" <michal.wilczynski@intel.com>
In-Reply-To: <20240123001555.4168188-1-michal.wilczynski@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dkIxSFg2aWUxZHVaemRjRGhoME14TWgwdkJzMGdCS2doVlNsaExFcjJhWW5k?=
 =?utf-8?B?cG1FUm84ejFrSG1ZbGlrd0E0UWV3UHVtaWNBa1pDZnIxcmw2RXlpa1BzQ094?=
 =?utf-8?B?VDg1ZUUvNTRIK0tZOW5kR1FjN0JFSVJmT2NpNDkvb0lHazQ1d2lXb2RyOWg3?=
 =?utf-8?B?YzhtSHQ4RVpvNXpWVEhSQW40WFdHMGlnTVBOOGpjcmF3eG1yZEtrMDZ5Zy9n?=
 =?utf-8?B?dVVnbzJJQmhERjRuYTU1Y1BtRWdjaGdSQmxRWlZ2a2RZNUtpVml6RkI4Rm02?=
 =?utf-8?B?WmhDNFkrNC9idHhSOHAwNXk3V3ZHS2pibG4yZFQzTGd5dTljNXlYR3dTVUx1?=
 =?utf-8?B?L3BXVTlRdThhSVNodTZmb3pDdGRPeHF6bnZkdWl6QVdhWkUvbzZiSlVFZjNr?=
 =?utf-8?B?V0t4WVFsRUJvalYwQWNZd1lTRGpVODA2ZHF5aTVmZURhSTRFTzRYWHF2WWI5?=
 =?utf-8?B?dlo0b3BsU0wzb3liOExraFREbktHdmNZVFg5TWV4SmRMN2taZnFFTyswUkRn?=
 =?utf-8?B?b3RqTlZqTnFWK1d3eFdVV2tML2liYkVKNmxUblVacm5pNmlqZ1grUDBtT051?=
 =?utf-8?B?d1JKdTI0MEJYRXNXckxNcExVQlRyNHErUVRrNXB2ZC85UzZHZ2xEOGl6Q3JX?=
 =?utf-8?B?RnBHOXliTjdRV2hSc2tLVVhVc01vdHQ0RjBjd0JJS0pseFV4SncrTVNNTkR1?=
 =?utf-8?B?eUVEQjQxcHMrbXlCelY1aVJEZTRRTUJIT0JiVWZlbEsxTTBoSzFGRG55UFht?=
 =?utf-8?B?MDdJN1pQcnpoa0QyMXVlYTcraXJrVjBJQnhDR3BZRUtMYjd3VjlhVjNMT2N4?=
 =?utf-8?B?MkxVRWpvRFd2VWRmRFhSOEgzdjhsS084YlI3NVBPOFpUcTVhZXBrVFRpQkVk?=
 =?utf-8?B?MHBQN2UwQk9xb3dxL0N5bTdOQkRSdlBuc3l1TEZtd2cySDZmQ0w1TDVVOHBo?=
 =?utf-8?B?alN4MXZDM00wYW5BbjhWYkVQUVlxUnNObGVrYkhqK1U4dGJzbHhlQXdhZXFy?=
 =?utf-8?B?dUxTdHUyUG1SRzE5OTE4dURoeERWVnlXWlRKTGxjSVhNUXRCS2p2SkdpMUx2?=
 =?utf-8?B?T1pXVS9ZNkxqTWszT0hXaitidThiOW9CUHhmQngwc3RNTDdJenQxMFpDQWhR?=
 =?utf-8?B?ZzJGT0hiMDRtcVh3bzdDUkdKd1JabDloKzI2dTFyOW1FempzWmthREpVUHhj?=
 =?utf-8?B?QVJhQzRkREdFVU1xb1FRWUFPcm5KWEJ5Q2Y5dDVFUjhyc2xZZGxncnhLUjE4?=
 =?utf-8?B?UTg0QW1vaFlSKzhRU28vbEZqMVI1azdXaXZIQnZtdmlRWnJ1emdaanM1ajBY?=
 =?utf-8?B?dlJuMU5OS0t6blBIYWVUMDJjenVEdE9WVmR4aXZFOUdwYVQyQXFOcDJaSy94?=
 =?utf-8?B?ajhNV0xJODYzOW81dHVDT282WGloNHBHOFIvUEo1SDZIeGxsVkhTNUhkRG5y?=
 =?utf-8?B?NkhBc2lTY2wrWnh4OGYzMzFDSnpzcGg2RWxEMWF3Sjh3bGNYRHJNMTFUVTZa?=
 =?utf-8?B?Z0sreG1jSEI5MlA5dmpOVmF1amtnRHlvK3h4cFJyZ1YwSGVZTnBySWxTUEdJ?=
 =?utf-8?B?ZWdUekgvMkJkZ0grMmgwczNpL1BCU2JVa254SExtbWJqQ0t0S1Y3Q3dlWWpT?=
 =?utf-8?B?R1BpalJidTh3bDNTaXlhZDIxQ2xZSS95ZzRhRWNudG5WeWdQSGViOWNjZ0N5?=
 =?utf-8?B?QmQyc003c2dnd2VtTVlrV3A0dVhKV3hmZTlDOHNOQ1pKWXZWQmZiM2MxYXJ5?=
 =?utf-8?B?UnNuTHMvTXEyYjlPbERWQlgvSjFDb3gyQzluc3Y2OElSYkYrdHJ0Um9yS2Ir?=
 =?utf-8?B?U3BRcnJ2dWZUbFdtTXlhbFhvQm1wZlBrQXlMMmFIdWVzbWI0U0hDTDJBWjVi?=
 =?utf-8?B?LzlmaUxWQ285WHExMTIyNDlVeW1jYkN3V1dqS04wVjIyMEJ3clVrTGJPRTlT?=
 =?utf-8?B?eUxGZFlocmp0SndUVllldkRYbFNCWmFSMU5JdFBBbWRvUW42Y1RLYXNqeHRF?=
 =?utf-8?B?cUVlcjd3aXRIb3A1Y1RRMWVTenVacHpiWFlweXd4OHdOZkJmZGZpV3BMM0dP?=
 =?utf-8?B?c2tsZlRqazRteGxyQzB6Q1FBZ1RPNE5sZ05zTDc2U0ZXOVBGZTk2aVVjeExy?=
 =?utf-8?B?WFhHQTBPOEd3RW80WWZiTnplazgvZmdvazk0WUhTdnNPZENiZnBtcGhSdmpu?=
 =?utf-8?B?N3c9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 30baf4e0-e35c-45c0-6423-08dc2885069e
X-MS-Exchange-CrossTenant-AuthSource: CO6PR11MB5603.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2024 09:04:58.9033
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FnBpkWtBC1M5nQmXdtGbOua5QvihV9N+dSvyF50CswRCOgttwmkQYjRZJlhfU2LYgey7+x+QhTblnjRg/+XoVkrX09Lv0w+ITxEEeA+5pxw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB8474
X-OriginatorOrg: intel.com


On 1/23/2024 1:15 AM, Michal Wilczynski wrote:
> Syzkaller found a warning triggered in nested_vmx_vmexit().
> vmx->nested.nested_run_pending is non-zero, even though we're in
> nested_vmx_vmexit(). Generally, trying  to cancel a pending entry is
> considered a bug. However in this particular scenario, the kernel
> behavior seems correct.
> 
> Syzkaller scenario:
> 1) Set up VCPU's
> 2) Run some code with KVM_RUN in L2 as a nested guest
> 3) Return from KVM_RUN
> 4) Inject KVM_SMI into the VCPU
> 5) Change the EFER register with KVM_SET_SREGS to value 0x2501
> 6) Run some code on the VCPU using KVM_RUN
> 7) Observe following behavior:
> 
> kvm_smm_transition: vcpu 0: entering SMM, smbase 0x30000
> kvm_entry: vcpu 0, rip 0x8000
> kvm_entry: vcpu 0, rip 0x8000
> kvm_entry: vcpu 0, rip 0x8002
> kvm_smm_transition: vcpu 0: leaving SMM, smbase 0x30000
> kvm_nested_vmenter: rip: 0x0000000000008002 vmcs: 0x0000000000007000
>                     nested_rip: 0x0000000000000000 int_ctl: 0x00000000
> 		    event_inj: 0x00000000 nested_ept=n guest
> 		    cr3: 0x0000000000002000
> kvm_nested_vmexit_inject: reason: TRIPLE_FAULT ext_inf1: 0x0000000000000000
>                           ext_inf2: 0x0000000000000000 ext_int: 0x00000000
> 			  ext_int_err: 0x00000000
> 
> What happened here is an SMI was injected immediately and the handler was
> called at address 0x8000; all is good. Later, an RSM instruction is
> executed in an emulator to return to the nested VM. em_rsm() is called,
> which leads to emulator_leave_smm(). A part of this function calls VMX/SVM
> callback, in this case vmx_leave_smm(). It attempts to set up a pending
> reentry to guest VM by calling nested_vmx_enter_non_root_mode() and sets
> vmx->nested.nested_run_pending to one. Unfortunately, later in
> emulator_leave_smm(), rsm_load_state_64() fails to write invalid EFER to
> the MSR. This results in em_rsm() calling triple_fault callback. At this
> point it's clear that the KVM should call the vmexit, but
> vmx->nested.nested_run_pending is left set to 1.
> 
> Similar flow goes for SVM, as the bug also reproduces on AMD platforms.
> 
> To address this issue, reset the nested_run_pending flag in the
> triple_fault handler. However, it's crucial to note that
> nested_pending_run cannot be cleared in all cases. It should only be
> cleared for the specific instruction requiring hardware VM-Enter to
> complete the emulation, such as RSM. Previously, there were instances
> where KVM prematurely synthesized a triple fault on nested VM-Enter. In
> these cases, it is not appropriate to zero the nested_pending_run.
> 
> To resolve this, introduce a new emulator flag indicating the need for
> HW VM-Enter to complete emulating RSM. Based on this flag, a decision can
> be made in vendor-specific triple fault handlers about whether
> nested_pending_run needs to be cleared.
> 
> Fixes: 759cbd59674a ("KVM: x86: nSVM/nVMX: set nested_run_pending on VM entry which is a result of RSM")
> Reported-by: Zheyu Ma <zheyuma97@gmail.com>
> Closes: https://lore.kernel.org/all/CAMhUBjmXMYsEoVYw_M8hSZjBMHh24i88QYm-RY6HDta5YZ7Wgw@mail.gmail.com
> Signed-off-by: Michal Wilczynski <michal.wilczynski@intel.com>
> ---
> v2:
>  - added new emulator flags indicating whether an instruction needs a
>    VM-Enter to complete emulation (Sean)
>  - fix in SVM nested triple_fault handler (Sean)
>  - only clear nested_run_pending on RSM instruction (Sean)
> 
>  arch/x86/kvm/emulate.c     |  4 +++-
>  arch/x86/kvm/kvm_emulate.h |  2 ++
>  arch/x86/kvm/svm/nested.c  |  9 +++++++++
>  arch/x86/kvm/vmx/nested.c  | 12 ++++++++++++
>  4 files changed, 26 insertions(+), 1 deletion(-)
> 

Hi Sean,
Friendly ping,

Regards,
Michał