Received-SPF: pass (google.com: domain of linux-kernel+bounces-173301-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Julian Stecklina <julian.stecklina@cyberus-technology.de>
To: Paolo Bonzini <pbonzini@redhat.com>,
	Jonathan Corbet <corbet@lwn.net>,
	Sean Christopherson <seanjc@google.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>,
	Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	x86@kernel.org,
	"H. Peter Anvin" <hpa@zytor.com>
Cc: Thomas Prescher <thomas.prescher@cyberus-technology.de>,
	Julian Stecklina <julian.stecklina@cyberus-technology.de>,
	kvm@vger.kernel.org,
	linux-doc@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: [PATCH] KVM: x86: add KVM_RUN_X86_GUEST_MODE kvm_run flag
Date: Wed,  8 May 2024 15:25:01 +0200
Message-ID: <20240508132502.184428-1-julian.stecklina@cyberus-technology.de>
Reply-To: <ZiFmNmlUxfQnXIua@google.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?ZLwhE37KXhY0JdZaSdAD7qQJ0XklPZrMkamMJRgKOahDe+aowE9y5VD0dtrj?=
 =?us-ascii?Q?fCDFVOCiHBVaDGdIMabBOMfbaQF1JkkvD/0gicNhagMs8qYPw9RhRYWaqvDZ?=
 =?us-ascii?Q?NjpsbIkvG5P9D6iRbkbF7Jyznqy7gRZCLJB273aSsiQKAhFWuWXWXJgx8Cet?=
 =?us-ascii?Q?hDlsHkIzYFvUWEN7mliRvXIU3HQLKB7m4OZ8Y4UIazWaUwyj/HfEhoUZnyas?=
 =?us-ascii?Q?k5FIDYpG2fGG2jTKXz++h0XWV+vwUt6m2ZZv+HUk3NqgszqXFBrXmE4583xu?=
 =?us-ascii?Q?j9Y7bRMHkj5TIy9JRAR97fbcgy6FvqTIfgcuUAk7JA30QzAMmnG+0sODABeU?=
 =?us-ascii?Q?RYd2czIzsd/EBbsRWfbneIM40RSNSKzZYEgyyxB2K8SYuie3DB//LzT2u71K?=
 =?us-ascii?Q?I/tlhu0UfJBW3o515+/0VVx62pwfvYHq6gVAZzl2HRAa6RMklnjsEsCD5jyn?=
 =?us-ascii?Q?c+/gfMw6kMljZOEI2FykSjg1T5I+fg9bbQZoS1ANzDnm8zld7UVCbD4FZg1J?=
 =?us-ascii?Q?jmojlEThOhg3XR6rPyqU0/g9+yKnFvITFzuEYQ8ACyQm3JHHr0NpHRV/qrIj?=
 =?us-ascii?Q?/1p45c4TaTA2AXURRdOv89LwElnpIZylUWdnSJ5gcgmeWznLWRo1qZWeDHul?=
 =?us-ascii?Q?zEKtdowM36fI3WV/QaUKO3lW9qWf8+Aam63u9gdlQL+/yAh2OyasTUxF62mj?=
 =?us-ascii?Q?HEVGYNfxcS3OehV6hoGcYSeCm6ttWYvH7eEff6JGOHJoJxX/8K/3kGLuLPGz?=
 =?us-ascii?Q?A8bLvIZctkKBTjiGrMMX4OSLa9HCN/DD8mvokBKYT1GNhzQ6MH4C6sPTIGT0?=
 =?us-ascii?Q?ZrJPLA2n/tbIsfTsk7YqDDONaxkl5JOks1/Itdbxce3aSQQiVlcKRLFWcLPH?=
 =?us-ascii?Q?Wf3QTzt8DpcpC5AONCCGDzB87AsXWYRG6TFfWbufLOcX1qS4+zOzyajLARW2?=
 =?us-ascii?Q?RXu+AIYEvvCyL0u1lq0UfLsue9dvkOTEOtSTvxTWmPSxPoz1FHJRchZPH/rY?=
 =?us-ascii?Q?1DCKGQgDnxAu4qvGStVz+Yakf+czxQQ4FgZ1hk8EgmOGGWIA6f2XMUMpsiGE?=
 =?us-ascii?Q?6oASgbD/ENMzqJ6wfznUSK5/0P1+JxDYkpE55BdJ8uICVGXTq+3xb29akPE/?=
 =?us-ascii?Q?5MDDeCMUksMwvNbaFM+2NGEI2BU7Gxn6Lm64zDW9DMWMZwAJ15xewwFbbu2K?=
 =?us-ascii?Q?m2jfllbMZlCANUWajP88kKC1ZSqdDWGtUf+Zdk9FwmrDlZh2XmcxPDA/9o9u?=
 =?us-ascii?Q?kBGsZ6lT4npCgBdGeQd+rMXWB2dckaa3ruFFbTAtEqLDpvqm+w2uE/MtnJrc?=
 =?us-ascii?Q?Mr/r0xIn9oO772X/EWT9+3f1dMrrQdp+qY6K+ha7mnWv90t+1ZmAhWu4U0+G?=
 =?us-ascii?Q?VJ58aAvKUFGE7KeXuHmisYkIRxYjMh20lukGybxwUBAp5qN0zWg6XyDiPhNK?=
 =?us-ascii?Q?OY8DE1WMW+6NROYkGo0Gy4t5xgjO36W4O9OnuPtfsJAoKjswlvApwzBc2Zuo?=
 =?us-ascii?Q?T76Vzn900TX7c8k7S1t+vqyqIiOtcCRi2+NfkQvgK3o10C0HGfumGqKI0blu?=
 =?us-ascii?Q?Zhsxz5EZGdpO/YyrLuBhGzDr8oloLEn2Eptj6z6xYGi5PgekifYS7Hn3cOa4?=
 =?us-ascii?Q?1JGGvhDcSKkVU3OIPBWOHsM=3D?=
X-OriginatorOrg: cyberus-technology.de
X-MS-Exchange-CrossTenant-Network-Message-Id: 07dcb821-65b5-449e-fece-08dc6f625338
X-MS-Exchange-CrossTenant-AuthSource: FR2P281MB2329.DEUP281.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2024 13:25:27.5434
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: f4e0f4e0-9d68-4bd6-a95b-0cba36dbac2e
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: dPG8q+gQyhlecBho6M4v1baOTfQHFa7tyr5Fncq1Aln5I9Wv3CAUvCxaG9u5cWxjOUWfEgi4Uz5yOa61zOoTxLEmO4I2SgbZ42U+Ap28HxDt6J4gxPha5mhx+diDdPPv
X-MS-Exchange-Transport-CrossTenantHeadersStamped: FR5P281MB4522

From: Thomas Prescher <thomas.prescher@cyberus-technology.de>

When a vCPU is interrupted by a signal while running a nested guest,
KVM will exit to userspace with L2 state. However, userspace has no
way to know whether it sees L1 or L2 state (besides calling
KVM_GET_STATS_FD, which does not have a stable ABI).

This causes multiple problems:

The simplest one is L2 state corruption when userspace marks the sregs
as dirty. See this mailing list thread [1] for a complete discussion.

Another problem is that if userspace decides to continue by emulating
instructions, it will unknowingly emulate with L2 state as if L1
doesn't exist, which can be considered a weird guest escape.

This patch introduces a new flag KVM_RUN_X86_GUEST_MODE in the kvm_run
data structure, which is set when the vCPU exited while running a
nested guest. Userspace can then handle this situation.

To see whether this functionality is available, this patch also
introduces a new capability KVM_CAP_X86_GUEST_MODE.

[1] https://lore.kernel.org/kvm/20240416123558.212040-1-julian.stecklina@cyberus-technology.de/T/#m280aadcb2e10ae02c191a7dc4ed4b711a74b1f55

Signed-off-by: Thomas Prescher <thomas.prescher@cyberus-technology.de>
Signed-off-by: Julian Stecklina <julian.stecklina@cyberus-technology.de>
---
 Documentation/virt/kvm/api.rst  | 17 +++++++++++++++++
 arch/x86/include/uapi/asm/kvm.h |  1 +
 arch/x86/kvm/x86.c              |  3 +++
 include/uapi/linux/kvm.h        |  1 +
 4 files changed, 22 insertions(+)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 0b5a33ee71ee..7748c3eb98e0 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6419,6 +6419,9 @@ affect the device's behavior. Current defined flags::
   #define KVM_RUN_X86_SMM     (1 << 0)
   /* x86, set if bus lock detected in VM */
   #define KVM_RUN_BUS_LOCK    (1 << 1)
+  /* x86, set if the VCPU exited from a nested (L2) guest */
+  #define KVM_RUN_X86_GUEST_MODE (1 << 2)
+
   /* arm64, set for KVM_EXIT_DEBUG */
   #define KVM_DEBUG_ARCH_HSR_HIGH_VALID  (1 << 0)
 
@@ -8063,6 +8066,20 @@ error/annotated fault.
 
 See KVM_EXIT_MEMORY_FAULT for more information.
 
+7.34 KVM_CAP_X86_GUEST_MODE
+------------------------------
+
+:Architectures: x86
+:Returns: Informational only, -EINVAL on direct KVM_ENABLE_CAP.
+
+The presence of this capability indicates that KVM_RUN will update the
+KVM_RUN_X86_GUEST_MODE bit in kvm_run.flags to indicate whether the
+vCPU was executing nested guest code when it exited.
+
+KVM exits with the register state of either the L1 or L2 guest
+depending on which executed at the time of an exit. Userspace must
+take care to differentiate between these cases.
+
 8. Other capabilities.
 ======================
 
diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h
index ef11aa4cab42..ff4ed82a2d06 100644
--- a/arch/x86/include/uapi/asm/kvm.h
+++ b/arch/x86/include/uapi/asm/kvm.h
@@ -106,6 +106,7 @@ struct kvm_ioapic_state {
 
 #define KVM_RUN_X86_SMM		 (1 << 0)
 #define KVM_RUN_X86_BUS_LOCK     (1 << 1)
+#define KVM_RUN_X86_GUEST_MODE   (1 << 2)
 
 /* for KVM_GET_REGS and KVM_SET_REGS */
 struct kvm_regs {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 91478b769af0..64f2cba9345e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4714,6 +4714,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_VM_DISABLE_NX_HUGE_PAGES:
 	case KVM_CAP_IRQFD_RESAMPLE:
 	case KVM_CAP_MEMORY_FAULT_INFO:
+	case KVM_CAP_X86_GUEST_MODE:
 		r = 1;
 		break;
 	case KVM_CAP_EXIT_HYPERCALL:
@@ -10200,6 +10201,8 @@ static void post_kvm_run_save(struct kvm_vcpu *vcpu)
 
 	if (is_smm(vcpu))
 		kvm_run->flags |= KVM_RUN_X86_SMM;
+	if (is_guest_mode(vcpu))
+		kvm_run->flags |= KVM_RUN_X86_GUEST_MODE;
 }
 
 static void update_cr8_intercept(struct kvm_vcpu *vcpu)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 2190adbe3002..ccb12f6a656d 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -917,6 +917,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_MEMORY_ATTRIBUTES 233
 #define KVM_CAP_GUEST_MEMFD 234
 #define KVM_CAP_VM_TYPES 235
+#define KVM_CAP_X86_GUEST_MODE 236
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
-- 
2.42.0