Received: by 2002:a05:7412:e794:b0:fa:551:50a7 with SMTP id o20csp1507907rdd; Thu, 11 Jan 2024 00:58:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IGH3S6k3qUW6JlZ+Wc0/Pz1cWXQ10JLluCe/WAfYvyx5IbLBVkQ9nf2ax1AVQew2vDP2yiC X-Received: by 2002:ad4:5aa7:0:b0:681:24f4:a631 with SMTP id u7-20020ad45aa7000000b0068124f4a631mr846635qvg.8.1704963523493; Thu, 11 Jan 2024 00:58:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704963523; cv=none; d=google.com; s=arc-20160816; b=JO9PUNSwo71Xl2maUO2hznxoZXhqdxlg+/Q03LjeCXU1wr73RsX968QrqkkXPEM1sA 7wtUdi3hya7nHYlgwsZlBX+lsNQjeWRu7/RaDOcLuSQBDfVKrRPOiNPXS6zN9f6NADfc q0Uv9vX/a5FK9+rNOzNhpDqM+HOWqiDAWcdBZIHr86jbdL5d9cnWu5a56AlzqKoHEPku dZph4EeTAkJSr8LnzD1otKaneFoOq9k1lGdVOJJYxhHTszvMvFoES53jWx2OveJ4Z0t0 n81zz3YRfs3JAyvjab52LCYOP7iVqhKus+LJODpPxJE8S6Svue3Qi43X5TBF4TAf2sRm qaxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=AAWYqQo49rSUP/liphSBHP0is1wvO5riamGNWMX1lTw=; fh=mM7j0AJ/S24jynU3+f8ZuQxBzhzn0ovRqHTZp+moQSw=; b=SH7lwhFus9dRQtWTLINbZOrcBLiiMITOwWUEfreQizgZsjZn67Fj8Ny0mPanatH0SR Yfg5xVgQ84zDAv7bp2R9tZw1nfsqTm7V6jgyI+ncen15XOMpJrX8eBxyMTEb0+Fkvoad F59a+457WmHq/M7JcBL5RArzOWVaTSKvXYtSS0CdrboIAc+PgUk+gNAGZHTMfSqR7aPG SUNGFZF+FpKTDfcIHpM8hdy7Hi/QaBJZJt79i8voLb+48AidoTLzcW0vv/Dq4DkscfAs dNmjdWs6NmGO9W89ZmxGutiHaC5fmRZwwSP6y5rd5KCj9dGgpegTDsxQhrs6NLEASd6y 1qYA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Bn0OsbXO; spf=pass (google.com: domain of linux-kernel+bounces-23261-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23261-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id y18-20020a0cf152000000b0067a360ada6esi387620qvl.255.2024.01.11.00.58.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jan 2024 00:58:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-23261-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Bn0OsbXO; spf=pass (google.com: domain of linux-kernel+bounces-23261-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23261-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 44A741C213C7 for ; Thu, 11 Jan 2024 08:58:43 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 435E2156DE; Thu, 11 Jan 2024 08:56:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Bn0OsbXO" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA456154BF; Thu, 11 Jan 2024 08:56:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1704963387; x=1736499387; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=/z2uYKhSyYcTU8/yd1Vq7zQ21LOb4cUygjDFi2E+u98=; b=Bn0OsbXOuHcmjyxhW4ej8etCRCRyt9chbCRy9agDpxsKaXyftP2GRFnW lPjPn00CMKTKqWh6RN4o2ChGYm6nAKQmug8E/qHeeEWqIWT1mh9YffXgd RuoYPSggzSTg1Yu7CVNlqIG8uGMCnE7sZlJttoG1nhj2XPeJwyOTzqxB0 Haa+3PvmXZRlvJL986Hv6OgqBa+KMSzy+emyJhjt3zKLHtfYsh3aKA8ks GFEG9M/MgMTWUA1jY1uFjT/QSczUVEcomPqtP17LDbAcXQhlfsp84zGtt VxP+pji+CbDsR7+GkfzNuDxTk3O7tk9TlKQa2eGDjTFUU49cn6+gFIJkF g==; X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="12141738" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="12141738" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:26 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="30929545" Received: from ericwong-mobl2.amr.corp.intel.com (HELO desk) ([10.209.43.169]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:24 -0800 Date: Thu, 11 Jan 2024 00:56:24 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Alyssa Milburn Subject: [PATCH v5 1/6] x86/bugs: Add asm helpers for executing VERW Message-ID: <20240111-delay-verw-v5-1-a3b234933ea6@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> MDS mitigation requires clearing the CPU buffers before returning to user. This needs to be done late in the exit-to-user path. Current location of VERW leaves a possibility of kernel data ending up in CPU buffers for memory accesses done after VERW such as: 1. Kernel data accessed by an NMI between VERW and return-to-user can remain in CPU buffers since NMI returning to kernel does not execute VERW to clear CPU buffers. 2. Alyssa reported that after VERW is executed, CONFIG_GCC_PLUGIN_STACKLEAK=y scrubs the stack used by a system call. Memory accesses during stack scrubbing can move kernel stack contents into CPU buffers. 3. When caller saved registers are restored after a return from function executing VERW, the kernel stack accesses can remain in CPU buffers(since they occur after VERW). To fix this VERW needs to be moved very late in exit-to-user path. In preparation for moving VERW to entry/exit asm code, create macros that can be used in asm. Also make VERW patching depend on a new feature flag X86_FEATURE_CLEAR_CPU_BUF. Reported-by: Alyssa Milburn Suggested-by: Andrew Cooper Suggested-by: Peter Zijlstra Signed-off-by: Pawan Gupta --- arch/x86/entry/entry.S | 22 ++++++++++++++++++++++ arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/include/asm/nospec-branch.h | 15 +++++++++++++++ 3 files changed, 38 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S index 8c8d38f0cb1d..bd8e77c5a375 100644 --- a/arch/x86/entry/entry.S +++ b/arch/x86/entry/entry.S @@ -6,6 +6,9 @@ #include #include #include +#include +#include +#include .pushsection .noinstr.text, "ax" @@ -20,3 +23,22 @@ SYM_FUNC_END(entry_ibpb) EXPORT_SYMBOL_GPL(entry_ibpb); .popsection + +/* + * Defines the VERW operand that is disguised as entry code so that + * it can be referenced with KPTI enabled. This ensures VERW can be + * used late in exit-to-user path after page tables are switched. + */ +.pushsection .entry.text, "ax" + +.align L1_CACHE_BYTES, 0xcc +SYM_CODE_START_NOALIGN(mds_verw_sel) + UNWIND_HINT_UNDEFINED + ANNOTATE_NOENDBR + .word __KERNEL_DS +.align L1_CACHE_BYTES, 0xcc +SYM_CODE_END(mds_verw_sel); +/* For KVM */ +EXPORT_SYMBOL_GPL(mds_verw_sel); + +.popsection diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 4af140cf5719..79a7e81b9458 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -308,10 +308,10 @@ #define X86_FEATURE_SMBA (11*32+21) /* "" Slow Memory Bandwidth Allocation */ #define X86_FEATURE_BMEC (11*32+22) /* "" Bandwidth Monitoring Event Configuration */ #define X86_FEATURE_USER_SHSTK (11*32+23) /* Shadow stack support for user mode applications */ - #define X86_FEATURE_SRSO (11*32+24) /* "" AMD BTB untrain RETs */ #define X86_FEATURE_SRSO_ALIAS (11*32+25) /* "" AMD BTB untrain RETs through aliasing */ #define X86_FEATURE_IBPB_ON_VMEXIT (11*32+26) /* "" Issue an IBPB only on VMEXIT */ +#define X86_FEATURE_CLEAR_CPU_BUF (11*32+27) /* "" Clear CPU buffers using VERW */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index f93e9b96927a..4ea4c310db52 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -315,6 +315,21 @@ #endif .endm +/* + * Macros to execute VERW instruction that mitigate transient data sampling + * attacks such as MDS. On affected systems a microcode update overloaded VERW + * instruction to also clear the CPU buffers. VERW clobbers CFLAGS.ZF. + * + * Note: Only the memory operand variant of VERW clears the CPU buffers. + */ +.macro EXEC_VERW + verw _ASM_RIP(mds_verw_sel) +.endm + +.macro CLEAR_CPU_BUFFERS + ALTERNATIVE "", __stringify(EXEC_VERW), X86_FEATURE_CLEAR_CPU_BUF +.endm + #else /* __ASSEMBLY__ */ #define ANNOTATE_RETPOLINE_SAFE \ -- 2.34.1