Received: by 2002:a05:6a10:1d13:0:0:0:0 with SMTP id pp19csp2916885pxb; Tue, 24 Aug 2021 10:32:18 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzgh6i0zGMZoh9Za2JQYcR0CGQ8+8gOXHQI6pU33BNM28VNBo/16PsFbSeF+uA28db2JY3n X-Received: by 2002:a92:c64c:: with SMTP id 12mr24717894ill.235.1629826337642; Tue, 24 Aug 2021 10:32:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629826337; cv=none; d=google.com; s=arc-20160816; b=AQJsQlDPraeI2Qj7DSsOjo6cP3t09o4VzDWzFsK3DybHL+5m6QHsYvSCBPqE+tmsxa JbfX9vBlUICGw4ROymBFQmemh6VQQn6LIQb+PaxmBU/ZHV0XamYnIdbOG17MOt/nzdTn 81bnlqZJbNUqmYI/Bz+eeRnj0gPe6H4/YzPuEGDICzgZDqKztsKJ3nqjgpD9gOtwWTVE hvZdhNPXaPaPFN9JO7bMtakNpT75tVFi8O2qh3lD7wO4on/9ue8asZuZMtyrcY6BKEvV b2DwZOarsdqXMM4o06/pA6lw0TqseKUWVhM7loT42PqGxpAmUGiycc4xhm5gvLAdIW5Q V+LQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=L7mVX+ytyB8NZ1UYmUM0H1vyMCDXNIn13Edl2zDtDr0=; b=NRQoRy7efLR/ijfx5oQRXiBU28rhP1CH7Q/wwJ4GtSgC+wPc/xn0wajiV/B/2dQJuq H2sPSEObwhB0I6/fT+bpzAQaiNRqoGmtR57man2BRPPBtZTRjzxbc1rboF2aMCkHLrQJ MWbeuLqDNqIH12ZPv3CBf40KBu1iHYtc1Affl+bcPMYLiNSvjtDmgRvjalmilTvC7hM7 LVR/zCP10OKIXU4INmpAL5+p68p21WjV16aZIIM+Zyb47+4/Pw+Np76JSyeAtFZWE/J2 g4VlTnYjtyHh1ryOrls3T3Eog+3SxPEWuONn2N5X77DwzUvhY0vCwbjOMhvOLDJwcBlX 8gbg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="dV/Wex0z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id p133si16184168jap.27.2021.08.24.10.32.06; Tue, 24 Aug 2021 10:32:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="dV/Wex0z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241741AbhHXRcK (ORCPT + 99 others); Tue, 24 Aug 2021 13:32:10 -0400 Received: from mail.kernel.org ([198.145.29.99]:34268 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241464AbhHXR3U (ORCPT ); Tue, 24 Aug 2021 13:29:20 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id F40CA61B43; Tue, 24 Aug 2021 17:05:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1629824730; bh=36eifsAOxTF4eQnSGTYznbnPJqSlDlev0XVPOLvtT0E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dV/Wex0z2mku+CBYJw9xnjxP5ELaFzgYBfJRf159TT63D97TcDFCf8qfmvdL/gbfN +MteTS9Pf94EQF0XjdWUDG98crk2Mf7kz4rGE1yAkTISHnucWpiY6a5ByQweNSKGaP L4zylc49cwJQIWN5+H32Wn05ai+DJ0ITQag2Wk3zDMv2HEt+yF8I9fSrQRunQ6YaDX 26g1OGUhtcDWj1S12sleziKJHwxH8Ynas7l+9Ts8KwXqk0kcAZz2VRf2lfZbKADkPw rH7bMxnm14i0/6/fQbGtsQkEJqy3Wm7oio7wPinGL6iUDh8057vvImtn6BSHH5upAd u+UD2r6CsimjQ== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Thomas Gleixner , Borislav Petkov , Greg Kroah-Hartman Subject: [PATCH 4.14 33/64] x86/fpu: Make init_fpstate correct with optimized XSAVE Date: Tue, 24 Aug 2021 13:04:26 -0400 Message-Id: <20210824170457.710623-34-sashal@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210824170457.710623-1-sashal@kernel.org> References: <20210824170457.710623-1-sashal@kernel.org> MIME-Version: 1.0 X-KernelTest-Patch: http://kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.14.245-rc1.gz X-KernelTest-Tree: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git X-KernelTest-Branch: linux-4.14.y X-KernelTest-Patches: git://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git X-KernelTest-Version: 4.14.245-rc1 X-KernelTest-Deadline: 2021-08-26T17:04+00:00 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Thomas Gleixner commit f9dfb5e390fab2df9f7944bb91e7705aba14cd26 upstream. The XSAVE init code initializes all enabled and supported components with XRSTOR(S) to init state. Then it XSAVEs the state of the components back into init_fpstate which is used in several places to fill in the init state of components. This works correctly with XSAVE, but not with XSAVEOPT and XSAVES because those use the init optimization and skip writing state of components which are in init state. So init_fpstate.xsave still contains all zeroes after this operation. There are two ways to solve that: 1) Use XSAVE unconditionally, but that requires to reshuffle the buffer when XSAVES is enabled because XSAVES uses compacted format. 2) Save the components which are known to have a non-zero init state by other means. Looking deeper, #2 is the right thing to do because all components the kernel supports have all-zeroes init state except the legacy features (FP, SSE). Those cannot be hard coded because the states are not identical on all CPUs, but they can be saved with FXSAVE which avoids all conditionals. Use FXSAVE to save the legacy FP/SSE components in init_fpstate along with a BUILD_BUG_ON() which reminds developers to validate that a newly added component has all zeroes init state. As a bonus remove the now unused copy_xregs_to_kernel_booting() crutch. The XSAVE and reshuffle method can still be implemented in the unlikely case that components are added which have a non-zero init state and no other means to save them. For now, FXSAVE is just simple and good enough. [ bp: Fix a typo or two in the text. ] Fixes: 6bad06b76892 ("x86, xsave: Use xsaveopt in context-switch path when supported") Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Reviewed-by: Borislav Petkov Cc: stable@vger.kernel.org Link: https://lkml.kernel.org/r/20210618143444.587311343@linutronix.de Signed-off-by: Greg Kroah-Hartman --- arch/x86/include/asm/fpu/internal.h | 30 ++++++----------------- arch/x86/kernel/fpu/xstate.c | 38 ++++++++++++++++++++++++++--- 2 files changed, 43 insertions(+), 25 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index b8c935033d21..4f274d851986 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -215,6 +215,14 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu) } } +static inline void fxsave(struct fxregs_state *fx) +{ + if (IS_ENABLED(CONFIG_X86_32)) + asm volatile( "fxsave %[fx]" : [fx] "=m" (*fx)); + else + asm volatile("fxsaveq %[fx]" : [fx] "=m" (*fx)); +} + /* These macros all use (%edi)/(%rdi) as the single memory argument. */ #define XSAVE ".byte " REX_PREFIX "0x0f,0xae,0x27" #define XSAVEOPT ".byte " REX_PREFIX "0x0f,0xae,0x37" @@ -283,28 +291,6 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu) : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory") -/* - * This function is called only during boot time when x86 caps are not set - * up and alternative can not be used yet. - */ -static inline void copy_xregs_to_kernel_booting(struct xregs_state *xstate) -{ - u64 mask = -1; - u32 lmask = mask; - u32 hmask = mask >> 32; - int err; - - WARN_ON(system_state != SYSTEM_BOOTING); - - if (static_cpu_has(X86_FEATURE_XSAVES)) - XSTATE_OP(XSAVES, xstate, lmask, hmask, err); - else - XSTATE_OP(XSAVE, xstate, lmask, hmask, err); - - /* We should never fault when copying to a kernel buffer: */ - WARN_ON_FPU(err); -} - /* * This function is called only during boot time when x86 caps are not set * up and alternative can not be used yet. diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 601a5da1d196..7d372db8bee1 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -404,6 +404,24 @@ static void __init print_xstate_offset_size(void) } } +/* + * All supported features have either init state all zeros or are + * handled in setup_init_fpu() individually. This is an explicit + * feature list and does not use XFEATURE_MASK*SUPPORTED to catch + * newly added supported features at build time and make people + * actually look at the init state for the new feature. + */ +#define XFEATURES_INIT_FPSTATE_HANDLED \ + (XFEATURE_MASK_FP | \ + XFEATURE_MASK_SSE | \ + XFEATURE_MASK_YMM | \ + XFEATURE_MASK_OPMASK | \ + XFEATURE_MASK_ZMM_Hi256 | \ + XFEATURE_MASK_Hi16_ZMM | \ + XFEATURE_MASK_PKRU | \ + XFEATURE_MASK_BNDREGS | \ + XFEATURE_MASK_BNDCSR) + /* * setup the xstate image representing the init state */ @@ -411,6 +429,8 @@ static void __init setup_init_fpu_buf(void) { static int on_boot_cpu __initdata = 1; + BUILD_BUG_ON(XCNTXT_MASK != XFEATURES_INIT_FPSTATE_HANDLED); + WARN_ON_FPU(!on_boot_cpu); on_boot_cpu = 0; @@ -429,10 +449,22 @@ static void __init setup_init_fpu_buf(void) copy_kernel_to_xregs_booting(&init_fpstate.xsave); /* - * Dump the init state again. This is to identify the init state - * of any feature which is not represented by all zero's. + * All components are now in init state. Read the state back so + * that init_fpstate contains all non-zero init state. This only + * works with XSAVE, but not with XSAVEOPT and XSAVES because + * those use the init optimization which skips writing data for + * components in init state. + * + * XSAVE could be used, but that would require to reshuffle the + * data when XSAVES is available because XSAVES uses xstate + * compaction. But doing so is a pointless exercise because most + * components have an all zeros init state except for the legacy + * ones (FP and SSE). Those can be saved with FXSAVE into the + * legacy area. Adding new features requires to ensure that init + * state is all zeroes or if not to add the necessary handling + * here. */ - copy_xregs_to_kernel_booting(&init_fpstate.xsave); + fxsave(&init_fpstate.fxsave); } static int xfeature_uncompacted_offset(int xfeature_nr) -- 2.30.2