Received: by 2002:a25:ef43:0:0:0:0:0 with SMTP id w3csp716239ybm; Thu, 28 May 2020 13:17:43 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwXspTLmWLboruRJ0k5cZFHs/k+TjzIeLmQVkfsF8dBN9zjXdLuyvkTzQfZOvM3AJrmDBJ+ X-Received: by 2002:a17:906:c108:: with SMTP id do8mr5009985ejc.134.1590697063640; Thu, 28 May 2020 13:17:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1590697063; cv=none; d=google.com; s=arc-20160816; b=b8CLxbIYig4UnH2qqtZLNKjQJKTEoQr+OdgF6RRyCmiagY7xn9Jl2i4roArFxni3gd NSBSTQD6Y0qO1lhogLvlLZczmwaJeqdbnesaXTh73jq7Y2EWYKRyU0dAdlESkpX7szYP NkTvt5/smY9T02WUCf29iWRa5mQ1dXwm4w1o2cMDimWvtI0joDg3yf7IBbt2cSjcjRLX 8D0CrFZogUIGehPg3zl7vVFVWQTCIVE6A4VZOcfh4SZL3Am+NHdUyM8UZqtxQxC51n+G EtWEqWSbR1LmR991tIu0mOFMwUP5SWAbAwWm/o2UCIJ2SqwHqQ3dHuR/ek1Sxd8oDlmJ e20A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ARG02polOIiyuVhQkPLNpw/1eCGG+2BXcoZaNik+YHw=; b=gQzhvLKbWTdZkjsDu0NnYs3tkJUhOdJi5d6jZm9JlVsnJATxqV9aykUKXOjIU3FRfG AVxfWfv7O13Qofu/MjG0NWuKR8CuP8ezgZi50PSPEKiij7zuEpxxqTvtMWe0la6rad7p xBk41Ydm2AJx0U+mGM2up6YAuIMyYLyoItHFTkOibBmcFwgw1F+nJ5OcSFFgdxwuXiR4 Jci3+2gMFPYWsBo8vkc8pkZhuAFEz/bL6uNzORfg8Iwya5ZHmqBIMZtaV+/Ve1p0jvxk ynLsY3QxrfCvP1ZNaP/KxdtflUCCjUI+VLqaL2DoBBqu/9aK630kUX/Pg3ZELlXsFDKM S0dQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=p9Fvr+Pl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id m11si1292307edp.600.2020.05.28.13.17.20; Thu, 28 May 2020 13:17:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=p9Fvr+Pl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2407044AbgE1UOj (ORCPT + 99 others); Thu, 28 May 2020 16:14:39 -0400 Received: from mail.kernel.org ([198.145.29.99]:41714 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2407008AbgE1UOU (ORCPT ); Thu, 28 May 2020 16:14:20 -0400 Received: from sasha-vm.mshome.net (c-73-47-72-35.hsd1.nh.comcast.net [73.47.72.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id F3AAF20B80; Thu, 28 May 2020 20:14:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1590696859; bh=iU9LWSJ5Vw8e1tY/J5Ku+EAQF/rbekFaByBUmBynFA8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p9Fvr+PlRCyIAqxQgBPrRZb89K7suMo51m7JARIGJWT/v91JQ3e9FculK2x0HNzmm rRDNzo/VbWdr9jIQLf5vQdMcA8oQdF/dY6d2RZ8EguY+pR9O5IQjjCXRXl5ccHebIJ yt9YkEnTwomvgP4mK95/U/VBEz5xmwDAp6s6z+5A= From: Sasha Levin To: tglx@linutronix.de, luto@kernel.org, ak@linux.intel.com Cc: corbet@lwn.net, mingo@redhat.com, bp@alien8.de, x86@kernel.org, shuah@kernel.org, gregkh@linuxfoundation.org, tony.luck@intel.com, chang.seok.bae@intel.com, dave.hansen@linux.intel.com, peterz@infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, jarkko.sakkinen@linux.intel.com, Ravi Shankar , Andrew Cooper , "H . Peter Anvin" , Sasha Levin Subject: [PATCH v13 04/16] x86/fsgsbase/64: Enable FSGSBASE instructions in helper functions Date: Thu, 28 May 2020 16:13:50 -0400 Message-Id: <20200528201402.1708239-5-sashal@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200528201402.1708239-1-sashal@kernel.org> References: <20200528201402.1708239-1-sashal@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: "Chang S. Bae" Add cpu feature conditional FSGSBASE access to the relevant helper functions. That allows to accelerate certain FS/GS base operations in subsequent changes. Note, that while possible, the user space entry/exit GSBASE operations are not going to use the new FSGSBASE instructions. The reason is that it would require additional storage for the user space value which adds more complexity to the low level code and experiments have shown marginal benefit. This may be revisited later but for now the SWAPGS based handling in the entry code is preserved except for the paranoid entry/exit code. To preserve the SWAPGS entry mechanism introduce __[rd|wr]gsbase_inactive() helpers. Note, for Xen PV, paravirt hooks can be added later as they might allow a very efficient but different implementation. [ tglx: Massaged changelog ] Signed-off-by: Chang S. Bae Signed-off-by: Thomas Gleixner Cc: Andy Lutomirski Cc: Andi Kleen Cc: Ravi Shankar Cc: Andrew Cooper Cc: H. Peter Anvin Link: https://lkml.kernel.org/r/1557309753-24073-7-git-send-email-chang.seok.bae@intel.com Signed-off-by: Sasha Levin --- arch/x86/include/asm/fsgsbase.h | 27 +++++++------- arch/x86/kernel/process_64.c | 64 +++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/fsgsbase.h b/arch/x86/include/asm/fsgsbase.h index fdd1177499b4..aefd53767a5d 100644 --- a/arch/x86/include/asm/fsgsbase.h +++ b/arch/x86/include/asm/fsgsbase.h @@ -49,35 +49,32 @@ static __always_inline void wrgsbase(unsigned long gsbase) asm volatile("wrgsbase %0" :: "r" (gsbase) : "memory"); } +#include + /* Helper functions for reading/writing FS/GS base */ static inline unsigned long x86_fsbase_read_cpu(void) { unsigned long fsbase; - rdmsrl(MSR_FS_BASE, fsbase); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + fsbase = rdfsbase(); + else + rdmsrl(MSR_FS_BASE, fsbase); return fsbase; } -static inline unsigned long x86_gsbase_read_cpu_inactive(void) -{ - unsigned long gsbase; - - rdmsrl(MSR_KERNEL_GS_BASE, gsbase); - - return gsbase; -} - static inline void x86_fsbase_write_cpu(unsigned long fsbase) { - wrmsrl(MSR_FS_BASE, fsbase); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + wrfsbase(fsbase); + else + wrmsrl(MSR_FS_BASE, fsbase); } -static inline void x86_gsbase_write_cpu_inactive(unsigned long gsbase) -{ - wrmsrl(MSR_KERNEL_GS_BASE, gsbase); -} +extern unsigned long x86_gsbase_read_cpu_inactive(void); +extern void x86_gsbase_write_cpu_inactive(unsigned long gsbase); #endif /* CONFIG_X86_64 */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 5ef9d8f25b0e..0624689825a3 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -150,6 +150,40 @@ enum which_selector { GS }; +/* + * Out of line to be protected from kprobes. It is not used on Xen + * paravirt. When paravirt support is needed, it needs to be renamed + * with native_ prefix. + */ +static noinline unsigned long __rdgsbase_inactive(void) +{ + unsigned long gsbase; + + lockdep_assert_irqs_disabled(); + + native_swapgs(); + gsbase = rdgsbase(); + native_swapgs(); + + return gsbase; +} +NOKPROBE_SYMBOL(__rdgsbase_inactive); + +/* + * Out of line to be protected from kprobes. It is not used on Xen + * paravirt. When paravirt support is needed, it needs to be renamed + * with native_ prefix. + */ +static noinline void __wrgsbase_inactive(unsigned long gsbase) +{ + lockdep_assert_irqs_disabled(); + + native_swapgs(); + wrgsbase(gsbase); + native_swapgs(); +} +NOKPROBE_SYMBOL(__wrgsbase_inactive); + /* * Saves the FS or GS base for an outgoing thread if FSGSBASE extensions are * not available. The goal is to be reasonably fast on non-FSGSBASE systems. @@ -328,6 +362,36 @@ static unsigned long x86_fsgsbase_read_task(struct task_struct *task, return base; } +unsigned long x86_gsbase_read_cpu_inactive(void) +{ + unsigned long gsbase; + + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + unsigned long flags; + + local_irq_save(flags); + gsbase = __rdgsbase_inactive(); + local_irq_restore(flags); + } else { + rdmsrl(MSR_KERNEL_GS_BASE, gsbase); + } + + return gsbase; +} + +void x86_gsbase_write_cpu_inactive(unsigned long gsbase) +{ + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + unsigned long flags; + + local_irq_save(flags); + __wrgsbase_inactive(gsbase); + local_irq_restore(flags); + } else { + wrmsrl(MSR_KERNEL_GS_BASE, gsbase); + } +} + unsigned long x86_fsbase_read_task(struct task_struct *task) { unsigned long fsbase; -- 2.25.1