Received: by 2002:ac0:a5b6:0:0:0:0:0 with SMTP id m51-v6csp883466imm; Thu, 31 May 2018 11:03:52 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJo8oK7+2jqf5komC152VAumIK340joThD/grUiQh31564Spnbo84VXWAtb+2iYgCPfBwl4 X-Received: by 2002:a63:4004:: with SMTP id n4-v6mr6410994pga.248.1527789832718; Thu, 31 May 2018 11:03:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527789832; cv=none; d=google.com; s=arc-20160816; b=bbWLwM8qq8MLsQ0E4DrQYAYg6dzcmQ86WuzNhUV5RPgOPOUpl/WlPWOCG1n7/MlBsD jxvklKeNh5VvKuCZCZHNGaxOS8dEOGGx2FskIkbOGO6aq9sEDFZj6Ss7qZFswZD0C9ok m0+4krLv352wXz3XMPIU4yJ1K/75SORRIEz2CbjzZQTJzcws2pz1bOUVTbT3QieP0zVY ZufoOTdBQFYLE2bRowDM3nB+ow4Omfak5Z7VGumXXOs3hcjsBVOA0np1TkZPadZcTOOr empeq89Tgw8qSu/qOj4E6hkr6eE7H5bP9gm/oGRjs/JezVfgR4o80/nacAbNRKtxwBOb 4aJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=WbuRgZKNeA9Nlbr5+yLXnzKVKOawEpyna/daxf61mGg=; b=MmhxfpYqhgLhIw7SXDdmZT7RSR0a2X98gDO/MY2SG2SaSbumyiW1asyFJw25Z9g3ky I5Vnwy8iSv4+vUpH4EbMmYpHbIblfnvJHmpk1V/+dqb108ylbBqTH89Ls8eT/yuuBA1Y roxfq3mf0PHYgsaQX6l56P/7poCbzx76HP5RN3RvSmw4+378VUkfO22UA9PwcLULUFl2 xoZFgQ2qwLwk/RhXPO2ABhw6qvgl9UtxorYgncAanpWAtd4RDsorvyGQ2fdv8hrkveJm MwZzhjwEhfb/e7r1HUf20zWFmyExZurW9+OROHxdbmNe2SbT/SFp0zSoUYarMQvChQVC aFXw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s2-v6si10118149plq.372.2018.05.31.11.03.38; Thu, 31 May 2018 11:03:52 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756009AbeEaSBM (ORCPT + 99 others); Thu, 31 May 2018 14:01:12 -0400 Received: from mga01.intel.com ([192.55.52.88]:18561 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755956AbeEaR7B (ORCPT ); Thu, 31 May 2018 13:59:01 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 31 May 2018 10:58:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,463,1520924400"; d="scan'208";a="60725528" Received: from chang-linux-2.sc.intel.com ([10.3.52.139]) by orsmga001.jf.intel.com with ESMTP; 31 May 2018 10:58:57 -0700 From: "Chang S. Bae" To: Andy Lutomirski , "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar Cc: Andi Kleen , Dave Hansen , Markus T Metzger , "Ravi V . Shankar" , "Chang S . Bae" , linux-kernel@vger.kernel.org Subject: [PATCH V2 09/15] x86/fsgsbase/64: Enable FSGSBASE instructions in helper functions Date: Thu, 31 May 2018 10:58:39 -0700 Message-Id: <1527789525-8857-10-git-send-email-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1527789525-8857-1-git-send-email-chang.seok.bae@intel.com> References: <1527789525-8857-1-git-send-email-chang.seok.bae@intel.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The helper functions switch on faster access to FS/GS, when FSGSBASE enabled. Accessing user GS base needs a couple of SWPAGS. It is avoidable if the user GS base is copied at kernel entry and updated as changed, and (actual) GS base is written back at kernel exit. However, it costs more cycles to do that. The measured overhead was (almost) offset to the benefit. Signed-off-by: Chang S. Bae Reviewed-by: Andi Kleen Cc: Any Lutomirski Cc: H. Peter Anvin Cc: Dave Hansen Cc: Thomas Gleixner Cc: Ingo Molnar --- arch/x86/include/asm/fsgsbase.h | 17 ++++------ arch/x86/kernel/process_64.c | 75 +++++++++++++++++++++++++++++++++++------ 2 files changed, 72 insertions(+), 20 deletions(-) diff --git a/arch/x86/include/asm/fsgsbase.h b/arch/x86/include/asm/fsgsbase.h index ed42015..903c7a0 100644 --- a/arch/x86/include/asm/fsgsbase.h +++ b/arch/x86/include/asm/fsgsbase.h @@ -54,26 +54,23 @@ static __always_inline void wrgsbase(unsigned long gsbase) : "memory"); } +#include + /* Helper functions for reading/writing FS/GS base */ static inline unsigned long read_fsbase(void) { unsigned long fsbase; - rdmsrl(MSR_FS_BASE, fsbase); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + fsbase = rdfsbase(); + else + rdmsrl(MSR_FS_BASE, fsbase); return fsbase; } void write_fsbase(unsigned long fsbase); - -static inline unsigned long read_inactive_gsbase(void) -{ - unsigned long gsbase; - - rdmsrl(MSR_KERNEL_GS_BASE, gsbase); - return gsbase; -} - +unsigned long read_inactive_gsbase(void); void write_inactive_gsbase(unsigned long gsbase); #endif /* CONFIG_X86_64 */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index cebf240..8ba947f 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -154,6 +154,38 @@ enum which_selector { }; /* + * Interrupts are disabled here. + * Out of line to be protected from kprobes. + */ +static noinline __kprobes unsigned long rd_inactive_gsbase(void) +{ + unsigned long gsbase, flags; + + local_irq_save(flags); + native_swapgs(); + gsbase = rdgsbase(); + native_swapgs(); + local_irq_restore(flags); + + return gsbase; +} + +/* + * Interrupts are disabled here. + * Out of line to be protected from kprobes. + */ +static noinline __kprobes void wr_inactive_gsbase(unsigned long gsbase) +{ + unsigned long flags; + + local_irq_save(flags); + native_swapgs(); + wrgsbase(gsbase); + native_swapgs(); + local_irq_restore(flags); +} + +/* * Saves the FS or GS base for an outgoing thread if FSGSBASE extensions are * not available. The goal is to be reasonably fast on non-FSGSBASE systems. * It's forcibly inlined because it'll generate better code and this function @@ -333,16 +365,35 @@ static unsigned long task_seg_base(struct task_struct *task, void write_fsbase(unsigned long fsbase) { - /* set the selector to 0 to not confuse __switch_to */ - loadseg(FS, 0); - wrmsrl(MSR_FS_BASE, fsbase); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + wrfsbase(fsbase); + } else { + /* set the selector to 0 to not confuse __switch_to */ + loadseg(FS, 0); + wrmsrl(MSR_FS_BASE, fsbase); + } +} + +unsigned long read_inactive_gsbase(void) +{ + unsigned long gsbase; + + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + gsbase = rd_inactive_gsbase(); + else + rdmsrl(MSR_KERNEL_GS_BASE, gsbase); + return gsbase; } void write_inactive_gsbase(unsigned long gsbase) { - /* set the selector to 0 to not confuse __switch_to */ - loadseg(GS, 0); - wrmsrl(MSR_KERNEL_GS_BASE, gsbase); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + wr_inactive_gsbase(gsbase); + } else { + /* set the selector to 0 to not confuse __switch_to */ + loadseg(GS, 0); + wrmsrl(MSR_KERNEL_GS_BASE, gsbase); + } } unsigned long read_task_fsbase(struct task_struct *task) @@ -351,7 +402,8 @@ unsigned long read_task_fsbase(struct task_struct *task) if (task == current) fsbase = read_fsbase(); - else if (task->thread.fsindex == 0) + else if (static_cpu_has(X86_FEATURE_FSGSBASE) || + (task->thread.fsindex == 0)) fsbase = task->thread.fsbase; else fsbase = task_seg_base(task, task->thread.fsindex); @@ -365,7 +417,8 @@ unsigned long read_task_gsbase(struct task_struct *task) if (task == current) gsbase = read_inactive_gsbase(); - else if (task->thread.gsindex == 0) + else if (static_cpu_has(X86_FEATURE_FSGSBASE) || + (task->thread.gsindex == 0)) gsbase = task->thread.gsbase; else gsbase = task_seg_base(task, task->thread.gsindex); @@ -388,7 +441,8 @@ int write_task_fsbase(struct task_struct *task, unsigned long fsbase) task->thread.fsbase = fsbase; if (task == current) write_fsbase(fsbase); - task->thread.fsindex = 0; + if (!static_cpu_has(X86_FEATURE_FSGSBASE)) + task->thread.fsindex = 0; put_cpu(); return 0; @@ -405,7 +459,8 @@ int write_task_gsbase(struct task_struct *task, unsigned long gsbase) task->thread.gsbase = gsbase; if (task == current) write_inactive_gsbase(gsbase); - task->thread.gsindex = 0; + if (!static_cpu_has(X86_FEATURE_FSGSBASE)) + task->thread.gsindex = 0; put_cpu(); return 0; -- 2.7.4