Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756674AbeAIM5m (ORCPT + 1 other); Tue, 9 Jan 2018 07:57:42 -0500 Received: from wtarreau.pck.nerim.net ([62.212.114.60]:39117 "EHLO 1wt.eu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753356AbeAIM5k (ORCPT ); Tue, 9 Jan 2018 07:57:40 -0500 From: Willy Tarreau To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Willy Tarreau , Andy Lutomirski , Borislav Petkov , Brian Gerst , Dave Hansen , Ingo Molnar , Linus Torvalds , Peter Zijlstra , Thomas Gleixner , Josh Poimboeuf , "H. Peter Anvin" , Kees Cook Subject: [RFC PATCH v2 5/6] x86/entry/pti: avoid setting CR3 when it's already correct Date: Tue, 9 Jan 2018 13:56:19 +0100 Message-Id: <1515502580-12261-6-git-send-email-w@1wt.eu> X-Mailer: git-send-email 2.8.0.rc2.1.gbe9624a In-Reply-To: <1515502580-12261-1-git-send-email-w@1wt.eu> References: <1515502580-12261-1-git-send-email-w@1wt.eu> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Return-Path: When entering the kernel with CR3 pointing to the kernel's PGD, there's no need to set it again. This will avoid a TLB flush on syscalls for tasks running with the kernel's PGD (see next patch). Signed-off-by: Willy Tarreau Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Dave Hansen Cc: Ingo Molnar Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Josh Poimboeuf Cc: "H. Peter Anvin" Cc: Kees Cook v2: - updated comments according to Ingo's suggestions - split the code to keep only the CR3 changes here --- arch/x86/entry/calling.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h index 45a63e0..2c0d3b5 100644 --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -214,6 +214,11 @@ .macro SWITCH_TO_KERNEL_CR3 scratch_reg:req ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI mov %cr3, \scratch_reg + + /* if we're already on the kernel PGD, we don't switch */ + testq $(PTI_SWITCH_PGTABLES_MASK), \scratch_reg + jz .Lend_\@ + ADJUST_KERNEL_CR3 \scratch_reg mov \scratch_reg, %cr3 .Lend_\@: @@ -262,6 +267,14 @@ ALTERNATIVE "jmp .Ldone_\@", "", X86_FEATURE_PTI movq %cr3, \scratch_reg movq \scratch_reg, \save_reg + + /* + * If we're already on the kernel PGD, we don't switch, + * we just save the current CR3. + */ + testq $(PTI_SWITCH_PGTABLES_MASK), \scratch_reg + jz .Ldone_\@ + /* * Is the "switch mask" all zero? That means that both of * these are zero: @@ -284,6 +297,13 @@ .macro RESTORE_CR3 scratch_reg:req save_reg:req ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI + /* + * If we saved a kernel context on entry, we didn't switch the CR3, + * so we don't need to restore it on the way out either: + */ + testq $(PTI_SWITCH_PGTABLES_MASK), \save_reg + jz .Lend_\@ + ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID /* -- 1.7.12.1