Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757874AbcLOQr1 (ORCPT ); Thu, 15 Dec 2016 11:47:27 -0500 Received: from Galois.linutronix.de ([146.0.238.70]:47771 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752327AbcLOQrW (ORCPT ); Thu, 15 Dec 2016 11:47:22 -0500 Message-Id: <20161215164240.904945084@linutronix.de> User-Agent: quilt/0.63-1 Date: Thu, 15 Dec 2016 16:44:03 -0000 From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Peter Zijlstra , Kyle Huey , Andy Lutomirski Subject: [patch 3/3] x86/process: Optimize TIF_NOTSC switch References: <20161215162648.061449202@linutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-15 Content-Disposition: inline; filename=x86-process--Optimize-TIF_NOTSC-switch.patch Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2112 Lines: 91 Provide and use a toggle helper instead of doing it with a branch. x86_64: 3662 8505 16 12183 2f97 Before 3646 8505 16 12167 2f87 After i386: 5906 9388 1804 17098 42ca Before 5834 9324 1740 16898 4202 After Signed-off-by: Thomas Gleixner --- arch/x86/include/asm/tlbflush.h | 10 ++++++++++ arch/x86/kernel/process.c | 22 ++++------------------ 2 files changed, 14 insertions(+), 18 deletions(-) --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -110,6 +110,16 @@ static inline void cr4_clear_bits(unsign } } +static inline void cr4_toggle_bits(unsigned long mask) +{ + unsigned long cr4; + + cr4 = this_cpu_read(cpu_tlbstate.cr4); + cr4 ^= mask; + this_cpu_write(cpu_tlbstate.cr4, cr4); + __write_cr4(cr4); +} + /* Read the CR4 shadow. */ static inline unsigned long cr4_read_shadow(void) { --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -116,11 +116,6 @@ void flush_thread(void) fpu__clear(&tsk->thread.fpu); } -static void hard_disable_TSC(void) -{ - cr4_set_bits(X86_CR4_TSD); -} - void disable_TSC(void) { preempt_disable(); @@ -129,15 +124,10 @@ void disable_TSC(void) * Must flip the CPU state synchronously with * TIF_NOTSC in the current running context. */ - hard_disable_TSC(); + cr4_set_bits(X86_CR4_TSD); preempt_enable(); } -static void hard_enable_TSC(void) -{ - cr4_clear_bits(X86_CR4_TSD); -} - static void enable_TSC(void) { preempt_disable(); @@ -146,7 +136,7 @@ static void enable_TSC(void) * Must flip the CPU state synchronously with * TIF_NOTSC in the current running context. */ - hard_enable_TSC(); + cr4_clear_bits(X86_CR4_TSD); preempt_enable(); } @@ -212,12 +202,8 @@ void __switch_to_xtra(struct task_struct if ((tifp ^ tifn) & _TIF_BLOCKSTEP) toggle_debugctlmsr(DEBUGCTLMSR_BTF); - if ((tifp ^ tifn) & _TIF_NOTSC) { - if (tifn & _TIF_NOTSC) - hard_disable_TSC(); - else - hard_enable_TSC(); - } + if ((tifp ^ tifn) & _TIF_NOTSC) + cr4_toggle_bits(X86_CR4_TSD); } /*