Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751474AbdFAHA2 (ORCPT ); Thu, 1 Jun 2017 03:00:28 -0400 Received: from mail.fireflyinternet.com ([109.228.58.192]:64415 "EHLO fireflyinternet.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751013AbdFAHA1 (ORCPT ); Thu, 1 Jun 2017 03:00:27 -0400 X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; From: Chris Wilson To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, intel-gfx@lists.freedesktop.org, Chris Wilson , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" Subject: [PATCH 1/3] x86-32: Teach copy_from_user to unroll .size=6/8 Date: Thu, 1 Jun 2017 07:58:41 +0100 Message-Id: <20170601065843.2392-2-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170601065843.2392-1-chris@chris-wilson.co.uk> References: <20170601065843.2392-1-chris@chris-wilson.co.uk> X-Originating-IP: 78.156.65.138 X-Country: code=GB country="United Kingdom" ip=78.156.65.138 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2156 Lines: 71 Two exception handling register moves are faster to inline than a call to __copy_user_ll(). We already apply the conversion for a get_user() call, so for symmetry we should also apply the optimisation to copy_from_user. Signed-off-by: Chris Wilson Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin" --- arch/x86/include/asm/uaccess_32.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/uaccess_32.h b/arch/x86/include/asm/uaccess_32.h index aeda9bb8af50..44d17d1ab07c 100644 --- a/arch/x86/include/asm/uaccess_32.h +++ b/arch/x86/include/asm/uaccess_32.h @@ -23,30 +23,47 @@ static __always_inline unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n) { if (__builtin_constant_p(n)) { - unsigned long ret; + unsigned long ret = 0; switch (n) { case 1: - ret = 0; __uaccess_begin(); __get_user_asm_nozero(*(u8 *)to, from, ret, "b", "b", "=q", 1); __uaccess_end(); return ret; case 2: - ret = 0; __uaccess_begin(); __get_user_asm_nozero(*(u16 *)to, from, ret, "w", "w", "=r", 2); __uaccess_end(); return ret; case 4: - ret = 0; __uaccess_begin(); __get_user_asm_nozero(*(u32 *)to, from, ret, "l", "k", "=r", 4); __uaccess_end(); return ret; + case 6: + __uaccess_begin(); + __get_user_asm_nozero(*(u32 *)to, from, ret, + "l", "k", "=r", 6); + if (likely(!ret)) + __get_user_asm_nozero(*(u16 *)(4 + (char *)to), + (u16 __user *)(4 + (char __user *)from), + ret, "w", "w", "=r", 2); + __uaccess_end(); + return ret; + case 8: + __uaccess_begin(); + __get_user_asm_nozero(*(u32 *)to, from, ret, + "l", "k", "=r", 8); + if (likely(!ret)) + __get_user_asm_nozero(*(u32 *)(4 + (char *)to), + (u32 __user *)(4 + (char __user *)from), + ret, "l", "k", "=r", 4); + __uaccess_end(); + return ret; } } return __copy_user_ll(to, (__force const void *)from, n); -- 2.11.0