Received: by 2002:ac0:a594:0:0:0:0:0 with SMTP id m20-v6csp3859231imm; Thu, 17 May 2018 16:30:12 -0700 (PDT) X-Google-Smtp-Source: AB8JxZoU1abpFNJ4wsA2k9bVX0UvysVSxikOJsZGFTmxfZ8l21pigFlGYGxsCK54G5qKt997OWDE X-Received: by 2002:a65:4945:: with SMTP id q5-v6mr3586488pgs.177.1526599812801; Thu, 17 May 2018 16:30:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1526599812; cv=none; d=google.com; s=arc-20160816; b=SVazg/EklkgO/jiNG3nrq2lUhy3HxU2M4k5pw7pdhoPt8o0/bNTSvouqf82JmUgm91 vJZbFnQq7BSMZ9zZEwbw6cdt91zz8UrpU5NBKjFGIe5AH+hmK8VhZtl+7HBT4TzcRv8T 3Js6Cmec+2KD20XiKh7HfeUMX/poQ9HPVlJ2nMtoyKRWiDJDS5Do9kYnRCvr2fl9Q+fR qrC11bIJFQ6XqN9szEP/xfBXB0pSUDCXo68NXw74SoTkfrLNPeT11+YJIoz0wzIcRVd/ h9cVWilZsgn2inpqIOW3oK+PpdHqXJkJLMoUOC4Qo++sJe0qBZh8PUXg5zk1Y41ZSEge peKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:arc-authentication-results; bh=hjL/Eoipaq4g5lZnvmMi/aOKz/WA8Uoddtwncrig6r0=; b=mbUA+LOtjXtYpuDcve4qL01Oqp0hO2hgPxAEulBwlBFOUAMdORjAMoTT7QRR/8sVtz yjlW8A/zf2a/7YpLduI/nx9b0Sh4i1M7vfRBnU3s68FBxM9umI93WtdQDPIhB5ehD+yO e7qSVFwabIbZgdc7AZ73jQSbvByBVEzhzi1arZUeUleLrA1fcYu1GO9relmnWmSTA+4s tPA6eX4k8B3wQcWSDsPb3l9lDOZNafpYnG/u3ky2zT10IrNhSq9HteyYCZY2QZh1Yg9V 4UWniBix+JVnV9xhoIXrhr3FrGKcweqYLYgl9sz/lsVq/7llYYjciDPpIVf4W/UtG6GN amVg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k6-v6si419313pgo.99.2018.05.17.16.29.58; Thu, 17 May 2018 16:30:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752342AbeEQX3N (ORCPT + 99 others); Thu, 17 May 2018 19:29:13 -0400 Received: from ex13-edg-ou-001.vmware.com ([208.91.0.189]:31398 "EHLO EX13-EDG-OU-001.vmware.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752204AbeEQX2o (ORCPT ); Thu, 17 May 2018 19:28:44 -0400 Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-001.vmware.com (10.113.208.155) with Microsoft SMTP Server id 15.0.1156.6; Thu, 17 May 2018 16:28:13 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 756B840785; Thu, 17 May 2018 16:28:40 -0700 (PDT) From: Nadav Amit To: , CC: , Nadav Amit , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , Kees Cook , Jan Beulich , Josh Poimboeuf Subject: [PATCH 5/6] x86: refcount: prevent gcc distortions Date: Thu, 17 May 2018 09:14:01 -0700 Message-ID: <20180517161402.78089-6-namit@vmware.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180517161402.78089-1-namit@vmware.com> References: <20180517161402.78089-1-namit@vmware.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: None (EX13-EDG-OU-001.vmware.com: namit@vmware.com does not designate permitted sender hosts) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org GCC considers the number of statements in inlined assembly blocks, according to new-lines and semicolons, as an indication to the cost of the block in time and space. This data is distorted by the kernel code, which puts information in alternative sections. As a result, the compiler may perform incorrect inlining and branch optimizations. The solution is to set an assembly macro and call it from the inlined assembly block. As a result GCC considers the inline assembly block as a single instruction. This patch allows to inline functions such as __get_seccomp_filter(). The effect of the patch is as follows on the kernel size: text data bss dec hex filename 18146418 10064100 2936832 31147350 1db4556 ./vmlinux before 18148228 10063968 2936832 31149028 1db4be4 ./vmlinux after (+1678) Static text symbols: Before: 39673 After: 39649 (-24) Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin" Cc: x86@kernel.org Cc: Kees Cook Cc: Jan Beulich Cc: Josh Poimboeuf Signed-off-by: Nadav Amit --- arch/x86/include/asm/refcount.h | 62 +++++++++++++++++++++------------ 1 file changed, 39 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/refcount.h b/arch/x86/include/asm/refcount.h index 4cf11d88d3b3..3126141f2c80 100644 --- a/arch/x86/include/asm/refcount.h +++ b/arch/x86/include/asm/refcount.h @@ -13,35 +13,49 @@ * the exception handler to use (in mm/extable.c), and then triggers the * central refcount exception. The fixup address for the exception points * back to the regular execution flow in .text. + * + * The logic and data are encapsulated within an assembly macro, which is then + * called on each use. This hack is necessary to prevent GCC from considering + * the inline assembly blocks as costly in time and space, which can prevent + * function inlining and lead to other bad compilation decisions. GCC computes + * inline assembly cost according to the number of perceived number of assembly + * instruction, based on the number of new-lines and semicolons in the assembly + * block. The macros will eventually be compiled into a single instruction that + * will be actually executed (unless an exception happens). This scheme allows + * GCC to better understand the inline asm cost. */ -#define _REFCOUNT_EXCEPTION \ - ".pushsection .text..refcount\n" \ - "111:\tlea %[counter], %%" _ASM_CX "\n" \ - "112:\t" ASM_UD2 "\n" \ - ASM_UNREACHABLE \ - ".popsection\n" \ - "113:\n" \ - _ASM_EXTABLE_REFCOUNT(112b, 113b) +asm(".macro __REFCOUNT_EXCEPTION counter:req\n\t" + ".pushsection .text..refcount\n" + "111:\tlea \\counter, %" _ASM_CX "\n" + "112:\t" ASM_UD2 "\n\t" + ASM_UNREACHABLE + ".popsection\n" + "113:\n\t" + _ASM_EXTABLE_REFCOUNT(112b, 113b) "\n\t" + ".endm"); /* Trigger refcount exception if refcount result is negative. */ -#define REFCOUNT_CHECK_LT_ZERO \ - "js 111f\n\t" \ - _REFCOUNT_EXCEPTION +asm(".macro __REFCOUNT_CHECK_LT_ZERO counter:req\n\t" + "js 111f\n\t" + "__REFCOUNT_EXCEPTION \\counter\n\t" + ".endm"); /* Trigger refcount exception if refcount result is zero or negative. */ -#define REFCOUNT_CHECK_LE_ZERO \ - "jz 111f\n\t" \ - REFCOUNT_CHECK_LT_ZERO +asm(".macro __REFCOUNT_CHECK_LE_ZERO counter:req\n\t" + "jz 111f\n\t" + "__REFCOUNT_CHECK_LT_ZERO counter=\\counter\n\t" + ".endm"); /* Trigger refcount exception unconditionally. */ -#define REFCOUNT_ERROR \ - "jmp 111f\n\t" \ - _REFCOUNT_EXCEPTION +asm(".macro __REFCOUNT_ERROR counter:req\n\t" + "jmp 111f\n\t" + "__REFCOUNT_EXCEPTION counter=\\counter\n\t" + ".endm"); static __always_inline void refcount_add(unsigned int i, refcount_t *r) { asm volatile(LOCK_PREFIX "addl %1,%0\n\t" - REFCOUNT_CHECK_LT_ZERO + "__REFCOUNT_CHECK_LT_ZERO %[counter]" : [counter] "+m" (r->refs.counter) : "ir" (i) : "cc", "cx"); @@ -50,7 +64,7 @@ static __always_inline void refcount_add(unsigned int i, refcount_t *r) static __always_inline void refcount_inc(refcount_t *r) { asm volatile(LOCK_PREFIX "incl %0\n\t" - REFCOUNT_CHECK_LT_ZERO + "__REFCOUNT_CHECK_LT_ZERO %[counter]" : [counter] "+m" (r->refs.counter) : : "cc", "cx"); } @@ -58,7 +72,7 @@ static __always_inline void refcount_inc(refcount_t *r) static __always_inline void refcount_dec(refcount_t *r) { asm volatile(LOCK_PREFIX "decl %0\n\t" - REFCOUNT_CHECK_LE_ZERO + "__REFCOUNT_CHECK_LE_ZERO %[counter]" : [counter] "+m" (r->refs.counter) : : "cc", "cx"); } @@ -66,13 +80,15 @@ static __always_inline void refcount_dec(refcount_t *r) static __always_inline __must_check bool refcount_sub_and_test(unsigned int i, refcount_t *r) { - GEN_BINARY_SUFFIXED_RMWcc(LOCK_PREFIX "subl", REFCOUNT_CHECK_LT_ZERO, + GEN_BINARY_SUFFIXED_RMWcc(LOCK_PREFIX "subl", + "__REFCOUNT_CHECK_LT_ZERO %[counter]", r->refs.counter, "er", i, "%0", e, "cx"); } static __always_inline __must_check bool refcount_dec_and_test(refcount_t *r) { - GEN_UNARY_SUFFIXED_RMWcc(LOCK_PREFIX "decl", REFCOUNT_CHECK_LT_ZERO, + GEN_UNARY_SUFFIXED_RMWcc(LOCK_PREFIX "decl", + "__REFCOUNT_CHECK_LT_ZERO %[counter]", r->refs.counter, "%0", e, "cx"); } @@ -90,7 +106,7 @@ bool refcount_add_not_zero(unsigned int i, refcount_t *r) /* Did we try to increment from/to an undesirable state? */ if (unlikely(c < 0 || c == INT_MAX || result < c)) { - asm volatile(REFCOUNT_ERROR + asm volatile("__REFCOUNT_ERROR %[counter]" : : [counter] "m" (r->refs.counter) : "cc", "cx"); break; -- 2.17.0