Received: by 10.213.65.68 with SMTP id h4csp1742072imn; Sun, 1 Apr 2018 13:44:44 -0700 (PDT) X-Google-Smtp-Source: AIpwx49jv/QIa5tyzh6qKNzSQg9YGhKqHc7/fHSBzyQBf+KhzgTaqig0Fxs8JH09Oe/hgvI2MEWQ X-Received: by 2002:a17:902:8490:: with SMTP id c16-v6mr7447971plo.132.1522615484292; Sun, 01 Apr 2018 13:44:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1522615484; cv=none; d=google.com; s=arc-20160816; b=Xu9VIur+crnz0cOfpLEGY801qjKhyjBpqQ8cP4y8PvKyC94Ywt0YS+MROUR5V+xu5q J65JilGkIHwR6Q62Ic9n3LmbIBKnvk+XWqwD52Q7eNVwe3GVy0mvZwcoHMbFFQRhMmKL V0JecJ5SV8PqMfZjZUAfH+VqtnALpszT9aIL0R3/Lg4WUmCz3jbtSpXoM503Ps0qmvwJ px4fKoRrAjyouqiUTYELt7n2Xn0m8DZEWw1iB4DH9wM9NzymZ2ZLRrB79YnND7vi7wwl blbyQorG0ehngcPLfCqg1/eBzTOtBxZ5wmDem05o2xjldJ3CUqB+Vlij8b0VT8FMbTNF nhOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:to:from:subject :organization:arc-authentication-results; bh=7HVZ6eoy//b6mk5A5WAKIZeMTMEYtev0Kbxm+RIOavU=; b=XBPrR3tYvqE+2ALGKki8HSNxleLTA7phW19fcunC/7+4hbHI/C+1ZV1uJ/bTBofslI /XsreLHuOh3uMcnpWrgJam45vF2LnGMSp2sMWJGtO07pj/ks6vlDcc7Q2WrR4/fZqYGZ SBe1bMlvMNK90dkX9NZuh50AIkII5oltQ/ZmxAsk9JVgchkJ+1fNHT7lNrVa8TGpUrOh 7gIUmjH3OW3ADhYyTuzHg+j5mDDK95jy5WZgb8WOCzLhiNVjVmkSVkaq3c3a8DN39YRu lzIyAMI9YOsw8q7ItenoVsS75G/JvA9x+sq5Q7+4wQxYeRJu2xAdFEoPOjIFRWE2IIkI 4vxA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z123si8777869pgb.466.2018.04.01.13.44.30; Sun, 01 Apr 2018 13:44:44 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754185AbeDAUlZ (ORCPT + 99 others); Sun, 1 Apr 2018 16:41:25 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:42132 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753951AbeDAUlX (ORCPT ); Sun, 1 Apr 2018 16:41:23 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 75DAB4023141 for ; Sun, 1 Apr 2018 20:41:22 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-120-59.rdu2.redhat.com [10.10.120.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0BDEED7DF5; Sun, 1 Apr 2018 20:41:21 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 11/45] C++: x86: Turn cmpxchg() & co. into inline template functions From: David Howells To: linux-kernel@vger.kernel.org Date: Sun, 01 Apr 2018 21:41:21 +0100 Message-ID: <152261528152.30503.6660578276914715570.stgit@warthog.procyon.org.uk> In-Reply-To: <152261521484.30503.16131389653845029164.stgit@warthog.procyon.org.uk> References: <152261521484.30503.16131389653845029164.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Sun, 01 Apr 2018 20:41:22 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Sun, 01 Apr 2018 20:41:22 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dhowells@redhat.com' RCPT:'' Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Turn cmpxchg() and similar functions into inline C++ template functions. This produces more robust source as the all the casting the C macros require is then unnecessary. Signed-off-by: David Howells --- arch/x86/include/asm/atomic.h | 2 arch/x86/include/asm/cmpxchg.h | 222 +++++++++++++--------------------------- 2 files changed, 75 insertions(+), 149 deletions(-) diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 72759f131cc5..20f225cd47e8 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -190,7 +190,7 @@ static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new) #define atomic_try_cmpxchg atomic_try_cmpxchg static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) { - return try_cmpxchg(&v->counter, old, new); + return try_cmpxchg(&v->counter, *old, new); } static inline int atomic_xchg(atomic_t *v, int new) diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h index 5e896c17476d..4bbf947c88a2 100644 --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -6,6 +6,12 @@ #include #include /* Provides LOCK_PREFIX */ +enum cmpxchg_lock { + __lock_local, + __lock_smp, + __lock_always +}; + /* * Non-existant functions to indicate usage errors at link time * (or compile-time if the compiler implements __compiletime_error(). @@ -20,22 +26,6 @@ extern void __add_wrong_size(void) __compiletime_error("Bad argument size for add"); /* - * Constants for operation sizes. On 32-bit, the 64-bit size it set to - * -1 because sizeof will never return -1, thereby making those switch - * case statements guaranteeed dead code which the compiler will - * eliminate, and allowing the "missing symbol in the default case" to - * indicate a usage error. - */ -#define __X86_CASE_B 1 -#define __X86_CASE_W 2 -#define __X86_CASE_L 4 -#ifdef CONFIG_64BIT -#define __X86_CASE_Q 8 -#else -#define __X86_CASE_Q -1 /* sizeof will never return -1 */ -#endif - -/* * An exchange-type operation, which takes a value and a pointer, and * returns the old value. */ @@ -107,67 +97,82 @@ static inline P xadd(P *ptr, N inc) return v; } +/* + * Atomic compare and exchange. + */ +template +static inline bool try_cmpxchg(P *ptr, P &old, N _rep, + enum cmpxchg_lock lock = __lock_smp) +{ + bool success; + P rep = _rep; + + if (sizeof(P) > sizeof(unsigned long)) + __cmpxchg_wrong_size(); + + switch (lock) { + case __lock_local: + asm volatile("cmpxchg %[rep], %[ptr]" + CC_SET(z) + : CC_OUT(z) (success), + [ptr] "+m" (*ptr), + [old] "+a" (old) + : [rep] "q" (rep) + : "memory"); + break; + case __lock_smp: + asm volatile(LOCK_PREFIX "cmpxchg %[rep], %[ptr]" + CC_SET(z) + : CC_OUT(z) (success), + [ptr] "+m" (*ptr), + [old] "+a" (old) + : [rep] "q" (rep) + : "memory"); + break; + case __lock_always: + asm volatile("lock; cmpxchg %[rep], %[ptr]" + CC_SET(z) + : CC_OUT(z) (success), + [ptr] "+m" (*ptr), + [old] "+a" (old) + : [rep] "q" (rep) + : "memory"); + break; + } + + return likely(success); +} + + /* * Atomic compare and exchange. Compare OLD with MEM, if identical, * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ -#define __raw_cmpxchg(ptr, old, new, size, lock) \ -({ \ - __typeof__(*(ptr)) __ret; \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - switch (size) { \ - case __X86_CASE_B: \ - { \ - volatile u8 *__ptr = (volatile u8 *)(ptr); \ - asm volatile(lock "cmpxchgb %2,%1" \ - : "=a" (__ret), "+m" (*__ptr) \ - : "q" (__new), "0" (__old) \ - : "memory"); \ - break; \ - } \ - case __X86_CASE_W: \ - { \ - volatile u16 *__ptr = (volatile u16 *)(ptr); \ - asm volatile(lock "cmpxchgw %2,%1" \ - : "=a" (__ret), "+m" (*__ptr) \ - : "r" (__new), "0" (__old) \ - : "memory"); \ - break; \ - } \ - case __X86_CASE_L: \ - { \ - volatile u32 *__ptr = (volatile u32 *)(ptr); \ - asm volatile(lock "cmpxchgl %2,%1" \ - : "=a" (__ret), "+m" (*__ptr) \ - : "r" (__new), "0" (__old) \ - : "memory"); \ - break; \ - } \ - case __X86_CASE_Q: \ - { \ - volatile u64 *__ptr = (volatile u64 *)(ptr); \ - asm volatile(lock "cmpxchgq %2,%1" \ - : "=a" (__ret), "+m" (*__ptr) \ - : "r" (__new), "0" (__old) \ - : "memory"); \ - break; \ - } \ - default: \ - __cmpxchg_wrong_size(); \ - } \ - __ret; \ -}) -#define __cmpxchg(ptr, old, new, size) \ - __raw_cmpxchg((ptr), (old), (new), (size), LOCK_PREFIX) +template +static inline P cmpxchg_local(P *ptr, P old, N rep) +{ + P ret = old; + try_cmpxchg(ptr, ret, rep, __lock_local); + return ret; +} -#define __sync_cmpxchg(ptr, old, new, size) \ - __raw_cmpxchg((ptr), (old), (new), (size), "lock; ") +template +static inline P cmpxchg(P *ptr, P old, N rep) +{ + P ret = old; + try_cmpxchg(ptr, ret, rep, __lock_smp); + return ret; +} -#define __cmpxchg_local(ptr, old, new, size) \ - __raw_cmpxchg((ptr), (old), (new), (size), "") +template +static inline P sync_cmpxchg(P *ptr, P old, N rep) +{ + P ret = old; + try_cmpxchg(ptr, ret, rep, __lock_always); + return ret; +} #ifdef CONFIG_X86_32 # include @@ -175,85 +180,6 @@ static inline P xadd(P *ptr, N inc) # include #endif -#define cmpxchg(ptr, old, new) \ - __cmpxchg(ptr, old, new, sizeof(*(ptr))) - -#define sync_cmpxchg(ptr, old, new) \ - __sync_cmpxchg(ptr, old, new, sizeof(*(ptr))) - -#define cmpxchg_local(ptr, old, new) \ - __cmpxchg_local(ptr, old, new, sizeof(*(ptr))) - - -#define __raw_try_cmpxchg(_ptr, _pold, _new, size, lock) \ -({ \ - bool success; \ - __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \ - __typeof__(*(_ptr)) __old = *_old; \ - __typeof__(*(_ptr)) __new = (_new); \ - switch (size) { \ - case __X86_CASE_B: \ - { \ - volatile u8 *__ptr = (volatile u8 *)(_ptr); \ - asm volatile(lock "cmpxchgb %[new], %[ptr]" \ - CC_SET(z) \ - : CC_OUT(z) (success), \ - [ptr] "+m" (*__ptr), \ - [old] "+a" (__old) \ - : [new] "q" (__new) \ - : "memory"); \ - break; \ - } \ - case __X86_CASE_W: \ - { \ - volatile u16 *__ptr = (volatile u16 *)(_ptr); \ - asm volatile(lock "cmpxchgw %[new], %[ptr]" \ - CC_SET(z) \ - : CC_OUT(z) (success), \ - [ptr] "+m" (*__ptr), \ - [old] "+a" (__old) \ - : [new] "r" (__new) \ - : "memory"); \ - break; \ - } \ - case __X86_CASE_L: \ - { \ - volatile u32 *__ptr = (volatile u32 *)(_ptr); \ - asm volatile(lock "cmpxchgl %[new], %[ptr]" \ - CC_SET(z) \ - : CC_OUT(z) (success), \ - [ptr] "+m" (*__ptr), \ - [old] "+a" (__old) \ - : [new] "r" (__new) \ - : "memory"); \ - break; \ - } \ - case __X86_CASE_Q: \ - { \ - volatile u64 *__ptr = (volatile u64 *)(_ptr); \ - asm volatile(lock "cmpxchgq %[new], %[ptr]" \ - CC_SET(z) \ - : CC_OUT(z) (success), \ - [ptr] "+m" (*__ptr), \ - [old] "+a" (__old) \ - : [new] "r" (__new) \ - : "memory"); \ - break; \ - } \ - default: \ - __cmpxchg_wrong_size(); \ - } \ - if (unlikely(!success)) \ - *_old = __old; \ - likely(success); \ -}) - -#define __try_cmpxchg(ptr, pold, new, size) \ - __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX) - -#define try_cmpxchg(ptr, pold, new) \ - __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr))) - #define __cmpxchg_double(pfx, p1, p2, o1, o2, n1, n2) \ ({ \ bool __ret; \