Received: by 2002:a25:1985:0:0:0:0:0 with SMTP id 127csp250151ybz; Tue, 21 Apr 2020 08:20:51 -0700 (PDT) X-Google-Smtp-Source: APiQypJhNG/J7WBiDalbzeF1MP6y6wzae9YgcPLcNSKZM4a+SS11cSm4mPXDKt6BW9YtTJWSCUx2 X-Received: by 2002:a17:906:b7da:: with SMTP id fy26mr22354080ejb.327.1587482451151; Tue, 21 Apr 2020 08:20:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587482451; cv=none; d=google.com; s=arc-20160816; b=di8ZmAVylmzUUJj7zGlWrMJfmABoi+XdEw3QRW51N2eBxRlKRv8R0DCer83FgsrVTk YtD3tgxVwmgTFoGhXKeeThbio8LM4z5H0SosZpcfXAi6u94S8hdrOozLWvnkXVRh1uVE x6qsyQsKX7kUfO/lbD/Fg/yVZcyGSlC3spCKkPGpxPxJxkwAfYaKceeLU/4crD6i6IWp 4HgMHzUUT/UBsxrfr8OoNSEBvT2uTS31LE9soBGKOY32ke4jhsQ7LOis7wzHVKzegB6u CcKueU7li8cgS7iLAIyc48cfnXeiwpBessB0oBCmADTu14XX7UxdJuFZvj8Vn2GHZ4qr qo3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=K78aOJfQiCgnuylBI1+JXEnr8c7X5JzHH+QhsSLjCxU=; b=HlsYUWu50IwCgfvfmu7YZB/itxZ6KUQyVv1HmmK4GmQzN2hwfi1AcgAP1auO71proQ gwwqDKCvnJJiVfoc/3S75I753n0oKI3YBfHQUXJ3lyrCjDFRkuNWCiWyRcDYeeu0JvLT eF9aIBhgyBu077nXiP1PBvCqR7O+FhCuSN0UeCkBy1KWHr6fkV7XnibCX6Lxr/OBajh9 7Z8Gyh3/0gDECgfZfD93Acu1DtbBabB40tpr2T0799eJLPZk4ZgkHWvJUeD8CLn8l0HW v9/08BgPJGbAmY/d+7LlRnI7jFmX3lZBxxL8Hb2CtDy/uQZw4nN8NAPoN4oCSknk7bX+ WGwg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=I5XWupvG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id s3si1906589ejs.302.2020.04.21.08.20.26; Tue, 21 Apr 2020 08:20:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=I5XWupvG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728473AbgDUPQN (ORCPT + 99 others); Tue, 21 Apr 2020 11:16:13 -0400 Received: from mail.kernel.org ([198.145.29.99]:35148 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728371AbgDUPQJ (ORCPT ); Tue, 21 Apr 2020 11:16:09 -0400 Received: from localhost.localdomain (236.31.169.217.in-addr.arpa [217.169.31.236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 849DD20724; Tue, 21 Apr 2020 15:16:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1587482169; bh=8fNQz1mwIal6Oqh/+LR51hNWYPT0sBjO+Q54FtI2Vvk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=I5XWupvGBhYFz463+rQtID9y2UoWBa7xQNiJga6LbbIoFhMVCvK6jgeaGwsVSOITX nmUdU14bBo7E5pUtpoJGryJM5L3QrnC1DQnsR4iJi+ykn/77G5vtp+KT97tUJsmwF+ v/xBeJ38BFa8KOuuQd/B6iYELibzNP45DvBaHE20= From: Will Deacon To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, kernel-team@android.com, Will Deacon , Mark Rutland , Michael Ellerman , Peter Zijlstra , Linus Torvalds , Segher Boessenkool , Christian Borntraeger , Luc Van Oostenryck , Arnd Bergmann , Peter Oberparleiter , Masahiro Yamada , Nick Desaulniers Subject: [PATCH v4 08/11] READ_ONCE: Drop pointer qualifiers when reading from scalar types Date: Tue, 21 Apr 2020 16:15:34 +0100 Message-Id: <20200421151537.19241-9-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200421151537.19241-1-will@kernel.org> References: <20200421151537.19241-1-will@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Passing a volatile-qualified pointer to READ_ONCE() is an absolute trainwreck for code generation: the use of 'typeof()' to define a temporary variable inside the macro means that the final evaluation in macro scope ends up forcing a read back from the stack. When stack protector is enabled (the default for arm64, at least), this causes the compiler to vomit up all sorts of junk. Unfortunately, dropping pointer qualifiers inside the macro poses quite a challenge, especially since the pointed-to type is permitted to be an aggregate, and this is relied upon by mm/ code accessing things like 'pmd_t'. Based on numerous hacks and discussions on the mailing list, this is the best I've managed to come up with. Introduce '__unqual_scalar_typeof()' which takes an expression and, if the expression is an optionally qualified 8, 16, 32 or 64-bit scalar type, evaluates to the unqualified type. Other input types, including aggregates, remain unchanged. Hopefully READ_ONCE() on volatile aggregate pointers isn't something we do on a fast-path. Cc: Peter Zijlstra Cc: Arnd Bergmann Suggested-by: Linus Torvalds Reported-by: Michael Ellerman Signed-off-by: Will Deacon --- include/linux/compiler.h | 6 +++--- include/linux/compiler_types.h | 21 +++++++++++++++++++++ 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/include/linux/compiler.h b/include/linux/compiler.h index 50bb2461648f..c363d8debc43 100644 --- a/include/linux/compiler.h +++ b/include/linux/compiler.h @@ -203,13 +203,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val, * atomicity or dependency ordering guarantees. Note that this may result * in tears! */ -#define __READ_ONCE(x) (*(const volatile typeof(x) *)&(x)) +#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x)) #define __READ_ONCE_SCALAR(x) \ ({ \ - typeof(x) __x = __READ_ONCE(x); \ + __unqual_scalar_typeof(x) __x = __READ_ONCE(x); \ smp_read_barrier_depends(); \ - __x; \ + (typeof(x))__x; \ }) #define READ_ONCE(x) \ diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index e970f97a7fcb..233066c92f6f 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -210,6 +210,27 @@ struct ftrace_likely_data { /* Are two types/vars the same type (ignoring qualifiers)? */ #define __same_type(a, b) __builtin_types_compatible_p(typeof(a), typeof(b)) +/* + * __unqual_scalar_typeof(x) - Declare an unqualified scalar type, leaving + * non-scalar types unchanged. + * + * We build this out of a couple of helper macros in a vain attempt to + * help you keep your lunch down while reading it. + */ +#define __pick_scalar_type(x, type, otherwise) \ + __builtin_choose_expr(__same_type(x, type), (type)0, otherwise) + +#define __pick_integer_type(x, type, otherwise) \ + __pick_scalar_type(x, unsigned type, \ + __pick_scalar_type(x, signed type, otherwise)) + +#define __unqual_scalar_typeof(x) typeof( \ + __pick_integer_type(x, char, \ + __pick_integer_type(x, short, \ + __pick_integer_type(x, int, \ + __pick_integer_type(x, long, \ + __pick_integer_type(x, long long, x)))))) + /* Is this type a native word size -- useful for atomic operations */ #define __native_word(t) \ (sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || \ -- 2.26.1.301.g55bc3eb7cb9-goog