Received: by 2002:a05:6a10:8395:0:0:0:0 with SMTP id n21csp610075pxh; Tue, 9 Nov 2021 16:08:31 -0800 (PST) X-Google-Smtp-Source: ABdhPJz24mTT+8TZoKUoxNrnwXHyJc8hKJaehnvSc2ad50HBoZI6AvN4h8U9kGxSyjxvd+izb7KC X-Received: by 2002:a92:7f0e:: with SMTP id a14mr8165235ild.215.1636502911344; Tue, 09 Nov 2021 16:08:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1636502911; cv=none; d=google.com; s=arc-20160816; b=0ZSdIKXGY6EF27+CiLizT7yVGE9Obg3fH7IhDfb3zbYqrWKtujVSc8zYDjP+b52JSf Go2N1cB1jiB0RLB1YgOD00jPlgsDGC3ez4LnutcOysAaE0kvdqoyZbyqiktCnrlyelKK 8n2LcKJhjoYr4E3Gn96QKOMBZe3aKqgAQkrLXSRbNn0jk5E4gwlzBl6XsA9Ucp6A4pap 6eqPRVkMYS2OXvKX2igWFR5s80eCDl2LimHJ150yTmIIvHeh7KgjiaFvnkmibwn0E0my miTzK8/XRhsZuWPjHP8TiSj60Z71PLS0tugRyPisVkirUzfFvEGlPCb4PcJC/MTEjNOc WqKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=m40p+Zg09jrou1nXj6JCS72BPa1wyD73g2+VWKrJgnk=; b=Aqrx21U2hdQMWaIXJGM7CRxNJNa9ca3o7CvjDhMBG8DbohJ6qGwVbAiwyQA0MW1p0S HpAxp/SocKZu14YUVaoyFqhKelQ0ZCGqw36PYnY3HLYqr9dHUh/YEEuaDtlqHvZZ/bUs FMbl8oeeUlYmjdXcKI/ytHJnAOdzSYqXAJ7gKwBJ/yRsP1+78t3e/pH4+vCsta9CxGZM zKHVHxrAZzPFluD1g6BYlot3BrKcZ0XDYI6BEQ50umnUrYwP3fYpsnm0UKqmq60++beY fMVXVU3mOYw8Tz1+m1QIGRz9lXQpJm2deXowGOgM5EkFJcmYdfyJz/GdS/M1A0DaIl6u Wa8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZGd5cLks; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id y5si44833229ill.62.2021.11.09.16.08.19; Tue, 09 Nov 2021 16:08:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZGd5cLks; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240395AbhKIQtJ (ORCPT + 97 others); Tue, 9 Nov 2021 11:49:09 -0500 Received: from mail.kernel.org ([198.145.29.99]:59894 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240237AbhKIQtE (ORCPT ); Tue, 9 Nov 2021 11:49:04 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 0B8E8610CF; Tue, 9 Nov 2021 16:46:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1636476378; bh=At1u3b+sYqBlXNyVUOqEsPvWFsVQ3X42m8iZpZWPAhg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZGd5cLks9EUe3zAi3JimnKMf6MRlw+jafWKpV36KAVE1sSZ4k+2cSOcNaV6d+TWuO QqQJWuRdAyoJ8qHEtbrBf6qQSfPESYUxQE/FC+zU9AXXv2WnOuLMINWgnARFqq6sOL s4LSz2w61FIQpG5rXishROE9i1o6Rj0u12lP5z01VBi2Q1kat2oAr3hmgTY0XiVJ9g Pj/WmkUaXpF0WNiYI1Y8ifuIsLwpsgs6lk0j/WYG3b7LTjOTum/JxgCGXraVuGrkM5 vhrotrfoaq5nXklf67YsEtDSopRi90R1dNJXOznAgjylJzNZouZZdFKubIaDvW/Np2 Hwg6NaDBY5FJg== From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Peter Zijlstra , Josh Poimboeuf , Jason Baron , Steven Rostedt , Mark Rutland , Kees Cook Subject: [RFC PATCH 4/7] static_call: fix broken static_call_query() for non-exported keys Date: Tue, 9 Nov 2021 17:45:46 +0100 Message-Id: <20211109164549.1724710-5-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211109164549.1724710-1-ardb@kernel.org> References: <20211109164549.1724710-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6201; h=from:subject; bh=At1u3b+sYqBlXNyVUOqEsPvWFsVQ3X42m8iZpZWPAhg=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhiqW236pCzHlHpC/JKEQCBJcnxNmBDaSOLbE42NWi XfPhW3WJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYYqltgAKCRDDTyI5ktmPJKkKDA CltCe4pjql1KgI3pIsdvxX2XQQsj0ncfZIrFcQl5ZzZ4YZukRfHkgKmqOcAhpHoC+O1gNJBp7PBQz6 QdWup52/U6TrP6oDcSXZeWWuXo7A8BrFDwFRlyvqU7yCRXosscLBgl5ik6ESm7fJmUNn+K1Q/U9tt5 KFO4+UN9KHeannGGbtNMjnHykhwW/FyE3So4oI1Q1CBhsI4++hL0vyQVmcxG97Ojmqs3mBIXYx+pAS RO+kXwtR0nMNENg+6zgcnrpeLU9sZoJmpihs8W+8Q651UElphwv+9EZ4G85IKUs58A1grr+SriPsBI WTHqlwx1e+j2PYqbzbdY/M8g66wxCDFQanLrvpW7WIdSYIct4IUzIn3GAI61eWAERFRaYSlOH5MLS9 Tv5LcSTHKFmx2Ey/uYUqblUBifu+6caRR5PiMLTKq8EDYr+Z03Mbi6fuAJym0YyOWPYiqu7irwF0aZ ebS2nV7KCRWUWK2v4FfAm+ZVCagnIZOvE3j8y+azUDs9U= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org static_call_query() accesses the func member of the static call key directly, which means that it is broken for cases where it is used from a module and the key resides elsewhere and is not exported. Let's add a helper that returns this value, and export it from the same module that the key resides in if the key itself is not exported. This way, we can always get the value regardless of whether the key is exported or not. Note that the non-NULL check of &STATIC_CALL_KEY(...) does not typically result in a load: for ISAs that support relocatable immediates, the address is patched into the instruction stream. For example, on ARM/Thumb2, we get 14a: f240 0300 movw r3, #0 14a: R_ARM_THM_MOVW_ABS_NC __SCK__pv_steal_clock 14e: f2c0 0300 movt r3, #0 14e: R_ARM_THM_MOVT_ABS __SCK__pv_steal_clock 152: b10b cbz r3, 158 154: 6818 ldr r0, [r3, #0] 156: 4770 bx lr 158: f7ff bffe b.w 0 <__SCQ__pv_steal_clock> 158: R_ARM_THM_JUMP24 __SCQ__pv_steal_clock I.e., it performs a conditional branch if zero (CBZ) on a quantity that was loaded using an MOVW/MOVT move-immediate pair, and either loads the func member directly, or tail calls __SCQ__pv_steal_clock if they key is not exported. Signed-off-by: Ard Biesheuvel --- include/linux/static_call.h | 9 ++++++++- include/linux/static_call_types.h | 18 +++++++++++++++++- tools/include/linux/static_call_types.h | 18 +++++++++++++++++- 3 files changed, 42 insertions(+), 3 deletions(-) diff --git a/include/linux/static_call.h b/include/linux/static_call.h index 3bba0bcba844..391f737496eb 100644 --- a/include/linux/static_call.h +++ b/include/linux/static_call.h @@ -149,7 +149,10 @@ extern void arch_static_call_transform(void *site, void *tramp, void *func, bool STATIC_CALL_TRAMP_ADDR(name), __F); \ }) -#define static_call_query(name) (READ_ONCE(STATIC_CALL_KEY(name).func)) +#define EXPORT_STATIC_CALL_QUERY(name, sfx) \ + typeof(STATIC_CALL_QUERY(name)()) STATIC_CALL_QUERY(name)(void) \ + { return STATIC_CALL_KEY(name).func; } \ + EXPORT_SYMBOL ## sfx (STATIC_CALL_QUERY(name)) #ifdef CONFIG_HAVE_STATIC_CALL_INLINE @@ -200,9 +203,11 @@ extern long __static_call_return0(void); /* Leave the key unexported, so modules can't change static call targets: */ #define EXPORT_STATIC_CALL_TRAMP(name) \ + EXPORT_STATIC_CALL_QUERY(name,); \ EXPORT_SYMBOL(STATIC_CALL_TRAMP(name)); \ EXPORT_STATIC_CALL_GETKEY_HELPER(name) #define EXPORT_STATIC_CALL_TRAMP_GPL(name) \ + EXPORT_STATIC_CALL_QUERY(name, _GPL); \ EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(name)); \ EXPORT_STATIC_CALL_GETKEY_HELPER(name) @@ -253,8 +258,10 @@ static inline long __static_call_return0(void) /* Leave the key unexported, so modules can't change static call targets: */ #define EXPORT_STATIC_CALL_TRAMP(name) \ + EXPORT_STATIC_CALL_QUERY(name,); \ EXPORT_SYMBOL(STATIC_CALL_TRAMP(name)) #define EXPORT_STATIC_CALL_TRAMP_GPL(name) \ + EXPORT_STATIC_CALL_QUERY(name, _GPL); \ EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(name)) #else /* Generic implementation */ diff --git a/include/linux/static_call_types.h b/include/linux/static_call_types.h index a31782909e43..2fce9aa8a995 100644 --- a/include/linux/static_call_types.h +++ b/include/linux/static_call_types.h @@ -23,6 +23,9 @@ #define STATIC_CALL_GETKEY_PREFIX_LEN (sizeof(STATIC_CALL_GETKEY_PREFIX_STR) - 1) #define STATIC_CALL_GETKEY(name) __PASTE(STATIC_CALL_GETKEY_PREFIX, name) +#define STATIC_CALL_QUERY_PREFIX __SCQ__ +#define STATIC_CALL_QUERY(name) __PASTE(STATIC_CALL_QUERY_PREFIX, name) + /* * Flags in the low bits of static_call_site::key. */ @@ -43,7 +46,20 @@ struct static_call_site { #define DECLARE_STATIC_CALL(name, func) \ extern __weak struct static_call_key STATIC_CALL_KEY(name); \ extern __weak struct static_call_key *STATIC_CALL_GETKEY(name)(void);\ - extern typeof(func) STATIC_CALL_TRAMP(name); + extern __weak typeof(func) *STATIC_CALL_QUERY(name)(void); \ + extern typeof(func) STATIC_CALL_TRAMP(name) + +#define __static_call_query(name) \ + ((typeof(STATIC_CALL_QUERY(name)()))READ_ONCE(STATIC_CALL_KEY(name).func)) + +#ifdef MODULE +/* the key might not be exported */ +#define static_call_query(name) \ + (&STATIC_CALL_KEY(name) ? __static_call_query(name) \ + : STATIC_CALL_QUERY(name)()) +#else +#define static_call_query(name) __static_call_query(name) +#endif #ifdef CONFIG_HAVE_STATIC_CALL diff --git a/tools/include/linux/static_call_types.h b/tools/include/linux/static_call_types.h index a31782909e43..2fce9aa8a995 100644 --- a/tools/include/linux/static_call_types.h +++ b/tools/include/linux/static_call_types.h @@ -23,6 +23,9 @@ #define STATIC_CALL_GETKEY_PREFIX_LEN (sizeof(STATIC_CALL_GETKEY_PREFIX_STR) - 1) #define STATIC_CALL_GETKEY(name) __PASTE(STATIC_CALL_GETKEY_PREFIX, name) +#define STATIC_CALL_QUERY_PREFIX __SCQ__ +#define STATIC_CALL_QUERY(name) __PASTE(STATIC_CALL_QUERY_PREFIX, name) + /* * Flags in the low bits of static_call_site::key. */ @@ -43,7 +46,20 @@ struct static_call_site { #define DECLARE_STATIC_CALL(name, func) \ extern __weak struct static_call_key STATIC_CALL_KEY(name); \ extern __weak struct static_call_key *STATIC_CALL_GETKEY(name)(void);\ - extern typeof(func) STATIC_CALL_TRAMP(name); + extern __weak typeof(func) *STATIC_CALL_QUERY(name)(void); \ + extern typeof(func) STATIC_CALL_TRAMP(name) + +#define __static_call_query(name) \ + ((typeof(STATIC_CALL_QUERY(name)()))READ_ONCE(STATIC_CALL_KEY(name).func)) + +#ifdef MODULE +/* the key might not be exported */ +#define static_call_query(name) \ + (&STATIC_CALL_KEY(name) ? __static_call_query(name) \ + : STATIC_CALL_QUERY(name)()) +#else +#define static_call_query(name) __static_call_query(name) +#endif #ifdef CONFIG_HAVE_STATIC_CALL -- 2.30.2