Received: by 2002:a05:6a10:2726:0:0:0:0 with SMTP id ib38csp689511pxb; Tue, 5 Apr 2022 18:54:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzURhAwprGJlIpjJwnhVZY9G0//TKAQsoxAT/oj2M0tHvlDV6JlStk1CbMbryhnc6dnMjcb X-Received: by 2002:a17:90a:9517:b0:1c9:6124:126 with SMTP id t23-20020a17090a951700b001c961240126mr7298165pjo.63.1649210066722; Tue, 05 Apr 2022 18:54:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1649210066; cv=none; d=google.com; s=arc-20160816; b=QfhSo7DQewVUx/QeveIlGnxJy+pQb3I2Y1mbD5VI4EtPdY63LWSfb6cyRoKH0ZoVTq tkNe9JWYpiho4gZ1BrQAX95XeOtU//7IQ79E66hFjgyToAaXZy50SPV09jgEWE2+SCSo IjSgD2IQ/mHnNSh7zoP9hS5S1989t4yyhYJOcmU1FWbKb7Q/ndDBMFa7oatqLMFGwHUq cRhDOCJN9bwPliqPVI4CO7JD+l1whd8RtQ4AisjwMeD14Wj1KCprLvjA6NyS6QM3ADXM GZHH0FrpP4ERZscPApXVD/rgGD4HAeXai4zl7x9tvbqLax+ruKLD1lJHfFLLCL1Bs7Pw JUOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=J9zhYN1holvn/XnjEynBUsx1ga2h9On1DO2V2VAmi/A=; b=eoMQfpF614KrxFWb60kx7xSO1qQgGyMqnOH9ABsVMcbKAx+2J1tQlGv65/I0pY7CiI YngkxnXkwSNjGdYlJ7DDxZz5ngWRgPaRjlDzb8dxI+T+EJQXMtEFdi7W3xEhuWNSQIiX jfWTzKIot8f8mZ71e1B9PPtkCeHMvRawazDPuKmD4i14qWwQe0t5AkmooAomS8bD3vWR o1TC38QVGKyCuQBOnxdSL3KXXEUhr4UCgGEpmKTpDIj865vZFKhJ3bLQFU5oWQudcRGk JpY1QVu77T6oKpoSsrc5fey6jxwSiP9hdAr6novpCndcBd3cl9AXCofNPg4A3JyuKl6j VXBA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=DYK3wZYW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h17-20020a635311000000b003816043ef08si14794283pgb.253.2022.04.05.18.54.12; Tue, 05 Apr 2022 18:54:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=DYK3wZYW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233578AbiDEIJ4 (ORCPT + 99 others); Tue, 5 Apr 2022 04:09:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232572AbiDEHyJ (ORCPT ); Tue, 5 Apr 2022 03:54:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4844B31904; Tue, 5 Apr 2022 00:50:12 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9B963615C3; Tue, 5 Apr 2022 07:50:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A5607C340EE; Tue, 5 Apr 2022 07:50:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1649145011; bh=YNF+91Oul/8FbqE+JE40TuO4ibnBfixTTRe5SmoLFNg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DYK3wZYWvDvtY5isj52j3OApA06do2VwmPWuJuavouOJwBgT2PvDU/raQEOXCnTJC jsTJ0ZlbtAWP9dWZKvXg5wa6tDFS9bj6CtXCPYg6D8lW1pz+euAv4S1/Tbzv9DRG0l DEUn6H8V5bT4TdGjlx1X6Aqg01RNv0qSxlvI0bTQ= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Mark Rutland , Ard Biesheuvel , Catalin Marinas , James Morse , Marc Zyngier , Mark Brown , Will Deacon , Sasha Levin Subject: [PATCH 5.17 0251/1126] arm64: prevent instrumentation of bp hardening callbacks Date: Tue, 5 Apr 2022 09:16:38 +0200 Message-Id: <20220405070414.977447597@linuxfoundation.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220405070407.513532867@linuxfoundation.org> References: <20220405070407.513532867@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland [ Upstream commit 614c0b9fee711dd89b1dd65c88ba83612a373fdc ] We may call arm64_apply_bp_hardening() early during entry (e.g. in el0_ia()) before it is safe to run instrumented code. Unfortunately this may result in running instrumented code in two cases: * The hardening callbacks called by arm64_apply_bp_hardening() are not marked as `noinstr`, and have been observed to be instrumented when compiled with either GCC or LLVM. * Since arm64_apply_bp_hardening() itself is only marked as `inline` rather than `__always_inline`, it is possible that the compiler decides to place it out-of-line, whereupon it may be instrumented. For example, with defconfig built with clang 13.0.0, call_hvc_arch_workaround_1() is compiled as: | : | d503233f paciasp | f81f0ffe str x30, [sp, #-16]! | 320183e0 mov w0, #0x80008000 | d503201f nop | d4000002 hvc #0x0 | f84107fe ldr x30, [sp], #16 | d50323bf autiasp | d65f03c0 ret ... but when CONFIG_FTRACE=y and CONFIG_KCOV=y this is compiled as: | : | d503245f bti c | d503201f nop | d503201f nop | d503233f paciasp | a9bf7bfd stp x29, x30, [sp, #-16]! | 910003fd mov x29, sp | 94000000 bl 0 <__sanitizer_cov_trace_pc> | 320183e0 mov w0, #0x80008000 | d503201f nop | d4000002 hvc #0x0 | a8c17bfd ldp x29, x30, [sp], #16 | d50323bf autiasp | d65f03c0 ret ... with a patchable function entry registered with ftrace, and a direct call to __sanitizer_cov_trace_pc(). Neither of these are safe early during entry sequences. This patch avoids the unsafe instrumentation by marking arm64_apply_bp_hardening() as `__always_inline` and by marking the hardening functions as `noinstr`. This avoids the potential for instrumentation, and causes clang to consistently generate the function as with the defconfig sample. Note: in the defconfig compilation, when CONFIG_SVE=y, x30 is spilled to the stack without being placed in a frame record, which will result in a missing entry if call_hvc_arch_workaround_1() is backtraced. Similar is true of qcom_link_stack_sanitisation(), where inline asm spills the LR to a GPR prior to corrupting it. This is not a significant issue presently as we will only backtrace here if an exception is taken, and in such cases we may omit entries for other reasons today. The relevant hardening functions were introduced in commits: ec82b567a74fbdff ("arm64: Implement branch predictor hardening for Falkor") b092201e00206141 ("arm64: Add ARM_SMCCC_ARCH_WORKAROUND_1 BP hardening support") ... and these were subsequently moved in commit: d4647f0a2ad71110 ("arm64: Rewrite Spectre-v2 mitigation code") The arm64_apply_bp_hardening() function was introduced in commit: 0f15adbb2861ce6f ("arm64: Add skeleton to harden the branch predictor against aliasing attacks") ... and was subsequently moved and reworked in commit: 6279017e807708a0 ("KVM: arm64: Move BP hardening helpers into spectre.h") Fixes: ec82b567a74fbdff ("arm64: Implement branch predictor hardening for Falkor") Fixes: b092201e00206141 ("arm64: Add ARM_SMCCC_ARCH_WORKAROUND_1 BP hardening support") Fixes: d4647f0a2ad71110 ("arm64: Rewrite Spectre-v2 mitigation code") Fixes: 0f15adbb2861ce6f ("arm64: Add skeleton to harden the branch predictor against aliasing attacks") Fixes: 6279017e807708a0 ("KVM: arm64: Move BP hardening helpers into spectre.h") Signed-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Marc Zyngier Cc: Mark Brown Cc: Will Deacon Acked-by: Marc Zyngier Reviewed-by: Mark Brown Link: https://lore.kernel.org/r/20220224181028.512873-1-mark.rutland@arm.com Signed-off-by: Will Deacon Signed-off-by: Sasha Levin --- arch/arm64/include/asm/spectre.h | 3 ++- arch/arm64/kernel/proton-pack.c | 9 ++++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/spectre.h b/arch/arm64/include/asm/spectre.h index 86e0cc9b9c68..aa3d3607d5c8 100644 --- a/arch/arm64/include/asm/spectre.h +++ b/arch/arm64/include/asm/spectre.h @@ -67,7 +67,8 @@ struct bp_hardening_data { DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data); -static inline void arm64_apply_bp_hardening(void) +/* Called during entry so must be __always_inline */ +static __always_inline void arm64_apply_bp_hardening(void) { struct bp_hardening_data *d; diff --git a/arch/arm64/kernel/proton-pack.c b/arch/arm64/kernel/proton-pack.c index 6d45c63c6454..5777929d35bf 100644 --- a/arch/arm64/kernel/proton-pack.c +++ b/arch/arm64/kernel/proton-pack.c @@ -233,17 +233,20 @@ static void install_bp_hardening_cb(bp_hardening_cb_t fn) __this_cpu_write(bp_hardening_data.slot, HYP_VECTOR_SPECTRE_DIRECT); } -static void call_smc_arch_workaround_1(void) +/* Called during entry so must be noinstr */ +static noinstr void call_smc_arch_workaround_1(void) { arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_1, NULL); } -static void call_hvc_arch_workaround_1(void) +/* Called during entry so must be noinstr */ +static noinstr void call_hvc_arch_workaround_1(void) { arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_WORKAROUND_1, NULL); } -static void qcom_link_stack_sanitisation(void) +/* Called during entry so must be noinstr */ +static noinstr void qcom_link_stack_sanitisation(void) { u64 tmp; -- 2.34.1