Received: by 2002:a05:6a10:413:0:0:0:0 with SMTP id 19csp2275321pxp; Mon, 21 Mar 2022 15:37:13 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz3hktJ9sf6fUt5JVAMfl6E8Zg41UT3/2C02Y2DIpk1CSQKkK24cKNC7oKdUVBqSSGOHNFT X-Received: by 2002:a17:902:d508:b0:154:2f99:a52e with SMTP id b8-20020a170902d50800b001542f99a52emr13164738plg.102.1647902233782; Mon, 21 Mar 2022 15:37:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647902233; cv=none; d=google.com; s=arc-20160816; b=Noy/HhDZRV8ks+wN8DBN6oSJaZGKSTZniFv8AiplETEzQ07QLWKKXYQ97M187ivfgp ZiHhxJN3iXXg8SbuWTVH0qSdRBrzvzt0ygYpeHC2ejIo2bOgN3SVVGCjjErdcXphlVtf kUnqPgktIwqk21JNf+vRWegWep0GQZXLiu6PjMHXsTgbtjQMI36BqOyiyZUx67ulao4I DiIvVNNvdvkN48hTdsqa7He5uehYvBlqH0OS5TqdyCy4iLgasOQZJnyElDN24YPeqz0l MykacMgHRVZEXnuMpy2Bgc1T42I2wklm2wuX8SCHjTx+ezCjOV72byFW7vh3ZfLbVu6o taJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=BWze4TYjVzjhRqyXvyuNuilP/KgRu+/ySgdeLf1w870=; b=QRsUWuEERgq+cyUwFApYn/HsirrNwKeCqdRZpqioYgFPzPqnb4PT6AhfRG1/aZbsri /8vwL38K+H/pCwykqNTuVm0kFjzjKKUiYb/H5FgoYeJlwO+mQsem3Tv+dhIhx+OyGxFL KluA6x9ToZqfRz48qVLm+xa36r1hu7hf4BdwXYqB+fXSb9CV/32b2Bne3S/nw7WiFQyo 664RDpD8irqZpzo5fdyQAX2SmSTEW/XHiYWu84uoooTzLGpEIhjgknsGjvHAs2xCAgB8 kYIp4+M7FBCQVSD1uSswA821VNhQIeJqLJcF/6LkZrfqq2rahwNnvL3gU5yE49hTS+KS bqpQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=MIqZKjfJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [2620:137:e000::1:18]) by mx.google.com with ESMTPS id kk18-20020a17090b4a1200b001bd14e030e1si559540pjb.185.2022.03.21.15.37.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Mar 2022 15:37:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18; Authentication-Results: mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=MIqZKjfJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id CC6FA3B6D92; Mon, 21 Mar 2022 14:48:39 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349274AbiCUODf (ORCPT + 99 others); Mon, 21 Mar 2022 10:03:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349069AbiCUN6z (ORCPT ); Mon, 21 Mar 2022 09:58:55 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60606764C; Mon, 21 Mar 2022 06:57:30 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F07BC6125C; Mon, 21 Mar 2022 13:57:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0716FC340E8; Mon, 21 Mar 2022 13:57:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1647871049; bh=vequwRd1eHlLFmd4IVJYLDGwGKy3Cw+YOfO+evt8qtc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MIqZKjfJ5rTD+xIN2tiyPWy1KBoUnKx+MvlY+a8sB+oyhAMOCfRN5lC2C+5EXiPY5 29GRZE6ThLTqh/M1O01sYTsURkISXzM/vbpGpuNfaTWR8FbiI8JERhPzUFsCIYr5kc RmPDlGZ71zyRqyYDfqVttgXTTey9QFArXrEaGjo0= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Greg Kroah-Hartman , Catalin Marinas , James Morse Subject: [PATCH 4.19 38/57] arm64: Add percpu vectors for EL1 Date: Mon, 21 Mar 2022 14:52:19 +0100 Message-Id: <20220321133223.098833153@linuxfoundation.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220321133221.984120927@linuxfoundation.org> References: <20220321133221.984120927@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: James Morse commit bd09128d16fac3c34b80bd6a29088ac632e8ce09 upstream. The Spectre-BHB workaround adds a firmware call to the vectors. This is needed on some CPUs, but not others. To avoid the unaffected CPU in a big/little pair from making the firmware call, create per cpu vectors. The per-cpu vectors only apply when returning from EL0. Systems using KPTI can use the canonical 'full-fat' vectors directly at EL1, the trampoline exit code will switch to this_cpu_vector on exit to EL0. Systems not using KPTI should always use this_cpu_vector. this_cpu_vector will point at a vector in tramp_vecs or __bp_harden_el1_vectors, depending on whether KPTI is in use. Reviewed-by: Catalin Marinas Signed-off-by: James Morse Signed-off-by: Greg Kroah-Hartman --- arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/include/asm/vectors.h | 27 +++++++++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ arch/arm64/kernel/entry.S | 16 ++++++++++------ arch/arm64/kvm/hyp/switch.c | 8 ++++++-- 5 files changed, 55 insertions(+), 9 deletions(-) --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -38,7 +38,7 @@ typedef struct { */ #define ASID(mm) ((mm)->context.id.counter & 0xffff) -static inline bool arm64_kernel_unmapped_at_el0(void) +static __always_inline bool arm64_kernel_unmapped_at_el0(void) { return IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0) && cpus_have_const_cap(ARM64_UNMAP_KERNEL_AT_EL0); --- a/arch/arm64/include/asm/vectors.h +++ b/arch/arm64/include/asm/vectors.h @@ -5,6 +5,15 @@ #ifndef __ASM_VECTORS_H #define __ASM_VECTORS_H +#include +#include + +#include + +extern char vectors[]; +extern char tramp_vectors[]; +extern char __bp_harden_el1_vectors[]; + /* * Note: the order of this enum corresponds to two arrays in entry.S: * tramp_vecs and __bp_harden_el1_vectors. By default the canonical @@ -31,4 +40,22 @@ enum arm64_bp_harden_el1_vectors { EL1_VECTOR_KPTI, }; +/* The vectors to use on return from EL0. e.g. to remap the kernel */ +DECLARE_PER_CPU_READ_MOSTLY(const char *, this_cpu_vector); + +#ifndef CONFIG_UNMAP_KERNEL_AT_EL0 +#define TRAMP_VALIAS 0 +#endif + +static inline const char * +arm64_get_bp_hardening_vector(enum arm64_bp_harden_el1_vectors slot) +{ + if (arm64_kernel_unmapped_at_el0()) + return (char *)TRAMP_VALIAS + SZ_2K * slot; + + WARN_ON_ONCE(slot == EL1_VECTOR_KPTI); + + return __bp_harden_el1_vectors + SZ_2K * slot; +} + #endif /* __ASM_VECTORS_H */ --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -20,11 +20,13 @@ #include #include +#include #include #include #include #include #include + #include #include #include @@ -33,6 +35,7 @@ #include #include #include +#include #include unsigned long elf_hwcap __read_mostly; @@ -51,6 +54,8 @@ unsigned int compat_elf_hwcap2 __read_mo DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS); EXPORT_SYMBOL(cpu_hwcaps); +DEFINE_PER_CPU_READ_MOSTLY(const char *, this_cpu_vector) = vectors; + /* * Flag to indicate if we have computed the system wide * capabilities based on the boot time active CPUs. This @@ -963,6 +968,12 @@ kpti_install_ng_mappings(const struct ar static bool kpti_applied = false; int cpu = smp_processor_id(); + if (__this_cpu_read(this_cpu_vector) == vectors) { + const char *v = arm64_get_bp_hardening_vector(EL1_VECTOR_KPTI); + + __this_cpu_write(this_cpu_vector, v); + } + if (kpti_applied) return; --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -71,7 +71,6 @@ .macro kernel_ventry, el, label, regsize = 64 .align 7 .Lventry_start\@: -#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 .if \el == 0 /* * This must be the first instruction of the EL0 vector entries. It is @@ -86,7 +85,6 @@ .endif .Lskip_tramp_vectors_cleanup\@: .endif -#endif sub sp, sp, #S_FRAME_SIZE #ifdef CONFIG_VMAP_STACK @@ -1033,10 +1031,14 @@ alternative_insn isb, nop, ARM64_WORKARO .endm .macro tramp_exit, regsize = 64 - adr x30, tramp_vectors -#ifdef CONFIG_MITIGATE_SPECTRE_BRANCH_HISTORY - add x30, x30, SZ_4K -#endif + tramp_data_read_var x30, this_cpu_vector +alternative_if_not ARM64_HAS_VIRT_HOST_EXTN + mrs x29, tpidr_el1 +alternative_else + mrs x29, tpidr_el2 +alternative_endif + ldr x30, [x30, x29] + msr vbar_el1, x30 ldr lr, [sp, #S_LR] tramp_unmap_kernel x29 @@ -1096,6 +1098,8 @@ __entry_tramp_data_vectors: __entry_tramp_data___sdei_asm_handler: .quad __sdei_asm_handler #endif /* CONFIG_ARM_SDE_INTERFACE */ +__entry_tramp_data_this_cpu_vector: + .quad this_cpu_vector .popsection // .rodata #endif /* CONFIG_RANDOMIZE_BASE */ #endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -34,6 +34,7 @@ #include #include #include +#include extern struct exception_table_entry __start___kvm_ex_table; extern struct exception_table_entry __stop___kvm_ex_table; @@ -155,10 +156,13 @@ static void __hyp_text __activate_traps( static void deactivate_traps_vhe(void) { - extern char vectors[]; /* kernel exception vectors */ + const char *host_vectors = vectors; write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2); write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1); - write_sysreg(vectors, vbar_el1); + + if (!arm64_kernel_unmapped_at_el0()) + host_vectors = __this_cpu_read(this_cpu_vector); + write_sysreg(host_vectors, vbar_el1); } NOKPROBE_SYMBOL(deactivate_traps_vhe);