Received: by 2002:ac0:a582:0:0:0:0:0 with SMTP id m2-v6csp181671imm; Fri, 5 Oct 2018 01:53:09 -0700 (PDT) X-Google-Smtp-Source: ACcGV62D6RfmDtOHmWB2WPs2pmKnUJAEXQqnb62nHW1iUY3cimlSkUgu9Ipv06GdPCkGG4tA7pZt X-Received: by 2002:a62:37c1:: with SMTP id e184-v6mr11028897pfa.112.1538729589613; Fri, 05 Oct 2018 01:53:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538729589; cv=none; d=google.com; s=arc-20160816; b=uSl3xgE9PKfXSPK7HU8QCa3SVLxDYskwW0T1xH9dHk7OS3INJZDKWbqBJIFAYFTMUF BC6buMvy8G21bAWxhxiVAYjiq7WS4MM/QLOKidvV65rclYbqPSIIOkBZ7ItVBzL+LhCd FRNfGfHGrgXLy+jy390M+PU6goaisq6djmDhOS/4faflfn3CwDrAVkSiih3di4vM/Wjs tt07Ivxqjp+JTLqJefXmRt+lAxof2xGKHO5uGOjSm8uHCNVSmNzMMyKQLARxA2LTeWan bs1kzkv3t51dBtAqkNU/WtDc4q0K6RXs1oYRax8/CrAK9Ctiq4U5xB1JbD2CCMboz/VN sPGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=C6x/TQUm9ZtEAE+LdWNDWRJqOYjlki5n7JYQdaR+hvY=; b=YBIAXqLIR/akTvZQFWtMyahnROhWRWyabjmp7Xv6U35siVgbhpAsgFIMxav05zPska 7B2jj8JZQvZuerdIp80yObd1kCPg53jrV4E4gZ4VWUEQ3m7ZwaOa6VEAdxQ9PwypFnTM NOG+ArMhOlDs2RTp+f/TcMWglg1UZOgEJmRzyv+TOE+eXOY8UIL7ZV5eU55lqYMDY4PV Be8JIWab8wNB6FaEFLSQ2+FqwCdzb4tvqe/uwgME3fELJjnRtoiVzf04AQVRSzY8Gjbb dYHE8iRKzJNnLPed3k1bkDN9rZ9KLRO6I/BuKD2BRL9e89oepYt3x5KhkocQevAYSKU3 op0A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y30-v6si7693678pgk.13.2018.10.05.01.52.53; Fri, 05 Oct 2018 01:53:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728986AbeJEPso (ORCPT + 99 others); Fri, 5 Oct 2018 11:48:44 -0400 Received: from foss.arm.com ([217.140.101.70]:48086 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728608AbeJEPsn (ORCPT ); Fri, 5 Oct 2018 11:48:43 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 54F6A1650; Fri, 5 Oct 2018 01:50:58 -0700 (PDT) Received: from moonbear.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 201433F5B3; Fri, 5 Oct 2018 01:50:54 -0700 (PDT) From: Kristina Martsenko To: linux-arm-kernel@lists.infradead.org Cc: Adam Wallis , Amit Kachhap , Andrew Jones , Ard Biesheuvel , Arnd Bergmann , Catalin Marinas , Christoffer Dall , Dave P Martin , Jacob Bramley , Kees Cook , Marc Zyngier , Mark Rutland , Ramana Radhakrishnan , "Suzuki K . Poulose" , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC 16/17] arm64: initialize and switch ptrauth kernel keys Date: Fri, 5 Oct 2018 09:47:53 +0100 Message-Id: <20181005084754.20950-17-kristina.martsenko@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20181005084754.20950-1-kristina.martsenko@arm.com> References: <20181005084754.20950-1-kristina.martsenko@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Set up keys to use pointer auth in the kernel. Each task has its own APIAKey, which is initialized during fork. The key is changed during context switch and on kernel entry from EL0. A function that changes the key cannot return, so inline such functions. Signed-off-by: Mark Rutland Signed-off-by: Kristina Martsenko --- arch/arm64/include/asm/pointer_auth.h | 9 ++++++++- arch/arm64/include/asm/ptrauth-asm.h | 13 +++++++++++++ arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/entry.S | 4 ++++ arch/arm64/kernel/process.c | 3 +++ arch/arm64/kernel/smp.c | 3 +++ 7 files changed, 33 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pointer_auth.h b/arch/arm64/include/asm/pointer_auth.h index 0634f06c3af2..e94ca7df8dab 100644 --- a/arch/arm64/include/asm/pointer_auth.h +++ b/arch/arm64/include/asm/pointer_auth.h @@ -50,12 +50,13 @@ do { \ write_sysreg_s(__pki_v.hi, SYS_ ## k ## KEYHI_EL1); \ } while (0) -static inline void ptrauth_keys_switch(struct ptrauth_keys *keys) +static __always_inline void ptrauth_keys_switch(struct ptrauth_keys *keys) { if (!cpus_have_const_cap(ARM64_HAS_ADDRESS_AUTH)) return; __ptrauth_key_install(APIA, keys->apia); + isb(); } static __always_inline void ptrauth_cpu_enable(void) @@ -85,11 +86,17 @@ static inline unsigned long ptrauth_strip_insn_pac(unsigned long ptr) #define ptrauth_task_init_user(tsk) \ ptrauth_keys_init(&(tsk)->thread_info.keys_user) +#define ptrauth_task_init_kernel(tsk) \ + ptrauth_keys_init(&(tsk)->thread_info.keys_kernel) +#define ptrauth_task_switch(tsk) \ + ptrauth_keys_switch(&(tsk)->thread_info.keys_kernel) #else /* CONFIG_ARM64_PTR_AUTH */ #define __no_ptrauth #define ptrauth_strip_insn_pac(lr) (lr) #define ptrauth_task_init_user(tsk) +#define ptrauth_task_init_kernel(tsk) +#define ptrauth_task_switch(tsk) #define ptrauth_cpu_enable(tsk) #endif /* CONFIG_ARM64_PTR_AUTH */ diff --git a/arch/arm64/include/asm/ptrauth-asm.h b/arch/arm64/include/asm/ptrauth-asm.h index f50bdfc4046c..3ef1cc8903d5 100644 --- a/arch/arm64/include/asm/ptrauth-asm.h +++ b/arch/arm64/include/asm/ptrauth-asm.h @@ -16,11 +16,24 @@ alternative_if ARM64_HAS_ADDRESS_AUTH alternative_else_nop_endif .endm + .macro ptrauth_keys_install_kernel tsk, tmp +alternative_if ARM64_HAS_ADDRESS_AUTH + ldr \tmp, [\tsk, #(TSK_TI_KEYS_KERNEL + PTRAUTH_KEY_APIALO)] + msr_s SYS_APIAKEYLO_EL1, \tmp + ldr \tmp, [\tsk, #(TSK_TI_KEYS_KERNEL + PTRAUTH_KEY_APIAHI)] + msr_s SYS_APIAKEYHI_EL1, \tmp + isb +alternative_else_nop_endif + .endm + #else /* CONFIG_ARM64_PTR_AUTH */ .macro ptrauth_keys_install_user tsk, tmp .endm + .macro ptrauth_keys_install_kernel tsk, tmp + .endm + #endif /* CONFIG_ARM64_PTR_AUTH */ #endif /* __ASM_PTRAUTH_ASM_H */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index ea9272fb52d4..e3ec5345addc 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -46,6 +46,7 @@ struct thread_info { int preempt_count; /* 0 => preemptable, <0 => bug */ #ifdef CONFIG_ARM64_PTR_AUTH struct ptrauth_keys keys_user; + struct ptrauth_keys keys_kernel; #endif }; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index b6be0dd037fd..6c61c9722b47 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -47,6 +47,7 @@ int main(void) #endif #ifdef CONFIG_ARM64_PTR_AUTH DEFINE(TSK_TI_KEYS_USER, offsetof(struct task_struct, thread_info.keys_user)); + DEFINE(TSK_TI_KEYS_KERNEL, offsetof(struct task_struct, thread_info.keys_kernel)); #endif DEFINE(TSK_STACK, offsetof(struct task_struct, stack)); BLANK(); diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 1e925f6d2978..a4503da445f7 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -250,6 +250,10 @@ alternative_else_nop_endif msr sp_el0, tsk .endif + .if \el == 0 + ptrauth_keys_install_kernel tsk, x20 + .endif + /* * Registers that may be useful after this macro is invoked: * diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 857ae05cd04c..a866996610de 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -330,6 +330,8 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start, */ fpsimd_flush_task_state(p); + ptrauth_task_init_kernel(p); + if (likely(!(p->flags & PF_KTHREAD))) { *childregs = *current_pt_regs(); childregs->regs[0] = 0; @@ -426,6 +428,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev, contextidr_thread_switch(next); entry_task_switch(next); uao_thread_switch(next); + ptrauth_task_switch(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 09690024dce8..d952dd62c780 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -212,6 +212,7 @@ asmlinkage notrace void secondary_start_kernel(void) */ check_local_cpu_capabilities(); + ptrauth_task_switch(current); ptrauth_cpu_enable(); if (cpu_ops[cpu]->cpu_postboot) @@ -418,6 +419,8 @@ void __init __no_ptrauth smp_prepare_boot_cpu(void) jump_label_init(); cpuinfo_store_boot_cpu(); + ptrauth_task_init_kernel(current); + ptrauth_task_switch(current); ptrauth_cpu_enable(); } -- 2.11.0