Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp9227164imu; Wed, 5 Dec 2018 00:54:22 -0800 (PST) X-Google-Smtp-Source: AFSGD/UB0Wbf11Rq3u5mBsi8kZxNNFIQcr963wbceJDpg/fct97bgMgM67k6w5XbKbFIKr3rSIC9 X-Received: by 2002:a17:902:6a4:: with SMTP id 33mr15068554plh.99.1544000062713; Wed, 05 Dec 2018 00:54:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544000062; cv=none; d=google.com; s=arc-20160816; b=T3OE197g0LbUp80CUaeFwSIjO4bv2kwjH83WwKdf45tvlpreSFT7Y5WKjC+0eg5h0S czCaK2hDCtO8ZqRfo0c57dhDF/UsX/Z31anvds2abr1RcIMIqmNMdkUb6Q3oJNJB13UO Vffy8gq1KKdt7jWwUiEEk9wKSR241bHP0Y940iIbL157KaBRmuWWrIXjQoVmdctUpPno /pdvy+YITAxXHYVG55IXxr9MDtzlixx4K9fqJTUq5pWAZ+sbN3++9IA0tRZI8CaZU6QC LZY5kSKCOMd4d55gRBk4LM6jja0tWR6Yw/rGHEwF7hbU0nM2L1RERxMiV64IMf/ECfMr 6uLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=I2PTggDWJV64JfGMj5JfLIBQT9vU+91aj3vSXffnLkg=; b=lfhWlx/DNa3ICN3Qicq+UqB0x1R+1Z2NS3DGqCb9AHfav8ctg57AtlTcRJh7lt++DZ yh4PqTumE75ZTdDJYnEw36D0Pry3t0r8d9pGliGefOERZ0hasjJ/fU+9EvWbyWrlMGdo 27caTAf3oJycd0oVltrfANIRJMcLAmz1LoRqbgWEoW049XNN3WwNv8YYhJxs/pdGaumy 6quo/RLdwuEtbsziipC51bxDWqV2FNLXpRHrqCl5tZ2TrX6KtcMXNqPS9Hj1wG59/Atf /NNZRKkDPq9ZcZzJdHMGQL+tbIkcJMcYvWNwZ8F7biPn9YjF8ruZt79OHIhXMGwQ51oe Tyqw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=vmware.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h9si20653863plb.180.2018.12.05.00.54.08; Wed, 05 Dec 2018 00:54:22 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=vmware.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727623AbeLEIxF (ORCPT + 99 others); Wed, 5 Dec 2018 03:53:05 -0500 Received: from ex13-edg-ou-001.vmware.com ([208.91.0.189]:33316 "EHLO EX13-EDG-OU-001.vmware.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727297AbeLEIwK (ORCPT ); Wed, 5 Dec 2018 03:52:10 -0500 Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-001.vmware.com (10.113.208.155) with Microsoft SMTP Server id 15.0.1156.6; Wed, 5 Dec 2018 00:52:02 -0800 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 1AF6F41280; Wed, 5 Dec 2018 00:52:05 -0800 (PST) From: Nadav Amit To: Ingo Molnar CC: , , "H. Peter Anvin" , Thomas Gleixner , Borislav Petkov , Andy Lutomirski , Nadav Amit , Dave Hansen , Peter Zijlstra , , , , Nadav Amit , Steven Rostedt Subject: [PATCH v7 08/14] x86/ftrace: Use text_poke_*() infrastructure Date: Tue, 4 Dec 2018 17:34:02 -0800 Message-ID: <20181205013408.47725-9-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181205013408.47725-1-namit@vmware.com> References: <20181205013408.47725-1-namit@vmware.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: None (EX13-EDG-OU-001.vmware.com: namit@vmware.com does not designate permitted sender hosts) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org A following patch is going to make module allocated memory non-executable. This requires to modify ftrace and make the memory executable again after it is configured. In addition, this patch makes ftrace use the general text poking infrastructure instead ftrace's homegrown text patching. This provides the advantages of having slightly "safer" code patching and avoiding races with module removal or other mechanisms that patch the kernel code. Cc: Steven Rostedt Signed-off-by: Nadav Amit --- arch/x86/kernel/ftrace.c | 74 +++++++++++++--------------------------- 1 file changed, 23 insertions(+), 51 deletions(-) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 01ebcb6f263e..f05a0f9e2837 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -22,6 +22,7 @@ #include #include #include +#include #include @@ -29,23 +30,10 @@ #include #include #include +#include #ifdef CONFIG_DYNAMIC_FTRACE -int ftrace_arch_code_modify_prepare(void) -{ - set_kernel_text_rw(); - set_all_modules_text_rw(); - return 0; -} - -int ftrace_arch_code_modify_post_process(void) -{ - set_all_modules_text_ro(); - set_kernel_text_ro(); - return 0; -} - union ftrace_code_union { char code[MCOUNT_INSN_SIZE]; struct { @@ -79,22 +67,6 @@ within(unsigned long addr, unsigned long start, unsigned long end) return addr >= start && addr < end; } -static unsigned long text_ip_addr(unsigned long ip) -{ - /* - * On x86_64, kernel text mappings are mapped read-only, so we use - * the kernel identity mapping instead of the kernel text mapping - * to modify the kernel text. - * - * For 32bit kernels, these mappings are same and we can use - * kernel identity mapping to modify code. - */ - if (within(ip, (unsigned long)_text, (unsigned long)_etext)) - ip = (unsigned long)__va(__pa_symbol(ip)); - - return ip; -} - static const unsigned char *ftrace_nop_replace(void) { return ideal_nops[NOP_ATOMIC5]; @@ -124,13 +96,8 @@ ftrace_modify_code_direct(unsigned long ip, unsigned const char *old_code, if (memcmp(replaced, old_code, MCOUNT_INSN_SIZE) != 0) return -EINVAL; - ip = text_ip_addr(ip); - /* replace the text with the new text */ - if (probe_kernel_write((void *)ip, new_code, MCOUNT_INSN_SIZE)) - return -EPERM; - - sync_core(); + text_poke_early((void *)ip, new_code, MCOUNT_INSN_SIZE); return 0; } @@ -302,10 +269,7 @@ int ftrace_int3_handler(struct pt_regs *regs) static int ftrace_write(unsigned long ip, const char *val, int size) { - ip = text_ip_addr(ip); - - if (probe_kernel_write((void *)ip, val, size)) - return -EPERM; + text_poke((void *)ip, val, size); return 0; } @@ -653,9 +617,11 @@ void arch_ftrace_update_code(int command) { /* See comment above by declaration of modifying_ftrace_code */ atomic_inc(&modifying_ftrace_code); + mutex_lock(&text_mutex); ftrace_modify_all_code(command); + mutex_unlock(&text_mutex); atomic_dec(&modifying_ftrace_code); } @@ -741,6 +707,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) unsigned long end_offset; unsigned long op_offset; unsigned long offset; + unsigned long npages; unsigned long size; unsigned long ip; unsigned long *ptr; @@ -748,7 +715,6 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* 48 8b 15 is movq (%rip), %rdx */ unsigned const char op_ref[] = { 0x48, 0x8b, 0x15 }; union ftrace_op_code_union op_ptr; - int ret; if (ops->flags & FTRACE_OPS_FL_SAVE_REGS) { start_offset = (unsigned long)ftrace_regs_caller; @@ -772,19 +738,16 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) return 0; *tramp_size = size + MCOUNT_INSN_SIZE + sizeof(void *); + npages = DIV_ROUND_UP(*tramp_size, PAGE_SIZE); /* Copy ftrace_caller onto the trampoline memory */ - ret = probe_kernel_read(trampoline, (void *)start_offset, size); - if (WARN_ON(ret < 0)) { - tramp_free(trampoline, *tramp_size); - return 0; - } + text_poke_early(trampoline, (void *)start_offset, size); ip = (unsigned long)trampoline + size; /* The trampoline ends with a jmp to ftrace_epilogue */ jmp = ftrace_jmp_replace(ip, (unsigned long)ftrace_epilogue); - memcpy(trampoline + size, jmp, MCOUNT_INSN_SIZE); + text_poke_early(trampoline + size, jmp, MCOUNT_INSN_SIZE); /* * The address of the ftrace_ops that is used for this trampoline @@ -813,11 +776,19 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) op_ptr.offset = offset; /* put in the new offset to the ftrace_ops */ - memcpy(trampoline + op_offset, &op_ptr, OP_REF_SIZE); + text_poke_early(trampoline + op_offset, &op_ptr, OP_REF_SIZE); /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP; + set_memory_ro((unsigned long)trampoline, npages); + + /* + * TODO: Once we have better code (and page-table) protection + * mechanisms, ensure that the code has not been tampered before. + */ + set_memory_x((unsigned long)trampoline, npages); + return (unsigned long)trampoline; } @@ -853,8 +824,6 @@ void arch_ftrace_update_trampoline(struct ftrace_ops *ops) */ if (!(ops->flags & FTRACE_OPS_FL_ALLOC_TRAMP)) return; - npages = PAGE_ALIGN(ops->trampoline_size) >> PAGE_SHIFT; - set_memory_rw(ops->trampoline, npages); } else { ops->trampoline = create_trampoline(ops, &size); if (!ops->trampoline) @@ -863,6 +832,8 @@ void arch_ftrace_update_trampoline(struct ftrace_ops *ops) npages = PAGE_ALIGN(size) >> PAGE_SHIFT; } + mutex_lock(&text_mutex); + offset = calc_trampoline_call_offset(ops->flags & FTRACE_OPS_FL_SAVE_REGS); ip = ops->trampoline + offset; @@ -871,7 +842,8 @@ void arch_ftrace_update_trampoline(struct ftrace_ops *ops) /* Do a safe modify in case the trampoline is executing */ new = ftrace_call_replace(ip, (unsigned long)func); ret = update_ftrace_func(ip, new); - set_memory_ro(ops->trampoline, npages); + + mutex_unlock(&text_mutex); /* The update should never fail */ WARN_ON(ret); -- 2.17.1