Received: by 2002:a05:6a10:206:0:0:0:0 with SMTP id 6csp1307689pxj; Fri, 18 Jun 2021 04:24:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz0I7AdaBpzcJz7EickZ4t/RWNAVRBzmvVxxRHrdODfFAKVno9rEtu/ONBszHtSl0WeFzHL X-Received: by 2002:a17:906:d1d2:: with SMTP id bs18mr10720506ejb.56.1624015488057; Fri, 18 Jun 2021 04:24:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624015488; cv=none; d=google.com; s=arc-20160816; b=mFOLqoG5/yErfZ+P7mdgaZCuXbz9vJpa6E2xcydZhOKxOuJyvktJBY0wzr/7RtH4AC oPDHyU9jbeWyr2WgoKoysXFiyhkgtEXGcTJ2ZwbV4cDqPbxJti6FXdqNe6S27Gxfq3op QsVg00L8AMSg2hv6lkNS8K30Ocqol1PBY7daUp/GzS66/IlmjqZjUg2tICeSYYq+jEJ6 YaOTSPbOnQlqiSEHaql+g1JCBH8Rdrfjqq2LdHbwdlBUv0Fi3dmzydpj1wWzDY3CeqK1 wXAe9k8pGIDHQeqt5oX4MYxj0CtedDg7+Qi1DIadKqUfJzLjdEtAqXdTNMFKrCJFl6yF koaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=psHGJXq914MrLs3Mv2Jw6hM5r5nO7Qx/cP7bWL7Q104=; b=uM0i8HV6dSYNk8j3TClKO0lF0eyYpBoVOJoRGyZKnMk4/3vDTaQ/JocFYFBiWCJ9xO xAyoRecDDEkuF5q/W6ql9ICE9UGjEtDctOV4zDAmvOO5V0Vpf8OAKgdpjSABKj/7ZMeg n7LnoaSB4qnVqzf3RaOlUrSqJdMJ5MsEAZJs9iJ4fCM1yq3E8zkOVIujqyZQ4GVSNING BTehcnA62TggyBpcxEoULVa+nKoNg8KJDNYfk0ktdu/SrwIbEETasmZzfgrXgEtStzLS //hVa8LrTDXS/qA32khX1Pt7qmXxVu4lF0i1zq+L8dCWqjy/XcqX1hJ86DOwYmw2NK6d p3TQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=KnrLjb9Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id l20si2268807ejn.131.2021.06.18.04.24.24; Fri, 18 Jun 2021 04:24:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=KnrLjb9Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233058AbhFRHJm (ORCPT + 99 others); Fri, 18 Jun 2021 03:09:42 -0400 Received: from mail.kernel.org ([198.145.29.99]:46378 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229782AbhFRHJi (ORCPT ); Fri, 18 Jun 2021 03:09:38 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 2509661369; Fri, 18 Jun 2021 07:07:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1624000049; bh=WFXe7meLPgEvkoFkT5K9EfHGJUd1NjvkS/nTh9M7lic=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KnrLjb9Q9xngUpKdtZTyZD+zlXo3Z7DmUZvteAH6jsB1WJPznPK+PjBCKSSSfYY+q 4yxnM5om0/1YbiYTrG8eVhFhR9fJlWsvp5Mc2wY4QzaPNycycNYtwaPn+xfdVn2AdZ gSQt1WaR8uIK9AnXkGP6RrLjFhEX5JGuyzvY1FhJkjgZaVPV1tPRAabmWEj16j9aWd pqmqCQ2QktHb0YIQwc1M0te2jT+WNK/9+k0WDNqTMN08zFOUZRxJNyAvuyAX+H8Ym6 WCM6n7Q+4yHoUamSXznQeiE5yDkh92bio2NIhnh+Mx+O52BQ9iHY3XIHVPjsZrw/c7 7uI7OKQPs8Akw== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 13/13] x86/kprobes: Fixup return address in generic trampoline handler Date: Fri, 18 Jun 2021 16:07:25 +0900 Message-Id: <162400004562.506599.7549585083316952768.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In x86, kretprobe trampoline address on the stack frame will be replaced with the real return address after returning from trampoline_handler. Before fixing the return address, the real return address can be found in the current->kretprobe_instances. However, since there is a window between updating the current->kretprobe_instances and fixing the address on the stack, if an interrupt caused at that timing and the interrupt handler does stacktrace, it may fail to unwind because it can not get the correct return address from current->kretprobe_instances. This will minimize that window by fixing the return address right before updating current->kretprobe_instances. Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryik --- Changes in v7: - Add a prototype for arch_kretprobe_fixup_return() --- arch/x86/kernel/kprobes/core.c | 15 +++++++++++++-- include/linux/kprobes.h | 3 +++ kernel/kprobes.c | 8 ++++++++ 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 4d040aaf969b..53c1dcfcb145 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -1032,6 +1032,7 @@ STACK_FRAME_NON_STANDARD(kretprobe_trampoline); #undef UNWIND_HINT_FUNC #define UNWIND_HINT_FUNC #endif + /* * When a retprobed function returns, this code saves registers and * calls trampoline_handler() runs, which calls the kretprobe's handler. @@ -1073,6 +1074,17 @@ asm( ); NOKPROBE_SYMBOL(kretprobe_trampoline); +void arch_kretprobe_fixup_return(struct pt_regs *regs, + unsigned long correct_ret_addr) +{ + unsigned long *frame_pointer; + + frame_pointer = ((unsigned long *)®s->sp) + 1; + + /* Replace fake return address with real one. */ + *frame_pointer = correct_ret_addr; +} + /* * Called from kretprobe_trampoline */ @@ -1090,8 +1102,7 @@ __used __visible void trampoline_handler(struct pt_regs *regs) regs->sp += sizeof(long); frame_pointer = ((unsigned long *)®s->sp) + 1; - /* Replace fake return address with real one. */ - *frame_pointer = kretprobe_trampoline_handler(regs, frame_pointer); + kretprobe_trampoline_handler(regs, frame_pointer); /* * Move flags to sp so that kretprobe_trapmoline can return * right after popf. diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 08d3415e4418..259bdc80e708 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -197,6 +197,9 @@ extern void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs); extern int arch_trampoline_kprobe(struct kprobe *p); +void arch_kretprobe_fixup_return(struct pt_regs *regs, + unsigned long correct_ret_addr); + void kretprobe_trampoline(void); /* * Since some architecture uses structured function pointer, diff --git a/kernel/kprobes.c b/kernel/kprobes.c index ba729ed05cb3..72e8125fb0e9 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1881,6 +1881,12 @@ unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, } NOKPROBE_SYMBOL(kretprobe_find_ret_addr); +void __weak arch_kretprobe_fixup_return(struct pt_regs *regs, + unsigned long correct_ret_addr) +{ + /* Do nothing by default. */ +} + unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, void *frame_pointer) { @@ -1922,6 +1928,8 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, first = first->next; } + arch_kretprobe_fixup_return(regs, (unsigned long)correct_ret_addr); + /* Unlink all nodes for this frame. */ first = current->kretprobe_instances.first; current->kretprobe_instances.first = node->next;