Received: by 2002:a25:8b12:0:0:0:0:0 with SMTP id i18csp2575401ybl; Mon, 19 Aug 2019 04:25:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqz53VFvgq0TQbUw9AZoOdSs/yNmBVmLljgewVw4Bb7ttrH+nFvX6R/Lxv363jOF1d6LhKdR X-Received: by 2002:a17:902:8492:: with SMTP id c18mr22325355plo.279.1566213902879; Mon, 19 Aug 2019 04:25:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566213902; cv=none; d=google.com; s=arc-20160816; b=qLpx5/8Ayzg2OprzJXliTz1J4tuwYM99YR9Up8xFOoboKASjFSAIiL+BaF7WTz2wGU a7tLyNFFgkAeuRSkeaoU1w6eFDmouL0tql0Pdl+9Ho+y6eol2WsrC2BMZHEONjw5Impk xYY5MpSJq2OjDyrjasGSPgR2t8twrROHXeHQEJlYNHQZcjkLtywcxBsDoxu+wNix9Xr5 w60GDjvAkcv+L/qLdUO54jG6OgBR9deCTwx7hERESP+yR705F0X0ExZVrn/fBbgmTPO9 3IMVH6MViScaY9SGaNWGYyEBvzDZF2sdCt7AGXb8zWPYrZiRiW3Pnt2CSPa807StTB6E 4dKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-disposition :content-transfer-encoding:mime-version:robot-unsubscribe:robot-id :git-commit-id:subject:to:references:in-reply-to:reply-to:cc :message-id:from:date:dkim-signature:dkim-filter; bh=iTq0Qt1POXu8C2JzCGJoRFCzdRhsh147MBhk8KR+y5E=; b=cxSgb6PcI9m/aPX6EwntJQQuQ7i0+yHyUzR28X83w77Fg+6NT79HzqswWKpuvk++vx FQXDhmVtRWh7ediRYO+64KpRfoSSYsbTidNKP/nuC4lIWSZkjoDMs7S8+n+4n9UoYCyk Qe36a+7g2JcqHFv4MYXycoJ7XH4w3j5w8jvSmYxxFE49OPMBXnvE8/T6gpsGKal6p/5P FGAnSuRHiiecBxZHh+20BINuMXF/YEshoP/wCEVmEtO1FVtzQ6X8m0jurjVVDtqcYCT+ 8/hBguXQsO/2M2lpaaiLmIIUeWNwXAsfoR8rgljxpfLRDIIxGTsjbZLhAMTTzT4HFt44 e+4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zytor.com header.s=2019071901 header.b=BOlBzzoL; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=zytor.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h5si2276399pjs.91.2019.08.19.04.24.47; Mon, 19 Aug 2019 04:25:02 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@zytor.com header.s=2019071901 header.b=BOlBzzoL; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=zytor.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727550AbfHSLWi (ORCPT + 99 others); Mon, 19 Aug 2019 07:22:38 -0400 Received: from terminus.zytor.com ([198.137.202.136]:47867 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727537AbfHSLWf (ORCPT ); Mon, 19 Aug 2019 07:22:35 -0400 Received: from terminus.zytor.com (localhost [127.0.0.1]) by terminus.zytor.com (8.15.2/8.15.2) with ESMTPS id x7JBM7E94114244 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 19 Aug 2019 04:22:08 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 terminus.zytor.com x7JBM7E94114244 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2019071901; t=1566213728; bh=iTq0Qt1POXu8C2JzCGJoRFCzdRhsh147MBhk8KR+y5E=; h=Date:From:Cc:Reply-To:In-Reply-To:References:To:Subject:From; b=BOlBzzoLQFUDG6n3Hy3L4KChScm7xXu3Szjby9DMoq4ueaxptvkCPb21rH1HX3bmb Nkah1/lSMCsIF92m5wagOKr+n0VYyeZ6WlrugUpMfn9OaYmWd/sdZULwNtSy5nDVPb 6N4LmT4uWWj/iXZ6qZ9nH/AnOWPx+OsR9k4NDVYBIkAQRoVLZJThjVB875Lm4pg5QZ Fie4/JByQHe7124zulMgLgVkNzmuyi39LERNBBfHCzv7Q53WKrrCZjpzOSWMC7snSG 5DJNO1h+hpjxLc6QwugvpebhG4chXtQGFiN/47nwI0tPrLlHLxPFNK5pAIVM/yAL3i ZCtyM+8IiTLwA== Received: (from tipbot@localhost) by terminus.zytor.com (8.15.2/8.15.2/Submit) id x7JBM6ZI4114239; Mon, 19 Aug 2019 04:22:06 -0700 Date: Mon, 19 Aug 2019 04:22:06 -0700 X-Authentication-Warning: terminus.zytor.com: tipbot set sender to tipbot@zytor.com using -f From: tip-bot for Andrea Righi Message-ID: Cc: anil.s.keshavamurthy@intel.com, andrea.righi@canonical.com, naveen.n.rao@linux.ibm.com, hpa@zytor.com, tglx@linutronix.de, linux-kernel@vger.kernel.org, davem@davemloft.net, torvalds@linux-foundation.org, mhiramat@kernel.org, peterz@infradead.org, mingo@kernel.org Reply-To: hpa@zytor.com, tglx@linutronix.de, andrea.righi@canonical.com, naveen.n.rao@linux.ibm.com, anil.s.keshavamurthy@intel.com, peterz@infradead.org, torvalds@linux-foundation.org, mhiramat@kernel.org, mingo@kernel.org, linux-kernel@vger.kernel.org, davem@davemloft.net In-Reply-To: <20190812184302.GA7010@xps-13> References: <20190812184302.GA7010@xps-13> To: linux-tip-commits@vger.kernel.org Subject: [tip:perf/urgent] kprobes: Fix potential deadlock in kprobe_optimizer() Git-Commit-ID: f1c6ece23729257fb46562ff9224cf5f61b818da X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED,BAYES_00, DATE_IN_FUTURE_96_Q,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on terminus.zytor.com Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: f1c6ece23729257fb46562ff9224cf5f61b818da Gitweb: https://git.kernel.org/tip/f1c6ece23729257fb46562ff9224cf5f61b818da Author: Andrea Righi AuthorDate: Mon, 12 Aug 2019 20:43:02 +0200 Committer: Ingo Molnar CommitDate: Mon, 19 Aug 2019 12:22:19 +0200 kprobes: Fix potential deadlock in kprobe_optimizer() lockdep reports the following deadlock scenario: WARNING: possible circular locking dependency detected kworker/1:1/48 is trying to acquire lock: 000000008d7a62b2 (text_mutex){+.+.}, at: kprobe_optimizer+0x163/0x290 but task is already holding lock: 00000000850b5e2d (module_mutex){+.+.}, at: kprobe_optimizer+0x31/0x290 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (module_mutex){+.+.}: __mutex_lock+0xac/0x9f0 mutex_lock_nested+0x1b/0x20 set_all_modules_text_rw+0x22/0x90 ftrace_arch_code_modify_prepare+0x1c/0x20 ftrace_run_update_code+0xe/0x30 ftrace_startup_enable+0x2e/0x50 ftrace_startup+0xa7/0x100 register_ftrace_function+0x27/0x70 arm_kprobe+0xb3/0x130 enable_kprobe+0x83/0xa0 enable_trace_kprobe.part.0+0x2e/0x80 kprobe_register+0x6f/0xc0 perf_trace_event_init+0x16b/0x270 perf_kprobe_init+0xa7/0xe0 perf_kprobe_event_init+0x3e/0x70 perf_try_init_event+0x4a/0x140 perf_event_alloc+0x93a/0xde0 __do_sys_perf_event_open+0x19f/0xf30 __x64_sys_perf_event_open+0x20/0x30 do_syscall_64+0x65/0x1d0 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #0 (text_mutex){+.+.}: __lock_acquire+0xfcb/0x1b60 lock_acquire+0xca/0x1d0 __mutex_lock+0xac/0x9f0 mutex_lock_nested+0x1b/0x20 kprobe_optimizer+0x163/0x290 process_one_work+0x22b/0x560 worker_thread+0x50/0x3c0 kthread+0x112/0x150 ret_from_fork+0x3a/0x50 other info that might help us debug this: Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(module_mutex); lock(text_mutex); lock(module_mutex); lock(text_mutex); *** DEADLOCK *** As a reproducer I've been using bcc's funccount.py (https://github.com/iovisor/bcc/blob/master/tools/funccount.py), for example: # ./funccount.py '*interrupt*' That immediately triggers the lockdep splat. Fix by acquiring text_mutex before module_mutex in kprobe_optimizer(). Signed-off-by: Andrea Righi Acked-by: Masami Hiramatsu Cc: Anil S Keshavamurthy Cc: David S. Miller Cc: Linus Torvalds Cc: Naveen N. Rao Cc: Peter Zijlstra Cc: Thomas Gleixner Fixes: d5b844a2cf50 ("ftrace/x86: Remove possible deadlock between register_kprobe() and ftrace_run_update_code()") Link: http://lkml.kernel.org/r/20190812184302.GA7010@xps-13 Signed-off-by: Ingo Molnar --- kernel/kprobes.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 9873fc627d61..d9770a5393c8 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -470,6 +470,7 @@ static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer); */ static void do_optimize_kprobes(void) { + lockdep_assert_held(&text_mutex); /* * The optimization/unoptimization refers online_cpus via * stop_machine() and cpu-hotplug modifies online_cpus. @@ -487,9 +488,7 @@ static void do_optimize_kprobes(void) list_empty(&optimizing_list)) return; - mutex_lock(&text_mutex); arch_optimize_kprobes(&optimizing_list); - mutex_unlock(&text_mutex); } /* @@ -500,6 +499,7 @@ static void do_unoptimize_kprobes(void) { struct optimized_kprobe *op, *tmp; + lockdep_assert_held(&text_mutex); /* See comment in do_optimize_kprobes() */ lockdep_assert_cpus_held(); @@ -507,7 +507,6 @@ static void do_unoptimize_kprobes(void) if (list_empty(&unoptimizing_list)) return; - mutex_lock(&text_mutex); arch_unoptimize_kprobes(&unoptimizing_list, &freeing_list); /* Loop free_list for disarming */ list_for_each_entry_safe(op, tmp, &freeing_list, list) { @@ -524,7 +523,6 @@ static void do_unoptimize_kprobes(void) } else list_del_init(&op->list); } - mutex_unlock(&text_mutex); } /* Reclaim all kprobes on the free_list */ @@ -556,6 +554,7 @@ static void kprobe_optimizer(struct work_struct *work) { mutex_lock(&kprobe_mutex); cpus_read_lock(); + mutex_lock(&text_mutex); /* Lock modules while optimizing kprobes */ mutex_lock(&module_mutex); @@ -583,6 +582,7 @@ static void kprobe_optimizer(struct work_struct *work) do_free_cleaned_kprobes(); mutex_unlock(&module_mutex); + mutex_unlock(&text_mutex); cpus_read_unlock(); mutex_unlock(&kprobe_mutex);