Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755411Ab1BNBZS (ORCPT ); Sun, 13 Feb 2011 20:25:18 -0500 Received: from mail4.hitachi.co.jp ([133.145.228.5]:52561 "EHLO mail4.hitachi.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755340Ab1BNBZL (ORCPT ); Sun, 13 Feb 2011 20:25:11 -0500 X-AuditID: b753bd60-a46baba000004916-ba-4d588474beaa X-AuditID: b753bd60-a46baba000004916-ba-4d588474beaa Message-ID: <4D588470.2040307@hitachi.com> Date: Mon, 14 Feb 2011 10:25:04 +0900 From: Masami Hiramatsu Organization: Systems Development Lab., Hitachi, Ltd., Japan User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Peter Zijlstra Cc: mingo@redhat.com, hpa@zytor.com, linux-kernel@vger.kernel.org, mathieu.desnoyers@efficios.com, rusty@rustcorp.com.au, ananth@in.ibm.com, fweisbec@gmail.com, rostedt@goodmis.org, jbeulich@novell.com, tglx@linutronix.de, jbaron@redhat.com, mhiramat@redhat.com, mingo@elte.hu, linux-tip-commits@vger.kernel.org, 2nddept-manager@sdl.hitachi.co.jp Subject: Re: [tip:perf/core] x86: Introduce text_poke_smp_batch() for batch-code modifying References: <20101203095422.2961.51217.stgit@ltc236.sdl.hitachi.co.jp> <1297458466.5226.93.camel@laptop> In-Reply-To: <1297458466.5226.93.camel@laptop> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: AAAAAA== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6018 Lines: 159 (2011/02/12 6:07), Peter Zijlstra wrote: > On Mon, 2010-12-06 at 18:17 +0000, tip-bot for Masami Hiramatsu wrote: > >> @@ -631,10 +639,13 @@ static int __kprobes stop_machine_text_poke(void *data) >> void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len) >> { >> struct text_poke_params tpp; >> + struct text_poke_param p; >> >> - tpp.addr = addr; >> - tpp.opcode = opcode; >> - tpp.len = len; >> + p.addr = addr; >> + p.opcode = opcode; >> + p.len = len; >> + tpp.params = &p; >> + tpp.nparams = 1; >> atomic_set(&stop_machine_first, 1); >> wrote_text = 0; >> /* Use __stop_machine() because the caller already got online_cpus. */ > > ^^^^^^^^^ > >> @@ -642,6 +653,26 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len) >> return addr; >> } >> >> +/** >> + * text_poke_smp_batch - Update instructions on a live kernel on SMP >> + * @params: an array of text_poke parameters >> + * @n: the number of elements in params. >> + * >> + * Modify multi-byte instruction by using stop_machine() on SMP. Since the >> + * stop_machine() is heavy task, it is better to aggregate text_poke requests >> + * and do it once if possible. >> + * >> + * Note: Must be called under get_online_cpus() and text_mutex. >> + */ >> +void __kprobes text_poke_smp_batch(struct text_poke_param *params, int n) >> +{ >> + struct text_poke_params tpp = {.params = params, .nparams = n}; >> + >> + atomic_set(&stop_machine_first, 1); >> + wrote_text = 0; >> + stop_machine(stop_machine_text_poke, (void *)&tpp, NULL); >> +} > > ^^^^^^^^^^^^^^ Oops! Indeed, it's my mistake. Thank you for reporting & fixing! > > > --- > Subject: x86: Fix text_poke_smp_batch() deadlock > > ======================================================= > [ INFO: possible circular locking dependency detected ] > 2.6.38-rc4-test+ #1 > ------------------------------------------------------- > bash/1850 is trying to acquire lock: > (text_mutex){+.+.+.}, at: [] return_to_handler+0x0/0x2f > > but task is already holding lock: > (smp_alt){+.+...}, at: [] return_to_handler+0x0/0x2f > > which lock already depends on the new lock. > > > the existing dependency chain (in reverse order) is: > > -> #2 (smp_alt){+.+...}: > [] lock_acquire+0xcd/0xf8 > [] __mutex_lock_common+0x4c/0x339 > [] mutex_lock_nested+0x3e/0x43 > [] alternatives_smp_switch+0x77/0x1d8 > [] do_boot_cpu+0xd7/0x762 > [] native_cpu_up+0xe6/0x16a > [] _cpu_up+0x9d/0xee > [] cpu_up+0xd3/0xe7 > [] kernel_init+0xe8/0x20a > [] kernel_thread_helper+0x4/0x10 > > -> #1 (cpu_hotplug.lock){+.+.+.}: > [] lock_acquire+0xcd/0xf8 > [] __mutex_lock_common+0x4c/0x339 > [] mutex_lock_nested+0x3e/0x43 > [] get_online_cpus+0x41/0x55 > [] stop_machine+0x1e/0x3e > [] text_poke_smp_batch+0x3a/0x3c > [] arch_optimize_kprobes+0x10d/0x11c > [] kprobe_optimizer+0x152/0x222 > [] process_one_work+0x1d3/0x335 > [] worker_thread+0x104/0x1a4 > [] kthread+0x9d/0xa5 > [] kernel_thread_helper+0x4/0x10 > > -> #0 (text_mutex){+.+.+.}: > > > other info that might help us debug this: > > 6 locks held by bash/1850: > #0: (&buffer->mutex){+.+.+.}, at: [] return_to_handler+0x0/0x2f > #1: (s_active#75){.+.+.+}, at: [] return_to_handler+0x0/0x2f > #2: (x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [] return_to_handler+0x0/0x2f > #3: (cpu_add_remove_lock){+.+.+.}, at: [] return_to_handler+0x0/0x2f > #4: (cpu_hotplug.lock){+.+.+.}, at: [] return_to_handler+0x0/0x2f > #5: (smp_alt){+.+...}, at: [] return_to_handler+0x0/0x2f > > stack backtrace: > Pid: 1850, comm: bash Not tainted 2.6.38-rc4-test+ #1 > Call Trace: > > [] print_circular_bug+0xa8/0xb7 > [] mutex_lock_nested+0x3e/0x43 > [] alternatives_smp_unlock+0x3d/0x93 > [] alternatives_smp_switch+0x198/0x1d8 > [] native_cpu_die+0x65/0x95 > [] _cpu_down+0x13e/0x202 > [] sysfs_write_file+0x108/0x144 > [] vfs_write+0xac/0xff > [] sys_write+0x4a/0x6e > > Reported-by: Steven Rostedt > Tested-by: Steven Rostedt > Signed-off-by: Peter Zijlstra Acked-by: Masami Hiramatsu > --- > arch/x86/kernel/alternative.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c > index 1236085..7038b95 100644 > --- a/arch/x86/kernel/alternative.c > +++ b/arch/x86/kernel/alternative.c > @@ -671,7 +671,7 @@ void __kprobes text_poke_smp_batch(struct text_poke_param *params, int n) > > atomic_set(&stop_machine_first, 1); > wrote_text = 0; > - stop_machine(stop_machine_text_poke, (void *)&tpp, NULL); > + __stop_machine(stop_machine_text_poke, (void *)&tpp, NULL); > } > > #if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL) > -- Masami HIRAMATSU 2nd Dept. Linux Technology Center Hitachi, Ltd., Systems Development Laboratory E-mail: masami.hiramatsu.pt@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/