Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754755AbaJVJxy (ORCPT ); Wed, 22 Oct 2014 05:53:54 -0400 Received: from cantor2.suse.de ([195.135.220.15]:35898 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752154AbaJVJxw (ORCPT ); Wed, 22 Oct 2014 05:53:52 -0400 Date: Wed, 22 Oct 2014 11:53:49 +0200 (CEST) From: Jiri Kosina To: Peter Zijlstra , Ingo Molnar , "Rafael J. Wysocki" , Pavel Machek , Steven Rostedt , Dave Jones , "Paul E. McKenney" , Daniel Lezcano , Nicolas Pitre cc: linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org Subject: Re: lockdep splat in CPU hotplug In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (LNX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 21 Oct 2014, Jiri Kosina wrote: > Hi, > > I am seeing the lockdep report below when resuming from suspend-to-disk > with current Linus' tree (c2661b80609). > > The reason for CCing Ingo and Peter is that I can't make any sense of one > of the stacktraces lockdep is providing. > > Please have a look at the very first stacktrace in the dump, where lockdep > is trying to explain where cpu_hotplug.lock#2 has been acquired. It seems > to imply that cpuidle_pause() is taking cpu_hotplug.lock, but that's not > the case at all. > > What am I missing? Okay, reverting 442bf3aaf55a ("sched: Let the scheduler see CPU idle states") and followup 83a0a96a5f26 ("sched/fair: Leverage the idle state info when choosing the "idlest" cpu") which depends on it makes the splat go away. Just for the sake of testing the hypothesis, I did just the minimal change below on top of current Linus' tree, and it also makes the splat go away (of course it's totally incorrect thing to do by itself alone): diff --git a/drivers/cpuidle/cpuidle.c b/drivers/cpuidle/cpuidle.c index 125150d..d31e04c 100644 --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -225,12 +225,6 @@ void cpuidle_uninstall_idle_handler(void) initialized = 0; wake_up_all_idle_cpus(); } - - /* - * Make sure external observers (such as the scheduler) - * are done looking at pointed idle states. - */ - synchronize_rcu(); } /** So indeed 442bf3aaf55a is guilty. Paul was stating yesterday that it can't be the try_get_online_cpus() in synchronize_sched_expedited(), as it's doing only trylock. There are however more places where synchronize_sched_expedited() is acquiring cpu_hotplug.lock unconditionally by calling put_online_cpus(), so the race seems real. Adding people involved in 442bf3aaf55a to CC. Still, the lockdep stacktrace is bogus and didn't really help understanding this. Any idea why it's wrong? > ====================================================== > [ INFO: possible circular locking dependency detected ] > 3.18.0-rc1-00069-gc2661b8 #1 Not tainted > ------------------------------------------------------- > do_s2disk/2367 is trying to acquire lock: > (cpuidle_lock){+.+.+.}, at: [] cpuidle_pause_and_lock+0x12/0x20 > > but task is already holding lock: > (cpu_hotplug.lock#2){+.+.+.}, at: [] cpu_hotplug_begin+0x4a/0x80 > > which lock already depends on the new lock. > > the existing dependency chain (in reverse order) is: > > -> #1 (cpu_hotplug.lock#2){+.+.+.}: > [] lock_acquire+0xac/0x130 > [] mutex_lock_nested+0x5c/0x3b0 > [] cpuidle_pause+0x12/0x30 > [] dpm_suspend_noirq+0x44/0x340 > [] dpm_suspend_end+0x38/0x80 > [] hibernation_snapshot+0xcd/0x370 > [] hibernate+0x168/0x210 > [] state_store+0xe4/0xf0 > [] kobj_attr_store+0xf/0x20 > [] sysfs_kf_write+0x43/0x60 > [] kernfs_fop_write+0xe7/0x170 > [] vfs_write+0xb2/0x1f0 > [] SyS_write+0x44/0xb0 > [] system_call_fastpath+0x16/0x1b > > -> #0 (cpuidle_lock){+.+.+.}: > [] __lock_acquire+0x1a03/0x1e30 > [] lock_acquire+0xac/0x130 > [] mutex_lock_nested+0x5c/0x3b0 > [] cpuidle_pause_and_lock+0x12/0x20 > [] acpi_processor_hotplug+0x45/0x8a [processor] > [] acpi_cpu_soft_notify+0xad/0xe3 [processor] > [] notifier_call_chain+0x53/0xa0 > [] __raw_notifier_call_chain+0x9/0x10 > [] cpu_notify+0x1e/0x40 > [] _cpu_up+0x148/0x160 > [] enable_nonboot_cpus+0xc9/0x1d0 > [] hibernation_snapshot+0x265/0x370 > [] hibernate+0x168/0x210 > [] state_store+0xe4/0xf0 > [] kobj_attr_store+0xf/0x20 > [] sysfs_kf_write+0x43/0x60 > [] kernfs_fop_write+0xe7/0x170 > [] vfs_write+0xb2/0x1f0 > [] SyS_write+0x44/0xb0 > [] system_call_fastpath+0x16/0x1b > > other info that might help us debug this: > > Possible unsafe locking scenario: > > CPU0 CPU1 > ---- ---- > lock(cpu_hotplug.lock#2); > lock(cpuidle_lock); > lock(cpu_hotplug.lock#2); > lock(cpuidle_lock); > > *** DEADLOCK *** > > 8 locks held by do_s2disk/2367: > #0: (sb_writers#6){.+.+.+}, at: [] vfs_write+0x1b3/0x1f0 > #1: (&of->mutex){+.+.+.}, at: [] kernfs_fop_write+0xbb/0x170 > #2: (s_active#188){.+.+.+}, at: [] kernfs_fop_write+0xc3/0x170 > #3: (pm_mutex){+.+.+.}, at: [] hibernate+0x4e/0x210 > #4: (device_hotplug_lock){+.+.+.}, at: [] lock_device_hotplug+0x12/0x20 > #5: (cpu_add_remove_lock){+.+.+.}, at: [] enable_nonboot_cpus+0x1f/0x1d0 > #6: (cpu_hotplug.lock){++++++}, at: [] cpu_hotplug_begin+0x0/0x80 > #7: (cpu_hotplug.lock#2){+.+.+.}, at: [] cpu_hotplug_begin+0x4a/0x80 > > stack backtrace: > CPU: 1 PID: 2367 Comm: do_s2disk Not tainted 3.18.0-rc1-00069-g4da0564 #1 > Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 > ffffffff823e4330 ffff8800789e7a48 ffffffff815b6754 0000000000001a69 > ffffffff823e4330 ffff8800789e7a98 ffffffff815b078b ffff8800741a5510 > ffff8800789e7af8 ffff8800741a5ea8 5a024e919538010b ffff8800741a5ea8 > Call Trace: > [] dump_stack+0x4e/0x68 > [] print_circular_bug+0x203/0x214 > [] __lock_acquire+0x1a03/0x1e30 > [] ? trace_hardirqs_on_caller+0xfd/0x1c0 > [] lock_acquire+0xac/0x130 > [] ? cpuidle_pause_and_lock+0x12/0x20 > [] mutex_lock_nested+0x5c/0x3b0 > [] ? cpuidle_pause_and_lock+0x12/0x20 > [] cpuidle_pause_and_lock+0x12/0x20 > [] acpi_processor_hotplug+0x45/0x8a [processor] > [] acpi_cpu_soft_notify+0xad/0xe3 [processor] > [] notifier_call_chain+0x53/0xa0 > [] __raw_notifier_call_chain+0x9/0x10 > [] cpu_notify+0x1e/0x40 > [] _cpu_up+0x148/0x160 > [] enable_nonboot_cpus+0xc9/0x1d0 > [] hibernation_snapshot+0x265/0x370 > [] hibernate+0x168/0x210 > [] state_store+0xe4/0xf0 > [] kobj_attr_store+0xf/0x20 > [] sysfs_kf_write+0x43/0x60 > [] kernfs_fop_write+0xe7/0x170 > [] vfs_write+0xb2/0x1f0 > [] ? sysret_check+0x1b/0x56 > [] SyS_write+0x44/0xb0 > [] system_call_fastpath+0x16/0x1b -- Jiri Kosina SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/