Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753431Ab3EFM77 (ORCPT ); Mon, 6 May 2013 08:59:59 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:20847 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752748Ab3EFM76 (ORCPT ); Mon, 6 May 2013 08:59:58 -0400 Date: Mon, 6 May 2013 08:59:37 -0400 From: Konrad Rzeszutek Wilk To: linux-kernel@vger.kernel.org, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com, x86@kernel.org, fenghua.yu@intel.com Cc: xen-devel@lists.xensource.com Subject: v3.9 - CPU hotplug and microcode earlier loading hits a mutex deadlock (x86_cpu_hotplug_driver_mutex) Message-ID: <20130506125937.GA14036@phenom.dumpdata.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: acsinet21.oracle.com [141.146.126.237] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4995 Lines: 114 Hey, As I was fixing the PV HVM's broken CPU hotplug mechanism I discovered a deadlock in the microcode and generic code. I am not sure if the ACPI generic mechanism would expose this, but looking at the flow (arch_register_cpu, then expecting user-space to call cpu_up), it should trigger this. Anyhow, this can easily be triggered if a new CPU is added and from user-space do: echo 1 > /sys/devices/system/cpu/cpu3/online on a newly appeared CPU. The deadlock is that the "store_online" in drivers/base/cpu.c takes the cpu_hotplug_driver_lock() lock, then calls "cpu_up". "cpu_up" eventually ends up calling "save_mc_for_early" which also takes the cpu_hotplug_driver_lock() lock. And here is that kernel thinks of it: smpboot: Stack at about ffff880075c39f44 smpboot: CPU3: has booted. microcode: CPU3 sig=0x206a7, pf=0x2, revision=0x25 ============================================= [ INFO: possible recursive locking detected ] 3.9.0upstream-10129-g167af0e #1 Not tainted --------------------------------------------- sh/2487 is trying to acquire lock: (x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [] cpu_hotplug_driver_lock+0x12/0x20 but task is already holding lock: (x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [] cpu_hotplug_driver_lock+0x12/0x20 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(x86_cpu_hotplug_driver_mutex); lock(x86_cpu_hotplug_driver_mutex); *** DEADLOCK *** May be due to missing lock nesting notation 6 locks held by sh/2487: #0: (sb_writers#5){.+.+.+}, at: [] vfs_write+0x17d/0x190 #1: (&buffer->mutex){+.+.+.}, at: [] sysfs_write_file+0x3f/0x160 #2: (s_active#20){.+.+.+}, at: [] sysfs_write_file+0xc8/0x160 #3: (x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [] cpu_hotplug_driver_lock+0x12/0x20 #4: (cpu_add_remove_lock){+.+.+.}, at: [] cpu_maps_update_begin+0x12/0x20 #5: (cpu_hotplug.lock){+.+.+.}, at: [] cpu_hotplug_begin+0x27/0x60 stack backtrace: CPU: 1 PID: 2487 Comm: sh Not tainted 3.9.0upstream-10129-g167af0e #1 Hardware name: Xen HVM domU, BIOS 4.3-unstable 05/03/2013 ffffffff8229b710 ffff880064e75538 ffffffff816fe47e ffff880064e75608 ffffffff81100cb6 ffff880064e75560 ffff88006670e290 ffff880064e75588 ffffffff00000000 ffff88006670e9b8 21710c800c83a1f5 ffff880064e75598 Call Trace: [] dump_stack+0x19/0x1b [] __lock_acquire+0x726/0x1890 [] ? is_module_text_address+0x22/0x40 [] ? __kernel_text_address+0x58/0x80 [] lock_acquire+0xaa/0x190 [] ? cpu_hotplug_driver_lock+0x12/0x20 [] __mutex_lock_common+0x5e/0x450 [] ? cpu_hotplug_driver_lock+0x12/0x20 [] ? sched_clock_local+0x25/0x90 [] ? cpu_hotplug_driver_lock+0x12/0x20 [] mutex_lock_nested+0x40/0x50 [] cpu_hotplug_driver_lock+0x12/0x20 [] save_mc_for_early+0x27/0xf0 [] ? mark_held_locks+0x90/0x150 [] ? get_page_from_freelist+0x46d/0x8e0 [] ? trace_hardirqs_on+0xd/0x10 [] ? sched_clock+0x9/0x10 [] ? __alloc_pages_nodemask+0x165/0xa30 [] ? sched_clock_local+0x25/0x90 [] ? sched_clock_cpu+0xb8/0x110 [] ? trace_hardirqs_off+0xd/0x10 [] ? vmap_page_range_noflush+0x279/0x370 [] ? map_vm_area+0x2d/0x50 [] ? __vmalloc_node_range+0x18e/0x260 [] ? generic_load_microcode+0xb8/0x1c0 [] generic_load_microcode+0x16c/0x1c0 [] ? generic_exec_single+0x7e/0xb0 [] ? request_microcode_user+0x20/0x20 [] request_microcode_fw+0x7f/0xa0 [] ? kobject_uevent+0xb/0x10 [] microcode_init_cpu+0xf4/0x110 [] mc_cpu_callback+0x5b/0xb3 [] notifier_call_chain+0x5c/0x120 [] __raw_notifier_call_chain+0x9/0x10 [] __cpu_notify+0x1b/0x30 [] _cpu_up+0x103/0x14b [] cpu_up+0xdb/0xee [] store_online+0xba/0x120 [] dev_attr_store+0x1b/0x20 [] sysfs_write_file+0xe1/0x160 [] vfs_write+0xdf/0x190 [] SyS_write+0x5d/0xa0 [] ? trace_hardirqs_on_thunk+0x3a/0x3f [] system_call_fastpath+0x16/0x1b Thoughts? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/