Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751865AbdHCJKV (ORCPT ); Thu, 3 Aug 2017 05:10:21 -0400 Received: from mx2.suse.de ([195.135.220.15]:37373 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751739AbdHCJKR (ORCPT ); Thu, 3 Aug 2017 05:10:17 -0400 Date: Thu, 3 Aug 2017 11:10:14 +0200 (CEST) From: Jiri Kosina X-X-Sender: jkosina@pobox.suse.cz To: Zhang Rui , Eduardo Valentin , Kalle Valo , Sara Sharon , Johannes Berg , Emmanuel Grumbach , Luca Coelho cc: linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, Intel Linux Wireless Subject: Re: x86/thermal: AB-BA dependency between mvm->mutex and tz->lock In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (LSU 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 8008 Lines: 211 On Mon, 31 Jul 2017, Jiri Kosina wrote: > Hi, > > booting current Linus' tree, I'm seeing lockdep splat (see the end of this > mail). > > Apparently, there is AB-BA between tz->lock and mvm->mutex through the CPU > hotplug lock. > > The obivous depency is: thermal_zone_get_temp() acquires tz->lock, and > then calls iwl_mvm_tzone_get_temp() (through tz->ops->get_temp() > callback), which acquires mvm->mutex > > The less obvious dependency is primarily caused by iwl_op_mode_mvm_start() > allocating workqueue (#2 stacktrace) while holding mvm->mutex (which is > broken, because that mutex is being taken also from CPU hotplug callback > path, hence the AB-BA). As the "central" part of the dependency is being added by iwlwifi driver (_iwl_pcie_rx_init() allocating workqueue while holding trans_pcie->mutex), I'm adding iwlwifi folks as well to CC. > > ====================================================== > WARNING: possible circular locking dependency detected > 4.13.0-rc2-00110-g0b5477d #347 Not tainted > ------------------------------------------------------ > modprobe/881 is trying to acquire lock: > (&mvm->mutex){+.+.+.}, at: [] iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm] > > but task is already holding lock: > (&tz->lock){+.+.+.}, at: [] thermal_zone_get_temp+0x41/0x70 > > which lock already depends on the new lock. > > > the existing dependency chain (in reverse order) is: > > -> #5 (&tz->lock){+.+.+.}: > lock_acquire+0xbd/0x220 > __mutex_lock+0x6e/0x900 > mutex_lock_nested+0x1b/0x20 > thermal_zone_get_temp+0x41/0x70 > thermal_zone_device_update+0x3c/0x280 > thermal_zone_device_register+0x3b8/0x610 > pkg_thermal_cpu_online+0x20b/0x284 [x86_pkg_temp_thermal] > cpuhp_invoke_callback+0xac/0x900 > cpuhp_thread_fun+0x79/0x160 > smpboot_thread_fn+0x156/0x220 > kthread+0x114/0x150 > ret_from_fork+0x2a/0x40 > > -> #4 (cpuhp_state){+.+.+.}: > lock_acquire+0xbd/0x220 > cpuhp_issue_call+0xea/0x170 > __cpuhp_setup_state_cpuslocked+0x12a/0x190 > __cpuhp_setup_state+0x46/0xc0 > page_writeback_init+0x43/0x67 > pagecache_init+0x39/0x3c > start_kernel+0x45a/0x4ae > x86_64_start_reservations+0x24/0x26 > x86_64_start_kernel+0x13d/0x14c > verify_cpu+0x0/0xf1 > > -> #3 (cpuhp_state_mutex){+.+.+.}: > lock_acquire+0xbd/0x220 > __mutex_lock+0x6e/0x900 > mutex_lock_nested+0x1b/0x20 > __cpuhp_setup_state_cpuslocked+0x4f/0x190 > __cpuhp_setup_state+0x46/0xc0 > page_alloc_init+0x28/0x30 > start_kernel+0x186/0x4ae > x86_64_start_reservations+0x24/0x26 > x86_64_start_kernel+0x13d/0x14c > verify_cpu+0x0/0xf1 > > -> #2 (cpu_hotplug_lock.rw_sem){++++++}: > lock_acquire+0xbd/0x220 > cpus_read_lock+0x46/0x90 > apply_workqueue_attrs+0x17/0x50 > __alloc_workqueue_key+0x195/0x4d0 > _iwl_pcie_rx_init+0x384/0x390 [iwlwifi] > iwl_pcie_rx_init+0x1e/0x380 [iwlwifi] > iwl_trans_pcie_start_fw+0x295/0x6f0 [iwlwifi] > iwl_mvm_load_ucode_wait_alive+0xe7/0x390 [iwlmvm] > iwl_run_init_mvm_ucode+0x84/0x320 [iwlmvm] > iwl_op_mode_mvm_start+0x964/0xd30 [iwlmvm] > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi] > iwl_opmode_register+0xaa/0xd0 [iwlwifi] > iwl_mvm_init+0x37/0x1000 [iwlmvm] > do_one_initcall+0x51/0x1a9 > do_init_module+0x60/0x20e > load_module+0x203f/0x2b50 > SYSC_finit_module+0x96/0xd0 > SyS_finit_module+0xe/0x10 > entry_SYSCALL_64_fastpath+0x23/0xc2 > > -> #1 (&trans_pcie->mutex){+.+.+.}: > lock_acquire+0xbd/0x220 > __mutex_lock+0x6e/0x900 > mutex_lock_nested+0x1b/0x20 > iwl_trans_pcie_start_fw+0x130/0x6f0 [iwlwifi] > iwl_mvm_load_ucode_wait_alive+0xe7/0x390 [iwlmvm] > iwl_run_init_mvm_ucode+0x84/0x320 [iwlmvm] > iwl_op_mode_mvm_start+0x964/0xd30 [iwlmvm] > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi] > iwl_opmode_register+0xaa/0xd0 [iwlwifi] > iwl_mvm_init+0x37/0x1000 [iwlmvm] > do_one_initcall+0x51/0x1a9 > do_init_module+0x60/0x20e > load_module+0x203f/0x2b50 > SYSC_finit_module+0x96/0xd0 > SyS_finit_module+0xe/0x10 > entry_SYSCALL_64_fastpath+0x23/0xc2 > > -> #0 (&mvm->mutex){+.+.+.}: > __lock_acquire+0x13e1/0x1400 > lock_acquire+0xbd/0x220 > __mutex_lock+0x6e/0x900 > mutex_lock_nested+0x1b/0x20 > iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm] > thermal_zone_get_temp+0x51/0x70 > thermal_zone_device_update+0x3c/0x280 > thermal_zone_device_register+0x3b8/0x610 > iwl_mvm_thermal_initialize+0x1d1/0x3a0 [iwlmvm] > iwl_op_mode_mvm_start+0xa1d/0xd30 [iwlmvm] > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi] > iwl_opmode_register+0xaa/0xd0 [iwlwifi] > iwl_mvm_init+0x37/0x1000 [iwlmvm] > do_one_initcall+0x51/0x1a9 > do_init_module+0x60/0x20e > load_module+0x203f/0x2b50 > SYSC_finit_module+0x96/0xd0 > SyS_finit_module+0xe/0x10 > entry_SYSCALL_64_fastpath+0x23/0xc2 > > other info that might help us debug this: > > Chain exists of: > &mvm->mutex --> cpuhp_state --> &tz->lock > > Possible unsafe locking scenario: > > CPU0 CPU1 > ---- ---- > lock(&tz->lock); > lock(cpuhp_state); > lock(&tz->lock); > lock(&mvm->mutex); > > *** DEADLOCK *** > > 2 locks held by modprobe/881: > #0: (&iwlwifi_opmode_table_mtx){+.+.+.}, at: [] iwl_opmode_register+0x24/0xd0 [iwlwifi] > #1: (&tz->lock){+.+.+.}, at: [] thermal_zone_get_temp+0x41/0x70 > > stack backtrace: > CPU: 3 PID: 881 Comm: modprobe Not tainted 4.13.0-rc2-00110-g0b5477d #347 > Hardware name: LENOVO 20K5S22R00/20K5S22R00, BIOS R0IET38W (1.16 ) 05/31/2017 > Call Trace: > dump_stack+0x85/0xc9 > print_circular_bug+0x1f9/0x207 > __lock_acquire+0x13e1/0x1400 > lock_acquire+0xbd/0x220 > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm] > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm] > __mutex_lock+0x6e/0x900 > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm] > ? thermal_zone_get_temp+0x41/0x70 > ? iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm] > ? thermal_zone_get_temp+0x41/0x70 > ? find_held_lock+0x39/0xb0 > mutex_lock_nested+0x1b/0x20 > iwl_mvm_tzone_get_temp+0x32/0x80 [iwlmvm] > thermal_zone_get_temp+0x51/0x70 > thermal_zone_device_update+0x3c/0x280 > thermal_zone_device_register+0x3b8/0x610 > iwl_mvm_thermal_initialize+0x1d1/0x3a0 [iwlmvm] > iwl_op_mode_mvm_start+0xa1d/0xd30 [iwlmvm] > _iwl_op_mode_start.isra.9+0x47/0xa0 [iwlwifi] > iwl_opmode_register+0xaa/0xd0 [iwlwifi] > iwl_mvm_init+0x37/0x1000 [iwlmvm] > ? 0xffffffffc0c87000 > do_one_initcall+0x51/0x1a9 > ? rcu_read_lock_sched_held+0x98/0xa0 > ? kmem_cache_alloc_trace+0x2a5/0x340 > do_init_module+0x60/0x20e > load_module+0x203f/0x2b50 > ? __symbol_put+0x50/0x50 > SYSC_finit_module+0x96/0xd0 > SyS_finit_module+0xe/0x10 > entry_SYSCALL_64_fastpath+0x23/0xc2 > RIP: 0033:0x7f2ef067cc89 > RSP: 002b:00007ffea2ea3d78 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 > RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2ef067cc89 > RDX: 0000000000000000 RSI: 000000000041af06 RDI: 0000000000000001 > RBP: 0000000000000005 R08: 0000000000000000 R09: 000000000096b230 > R10: 0000000000000001 R11: 0000000000000246 R12: 00007ffea2ea2d80 > R13: 00007ffea2ea2d60 R14: 0000000000000005 R15: 000000000096f3c0 > thermal thermal_zone3: failed to read out thermal zone (-5) > > > -- > Jiri Kosina > SUSE Labs > -- Jiri Kosina SUSE Labs