Following deadlock warning noticed on arm64 Qcom APQ 8016 SBC device with
a given config (kselftest merge configs) while booting.
This is almost always reproducible with a given config.
started happening from stable-rc v6.1.55.
Reported-by: Linux Kernel Functional Testing <[email protected]>
Boot log:
-------
[ 28.082133] ======================================================
[ 28.082673] WARNING: possible circular locking dependency detected
[ 28.088664] 6.1.55 #1 Not tainted
[ 28.094813] ------------------------------------------------------
[ 28.098211] (udev-worker)/278 is trying to acquire lock:
[ 28.104285] ffff80000b2b2d30 (rpm_smd_clk_lock){+.+.}-{3:3}, at:
clk_smd_rpm_set_rate (drivers/clk/qcom/clk-smd-rpm.c:321)
[ 28.109772]
[ 28.109772] but task is already holding lock:
[ 28.118525] ffff80000b49f3e0 (icc_bw_lock){+.+.}-{3:3}, at:
icc_node_add (drivers/interconnect/core.c:973)
[ 28.124175]
[ 28.124175] which lock already depends on the new lock.
[ 28.124175]
[ 28.131729]
[ 28.131729] the existing dependency chain (in reverse order) is:
[ 28.140058]
[ 28.140058] -> #2 (icc_bw_lock){+.+.}-{3:3}:
[ 28.147518] icc_init (drivers/interconnect/core.c:1159 (discriminator 1))
[ 28.153235] do_one_initcall (init/main.c:1292)
[ 28.156884] kernel_init_freeable (init/main.c:1364 (discriminator
3) init/main.c:1381 (discriminator 3) init/main.c:1400 (discriminator
3) init/main.c:1620 (discriminator 3))
[ 28.161227] kernel_init (init/main.c:1510)
[ 28.165910] ret_from_fork (arch/arm64/kernel/entry.S:861)
[ 28.170077]
[ 28.170077] -> #1 (fs_reclaim){+.+.}-{0:0}:
[ 28.174079] fs_reclaim_acquire (mm/page_alloc.c:4700 mm/page_alloc.c:4691)
[ 28.179890] __kmem_cache_alloc_node (include/linux/sched/mm.h:272
mm/slab.h:710 mm/slub.c:3318 mm/slub.c:3437)
[ 28.184492] __kmalloc (mm/slab_common.c:956 mm/slab_common.c:968)
[ 28.189434] qcom_rpm_smd_write (include/linux/slab.h:558
drivers/soc/qcom/smd-rpm.c:116)
[ 28.193084] clk_smd_rpm_prepare (drivers/clk/qcom/clk-smd-rpm.c:192
drivers/clk/qcom/clk-smd-rpm.c:253)
[ 28.197945] clk_core_prepare (drivers/clk/clk.c:971)
[ 28.202545] clk_prepare (drivers/clk/clk.c:1004
drivers/clk/clk.c:1027 drivers/clk/clk.c:1022)
[ 28.206881] amba_get_enable_pclk (include/linux/clk.h:1072
drivers/amba/bus.c:70)
[ 28.211052] amba_read_periphid (drivers/amba/bus.c:146)
[ 28.215653] amba_match (drivers/amba/bus.c:222)
[ 28.220336] __device_attach_driver (drivers/base/base.h:147
(discriminator 1) drivers/base/dd.c:913 (discriminator 1))
[ 28.223900] bus_for_each_drv (drivers/base/bus.c:427)
[ 28.229106] __device_attach (drivers/base/dd.c:1017)
[ 28.233359] device_initial_probe (drivers/base/dd.c:1065)
[ 28.237699] bus_probe_device (drivers/base/bus.c:489)
[ 28.242385] deferred_probe_work_func (drivers/base/dd.c:124)
[ 28.246729] process_one_work (kernel/workqueue.c:2297)
[ 28.251761] worker_thread (include/linux/list.h:292 (discriminator
2) kernel/workqueue.c:2352 (discriminator 2) kernel/workqueue.c:2444
(discriminator 2))
[ 28.256186] kthread (kernel/kthread.c:376)
[ 28.260697] ret_from_fork (arch/arm64/kernel/entry.S:861)
[ 28.264258]
[ 28.264258] -> #0 (rpm_smd_clk_lock){+.+.}-{3:3}:
[ 28.268351] __lock_acquire (kernel/locking/lockdep.c:3091
(discriminator 1) kernel/locking/lockdep.c:3209 (discriminator 1)
kernel/locking/lockdep.c:3824 (discriminator 1)
kernel/locking/lockdep.c:5048 (discriminator 1))
[ 28.274503] lock_acquire (arch/arm64/include/asm/percpu.h:40
kernel/locking/lockdep.c:467 kernel/locking/lockdep.c:5663
kernel/locking/lockdep.c:5626)
[ 28.278929] __mutex_lock (kernel/locking/mutex.c:605
kernel/locking/mutex.c:747)
[ 28.283095] mutex_lock_nested (kernel/locking/mutex.c:800)
[ 28.287263] clk_smd_rpm_set_rate (drivers/clk/qcom/clk-smd-rpm.c:321)
[ 28.291691] clk_change_rate (drivers/clk/clk.c:2278)
[ 28.296464] clk_core_set_rate_nolock (drivers/clk/clk.c:2382
drivers/clk/clk.c:2340)
[ 28.300895] clk_set_rate (drivers/clk/clk.c:2423 drivers/clk/clk.c:2410)
[ 28.306012] qcom_icc_set+0x1d4/0x270 icc_smd_rpm
[ 28.310186] icc_node_add (drivers/interconnect/core.c:994)
[ 28.315561] qnoc_probe+0x240/0x420 icc_smd_rpm
[ 28.319558] platform_probe (drivers/base/platform.c:1400)
[ 28.324675] really_probe (drivers/base/dd.c:560 drivers/base/dd.c:639)
[ 28.328841] __driver_probe_device (drivers/base/dd.c:785)
[ 28.333011] driver_probe_device (drivers/base/dd.c:815)
[ 28.337786] __driver_attach (drivers/base/dd.c:1202)
[ 28.342646] bus_for_each_dev (drivers/base/bus.c:301)
[ 28.346984] driver_attach (drivers/base/dd.c:1219)
[ 28.351411] bus_add_driver (drivers/base/bus.c:618)
[ 28.355404] driver_register (drivers/base/driver.c:246)
[ 28.359832] __platform_driver_register (drivers/base/platform.c:868)
[ 28.364177] msm8916_noc_driver_init+0x28/0x1000 qnoc_msm8916
[ 28.369217] do_one_initcall (init/main.c:1292)
[ 28.375804] do_init_module (kernel/module/main.c:2464)
[ 28.380144] load_module (kernel/module/main.c:2873)
[ 28.384485] __do_sys_finit_module (kernel/module/main.c:2973
(discriminator 1))
[ 28.388742] __arm64_sys_finit_module (kernel/module/main.c:2940)
[ 28.393342] invoke_syscall (arch/arm64/include/asm/current.h:19
arch/arm64/kernel/syscall.c:57)
[ 28.398546] el0_svc_common.constprop.0 (arch/arm64/kernel/syscall.c:149)
[ 28.402894] do_el0_svc (arch/arm64/kernel/syscall.c:207)
[ 28.408179] el0_svc (arch/arm64/include/asm/daifflags.h:28
arch/arm64/kernel/entry-common.c:133
arch/arm64/kernel/entry-common.c:142
arch/arm64/kernel/entry-common.c:638)
[ 28.411825] el0t_64_sync_handler (arch/arm64/kernel/entry-common.c:656)
[ 28.415651] el0t_64_sync (arch/arm64/kernel/entry.S:581)
[ 28.420248]
[ 28.420248] other info that might help us debug this:
[ 28.420248]
[ 28.424438] Chain exists of:
[ 28.424438] rpm_smd_clk_lock --> fs_reclaim --> icc_bw_lock
[ 28.424438]
[ 28.432432] Possible unsafe locking scenario:
[ 28.432432]
[ 28.442481] CPU0 CPU1
[ 28.448374] ---- ----
[ 28.452888] lock(icc_bw_lock);
Linux runner-vwmj3eza-project-40964107-concurrent-0 6.1.55 #1 SMP
PREEMPT @1695871287 aarch6[ 28.457396]
lock(fs_reclaim);
4
[ 28.475726] lock(rpm_smd_clk_lock);
[ 28.475770]
[ 28.475770] *** DEADLOCK ***
[ 28.475770]
[ 28.478630] 4 locks held by (udev-worker)/278:
[ 28.484350] #0: ffff000004e968f8 (&dev->mutex){....}-{3:3}, at:
__driver_attach (drivers/base/dd.c:1201)
[ 28.488885] #1: ffff80000b49f2e8 (icc_lock){+.+.}-{3:3}, at:
icc_node_add (drivers/interconnect/core.c:971)
[ 28.497475] #2: ffff80000b49f3e0 (icc_bw_lock){+.+.}-{3:3}, at:
icc_node_add (drivers/interconnect/core.c:973)
[ 28.505202] #3: ffff80000b23eeb8 (prepare_lock){+.+.}-{3:3}, at:
clk_prepare_lock (drivers/clk/clk.c:129 (discriminator 1))
[ 28.513363]
[ 28.513363] stack backtrace:
[ 28.521766] CPU: 3 PID: 278 Comm: (udev-worker) Not tainted 6.1.55 #1
[ 28.526028] Hardware name: Qualcomm Technologies, Inc. APQ 8016 SBC (DT)
[ 28.532453] Call trace:
[ 28.539202] dump_backtrace (arch/arm64/kernel/stacktrace.c:160
arch/arm64/kernel/stacktrace.c:143)
[ 28.541379] show_stack (arch/arm64/kernel/stacktrace.c:167)
[ 28.545194] dump_stack_lvl (lib/dump_stack.c:107)
[ 28.548582] dump_stack (lib/dump_stack.c:114)
[ 28.552225] print_circular_bug (kernel/locking/lockdep.c:2048)
[ 28.555529] check_noncircular (kernel/locking/lockdep.c:2173)
[ 28.559521] __lock_acquire (kernel/locking/lockdep.c:3091
(discriminator 1) kernel/locking/lockdep.c:3209 (discriminator 1)
kernel/locking/lockdep.c:3824 (discriminator 1)
kernel/locking/lockdep.c:5048 (discriminator 1))
[ 28.563687] lock_acquire (arch/arm64/include/asm/percpu.h:40
kernel/locking/lockdep.c:467 kernel/locking/lockdep.c:5663
kernel/locking/lockdep.c:5626)
[ 28.567767] __mutex_lock (kernel/locking/mutex.c:605
kernel/locking/mutex.c:747)
[ 28.571411] mutex_lock_nested (kernel/locking/mutex.c:800)
[ 28.575059] clk_smd_rpm_set_rate (drivers/clk/qcom/clk-smd-rpm.c:321)
[ 28.578968] clk_change_rate (drivers/clk/clk.c:2278)
[ 28.583220] clk_core_set_rate_nolock (drivers/clk/clk.c:2382
drivers/clk/clk.c:2340)
[ 28.587128] clk_set_rate (drivers/clk/clk.c:2423 drivers/clk/clk.c:2410)
[ 28.591724] qcom_icc_set+0x1d4/0x270 icc_smd_rpm
[ 28.595029] icc_node_add (drivers/interconnect/core.c:994)
[ 28.599884] qnoc_probe+0x240/0x420 icc_smd_rpm
[ 28.603709] platform_probe (drivers/base/platform.c:1400)
[ 28.608305] really_probe (drivers/base/dd.c:560 drivers/base/dd.c:639)
[ 28.611949] __driver_probe_device (drivers/base/dd.c:785)
[ 28.615599] driver_probe_device (drivers/base/dd.c:815)
[ 28.619851] __driver_attach (drivers/base/dd.c:1202)
[ 28.623844] bus_for_each_dev (drivers/base/bus.c:301)
[ 28.628010] driver_attach (drivers/base/dd.c:1219)
[ 28.631568] bus_add_driver (drivers/base/bus.c:618)
[ 28.635388] driver_register (drivers/base/driver.c:246)
[ 28.638948] __platform_driver_register (drivers/base/platform.c:868)
[ 28.642771] msm8916_noc_driver_init+0x28/0x1000 qnoc_msm8916
[ 28.647638] do_one_initcall (init/main.c:1292)
[ 28.653356] do_init_module (kernel/module/main.c:2464)
[ 28.657176] load_module (kernel/module/main.c:2873)
[ 28.660995] __do_sys_finit_module (kernel/module/main.c:2973
(discriminator 1))
[ 28.664732] __arm64_sys_finit_module (kernel/module/main.c:2940)
[ 28.669159] invoke_syscall (arch/arm64/include/asm/current.h:19
arch/arm64/kernel/syscall.c:57)
[ 28.673496] el0_svc_common.constprop.0 (arch/arm64/kernel/syscall.c:149)
[ 28.677321] do_el0_svc (arch/arm64/kernel/syscall.c:207)
[ 28.682087] el0_svc (arch/arm64/include/asm/daifflags.h:28
arch/arm64/kernel/entry-common.c:133
arch/arm64/kernel/entry-common.c:142
arch/arm64/kernel/entry-common.c:638)
[ 28.685558] el0t_64_sync_handler (arch/arm64/kernel/entry-common.c:656)
[ 28.688516] el0t_64_sync (arch/arm64/kernel/entry.S:581)
Links:
- https://lkft.validation.linaro.org/scheduler/job/6819688#L3326
- https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.1.y/build/v6.1.57-7-g3fe61dd155ac/testrun/20473398/suite/log-parser-test/tests/
metadata:
git_ref: linux-6.1.y
git_repo: https://gitlab.com/Linaro/lkft/mirrors/stable/linux-stable-rc
git_sha: d23900f974e0fb995b36ef47283a5aa74ca25f51
git_describe: v6.1.55
kernel_version: 6.1.55
kernel-config:
https://storage.tuxsuite.com/public/linaro/lkft/builds/2W0YTi5jyhwfHW8YVU86FdmCI85/config
artifact-location:
https://storage.tuxsuite.com/public/linaro/lkft/builds/2W0YTi5jyhwfHW8YVU86FdmCI85/
toolchain: gcc-13
--
Linaro LKFT
https://lkft.linaro.org