Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 86895e1e85b3a8220936304841b20d01a6239976 ("[PATCH 3/4] mm/memcg: Add a local_lock_t for IRQ and TASK object.")
url: https://github.com/0day-ci/linux/commits/Sebastian-Andrzej-Siewior/mm-memcg-Address-PREEMPT_RT-problems-instead-of-disabling-it/20220126-004438
patch link: https://lore.kernel.org/linux-mm/[email protected]
in testcase: kernel-selftests
version: kernel-selftests-x86_64-f050cde9-1_20220127
with following parameters:
group: cgroup
ucode: 0xe2
test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel.
test-url: https://www.kernel.org/doc/Documentation/kselftest.txt
on test machine: 8 threads Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz with 16G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
[ 698.190282][ T6895] WARNING: possible circular locking dependency detected
[ 698.197143][ T6895] 5.17.0-rc1-00003-g86895e1e85b3 #1 Not tainted
[ 698.203222][ T6895] ------------------------------------------------------
[ 698.210077][ T6895] with_stress.sh/6895 is trying to acquire lock:
[ 698.216247][ T6895] ffff888100b42428 (&mm->mmap_lock#2){++++}-{3:3}, at: mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384)
[ 698.225194][ T6895]
[ 698.225194][ T6895] but task is already holding lock:
[ 698.232394][ T6895] ffffffff8569b1f0 (&cpuset_rwsem){++++}-{0:0}, at: cpuset_attach (kernel/cgroup/cpuset.c:2257)
[ 698.240991][ T6895]
[ 698.240991][ T6895] which lock already depends on the new lock.
[ 698.240991][ T6895]
[ 698.251226][ T6895]
[ 698.251226][ T6895] the existing dependency chain (in reverse order) is:
[ 698.260076][ T6895]
[ 698.260076][ T6895] -> #2 (&cpuset_rwsem){++++}-{0:0}:
[ 698.267369][ T6895] lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604)
[ 698.272241][ T6895] percpu_down_write (kernel/locking/percpu-rwsem.c:220)
[ 698.277455][ T6895] cpuset_css_online (include/linux/instrumented.h:86 include/asm-generic/bitops/instrumented-atomic.h:28 kernel/cgroup/cpuset.c:2820)
[ 698.282667][ T6895] online_css (kernel/cgroup/cgroup.c:5250)
[ 698.287277][ T6895] cgroup_apply_control_enable (kernel/cgroup/cgroup.c:5321 kernel/cgroup/cgroup.c:3124)
[ 698.293448][ T6895] cgroup_apply_control (kernel/cgroup/cgroup.c:3207)
[ 698.298830][ T6895] cgroup_subtree_control_write (kernel/cgroup/cgroup.c:3232 kernel/cgroup/cgroup.c:3365)
[ 698.305080][ T6895] cgroup_file_write (kernel/cgroup/cgroup.c:3855)
[ 698.310378][ T6895] kernfs_fop_write_iter (fs/kernfs/file.c:300)
[ 698.316020][ T6895] new_sync_write (fs/read_write.c:504 (discriminator 1))
[ 698.321059][ T6895] vfs_write (fs/read_write.c:590)
[ 698.325667][ T6895] ksys_write (fs/read_write.c:643)
[ 698.330276][ T6895] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80)
[ 698.335065][ T6895] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113)
[ 698.341314][ T6895]
[ 698.341314][ T6895] -> #1 (cpu_hotplug_lock){++++}-{0:0}:
[ 698.348869][ T6895] lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604)
[ 698.353733][ T6895] cpus_read_lock (arch/x86/include/asm/preempt.h:80 include/linux/percpu-rwsem.h:53 kernel/cpu.c:309)
[ 698.358600][ T6895] drain_all_stock (mm/memcontrol.c:2207 mm/memcontrol.c:2193)
[ 698.363641][ T6895] try_charge_memcg (mm/memcontrol.c:2563)
[ 698.368941][ T6895] charge_memcg (mm/memcontrol.c:6766)
[ 698.373724][ T6895] __mem_cgroup_charge (include/linux/cgroup.h:403 mm/memcontrol.c:6787)
[ 698.379116][ T6895] do_anonymous_page (mm/memory.c:3783)
[ 698.384503][ T6895] __handle_mm_fault (mm/memory.c:4568 mm/memory.c:4705)
[ 698.389976][ T6895] handle_mm_fault (mm/memory.c:4803)
[ 698.395102][ T6895] do_user_addr_fault (arch/x86/mm/fault.c:1397)
[ 698.400486][ T6895] exc_page_fault (arch/x86/include/asm/irqflags.h:29 arch/x86/include/asm/irqflags.h:70 arch/x86/include/asm/irqflags.h:130 arch/x86/mm/fault.c:1492 arch/x86/mm/fault.c:1540)
[ 698.405440][ T6895] asm_exc_page_fault (arch/x86/include/asm/idtentry.h:568)
[ 698.410651][ T6895]
[ 698.410651][ T6895] -> #0 (&mm->mmap_lock#2){++++}-{3:3}:
[ 698.418210][ T6895] check_prev_add (kernel/locking/lockdep.c:3064)
[ 698.423341][ T6895] __lock_acquire (kernel/locking/lockdep.c:3187 kernel/locking/lockdep.c:3801 kernel/locking/lockdep.c:5027)
[ 698.428552][ T6895] lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604)
[ 698.433419][ T6895] down_write (include/linux/instrumented.h:101 include/linux/atomic/atomic-instrumented.h:1779 kernel/locking/rwsem.c:254 kernel/locking/rwsem.c:1258 kernel/locking/rwsem.c:1268 kernel/locking/rwsem.c:1515)
[ 698.438033][ T6895] mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384)
[ 698.442995][ T6895] cpuset_attach (include/linux/instrumented.h:71 include/asm-generic/bitops/instrumented-non-atomic.h:134 kernel/cgroup/cpuset.c:258 kernel/cgroup/cpuset.c:2293)
[ 698.447947][ T6895] cgroup_migrate_execute (kernel/cgroup/cgroup.c:2513)
[ 698.453764][ T6895] cgroup_update_dfl_csses (kernel/cgroup/cgroup.c:2975)
[ 698.459586][ T6895] cgroup_subtree_control_write (kernel/cgroup/cgroup.c:3232 kernel/cgroup/cgroup.c:3365)
[ 698.465835][ T6895] cgroup_file_write (kernel/cgroup/cgroup.c:3855)
[ 698.471130][ T6895] kernfs_fop_write_iter (fs/kernfs/file.c:300)
[ 698.476775][ T6895] new_sync_write (fs/read_write.c:504 (discriminator 1))
[ 698.481811][ T6895] vfs_write (fs/read_write.c:590)
[ 698.486417][ T6895] ksys_write (fs/read_write.c:643)
[ 698.491022][ T6895] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80)
[ 698.495802][ T6895] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113)
[ 698.502050][ T6895]
[ 698.502050][ T6895] other info that might help us debug this:
[ 698.502050][ T6895]
[ 698.512106][ T6895] Chain exists of:
[ 698.512106][ T6895] &mm->mmap_lock#2 --> cpu_hotplug_lock --> &cpuset_rwsem
[ 698.512106][ T6895]
[ 698.524934][ T6895] Possible unsafe locking scenario:
[ 698.524934][ T6895]
[ 698.532226][ T6895] CPU0 CPU1
[ 698.537436][ T6895] ---- ----
[ 698.542648][ T6895] lock(&cpuset_rwsem);
[ 698.546735][ T6895] lock(cpu_hotplug_lock);
[ 698.553593][ T6895] lock(&cpuset_rwsem);
[ 698.560197][ T6895] lock(&mm->mmap_lock#2);
[ 698.564550][ T6895]
[ 698.564550][ T6895] *** DEADLOCK ***
[ 698.564550][ T6895]
[ 698.572532][ T6895] 5 locks held by with_stress.sh/6895:
[ 698.577839][ T6895] #0: ffff8881013f2448 (sb_writers#7){.+.+}-{0:0}, at: ksys_write (fs/read_write.c:643)
[ 698.586519][ T6895] #1: ffff88817c726088 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter (fs/kernfs/file.c:288)
[ 698.596064][ T6895] #2: ffffffff8568d788 (cgroup_mutex){+.+.}-{3:3}, at: cgroup_lock_and_drain_offline (kernel/cgroup/cgroup.c:3001)
[ 698.606385][ T6895] #3: ffffffff8568d550 (cgroup_threadgroup_rwsem){++++}-{0:0}, at: cgroup_update_dfl_csses (include/linux/spinlock.h:374 (discriminator 9) kernel/cgroup/cgroup.c:2951 (discriminator 9))
[ 698.617312][ T6895] #4: ffffffff8569b1f0 (&cpuset_rwsem){++++}-{0:0}, at: cpuset_attach (kernel/cgroup/cpuset.c:2257)
[ 698.626335][ T6895]
[ 698.626335][ T6895] stack backtrace:
[ 698.632074][ T6895] CPU: 1 PID: 6895 Comm: with_stress.sh Not tainted 5.17.0-rc1-00003-g86895e1e85b3 #1
[ 698.641447][ T6895] Hardware name: HP HP Z240 SFF Workstation/802E, BIOS N51 Ver. 01.63 10/05/2017
[ 698.650384][ T6895] Call Trace:
[ 698.653526][ T6895] <TASK>
[ 698.656319][ T6895] dump_stack_lvl (lib/dump_stack.c:107)
[ 698.660669][ T6895] check_noncircular (kernel/locking/lockdep.c:2143)
[ 698.665449][ T6895] ? print_circular_bug+0x480/0x480
[ 698.671103][ T6895] ? mark_lock+0xca/0x13c0
[ 698.675976][ T6895] ? mark_lock_irq (kernel/locking/lockdep.c:4564)
[ 698.680765][ T6895] check_prev_add (kernel/locking/lockdep.c:3064)
[ 698.685379][ T6895] ? __lock_acquire (arch/x86/include/asm/bitops.h:214 include/asm-generic/bitops/instrumented-non-atomic.h:135 kernel/locking/lockdep.c:199 kernel/locking/lockdep.c:3713 kernel/locking/lockdep.c:3769 kernel/locking/lockdep.c:5027)
[ 698.690247][ T6895] __lock_acquire (kernel/locking/lockdep.c:3187 kernel/locking/lockdep.c:3801 kernel/locking/lockdep.c:5027)
[ 698.694949][ T6895] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4885)
[ 698.700772][ T6895] ? lock_is_held_type (kernel/locking/lockdep.c:5380 kernel/locking/lockdep.c:5682)
[ 698.705645][ T6895] ? rcu_read_lock_sched_held (include/linux/lockdep.h:283 kernel/rcu/update.c:125)
[ 698.711128][ T6895] lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604)
[ 698.715481][ T6895] ? mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384)
[ 698.720096][ T6895] ? rcu_read_unlock (include/linux/rcupdate.h:723 (discriminator 5))
[ 698.724709][ T6895] ? get_task_mm (kernel/fork.c:1311)
[ 698.728972][ T6895] ? rcu_read_lock_sched_held (include/linux/lockdep.h:283 kernel/rcu/update.c:125)
[ 698.734445][ T6895] ? rcu_read_lock_bh_held (kernel/rcu/update.c:120)
[ 698.739578][ T6895] ? lock_acquire (kernel/locking/lockdep.c:438 kernel/locking/lockdep.c:5641 kernel/locking/lockdep.c:5604)
[ 698.744098][ T6895] ? lock_is_held_type (kernel/locking/lockdep.c:5380 kernel/locking/lockdep.c:5682)
[ 698.748968][ T6895] down_write (include/linux/instrumented.h:101 include/linux/atomic/atomic-instrumented.h:1779 kernel/locking/rwsem.c:254 kernel/locking/rwsem.c:1258 kernel/locking/rwsem.c:1268 kernel/locking/rwsem.c:1515)
[ 698.753063][ T6895] ? mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384)
[ 698.757668][ T6895] ? down_write_killable (kernel/locking/rwsem.c:1512)
[ 698.762797][ T6895] ? rwlock_bug+0xc0/0xc0
[ 698.767581][ T6895] mpol_rebind_mm (arch/x86/include/asm/jump_label.h:27 include/linux/jump_label.h:212 include/linux/mmap_lock.h:35 include/linux/mmap_lock.h:72 mm/mempolicy.c:384)
[ 698.772023][ T6895] cpuset_attach (include/linux/instrumented.h:71 include/asm-generic/bitops/instrumented-non-atomic.h:134 kernel/cgroup/cpuset.c:258 kernel/cgroup/cpuset.c:2293)
[ 698.776463][ T6895] ? guarantee_online_cpus (kernel/cgroup/cpuset.c:2243)
[ 698.781763][ T6895] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4232 kernel/locking/lockdep.c:4292 kernel/locking/lockdep.c:4244)
[ 698.787587][ T6895] cgroup_migrate_execute (kernel/cgroup/cgroup.c:2513)
[ 698.792895][ T6895] ? _raw_spin_unlock_irq (arch/x86/include/asm/irqflags.h:45 arch/x86/include/asm/irqflags.h:80 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202)
[ 698.797942][ T6895] cgroup_update_dfl_csses (kernel/cgroup/cgroup.c:2975)
[ 698.803249][ T6895] ? cgroup_apply_control_enable (kernel/cgroup/cgroup.c:2939)
[ 698.809075][ T6895] ? css_next_descendant_pre (kernel/cgroup/cgroup.c:4394 kernel/cgroup/cgroup.c:4375)
[ 698.814468][ T6895] cgroup_subtree_control_write (kernel/cgroup/cgroup.c:3232 kernel/cgroup/cgroup.c:3365)
[ 698.820203][ T6895] ? cgroup_max_descendants_write (kernel/cgroup/cgroup.c:3283)
[ 698.826116][ T6895] cgroup_file_write (kernel/cgroup/cgroup.c:3855)
[ 698.830903][ T6895] ? css_release_work_fn (kernel/cgroup/cgroup.c:3855)
[ 698.836028][ T6895] ? __kasan_slab_alloc (mm/kasan/common.c:431 mm/kasan/common.c:469)
[ 698.840896][ T6895] ? lock_is_held_type (kernel/locking/lockdep.c:5380 kernel/locking/lockdep.c:5682)
[ 698.845762][ T6895] ? css_release_work_fn (kernel/cgroup/cgroup.c:3855)
[ 698.850886][ T6895] kernfs_fop_write_iter (fs/kernfs/file.c:300)
[ 698.856015][ T6895] new_sync_write (fs/read_write.c:504 (discriminator 1))
[ 698.860544][ T6895] ? new_sync_read (fs/read_write.c:493)
[ 698.865151][ T6895] ? ksys_write (fs/read_write.c:643)
[ 698.869413][ T6895] ? rcu_read_unlock (include/linux/rcupdate.h:723 (discriminator 5))
[ 698.874027][ T6895] ? lock_is_held_type (kernel/locking/lockdep.c:5380 kernel/locking/lockdep.c:5682)
[ 698.878895][ T6895] ? rcu_read_lock_held (kernel/rcu/update.c:341)
[ 698.883772][ T6895] vfs_write (fs/read_write.c:590)
[ 698.887867][ T6895] ksys_write (fs/read_write.c:643)
[ 698.891963][ T6895] ? __ia32_sys_read (fs/read_write.c:633)
[ 698.896570][ T6895] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4232 kernel/locking/lockdep.c:4292 kernel/locking/lockdep.c:4244)
[ 698.902386][ T6895] ? syscall_enter_from_user_mode (arch/x86/include/asm/irqflags.h:45 arch/x86/include/asm/irqflags.h:80 kernel/entry/common.c:107)
[ 698.908117][ T6895] do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80)
[ 698.912384][ T6895] ? lockdep_hardirqs_on_prepare (kernel/locking/lockdep.c:4232 kernel/locking/lockdep.c:4292 kernel/locking/lockdep.c:4244)
[ 698.918203][ T6895] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:113)
[ 698.923935][ T6895] RIP: 0033:0x7fe2d6264504
[ 698.928205][ T6895] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 f9 61 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
All code
========
0: 00 f7 add %dh,%bh
2: d8 64 89 02 fsubs 0x2(%rcx,%rcx,4)
6: 48 c7 c0 ff ff ff ff mov $0xffffffffffffffff,%rax
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
sudo bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
sudo bin/lkp run generated-yaml-file
# if come across any failure that blocks the test,
# please remove ~/.lkp and /lkp dir to run from a clean state.
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation
Thanks,
Oliver Sang