Hello,
Recently, our machine triggered a warning in __disable_runtime. The
dmesg are as follow:
[ 991.697692] WARNING: CPU: 0 PID: 49573 at kernel/sched/rt.c:802
rq_offline_rt+0x24d/0x260
[ 991.697795] CPU: 0 PID: 49573 Comm: kworker/1:0 Kdump: loaded Not
tainted 6.9.0-rc1+ #4
[ 991.697798] Hardware name: SuperCloud R5210 G12/X12DPi-N6, BIOS 1.1c
08/30/2021
[ 991.697800] Workqueue: events cpuset_hotplug_workfn
[ 991.697803] RIP: 0010:rq_offline_rt+0x24d/0x260
[ 991.697825] Call Trace:
[ 991.697827] <TASK>
[ 991.697830] ? __warn+0x7c/0x130
[ 991.697835] ? rq_offline_rt+0x24d/0x260
[ 991.697837] ? report_bug+0xf8/0x1e0
[ 991.697842] ? handle_bug+0x3f/0x70
[ 991.697858] set_rq_offline.part.125+0x2d/0x70
[ 991.697864] rq_attach_root+0xda/0x110
[ 991.697867] cpu_attach_domain+0x433/0x860
[ 991.697870] ? psi_task_switch+0x11d/0x260
[ 991.697873] ? __kmalloc_node+0x1dc/0x390
[ 991.697877] ? alloc_cpumask_var_node+0x1b/0x30
[ 991.697880] partition_sched_domains_locked+0x2a8/0x3a0
[ 991.697883] ? css_next_child+0x61/0x80
[ 991.697885] rebuild_sched_domains_locked+0x608/0x800
[ 991.697890] ? percpu_rwsem_wait+0x160/0x160
[ 991.697895] rebuild_sched_domains+0x1b/0x30
[ 991.697897] cpuset_hotplug_workfn+0x4b6/0x1160
[ 991.697899] ? balance_push+0x4e/0x120
[ 991.697903] ? finish_task_switch+0x8d/0x2d0
[ 991.697905] ? __switch_to+0x126/0x4f0
[ 991.697909] process_scheduled_works+0xad/0x430
[ 991.697917] worker_thread+0x105/0x270
[ 991.697922] kthread+0xde/0x110
[ 991.697928] ret_from_fork+0x2d/0x50
[ 991.697935] ret_from_fork_asm+0x11/0x20
[ 991.697940] </TASK>
[ 991.697941] ---[ end trace 0000000000000000 ]---
The corresponding code is :
802 WARN_ON_ONCE(want);
Because this WARN_ON_ONCE hasn’t changed from BUG_ON under linux-6.1, it
will trigger panic in those version.
More information:
1. RT_RUNTIME_SHARE is enabled.
2. We continuously create and remove cpu cgroups. We use cgexec to do
some tasks like "tree" or "ps" in these cgroups and the rt_runtime_us in
these cgroups are set to 2000~6000.
3. There are frequent cpu offline/online operations, so it will trigger
__disable_runtime.
Every time we run these operations after reboot, this warning will
happen easily.
---
Regards
Zhao Wenhui
在 2024/4/7 22:06, zhaowenhui (A) 写道:
> More information:
> 1. RT_RUNTIME_SHARE is enabled.
> 2. We continuously create and remove cpu cgroups. We use cgexec to do
> some tasks like "tree" or "ps" in these cgroups and the rt_runtime_us in
> these cgroups are set to 2000~6000.
> 3. There are frequent cpu offline/online operations, so it will trigger
> __disable_runtime.
>
> Every time we run these operations after reboot, this warning will
> happen easily.
>
--------reproduce shell scripts----------
[1] on_off.sh
#!/bin/bash
let cpu_end=8
while true
do
for i in `seq 0 $cpu_end`
do
echo $i
let j=i+1
echo 0 > /sys/devices/system/cpu/cpu${i}/online
echo 0 > /sys/devices/system/cpu/cpu${j}/online
sleep 0.5
echo 1 > /sys/devices/system/cpu/cpu${i}/online
echo 1 > /sys/devices/system/cpu/cpu${j}/online
done
done
[2] rt_test.sh
#!/bin/bsh
# set -x
name=test$$
cpu_dir=/sys/fs/cgroup/cpu/$name
mkdir $cpu_dir
echo $rt_runtime_us > $cpu_dir/cpu.rt_runtime_us &>/dev/null
function cg_change_rt_runtime_us()
{
while true
do
let tmp=2+RANDOM%4
let quota=tmp*1000
echo $quota > $cpu_dir/cpu.rt_runtime_us
done
}
function rt_test1()
{
while true
do
cgexec -g cpu:/${name} chrt 20 tree /sys/fs/cgroup/cpu
&>/dev/null
done
}
function rt_test2()
{
while true
do
cgexec -g cpu:/${name} chrt 30 ps aux &>/dev/null
done
}
rt_test1 &
rt_test2 &
cg_change_rt_runtime_us &> /dev/null &
[3] start_repro.sh
function set_env(){
echo RT_RUNTIME_SHARE > /sys/kernel/debug/sched/features
echo RT_RUNTIME_SHARE > /sys/kernel/debug/sched_features
}
function run_test(){
for i in 0 1
do
nohup sh rt_test.sh &
nohup sh rt_test.sh &
nohup sh rt_test.sh &
nohup sh rt_test.sh &
done
nohup sh on_off.sh &>/dev/null &
}
set_env
run_test
--------reproduce shell scripts----------
Running the start_repro.sh can easily reproduce this issue.
And when I debug in __disable_runtime, I found that the rtrq have
borrow some runtime, because rtrq->runtime is greater than
rtb->rt_runtime, but its span has only one cpu which is itself.
[ 43.686441] [__disable_runtime] begin, rq: ff3113087e5b4ec0 , cpu: 159
[ 43.686449] [__disable_runtime] rtrq: ff3112c9e9259800 ,
rtrq->runtime :3037500
[ 43.686450] [__disable_runtime] rtb->period: 1000000000 ,
rtb->rt_runtime: 3000000
[ 43.686451] [__disable_runtime] want original : -37500
[ 43.686451] [__disable_runtime] after get want, rtrq:
ff3112c9e9259800 , rtrq->runtime :3037500
[ 43.686453] [ERROR!] span have cpu: 159
Do you have any idea about this? ????
---
Regards
Zhao Wenhui