2021-07-14 08:40:35

by kernel test robot

[permalink] [raw]
Subject: [clocksource] db3a34e174: stress-ng.lockbus.ops_per_sec -10.1% regression



Greeting,

FYI, we noticed a -10.1% regression of stress-ng.lockbus.ops_per_sec due to commit:


commit: db3a34e17433de2390eb80d436970edcebd0ca3e ("clocksource: Retry clock read if long delays detected")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: stress-ng
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

nr_threads: 100%
testtime: 60s
class: cpu-cache
test: lockbus
cpufreq_governor: performance
ucode: 0x5003006




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
bin/lkp run generated-yaml-file

=========================================================================================
class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
cpu-cache/gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/lockbus/stress-ng/60s/0x5003006

commit:
64ab707125 ("clockevents: Add missing parameter documentation")
db3a34e174 ("clocksource: Retry clock read if long delays detected")

64ab7071254c178e db3a34e17433de2390eb80d4369
---------------- ---------------------------
%stddev %change %stddev
\ | \
4185 ? 3% -10.1% 3763 stress-ng.lockbus.ops_per_sec
6281 ? 2% +34.7% 8459 stress-ng.time.percent_of_cpu_this_job_got
208.68 ? 10% +60.1% 334.06 ? 8% stress-ng.time.system_time
3778 ? 4% +30.9% 4944 stress-ng.time.user_time
386955 ? 90% +405.3% 1955231 ? 22% cpuidle.POLL.time
375.17 +1.2% 379.78 pmeter.Average_Active_Power
181590 ? 8% +20.4% 218544 ? 4% softirqs.RCU
228.90 +1.9% 233.14 turbostat.PkgWatt
149585 ? 3% +13.4% 169573 ? 2% numa-vmstat.node1.nr_mapped
150816 ? 5% +13.4% 171081 ? 4% numa-vmstat.node1.nr_shmem
59.33 ? 2% +32.6% 78.67 vmstat.cpu.us
3011 ? 5% +13.2% 3409 ? 4% vmstat.system.cs
20.50 ? 2% -5.5 14.97 ? 2% mpstat.cpu.all.idle%
18.03 ? 15% -17.5 0.57 ? 96% mpstat.cpu.all.irq%
2.24 ? 9% +0.5 2.75 ? 5% mpstat.cpu.all.soft%
52.98 ? 3% +21.7 74.71 mpstat.cpu.all.usr%
0.01 ? 24% +446.5% 0.04 ? 51% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.01 ? 15% +562.3% 0.06 ? 54% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.01 ? 60% +5414.1% 0.72 ?205% perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
1006 ? 17% -36.1% 643.00 ? 31% perf-sched.total_wait_and_delay.count.ms
79.50 ? 43% -76.7% 18.50 ? 96% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
906386 ? 2% +3.0% 933337 proc-vmstat.nr_file_pages
317946 ? 6% +9.0% 346532 ? 2% proc-vmstat.nr_mapped
313202 ? 6% +8.6% 340155 ? 2% proc-vmstat.nr_shmem
186975 ? 90% +90.1% 355419 ? 63% proc-vmstat.pgfree
5317 ? 7% +34.6% 7158 ? 6% proc-vmstat.pgreuse
4.81 ? 13% -63.0% 1.78 ? 25% perf-stat.i.MPKI
7.693e+08 ? 10% +46.8% 1.129e+09 ? 9% perf-stat.i.branch-instructions
2.46 ? 6% -1.5 0.92 ? 12% perf-stat.i.branch-miss-rate%
26569766 ? 13% -48.6% 13652436 ? 11% perf-stat.i.branch-misses
268.47 ? 6% -78.4% 57.98 ? 17% perf-stat.i.cpi
128225 ? 6% -14.4% 109777 perf-stat.i.cpu-clock
2.23e+11 ? 3% +12.3% 2.504e+11 perf-stat.i.cpu-cycles
279790 ? 6% -20.3% 222991 ? 4% perf-stat.i.cycles-between-cache-misses
0.03 ? 42% -0.0 0.01 ? 48% perf-stat.i.dTLB-load-miss-rate%
3.387e+08 ? 12% -41.3% 1.99e+08 ? 6% perf-stat.i.dTLB-stores
73.90 +11.4 85.35 perf-stat.i.iTLB-load-miss-rate%
626026 ? 10% -53.1% 293497 ? 11% perf-stat.i.iTLB-loads
3.75e+09 ? 10% +38.2% 5.182e+09 ? 9% perf-stat.i.instructions
3635 ? 8% +107.0% 7523 ? 3% perf-stat.i.instructions-per-iTLB-miss
0.27 ? 15% -48.1% 0.14 ? 8% perf-stat.i.ipc
98.69 ? 18% -52.6% 46.80 ? 10% perf-stat.i.major-faults
1.67 ? 7% +39.5% 2.33 ? 2% perf-stat.i.metric.GHz
15.26 ? 5% +26.7% 19.34 ? 7% perf-stat.i.metric.M/sec
4933 ? 12% -34.4% 3234 ? 3% perf-stat.i.minor-faults
71.87 ? 3% +11.0 82.86 perf-stat.i.node-load-miss-rate%
129202 ? 15% -37.8% 80341 ? 5% perf-stat.i.node-loads
47.00 ? 6% +27.4 74.36 ? 3% perf-stat.i.node-store-miss-rate%
416773 ? 22% -51.8% 200689 ? 10% perf-stat.i.node-stores
5032 ? 12% -34.8% 3280 ? 3% perf-stat.i.page-faults
128249 ? 6% -14.4% 109787 perf-stat.i.task-clock
4.56 ? 18% -68.8% 1.42 ? 19% perf-stat.overall.MPKI
2.76 ? 10% -2.2 0.55 ? 9% perf-stat.overall.branch-miss-rate%
190.97 ? 8% -69.4% 58.36 ? 12% perf-stat.overall.cpi
0.03 ? 43% -0.0 0.01 ? 41% perf-stat.overall.dTLB-load-miss-rate%
78.39 ? 2% +6.7 85.12 perf-stat.overall.iTLB-load-miss-rate%
2230 ? 11% +238.1% 7543 ? 4% perf-stat.overall.instructions-per-iTLB-miss
0.01 ? 8% +229.2% 0.02 ? 11% perf-stat.overall.ipc
71.24 ? 3% +11.9 83.18 perf-stat.overall.node-load-miss-rate%
31.44 ? 17% +32.9 64.30 ? 4% perf-stat.overall.node-store-miss-rate%
2.064e+08 ? 13% +346.8% 9.224e+08 ? 11% perf-stat.ps.branch-instructions
752036 ? 14% +44.3% 1085363 ? 4% perf-stat.ps.cache-misses
2030 ? 8% +17.0% 2376 perf-stat.ps.context-switches
1.89e+11 ? 11% +26.5% 2.392e+11 perf-stat.ps.cpu-cycles
2.345e+08 ? 12% +148.7% 5.832e+08 ? 10% perf-stat.ps.dTLB-loads
9.969e+08 ? 12% +317.0% 4.157e+09 ? 11% perf-stat.ps.instructions
1386 ? 14% +24.4% 1725 ? 5% perf-stat.ps.minor-faults
93040 ? 10% +128.6% 212694 ? 7% perf-stat.ps.node-load-misses
43018 ? 16% +177.4% 119313 ? 9% perf-stat.ps.node-store-misses
1404 ? 14% +23.7% 1737 ? 5% perf-stat.ps.page-faults
4.134e+10 ? 36% +514.1% 2.539e+11 ? 12% perf-stat.total.instructions
9296 ? 26% -61.4% 3591 ? 29% interrupts.CPU0.NMI:Non-maskable_interrupts
9296 ? 26% -61.4% 3591 ? 29% interrupts.CPU0.PMI:Performance_monitoring_interrupts
9800 ? 21% -59.1% 4007 ? 26% interrupts.CPU10.NMI:Non-maskable_interrupts
9800 ? 21% -59.1% 4007 ? 26% interrupts.CPU10.PMI:Performance_monitoring_interrupts
10056 ? 31% -57.1% 4312 ? 14% interrupts.CPU12.NMI:Non-maskable_interrupts
10056 ? 31% -57.1% 4312 ? 14% interrupts.CPU12.PMI:Performance_monitoring_interrupts
9164 ? 31% -56.0% 4033 ? 25% interrupts.CPU14.NMI:Non-maskable_interrupts
9164 ? 31% -56.0% 4033 ? 25% interrupts.CPU14.PMI:Performance_monitoring_interrupts
9386 ? 22% -57.2% 4021 ? 26% interrupts.CPU15.NMI:Non-maskable_interrupts
9386 ? 22% -57.2% 4021 ? 26% interrupts.CPU15.PMI:Performance_monitoring_interrupts
9665 ? 25% -66.2% 3264 ? 34% interrupts.CPU16.NMI:Non-maskable_interrupts
9665 ? 25% -66.2% 3264 ? 34% interrupts.CPU16.PMI:Performance_monitoring_interrupts
9907 ? 21% -64.3% 3532 ? 25% interrupts.CPU17.NMI:Non-maskable_interrupts
9907 ? 21% -64.3% 3532 ? 25% interrupts.CPU17.PMI:Performance_monitoring_interrupts
9510 ? 24% -54.3% 4343 ? 12% interrupts.CPU18.NMI:Non-maskable_interrupts
9510 ? 24% -54.3% 4343 ? 12% interrupts.CPU18.PMI:Performance_monitoring_interrupts
8709 ? 26% -51.1% 4259 ? 13% interrupts.CPU19.NMI:Non-maskable_interrupts
8709 ? 26% -51.1% 4259 ? 13% interrupts.CPU19.PMI:Performance_monitoring_interrupts
8427 ? 25% -53.5% 3918 ? 18% interrupts.CPU2.NMI:Non-maskable_interrupts
8427 ? 25% -53.5% 3918 ? 18% interrupts.CPU2.PMI:Performance_monitoring_interrupts
8683 ? 24% -58.4% 3614 ? 34% interrupts.CPU20.NMI:Non-maskable_interrupts
8683 ? 24% -58.4% 3614 ? 34% interrupts.CPU20.PMI:Performance_monitoring_interrupts
9291 ? 23% -62.0% 3534 ? 25% interrupts.CPU21.NMI:Non-maskable_interrupts
9291 ? 23% -62.0% 3534 ? 25% interrupts.CPU21.PMI:Performance_monitoring_interrupts
9050 ? 31% -56.7% 3919 ? 23% interrupts.CPU22.NMI:Non-maskable_interrupts
9050 ? 31% -56.7% 3919 ? 23% interrupts.CPU22.PMI:Performance_monitoring_interrupts
8484 ? 21% -53.7% 3928 ? 24% interrupts.CPU23.NMI:Non-maskable_interrupts
8484 ? 21% -53.7% 3928 ? 24% interrupts.CPU23.PMI:Performance_monitoring_interrupts
9342 ? 24% -53.6% 4331 ? 13% interrupts.CPU24.NMI:Non-maskable_interrupts
9342 ? 24% -53.6% 4331 ? 13% interrupts.CPU24.PMI:Performance_monitoring_interrupts
9322 ? 23% -57.6% 3957 ? 19% interrupts.CPU25.NMI:Non-maskable_interrupts
9322 ? 23% -57.6% 3957 ? 19% interrupts.CPU25.PMI:Performance_monitoring_interrupts
8548 ? 23% -49.3% 4331 ? 13% interrupts.CPU28.NMI:Non-maskable_interrupts
8548 ? 23% -49.3% 4331 ? 13% interrupts.CPU28.PMI:Performance_monitoring_interrupts
9804 ? 22% -59.2% 4002 ? 26% interrupts.CPU3.NMI:Non-maskable_interrupts
9804 ? 22% -59.2% 4002 ? 26% interrupts.CPU3.PMI:Performance_monitoring_interrupts
8523 ? 21% -53.7% 3946 ? 28% interrupts.CPU30.NMI:Non-maskable_interrupts
8523 ? 21% -53.7% 3946 ? 28% interrupts.CPU30.PMI:Performance_monitoring_interrupts
9273 ? 23% -64.9% 3258 ? 35% interrupts.CPU32.NMI:Non-maskable_interrupts
9273 ? 23% -64.9% 3258 ? 35% interrupts.CPU32.PMI:Performance_monitoring_interrupts
8605 ? 27% -55.6% 3825 ? 18% interrupts.CPU33.NMI:Non-maskable_interrupts
8605 ? 27% -55.6% 3825 ? 18% interrupts.CPU33.PMI:Performance_monitoring_interrupts
9126 ? 22% -57.3% 3893 ? 28% interrupts.CPU34.NMI:Non-maskable_interrupts
9126 ? 22% -57.3% 3893 ? 28% interrupts.CPU34.PMI:Performance_monitoring_interrupts
8862 ? 21% -56.4% 3866 ? 29% interrupts.CPU35.NMI:Non-maskable_interrupts
8862 ? 21% -56.4% 3866 ? 29% interrupts.CPU35.PMI:Performance_monitoring_interrupts
9300 ? 22% -58.7% 3837 ? 18% interrupts.CPU37.NMI:Non-maskable_interrupts
9300 ? 22% -58.7% 3837 ? 18% interrupts.CPU37.PMI:Performance_monitoring_interrupts
9118 ? 23% -53.5% 4239 ? 14% interrupts.CPU38.NMI:Non-maskable_interrupts
9118 ? 23% -53.5% 4239 ? 14% interrupts.CPU38.PMI:Performance_monitoring_interrupts
9411 ? 24% -58.0% 3949 ? 27% interrupts.CPU39.NMI:Non-maskable_interrupts
9411 ? 24% -58.0% 3949 ? 27% interrupts.CPU39.PMI:Performance_monitoring_interrupts
9106 ? 21% -61.3% 3528 ? 31% interrupts.CPU41.NMI:Non-maskable_interrupts
9106 ? 21% -61.3% 3528 ? 31% interrupts.CPU41.PMI:Performance_monitoring_interrupts
9218 ? 25% -54.1% 4232 ? 14% interrupts.CPU42.NMI:Non-maskable_interrupts
9218 ? 25% -54.1% 4232 ? 14% interrupts.CPU42.PMI:Performance_monitoring_interrupts
9326 ? 21% -58.3% 3886 ? 24% interrupts.CPU43.NMI:Non-maskable_interrupts
9326 ? 21% -58.3% 3886 ? 24% interrupts.CPU43.PMI:Performance_monitoring_interrupts
9078 ? 26% -53.7% 4207 ? 14% interrupts.CPU44.NMI:Non-maskable_interrupts
9078 ? 26% -53.7% 4207 ? 14% interrupts.CPU44.PMI:Performance_monitoring_interrupts
11918 ? 22% -60.5% 4707 ? 12% interrupts.CPU47.NMI:Non-maskable_interrupts
11918 ? 22% -60.5% 4707 ? 12% interrupts.CPU47.PMI:Performance_monitoring_interrupts
313.50 ? 20% +22.1% 382.83 ? 5% interrupts.CPU47.RES:Rescheduling_interrupts
8812 ? 27% -51.8% 4249 ? 15% interrupts.CPU48.NMI:Non-maskable_interrupts
8812 ? 27% -51.8% 4249 ? 15% interrupts.CPU48.PMI:Performance_monitoring_interrupts
9771 ? 24% -67.9% 3140 ? 37% interrupts.CPU49.NMI:Non-maskable_interrupts
9771 ? 24% -67.9% 3140 ? 37% interrupts.CPU49.PMI:Performance_monitoring_interrupts
9571 ? 22% -54.5% 4350 ? 15% interrupts.CPU5.NMI:Non-maskable_interrupts
9571 ? 22% -54.5% 4350 ? 15% interrupts.CPU5.PMI:Performance_monitoring_interrupts
8607 ? 28% -55.0% 3869 ? 23% interrupts.CPU50.NMI:Non-maskable_interrupts
8607 ? 28% -55.0% 3869 ? 23% interrupts.CPU50.PMI:Performance_monitoring_interrupts
9256 ? 22% -54.3% 4233 ? 11% interrupts.CPU51.NMI:Non-maskable_interrupts
9256 ? 22% -54.3% 4233 ? 11% interrupts.CPU51.PMI:Performance_monitoring_interrupts
8684 ? 26% -54.9% 3916 ? 27% interrupts.CPU55.NMI:Non-maskable_interrupts
8684 ? 26% -54.9% 3916 ? 27% interrupts.CPU55.PMI:Performance_monitoring_interrupts
9258 ? 23% -54.2% 4238 ? 15% interrupts.CPU56.NMI:Non-maskable_interrupts
9258 ? 23% -54.2% 4238 ? 15% interrupts.CPU56.PMI:Performance_monitoring_interrupts
9609 ? 22% -59.7% 3869 ? 24% interrupts.CPU58.NMI:Non-maskable_interrupts
9609 ? 22% -59.7% 3869 ? 24% interrupts.CPU58.PMI:Performance_monitoring_interrupts
8719 ? 25% -51.6% 4221 ? 14% interrupts.CPU59.NMI:Non-maskable_interrupts
8719 ? 25% -51.6% 4221 ? 14% interrupts.CPU59.PMI:Performance_monitoring_interrupts
9685 ? 23% -59.8% 3894 ? 27% interrupts.CPU61.NMI:Non-maskable_interrupts
9685 ? 23% -59.8% 3894 ? 27% interrupts.CPU61.PMI:Performance_monitoring_interrupts
8818 ? 31% -54.8% 3982 ? 27% interrupts.CPU62.NMI:Non-maskable_interrupts
8818 ? 31% -54.8% 3982 ? 27% interrupts.CPU62.PMI:Performance_monitoring_interrupts
9598 ? 21% -59.2% 3920 ? 23% interrupts.CPU63.NMI:Non-maskable_interrupts
9598 ? 21% -59.2% 3920 ? 23% interrupts.CPU63.PMI:Performance_monitoring_interrupts
9667 ? 23% -55.3% 4318 ? 13% interrupts.CPU66.NMI:Non-maskable_interrupts
9667 ? 23% -55.3% 4318 ? 13% interrupts.CPU66.PMI:Performance_monitoring_interrupts
7886 ? 27% -46.2% 4242 ? 12% interrupts.CPU67.NMI:Non-maskable_interrupts
7886 ? 27% -46.2% 4242 ? 12% interrupts.CPU67.PMI:Performance_monitoring_interrupts
9656 ? 19% -59.9% 3873 ? 23% interrupts.CPU68.NMI:Non-maskable_interrupts
9656 ? 19% -59.9% 3873 ? 23% interrupts.CPU68.PMI:Performance_monitoring_interrupts
8746 ? 21% -55.6% 3885 ? 19% interrupts.CPU69.NMI:Non-maskable_interrupts
8746 ? 21% -55.6% 3885 ? 19% interrupts.CPU69.PMI:Performance_monitoring_interrupts
9735 ? 25% -58.8% 4007 ? 21% interrupts.CPU70.NMI:Non-maskable_interrupts
9735 ? 25% -58.8% 4007 ? 21% interrupts.CPU70.PMI:Performance_monitoring_interrupts
10019 ? 22% -57.2% 4292 ? 13% interrupts.CPU71.NMI:Non-maskable_interrupts
10019 ? 22% -57.2% 4292 ? 13% interrupts.CPU71.PMI:Performance_monitoring_interrupts
9357 ? 26% -55.0% 4209 ? 13% interrupts.CPU72.NMI:Non-maskable_interrupts
9357 ? 26% -55.0% 4209 ? 13% interrupts.CPU72.PMI:Performance_monitoring_interrupts
8721 ? 24% -59.5% 3531 ? 27% interrupts.CPU75.NMI:Non-maskable_interrupts
8721 ? 24% -59.5% 3531 ? 27% interrupts.CPU75.PMI:Performance_monitoring_interrupts
8499 ? 24% -57.1% 3646 ? 37% interrupts.CPU77.NMI:Non-maskable_interrupts
8499 ? 24% -57.1% 3646 ? 37% interrupts.CPU77.PMI:Performance_monitoring_interrupts
9140 ? 23% -58.3% 3810 ? 19% interrupts.CPU78.NMI:Non-maskable_interrupts
9140 ? 23% -58.3% 3810 ? 19% interrupts.CPU78.PMI:Performance_monitoring_interrupts
8348 ? 27% -49.0% 4258 ? 12% interrupts.CPU79.NMI:Non-maskable_interrupts
8348 ? 27% -49.0% 4258 ? 12% interrupts.CPU79.PMI:Performance_monitoring_interrupts
8910 ? 22% -59.3% 3626 ? 33% interrupts.CPU80.NMI:Non-maskable_interrupts
8910 ? 22% -59.3% 3626 ? 33% interrupts.CPU80.PMI:Performance_monitoring_interrupts
8093 ? 19% -57.0% 3476 ? 25% interrupts.CPU82.NMI:Non-maskable_interrupts
8093 ? 19% -57.0% 3476 ? 25% interrupts.CPU82.PMI:Performance_monitoring_interrupts
8221 ? 22% -51.8% 3962 ? 27% interrupts.CPU83.NMI:Non-maskable_interrupts
8221 ? 22% -51.8% 3962 ? 27% interrupts.CPU83.PMI:Performance_monitoring_interrupts
8911 ? 30% -52.6% 4227 ? 13% interrupts.CPU85.NMI:Non-maskable_interrupts
8911 ? 30% -52.6% 4227 ? 13% interrupts.CPU85.PMI:Performance_monitoring_interrupts
8535 ? 39% -54.6% 3871 ? 24% interrupts.CPU86.NMI:Non-maskable_interrupts
8535 ? 39% -54.6% 3871 ? 24% interrupts.CPU86.PMI:Performance_monitoring_interrupts
9061 ? 32% -59.4% 3681 ? 37% interrupts.CPU88.NMI:Non-maskable_interrupts
9061 ? 32% -59.4% 3681 ? 37% interrupts.CPU88.PMI:Performance_monitoring_interrupts
9533 ? 34% -66.2% 3220 ? 35% interrupts.CPU89.NMI:Non-maskable_interrupts
9533 ? 34% -66.2% 3220 ? 35% interrupts.CPU89.PMI:Performance_monitoring_interrupts
8924 ? 30% -55.8% 3944 ? 26% interrupts.CPU91.NMI:Non-maskable_interrupts
8924 ? 30% -55.8% 3944 ? 26% interrupts.CPU91.PMI:Performance_monitoring_interrupts
9693 ? 22% -59.2% 3956 ? 27% interrupts.CPU92.NMI:Non-maskable_interrupts
9693 ? 22% -59.2% 3956 ? 27% interrupts.CPU92.PMI:Performance_monitoring_interrupts
8789 ? 26% -56.2% 3849 ? 20% interrupts.CPU93.NMI:Non-maskable_interrupts
8789 ? 26% -56.2% 3849 ? 20% interrupts.CPU93.PMI:Performance_monitoring_interrupts
8107 ? 28% -48.2% 4202 ? 14% interrupts.CPU94.NMI:Non-maskable_interrupts
8107 ? 28% -48.2% 4202 ? 14% interrupts.CPU94.PMI:Performance_monitoring_interrupts
803.00 ? 3% -23.8% 611.67 ? 17% interrupts.CPU95.CAL:Function_call_interrupts
9453 ? 23% -53.5% 4398 ? 14% interrupts.CPU95.NMI:Non-maskable_interrupts
9453 ? 23% -53.5% 4398 ? 14% interrupts.CPU95.PMI:Performance_monitoring_interrupts
219.50 ? 16% -58.8% 90.33 ? 5% interrupts.IWI:IRQ_work_interrupts
858269 ? 23% -56.1% 377152 ? 14% interrupts.NMI:Non-maskable_interrupts
858269 ? 23% -56.1% 377152 ? 14% interrupts.PMI:Performance_monitoring_interrupts
1946 ? 6% +14.1% 2220 interrupts.RES:Rescheduling_interrupts
412.83 ? 22% -49.6% 208.17 ? 43% interrupts.TLB:TLB_shootdowns
35.40 ? 16% -13.1 22.35 ? 36% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
34.93 ? 16% -12.8 22.18 ? 36% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
20.37 ? 16% -9.0 11.36 ? 41% perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
5.09 ? 28% -4.1 0.97 ? 20% perf-profile.calltrace.cycles-pp.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
4.54 ? 21% -2.7 1.88 ? 33% perf-profile.calltrace.cycles-pp.hrtimer_active.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
3.09 ? 25% -1.5 1.56 ? 20% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
3.08 ? 25% -1.5 1.55 ? 20% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.88 ? 31% -1.4 0.45 ? 71% perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
2.00 ? 14% -1.1 0.93 ? 51% perf-profile.calltrace.cycles-pp.update_cfs_group.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
3.69 ? 15% -0.9 2.82 ? 9% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
2.95 ? 15% -0.8 2.11 ? 9% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt
1.86 ? 18% -0.7 1.18 ? 15% perf-profile.calltrace.cycles-pp.execve
1.85 ? 18% -0.7 1.18 ? 15% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
1.85 ? 18% -0.7 1.18 ? 15% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
1.85 ? 18% -0.7 1.18 ? 15% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
1.84 ? 18% -0.7 1.18 ? 15% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
2.01 ? 16% -0.6 1.36 ? 10% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu
1.52 ? 18% -0.6 0.97 ? 17% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.38 ? 18% -0.5 0.86 ? 18% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
1.33 ? 19% -0.5 0.83 ? 19% perf-profile.calltrace.cycles-pp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
0.90 ? 15% -0.5 0.42 ? 73% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
1.64 ? 22% -0.5 1.17 ? 13% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start
0.78 ? 14% -0.5 0.32 ?101% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.78 ? 14% -0.5 0.32 ?101% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.78 ? 14% -0.5 0.32 ?101% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.85 ? 14% -0.4 0.44 ? 45% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains
1.22 ? 13% -0.4 0.84 ? 19% perf-profile.calltrace.cycles-pp.asm_exc_page_fault
1.16 ? 15% -0.4 0.79 ? 18% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
1.16 ? 15% -0.4 0.79 ? 18% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
0.97 ? 15% -0.3 0.64 ? 19% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
1.04 ? 16% -0.3 0.79 ? 16% perf-profile.calltrace.cycles-pp.__libc_fork
0.80 ? 16% -0.2 0.58 ? 17% perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
0.80 ? 16% -0.2 0.58 ? 16% perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
0.80 ? 16% -0.2 0.59 ? 17% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork
0.80 ? 16% -0.2 0.59 ? 17% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
1.38 ? 9% +0.3 1.68 ? 15% perf-profile.calltrace.cycles-pp.run_rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.50 ? 45% +0.6 1.11 ? 41% perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +4.0 4.01 ? 24% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
0.10 ?223% +4.4 4.45 ? 25% perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
1.11 ? 26% +5.3 6.38 ? 48% perf-profile.calltrace.cycles-pp.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
0.00 +5.9 5.87 ? 47% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt
0.00 +6.0 5.98 ? 29% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times
0.00 +6.8 6.77 ? 30% perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle
0.00 +6.9 6.90 ? 30% perf-profile.calltrace.cycles-pp.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
2.53 ? 14% +9.8 12.36 ? 26% perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +11.3 11.32 ? 27% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
65.37 ? 5% +11.8 77.13 ? 4% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
56.02 ? 7% +14.9 70.94 ? 6% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
55.81 ? 7% +15.1 70.87 ? 6% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
7.06 ? 17% +15.6 22.63 ? 15% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +18.3 18.30 ? 15% perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt
1.43 ? 12% +18.9 20.29 ? 13% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
40.00 ? 14% -15.1 24.85 ? 34% perf-profile.children.cycles-pp.tick_sched_handle
39.46 ? 14% -14.8 24.66 ? 34% perf-profile.children.cycles-pp.update_process_times
22.79 ? 15% -10.3 12.46 ? 39% perf-profile.children.cycles-pp.task_tick_fair
5.39 ? 22% -4.3 1.11 ? 20% perf-profile.children.cycles-pp.perf_event_task_tick
4.78 ? 17% -3.2 1.57 ? 36% perf-profile.children.cycles-pp.irqtime_account_irq
4.26 ? 17% -3.1 1.20 ? 23% perf-profile.children.cycles-pp.ghes_notify_nmi
3.10 ? 25% -2.8 0.28 ? 36% perf-profile.children.cycles-pp.__intel_pmu_enable_all
4.86 ? 21% -2.8 2.05 ? 31% perf-profile.children.cycles-pp.hrtimer_active
10.39 ? 13% -2.5 7.92 ? 14% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
10.38 ? 13% -2.5 7.91 ? 14% perf-profile.children.cycles-pp.do_syscall_64
2.15 ? 28% -1.5 0.64 ? 31% perf-profile.children.cycles-pp.rcu_sched_clock_irq
1.70 ? 21% -1.3 0.40 ? 33% perf-profile.children.cycles-pp.update_rq_clock
2.26 ? 12% -1.2 1.10 ? 33% perf-profile.children.cycles-pp.update_cfs_group
1.78 ? 9% -1.2 0.63 ? 18% perf-profile.children.cycles-pp._raw_spin_lock
2.70 ? 20% -1.1 1.56 ? 31% perf-profile.children.cycles-pp.native_sched_clock
4.30 ? 14% -1.1 3.18 ? 10% perf-profile.children.cycles-pp.rebalance_domains
3.42 ? 15% -1.1 2.37 ? 9% perf-profile.children.cycles-pp.load_balance
1.63 ? 40% -1.0 0.64 ? 12% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.31 ? 10% -1.0 0.34 ? 26% perf-profile.children.cycles-pp.trigger_load_balance
1.70 ? 10% -0.9 0.78 ? 26% perf-profile.children.cycles-pp.native_flush_tlb_one_user
1.08 ? 3% -0.8 0.23 ? 20% perf-profile.children.cycles-pp.__remove_hrtimer
1.26 ? 48% -0.8 0.42 ? 20% perf-profile.children.cycles-pp.unmap_vmas
1.18 ? 6% -0.8 0.36 ? 31% perf-profile.children.cycles-pp.sched_clock_cpu
1.19 ? 52% -0.8 0.38 ? 19% perf-profile.children.cycles-pp.unmap_page_range
1.15 ? 54% -0.8 0.35 ? 19% perf-profile.children.cycles-pp.zap_pte_range
2.32 ? 16% -0.8 1.54 ? 10% perf-profile.children.cycles-pp.find_busiest_group
1.26 ? 6% -0.8 0.47 ? 16% perf-profile.children.cycles-pp.hrtimer_update_next_event
1.99 ? 15% -0.8 1.20 ? 16% perf-profile.children.cycles-pp.__x64_sys_execve
1.98 ? 15% -0.8 1.20 ? 16% perf-profile.children.cycles-pp.do_execveat_common
0.94 ? 70% -0.8 0.17 ? 25% perf-profile.children.cycles-pp.__do_munmap
1.48 ? 37% -0.7 0.73 ? 25% perf-profile.children.cycles-pp.paranoid_entry
1.08 ? 23% -0.7 0.34 ? 34% perf-profile.children.cycles-pp.nmi_handle
2.17 ? 16% -0.7 1.44 ? 10% perf-profile.children.cycles-pp.update_sd_lb_stats
0.77 ? 83% -0.7 0.06 ? 53% perf-profile.children.cycles-pp.unmap_region
1.08 ? 25% -0.7 0.38 ? 41% perf-profile.children.cycles-pp.arch_scale_freq_tick
0.86 ? 29% -0.7 0.15 ? 29% perf-profile.children.cycles-pp.sync_regs
0.72 ? 89% -0.7 0.03 ?100% perf-profile.children.cycles-pp.__x64_sys_munmap
0.77 ? 85% -0.7 0.08 ? 24% perf-profile.children.cycles-pp.__vm_munmap
1.86 ? 18% -0.7 1.18 ? 15% perf-profile.children.cycles-pp.execve
0.83 ? 3% -0.7 0.16 ? 22% perf-profile.children.cycles-pp.timerqueue_del
1.63 ? 16% -0.6 0.98 ? 17% perf-profile.children.cycles-pp.bprm_execve
0.84 ? 24% -0.6 0.24 ? 18% perf-profile.children.cycles-pp.update_irq_load_avg
1.47 ? 16% -0.6 0.88 ? 19% perf-profile.children.cycles-pp.exec_binprm
0.86 ? 12% -0.6 0.27 ? 14% perf-profile.children.cycles-pp.enqueue_hrtimer
1.43 ? 16% -0.6 0.85 ? 20% perf-profile.children.cycles-pp.load_elf_binary
0.78 ? 13% -0.6 0.24 ? 14% perf-profile.children.cycles-pp.timerqueue_add
2.39 ? 15% -0.5 1.90 ? 19% perf-profile.children.cycles-pp.asm_exc_page_fault
1.90 ? 16% -0.5 1.40 ? 18% perf-profile.children.cycles-pp.handle_mm_fault
1.76 ? 15% -0.5 1.27 ? 18% perf-profile.children.cycles-pp.__handle_mm_fault
2.23 ? 16% -0.5 1.75 ? 18% perf-profile.children.cycles-pp.exc_page_fault
2.22 ? 16% -0.5 1.74 ? 18% perf-profile.children.cycles-pp.do_user_addr_fault
0.60 ? 21% -0.5 0.12 ? 38% perf-profile.children.cycles-pp.account_process_tick
1.30 ? 14% -0.5 0.83 ? 18% perf-profile.children.cycles-pp.mmput
1.29 ? 14% -0.5 0.82 ? 18% perf-profile.children.cycles-pp.exit_mmap
0.88 ? 10% -0.5 0.42 ? 11% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.72 ? 8% -0.4 0.28 ? 16% perf-profile.children.cycles-pp.__hrtimer_next_event_base
0.99 ? 15% -0.4 0.58 ? 30% perf-profile.children.cycles-pp.acpi_os_read_memory
0.86 ? 19% -0.4 0.48 ? 21% perf-profile.children.cycles-pp.do_filp_open
0.85 ? 19% -0.4 0.48 ? 21% perf-profile.children.cycles-pp.path_openat
0.84 ? 18% -0.4 0.47 ? 21% perf-profile.children.cycles-pp.do_sys_open
0.84 ? 18% -0.4 0.47 ? 22% perf-profile.children.cycles-pp.do_sys_openat2
0.44 ? 10% -0.3 0.10 ? 25% perf-profile.children.cycles-pp.ghes_copy_tofrom_phys
0.75 ? 15% -0.3 0.43 ? 17% perf-profile.children.cycles-pp.begin_new_exec
0.78 ? 17% -0.3 0.46 ? 22% perf-profile.children.cycles-pp.try_to_wake_up
0.50 ? 23% -0.3 0.19 ? 37% perf-profile.children.cycles-pp.account_user_time
0.41 ? 16% -0.3 0.11 ? 27% perf-profile.children.cycles-pp.tick_program_event
0.66 ? 19% -0.3 0.38 ? 22% perf-profile.children.cycles-pp.vm_mmap_pgoff
0.49 ? 7% -0.3 0.21 ? 18% perf-profile.children.cycles-pp.__hrtimer_get_next_event
0.53 ? 23% -0.3 0.25 ? 33% perf-profile.children.cycles-pp.write
0.51 ? 16% -0.3 0.24 ? 18% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.43 ? 14% -0.3 0.17 ? 25% perf-profile.children.cycles-pp.__native_set_fixmap
0.62 ? 19% -0.3 0.36 ? 22% perf-profile.children.cycles-pp.do_mmap
0.37 ? 48% -0.3 0.11 ? 21% perf-profile.children.cycles-pp.native_irq_return_iret
1.04 ? 16% -0.3 0.79 ? 16% perf-profile.children.cycles-pp.__libc_fork
0.43 ? 66% -0.2 0.18 ? 44% perf-profile.children.cycles-pp.exc_nmi
0.36 ? 42% -0.2 0.12 ? 42% perf-profile.children.cycles-pp.__intel_pmu_disable_all
0.94 ? 14% -0.2 0.70 ? 17% perf-profile.children.cycles-pp.do_fault
0.56 ? 19% -0.2 0.32 ? 23% perf-profile.children.cycles-pp.mmap_region
0.80 ? 14% -0.2 0.57 ? 17% perf-profile.children.cycles-pp.__x64_sys_exit_group
0.80 ? 14% -0.2 0.57 ? 17% perf-profile.children.cycles-pp.do_group_exit
0.34 ? 10% -0.2 0.11 ? 12% perf-profile.children.cycles-pp.hrtimer_forward
0.80 ? 14% -0.2 0.57 ? 17% perf-profile.children.cycles-pp.do_exit
0.77 ? 16% -0.2 0.54 ? 17% perf-profile.children.cycles-pp.copy_process
0.80 ? 16% -0.2 0.58 ? 16% perf-profile.children.cycles-pp.__do_sys_clone
0.80 ? 16% -0.2 0.58 ? 17% perf-profile.children.cycles-pp.kernel_clone
0.50 ? 19% -0.2 0.28 ? 25% perf-profile.children.cycles-pp.ksys_mmap_pgoff
0.37 ? 5% -0.2 0.16 ? 38% perf-profile.children.cycles-pp.run_posix_cpu_timers
0.44 ? 20% -0.2 0.24 ? 24% perf-profile.children.cycles-pp.link_path_walk
0.62 ? 13% -0.2 0.42 ? 18% perf-profile.children.cycles-pp.filemap_map_pages
0.43 ? 20% -0.2 0.24 ? 23% perf-profile.children.cycles-pp.walk_component
0.53 ? 17% -0.2 0.35 ? 20% perf-profile.children.cycles-pp.dup_mm
0.22 ? 17% -0.2 0.04 ? 72% perf-profile.children.cycles-pp.nmi_restore
0.28 ? 24% -0.2 0.10 ? 19% perf-profile.children.cycles-pp.__poll
0.46 ? 20% -0.2 0.28 ? 18% perf-profile.children.cycles-pp.kmem_cache_alloc
0.44 ? 17% -0.2 0.26 ? 23% perf-profile.children.cycles-pp.setlocale
0.50 ? 17% -0.2 0.33 ? 20% perf-profile.children.cycles-pp.dup_mmap
0.38 ? 17% -0.2 0.22 ? 15% perf-profile.children.cycles-pp.free_pgtables
0.49 ? 14% -0.1 0.35 ? 19% perf-profile.children.cycles-pp.__alloc_pages
0.23 ? 13% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.rb_insert_color
0.24 ? 26% -0.1 0.10 ? 26% perf-profile.children.cycles-pp.enqueue_task_fair
0.22 ? 27% -0.1 0.09 ? 32% perf-profile.children.cycles-pp.ttwu_do_activate
0.29 ? 18% -0.1 0.17 ? 21% perf-profile.children.cycles-pp.__open64_nocancel
0.35 ? 15% -0.1 0.22 ? 16% perf-profile.children.cycles-pp.get_page_from_freelist
0.30 ? 26% -0.1 0.18 ? 18% perf-profile.children.cycles-pp.pipe_write
0.35 ? 18% -0.1 0.24 ? 24% perf-profile.children.cycles-pp.next_uptodate_page
0.18 ? 23% -0.1 0.07 ? 31% perf-profile.children.cycles-pp.native_set_fixmap
0.26 ? 27% -0.1 0.14 ? 20% perf-profile.children.cycles-pp.__wake_up_common_lock
0.24 ? 11% -0.1 0.14 ? 22% perf-profile.children.cycles-pp.tlb_flush_mmu
0.34 ? 20% -0.1 0.24 ? 23% perf-profile.children.cycles-pp.call_timer_fn
0.22 ? 17% -0.1 0.12 ? 15% perf-profile.children.cycles-pp.filename_lookup
0.24 ? 21% -0.1 0.14 ? 19% perf-profile.children.cycles-pp.release_pages
0.21 ? 17% -0.1 0.12 ? 15% perf-profile.children.cycles-pp.path_lookupat
0.19 ? 18% -0.1 0.10 ? 33% perf-profile.children.cycles-pp.step_into
0.17 ? 30% -0.1 0.08 ? 25% perf-profile.children.cycles-pp.__fput
0.17 ? 26% -0.1 0.08 ? 52% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.23 ? 20% -0.1 0.14 ? 17% perf-profile.children.cycles-pp.elf_map
0.17 ? 13% -0.1 0.09 ? 25% perf-profile.children.cycles-pp.__do_sys_newstat
0.23 ? 14% -0.1 0.14 ? 20% perf-profile.children.cycles-pp.tlb_finish_mmu
0.23 ? 22% -0.1 0.15 ? 27% perf-profile.children.cycles-pp.__split_vma
0.19 ? 13% -0.1 0.10 ? 16% perf-profile.children.cycles-pp.unlink_anon_vmas
0.17 ? 14% -0.1 0.08 ? 26% perf-profile.children.cycles-pp.vfs_statx
0.18 ? 19% -0.1 0.10 ? 23% perf-profile.children.cycles-pp.dput
0.15 ? 31% -0.1 0.07 ? 22% perf-profile.children.cycles-pp.__wake_up_common
0.14 ? 33% -0.1 0.06 ? 49% perf-profile.children.cycles-pp.autoremove_wake_function
0.22 ? 24% -0.1 0.14 ? 18% perf-profile.children.cycles-pp.select_task_rq_fair
0.18 ? 16% -0.1 0.10 ? 27% perf-profile.children.cycles-pp.__mmap
0.21 ? 11% -0.1 0.13 ? 25% perf-profile.children.cycles-pp.do_anonymous_page
0.18 ? 16% -0.1 0.11 ? 12% perf-profile.children.cycles-pp.copy_strings
0.15 ? 24% -0.1 0.07 ? 26% perf-profile.children.cycles-pp.enqueue_entity
0.19 ? 24% -0.1 0.12 ? 29% perf-profile.children.cycles-pp.__vma_adjust
0.17 ? 25% -0.1 0.10 ? 27% perf-profile.children.cycles-pp.select_idle_sibling
0.14 ? 16% -0.1 0.07 ? 20% perf-profile.children.cycles-pp.do_open
0.17 ? 25% -0.1 0.10 ? 22% perf-profile.children.cycles-pp.alloc_empty_file
0.16 ? 17% -0.1 0.09 ? 21% perf-profile.children.cycles-pp.perf_event_mmap
0.22 ? 26% -0.1 0.15 ? 16% perf-profile.children.cycles-pp.pipe_read
0.19 ? 19% -0.1 0.12 ? 30% perf-profile.children.cycles-pp.__x64_sys_mprotect
0.19 ? 19% -0.1 0.12 ? 30% perf-profile.children.cycles-pp.do_mprotect_pkey
0.16 ? 14% -0.1 0.09 ? 22% perf-profile.children.cycles-pp.__x64_sys_poll
0.16 ? 14% -0.1 0.09 ? 22% perf-profile.children.cycles-pp.do_sys_poll
0.15 ? 21% -0.1 0.08 ? 13% perf-profile.children.cycles-pp.unlink_file_vma
0.16 ? 22% -0.1 0.10 ? 22% perf-profile.children.cycles-pp.__lookup_slow
0.09 ? 15% -0.1 0.02 ? 99% perf-profile.children.cycles-pp.filemap_read
0.17 ? 18% -0.1 0.11 ? 27% perf-profile.children.cycles-pp.vma_link
0.13 ? 22% -0.1 0.07 ? 27% perf-profile.children.cycles-pp.d_alloc_parallel
0.09 ? 19% -0.1 0.03 ?101% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.12 ? 25% -0.1 0.06 ? 53% perf-profile.children.cycles-pp.select_idle_cpu
0.20 ? 18% -0.1 0.13 ? 17% perf-profile.children.cycles-pp.page_remove_rmap
0.10 ? 13% -0.1 0.04 ? 72% perf-profile.children.cycles-pp.perf_poll
0.16 ? 21% -0.1 0.10 ? 28% perf-profile.children.cycles-pp.lookup_fast
0.16 ? 17% -0.1 0.10 ? 25% perf-profile.children.cycles-pp.vma_interval_tree_insert
0.17 ? 19% -0.1 0.11 ? 28% perf-profile.children.cycles-pp.mprotect_fixup
0.11 ? 22% -0.1 0.05 ? 51% perf-profile.children.cycles-pp.d_alloc
0.09 ? 19% -0.1 0.03 ?101% perf-profile.children.cycles-pp.available_idle_cpu
0.10 ? 11% -0.1 0.04 ? 75% perf-profile.children.cycles-pp.open64
0.08 ? 20% -0.1 0.03 ?100% perf-profile.children.cycles-pp.anon_vma_clone
0.12 ? 21% -0.1 0.06 ? 51% perf-profile.children.cycles-pp.remove_vma
0.12 ? 16% -0.1 0.07 ? 53% perf-profile.children.cycles-pp.__clear_user
0.16 ? 12% -0.1 0.11 ? 18% perf-profile.children.cycles-pp.prep_new_page
0.14 ? 17% -0.1 0.09 ? 17% perf-profile.children.cycles-pp.dequeue_task_fair
0.14 ? 19% -0.1 0.08 ? 26% perf-profile.children.cycles-pp.rmqueue
0.08 ? 14% -0.1 0.04 ? 71% perf-profile.children.cycles-pp.do_dentry_open
0.12 ? 22% -0.0 0.06 ? 14% perf-profile.children.cycles-pp.vma_interval_tree_remove
0.12 ? 11% -0.0 0.07 ? 23% perf-profile.children.cycles-pp.proc_reg_read_iter
0.12 ? 15% -0.0 0.08 ? 6% perf-profile.children.cycles-pp.get_arg_page
0.12 ? 15% -0.0 0.08 ? 6% perf-profile.children.cycles-pp.__get_user_pages_remote
0.12 ? 14% -0.0 0.07 ? 10% perf-profile.children.cycles-pp.__get_user_pages
0.12 ? 21% -0.0 0.08 ? 17% perf-profile.children.cycles-pp.do_open_execat
0.07 ? 20% -0.0 0.03 ?100% perf-profile.children.cycles-pp.__xstat64
0.11 ? 24% -0.0 0.06 ? 17% perf-profile.children.cycles-pp.anon_vma_fork
0.11 ? 21% -0.0 0.06 ? 23% perf-profile.children.cycles-pp.lru_add_drain_cpu
0.08 ? 14% -0.0 0.04 ? 72% perf-profile.children.cycles-pp.__check_object_size
0.07 ? 13% -0.0 0.03 ?100% perf-profile.children.cycles-pp.__close
0.07 ? 15% -0.0 0.03 ?100% perf-profile.children.cycles-pp.perf_iterate_sb
0.09 ? 15% -0.0 0.04 ? 75% perf-profile.children.cycles-pp.getname_flags
0.12 ? 20% -0.0 0.08 ? 26% perf-profile.children.cycles-pp.lru_add_drain
0.11 ? 20% -0.0 0.07 ? 21% perf-profile.children.cycles-pp.vsnprintf
0.14 ? 13% -0.0 0.10 ? 19% perf-profile.children.cycles-pp.clear_page_erms
0.08 ? 17% -0.0 0.05 ? 47% perf-profile.children.cycles-pp.strnlen_user
0.12 ? 18% -0.0 0.08 ? 17% perf-profile.children.cycles-pp.pte_alloc_one
0.09 ? 21% -0.0 0.06 ? 53% perf-profile.children.cycles-pp.malloc
0.10 ? 14% -0.0 0.07 ? 11% perf-profile.children.cycles-pp.dup_task_struct
0.08 ? 17% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.common_file_perm
0.10 ? 19% -0.0 0.06 ? 14% perf-profile.children.cycles-pp.down_write
0.08 ? 18% -0.0 0.04 ? 47% perf-profile.children.cycles-pp.copy_string_kernel
0.09 ? 9% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.touch_atime
0.11 ? 16% -0.0 0.08 ? 21% perf-profile.children.cycles-pp.___perf_sw_event
0.07 ? 16% -0.0 0.05 ? 47% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.05 ? 51% +0.1 0.11 ? 20% perf-profile.children.cycles-pp.__set_cpus_allowed_ptr
0.04 ? 72% +0.1 0.10 ? 19% perf-profile.children.cycles-pp.affine_move_task
0.24 ? 15% +0.1 0.30 ? 13% perf-profile.children.cycles-pp.rcu_do_batch
0.09 ? 19% +0.1 0.15 ? 22% perf-profile.children.cycles-pp.__x64_sys_sched_setaffinity
0.02 ?144% +0.1 0.11 ? 20% perf-profile.children.cycles-pp.perf_read
0.17 ? 24% +0.1 0.27 ? 33% perf-profile.children.cycles-pp.start_secondary
0.17 ? 21% +0.1 0.28 ? 34% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
0.17 ? 21% +0.1 0.28 ? 34% perf-profile.children.cycles-pp.cpu_startup_entry
0.17 ? 21% +0.1 0.28 ? 34% perf-profile.children.cycles-pp.do_idle
0.13 ? 35% +0.1 0.24 ? 23% perf-profile.children.cycles-pp.drm_fb_helper_damage_work
0.12 ? 38% +0.1 0.23 ? 23% perf-profile.children.cycles-pp.memcpy_toio
0.12 ? 31% +0.1 0.24 ? 31% perf-profile.children.cycles-pp.evsel__read_counter
0.04 ?118% +0.1 0.16 ? 36% perf-profile.children.cycles-pp.cpuidle_enter_state
0.04 ?118% +0.1 0.16 ? 35% perf-profile.children.cycles-pp.cpuidle_enter
0.18 ? 18% +0.1 0.31 ? 24% perf-profile.children.cycles-pp.sched_setaffinity
0.06 ?109% +0.2 0.28 ? 38% perf-profile.children.cycles-pp.collapse_huge_page
0.07 ?109% +0.2 0.31 ? 37% perf-profile.children.cycles-pp.khugepaged
0.06 ?110% +0.2 0.30 ? 38% perf-profile.children.cycles-pp.khugepaged_scan_mm_slot
0.06 ?110% +0.2 0.30 ? 38% perf-profile.children.cycles-pp.khugepaged_scan_pmd
0.04 ? 71% +0.2 0.28 ? 30% perf-profile.children.cycles-pp.get_cpu_idle_time_us
0.04 ? 71% +0.2 0.28 ? 30% perf-profile.children.cycles-pp.get_idle_time
0.26 ? 27% +0.3 0.51 ? 31% perf-profile.children.cycles-pp.read_counters
0.26 ? 27% +0.3 0.52 ? 31% perf-profile.children.cycles-pp.dispatch_events
0.26 ? 27% +0.3 0.52 ? 31% perf-profile.children.cycles-pp.process_interval
0.26 ? 27% +0.3 0.52 ? 31% perf-profile.children.cycles-pp.cmd_stat
0.00 +0.3 0.29 ? 32% perf-profile.children.cycles-pp.get_cpu_iowait_time_us
0.00 +0.3 0.30 ? 32% perf-profile.children.cycles-pp.get_iowait_time
0.76 ? 16% +0.6 1.36 ? 37% perf-profile.children.cycles-pp.rcu_core
0.00 +0.6 0.62 ? 35% perf-profile.children.cycles-pp.sched_clock_local
0.00 +7.8 7.76 ? 28% perf-profile.children.cycles-pp.sched_clock_tick
2.81 ? 12% +11.1 13.92 ? 25% perf-profile.children.cycles-pp.ktime_get_update_offsets_now
74.12 ? 5% +11.8 85.94 ? 3% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
63.64 ? 6% +15.5 79.18 ? 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
63.41 ? 6% +15.7 79.11 ? 5% perf-profile.children.cycles-pp.hrtimer_interrupt
7.49 ? 15% +17.5 25.00 ? 15% perf-profile.children.cycles-pp.clockevents_program_event
3.42 ? 9% +39.4 42.77 ? 6% perf-profile.children.cycles-pp.ktime_get
0.00 +50.9 50.86 ? 10% perf-profile.children.cycles-pp.read_hpet
4.26 ? 17% -3.1 1.20 ? 23% perf-profile.self.cycles-pp.ghes_notify_nmi
3.39 ? 14% -2.6 0.80 ? 16% perf-profile.self.cycles-pp.perf_event_task_tick
4.11 ? 27% -2.4 1.70 ? 29% perf-profile.self.cycles-pp.hrtimer_active
3.52 ? 24% -2.2 1.32 ? 34% perf-profile.self.cycles-pp.irqtime_account_irq
2.43 ? 19% -2.2 0.25 ? 31% perf-profile.self.cycles-pp.__intel_pmu_enable_all
3.14 ? 24% -2.0 1.18 ? 36% perf-profile.self.cycles-pp.clockevents_program_event
1.68 ? 28% -1.4 0.23 ? 30% perf-profile.self.cycles-pp.update_process_times
2.38 ? 21% -1.4 1.01 ? 38% perf-profile.self.cycles-pp.task_tick_fair
2.39 ? 16% -1.3 1.08 ? 19% perf-profile.self.cycles-pp.ktime_get_update_offsets_now
1.76 ? 32% -1.2 0.54 ? 30% perf-profile.self.cycles-pp.rcu_sched_clock_irq
2.48 ? 20% -1.0 1.48 ? 31% perf-profile.self.cycles-pp.native_sched_clock
1.86 ? 20% -0.9 0.92 ? 30% perf-profile.self.cycles-pp.update_cfs_group
1.70 ? 10% -0.9 0.78 ? 26% perf-profile.self.cycles-pp.native_flush_tlb_one_user
1.23 ? 6% -0.8 0.43 ? 13% perf-profile.self.cycles-pp._raw_spin_lock
1.48 ? 37% -0.7 0.73 ? 25% perf-profile.self.cycles-pp.paranoid_entry
1.08 ? 23% -0.7 0.34 ? 34% perf-profile.self.cycles-pp.nmi_handle
0.98 ? 12% -0.7 0.25 ? 25% perf-profile.self.cycles-pp.trigger_load_balance
0.82 ? 31% -0.7 0.15 ? 28% perf-profile.self.cycles-pp.sync_regs
0.76 ? 23% -0.5 0.26 ? 34% perf-profile.self.cycles-pp.arch_scale_freq_tick
0.67 ? 10% -0.5 0.18 ? 10% perf-profile.self.cycles-pp.__hrtimer_run_queues
0.99 ? 15% -0.4 0.58 ? 30% perf-profile.self.cycles-pp.acpi_os_read_memory
0.57 ? 26% -0.4 0.16 ? 12% perf-profile.self.cycles-pp.update_irq_load_avg
0.51 ? 25% -0.4 0.11 ? 34% perf-profile.self.cycles-pp.account_process_tick
0.96 ? 11% -0.3 0.61 ? 9% perf-profile.self.cycles-pp.update_sd_lb_stats
0.55 ? 10% -0.3 0.21 ? 30% perf-profile.self.cycles-pp.scheduler_tick
0.44 ? 10% -0.3 0.10 ? 25% perf-profile.self.cycles-pp.ghes_copy_tofrom_phys
0.36 ? 20% -0.3 0.04 ? 72% perf-profile.self.cycles-pp.update_rq_clock
0.65 ? 15% -0.3 0.33 ? 11% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.61 ? 12% -0.3 0.29 ? 31% perf-profile.self.cycles-pp.hrtimer_interrupt
0.50 ? 6% -0.3 0.21 ? 14% perf-profile.self.cycles-pp.__hrtimer_next_event_base
0.40 ? 12% -0.3 0.12 ? 16% perf-profile.self.cycles-pp.timerqueue_add
0.59 ? 4% -0.3 0.32 ? 8% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.36 ? 49% -0.3 0.10 ? 21% perf-profile.self.cycles-pp.native_irq_return_iret
0.43 ? 14% -0.3 0.17 ? 25% perf-profile.self.cycles-pp.__native_set_fixmap
0.30 ? 29% -0.3 0.05 ?111% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
0.43 ? 66% -0.2 0.18 ? 44% perf-profile.self.cycles-pp.exc_nmi
0.33 ? 40% -0.2 0.12 ? 41% perf-profile.self.cycles-pp.__intel_pmu_disable_all
0.28 ? 15% -0.2 0.07 ? 48% perf-profile.self.cycles-pp.tick_sched_timer
0.27 ? 13% -0.2 0.08 ? 25% perf-profile.self.cycles-pp.tick_program_event
0.22 ? 17% -0.2 0.04 ? 72% perf-profile.self.cycles-pp.nmi_restore
0.21 ? 25% -0.2 0.05 ? 72% perf-profile.self.cycles-pp.account_user_time
0.24 ? 6% -0.2 0.08 ? 5% perf-profile.self.cycles-pp.hrtimer_forward
0.22 ? 8% -0.1 0.08 ? 24% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.26 ? 9% -0.1 0.12 ? 31% perf-profile.self.cycles-pp.run_posix_cpu_timers
0.22 ? 10% -0.1 0.09 ? 17% perf-profile.self.cycles-pp.timerqueue_del
0.18 ? 13% -0.1 0.05 ? 47% perf-profile.self.cycles-pp.__remove_hrtimer
0.18 ? 31% -0.1 0.06 ? 46% perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
0.18 ? 13% -0.1 0.06 ? 48% perf-profile.self.cycles-pp.hrtimer_update_next_event
0.18 ? 23% -0.1 0.07 ? 31% perf-profile.self.cycles-pp.native_set_fixmap
0.20 ? 7% -0.1 0.09 ? 20% perf-profile.self.cycles-pp.__hrtimer_get_next_event
0.16 ? 8% -0.1 0.06 ? 17% perf-profile.self.cycles-pp.rb_insert_color
0.12 ? 27% -0.1 0.05 ? 71% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.08 ? 17% -0.0 0.03 ?100% perf-profile.self.cycles-pp.available_idle_cpu
0.14 ? 10% -0.0 0.09 ? 21% perf-profile.self.cycles-pp.next_uptodate_page
0.08 ? 14% -0.0 0.03 ?101% perf-profile.self.cycles-pp.finish_task_switch
0.07 ? 9% -0.0 0.03 ?100% perf-profile.self.cycles-pp.find_busiest_group
0.09 ? 11% -0.0 0.05 ? 47% perf-profile.self.cycles-pp.kmem_cache_alloc
0.10 ? 11% -0.0 0.07 ? 16% perf-profile.self.cycles-pp.zap_pte_range
0.04 ? 44% +0.0 0.07 ? 14% perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
0.04 ? 71% +0.1 0.13 ? 12% perf-profile.self.cycles-pp.run_rebalance_domains
0.02 ?142% +0.1 0.11 ? 23% perf-profile.self.cycles-pp.memcpy_toio
0.00 +0.3 0.26 ? 31% perf-profile.self.cycles-pp.sched_clock_local
0.15 ? 18% +0.3 0.43 ? 15% perf-profile.self.cycles-pp.__softirqentry_text_start
2.38 ? 7% +1.5 3.88 ? 7% perf-profile.self.cycles-pp.ktime_get
0.00 +46.9 46.94 ? 12% perf-profile.self.cycles-pp.read_hpet



stress-ng.time.user_time

6000 +--------------------------------------------------------------------+
| |
5000 |OOOO O OOOOOO OOOO O OOOOOO O O OO OO OO OO O OOO |
| OO O OOO O O O OO O OO O O OO|
| + + |
4000 |++++++.+++ ++++ ++++.+ :+ ++ +++ + ++ |
| + + + : : + + ++ |
3000 |-+ : : |
| : : |
2000 |-+ : : |
| : |
| : |
1000 |-+ : |
| : |
0 +--------------------------------------------------------------------+


stress-ng.time.percent_of_cpu_this_job_got

9000 +--------------------------------------------------------------------+
|OOOO O OOOOOO OOOOOO O OOOOOOOOOO O O OOO OO OOOOOOOOOOOOO OOOOOO|
8000 |-+ |
7000 |-+ + |
|+ ++++.+++ ++++ ++++.+ :+ ++ ++++. +++ |
6000 |-+ + + : : + + + |
5000 |-+ : : |
| : : |
4000 |-+ : : |
3000 |-+ : |
| : |
2000 |-+ : |
1000 |-+ : |
| : |
0 +--------------------------------------------------------------------+


stress-ng.lockbus.ops

300000 +------------------------------------------------------------------+
| +. |
250000 |++++++++++.++++++++++ +++++++++ ++++ |
| O O O OO: :OOOO OO O O OOO OO OO OO OOO O OOOOOOO|
|OOOO OOOO O OO O : : O O O O O OO |
200000 |-+ : : |
| : : |
150000 |-+ : : |
| :: |
100000 |-+ : |
| : |
| : |
50000 |-+ : |
| : |
0 +------------------------------------------------------------------+


stress-ng.lockbus.ops_per_sec

4500 +--------------------------------------------------------------------+
|++++++.+++++++++++++.+ +++++++++ ++++ |
4000 |OOOO O OOOO OOOOOO : :OOOOOOOOO O O OOO OO OO OO OOOOO O OOOOOO|
3500 |-+ O O : : O O O |
| : : |
3000 |-+ : : |
2500 |-+ : : |
| : : |
2000 |-+ : |
1500 |-+ : |
| : |
1000 |-+ : |
500 |-+ : |
| : |
0 +--------------------------------------------------------------------+


[*] bisect-good sample
[O] bisect-bad sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation

Thanks,
Oliver Sang


Attachments:
(No filename) (61.98 kB)
config-5.13.0-rc4-00023-gdb3a34e17433 (176.94 kB)
job-script (8.19 kB)
job.yaml (5.52 kB)
reproduce (350.00 B)
Download all attachments