2022-07-04 15:03:28

by kernel test robot

[permalink] [raw]
Subject: [sched] f3dd3f6745: fxmark.ssd_ext4_no_jnl_DWSL_72_directio.works/sec -33.0% regression



Greeting,

FYI, we noticed a -33.0% regression of fxmark.ssd_ext4_no_jnl_DWSL_72_directio.works/sec due to commit:


commit: f3dd3f674555bd9455c5ae7fafce0696bd9931b3 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core

in testcase: fxmark
on test machine: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with 128G memory
with following parameters:

disk: 1SSD
media: ssd
test: DWSL
fstype: ext4_no_jnl
directio: directio
cpufreq_governor: performance
ucode: 0xd000331

test-description: FxMark is a filesystem benchmark that test multicore scalability.
test-url: https://github.com/sslab-gatech/fxmark

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_tps 1.3% improvement |
| test machine | 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G memory |
| test parameters | cluster=cs-localhost |
| | cpufreq_governor=performance |
| | ip=ipv4 |
| | nr_threads=25% |
| | runtime=300s |
| | test=UDP_RR |
| | ucode=0xd000331 |
+------------------+---------------------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min 246.4% improvement |
| test machine | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory |
| test parameters | cpufreq_governor=performance |
| | disk=4BRD_12G |
| | fs=f2fs |
| | load=100 |
| | md=RAID1 |
| | test=sync_disk_rw |
| | ucode=0x500320a |
+------------------+---------------------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min 3.9% improvement |
| test machine | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory |
| test parameters | cpufreq_governor=performance |
| | disk=4BRD_12G |
| | fs=xfs |
| | load=300 |
| | md=RAID1 |
| | test=sync_disk_rw |
| | ucode=0x500320a |
+------------------+---------------------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput 7.2% improvement |
| test machine | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory |
| test parameters | cpufreq_governor=performance |
| | ipc=pipe |
| | iterations=8 |
| | mode=process |
| | nr_threads=50% |
| | ucode=0x500320a |
+------------------+---------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
sudo bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
sudo bin/lkp run generated-yaml-file

# if come across any failure that blocks the test,
# please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
gcc-11/performance/directio/1SSD/ext4_no_jnl/x86_64-rhel-8.3/ssd/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp5/DWSL/fxmark/0xd000331

commit:
28156108fe ("sched: Fix the check of nr_running at queue wakelist")
f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc
---------------- ---------------------------
%stddev %change %stddev
\ | \
518.88 ? 5% -99.9% 0.48 fxmark.ssd_ext4_no_jnl_DWSL_18_directio.idle_sec
57.91 ? 5% -99.9% 0.05 fxmark.ssd_ext4_no_jnl_DWSL_18_directio.idle_util
348.18 ? 8% +148.6% 865.70 fxmark.ssd_ext4_no_jnl_DWSL_18_directio.iowait_sec
38.86 ? 8% +148.6% 96.59 fxmark.ssd_ext4_no_jnl_DWSL_18_directio.iowait_util
3.62 -25.6% 2.70 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_18_directio.softirq_sec
0.40 -25.6% 0.30 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_18_directio.softirq_util
37.96 ? 2% -99.9% 0.03 fxmark.ssd_ext4_no_jnl_DWSL_2_directio.idle_sec
38.38 ? 2% -99.9% 0.03 fxmark.ssd_ext4_no_jnl_DWSL_2_directio.idle_util
53.08 ? 2% +72.0% 91.31 fxmark.ssd_ext4_no_jnl_DWSL_2_directio.iowait_sec
53.66 ? 2% +71.6% 92.06 fxmark.ssd_ext4_no_jnl_DWSL_2_directio.iowait_util
0.98 ? 5% -15.1% 0.83 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_2_directio.softirq_sec
0.99 ? 5% -15.4% 0.84 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_2_directio.softirq_util
865.64 ? 5% -99.9% 0.98 fxmark.ssd_ext4_no_jnl_DWSL_36_directio.idle_sec
48.76 ? 5% -99.9% 0.06 fxmark.ssd_ext4_no_jnl_DWSL_36_directio.idle_util
854.04 ? 5% +100.5% 1712 fxmark.ssd_ext4_no_jnl_DWSL_36_directio.iowait_sec
48.11 ? 5% +100.5% 96.47 fxmark.ssd_ext4_no_jnl_DWSL_36_directio.iowait_util
8.44 -46.7% 4.50 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_36_directio.softirq_sec
0.48 -46.7% 0.25 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_36_directio.softirq_util
28.81 +31.1% 37.76 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_36_directio.sys_sec
1.62 +31.1% 2.13 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_36_directio.sys_util
975762 -23.4% 747169 ? 3% fxmark.ssd_ext4_no_jnl_DWSL_36_directio.works
19514 -23.4% 14942 ? 3% fxmark.ssd_ext4_no_jnl_DWSL_36_directio.works/sec
112.89 -99.9% 0.08 ? 5% fxmark.ssd_ext4_no_jnl_DWSL_4_directio.idle_sec
56.95 -99.9% 0.04 ? 5% fxmark.ssd_ext4_no_jnl_DWSL_4_directio.idle_util
75.46 ? 2% +150.0% 188.64 fxmark.ssd_ext4_no_jnl_DWSL_4_directio.iowait_sec
38.07 +149.7% 95.07 fxmark.ssd_ext4_no_jnl_DWSL_4_directio.iowait_util
1.34 ? 3% -20.2% 1.07 ? 4% fxmark.ssd_ext4_no_jnl_DWSL_4_directio.softirq_sec
0.68 ? 3% -20.3% 0.54 ? 4% fxmark.ssd_ext4_no_jnl_DWSL_4_directio.softirq_util
508.25 ? 9% -99.7% 1.46 fxmark.ssd_ext4_no_jnl_DWSL_54_directio.idle_sec
19.07 ? 9% -99.7% 0.05 fxmark.ssd_ext4_no_jnl_DWSL_54_directio.idle_util
2075 ? 2% +25.1% 2595 fxmark.ssd_ext4_no_jnl_DWSL_54_directio.iowait_sec
77.86 ? 2% +24.5% 96.91 fxmark.ssd_ext4_no_jnl_DWSL_54_directio.iowait_util
9.07 ? 4% -31.1% 6.26 ? 4% fxmark.ssd_ext4_no_jnl_DWSL_54_directio.softirq_sec
0.34 ? 4% -31.4% 0.23 ? 4% fxmark.ssd_ext4_no_jnl_DWSL_54_directio.softirq_util
816890 -29.5% 575621 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_54_directio.works
16336 -29.5% 11511 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_54_directio.works/sec
507.19 ? 10% -99.6% 1.97 fxmark.ssd_ext4_no_jnl_DWSL_72_directio.idle_sec
14.21 ? 10% -99.6% 0.06 fxmark.ssd_ext4_no_jnl_DWSL_72_directio.idle_util
2974 +17.2% 3486 fxmark.ssd_ext4_no_jnl_DWSL_72_directio.iowait_sec
83.34 +16.9% 97.44 fxmark.ssd_ext4_no_jnl_DWSL_72_directio.iowait_util
31.28 +23.0% 38.50 ? 8% fxmark.ssd_ext4_no_jnl_DWSL_72_directio.irq_sec
0.88 +22.7% 1.08 ? 8% fxmark.ssd_ext4_no_jnl_DWSL_72_directio.irq_util
9.45 -29.6% 6.66 ? 3% fxmark.ssd_ext4_no_jnl_DWSL_72_directio.softirq_sec
0.26 -29.7% 0.19 ? 3% fxmark.ssd_ext4_no_jnl_DWSL_72_directio.softirq_util
826258 -33.0% 553194 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_72_directio.works
16523 -33.0% 11062 ? 2% fxmark.ssd_ext4_no_jnl_DWSL_72_directio.works/sec
1930279 ? 3% -6.2% 1811239 ? 2% fxmark.time.voluntary_context_switches
55807773 -11.6% 49357289 ? 4% cpuidle..usage
53054978 -11.7% 46862001 proc-vmstat.pgpgout
830.64 ? 3% -21.2% 654.40 ? 5% uptime.idle
46.17 ? 2% -46.7% 24.59 ? 2% iostat.cpu.idle
48.27 ? 2% +44.6% 69.80 iostat.cpu.iowait
47.91 ? 2% -22.1 25.77 ? 2% mpstat.cpu.all.idle%
47.73 ? 2% +22.1 69.87 mpstat.cpu.all.iowait%
0.39 ? 3% -0.1 0.31 ? 2% mpstat.cpu.all.soft%
98555 -11.8% 86971 ? 3% vmstat.io.bo
102932 -19.4% 82934 ? 2% vmstat.system.cs
76098 -7.5% 70394 ? 6% vmstat.system.in
102783 -19.4% 82894 ? 2% perf-stat.i.context-switches
54.23 -5.3% 51.38 perf-stat.i.cpu-migrations
102570 -19.4% 82720 ? 2% perf-stat.ps.context-switches
54.15 -5.2% 51.31 perf-stat.ps.cpu-migrations
1.39 ? 5% -13.0% 1.21 ? 7% sched_debug.cfs_rq:/.h_nr_running.max
1095631 ?166% -90.9% 100111 ? 17% sched_debug.cfs_rq:/.load.avg
1990 ? 9% -22.9% 1535 ? 17% sched_debug.cfs_rq:/.load_avg.max
306.85 ? 10% -17.4% 253.49 ? 15% sched_debug.cfs_rq:/.load_avg.stddev
6.20 ? 9% -21.5% 4.86 ? 25% sched_debug.cfs_rq:/.util_est_enqueued.avg
2873 ? 10% -29.5% 2026 ? 10% sched_debug.cpu.clock_task.stddev
3837026 ? 7% -23.4% 2940395 ? 8% sched_debug.cpu.nr_switches.max
43.97 ? 42% -13.1 30.88 ? 10% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
50.19 ? 33% -12.7 37.50 ? 14% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
49.35 ? 33% -12.4 36.96 ? 14% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
49.34 ? 33% -12.4 36.96 ? 14% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.41 ? 60% -0.6 0.78 ? 15% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
0.00 +0.8 0.75 ? 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.dd_insert_requests.blk_mq_sched_insert_request.blk_mq_submit_bio
0.00 +0.8 0.80 ? 10% perf-profile.calltrace.cycles-pp._raw_spin_lock.dd_insert_requests.blk_mq_sched_insert_request.blk_mq_submit_bio.submit_bio_noacct_nocheck
0.08 ?223% +0.8 0.93 ? 13% perf-profile.calltrace.cycles-pp.dd_insert_requests.blk_mq_sched_insert_request.blk_mq_submit_bio.submit_bio_noacct_nocheck.submit_bh_wbc
0.08 ?223% +0.9 0.94 ? 13% perf-profile.calltrace.cycles-pp.blk_mq_sched_insert_request.blk_mq_submit_bio.submit_bio_noacct_nocheck.submit_bh_wbc.__sync_dirty_buffer
44.82 ? 42% -13.4 31.39 ? 10% perf-profile.children.cycles-pp.cpuidle_idle_call
50.17 ? 33% -12.7 37.48 ? 14% perf-profile.children.cycles-pp.do_idle
50.19 ? 33% -12.7 37.50 ? 14% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
50.19 ? 33% -12.7 37.50 ? 14% perf-profile.children.cycles-pp.cpu_startup_entry
49.35 ? 33% -12.4 36.96 ? 14% perf-profile.children.cycles-pp.start_secondary
24.38 ? 33% -6.7 17.72 ? 14% perf-profile.children.cycles-pp.mwait_idle_with_hints
1.78 ? 59% -0.8 1.01 ? 11% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
1.45 ? 59% -0.6 0.81 ? 15% perf-profile.children.cycles-pp.tick_nohz_next_event
0.70 ? 77% -0.4 0.32 ? 31% perf-profile.children.cycles-pp.timekeeping_max_deferment
0.15 ? 13% -0.1 0.09 ? 46% perf-profile.children.cycles-pp.nr_iowait_cpu
0.38 ? 45% +0.6 0.94 ? 13% perf-profile.children.cycles-pp.blk_mq_sched_insert_request
0.66 ? 45% +0.8 1.47 ? 16% perf-profile.children.cycles-pp.dd_insert_requests
0.70 ? 77% -0.4 0.32 ? 30% perf-profile.self.cycles-pp.timekeeping_max_deferment
0.26 ? 63% -0.1 0.14 ? 10% perf-profile.self.cycles-pp.tick_nohz_next_event


***************************************************************************************************
lkp-icl-2sp2: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
cs-localhost/gcc-11/performance/ipv4/x86_64-rhel-8.3/25%/debian-11.1-x86_64-20220510.cgz/300s/lkp-icl-2sp2/UDP_RR/netperf/0xd000331

commit:
28156108fe ("sched: Fix the check of nr_running at queue wakelist")
f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc
---------------- ---------------------------
%stddev %change %stddev
\ | \
2897681 +1.3% 2934351 netperf.Throughput_total_tps
90552 +1.3% 91698 netperf.Throughput_tps
10378 +4.8% 10881 netperf.time.minor_page_faults
1511 -39.2% 918.83 netperf.time.percent_of_cpu_this_job_got
4215 -39.9% 2532 netperf.time.system_time
337.04 -30.3% 235.04 netperf.time.user_time
8.693e+08 +1.3% 8.803e+08 netperf.time.voluntary_context_switches
8.693e+08 +1.3% 8.803e+08 netperf.workload
20979 ? 4% -9.0% 19095 ? 5% proc-vmstat.pgactivate
73.20 +3.1 76.26 turbostat.C1%
0.78 +2.9 3.64 mpstat.cpu.all.irq%
6.06 -1.6 4.47 mpstat.cpu.all.soft%
16.74 -2.0 14.74 mpstat.cpu.all.sys%
38.00 -28.1% 27.33 vmstat.procs.r
11439523 +1.3% 11584593 vmstat.system.cs
5951961 +1.2% 6025940 vmstat.system.in
119282 ? 8% -75.4% 29291 ? 20% sched_debug.cfs_rq:/.MIN_vruntime.avg
1541193 ? 12% -63.0% 570028 ? 9% sched_debug.cfs_rq:/.MIN_vruntime.max
354476 ? 7% -68.6% 111435 ? 14% sched_debug.cfs_rq:/.MIN_vruntime.stddev
0.32 ? 3% -21.1% 0.26 ? 5% sched_debug.cfs_rq:/.h_nr_running.avg
9525 ? 3% +20.0% 11432 ? 4% sched_debug.cfs_rq:/.load.avg
33675 ? 10% +114.1% 72104 ? 87% sched_debug.cfs_rq:/.load.max
12820 ? 2% +57.4% 20178 ? 26% sched_debug.cfs_rq:/.load.stddev
119282 ? 8% -75.4% 29291 ? 20% sched_debug.cfs_rq:/.max_vruntime.avg
1541193 ? 12% -63.0% 570028 ? 9% sched_debug.cfs_rq:/.max_vruntime.max
354476 ? 7% -68.6% 111435 ? 14% sched_debug.cfs_rq:/.max_vruntime.stddev
1104703 ? 10% -59.1% 451566 ? 10% sched_debug.cfs_rq:/.min_vruntime.avg
1722807 ? 12% -54.9% 776144 ? 20% sched_debug.cfs_rq:/.min_vruntime.max
459283 ? 14% -50.0% 229644 ? 22% sched_debug.cfs_rq:/.min_vruntime.min
288286 ? 12% -67.7% 93188 ? 9% sched_debug.cfs_rq:/.min_vruntime.stddev
0.32 ? 2% -21.1% 0.26 ? 5% sched_debug.cfs_rq:/.nr_running.avg
296.34 -30.8% 204.95 ? 2% sched_debug.cfs_rq:/.runnable_avg.avg
263.94 -27.6% 191.18 sched_debug.cfs_rq:/.runnable_avg.stddev
810487 ? 23% -51.7% 391282 ? 33% sched_debug.cfs_rq:/.spread0.max
-453070 -65.7% -155225 sched_debug.cfs_rq:/.spread0.min
288295 ? 12% -67.7% 93193 ? 9% sched_debug.cfs_rq:/.spread0.stddev
295.88 -30.8% 204.74 ? 2% sched_debug.cfs_rq:/.util_avg.avg
263.59 -27.5% 191.03 sched_debug.cfs_rq:/.util_avg.stddev
156.07 ? 3% -50.1% 77.80 ? 6% sched_debug.cfs_rq:/.util_est_enqueued.avg
735.13 ? 5% -11.2% 652.93 ? 8% sched_debug.cfs_rq:/.util_est_enqueued.max
224.80 -36.1% 143.75 ? 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev
1094 ? 3% -13.0% 951.53 ? 7% sched_debug.cpu.curr->pid.avg
0.28 ? 2% -24.9% 0.21 ? 6% sched_debug.cpu.nr_running.avg
9.67 -26.9% 7.07 perf-stat.i.MPKI
1.954e+10 +3.9% 2.031e+10 perf-stat.i.branch-instructions
1.33 -0.1 1.27 perf-stat.i.branch-miss-rate%
1.25 ? 2% +0.3 1.57 perf-stat.i.cache-miss-rate%
9.408e+08 -24.0% 7.152e+08 perf-stat.i.cache-references
11524667 +1.3% 11671113 perf-stat.i.context-switches
1.29 -4.6% 1.23 perf-stat.i.cpi
366.03 +7.3% 392.62 perf-stat.i.cpu-migrations
2.865e+10 +4.1% 2.984e+10 perf-stat.i.dTLB-loads
1.744e+10 +3.7% 1.809e+10 perf-stat.i.dTLB-stores
9.917e+10 +3.9% 1.03e+11 perf-stat.i.instructions
0.79 +4.7% 0.82 perf-stat.i.ipc
520.11 +3.6% 538.63 perf-stat.i.metric.M/sec
2630922 -4.1% 2523585 perf-stat.i.node-store-misses
9.49 -26.8% 6.94 perf-stat.overall.MPKI
1.29 -0.1 1.23 perf-stat.overall.branch-miss-rate%
1.18 ? 3% +0.3 1.50 perf-stat.overall.cache-miss-rate%
1.27 -4.5% 1.21 perf-stat.overall.cpi
0.79 +4.7% 0.82 perf-stat.overall.ipc
34314 +2.6% 35206 perf-stat.overall.path-length
1.948e+10 +3.9% 2.024e+10 perf-stat.ps.branch-instructions
9.376e+08 -24.0% 7.128e+08 perf-stat.ps.cache-references
11486058 +1.3% 11632058 perf-stat.ps.context-switches
365.07 +7.3% 391.61 perf-stat.ps.cpu-migrations
2.856e+10 +4.1% 2.974e+10 perf-stat.ps.dTLB-loads
1.738e+10 +3.7% 1.803e+10 perf-stat.ps.dTLB-stores
9.884e+10 +3.9% 1.027e+11 perf-stat.ps.instructions
2622155 -4.1% 2515203 perf-stat.ps.node-store-misses
2.983e+13 +3.9% 3.099e+13 perf-stat.total.instructions
48.27 -48.3 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_reschedule_ipi.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
18.85 -4.4 14.41 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg
19.51 -4.4 15.08 perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
19.97 -4.4 15.55 perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
15.81 -4.4 11.39 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_send_skb.udp_send_skb
16.00 -4.4 11.60 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg
30.37 -4.4 25.98 perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
31.58 -4.4 27.21 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
15.53 -4.4 11.17 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_send_skb
29.41 -4.3 25.08 perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
13.59 -4.0 9.59 ? 2% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
14.18 -4.0 10.18 ? 2% perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
13.68 -4.0 9.70 ? 2% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
10.94 -3.7 7.20 ? 2% perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
10.86 -3.7 7.13 ? 2% perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
9.47 -3.7 5.82 ? 2% perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv
9.76 -3.6 6.12 ? 2% perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu
11.76 -3.6 8.12 perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
9.20 -3.6 5.56 ? 2% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb
11.88 -3.6 8.25 ? 2% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
8.12 -3.6 4.49 ? 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
11.94 -3.6 8.31 ? 2% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
8.94 -3.6 5.32 ? 2% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb
8.28 -3.6 4.68 ? 2% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb
12.54 -3.4 9.13 ? 2% perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
26.78 ? 3% -3.1 23.71 ? 3% perf-profile.calltrace.cycles-pp.send_udp_rr.main.__libc_start_main
26.50 ? 3% -3.1 23.44 ? 3% perf-profile.calltrace.cycles-pp.send_omni_inner.send_udp_rr.main.__libc_start_main
15.83 ? 3% -2.6 13.24 ? 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto.send_omni_inner.send_udp_rr.main
15.98 ? 3% -2.5 13.48 ? 3% perf-profile.calltrace.cycles-pp.sendto.send_omni_inner.send_udp_rr.main.__libc_start_main
15.88 -2.4 13.50 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.send_omni_inner
16.10 -2.4 13.73 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.send_omni_inner.send_udp_rr
5.76 -2.3 3.44 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
6.01 -2.3 3.69 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
15.89 -2.0 13.90 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.recv_omni
16.11 -2.0 14.12 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.recv_omni.process_requests
16.27 -2.0 14.29 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto.recv_omni.process_requests.spawn_child
16.51 -2.0 14.54 perf-profile.calltrace.cycles-pp.sendto.recv_omni.process_requests.spawn_child.accept_connection
2.97 -1.8 1.19 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
2.16 -1.3 0.84 perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_idle
6.30 -1.3 4.99 ? 2% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp
6.46 -1.2 5.24 ? 2% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
6.54 -1.1 5.47 ? 2% perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg
8.34 -0.9 7.48 ? 2% perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom
10.27 -0.8 9.44 perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
14.52 -0.6 13.91 perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
14.60 -0.6 14.00 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
16.43 -0.6 15.88 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom
8.58 ? 2% -0.4 8.18 ? 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom.send_omni_inner.send_udp_rr
2.41 -0.2 2.19 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets
8.36 -0.2 8.16 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom.recv_omni
2.08 -0.2 1.89 perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
8.65 -0.2 8.46 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom.recv_omni.process_requests
8.79 -0.2 8.61 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvfrom.recv_omni.process_requests.spawn_child
9.72 -0.1 9.58 perf-profile.calltrace.cycles-pp.recvfrom.recv_omni.process_requests.spawn_child.accept_connection
2.13 +0.0 2.18 perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
0.62 +0.3 0.96 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
0.00 +0.5 0.51 perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule_idle.do_idle
0.00 +0.5 0.52 ? 2% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending
0.00 +0.8 0.77 perf-profile.calltrace.cycles-pp.resched_curr.check_preempt_curr.ttwu_do_wakeup.sched_ttwu_pending.__sysvec_call_function_single
0.00 +0.8 0.82 perf-profile.calltrace.cycles-pp.update_rq_clock.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
0.00 +0.9 0.86 ? 2% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry
0.00 +0.9 0.88 perf-profile.calltrace.cycles-pp.irqentry_enter.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter
0.00 +0.9 0.88 perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single
0.00 +0.9 0.94 perf-profile.calltrace.cycles-pp.llist_reverse_order.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
0.00 +0.9 0.94 perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
0.00 +1.2 1.20 ? 2% perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter
0.00 +1.6 1.61 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single
0.00 +1.7 1.71 ? 2% perf-profile.calltrace.cycles-pp.ttwu_queue_wakelist.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.00 +1.8 1.81 perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry
0.00 +2.0 1.97 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single
0.00 +2.1 2.06 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
0.00 +4.3 4.35 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry
42.79 +4.4 47.19 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
43.00 +4.4 47.41 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
43.06 +4.4 47.46 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
43.31 +4.4 47.73 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
0.00 +6.8 6.81 perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter
6.60 +7.3 13.93 perf-profile.calltrace.cycles-pp.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
31.15 +7.5 38.70 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
29.32 +7.6 36.88 perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
30.97 +7.6 38.54 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
34.13 +7.6 41.74 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +9.7 9.67 perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state
0.00 +56.3 56.35 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
25.79 -25.8 0.00 perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
49.81 -4.6 45.22 perf-profile.children.cycles-pp.do_syscall_64
50.35 -4.6 45.78 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
15.98 -4.5 11.51 perf-profile.children.cycles-pp.__softirqentry_text_start
16.04 -4.4 11.59 perf-profile.children.cycles-pp.do_softirq
16.31 -4.4 11.90 perf-profile.children.cycles-pp.__local_bh_enable_ip
19.02 -4.4 14.66 perf-profile.children.cycles-pp.ip_finish_output2
19.69 -4.4 15.34 perf-profile.children.cycles-pp.ip_send_skb
20.14 -4.3 15.81 perf-profile.children.cycles-pp.udp_send_skb
29.63 -4.2 25.43 perf-profile.children.cycles-pp.udp_sendmsg
30.46 -4.2 26.29 perf-profile.children.cycles-pp.sock_sendmsg
31.69 -4.1 27.54 perf-profile.children.cycles-pp.__sys_sendto
31.90 -4.1 27.75 perf-profile.children.cycles-pp.__x64_sys_sendto
33.74 -4.1 29.64 perf-profile.children.cycles-pp.sendto
14.01 -4.1 9.95 perf-profile.children.cycles-pp.__napi_poll
14.48 -4.0 10.45 perf-profile.children.cycles-pp.net_rx_action
13.86 -4.0 9.83 perf-profile.children.cycles-pp.process_backlog
9.73 -3.7 6.03 perf-profile.children.cycles-pp.sock_def_readable
8.39 -3.7 4.69 perf-profile.children.cycles-pp.try_to_wake_up
10.04 -3.7 6.35 perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb
11.14 -3.7 7.46 perf-profile.children.cycles-pp.udp_unicast_rcv_skb
9.47 -3.7 5.78 perf-profile.children.cycles-pp.__wake_up_common_lock
11.09 -3.7 7.41 perf-profile.children.cycles-pp.udp_queue_rcv_one_skb
9.19 -3.7 5.51 perf-profile.children.cycles-pp.__wake_up_common
12.00 -3.7 8.33 perf-profile.children.cycles-pp.__udp4_lib_rcv
8.52 -3.7 4.86 perf-profile.children.cycles-pp.autoremove_wake_function
12.42 -3.7 8.77 perf-profile.children.cycles-pp.__schedule
12.16 -3.6 8.52 perf-profile.children.cycles-pp.ip_local_deliver_finish
12.11 -3.6 8.46 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
12.78 -3.4 9.36 perf-profile.children.cycles-pp.__netif_receive_skb_one_core
27.78 -2.6 25.14 perf-profile.children.cycles-pp.send_udp_rr
27.69 -2.4 25.32 perf-profile.children.cycles-pp.send_omni_inner
6.07 -2.3 3.74 perf-profile.children.cycles-pp.schedule_idle
27.42 -2.1 25.31 perf-profile.children.cycles-pp.accept_connection
27.42 -2.1 25.31 perf-profile.children.cycles-pp.spawn_child
27.42 -2.1 25.31 perf-profile.children.cycles-pp.process_requests
27.40 -2.1 25.30 perf-profile.children.cycles-pp.recv_omni
3.50 -1.6 1.94 ? 2% perf-profile.children.cycles-pp.pick_next_task_fair
2.25 -1.3 0.93 perf-profile.children.cycles-pp.set_next_entity
6.65 -1.2 5.43 perf-profile.children.cycles-pp.schedule
2.41 -1.1 1.29 perf-profile.children.cycles-pp.update_rq_clock
1.08 -1.0 0.12 ? 4% perf-profile.children.cycles-pp.tick_nohz_idle_exit
7.25 -0.9 6.37 perf-profile.children.cycles-pp.schedule_timeout
8.59 -0.8 7.75 perf-profile.children.cycles-pp.__skb_wait_for_more_packets
10.53 -0.8 9.74 perf-profile.children.cycles-pp.__skb_recv_udp
14.82 -0.7 14.16 perf-profile.children.cycles-pp.udp_recvmsg
14.88 -0.7 14.23 perf-profile.children.cycles-pp.inet_recvmsg
2.66 -0.6 2.10 perf-profile.children.cycles-pp.ttwu_do_activate
2.55 -0.5 2.03 perf-profile.children.cycles-pp.enqueue_task_fair
16.68 -0.5 16.16 perf-profile.children.cycles-pp.__sys_recvfrom
16.84 -0.5 16.33 perf-profile.children.cycles-pp.__x64_sys_recvfrom
0.74 ? 2% -0.5 0.28 ? 3% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.58 -0.4 0.14 ? 7% perf-profile.children.cycles-pp.pick_next_entity
0.79 ? 3% -0.4 0.36 perf-profile.children.cycles-pp.finish_task_switch
0.64 ? 2% -0.4 0.22 ? 4% perf-profile.children.cycles-pp.switch_mm_irqs_off
20.21 -0.4 19.80 perf-profile.children.cycles-pp.recvfrom
1.03 -0.4 0.63 perf-profile.children.cycles-pp.native_apic_msr_eoi_write
2.02 -0.3 1.71 perf-profile.children.cycles-pp.update_load_avg
1.22 -0.3 0.96 perf-profile.children.cycles-pp.ttwu_do_wakeup
1.14 ? 2% -0.2 0.90 perf-profile.children.cycles-pp.check_preempt_curr
2.50 -0.2 2.28 perf-profile.children.cycles-pp.dequeue_task_fair
1.94 -0.2 1.72 perf-profile.children.cycles-pp.enqueue_entity
0.71 -0.2 0.52 ? 4% perf-profile.children.cycles-pp.nr_iowait_cpu
2.21 -0.2 2.03 perf-profile.children.cycles-pp.dequeue_entity
1.02 -0.1 0.91 perf-profile.children.cycles-pp.irqentry_enter
0.94 -0.0 0.90 ? 2% perf-profile.children.cycles-pp.update_curr
0.12 ? 6% -0.0 0.08 ? 8% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.57 ? 2% -0.0 0.53 perf-profile.children.cycles-pp.__update_load_avg_se
1.39 -0.0 1.35 perf-profile.children.cycles-pp.native_irq_return_iret
0.19 ? 3% -0.0 0.16 ? 9% perf-profile.children.cycles-pp.fib_lookup_good_nhc
0.21 -0.0 0.18 ? 4% perf-profile.children.cycles-pp.__list_del_entry_valid
0.24 ? 2% -0.0 0.22 ? 5% perf-profile.children.cycles-pp.rcu_nmi_enter
0.14 ? 4% -0.0 0.12 perf-profile.children.cycles-pp.error_entry
0.22 ? 2% -0.0 0.20 ? 2% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.06 ? 8% +0.0 0.07 ? 5% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.47 +0.0 0.48 perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.37 +0.0 0.39 ? 2% perf-profile.children.cycles-pp.netif_rx_internal
0.76 +0.0 0.78 perf-profile.children.cycles-pp.ip_generic_getfrag
0.40 +0.0 0.42 ? 2% perf-profile.children.cycles-pp.__netif_rx
0.34 +0.0 0.36 ? 2% perf-profile.children.cycles-pp.enqueue_to_backlog
0.09 ? 4% +0.0 0.11 ? 3% perf-profile.children.cycles-pp.place_entity
0.16 ? 5% +0.0 0.18 ? 5% perf-profile.children.cycles-pp.__get_user_4
0.08 ? 4% +0.0 0.10 ? 7% perf-profile.children.cycles-pp.ip_rcv_finish_core
0.08 ? 6% +0.0 0.10 ? 3% perf-profile.children.cycles-pp.clear_buddies
0.06 ? 6% +0.0 0.09 ? 5% perf-profile.children.cycles-pp.menu_reflect
0.53 +0.0 0.56 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.47 +0.0 0.50 perf-profile.children.cycles-pp.aa_sk_perm
0.44 ? 2% +0.0 0.47 perf-profile.children.cycles-pp.__might_fault
0.94 +0.0 0.97 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.74 +0.0 0.78 perf-profile.children.cycles-pp.check_heap_object
0.06 ? 7% +0.0 0.10 ? 3% perf-profile.children.cycles-pp.rb_erase
0.09 ? 5% +0.0 0.13 ? 5% perf-profile.children.cycles-pp.__list_add_valid
1.68 +0.0 1.72 perf-profile.children.cycles-pp.loopback_xmit
0.20 ? 3% +0.0 0.24 ? 2% perf-profile.children.cycles-pp.__netif_receive_skb_core
0.74 +0.0 0.79 ? 3% perf-profile.children.cycles-pp.select_idle_sibling
1.10 +0.0 1.15 perf-profile.children.cycles-pp.__consume_stateless_skb
0.74 +0.0 0.79 perf-profile.children.cycles-pp.___perf_sw_event
2.19 +0.0 2.24 perf-profile.children.cycles-pp.menu_select
1.09 +0.0 1.14 perf-profile.children.cycles-pp.skb_release_data
0.38 ? 2% +0.0 0.43 ? 11% perf-profile.children.cycles-pp._copy_to_user
0.00 +0.1 0.05 perf-profile.children.cycles-pp.idle_cpu
1.30 +0.1 1.35 perf-profile.children.cycles-pp.__check_object_size
0.21 ? 3% +0.1 0.26 ? 4% perf-profile.children.cycles-pp.update_cfs_group
0.00 +0.1 0.05 ? 7% perf-profile.children.cycles-pp.flush_smp_call_function_queue
1.01 +0.1 1.06 ? 2% perf-profile.children.cycles-pp.select_task_rq_fair
0.00 +0.1 0.05 ? 8% perf-profile.children.cycles-pp.tick_nohz_idle_got_tick
0.10 ? 4% +0.1 0.16 ? 2% perf-profile.children.cycles-pp.sched_clock
0.65 +0.1 0.71 ? 2% perf-profile.children.cycles-pp.__switch_to_asm
0.00 +0.1 0.06 ? 17% perf-profile.children.cycles-pp.__update_blocked_fair
0.09 ? 7% +0.1 0.15 ? 13% perf-profile.children.cycles-pp.update_blocked_averages
4.40 +0.1 4.47 perf-profile.children.cycles-pp.__ip_append_data
0.16 ? 4% +0.1 0.24 ? 2% perf-profile.children.cycles-pp.x2apic_send_IPI
0.08 ? 7% +0.1 0.16 ? 10% perf-profile.children.cycles-pp.update_sg_lb_stats
0.99 +0.1 1.07 ? 5% perf-profile.children.cycles-pp.move_addr_to_user
0.00 +0.1 0.09 ? 8% perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
0.04 ? 45% +0.1 0.13 ? 14% perf-profile.children.cycles-pp.cpuacct_charge
0.17 ? 2% +0.1 0.26 ? 2% perf-profile.children.cycles-pp.ip_rcv_core
0.10 ? 5% +0.1 0.19 ? 7% perf-profile.children.cycles-pp.update_sd_lb_stats
0.10 ? 6% +0.1 0.21 ? 10% perf-profile.children.cycles-pp.find_busiest_group
0.00 +0.1 0.12 ? 3% perf-profile.children.cycles-pp.send_call_function_single_ipi
0.71 +0.1 0.83 perf-profile.children.cycles-pp.__x2apic_send_IPI_dest
0.12 ? 7% +0.1 0.25 ? 9% perf-profile.children.cycles-pp.load_balance
0.82 +0.2 0.98 perf-profile.children.cycles-pp._raw_spin_lock
0.56 ? 3% +0.2 0.71 ? 2% perf-profile.children.cycles-pp.available_idle_cpu
7.00 +0.2 7.15 perf-profile.children.cycles-pp.ip_make_skb
0.33 +0.2 0.50 ? 2% perf-profile.children.cycles-pp.ip_rcv
1.10 ? 2% +0.2 1.32 perf-profile.children.cycles-pp.prepare_task_switch
0.00 +0.2 0.23 perf-profile.children.cycles-pp.llist_add_batch
0.38 ? 5% +0.2 0.62 ? 8% perf-profile.children.cycles-pp.newidle_balance
0.00 +0.3 0.26 perf-profile.children.cycles-pp.__smp_call_single_queue
0.18 ? 4% +0.3 0.51 perf-profile.children.cycles-pp.__irq_exit_rcu
0.97 +0.5 1.44 perf-profile.children.cycles-pp.native_sched_clock
1.16 +0.6 1.73 perf-profile.children.cycles-pp.sched_clock_cpu
0.13 ? 8% +0.6 0.78 perf-profile.children.cycles-pp.resched_curr
0.06 ? 13% +0.9 0.95 ? 2% perf-profile.children.cycles-pp.tick_irq_enter
0.00 +1.0 0.95 perf-profile.children.cycles-pp.llist_reverse_order
0.07 ? 12% +1.2 1.27 ? 2% perf-profile.children.cycles-pp.irq_enter_rcu
0.07 ? 5% +1.7 1.79 perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.00 +1.8 1.84 perf-profile.children.cycles-pp.__flush_smp_call_function_queue
43.06 +4.4 47.46 perf-profile.children.cycles-pp.start_secondary
43.19 +4.4 47.61 perf-profile.children.cycles-pp.do_idle
0.00 +4.4 4.42 perf-profile.children.cycles-pp.sched_ttwu_pending
43.31 +4.4 47.73 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
43.31 +4.4 47.73 perf-profile.children.cycles-pp.cpu_startup_entry
0.00 +6.9 6.87 perf-profile.children.cycles-pp.__sysvec_call_function_single
31.36 +7.6 38.95 perf-profile.children.cycles-pp.cpuidle_enter
29.52 +7.6 37.11 perf-profile.children.cycles-pp.acpi_idle_enter
31.21 +7.6 38.82 perf-profile.children.cycles-pp.cpuidle_enter_state
34.43 +7.7 42.08 perf-profile.children.cycles-pp.cpuidle_idle_call
29.00 +7.8 36.80 perf-profile.children.cycles-pp.acpi_idle_do_entry
0.00 +9.8 9.77 perf-profile.children.cycles-pp.sysvec_call_function_single
0.00 +33.6 33.56 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
1.56 -1.3 0.23 ? 3% perf-profile.self.cycles-pp.set_next_entity
1.92 -1.2 0.76 perf-profile.self.cycles-pp.update_rq_clock
1.65 -0.6 1.05 perf-profile.self.cycles-pp.__schedule
0.73 ? 2% -0.5 0.28 ? 2% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.57 -0.4 0.12 ? 7% perf-profile.self.cycles-pp.pick_next_entity
0.64 ? 3% -0.4 0.21 ? 3% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.72 ? 3% -0.4 0.30 perf-profile.self.cycles-pp.finish_task_switch
1.02 -0.4 0.62 perf-profile.self.cycles-pp.native_apic_msr_eoi_write
0.61 -0.4 0.23 ? 3% perf-profile.self.cycles-pp.irqtime_account_irq
0.63 ? 2% -0.3 0.33 ? 4% perf-profile.self.cycles-pp.enqueue_task_fair
0.88 -0.3 0.60 perf-profile.self.cycles-pp.update_load_avg
0.52 ? 2% -0.2 0.32 ? 3% perf-profile.self.cycles-pp.acpi_idle_enter
0.70 -0.2 0.50 ? 4% perf-profile.self.cycles-pp.nr_iowait_cpu
0.86 -0.1 0.74 ? 2% perf-profile.self.cycles-pp.enqueue_entity
0.46 ? 2% -0.1 0.35 ? 3% perf-profile.self.cycles-pp.update_curr
0.21 ? 3% -0.1 0.12 ? 5% perf-profile.self.cycles-pp.__napi_poll
0.29 ? 3% -0.1 0.20 ? 3% perf-profile.self.cycles-pp.process_backlog
0.77 -0.1 0.68 perf-profile.self.cycles-pp.irqentry_enter
0.43 ? 3% -0.1 0.35 ? 3% perf-profile.self.cycles-pp.select_task_rq
0.97 -0.1 0.91 perf-profile.self.cycles-pp.udp_sendmsg
0.36 ? 4% -0.1 0.30 ? 2% perf-profile.self.cycles-pp.dequeue_entity
0.79 ? 2% -0.0 0.74 ? 2% perf-profile.self.cycles-pp.try_to_wake_up
0.51 ? 2% -0.0 0.47 perf-profile.self.cycles-pp.__update_load_avg_se
0.25 ? 2% -0.0 0.21 ? 6% perf-profile.self.cycles-pp.dequeue_task_fair
0.12 ? 5% -0.0 0.08 ? 14% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.14 ? 3% -0.0 0.10 ? 7% perf-profile.self.cycles-pp.tick_nohz_idle_exit
0.10 ? 4% -0.0 0.08 ? 6% perf-profile.self.cycles-pp.ttwu_do_activate
1.38 -0.0 1.35 perf-profile.self.cycles-pp.native_irq_return_iret
0.18 ? 3% -0.0 0.15 ? 9% perf-profile.self.cycles-pp.fib_lookup_good_nhc
0.13 ? 3% -0.0 0.11 perf-profile.self.cycles-pp.error_entry
0.19 -0.0 0.16 ? 4% perf-profile.self.cycles-pp.__list_del_entry_valid
0.08 ? 5% -0.0 0.06 ? 7% perf-profile.self.cycles-pp.ttwu_do_wakeup
0.22 ? 3% -0.0 0.20 ? 3% perf-profile.self.cycles-pp.pick_next_task_fair
0.62 -0.0 0.60 perf-profile.self.cycles-pp.skb_set_owner_w
0.07 ? 5% +0.0 0.08 ? 4% perf-profile.self.cycles-pp.place_entity
0.45 +0.0 0.46 perf-profile.self.cycles-pp.__might_resched
0.56 +0.0 0.58 perf-profile.self.cycles-pp.check_heap_object
0.08 +0.0 0.10 ? 6% perf-profile.self.cycles-pp.ip_rcv_finish_core
0.12 ? 4% +0.0 0.14 ? 3% perf-profile.self.cycles-pp.__hrtimer_next_event_base
0.53 +0.0 0.55 perf-profile.self.cycles-pp.udp_recvmsg
0.16 ? 2% +0.0 0.18 ? 4% perf-profile.self.cycles-pp.skb_release_data
0.16 ? 3% +0.0 0.18 ? 3% perf-profile.self.cycles-pp.__get_user_4
0.20 ? 3% +0.0 0.23 ? 5% perf-profile.self.cycles-pp.do_softirq
0.91 +0.0 0.93 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.55 +0.0 0.57 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.44 ? 2% +0.0 0.46 ? 2% perf-profile.self.cycles-pp.net_rx_action
0.04 ? 44% +0.0 0.07 ? 5% perf-profile.self.cycles-pp.clear_buddies
0.08 ? 6% +0.0 0.11 ? 3% perf-profile.self.cycles-pp.__list_add_valid
0.26 ? 2% +0.0 0.29 perf-profile.self.cycles-pp.__local_bh_enable_ip
0.62 ? 2% +0.0 0.65 perf-profile.self.cycles-pp.__skb_recv_udp
0.20 ? 4% +0.0 0.24 ? 4% perf-profile.self.cycles-pp.__sys_recvfrom
0.20 ? 4% +0.0 0.24 perf-profile.self.cycles-pp.__netif_receive_skb_core
0.66 ? 2% +0.0 0.70 perf-profile.self.cycles-pp.___perf_sw_event
0.03 ? 70% +0.0 0.08 ? 4% perf-profile.self.cycles-pp.rb_erase
0.00 +0.1 0.05 perf-profile.self.cycles-pp.__irq_exit_rcu
0.18 ? 4% +0.1 0.23 ? 6% perf-profile.self.cycles-pp.update_cfs_group
0.00 +0.1 0.05 ? 8% perf-profile.self.cycles-pp.sched_clock
0.06 ? 6% +0.1 0.12 ? 9% perf-profile.self.cycles-pp.update_sg_lb_stats
0.00 +0.1 0.06 perf-profile.self.cycles-pp.__sysvec_call_function_single
0.63 ? 2% +0.1 0.70 ? 2% perf-profile.self.cycles-pp.__switch_to_asm
0.08 ? 4% +0.1 0.14 ? 3% perf-profile.self.cycles-pp.ip_rcv
0.15 ? 5% +0.1 0.22 ? 2% perf-profile.self.cycles-pp.sched_clock_cpu
0.16 ? 3% +0.1 0.23 ? 3% perf-profile.self.cycles-pp.x2apic_send_IPI
0.00 +0.1 0.08 ? 13% perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
0.00 +0.1 0.08 ? 5% perf-profile.self.cycles-pp.tick_irq_enter
0.04 ? 45% +0.1 0.13 ? 15% perf-profile.self.cycles-pp.cpuacct_charge
0.17 ? 2% +0.1 0.26 ? 2% perf-profile.self.cycles-pp.ip_rcv_core
0.09 ? 6% +0.1 0.18 ? 2% perf-profile.self.cycles-pp.schedule
0.71 +0.1 0.82 perf-profile.self.cycles-pp.__x2apic_send_IPI_dest
0.00 +0.1 0.12 ? 5% perf-profile.self.cycles-pp.send_call_function_single_ipi
0.54 ? 3% +0.2 0.70 ? 3% perf-profile.self.cycles-pp.available_idle_cpu
0.77 +0.2 0.94 ? 2% perf-profile.self.cycles-pp._raw_spin_lock
0.35 ? 7% +0.2 0.52 perf-profile.self.cycles-pp.prepare_task_switch
0.07 ? 5% +0.2 0.25 perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.00 +0.2 0.21 ? 3% perf-profile.self.cycles-pp.sched_ttwu_pending
0.00 +0.2 0.22 perf-profile.self.cycles-pp.llist_add_batch
0.00 +0.3 0.29 perf-profile.self.cycles-pp.sysvec_call_function_single
0.00 +0.3 0.32 perf-profile.self.cycles-pp.irq_enter_rcu
0.60 ? 4% +0.3 0.94 perf-profile.self.cycles-pp.schedule_timeout
0.00 +0.4 0.38 ? 3% perf-profile.self.cycles-pp.asm_sysvec_call_function_single
23.90 +0.4 24.31 perf-profile.self.cycles-pp.acpi_idle_do_entry
0.93 +0.4 1.38 perf-profile.self.cycles-pp.native_sched_clock
0.13 ? 8% +0.6 0.78 perf-profile.self.cycles-pp.resched_curr
0.00 +0.9 0.88 perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.00 +1.0 0.95 perf-profile.self.cycles-pp.llist_reverse_order



***************************************************************************************************
lkp-csl-2sp9: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
gcc-11/performance/4BRD_12G/f2fs/x86_64-rhel-8.3/100/RAID1/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp9/sync_disk_rw/aim7/0x500320a

commit:
28156108fe ("sched: Fix the check of nr_running at queue wakelist")
f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc
---------------- ---------------------------
%stddev %change %stddev
\ | \
1526 ? 10% +246.4% 5285 ? 25% aim7.jobs-per-min
398.30 ? 11% -69.4% 121.76 ? 26% aim7.time.elapsed_time
398.30 ? 11% -69.4% 121.76 ? 26% aim7.time.elapsed_time.max
86549701 -7.5% 80055785 aim7.time.file_system_outputs
4061406 ? 10% -60.0% 1624315 ? 22% aim7.time.involuntary_context_switches
13445 ? 3% -10.5% 12039 ? 4% aim7.time.minor_page_faults
1767 ? 9% -32.9% 1186 ? 15% aim7.time.system_time
3.832e+08 ? 12% -67.8% 1.232e+08 ? 30% aim7.time.voluntary_context_switches
3.208e+10 ? 12% -72.0% 8.998e+09 ? 27% cpuidle..time
5.072e+08 ? 13% -69.4% 1.55e+08 ? 31% cpuidle..usage
43.08 -24.9% 32.37 ? 21% iostat.cpu.idle
6.13 ? 3% +119.1% 13.43 ? 11% iostat.cpu.system
430.38 ? 11% -64.2% 153.88 ? 20% uptime.boot
17079 ? 11% -67.9% 5479 ? 2% uptime.idle
42.81 -11.5 31.28 ? 22% mpstat.cpu.all.idle%
0.76 ? 2% +0.2 0.93 ? 7% mpstat.cpu.all.irq%
0.05 +0.0 0.06 ? 7% mpstat.cpu.all.soft%
5.34 ? 3% +7.3 12.64 ? 12% mpstat.cpu.all.sys%
0.09 ? 8% +0.1 0.20 ? 10% mpstat.cpu.all.usr%
42.67 -25.4% 31.83 ? 22% vmstat.cpu.id
108692 ? 9% +212.7% 339895 ? 24% vmstat.io.bo
1.162e+08 ? 2% -16.3% 97235935 ? 3% vmstat.memory.free
88.33 ? 2% -33.6% 58.67 ? 14% vmstat.procs.b
5.17 ? 7% +100.0% 10.33 ? 14% vmstat.procs.r
185561 +14.2% 211999 ? 3% vmstat.system.in
10476 ? 8% +61.0% 16866 ? 12% numa-meminfo.node0.Dirty
31580 ? 9% +18.1% 37306 ? 8% numa-meminfo.node0.Inactive(file)
54824112 ? 7% -19.4% 44208669 ? 8% numa-meminfo.node0.MemFree
10835242 ? 36% +98.0% 21450685 ? 17% numa-meminfo.node0.MemUsed
9855 ? 12% +67.2% 16479 ? 13% numa-meminfo.node1.Dirty
29343 ? 9% +26.5% 37127 ? 6% numa-meminfo.node1.Inactive(file)
61393162 ? 4% -13.5% 53116009 ? 10% numa-meminfo.node1.MemFree
4621337 ? 56% +179.1% 12898489 ? 41% numa-meminfo.node1.MemUsed
180156 ? 2% -34.3% 118377 ? 13% meminfo.AnonHugePages
20545 ? 3% +58.5% 32563 ? 12% meminfo.Dirty
60890 +22.1% 74342 ? 6% meminfo.Inactive(file)
138527 ? 3% +29.7% 179674 ? 4% meminfo.KReclaimable
1.157e+08 ? 2% -16.3% 96822390 ? 3% meminfo.MemAvailable
1.162e+08 ? 2% -16.3% 97308937 ? 3% meminfo.MemFree
15456872 ? 15% +122.3% 34364917 ? 10% meminfo.Memused
138527 ? 3% +29.7% 179674 ? 4% meminfo.SReclaimable
322884 +13.2% 365463 ? 2% meminfo.Slab
17500146 ? 16% +99.0% 34825108 ? 11% meminfo.max_used_kB
2613 ? 8% +61.2% 4213 ? 13% numa-vmstat.node0.nr_dirty
13706147 ? 7% -19.4% 11052112 ? 8% numa-vmstat.node0.nr_free_pages
7894 ? 9% +18.1% 9326 ? 8% numa-vmstat.node0.nr_inactive_file
97.83 ? 67% -100.0% 0.00 numa-vmstat.node0.nr_mlock
7894 ? 9% +18.1% 9326 ? 8% numa-vmstat.node0.nr_zone_inactive_file
621.67 ? 18% -63.2% 228.50 ? 72% numa-vmstat.node0.nr_zone_write_pending
2462 ? 12% +67.5% 4126 ? 13% numa-vmstat.node1.nr_dirty
15348666 ? 4% -13.5% 13279101 ? 10% numa-vmstat.node1.nr_free_pages
7336 ? 9% +26.6% 9285 ? 6% numa-vmstat.node1.nr_inactive_file
7336 ? 9% +26.6% 9285 ? 6% numa-vmstat.node1.nr_zone_inactive_file
568.00 ? 11% -68.1% 181.17 ? 64% numa-vmstat.node1.nr_zone_write_pending
444.83 +52.0% 676.33 ? 4% turbostat.Avg_MHz
15.96 +8.4 24.38 ? 4% turbostat.Busy%
2.476e+08 ? 12% -67.0% 81669432 ? 35% turbostat.C1
46.80 -6.9 39.93 ? 12% turbostat.C1%
1.495e+08 ? 12% -74.3% 38375703 ? 18% turbostat.C1E
84.02 -11.2% 74.64 ? 2% turbostat.CPU%c1
0.02 +4791.7% 0.98 ?133% turbostat.CPU%c6
74404370 ? 11% -64.8% 26214041 ? 22% turbostat.IRQ
1.096e+08 ? 18% -68.6% 34360239 ? 35% turbostat.POLL
53.17 +3.4% 55.00 turbostat.PkgTmp
163.95 +6.6% 174.70 turbostat.PkgWatt
14.14 +6.7% 15.08 turbostat.RAMWatt
63785 +3.3% 65914 proc-vmstat.nr_anon_pages
10809810 -7.6% 9991919 proc-vmstat.nr_dirtied
5118 ? 3% +59.4% 8158 ? 12% proc-vmstat.nr_dirty
2887461 ? 2% -16.3% 2415794 ? 3% proc-vmstat.nr_dirty_background_threshold
5781983 ? 2% -16.3% 4837496 ? 3% proc-vmstat.nr_dirty_threshold
29054604 ? 2% -16.3% 24327424 ? 3% proc-vmstat.nr_free_pages
15227 +22.0% 18583 ? 6% proc-vmstat.nr_inactive_file
114.67 ? 44% -100.0% 0.00 proc-vmstat.nr_mlock
34630 ? 3% +29.7% 44918 ? 4% proc-vmstat.nr_slab_reclaimable
10774150 -7.6% 9955315 proc-vmstat.nr_written
15227 +22.0% 18583 ? 6% proc-vmstat.nr_zone_inactive_file
1097 ? 12% -62.0% 417.00 ? 80% proc-vmstat.nr_zone_write_pending
6341509 ? 8% +89.3% 12003005 ? 6% proc-vmstat.numa_hit
6264928 ? 8% +90.3% 11922778 ? 6% proc-vmstat.numa_local
6341353 ? 8% +89.3% 12002253 ? 6% proc-vmstat.pgalloc_normal
3894 ? 22% +97.2% 7678 ? 6% proc-vmstat.pgdeactivate
1283415 ? 6% -54.3% 586146 ? 37% proc-vmstat.pgfault
3193580 -26.2% 2357922 ? 26% proc-vmstat.pgfree
43111608 -7.6% 39849599 proc-vmstat.pgpgout
46760 ? 10% -57.3% 19952 ? 15% proc-vmstat.pgreuse
7619 ? 23% +100.4% 15269 ? 6% proc-vmstat.pgrotated
0.09 ? 15% +81.1% 0.16 ? 16% sched_debug.cfs_rq:/.h_nr_running.avg
0.28 ? 6% +31.3% 0.37 ? 7% sched_debug.cfs_rq:/.h_nr_running.stddev
0.05 ?142% +1084.6% 0.61 ? 46% sched_debug.cfs_rq:/.load_avg.min
8133 ? 17% -31.6% 5564 ? 16% sched_debug.cfs_rq:/.min_vruntime.stddev
0.09 ? 14% +80.4% 0.16 ? 16% sched_debug.cfs_rq:/.nr_running.avg
0.28 ? 6% +31.0% 0.37 ? 7% sched_debug.cfs_rq:/.nr_running.stddev
8.27 ? 44% +243.8% 28.42 ? 47% sched_debug.cfs_rq:/.removed.load_avg.avg
289.86 ? 31% +125.7% 654.22 ? 43% sched_debug.cfs_rq:/.removed.load_avg.max
44.17 ? 30% +178.7% 123.10 ? 35% sched_debug.cfs_rq:/.removed.load_avg.stddev
3.51 ? 40% +254.2% 12.43 ? 39% sched_debug.cfs_rq:/.removed.runnable_avg.avg
144.11 ? 30% +121.1% 318.69 ? 41% sched_debug.cfs_rq:/.removed.runnable_avg.max
19.97 ? 28% +179.9% 55.88 ? 32% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
3.51 ? 40% +254.2% 12.43 ? 39% sched_debug.cfs_rq:/.removed.util_avg.avg
144.11 ? 30% +121.1% 318.69 ? 41% sched_debug.cfs_rq:/.removed.util_avg.max
19.97 ? 28% +179.9% 55.88 ? 32% sched_debug.cfs_rq:/.removed.util_avg.stddev
101.10 ? 4% +109.3% 211.65 ? 11% sched_debug.cfs_rq:/.runnable_avg.avg
1066 ? 3% +8.4% 1155 ? 7% sched_debug.cfs_rq:/.runnable_avg.max
0.12 ?170% +6854.8% 8.56 ?140% sched_debug.cfs_rq:/.runnable_avg.min
155.43 ? 3% +40.0% 217.67 ? 7% sched_debug.cfs_rq:/.runnable_avg.stddev
8133 ? 17% -31.6% 5564 ? 16% sched_debug.cfs_rq:/.spread0.stddev
100.71 ? 4% +109.5% 211.01 ? 11% sched_debug.cfs_rq:/.util_avg.avg
1065 ? 3% +8.4% 1155 ? 7% sched_debug.cfs_rq:/.util_avg.max
0.17 ?154% +4752.3% 8.47 ?141% sched_debug.cfs_rq:/.util_avg.min
155.41 ? 3% +40.1% 217.75 ? 7% sched_debug.cfs_rq:/.util_avg.stddev
17.21 ? 4% +71.8% 29.56 ? 8% sched_debug.cfs_rq:/.util_est_enqueued.avg
104.99 ? 3% +17.1% 122.98 ? 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev
203519 ? 6% +86.0% 378456 ? 15% sched_debug.cpu.avg_idle.avg
23207 ? 10% -34.1% 15298 ? 15% sched_debug.cpu.avg_idle.min
215927 ? 12% -64.8% 75933 ? 19% sched_debug.cpu.clock.avg
215930 ? 12% -64.8% 75937 ? 19% sched_debug.cpu.clock.max
215922 ? 12% -64.8% 75928 ? 19% sched_debug.cpu.clock.min
214420 ? 12% -64.8% 75461 ? 19% sched_debug.cpu.clock_task.avg
214557 ? 12% -64.8% 75513 ? 19% sched_debug.cpu.clock_task.max
213787 ? 12% -65.0% 74884 ? 19% sched_debug.cpu.clock_task.min
489.12 ? 10% +77.2% 866.67 ? 22% sched_debug.cpu.curr->pid.avg
9698 ? 7% -39.1% 5905 ? 7% sched_debug.cpu.curr->pid.max
538311 ? 3% +87.0% 1006375 ? 51% sched_debug.cpu.max_idle_balance_cost.max
5365 ? 62% +1667.9% 94854 ?105% sched_debug.cpu.max_idle_balance_cost.stddev
0.09 ? 8% +87.6% 0.17 ? 21% sched_debug.cpu.nr_running.avg
0.28 ? 3% +30.1% 0.37 ? 8% sched_debug.cpu.nr_running.stddev
3886268 ? 16% -74.8% 979579 ? 36% sched_debug.cpu.nr_switches.avg
4753123 ? 19% -76.3% 1127185 ? 40% sched_debug.cpu.nr_switches.max
2510015 ? 10% -69.0% 778502 ? 32% sched_debug.cpu.nr_switches.min
513442 ? 44% -85.2% 75784 ? 69% sched_debug.cpu.nr_switches.stddev
215923 ? 12% -64.8% 75929 ? 19% sched_debug.cpu_clk
215350 ? 12% -65.0% 75355 ? 19% sched_debug.ktime
216859 ? 12% -64.6% 76708 ? 19% sched_debug.sched_clk
3.027e+09 +37.7% 4.168e+09 perf-stat.i.branch-instructions
1.19 +0.5 1.68 ? 20% perf-stat.i.branch-miss-rate%
35249374 ? 2% +67.7% 59107652 ? 8% perf-stat.i.branch-misses
18.14 ? 4% +5.0 23.10 ? 5% perf-stat.i.cache-miss-rate%
51539072 ? 2% +75.3% 90324359 ? 6% perf-stat.i.cache-misses
2.857e+08 ? 4% +33.4% 3.811e+08 ? 8% perf-stat.i.cache-references
1945960 +5.2% 2046993 ? 3% perf-stat.i.context-switches
2.88 +8.2% 3.12 ? 4% perf-stat.i.cpi
3.961e+10 +52.3% 6.03e+10 ? 4% perf-stat.i.cpu-cycles
6686 ? 11% +327.7% 28595 ? 20% perf-stat.i.cpu-migrations
0.01 ? 23% +0.0 0.06 ? 59% perf-stat.i.dTLB-load-miss-rate%
565205 ? 16% +332.6% 2444922 ? 28% perf-stat.i.dTLB-load-misses
3.747e+09 +45.1% 5.438e+09 ? 2% perf-stat.i.dTLB-loads
0.00 ? 24% +0.0 0.01 ? 68% perf-stat.i.dTLB-store-miss-rate%
51172 ? 14% +216.7% 162069 ? 29% perf-stat.i.dTLB-store-misses
1.864e+09 +36.4% 2.544e+09 perf-stat.i.dTLB-stores
33.72 +1.2 34.89 ? 2% perf-stat.i.iTLB-load-miss-rate%
7385612 +21.5% 8974148 perf-stat.i.iTLB-load-misses
14725202 +20.9% 17808328 perf-stat.i.iTLB-loads
1.387e+10 +42.1% 1.972e+10 perf-stat.i.instructions
1941 +24.1% 2408 ? 3% perf-stat.i.instructions-per-iTLB-miss
0.60 ? 10% +181.3% 1.68 ? 6% perf-stat.i.major-faults
0.45 +52.2% 0.69 ? 4% perf-stat.i.metric.GHz
558.11 +57.7% 880.04 ? 8% perf-stat.i.metric.K/sec
101.40 +40.4% 142.35 perf-stat.i.metric.M/sec
94.97 -3.4 91.55 perf-stat.i.node-load-miss-rate%
22231769 +56.6% 34809028 ? 5% perf-stat.i.node-load-misses
1074012 ? 7% +143.7% 2617508 ? 13% perf-stat.i.node-loads
92.47 -4.1 88.35 ? 2% perf-stat.i.node-store-miss-rate%
8136157 +87.1% 15224518 ? 8% perf-stat.i.node-store-misses
631409 ? 13% +160.4% 1644054 ? 20% perf-stat.i.node-stores
1.16 +0.3 1.42 ? 7% perf-stat.overall.branch-miss-rate%
18.07 ? 4% +5.7 23.74 ? 3% perf-stat.overall.cache-miss-rate%
2.86 ? 2% +7.1% 3.06 ? 2% perf-stat.overall.cpi
768.67 -13.0% 668.99 ? 3% perf-stat.overall.cycles-between-cache-misses
0.02 ? 15% +0.0 0.04 ? 26% perf-stat.overall.dTLB-load-miss-rate%
0.00 ? 12% +0.0 0.01 ? 28% perf-stat.overall.dTLB-store-miss-rate%
1878 +16.9% 2197 perf-stat.overall.instructions-per-iTLB-miss
0.35 ? 2% -6.6% 0.33 ? 2% perf-stat.overall.ipc
95.39 -2.4 93.04 perf-stat.overall.node-load-miss-rate%
92.80 -2.5 90.35 perf-stat.overall.node-store-miss-rate%
3.019e+09 +36.9% 4.132e+09 perf-stat.ps.branch-instructions
35157825 ? 2% +66.6% 58574778 ? 7% perf-stat.ps.branch-misses
51407192 ? 2% +74.1% 89524967 ? 6% perf-stat.ps.cache-misses
2.85e+08 ? 4% +32.5% 3.777e+08 ? 8% perf-stat.ps.cache-references
1940981 +4.5% 2029274 ? 3% perf-stat.ps.context-switches
3.951e+10 +51.3% 5.977e+10 ? 4% perf-stat.ps.cpu-cycles
6668 ? 11% +324.9% 28333 ? 20% perf-stat.ps.cpu-migrations
563754 ? 16% +329.7% 2422257 ? 28% perf-stat.ps.dTLB-load-misses
3.738e+09 +44.2% 5.39e+09 ? 2% perf-stat.ps.dTLB-loads
51038 ? 14% +214.6% 160565 ? 28% perf-stat.ps.dTLB-store-misses
1.86e+09 +35.6% 2.521e+09 perf-stat.ps.dTLB-stores
7366790 +20.8% 8895533 perf-stat.ps.iTLB-load-misses
14687561 +20.2% 17652009 perf-stat.ps.iTLB-loads
1.384e+10 +41.2% 1.954e+10 perf-stat.ps.instructions
0.60 ? 10% +179.5% 1.66 ? 6% perf-stat.ps.major-faults
22174931 +55.6% 34501460 ? 5% perf-stat.ps.node-load-misses
1071229 ? 7% +142.1% 2593745 ? 13% perf-stat.ps.node-loads
8115368 +85.9% 15089754 ? 7% perf-stat.ps.node-store-misses
629775 ? 13% +158.6% 1628862 ? 20% perf-stat.ps.node-stores
5.52e+12 ? 10% -56.7% 2.392e+12 ? 24% perf-stat.total.instructions
73.04 ? 5% -19.4 53.65 ? 7% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
72.28 ? 5% -19.2 53.04 ? 7% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
72.26 ? 5% -19.2 53.03 ? 7% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
72.23 ? 5% -19.2 53.01 ? 7% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
67.43 ? 5% -18.3 49.10 ? 7% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
66.16 ? 5% -17.8 48.37 ? 7% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
65.59 ? 5% -17.6 48.02 ? 7% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
15.97 ? 16% -8.8 7.15 ? 34% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
25.00 ? 7% -7.9 17.06 ? 23% perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
8.52 ? 23% -3.5 5.00 ? 13% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter
3.83 ? 28% -2.9 0.92 ? 38% perf-profile.calltrace.cycles-pp.folio_wake_bit.folio_end_writeback.f2fs_write_end_io.raid_end_bio_io.raid1_end_write_request
3.75 ? 28% -2.9 0.86 ? 40% perf-profile.calltrace.cycles-pp.__wake_up_common.folio_wake_bit.folio_end_writeback.f2fs_write_end_io.raid_end_bio_io
3.68 ? 28% -2.8 0.83 ? 40% perf-profile.calltrace.cycles-pp.wake_page_function.__wake_up_common.folio_wake_bit.folio_end_writeback.f2fs_write_end_io
3.88 ? 27% -2.8 1.04 ? 32% perf-profile.calltrace.cycles-pp.folio_end_writeback.f2fs_write_end_io.raid_end_bio_io.raid1_end_write_request.__submit_bio
3.56 ? 28% -2.8 0.80 ? 41% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_page_function.__wake_up_common.folio_wake_bit.folio_end_writeback
3.97 ? 26% -2.7 1.26 ? 24% perf-profile.calltrace.cycles-pp.f2fs_write_end_io.raid_end_bio_io.raid1_end_write_request.__submit_bio.__submit_bio_noacct
4.08 ? 25% -2.7 1.37 ? 21% perf-profile.calltrace.cycles-pp.raid1_end_write_request.__submit_bio.__submit_bio_noacct.flush_bio_list.flush_pending_writes
5.24 ? 6% -2.4 2.80 ? 22% perf-profile.calltrace.cycles-pp.io_schedule.folio_wait_bit_common.folio_wait_writeback.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback
5.19 ? 6% -2.4 2.78 ? 22% perf-profile.calltrace.cycles-pp.schedule.io_schedule.folio_wait_bit_common.folio_wait_writeback.f2fs_wait_on_page_writeback
5.13 ? 6% -2.4 2.76 ? 22% perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.folio_wait_bit_common.folio_wait_writeback
4.01 ? 26% -2.3 1.67 ? 7% perf-profile.calltrace.cycles-pp.raid_end_bio_io.raid1_end_write_request.__submit_bio.__submit_bio_noacct.flush_bio_list
4.55 ? 21% -2.2 2.31 ? 6% perf-profile.calltrace.cycles-pp.flush_pending_writes.raid1d.md_thread.kthread.ret_from_fork
4.55 ? 20% -2.2 2.31 ? 6% perf-profile.calltrace.cycles-pp.raid1d.md_thread.kthread.ret_from_fork
4.55 ? 21% -2.2 2.31 ? 6% perf-profile.calltrace.cycles-pp.flush_bio_list.flush_pending_writes.raid1d.md_thread.kthread
4.51 ? 21% -2.2 2.27 ? 7% perf-profile.calltrace.cycles-pp.__submit_bio_noacct.flush_bio_list.flush_pending_writes.raid1d.md_thread
4.50 ? 21% -2.2 2.27 ? 7% perf-profile.calltrace.cycles-pp.__submit_bio.__submit_bio_noacct.flush_bio_list.flush_pending_writes.raid1d
4.56 ? 20% -2.2 2.33 ? 6% perf-profile.calltrace.cycles-pp.md_thread.kthread.ret_from_fork
4.94 ? 15% -1.7 3.20 ? 3% perf-profile.calltrace.cycles-pp.ret_from_fork
4.94 ? 15% -1.7 3.20 ? 3% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
3.31 ? 4% -1.3 2.04 ? 20% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.io_schedule.folio_wait_bit_common
3.28 ? 4% -1.3 2.03 ? 20% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.io_schedule
1.98 ? 5% -0.8 1.14 ? 3% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.93 ? 4% -0.8 1.10 ? 3% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
1.26 ? 19% -0.7 0.54 ? 52% perf-profile.calltrace.cycles-pp.down_read.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file
1.55 ? 9% -0.7 0.88 ? 9% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
2.07 ? 2% +0.3 2.36 ? 10% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.77 ? 7% +0.4 1.15 ? 10% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry
0.63 ? 7% +0.4 1.03 ? 7% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle
1.28 ? 4% +0.4 1.69 ? 10% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
0.63 ? 12% +0.5 1.12 ? 11% perf-profile.calltrace.cycles-pp.prepare_write_begin.f2fs_write_begin.generic_perform_write.f2fs_buffered_write_iter.f2fs_file_write_iter
2.95 ? 6% +0.5 3.48 ? 6% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
3.00 ? 6% +0.5 3.52 ? 6% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule
0.26 ?100% +0.6 0.82 ? 8% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue
0.69 ? 15% +0.6 1.30 ? 11% perf-profile.calltrace.cycles-pp.f2fs_write_begin.generic_perform_write.f2fs_buffered_write_iter.f2fs_file_write_iter.new_sync_write
0.27 ?100% +0.6 0.88 ? 17% perf-profile.calltrace.cycles-pp.brd_submit_bio.__submit_bio.__submit_bio_noacct.flush_bio_list.flush_pending_writes
0.00 +0.6 0.62 ? 10% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback
0.00 +0.7 0.66 ? 9% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback
0.10 ?223% +0.7 0.84 ? 17% perf-profile.calltrace.cycles-pp.copy_to_brd.brd_do_bvec.brd_submit_bio.__submit_bio.__submit_bio_noacct
0.10 ?223% +0.8 0.86 ? 17% perf-profile.calltrace.cycles-pp.raid1_end_write_request.__submit_bio.__submit_bio_noacct.flush_bio_list.raid1_unplug
2.10 ? 7% +0.8 2.88 ? 9% perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_file_write_iter
0.00 +0.8 0.78 ? 14% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.rwsem_down_read_slowpath.__submit_merged_write_cond
0.70 ? 5% +0.8 1.51 ? 14% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
0.08 ?223% +0.8 0.93 ? 18% perf-profile.calltrace.cycles-pp.f2fs_merge_page_bio.f2fs_inplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages
0.00 +0.9 0.95 ? 39% perf-profile.calltrace.cycles-pp.__percpu_counter_sum.f2fs_space_for_roll_forward.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
0.10 ?223% +1.0 1.05 ? 17% perf-profile.calltrace.cycles-pp.f2fs_inplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.__f2fs_write_data_pages
0.00 +1.0 0.95 ? 13% perf-profile.calltrace.cycles-pp.__schedule.schedule.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback
0.00 +1.0 0.96 ? 13% perf-profile.calltrace.cycles-pp.schedule.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback
0.00 +1.0 0.96 ? 20% perf-profile.calltrace.cycles-pp.f2fs_outplace_write_data.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.__f2fs_write_data_pages
0.94 ? 22% +1.0 1.91 ? 13% perf-profile.calltrace.cycles-pp.generic_perform_write.f2fs_buffered_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write
0.00 +1.0 0.98 ? 37% perf-profile.calltrace.cycles-pp.f2fs_space_for_roll_forward.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
0.00 +1.0 0.98 ? 16% perf-profile.calltrace.cycles-pp.f2fs_submit_merged_ipu_write.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_file_write_iter
0.95 ? 23% +1.0 1.94 ? 13% perf-profile.calltrace.cycles-pp.f2fs_buffered_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
0.12 ?223% +1.1 1.18 ? 11% perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
3.13 ? 7% +1.1 4.20 ? 13% perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
0.00 +1.1 1.08 ? 41% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.rwsem_down_read_slowpath
0.39 ?105% +1.1 1.47 ? 17% perf-profile.calltrace.cycles-pp.brd_submit_bio.__submit_bio.__submit_bio_noacct.flush_bio_list.raid1_unplug
0.09 ?223% +1.1 1.19 ? 16% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page
0.13 ?223% +1.1 1.28 ? 30% perf-profile.calltrace.cycles-pp.f2fs_is_checkpointed_node.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
0.13 ?223% +1.2 1.37 ? 18% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
0.13 ?223% +1.2 1.37 ? 18% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.__wait_for_common
0.00 +1.3 1.26 ? 12% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file.f2fs_file_write_iter
0.39 ?104% +1.3 1.74 ? 17% perf-profile.calltrace.cycles-pp.f2fs_submit_merged_ipu_write.f2fs_write_cache_pages.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc
0.12 ?223% +1.4 1.54 ? 15% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page
0.15 ?223% +1.4 1.59 ? 18% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__wait_for_common.f2fs_issue_flush
0.15 ?223% +1.4 1.60 ? 18% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__wait_for_common.f2fs_issue_flush.f2fs_do_sync_file
0.15 ?223% +1.4 1.60 ? 18% perf-profile.calltrace.cycles-pp.schedule_timeout.__wait_for_common.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter
0.16 ?223% +1.5 1.64 ? 18% perf-profile.calltrace.cycles-pp.__wait_for_common.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
0.39 ?110% +1.5 1.88 ? 17% perf-profile.calltrace.cycles-pp.f2fs_issue_flush.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
0.27 ?155% +1.5 1.79 ? 33% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.set_node_addr.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file
0.30 ?154% +1.6 1.93 ? 32% perf-profile.calltrace.cycles-pp.set_node_addr.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
0.65 ?102% +1.7 2.33 ? 17% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_submit_bio.__submit_bio.__submit_bio_noacct.flush_bio_list
0.10 ?223% +1.7 1.84 ? 10% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file
0.61 ?106% +1.8 2.40 ? 17% perf-profile.calltrace.cycles-pp.__submit_bio.__submit_bio_noacct.flush_bio_list.raid1_unplug.__blk_flush_plug
0.61 ?106% +1.8 2.40 ? 17% perf-profile.calltrace.cycles-pp.__submit_bio_noacct.flush_bio_list.raid1_unplug.__blk_flush_plug.blk_finish_plug
0.73 ? 86% +1.8 2.56 ? 17% perf-profile.calltrace.cycles-pp.blk_finish_plug.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
0.64 ?106% +1.9 2.53 ? 17% perf-profile.calltrace.cycles-pp.flush_bio_list.raid1_unplug.__blk_flush_plug.blk_finish_plug.__f2fs_write_data_pages
0.15 ?223% +1.9 2.04 ? 15% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page.__write_node_page
0.64 ?106% +1.9 2.54 ? 17% perf-profile.calltrace.cycles-pp.raid1_unplug.__blk_flush_plug.blk_finish_plug.__f2fs_write_data_pages.do_writepages
0.65 ?106% +1.9 2.56 ? 17% perf-profile.calltrace.cycles-pp.__blk_flush_plug.blk_finish_plug.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc
0.36 ?111% +1.9 2.29 ? 15% perf-profile.calltrace.cycles-pp.f2fs_submit_page_write.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages
0.14 ?223% +2.1 2.26 ? 24% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.f2fs_allocate_data_block.do_write_page.f2fs_do_write_node_page
0.36 ?109% +2.3 2.63 ? 23% perf-profile.calltrace.cycles-pp.__mutex_lock.f2fs_allocate_data_block.do_write_page.f2fs_do_write_node_page.__write_node_page
0.58 ?107% +2.6 3.21 ? 19% perf-profile.calltrace.cycles-pp.f2fs_do_write_data_page.f2fs_write_single_data_page.f2fs_write_cache_pages.__f2fs_write_data_pages.do_writepages
0.48 ?109% +2.7 3.20 ? 22% perf-profile.calltrace.cycles-pp.f2fs_allocate_data_block.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages
0.62 ?107% +2.8 3.43 ? 19% perf-profile.calltrace.cycles-pp.f2fs_write_single_data_page.f2fs_write_cache_pages.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc
1.39 ? 69% +4.2 5.61 ? 18% perf-profile.calltrace.cycles-pp.f2fs_write_cache_pages.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
1.02 ? 91% +4.9 5.89 ? 18% perf-profile.calltrace.cycles-pp.do_write_page.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file
1.02 ? 91% +4.9 5.89 ? 18% perf-profile.calltrace.cycles-pp.f2fs_do_write_node_page.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter
2.34 ? 60% +6.0 8.30 ? 17% perf-profile.calltrace.cycles-pp.__f2fs_write_data_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range
2.35 ? 60% +6.0 8.30 ? 17% perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file
2.36 ? 60% +6.0 8.32 ? 17% perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter
2.36 ? 60% +6.0 8.33 ? 17% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
2.39 ? 60% +6.1 8.47 ? 17% perf-profile.calltrace.cycles-pp.file_write_and_wait_range.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
2.00 ? 86% +8.0 9.96 ? 22% perf-profile.calltrace.cycles-pp.__write_node_page.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write
2.89 ? 85% +10.6 13.51 ? 22% perf-profile.calltrace.cycles-pp.f2fs_fsync_node_pages.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write
19.95 ? 22% +19.9 39.80 ? 9% perf-profile.calltrace.cycles-pp.f2fs_do_sync_file.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
20.97 ? 22% +21.0 41.94 ? 9% perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
20.97 ? 22% +21.0 41.96 ? 9% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
21.37 ? 21% +21.0 42.39 ? 9% perf-profile.calltrace.cycles-pp.write
21.00 ? 22% +21.0 42.02 ? 9% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
21.00 ? 22% +21.0 42.04 ? 9% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
21.02 ? 22% +21.1 42.09 ? 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
21.02 ? 22% +21.1 42.10 ? 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
73.04 ? 5% -19.4 53.65 ? 7% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
73.04 ? 5% -19.4 53.65 ? 7% perf-profile.children.cycles-pp.cpu_startup_entry
73.01 ? 5% -19.4 53.63 ? 7% perf-profile.children.cycles-pp.do_idle
72.28 ? 5% -19.2 53.04 ? 7% perf-profile.children.cycles-pp.start_secondary
68.17 ? 5% -18.5 49.67 ? 7% perf-profile.children.cycles-pp.cpuidle_idle_call
66.30 ? 5% -17.7 48.57 ? 7% perf-profile.children.cycles-pp.cpuidle_enter
66.28 ? 5% -17.7 48.56 ? 7% perf-profile.children.cycles-pp.cpuidle_enter_state
16.17 ? 16% -8.9 7.24 ? 34% perf-profile.children.cycles-pp.poll_idle
49.10 ? 2% -8.4 40.65 ? 3% perf-profile.children.cycles-pp.mwait_idle_with_hints
25.27 ? 7% -8.0 17.24 ? 23% perf-profile.children.cycles-pp.intel_idle_irq
4.32 ? 26% -3.2 1.12 ? 39% perf-profile.children.cycles-pp.folio_wake_bit
3.82 ? 27% -2.9 0.94 ? 37% perf-profile.children.cycles-pp.__wake_up_common
3.69 ? 28% -2.8 0.85 ? 39% perf-profile.children.cycles-pp.wake_page_function
3.96 ? 25% -2.7 1.27 ? 23% perf-profile.children.cycles-pp.folio_end_writeback
4.09 ? 23% -2.5 1.64 ? 14% perf-profile.children.cycles-pp.f2fs_write_end_io
4.17 ? 23% -2.4 1.80 ? 12% perf-profile.children.cycles-pp.raid_end_bio_io
5.26 ? 6% -2.4 2.90 ? 21% perf-profile.children.cycles-pp.io_schedule
4.55 ? 21% -2.2 2.31 ? 6% perf-profile.children.cycles-pp.flush_pending_writes
4.55 ? 20% -2.2 2.31 ? 6% perf-profile.children.cycles-pp.raid1d
4.56 ? 20% -2.2 2.33 ? 6% perf-profile.children.cycles-pp.md_thread
5.54 ? 17% -2.1 3.43 ? 9% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
4.33 ? 20% -2.1 2.23 ? 6% perf-profile.children.cycles-pp.raid1_end_write_request
4.01 ? 17% -2.0 1.97 ? 6% perf-profile.children.cycles-pp.try_to_wake_up
4.94 ? 15% -1.7 3.20 ? 3% perf-profile.children.cycles-pp.ret_from_fork
4.94 ? 15% -1.7 3.20 ? 3% perf-profile.children.cycles-pp.kthread
2.01 ? 5% -0.9 1.15 ? 3% perf-profile.children.cycles-pp.schedule_idle
1.58 ? 9% -0.7 0.90 ? 9% perf-profile.children.cycles-pp.menu_select
1.84 ? 9% -0.6 1.29 ? 7% perf-profile.children.cycles-pp.ttwu_do_activate
1.51 ? 7% -0.5 1.00 ? 4% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
1.61 ? 9% -0.4 1.17 ? 5% perf-profile.children.cycles-pp.enqueue_task_fair
0.76 ? 6% -0.4 0.38 ? 4% perf-profile.children.cycles-pp.update_rq_clock
1.05 ? 8% -0.3 0.71 ? 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
1.38 ? 12% -0.3 1.05 ? 8% perf-profile.children.cycles-pp.down_read
1.04 ? 8% -0.3 0.70 ? 5% perf-profile.children.cycles-pp.hrtimer_interrupt
0.54 ? 4% -0.3 0.22 ? 9% perf-profile.children.cycles-pp.set_next_entity
1.26 ? 8% -0.3 0.94 ? 5% perf-profile.children.cycles-pp.enqueue_entity
0.51 ? 20% -0.3 0.23 ? 32% perf-profile.children.cycles-pp.f2fs_put_page
0.57 ? 14% -0.3 0.30 ? 9% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.36 ? 11% -0.2 0.12 ? 13% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.41 ? 16% -0.2 0.18 ? 10% perf-profile.children.cycles-pp.tick_nohz_next_event
0.56 ? 9% -0.2 0.35 ? 8% perf-profile.children.cycles-pp.ktime_get
0.55 ? 8% -0.2 0.40 ? 4% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.18 ? 13% -0.1 0.03 ? 70% perf-profile.children.cycles-pp.check_preempt_curr
0.23 ? 12% -0.1 0.09 ? 12% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.20 ? 14% -0.1 0.06 ? 9% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.37 ? 10% -0.1 0.24 ? 9% perf-profile.children.cycles-pp.clockevents_program_event
0.42 ? 7% -0.1 0.29 ? 10% perf-profile.children.cycles-pp.sched_clock_cpu
0.40 ? 5% -0.1 0.28 ? 11% perf-profile.children.cycles-pp.__switch_to_asm
0.96 ? 3% -0.1 0.84 ? 3% perf-profile.children.cycles-pp.update_load_avg
0.40 ? 9% -0.1 0.29 ? 5% perf-profile.children.cycles-pp.tick_sched_timer
0.36 ? 6% -0.1 0.26 ? 8% perf-profile.children.cycles-pp.native_sched_clock
0.37 ? 5% -0.1 0.26 ? 8% perf-profile.children.cycles-pp.__switch_to
1.02 ? 2% -0.1 0.93 ? 4% perf-profile.children.cycles-pp.dequeue_entity
0.18 ? 9% -0.1 0.10 ? 15% perf-profile.children.cycles-pp.reweight_entity
0.32 ? 10% -0.1 0.23 ? 5% perf-profile.children.cycles-pp.tick_sched_handle
0.31 ? 12% -0.1 0.22 ? 6% perf-profile.children.cycles-pp.update_process_times
0.26 ? 3% -0.1 0.20 ? 6% perf-profile.children.cycles-pp.__irq_exit_rcu
0.45 ? 3% -0.1 0.39 ? 3% perf-profile.children.cycles-pp.update_curr
0.14 ? 14% -0.1 0.08 ? 13% perf-profile.children.cycles-pp.rcu_idle_exit
0.24 ? 5% -0.1 0.18 ? 5% perf-profile.children.cycles-pp.__update_load_avg_se
0.15 ? 4% -0.1 0.10 ? 14% perf-profile.children.cycles-pp.read_tsc
0.14 ? 13% -0.1 0.09 ? 10% perf-profile.children.cycles-pp.get_next_timer_interrupt
1.14 ? 2% -0.0 1.09 ? 3% perf-profile.children.cycles-pp.dequeue_task_fair
0.12 ? 6% -0.0 0.07 ? 9% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.20 ? 3% -0.0 0.16 ? 4% perf-profile.children.cycles-pp.__softirqentry_text_start
0.08 ? 7% -0.0 0.03 ? 70% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.07 ? 12% -0.0 0.03 ?102% perf-profile.children.cycles-pp.call_cpuidle
0.20 ? 11% -0.0 0.15 ? 8% perf-profile.children.cycles-pp.scheduler_tick
0.07 ? 15% -0.0 0.03 ?100% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.08 ? 14% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.rcu_eqs_exit
0.06 ? 7% -0.0 0.02 ? 99% perf-profile.children.cycles-pp.put_prev_task_fair
0.12 ? 6% -0.0 0.08 ? 7% perf-profile.children.cycles-pp.rebalance_domains
0.09 ? 16% -0.0 0.06 ? 13% perf-profile.children.cycles-pp.rcu_dynticks_inc
0.11 ? 6% -0.0 0.07 ? 12% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.09 ? 5% -0.0 0.05 ? 7% perf-profile.children.cycles-pp.lapic_next_deadline
0.10 ? 8% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.__calc_delta
0.07 ? 15% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.hrtimer_get_next_event
0.08 ? 5% -0.0 0.06 ? 8% perf-profile.children.cycles-pp.native_irq_return_iret
0.08 ? 11% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.update_min_vruntime
0.26 -0.0 0.24 perf-profile.children.cycles-pp.prepare_task_switch
0.05 ? 47% +0.0 0.08 ? 16% perf-profile.children.cycles-pp.__slab_free
0.09 ? 14% +0.0 0.12 ? 7% perf-profile.children.cycles-pp.__might_resched
0.06 ? 21% +0.0 0.11 ? 12% perf-profile.children.cycles-pp.__update_blocked_fair
0.02 ?144% +0.1 0.07 ? 16% perf-profile.children.cycles-pp.bio_associate_blkg_from_css
0.04 ? 75% +0.1 0.09 ? 19% perf-profile.children.cycles-pp.kmem_cache_alloc
0.04 ? 75% +0.1 0.10 ? 15% perf-profile.children.cycles-pp.queue_work_on
0.03 ?103% +0.1 0.09 ? 15% perf-profile.children.cycles-pp.__queue_work
0.01 ?223% +0.1 0.06 ? 11% perf-profile.children.cycles-pp._wait_barrier
0.10 ? 14% +0.1 0.16 ? 14% perf-profile.children.cycles-pp.kmem_cache_free
0.01 ?223% +0.1 0.07 ? 17% perf-profile.children.cycles-pp.f2fs_lookup_rb_tree
0.01 ?223% +0.1 0.07 ? 14% perf-profile.children.cycles-pp.folio_clear_dirty_for_io
0.01 ?223% +0.1 0.07 ? 14% perf-profile.children.cycles-pp.update_sd_pick_busiest
0.01 ?223% +0.1 0.07 ? 14% perf-profile.children.cycles-pp.sb_mark_inode_writeback
0.06 ? 56% +0.1 0.13 ? 15% perf-profile.children.cycles-pp.md_flush_request
0.03 ?100% +0.1 0.09 ? 14% perf-profile.children.cycles-pp.__bio_alloc
0.01 ?223% +0.1 0.08 ? 16% perf-profile.children.cycles-pp.mutex_lock
0.02 ?141% +0.1 0.08 ? 17% perf-profile.children.cycles-pp.__mark_inode_dirty
0.06 ? 51% +0.1 0.13 ? 15% perf-profile.children.cycles-pp.mempool_alloc
0.02 ?142% +0.1 0.09 ? 15% perf-profile.children.cycles-pp.copyin
0.02 ?142% +0.1 0.09 ? 15% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.01 ?223% +0.1 0.08 ? 13% perf-profile.children.cycles-pp.cpumask_next
0.40 ? 3% +0.1 0.46 ? 5% perf-profile.children.cycles-pp.up_read
0.01 ?223% +0.1 0.08 ? 17% perf-profile.children.cycles-pp.rmqueue
0.02 ?152% +0.1 0.10 ? 23% perf-profile.children.cycles-pp.xas_load
0.01 ?223% +0.1 0.08 ? 22% perf-profile.children.cycles-pp.f2fs_update_dirty_folio
0.04 ?104% +0.1 0.11 ? 15% perf-profile.children.cycles-pp.submit_bio_noacct
0.02 ?144% +0.1 0.10 ? 19% perf-profile.children.cycles-pp.copy_page_from_iter_atomic
0.01 ?223% +0.1 0.09 ? 20% perf-profile.children.cycles-pp.read_node_page
0.01 ?223% +0.1 0.08 ? 22% perf-profile.children.cycles-pp.percpu_counter_add_batch
0.05 ? 73% +0.1 0.13 ? 14% perf-profile.children.cycles-pp.bio_alloc_clone
0.01 ?223% +0.1 0.09 ? 15% perf-profile.children.cycles-pp.f2fs_update_data_blkaddr
0.03 ?102% +0.1 0.11 ? 18% perf-profile.children.cycles-pp.__mod_lruvec_page_state
0.01 ?223% +0.1 0.09 ? 23% perf-profile.children.cycles-pp.f2fs_get_node_page
0.01 ?223% +0.1 0.09 ? 17% perf-profile.children.cycles-pp.wake_q_add
0.00 +0.1 0.08 ? 20% perf-profile.children.cycles-pp.osq_unlock
0.04 ? 78% +0.1 0.13 ? 18% perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush
0.16 ? 17% +0.1 0.25 ? 9% perf-profile.children.cycles-pp.update_blocked_averages
0.10 ? 34% +0.1 0.19 ? 14% perf-profile.children.cycles-pp.bio_alloc_bioset
0.16 ? 32% +0.1 0.25 ? 14% perf-profile.children.cycles-pp.process_one_work
0.04 ?105% +0.1 0.13 ? 31% perf-profile.children.cycles-pp.f2fs_invalidate_blocks
0.01 ?223% +0.1 0.11 ? 21% perf-profile.children.cycles-pp.wake_affine
0.01 ?223% +0.1 0.11 ? 17% perf-profile.children.cycles-pp.__is_cp_guaranteed
0.07 ? 61% +0.1 0.17 ? 18% perf-profile.children.cycles-pp.mem_cgroup_wb_stats
0.07 ? 58% +0.1 0.17 ? 18% perf-profile.children.cycles-pp.cgroup_rstat_flush_locked
0.01 ?223% +0.1 0.12 ? 10% perf-profile.children.cycles-pp.__remove_ino_entry
0.07 ? 58% +0.1 0.17 ? 18% perf-profile.children.cycles-pp.__mem_cgroup_flush_stats
0.07 ? 58% +0.1 0.17 ? 18% perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe
0.04 ?104% +0.1 0.14 ? 30% perf-profile.children.cycles-pp.f2fs_truncate
0.04 ?104% +0.1 0.14 ? 30% perf-profile.children.cycles-pp.f2fs_do_truncate_blocks
0.04 ?104% +0.1 0.14 ? 30% perf-profile.children.cycles-pp.f2fs_truncate_data_blocks_range
0.07 ? 61% +0.1 0.17 ? 17% perf-profile.children.cycles-pp.balance_dirty_pages
0.06 ? 83% +0.1 0.16 ? 19% perf-profile.children.cycles-pp.__wake_up_common_lock
0.04 ? 76% +0.1 0.15 ? 16% perf-profile.children.cycles-pp.f2fs_del_fsync_node_entry
0.01 ?223% +0.1 0.13 ? 22% perf-profile.children.cycles-pp.__filemap_fdatawait_range
0.00 +0.1 0.12 ? 22% perf-profile.children.cycles-pp.__insert_extent_tree
0.15 ? 31% +0.1 0.27 ? 10% perf-profile.children.cycles-pp.__submit_merged_bio
0.03 ?106% +0.1 0.15 ? 18% perf-profile.children.cycles-pp.up_write
0.04 ?109% +0.1 0.15 ? 19% perf-profile.children.cycles-pp.update_sit_entry
0.14 ? 36% +0.1 0.26 ? 16% perf-profile.children.cycles-pp.submit_bio_wait
0.14 ? 33% +0.1 0.26 ? 16% perf-profile.children.cycles-pp.blkdev_issue_flush
0.14 ? 35% +0.1 0.26 ? 17% perf-profile.children.cycles-pp.__submit_flush_wait
0.08 ? 60% +0.1 0.21 ? 14% perf-profile.children.cycles-pp.__has_merged_page
0.08 ? 59% +0.1 0.21 ? 19% perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
0.00 +0.1 0.13 ? 19% perf-profile.children.cycles-pp.f2fs_update_extent_tree_range
0.06 ? 82% +0.1 0.19 ? 14% perf-profile.children.cycles-pp.f2fs_update_inode
0.04 ?103% +0.1 0.17 ? 17% perf-profile.children.cycles-pp.__folio_mark_dirty
0.04 ?115% +0.1 0.17 ? 20% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.08 ? 38% +0.1 0.22 ? 21% perf-profile.children.cycles-pp.clear_page_erms
0.16 ? 28% +0.1 0.30 ? 12% perf-profile.children.cycles-pp.f2fs_write_inode
0.10 ? 52% +0.1 0.24 ? 17% perf-profile.children.cycles-pp.__filemap_get_folio
0.10 ? 52% +0.1 0.24 ? 16% perf-profile.children.cycles-pp.pagecache_get_page
0.04 ?112% +0.2 0.19 ? 20% perf-profile.children.cycles-pp.sysvec_call_function_single
0.05 ?106% +0.2 0.20 ? 14% perf-profile.children.cycles-pp.down_write
0.05 ?104% +0.2 0.20 ? 20% perf-profile.children.cycles-pp.f2fs_dirty_data_folio
0.03 ?155% +0.2 0.18 ? 18% perf-profile.children.cycles-pp.f2fs_balance_fs
0.05 ?117% +0.2 0.21 ? 27% perf-profile.children.cycles-pp.xas_find_marked
0.05 ?109% +0.2 0.21 ? 18% perf-profile.children.cycles-pp.complete
0.08 ? 72% +0.2 0.25 ? 14% perf-profile.children.cycles-pp.__get_node_page
0.07 ? 82% +0.2 0.24 ? 25% perf-profile.children.cycles-pp.__dentry_kill
0.07 ? 82% +0.2 0.24 ? 25% perf-profile.children.cycles-pp.evict
0.07 ? 82% +0.2 0.24 ? 25% perf-profile.children.cycles-pp.f2fs_evict_inode
0.22 ? 33% +0.2 0.38 ? 14% perf-profile.children.cycles-pp.worker_thread
0.11 ? 54% +0.2 0.28 ? 19% perf-profile.children.cycles-pp.f2fs_write_end
0.08 ? 65% +0.2 0.25 ? 14% perf-profile.children.cycles-pp.f2fs_update_inode_page
0.07 ? 81% +0.2 0.24 ? 25% perf-profile.children.cycles-pp.exit_to_user_mode_loop
0.07 ? 82% +0.2 0.24 ? 26% perf-profile.children.cycles-pp.dput
0.07 ? 82% +0.2 0.24 ? 26% perf-profile.children.cycles-pp.__fput
0.07 ? 82% +0.2 0.24 ? 26% perf-profile.children.cycles-pp.dentry_kill
0.07 ? 82% +0.2 0.24 ? 25% perf-profile.children.cycles-pp.task_work_run
0.07 ? 82% +0.2 0.24 ? 25% perf-profile.children.cycles-pp.__close
0.05 ?109% +0.2 0.23 ? 15% perf-profile.children.cycles-pp.f2fs_dirty_node_folio
0.21 ? 26% +0.2 0.40 ? 15% perf-profile.children.cycles-pp.finish_task_switch
0.05 ?110% +0.2 0.24 ? 19% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.08 ? 82% +0.2 0.28 ? 24% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
0.09 ? 66% +0.2 0.28 ? 16% perf-profile.children.cycles-pp.filemap_dirty_folio
0.10 ? 63% +0.2 0.30 ? 18% perf-profile.children.cycles-pp.__folio_end_writeback
0.12 ? 39% +0.2 0.32 ? 20% perf-profile.children.cycles-pp.get_page_from_freelist
0.08 ? 83% +0.2 0.29 ? 23% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.12 ? 44% +0.2 0.32 ? 13% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
0.05 ?111% +0.2 0.26 ? 20% perf-profile.children.cycles-pp.f2fs_is_valid_blkaddr
0.13 ? 41% +0.2 0.34 ? 21% perf-profile.children.cycles-pp.__alloc_pages
0.26 ? 25% +0.2 0.48 ? 10% perf-profile.children.cycles-pp._find_next_bit
0.04 ?114% +0.2 0.28 ? 26% perf-profile.children.cycles-pp.__lookup_nat_cache
0.08 ? 74% +0.2 0.33 ? 14% perf-profile.children.cycles-pp.rwsem_mark_wake
0.16 ? 40% +0.2 0.41 ? 16% perf-profile.children.cycles-pp.__radix_tree_lookup
0.34 ? 21% +0.2 0.58 ? 10% perf-profile.children.cycles-pp.cpumask_next_and
0.06 ?117% +0.2 0.31 ? 29% perf-profile.children.cycles-pp.release_pages
0.09 ? 88% +0.2 0.34 ? 15% perf-profile.children.cycles-pp.find_busiest_queue
0.07 ?115% +0.2 0.32 ? 29% perf-profile.children.cycles-pp.__pagevec_release
0.09 ? 70% +0.3 0.34 ? 16% perf-profile.children.cycles-pp.__folio_start_writeback
0.70 ? 14% +0.3 0.96 ? 8% perf-profile.children.cycles-pp.idle_cpu
0.24 ? 38% +0.3 0.55 ? 12% perf-profile.children.cycles-pp.raid1_write_request
2.08 ? 2% +0.3 2.40 ? 10% perf-profile.children.cycles-pp.flush_smp_call_function_queue
0.06 ?114% +0.3 0.40 ? 21% perf-profile.children.cycles-pp.mutex_spin_on_owner
0.13 ? 89% +0.3 0.47 ? 19% perf-profile.children.cycles-pp.issue_flush_thread
0.34 ? 40% +0.4 0.77 ? 13% perf-profile.children.cycles-pp.raid1_make_request
0.37 ? 40% +0.5 0.85 ? 14% perf-profile.children.cycles-pp.md_handle_request
0.63 ? 12% +0.5 1.12 ? 11% perf-profile.children.cycles-pp.prepare_write_begin
0.40 ? 40% +0.5 0.91 ? 14% perf-profile.children.cycles-pp.md_submit_bio
1.32 ? 3% +0.5 1.83 ? 8% perf-profile.children.cycles-pp.sched_ttwu_pending
0.14 ? 79% +0.5 0.66 ? 18% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.41 ? 37% +0.5 0.95 ? 18% perf-profile.children.cycles-pp.brd_insert_page
0.35 ? 78% +0.6 0.94 ? 17% perf-profile.children.cycles-pp.wake_up_q
0.69 ? 15% +0.6 1.30 ? 11% perf-profile.children.cycles-pp.f2fs_write_begin
0.21 ?108% +0.6 0.84 ? 30% perf-profile.children.cycles-pp.find_get_pages_range_tag
0.21 ?108% +0.6 0.84 ? 29% perf-profile.children.cycles-pp.pagevec_lookup_range_tag
0.36 ? 51% +0.6 0.99 ? 17% perf-profile.children.cycles-pp.copy_to_brd
0.24 ? 69% +0.7 0.93 ? 18% perf-profile.children.cycles-pp.f2fs_merge_page_bio
0.20 ? 69% +0.8 0.95 ? 38% perf-profile.children.cycles-pp.__percpu_counter_sum
0.19 ?109% +0.8 0.96 ? 30% perf-profile.children.cycles-pp.f2fs_need_inode_block_update
0.27 ? 69% +0.8 1.05 ? 17% perf-profile.children.cycles-pp.f2fs_inplace_write_data
0.20 ? 69% +0.8 0.98 ? 37% perf-profile.children.cycles-pp.f2fs_space_for_roll_forward
0.18 ? 73% +0.8 0.97 ? 20% perf-profile.children.cycles-pp.f2fs_outplace_write_data
0.44 ? 75% +0.9 1.32 ? 17% perf-profile.children.cycles-pp.rwsem_wake
0.17 ? 78% +0.9 1.12 ? 21% perf-profile.children.cycles-pp.f2fs_lookup_extent_tree
0.95 ? 22% +1.0 1.92 ? 13% perf-profile.children.cycles-pp.generic_perform_write
0.95 ? 23% +1.0 1.94 ? 13% perf-profile.children.cycles-pp.f2fs_buffered_write_iter
0.27 ?102% +1.0 1.28 ? 30% perf-profile.children.cycles-pp.f2fs_is_checkpointed_node
0.25 ?108% +1.1 1.31 ? 34% perf-profile.children.cycles-pp.f2fs_get_node_info
0.34 ? 93% +1.3 1.62 ? 18% perf-profile.children.cycles-pp.schedule_timeout
0.38 ? 89% +1.3 1.71 ? 18% perf-profile.children.cycles-pp.__wait_for_common
1.22 ? 16% +1.4 2.60 ? 13% perf-profile.children.cycles-pp._raw_spin_lock
0.48 ? 73% +1.4 1.88 ? 17% perf-profile.children.cycles-pp.f2fs_issue_flush
0.93 ? 42% +1.4 2.33 ? 17% perf-profile.children.cycles-pp.brd_do_bvec
0.94 ? 42% +1.4 2.35 ? 17% perf-profile.children.cycles-pp.brd_submit_bio
0.42 ?100% +1.5 1.94 ? 32% perf-profile.children.cycles-pp.set_node_addr
2.08 ? 5% +1.6 3.66 ? 8% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.86 ? 57% +1.7 2.54 ? 17% perf-profile.children.cycles-pp.raid1_unplug
0.87 ? 56% +1.7 2.56 ? 17% perf-profile.children.cycles-pp.blk_finish_plug
0.87 ? 56% +1.7 2.56 ? 17% perf-profile.children.cycles-pp.__blk_flush_plug
2.59 ? 6% +1.8 4.37 ? 5% perf-profile.children.cycles-pp.__submit_merged_write_cond
0.78 ? 55% +1.9 2.73 ? 14% perf-profile.children.cycles-pp.f2fs_submit_merged_ipu_write
0.46 ? 81% +2.0 2.50 ? 15% perf-profile.children.cycles-pp.f2fs_submit_page_write
3.67 ? 20% +2.1 5.78 ? 8% perf-profile.children.cycles-pp.update_sg_lb_stats
8.63 ? 10% +2.1 10.74 ? 6% perf-profile.children.cycles-pp.__schedule
3.82 ? 20% +2.2 6.05 ? 8% perf-profile.children.cycles-pp.update_sd_lb_stats
3.87 ? 20% +2.3 6.14 ? 8% perf-profile.children.cycles-pp.find_busiest_group
0.47 ? 75% +2.4 2.91 ? 23% perf-profile.children.cycles-pp.__mutex_lock
0.75 ? 64% +2.5 3.21 ? 19% perf-profile.children.cycles-pp.f2fs_do_write_data_page
0.80 ? 65% +2.6 3.43 ? 19% perf-profile.children.cycles-pp.f2fs_write_single_data_page
5.18 ? 18% +2.7 7.84 ? 8% perf-profile.children.cycles-pp.pick_next_task_fair
4.12 ? 22% +2.8 6.91 ? 9% perf-profile.children.cycles-pp.load_balance
6.70 ? 15% +2.9 9.64 ? 7% perf-profile.children.cycles-pp.schedule
0.71 ? 82% +2.9 3.65 ? 18% perf-profile.children.cycles-pp.rwsem_optimistic_spin
0.69 ? 72% +3.0 3.72 ? 22% perf-profile.children.cycles-pp.f2fs_allocate_data_block
4.44 ? 22% +3.1 7.50 ? 9% perf-profile.children.cycles-pp.newidle_balance
2.65 ? 24% +3.4 6.02 ? 5% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.86 ? 78% +4.2 5.04 ? 21% perf-profile.children.cycles-pp.osq_lock
1.44 ? 62% +4.2 5.62 ? 18% perf-profile.children.cycles-pp.f2fs_write_cache_pages
1.14 ? 84% +4.6 5.72 ? 19% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
1.12 ? 75% +4.8 5.89 ? 18% perf-profile.children.cycles-pp.f2fs_do_write_node_page
1.26 ? 74% +5.4 6.64 ? 18% perf-profile.children.cycles-pp.do_write_page
1.35 ? 85% +5.8 7.16 ? 19% perf-profile.children.cycles-pp.rwsem_down_read_slowpath
3.33 ? 24% +5.9 9.20 ? 5% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
2.34 ? 60% +6.0 8.30 ? 17% perf-profile.children.cycles-pp.__f2fs_write_data_pages
2.35 ? 60% +6.0 8.30 ? 17% perf-profile.children.cycles-pp.do_writepages
2.36 ? 60% +6.0 8.32 ? 17% perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
2.36 ? 60% +6.0 8.33 ? 17% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
2.39 ? 60% +6.1 8.47 ? 17% perf-profile.children.cycles-pp.file_write_and_wait_range
2.03 ? 84% +7.9 9.96 ? 22% perf-profile.children.cycles-pp.__write_node_page
2.95 ? 81% +10.6 13.52 ? 22% perf-profile.children.cycles-pp.f2fs_fsync_node_pages
19.95 ? 22% +19.9 39.80 ? 9% perf-profile.children.cycles-pp.f2fs_do_sync_file
20.97 ? 22% +21.0 41.94 ? 9% perf-profile.children.cycles-pp.f2fs_file_write_iter
20.98 ? 22% +21.0 41.97 ? 9% perf-profile.children.cycles-pp.new_sync_write
21.00 ? 22% +21.0 42.04 ? 9% perf-profile.children.cycles-pp.vfs_write
21.01 ? 22% +21.0 42.05 ? 9% perf-profile.children.cycles-pp.ksys_write
21.37 ? 21% +21.0 42.41 ? 9% perf-profile.children.cycles-pp.write
21.18 ? 22% +21.3 42.45 ? 9% perf-profile.children.cycles-pp.do_syscall_64
21.19 ? 22% +21.3 42.46 ? 9% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
15.91 ? 16% -8.8 7.11 ? 34% perf-profile.self.cycles-pp.poll_idle
48.31 ? 2% -8.0 40.27 ? 2% perf-profile.self.cycles-pp.mwait_idle_with_hints
1.05 ? 8% -0.5 0.52 ? 31% perf-profile.self.cycles-pp.folio_wait_bit_common
0.87 ? 11% -0.4 0.46 ? 24% perf-profile.self.cycles-pp.f2fs_wait_on_node_pages_writeback
0.91 ? 8% -0.4 0.53 ? 10% perf-profile.self.cycles-pp.menu_select
0.60 ? 7% -0.3 0.27 ? 4% perf-profile.self.cycles-pp.update_rq_clock
1.27 ? 13% -0.3 0.96 ? 7% perf-profile.self.cycles-pp.down_read
0.35 ? 11% -0.2 0.12 ? 13% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.69 -0.2 0.47 ? 2% perf-profile.self.cycles-pp.__schedule
0.27 ? 8% -0.2 0.05 ? 8% perf-profile.self.cycles-pp.set_next_entity
0.31 ? 16% -0.2 0.12 ? 39% perf-profile.self.cycles-pp.folio_wait_writeback
0.52 ? 9% -0.2 0.35 ? 6% perf-profile.self.cycles-pp.enqueue_entity
0.42 ? 12% -0.2 0.25 ? 10% perf-profile.self.cycles-pp.ktime_get
0.36 ? 12% -0.1 0.23 ? 4% perf-profile.self.cycles-pp.enqueue_task_fair
0.40 ? 5% -0.1 0.28 ? 11% perf-profile.self.cycles-pp.__switch_to_asm
0.22 ? 9% -0.1 0.10 ? 9% perf-profile.self.cycles-pp.do_idle
0.36 ? 5% -0.1 0.26 ? 8% perf-profile.self.cycles-pp.__switch_to
0.23 ? 16% -0.1 0.12 ? 32% perf-profile.self.cycles-pp.ttwu_do_activate
0.35 ? 6% -0.1 0.24 ? 7% perf-profile.self.cycles-pp.native_sched_clock
0.36 ? 5% -0.1 0.27 ? 11% perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.18 ? 9% -0.1 0.10 ? 14% perf-profile.self.cycles-pp.reweight_entity
1.12 ? 2% -0.1 1.04 ? 3% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.24 ? 5% -0.1 0.18 ? 5% perf-profile.self.cycles-pp.__update_load_avg_se
0.12 ? 8% -0.1 0.06 ? 49% perf-profile.self.cycles-pp.f2fs_wait_on_page_writeback
0.15 ? 4% -0.1 0.10 ? 10% perf-profile.self.cycles-pp.read_tsc
0.31 ? 12% -0.1 0.26 ? 4% perf-profile.self.cycles-pp.llist_reverse_order
0.13 ? 10% -0.1 0.08 ? 14% perf-profile.self.cycles-pp.cpuidle_idle_call
0.07 ? 12% -0.0 0.03 ?102% perf-profile.self.cycles-pp.call_cpuidle
0.09 ? 17% -0.0 0.05 ? 47% perf-profile.self.cycles-pp.rcu_dynticks_inc
0.13 ? 11% -0.0 0.09 ? 7% perf-profile.self.cycles-pp.cpuidle_enter_state
0.11 ? 6% -0.0 0.07 ? 12% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.09 ? 5% -0.0 0.05 ? 7% perf-profile.self.cycles-pp.lapic_next_deadline
0.27 ? 5% -0.0 0.23 ? 2% perf-profile.self.cycles-pp.__list_add_valid
0.09 ? 11% -0.0 0.06 ? 9% perf-profile.self.cycles-pp.__calc_delta
0.07 ? 6% -0.0 0.04 ? 45% perf-profile.self.cycles-pp.update_min_vruntime
0.08 ? 5% -0.0 0.06 ? 8% perf-profile.self.cycles-pp.native_irq_return_iret
0.12 ? 6% -0.0 0.11 ? 4% perf-profile.self.cycles-pp.dequeue_entity
0.15 ? 4% +0.0 0.18 ? 6% perf-profile.self.cycles-pp.update_curr
0.13 ? 12% +0.0 0.16 ? 5% perf-profile.self.cycles-pp.finish_task_switch
0.08 ? 11% +0.0 0.12 ? 4% perf-profile.self.cycles-pp.__might_resched
0.10 ? 9% +0.0 0.14 ? 4% perf-profile.self.cycles-pp.dequeue_task_fair
0.04 ? 71% +0.0 0.08 ? 10% perf-profile.self.cycles-pp.find_busiest_group
0.42 ? 4% +0.0 0.46 ? 2% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.10 ? 13% +0.0 0.14 ? 5% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.02 ?144% +0.1 0.07 ? 16% perf-profile.self.cycles-pp.bio_associate_blkg_from_css
0.01 ?223% +0.1 0.06 ? 11% perf-profile.self.cycles-pp._wait_barrier
0.15 ? 6% +0.1 0.20 ? 9% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.01 ?223% +0.1 0.07 ? 14% perf-profile.self.cycles-pp.mutex_lock
0.10 ? 23% +0.1 0.16 ? 10% perf-profile.self.cycles-pp.cpumask_next_and
0.40 ? 3% +0.1 0.46 ? 4% perf-profile.self.cycles-pp.up_read
0.01 ?223% +0.1 0.07 ? 17% perf-profile.self.cycles-pp.f2fs_lookup_rb_tree
0.00 +0.1 0.06 ? 11% perf-profile.self.cycles-pp.update_sd_pick_busiest
0.01 ?223% +0.1 0.08 ? 20% perf-profile.self.cycles-pp.__folio_start_writeback
0.00 +0.1 0.07 ? 14% perf-profile.self.cycles-pp.f2fs_file_write_iter
0.02 ?142% +0.1 0.08 ? 20% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.01 ?223% +0.1 0.08 ? 20% perf-profile.self.cycles-pp.percpu_counter_add_batch
0.08 ? 24% +0.1 0.15 ? 7% perf-profile.self.cycles-pp.load_balance
0.01 ?223% +0.1 0.08 ? 13% perf-profile.self.cycles-pp.raid_end_bio_io
0.01 ?223% +0.1 0.08 ? 17% perf-profile.self.cycles-pp.md_handle_request
0.00 +0.1 0.07 ? 14% perf-profile.self.cycles-pp.f2fs_issue_flush
0.03 ?102% +0.1 0.11 ? 21% perf-profile.self.cycles-pp.brd_do_bvec
0.01 ?223% +0.1 0.09 ? 17% perf-profile.self.cycles-pp.f2fs_submit_page_write
0.04 ? 76% +0.1 0.12 ? 18% perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush
0.01 ?223% +0.1 0.10 ? 21% perf-profile.self.cycles-pp.__write_node_page
0.00 +0.1 0.08 ? 19% perf-profile.self.cycles-pp.osq_unlock
0.01 ?223% +0.1 0.09 ? 17% perf-profile.self.cycles-pp.wake_q_add
0.17 ? 15% +0.1 0.26 ? 4% perf-profile.self.cycles-pp.newidle_balance
0.01 ?223% +0.1 0.10 ? 13% perf-profile.self.cycles-pp.f2fs_inplace_write_data
0.01 ?223% +0.1 0.10 ? 22% perf-profile.self.cycles-pp.f2fs_write_end_io
0.00 +0.1 0.09 ? 23% perf-profile.self.cycles-pp.f2fs_get_node_page
0.12 ? 12% +0.1 0.21 ? 14% perf-profile.self.cycles-pp.f2fs_submit_merged_ipu_write
0.02 ?144% +0.1 0.11 ? 20% perf-profile.self.cycles-pp.__f2fs_write_data_pages
0.12 ? 23% +0.1 0.21 ? 9% perf-profile.self.cycles-pp.update_sd_lb_stats
0.03 ?105% +0.1 0.13 ? 17% perf-profile.self.cycles-pp.__filemap_get_folio
0.01 ?223% +0.1 0.11 ? 17% perf-profile.self.cycles-pp.__is_cp_guaranteed
0.01 ?223% +0.1 0.12 ? 17% perf-profile.self.cycles-pp.rwsem_down_write_slowpath
0.03 ?106% +0.1 0.15 ? 18% perf-profile.self.cycles-pp.update_sit_entry
0.03 ?106% +0.1 0.15 ? 20% perf-profile.self.cycles-pp.up_write
0.08 ? 63% +0.1 0.20 ? 14% perf-profile.self.cycles-pp.__has_merged_page
0.03 ?111% +0.1 0.16 ? 13% perf-profile.self.cycles-pp.f2fs_do_sync_file
0.04 ?115% +0.1 0.18 ? 25% perf-profile.self.cycles-pp.f2fs_fsync_node_pages
0.04 ?110% +0.1 0.18 ? 14% perf-profile.self.cycles-pp.down_write
0.03 ?155% +0.1 0.16 ? 19% perf-profile.self.cycles-pp.f2fs_allocate_data_block
0.07 ? 59% +0.1 0.21 ? 21% perf-profile.self.cycles-pp.clear_page_erms
0.05 ?117% +0.2 0.21 ? 27% perf-profile.self.cycles-pp.xas_find_marked
0.04 ?112% +0.2 0.20 ? 13% perf-profile.self.cycles-pp.rwsem_mark_wake
0.10 ? 45% +0.2 0.27 ? 12% perf-profile.self.cycles-pp.raid1_write_request
0.02 ?223% +0.2 0.18 ? 17% perf-profile.self.cycles-pp.f2fs_balance_fs
0.08 ? 81% +0.2 0.27 ? 28% perf-profile.self.cycles-pp.__percpu_counter_sum
0.24 ? 26% +0.2 0.45 ? 10% perf-profile.self.cycles-pp._find_next_bit
0.05 ?111% +0.2 0.26 ? 20% perf-profile.self.cycles-pp.f2fs_is_valid_blkaddr
0.08 ? 87% +0.2 0.29 ? 16% perf-profile.self.cycles-pp.find_busiest_queue
0.05 ?112% +0.2 0.26 ? 19% perf-profile.self.cycles-pp.f2fs_lookup_extent_tree
0.13 ? 46% +0.2 0.36 ? 17% perf-profile.self.cycles-pp.raid1_end_write_request
0.06 ?120% +0.2 0.29 ? 29% perf-profile.self.cycles-pp.release_pages
0.16 ? 40% +0.2 0.40 ? 16% perf-profile.self.cycles-pp.__radix_tree_lookup
0.68 ? 14% +0.2 0.93 ? 7% perf-profile.self.cycles-pp.idle_cpu
0.08 ? 94% +0.3 0.37 ? 19% perf-profile.self.cycles-pp.rwsem_optimistic_spin
0.06 ?114% +0.3 0.40 ? 21% perf-profile.self.cycles-pp.mutex_spin_on_owner
0.09 ? 77% +0.4 0.46 ? 17% perf-profile.self.cycles-pp.rwsem_down_read_slowpath
0.15 ?117% +0.5 0.63 ? 31% perf-profile.self.cycles-pp.find_get_pages_range_tag
0.13 ? 78% +0.5 0.61 ? 18% perf-profile.self.cycles-pp.rwsem_spin_on_owner
1.07 ? 10% +0.5 1.56 ? 9% perf-profile.self.cycles-pp._raw_spin_lock
0.34 ? 50% +0.6 0.95 ? 17% perf-profile.self.cycles-pp.copy_to_brd
2.70 ? 22% +1.6 4.35 ? 9% perf-profile.self.cycles-pp.update_sg_lb_stats
0.86 ? 78% +4.1 5.01 ? 21% perf-profile.self.cycles-pp.osq_lock
3.32 ? 24% +5.9 9.19 ? 5% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



***************************************************************************************************
lkp-csl-2sp9: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
gcc-11/performance/4BRD_12G/xfs/x86_64-rhel-8.3/300/RAID1/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp9/sync_disk_rw/aim7/0x500320a

commit:
28156108fe ("sched: Fix the check of nr_running at queue wakelist")
f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc
---------------- ---------------------------
%stddev %change %stddev
\ | \
6574 +3.9% 6832 aim7.jobs-per-min
273.98 -3.8% 263.61 aim7.time.elapsed_time
273.98 -3.8% 263.61 aim7.time.elapsed_time.max
980796 +6.2% 1041949 aim7.time.involuntary_context_switches
47635 ? 2% -7.0% 44310 aim7.time.minor_page_faults
6110 ? 2% -8.0% 5624 aim7.time.system_time
69966039 +2.2% 71483752 aim7.time.voluntary_context_switches
34031 ? 6% +18.8% 40422 ? 10% meminfo.KernelStack
16884 ? 10% +26.3% 21319 ? 14% numa-meminfo.node1.KernelStack
16885 ? 10% +26.2% 21316 ? 14% numa-vmstat.node1.nr_kernel_stack
5.77 ? 9% +27.8% 7.38 ? 14% sched_debug.cpu.clock.stddev
9750 -14.0% 8382 uptime.idle
31.73 ? 2% -15.0% 26.97 iostat.cpu.idle
40.12 +13.8% 45.67 iostat.cpu.iowait
31.25 ? 3% -4.8 26.44 mpstat.cpu.all.idle%
40.41 +5.6 46.02 mpstat.cpu.all.iowait%
34031 ? 6% +18.8% 40418 ? 10% proc-vmstat.nr_kernel_stack
56829 +3.4% 58750 ? 2% proc-vmstat.nr_slab_unreclaimable
465583 +3.6% 482389 proc-vmstat.pgactivate
34967 -3.0% 33923 proc-vmstat.pgreuse
31.00 ? 3% -15.1% 26.33 vmstat.cpu.id
39.50 +14.8% 45.33 vmstat.cpu.wa
273156 +4.0% 284008 vmstat.io.bo
509661 +5.3% 536464 vmstat.system.cs
200604 +4.7% 210117 vmstat.system.in
38505571 +7.8% 41516807 perf-stat.i.branch-misses
18.74 +0.9 19.61 ? 2% perf-stat.i.cache-miss-rate%
45792562 +6.5% 48766141 perf-stat.i.cache-misses
514616 +5.2% 541341 perf-stat.i.context-switches
4.99 -3.4% 4.81 perf-stat.i.cpi
89102 +7.5% 95767 perf-stat.i.cpu-migrations
1.291e+09 +6.3% 1.373e+09 perf-stat.i.dTLB-stores
4605702 +6.9e+06% 3.2e+11 ?223% perf-stat.i.iTLB-load-misses
3333 +1.5% 3385 perf-stat.i.minor-faults
16366919 +7.2% 17545996 perf-stat.i.node-load-misses
2107545 +5.4% 2222279 perf-stat.i.node-loads
73.73 +2.4 76.10 perf-stat.i.node-store-miss-rate%
8621292 ? 2% +9.0% 9396473 perf-stat.i.node-store-misses
3335 +1.5% 3386 perf-stat.i.page-faults
1.30 +0.1 1.40 perf-stat.overall.branch-miss-rate%
18.82 +0.9 19.72 perf-stat.overall.cache-miss-rate%
4.97 -3.0% 4.82 perf-stat.overall.cpi
1533 -8.5% 1402 perf-stat.overall.cycles-between-cache-misses
0.20 +3.1% 0.21 perf-stat.overall.ipc
74.18 ? 2% +2.2 76.38 perf-stat.overall.node-store-miss-rate%
38363874 +7.8% 41358628 perf-stat.ps.branch-misses
45621719 +6.5% 48579204 perf-stat.ps.cache-misses
512684 +5.2% 539255 perf-stat.ps.context-switches
88768 +7.5% 95397 perf-stat.ps.cpu-migrations
1.286e+09 +6.3% 1.368e+09 perf-stat.ps.dTLB-stores
4588811 +7e+06% 3.19e+11 ?223% perf-stat.ps.iTLB-load-misses
16305621 +7.2% 17478491 perf-stat.ps.node-load-misses
2099675 +5.4% 2213764 perf-stat.ps.node-loads
8589098 ? 2% +9.0% 9360368 perf-stat.ps.node-store-misses
3.869e+12 -3.3% 3.743e+12 perf-stat.total.instructions
5.91 ? 2% -1.2 4.68 ? 3% perf-profile.calltrace.cycles-pp.__mutex_lock.__flush_workqueue.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq
5.52 ? 2% -1.2 4.31 ? 3% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.__flush_workqueue.xlog_cil_push_now.xlog_cil_force_seq
7.38 ? 2% -1.0 6.42 ? 3% perf-profile.calltrace.cycles-pp.__flush_workqueue.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync
2.32 ? 2% -1.0 1.37 ? 2% perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.xfs_file_fsync.xfs_file_buffered_write.new_sync_write.vfs_write
2.01 -0.9 1.14 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.xfs_file_fsync
2.01 -0.9 1.15 ? 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.xfs_file_fsync.xfs_file_buffered_write
2.01 -0.9 1.15 ? 2% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.xfs_file_fsync.xfs_file_buffered_write.new_sync_write
8.08 -0.8 7.28 ? 3% perf-profile.calltrace.cycles-pp.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write
1.07 ? 2% -0.5 0.59 ? 4% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write
1.07 ? 2% -0.5 0.59 ? 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync
1.06 ? 3% -0.5 0.58 ? 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_force_lsn.xfs_log_force_seq
1.50 -0.4 1.11 perf-profile.calltrace.cycles-pp.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write.new_sync_write
0.72 ? 4% -0.1 0.63 ? 5% perf-profile.calltrace.cycles-pp.xlog_ioend_work.process_one_work.worker_thread.kthread.ret_from_fork
0.59 ? 3% +0.1 0.65 ? 2% perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write
0.84 ? 7% +0.1 0.93 ? 4% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_vn_update_time.file_update_time
1.15 ? 4% +0.1 1.25 ? 4% perf-profile.calltrace.cycles-pp.xlog_cil_commit.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_write_checks
1.22 ? 4% +0.1 1.32 ? 4% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_write_checks.xfs_file_buffered_write
0.65 ? 6% +0.2 0.81 ? 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq
0.68 ? 6% +0.2 0.84 ? 5% perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_push_now.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync
0.72 ? 4% +0.6 1.32 ? 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_seq.xfs_log_force_seq
0.72 ? 4% +0.6 1.34 ? 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync
0.73 ? 4% +0.6 1.34 ? 4% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_seq.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write
10.11 +0.6 10.74 perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.xfs_log_force_seq.xfs_file_fsync.xfs_file_buffered_write.new_sync_write
0.00 +0.7 0.66 ? 3% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
14.74 ? 2% +0.9 15.61 ? 2% perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_write.new_sync_write.vfs_write
0.00 +0.9 0.87 ? 2% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
5.91 ? 2% -1.2 4.68 ? 3% perf-profile.children.cycles-pp.__mutex_lock
5.56 ? 2% -1.2 4.35 ? 3% perf-profile.children.cycles-pp.osq_lock
7.38 ? 2% -1.0 6.42 ? 3% perf-profile.children.cycles-pp.__flush_workqueue
8.08 -0.8 7.28 ? 3% perf-profile.children.cycles-pp.xlog_cil_push_now
2.02 ? 2% -0.5 1.51 perf-profile.children.cycles-pp.xlog_force_lsn
1.46 -0.2 1.27 ? 2% perf-profile.children.cycles-pp.try_to_wake_up
1.16 ? 2% -0.2 1.02 ? 3% perf-profile.children.cycles-pp.__wake_up_common
0.72 ? 4% -0.1 0.63 ? 5% perf-profile.children.cycles-pp.xlog_ioend_work
1.16 ? 2% -0.1 1.10 ? 4% perf-profile.children.cycles-pp.__wake_up_common_lock
0.56 -0.1 0.50 ? 2% perf-profile.children.cycles-pp.schedule_idle
0.28 ? 4% -0.1 0.22 ? 5% perf-profile.children.cycles-pp.folio_wake_bit
0.23 ? 4% -0.1 0.17 ? 6% perf-profile.children.cycles-pp.wake_page_function
0.47 ? 4% -0.1 0.41 ? 5% perf-profile.children.cycles-pp.iomap_finish_ioend
0.40 ? 4% -0.1 0.34 ? 4% perf-profile.children.cycles-pp.folio_end_writeback
0.14 ? 6% -0.1 0.09 ? 10% perf-profile.children.cycles-pp.set_next_entity
0.45 ? 3% -0.0 0.40 ? 6% perf-profile.children.cycles-pp.xlog_state_clean_iclog
0.45 ? 4% -0.0 0.41 ? 3% perf-profile.children.cycles-pp.raid_end_bio_io
0.12 ? 6% -0.0 0.08 ? 5% perf-profile.children.cycles-pp.xlog_state_done_syncing
0.36 ? 3% -0.0 0.32 ? 2% perf-profile.children.cycles-pp.complete
0.08 ? 5% +0.0 0.10 ? 5% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
0.14 ? 4% +0.0 0.16 ? 3% perf-profile.children.cycles-pp.sched_clock_cpu
0.08 ? 7% +0.0 0.10 ? 5% perf-profile.children.cycles-pp.__update_blocked_fair
0.15 ? 2% +0.0 0.16 ? 4% perf-profile.children.cycles-pp.iomap_write_end
0.11 ? 8% +0.0 0.12 ? 4% perf-profile.children.cycles-pp.__switch_to
0.10 ? 4% +0.0 0.12 ? 4% perf-profile.children.cycles-pp.prepare_task_switch
0.18 ? 5% +0.0 0.20 ? 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.18 ? 3% +0.0 0.20 ? 4% perf-profile.children.cycles-pp.bio_alloc_bioset
0.15 ? 5% +0.0 0.17 ? 6% perf-profile.children.cycles-pp.finish_task_switch
0.06 ? 8% +0.0 0.08 ? 4% perf-profile.children.cycles-pp.llist_reverse_order
0.12 ? 7% +0.0 0.14 ? 4% perf-profile.children.cycles-pp.wake_affine
0.03 ? 70% +0.0 0.06 ? 6% perf-profile.children.cycles-pp.___perf_sw_event
0.26 ? 3% +0.0 0.29 ? 4% perf-profile.children.cycles-pp.update_curr
0.15 ? 7% +0.0 0.18 ? 4% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.33 ? 4% +0.0 0.36 ? 4% perf-profile.children.cycles-pp.xlog_cil_push_work
0.51 ? 2% +0.0 0.55 ? 3% perf-profile.children.cycles-pp.dequeue_entity
0.46 ? 4% +0.0 0.51 ? 3% perf-profile.children.cycles-pp.select_task_rq
0.15 ? 5% +0.0 0.20 ? 4% perf-profile.children.cycles-pp.xlog_write
0.09 ? 5% +0.0 0.14 ? 6% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.48 ? 3% +0.1 0.53 ? 4% perf-profile.children.cycles-pp.iomap_write_iter
0.08 ? 9% +0.1 0.13 ? 4% perf-profile.children.cycles-pp.llist_add_batch
0.08 ? 11% +0.1 0.13 ? 3% perf-profile.children.cycles-pp.__smp_call_single_queue
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.xlog_cil_write_commit_record
0.00 +0.1 0.06 ? 8% perf-profile.children.cycles-pp.xlog_cil_set_ctx_write_state
0.00 +0.1 0.06 ? 8% perf-profile.children.cycles-pp.xlog_write_get_more_iclog_space
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.xlog_write_partial
0.59 ? 3% +0.1 0.65 ? 2% perf-profile.children.cycles-pp.iomap_file_buffered_write
0.64 +0.1 0.70 ? 3% perf-profile.children.cycles-pp.dequeue_task_fair
0.47 ? 2% +0.1 0.54 ? 2% perf-profile.children.cycles-pp.enqueue_entity
0.60 ? 2% +0.1 0.67 ? 3% perf-profile.children.cycles-pp.ttwu_do_activate
0.14 ? 7% +0.1 0.23 ? 3% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.62 +0.1 0.71 ? 3% perf-profile.children.cycles-pp.enqueue_task_fair
0.89 ? 7% +0.1 0.99 ? 4% perf-profile.children.cycles-pp.xlog_cil_insert_items
1.22 ? 3% +0.1 1.34 ? 5% perf-profile.children.cycles-pp.xlog_cil_commit
1.30 ? 3% +0.1 1.42 ? 4% perf-profile.children.cycles-pp.__xfs_trans_commit
3.34 ? 3% +0.4 3.72 ? 2% perf-profile.children.cycles-pp._raw_spin_lock
0.37 ? 2% +0.4 0.78 ? 3% perf-profile.children.cycles-pp.sched_ttwu_pending
0.41 +0.5 0.88 ? 2% perf-profile.children.cycles-pp.flush_smp_call_function_queue
14.75 ? 2% +0.9 15.61 ? 2% perf-profile.children.cycles-pp.file_write_and_wait_range
5.52 ? 2% -1.2 4.32 ? 3% perf-profile.self.cycles-pp.osq_lock
0.11 ? 6% +0.0 0.13 ? 6% perf-profile.self.cycles-pp.enqueue_entity
0.17 ? 4% +0.0 0.19 ? 4% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.05 ? 7% +0.0 0.08 ? 6% perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.06 ? 9% +0.0 0.08 ? 4% perf-profile.self.cycles-pp.llist_reverse_order
0.01 ?223% +0.0 0.06 ? 9% perf-profile.self.cycles-pp.___perf_sw_event
0.08 ? 9% +0.1 0.13 ? 4% perf-profile.self.cycles-pp.llist_add_batch
0.66 ? 2% +0.1 0.72 ? 3% perf-profile.self.cycles-pp._raw_spin_lock
0.00 +0.1 0.06 ? 6% perf-profile.self.cycles-pp.sched_ttwu_pending
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.__flush_smp_call_function_queue



***************************************************************************************************
lkp-csl-2sp9: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
gcc-11/performance/pipe/8/x86_64-rhel-8.3/process/50%/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp9/hackbench/0x500320a

commit:
28156108fe ("sched: Fix the check of nr_running at queue wakelist")
f3dd3f6745 ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle")

28156108fecb1f80 f3dd3f674555bd9455c5ae7fafc
---------------- ---------------------------
%stddev %change %stddev
\ | \
317933 +7.2% 340823 ? 2% hackbench.throughput
281.33 -5.6% 265.66 hackbench.time.elapsed_time
281.33 -5.6% 265.66 hackbench.time.elapsed_time.max
9.332e+08 ? 2% -10.1% 8.387e+08 ? 2% hackbench.time.involuntary_context_switches
22332 -6.2% 20950 hackbench.time.system_time
1668 -3.3% 1613 hackbench.time.user_time
2.484e+09 -7.0% 2.311e+09 hackbench.time.voluntary_context_switches
0.99 ? 2% +0.3 1.33 mpstat.cpu.all.irq%
4.607e+08 ? 3% -14.8% 3.927e+08 cpuidle..time
59750049 -14.4% 51160896 cpuidle..usage
107526 ? 46% -44.8% 59395 ? 13% numa-meminfo.node0.Mapped
886278 ? 86% -81.7% 162163 ? 69% numa-meminfo.node1.Mapped
26816 ? 46% -44.4% 14905 ? 14% numa-vmstat.node0.nr_mapped
220693 ? 86% -81.7% 40319 ? 67% numa-vmstat.node1.nr_mapped
1851 ? 4% +30.6% 2418 ? 3% sched_debug.cpu.avg_idle.min
86.08 ? 9% +29.5% 111.46 ? 11% sched_debug.cpu.clock_task.stddev
20680126 ? 3% -49.3% 10478464 ? 2% turbostat.C1
0.43 ? 22% -0.2 0.22 ? 3% turbostat.C1%
0.38 ? 2% -0.0 0.35 ? 3% turbostat.POLL%
4953449 ? 39% -34.7% 3235753 ? 7% vmstat.memory.cache
12228000 -2.2% 11965069 vmstat.system.cs
1391657 +4.2% 1450409 vmstat.system.in
359385 ? 9% -10.0% 323464 ? 2% meminfo.AnonPages
4834275 ? 40% -35.4% 3122465 ? 8% meminfo.Cached
3260439 ? 61% -53.9% 1501744 ? 17% meminfo.Committed_AS
2453612 ? 81% -70.9% 713687 ? 36% meminfo.Inactive
2453432 ? 81% -70.9% 713334 ? 36% meminfo.Inactive(anon)
993573 ? 79% -77.7% 221100 ? 53% meminfo.Mapped
6637365 ? 31% -27.4% 4821818 ? 5% meminfo.Memused
2183740 ? 90% -78.4% 471571 ? 53% meminfo.Shmem
7821764 ? 17% -21.1% 6170585 ? 6% meminfo.max_used_kB
89894 ? 9% -10.0% 80918 ? 2% proc-vmstat.nr_anon_pages
1209129 ? 40% -35.5% 779734 ? 8% proc-vmstat.nr_file_pages
613996 ? 81% -71.0% 178154 ? 36% proc-vmstat.nr_inactive_anon
248662 ? 79% -77.7% 55432 ? 52% proc-vmstat.nr_mapped
546495 ? 90% -78.6% 117009 ? 53% proc-vmstat.nr_shmem
613996 ? 81% -71.0% 178154 ? 36% proc-vmstat.nr_zone_inactive_anon
303380 ? 16% -39.0% 185031 ? 21% proc-vmstat.numa_hint_faults
200464 ? 20% -42.8% 114585 ? 22% proc-vmstat.numa_hint_faults_local
89782752 +3.6% 93042546 proc-vmstat.numa_hit
89705243 +3.6% 92968283 proc-vmstat.numa_local
672643 ? 37% -45.7% 365098 ? 9% proc-vmstat.numa_pte_updates
89776545 +3.6% 93036044 proc-vmstat.pgalloc_normal
1626079 ? 4% -10.8% 1450258 ? 3% proc-vmstat.pgfault
89378790 +4.1% 93037516 proc-vmstat.pgfree
15.75 -2.9% 15.30 perf-stat.i.MPKI
2.723e+10 +2.2% 2.782e+10 perf-stat.i.branch-instructions
1.15 -0.0 1.13 perf-stat.i.branch-miss-rate%
2.15 ? 5% +0.2 2.34 ? 4% perf-stat.i.cache-miss-rate%
42514112 ? 4% +8.4% 46069933 ? 5% perf-stat.i.cache-misses
12338031 -2.2% 12068469 perf-stat.i.context-switches
1.80 -1.9% 1.76 perf-stat.i.cpi
3.993e+10 +2.1% 4.076e+10 perf-stat.i.dTLB-loads
2.294e+10 +1.8% 2.336e+10 perf-stat.i.dTLB-stores
66.43 +1.1 67.51 perf-stat.i.iTLB-load-miss-rate%
76738049 -3.1% 74356040 perf-stat.i.iTLB-loads
1.353e+11 +2.1% 1.382e+11 perf-stat.i.instructions
0.56 +2.1% 0.57 perf-stat.i.ipc
1047 +2.0% 1068 perf-stat.i.metric.M/sec
5305 ? 3% -5.9% 4990 ? 3% perf-stat.i.minor-faults
8761133 ? 3% +9.8% 9619722 ? 4% perf-stat.i.node-load-misses
5459816 ? 2% +7.2% 5852880 ? 4% perf-stat.i.node-store-misses
5307 ? 3% -5.9% 4993 ? 3% perf-stat.i.page-faults
15.51 -3.0% 15.05 perf-stat.overall.MPKI
1.13 -0.0 1.12 perf-stat.overall.branch-miss-rate%
2.03 ? 4% +0.2 2.22 ? 4% perf-stat.overall.cache-miss-rate%
1.78 -2.0% 1.75 perf-stat.overall.cpi
0.12 ? 6% -0.0 0.11 ? 2% perf-stat.overall.dTLB-load-miss-rate%
66.43 +1.1 67.51 perf-stat.overall.iTLB-load-miss-rate%
0.56 +2.1% 0.57 perf-stat.overall.ipc
2.714e+10 +2.1% 2.771e+10 perf-stat.ps.branch-instructions
42404117 ? 4% +8.3% 45914737 ? 5% perf-stat.ps.cache-misses
12297697 -2.2% 12031391 perf-stat.ps.context-switches
3.978e+10 +2.0% 4.06e+10 perf-stat.ps.dTLB-loads
2.286e+10 +1.8% 2.327e+10 perf-stat.ps.dTLB-stores
76489212 -3.1% 74128795 perf-stat.ps.iTLB-loads
1.349e+11 +2.0% 1.376e+11 perf-stat.ps.instructions
5314 ? 3% -6.6% 4961 ? 3% perf-stat.ps.minor-faults
8734611 ? 3% +9.7% 9580673 ? 4% perf-stat.ps.node-load-misses
5441379 ? 2% +7.1% 5827671 ? 4% perf-stat.ps.node-store-misses
5317 ? 3% -6.6% 4964 ? 3% perf-stat.ps.page-faults
3.807e+13 -3.7% 3.667e+13 perf-stat.total.instructions
13.75 +0.3 14.09 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.new_sync_read
15.17 ? 3% +0.5 15.70 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
15.28 ? 3% +0.5 15.82 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
5.21 ? 4% +0.7 5.91 ? 5% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.exit_to_user_mode_loop
5.42 ? 4% +0.7 6.14 ? 5% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
6.62 ? 9% +0.8 7.45 ? 6% perf-profile.calltrace.cycles-pp.__get_user_nocheck_8.perf_callchain_user.get_perf_callchain.perf_callchain.perf_prepare_sample
3.28 ? 16% +1.0 4.24 ? 5% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_migrate_task
3.38 ? 16% +1.0 4.34 ? 5% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_migrate_task.set_task_cpu.try_to_wake_up
3.36 ? 16% +1.0 4.33 ? 5% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_migrate_task.set_task_cpu
3.44 ? 16% +1.0 4.44 ? 5% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_migrate_task.set_task_cpu.try_to_wake_up.autoremove_wake_function
3.49 ? 16% +1.0 4.49 ? 5% perf-profile.calltrace.cycles-pp.perf_trace_sched_migrate_task.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common
4.06 ? 15% +1.1 5.14 ? 2% perf-profile.calltrace.cycles-pp.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
9.21 ? 4% +1.1 10.30 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
9.10 ? 4% +1.1 10.19 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
57.10 +1.1 58.22 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
56.86 +1.1 57.98 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
8.61 ? 4% +1.1 9.74 ? 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
8.67 ? 4% +1.1 9.81 ? 3% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
58.57 +1.2 59.74 perf-profile.calltrace.cycles-pp.__libc_write
8.76 ? 4% +1.2 9.99 ? 3% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.65 ? 69% -0.6 0.05 ? 77% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
0.65 ? 69% -0.6 0.05 ? 77% perf-profile.children.cycles-pp.cpu_startup_entry
0.65 ? 69% -0.6 0.05 ? 77% perf-profile.children.cycles-pp.do_idle
0.64 ? 69% -0.6 0.05 ? 77% perf-profile.children.cycles-pp.start_secondary
1.24 ? 8% -0.2 1.04 ? 4% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.28 ? 50% -0.2 0.09 ? 27% perf-profile.children.cycles-pp.__mutex_lock
0.23 ? 41% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.record__mmap_read_evlist
0.23 ? 41% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.perf_mmap__push
0.23 ? 41% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.record__pushfn
0.23 ? 40% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.generic_file_write_iter
0.23 ? 40% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.__generic_file_write_iter
0.23 ? 40% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.generic_perform_write
0.53 ? 11% -0.1 0.42 ? 5% perf-profile.children.cycles-pp.copyin
0.24 ? 37% -0.1 0.13 ? 28% perf-profile.children.cycles-pp.__mutex_unlock_slowpath
0.16 ? 55% -0.1 0.06 ? 48% perf-profile.children.cycles-pp.wake_up_q
15.64 +0.4 16.08 perf-profile.children.cycles-pp.enqueue_task_fair
15.75 +0.4 16.20 perf-profile.children.cycles-pp.ttwu_do_activate
3.51 ? 16% +1.0 4.50 ? 5% perf-profile.children.cycles-pp.perf_trace_sched_migrate_task
59.07 +1.0 60.08 perf-profile.children.cycles-pp.__libc_write
4.08 ? 15% +1.1 5.15 ? 2% perf-profile.children.cycles-pp.set_task_cpu
39.22 +1.1 40.33 perf-profile.children.cycles-pp.schedule
9.11 ? 5% +1.2 10.27 perf-profile.children.cycles-pp.exit_to_user_mode_loop
10.37 ? 5% +1.2 11.60 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
10.18 ? 5% +1.2 11.41 perf-profile.children.cycles-pp.exit_to_user_mode_prepare
0.16 ? 3% -0.0 0.14 ? 4% perf-profile.self.cycles-pp.finish_task_switch
0.06 ? 6% +0.0 0.08 ? 10% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.21 ? 8% +0.0 0.25 ? 6% perf-profile.self.cycles-pp.set_next_buddy





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


--
0-DAY CI Kernel Test Service
https://01.org/lkp



Attachments:
(No filename) (154.63 kB)
config-5.19.0-rc2-00010-gf3dd3f674555 (166.39 kB)
job-script (8.24 kB)
job.yaml (5.70 kB)
reproduce (263.00 B)
Download all attachments