Greeting,
FYI, we noticed a -7.9% regression of hackbench.throughput due to commit:
commit: 829c1651e9c4a6f78398d3e67651cef9bb6b42cc ("sched/fair: sanitize vruntime of entity being placed")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: hackbench
on test machine: 104 threads 2 sockets (Skylake) with 192G memory
with following parameters:
nr_threads: 100%
iterations: 4
mode: process
ipc: socket
cpufreq_governor: performance
In addition to that, the commit also has significant impact on the following tests:
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.rawudp.ops_per_sec 74.0% improvement |
| test machine | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz (Cascade Lake) with 128G memory |
| test parameters | class=network |
| | cpufreq_governor=performance |
| | nr_threads=100% |
| | test=rawudp |
| | testtime=60s |
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput 125.5% improvement |
| test machine | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz (Cascade Lake) with 128G memory |
| test parameters | cpufreq_governor=performance |
| | ipc=pipe |
| | iterations=8 |
| | mode=process |
| | nr_threads=50% |
+------------------+------------------------------------------------------------------------------------------------+
If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-lkp/[email protected]
Details are as below:
=========================================================================================
compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase:
gcc-11/performance/socket/4/x86_64-rhel-8.3/process/100%/debian-11.1-x86_64-20220510.cgz/lkp-skl-fpga01/hackbench
commit:
a2e90611b9 ("sched/fair: Remove capacity inversion detection")
829c1651e9 ("sched/fair: sanitize vruntime of entity being placed")
a2e90611b9f425ad 829c1651e9c4a6f78398d3e6765
---------------- ---------------------------
%stddev %change %stddev
\ | \
332.56 +19.0% 395.75 uptime.boot
591284 ? 13% +45.9% 862691 ? 13% numa-numastat.node0.local_node
633561 ? 15% +43.8% 911278 ? 13% numa-numastat.node0.numa_hit
1.25 ? 6% -0.2 1.06 ? 10% mpstat.cpu.all.idle%
0.03 -0.0 0.03 mpstat.cpu.all.soft%
22.68 -2.8 19.86 mpstat.cpu.all.usr%
3746600 ? 13% -15.3% 3174871 vmstat.memory.cache
1751 +30.0% 2277 vmstat.procs.r
1628685 +277.9% 6154443 ? 3% vmstat.system.cs
320461 +58.9% 509072 ? 4% vmstat.system.in
3618671 ? 13% -16.1% 3035929 meminfo.Cached
2906667 ? 18% -23.1% 2234788 ? 2% meminfo.Committed_AS
1295992 ? 39% -47.9% 675180 ? 7% meminfo.Inactive
1295802 ? 39% -47.9% 674982 ? 7% meminfo.Inactive(anon)
535486 ? 32% -55.2% 240147 ? 23% meminfo.Mapped
6007002 ? 9% -13.2% 5213349 meminfo.Memused
888389 ? 56% -65.6% 305644 ? 13% meminfo.Shmem
468144 ? 9% -20.5% 372290 ? 11% numa-meminfo.node1.AnonPages.max
1789452 ? 60% -80.3% 353145 ? 19% numa-meminfo.node1.FilePages
1090426 ? 46% -59.8% 438087 ? 19% numa-meminfo.node1.Inactive
1090417 ? 46% -59.8% 437944 ? 19% numa-meminfo.node1.Inactive(anon)
422719 ? 33% -64.5% 150054 ? 29% numa-meminfo.node1.Mapped
3077746 ? 32% -55.8% 1361807 ? 10% numa-meminfo.node1.MemUsed
877875 ? 56% -66.4% 295052 ? 13% numa-meminfo.node1.Shmem
633532 ? 15% +43.9% 911355 ? 13% numa-vmstat.node0.numa_hit
591255 ? 13% +45.9% 862767 ? 13% numa-vmstat.node0.numa_local
446087 ? 61% -79.9% 89778 ? 18% numa-vmstat.node1.nr_file_pages
271096 ? 46% -59.1% 110864 ? 20% numa-vmstat.node1.nr_inactive_anon
105177 ? 33% -63.0% 38966 ? 29% numa-vmstat.node1.nr_mapped
218193 ? 57% -65.5% 75255 ? 13% numa-vmstat.node1.nr_shmem
271096 ? 46% -59.1% 110864 ? 20% numa-vmstat.node1.nr_zone_inactive_anon
1087461 ? 8% +19.9% 1304147 ? 4% turbostat.C1
238405 ? 3% +48.4% 353679 ? 6% turbostat.C1E
1.17 ? 4% -0.2 0.99 ? 5% turbostat.C6%
0.26 ? 7% -22.8% 0.20 ? 7% turbostat.CPU%c6
91428380 +93.8% 1.772e+08 ? 5% turbostat.IRQ
446543 ? 6% -40.4% 266049 ? 8% turbostat.POLL
34.72 -6.8% 32.36 turbostat.RAMWatt
176568 -7.9% 162531 hackbench.throughput
173811 -18.4% 141887 hackbench.throughput_avg
176568 -7.9% 162531 hackbench.throughput_best
167465 -36.9% 105686 hackbench.throughput_worst
282.47 +22.4% 345.62 hackbench.time.elapsed_time
282.47 +22.4% 345.62 hackbench.time.elapsed_time.max
1.08e+08 ? 2% +648.7% 8.084e+08 ? 5% hackbench.time.involuntary_context_switches
22102 +26.9% 28058 hackbench.time.system_time
6651 +7.9% 7177 hackbench.time.user_time
3.542e+08 +276.0% 1.332e+09 ? 4% hackbench.time.voluntary_context_switches
121493 ? 4% -8.7% 110874 ? 2% proc-vmstat.nr_anon_pages
905193 ? 14% -16.0% 760560 proc-vmstat.nr_file_pages
324344 ? 39% -47.5% 170335 ? 7% proc-vmstat.nr_inactive_anon
82632 -2.0% 80963 proc-vmstat.nr_kernel_stack
133808 ? 32% -54.1% 61447 ? 23% proc-vmstat.nr_mapped
222621 ? 57% -65.0% 77988 ? 13% proc-vmstat.nr_shmem
31327 -1.5% 30854 proc-vmstat.nr_slab_reclaimable
118181 -7.8% 108941 proc-vmstat.nr_slab_unreclaimable
324344 ? 39% -47.5% 170335 ? 7% proc-vmstat.nr_zone_inactive_anon
146649 ? 4% -23.3% 112510 ? 17% proc-vmstat.numa_hint_faults
95646 ? 6% -42.3% 55232 ? 23% proc-vmstat.numa_hint_faults_local
2501175 ? 6% +34.1% 3353611 ? 4% proc-vmstat.pgalloc_normal
2282129 ? 10% +44.0% 3287348 ? 4% proc-vmstat.pgfree
111663 ? 2% +9.6% 122351 ? 6% proc-vmstat.pgreuse
2184832 +21.6% 2657792 proc-vmstat.unevictable_pgs_scanned
14.29 ? 5% +21.1% 17.31 ? 5% sched_debug.cfs_rq:/.h_nr_running.avg
8.90 ? 3% -16.9% 7.40 ? 5% sched_debug.cfs_rq:/.h_nr_running.stddev
59684 ?126% -61.8% 22786 ? 7% sched_debug.cfs_rq:/.load.max
7944 ? 90% -49.3% 4026 ? 9% sched_debug.cfs_rq:/.load.stddev
11840155 +39.0% 16456271 sched_debug.cfs_rq:/.min_vruntime.avg
13870482 ? 3% +115.5% 29897645 ? 16% sched_debug.cfs_rq:/.min_vruntime.max
11419937 +32.2% 15100337 sched_debug.cfs_rq:/.min_vruntime.min
302622 ? 14% +434.1% 1616191 ? 21% sched_debug.cfs_rq:/.min_vruntime.stddev
14521 ? 3% +23.1% 17873 ? 4% sched_debug.cfs_rq:/.runnable_avg.avg
24482 ? 2% +16.6% 28534 ? 8% sched_debug.cfs_rq:/.runnable_avg.max
3781 ? 4% +15.6% 4372 ? 8% sched_debug.cfs_rq:/.runnable_avg.stddev
-1538686 +365.4% -7160429 sched_debug.cfs_rq:/.spread0.min
307224 ? 14% +426.4% 1617267 ? 22% sched_debug.cfs_rq:/.spread0.stddev
1557 ? 4% -11.5% 1379 ? 3% sched_debug.cfs_rq:/.util_avg.max
308.20 ? 16% +63.4% 503.64 ? 11% sched_debug.cfs_rq:/.util_avg.min
227.94 ? 3% -29.5% 160.63 ? 4% sched_debug.cfs_rq:/.util_avg.stddev
506.05 ? 5% +31.3% 664.53 ? 4% sched_debug.cfs_rq:/.util_est_enqueued.avg
10.67 ? 46% +452.6% 58.94 ? 48% sched_debug.cfs_rq:/.util_est_enqueued.min
340.86 ? 3% -9.9% 307.11 ? 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev
173106 +20.4% 208437 sched_debug.cpu.clock.avg
173428 +20.7% 209392 sched_debug.cpu.clock.max
172807 +20.0% 207372 sched_debug.cpu.clock.min
188.14 ? 45% +211.4% 585.91 ? 18% sched_debug.cpu.clock.stddev
171366 +20.4% 206279 sched_debug.cpu.clock_task.avg
171999 +20.7% 207633 sched_debug.cpu.clock_task.max
155591 +21.8% 189434 sched_debug.cpu.clock_task.min
1596 ? 3% +17.2% 1871 ? 4% sched_debug.cpu.clock_task.stddev
6694 ? 16% -32.3% 4534 ? 24% sched_debug.cpu.max_idle_balance_cost.stddev
0.00 ? 43% +205.0% 0.00 ? 18% sched_debug.cpu.next_balance.stddev
14.30 ? 5% +20.6% 17.26 ? 5% sched_debug.cpu.nr_running.avg
8.91 ? 3% -17.0% 7.40 ? 5% sched_debug.cpu.nr_running.stddev
1834484 ? 2% +437.1% 9853555 ? 5% sched_debug.cpu.nr_switches.avg
2261734 ? 5% +516.3% 13939145 ? 6% sched_debug.cpu.nr_switches.max
1565403 +370.7% 7368423 ? 6% sched_debug.cpu.nr_switches.min
133984 ? 18% +870.5% 1300270 ? 4% sched_debug.cpu.nr_switches.stddev
172800 +20.0% 207351 sched_debug.cpu_clk
169631 +20.4% 204181 sched_debug.ktime
173545 +19.9% 208099 sched_debug.sched_clk
7.31 +21.4% 8.87 perf-stat.i.MPKI
1.916e+10 +15.5% 2.213e+10 perf-stat.i.branch-instructions
2.17 -0.1 2.07 perf-stat.i.branch-miss-rate%
4.135e+08 +7.2% 4.435e+08 perf-stat.i.branch-misses
23.00 -9.2 13.78 ? 3% perf-stat.i.cache-miss-rate%
1.699e+08 -36.4% 1.081e+08 ? 2% perf-stat.i.cache-misses
7.463e+08 +34.9% 1.007e+09 perf-stat.i.cache-references
1631170 +385.1% 7912003 ? 4% perf-stat.i.context-switches
2.75 -7.7% 2.54 perf-stat.i.cpi
104173 +1.1% 105339 perf-stat.i.cpu-clock
128043 ? 2% +289.0% 498025 ? 6% perf-stat.i.cpu-migrations
1686 +129.8% 3875 ? 9% perf-stat.i.cycles-between-cache-misses
0.50 ? 2% +0.6 1.13 perf-stat.i.dTLB-load-miss-rate%
1.456e+08 ? 2% +156.8% 3.738e+08 perf-stat.i.dTLB-load-misses
2.903e+10 +10.0% 3.194e+10 perf-stat.i.dTLB-loads
0.13 ? 7% +0.1 0.21 ? 2% perf-stat.i.dTLB-store-miss-rate%
22070920 ? 7% +84.1% 40627527 ? 2% perf-stat.i.dTLB-store-misses
1.752e+10 +9.5% 1.92e+10 perf-stat.i.dTLB-stores
29.21 +3.0 32.23 perf-stat.i.iTLB-load-miss-rate%
57279315 +57.3% 90075362 ? 3% perf-stat.i.iTLB-load-misses
1.383e+08 +31.1% 1.813e+08 perf-stat.i.iTLB-loads
1.029e+11 +11.6% 1.148e+11 perf-stat.i.instructions
1821 -19.0% 1475 ? 3% perf-stat.i.instructions-per-iTLB-miss
0.36 +11.6% 0.41 perf-stat.i.ipc
1096 -4.7% 1044 perf-stat.i.metric.K/sec
639.86 +11.1% 710.59 perf-stat.i.metric.M/sec
4992 -3.8% 4802 ? 2% perf-stat.i.minor-faults
22.97 +22.1 45.10 ? 5% perf-stat.i.node-load-miss-rate%
11795656 +9.0% 12859890 ? 9% perf-stat.i.node-load-misses
40616847 -46.0% 21938686 ? 4% perf-stat.i.node-loads
8.56 ? 3% +29.5 38.06 ? 9% perf-stat.i.node-store-miss-rate%
2902019 ? 2% +60.8% 4666181 ? 13% perf-stat.i.node-store-misses
33981660 -50.6% 16788672 ? 5% perf-stat.i.node-stores
5022 -3.7% 4834 ? 2% perf-stat.i.page-faults
104173 +1.1% 105340 perf-stat.i.task-clock
7.21 +11.6% 8.05 perf-stat.overall.MPKI
2.15 -0.1 2.05 perf-stat.overall.branch-miss-rate%
22.82 -9.3 13.53 ? 3% perf-stat.overall.cache-miss-rate%
2.74 -6.3% 2.56 perf-stat.overall.cpi
1661 +41.7% 2354 perf-stat.overall.cycles-between-cache-misses
0.50 ? 2% +0.5 0.99 perf-stat.overall.dTLB-load-miss-rate%
0.13 ? 7% +0.1 0.19 ? 2% perf-stat.overall.dTLB-store-miss-rate%
29.28 +2.2 31.47 perf-stat.overall.iTLB-load-miss-rate%
1804 -23.8% 1374 ? 2% perf-stat.overall.instructions-per-iTLB-miss
0.37 +6.8% 0.39 perf-stat.overall.ipc
22.25 +7.3 29.59 ? 6% perf-stat.overall.node-load-miss-rate%
7.63 ? 2% +7.9 15.50 ? 12% perf-stat.overall.node-store-miss-rate%
1.914e+10 +9.2% 2.089e+10 perf-stat.ps.branch-instructions
4.122e+08 +4.0% 4.286e+08 perf-stat.ps.branch-misses
1.692e+08 -29.7% 1.19e+08 perf-stat.ps.cache-misses
7.412e+08 +18.7% 8.8e+08 perf-stat.ps.cache-references
1614353 +282.0% 6167224 ? 4% perf-stat.ps.context-switches
124528 ? 2% +181.4% 350438 ? 6% perf-stat.ps.cpu-migrations
1.448e+08 ? 2% +111.2% 3.06e+08 perf-stat.ps.dTLB-load-misses
2.898e+10 +5.6% 3.059e+10 perf-stat.ps.dTLB-loads
21927790 ? 7% +60.0% 35093075 ? 2% perf-stat.ps.dTLB-store-misses
1.748e+10 +5.3% 1.84e+10 perf-stat.ps.dTLB-stores
56957377 +39.7% 79589638 ? 2% perf-stat.ps.iTLB-load-misses
1.376e+08 +25.9% 1.732e+08 perf-stat.ps.iTLB-loads
1.027e+11 +6.4% 1.093e+11 perf-stat.ps.instructions
23.77 ? 9% -14.8% 20.26 ? 3% perf-stat.ps.major-faults
4672 -20.7% 3705 ? 2% perf-stat.ps.minor-faults
40554146 -33.5% 26953905 ? 3% perf-stat.ps.node-loads
2797533 ? 2% +36.8% 3825728 ? 11% perf-stat.ps.node-store-misses
33847416 -38.3% 20890250 ? 3% perf-stat.ps.node-stores
4696 -20.7% 3725 ? 2% perf-stat.ps.page-faults
2.906e+13 +30.2% 3.782e+13 perf-stat.total.instructions
15.22 ? 2% -8.6 6.66 ? 3% perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
13.83 ? 2% -7.8 5.98 ? 4% perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
13.60 ? 2% -7.8 5.85 ? 4% perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
8.99 ? 2% -5.4 3.56 ? 4% perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
6.45 ? 3% -4.3 2.19 ? 5% perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
5.88 ? 4% -3.4 2.48 ? 3% perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
5.76 ? 3% -3.3 2.43 ? 3% perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
5.57 ? 5% -3.3 2.30 ? 3% perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
5.14 ? 5% -3.1 2.03 ? 4% perf-profile.calltrace.cycles-pp.__kmem_cache_alloc_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
6.24 ? 2% -2.9 3.31 ? 2% perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
6.18 ? 2% -2.9 3.29 ? 2% perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
6.12 ? 2% -2.9 3.23 ? 2% perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg
51.05 -2.4 48.66 perf-profile.calltrace.cycles-pp.__libc_read
5.65 -2.3 3.37 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_read
5.51 ? 2% -2.2 3.27 ? 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_write
3.19 ? 3% -1.8 1.41 ? 3% perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
2.79 ? 3% -1.7 1.14 ? 3% perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
2.66 ? 4% -1.6 1.06 ? 3% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
2.68 ? 2% -1.3 1.36 ? 6% perf-profile.calltrace.cycles-pp.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
2.98 ? 2% -1.2 1.82 perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
2.25 ? 2% -1.1 1.15 ? 2% perf-profile.calltrace.cycles-pp.skb_release_head_state.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
2.72 ? 2% -1.1 1.63 ? 2% perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
2.79 ? 2% -1.1 1.71 ? 2% perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
2.18 ? 2% -1.1 1.11 ? 2% perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
1.98 ? 3% -1.0 0.95 ? 3% perf-profile.calltrace.cycles-pp.sock_wfree.unix_destruct_scm.skb_release_head_state.consume_skb.unix_stream_read_generic
41.54 -1.0 40.55 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
2.06 -1.0 1.08 ? 3% perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
1.18 ? 5% -0.9 0.28 ?100% perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
2.06 -0.8 1.23 perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_write
2.16 ? 2% -0.7 1.46 perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_read
2.19 ? 2% -0.6 1.54 ? 4% perf-profile.calltrace.cycles-pp.__kmem_cache_free.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
1.45 ? 2% -0.6 0.83 perf-profile.calltrace.cycles-pp._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
4.82 -0.5 4.27 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
1.32 ? 2% -0.5 0.86 ? 2% perf-profile.calltrace.cycles-pp.__check_object_size.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
0.94 ? 3% -0.4 0.54 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg
1.14 ? 2% -0.4 0.76 ? 4% perf-profile.calltrace.cycles-pp.memcg_slab_post_alloc_hook.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
1.00 ? 3% -0.4 0.65 ? 6% perf-profile.calltrace.cycles-pp.memcg_slab_post_alloc_hook.__kmem_cache_alloc_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
1.30 ? 3% -0.3 0.99 ? 3% perf-profile.calltrace.cycles-pp.sock_recvmsg.sock_read_iter.vfs_read.ksys_read.do_syscall_64
0.98 -0.3 0.69 ? 2% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.90 ? 3% -0.2 0.68 perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
1.15 ? 3% -0.2 0.93 ? 3% perf-profile.calltrace.cycles-pp.security_socket_recvmsg.sock_recvmsg.sock_read_iter.vfs_read.ksys_read
0.85 -0.2 0.64 ? 2% perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
0.56 ? 3% -0.2 0.36 ? 70% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
0.84 ? 3% -0.2 0.67 ? 4% perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter.vfs_read
1.06 -0.2 0.90 ? 3% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.22 ? 2% -0.2 1.07 perf-profile.calltrace.cycles-pp.security_socket_sendmsg.sock_sendmsg.sock_write_iter.vfs_write.ksys_write
0.88 -0.1 0.80 ? 5% perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64
0.93 ? 2% -0.1 0.88 ? 2% perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
0.63 +0.3 0.90 perf-profile.calltrace.cycles-pp.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
0.00 +0.5 0.54 perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
30.43 +0.6 31.02 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
0.00 +0.6 0.60 ? 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg
0.00 +0.6 0.61 ? 3% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
0.00 +0.6 0.61 ? 3% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop
0.00 +0.7 0.66 ? 4% perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
0.00 +0.7 0.66 ? 4% perf-profile.calltrace.cycles-pp.__switch_to_asm.__libc_write
0.00 +0.8 0.76 ? 3% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
0.00 +0.8 0.76 ? 5% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop
0.00 +0.8 0.77 ? 2% perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.schedule_timeout
32.79 +0.8 33.56 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
0.00 +0.8 0.80 ? 3% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
0.00 +0.8 0.81 ? 3% perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
0.00 +0.8 0.81 ? 3% perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
0.00 +0.8 0.83 perf-profile.calltrace.cycles-pp.__switch_to_asm.__libc_read
0.00 +0.8 0.84 ? 3% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
0.00 +0.9 0.88 ? 2% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
29.52 +0.9 30.42 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
31.98 +0.9 32.88 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
29.91 +0.9 30.82 perf-profile.calltrace.cycles-pp.sock_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.9 0.94 ? 22% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function
0.00 +1.0 0.97 ? 3% perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.00 +1.0 0.98 perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.schedule_timeout
0.00 +1.0 1.00 perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
0.00 +1.1 1.07 ? 3% perf-profile.calltrace.cycles-pp.prepare_to_wait.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
27.84 +1.4 29.27 perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_read_iter.vfs_read.ksys_read.do_syscall_64
0.00 +1.4 1.43 ? 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common
27.39 +1.5 28.87 perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read.ksys_read
0.00 +1.5 1.50 ? 3% perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.schedule_timeout.unix_stream_data_wait
0.00 +1.6 1.58 ? 16% perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
0.00 +1.6 1.62 ? 3% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
27.43 +1.6 29.06 perf-profile.calltrace.cycles-pp.sock_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +1.6 1.63 ? 6% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
0.00 +1.7 1.68 ? 3% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
0.00 +1.8 1.77 ? 2% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.00 +1.9 1.88 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.unix_stream_data_wait
0.00 +1.9 1.93 ? 2% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.schedule_timeout.unix_stream_data_wait
0.00 +2.0 1.98 ? 11% perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.00 +2.0 1.99 ? 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
26.48 +2.0 28.48 perf-profile.calltrace.cycles-pp.sock_sendmsg.sock_write_iter.vfs_write.ksys_write.do_syscall_64
0.00 +2.1 2.12 ? 4% perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
25.07 +2.2 27.30 perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write.ksys_write
0.00 +2.3 2.30 ? 10% perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
47.96 +2.5 50.45 perf-profile.calltrace.cycles-pp.__libc_write
0.00 +2.9 2.92 ? 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
0.00 +3.4 3.44 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
39.02 +4.6 43.61 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
0.70 ? 13% +5.0 5.67 ? 2% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.unix_stream_data_wait
4.80 +5.1 9.86 ? 3% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
0.70 ? 14% +5.2 5.94 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
35.89 +5.6 41.48 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
0.00 +6.0 5.98 ? 5% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
0.00 +6.1 6.14 ? 5% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
0.83 ? 14% +6.3 7.10 ? 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.00 +6.3 6.32 ? 5% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.18 ?141% +7.0 7.14 ? 5% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
1.98 ? 13% +12.1 14.07 ? 2% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.unix_stream_data_wait.unix_stream_read_generic
1.78 ? 16% +12.1 13.90 ? 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
2.02 ? 13% +12.3 14.34 ? 2% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg
1.82 ? 16% +12.4 14.22 ? 2% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg
2.06 ? 13% +12.5 14.58 ? 2% perf-profile.calltrace.cycles-pp.schedule_timeout.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
1.87 ? 15% +12.7 14.60 ? 2% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg
2.61 ? 10% +13.2 15.79 ? 2% perf-profile.calltrace.cycles-pp.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.vfs_write
2.08 ? 14% +13.2 15.29 ? 2% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
2.40 ? 13% +14.6 17.02 ? 2% perf-profile.calltrace.cycles-pp.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.vfs_read
15.24 ? 2% -8.6 6.67 ? 3% perf-profile.children.cycles-pp.sock_alloc_send_pskb
13.84 ? 2% -7.8 5.99 ? 4% perf-profile.children.cycles-pp.alloc_skb_with_frags
13.68 ? 2% -7.8 5.90 ? 4% perf-profile.children.cycles-pp.__alloc_skb
9.01 ? 2% -5.4 3.56 ? 4% perf-profile.children.cycles-pp.consume_skb
6.76 ? 13% -4.6 2.18 ? 2% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
11.24 -4.5 6.70 perf-profile.children.cycles-pp.syscall_return_via_sysret
6.46 ? 3% -4.3 2.20 ? 5% perf-profile.children.cycles-pp.skb_release_data
4.16 ? 12% -4.0 0.14 ? 59% perf-profile.children.cycles-pp.___slab_alloc
5.74 ? 16% -3.9 1.89 ? 3% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
5.92 ? 4% -3.4 2.50 ? 3% perf-profile.children.cycles-pp.kmalloc_reserve
5.91 ? 3% -3.4 2.51 ? 3% perf-profile.children.cycles-pp.kmem_cache_alloc_node
5.70 ? 5% -3.3 2.37 ? 4% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
5.26 ? 5% -3.2 2.11 ? 3% perf-profile.children.cycles-pp.__kmem_cache_alloc_node
6.24 ? 2% -2.9 3.32 ? 2% perf-profile.children.cycles-pp.unix_stream_read_actor
6.19 ? 2% -2.9 3.29 ? 2% perf-profile.children.cycles-pp.skb_copy_datagram_iter
6.12 ? 2% -2.9 3.24 ? 2% perf-profile.children.cycles-pp.__skb_datagram_iter
3.08 ? 2% -2.4 0.64 ? 12% perf-profile.children.cycles-pp.__slab_free
51.44 -2.3 49.10 perf-profile.children.cycles-pp.__libc_read
5.34 ? 2% -2.1 3.20 perf-profile.children.cycles-pp.__entry_text_start
3.61 ? 3% -2.0 1.60 ? 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
3.21 ? 3% -1.8 1.42 ? 3% perf-profile.children.cycles-pp._copy_to_iter
2.80 ? 3% -1.7 1.15 ? 3% perf-profile.children.cycles-pp.copyout
4.12 ? 2% -1.5 2.58 ? 2% perf-profile.children.cycles-pp.__check_object_size
2.70 ? 2% -1.3 1.37 ? 6% perf-profile.children.cycles-pp.kmem_cache_free
2.80 -1.2 1.60 ? 3% perf-profile.children.cycles-pp.check_heap_object
3.00 ? 2% -1.2 1.83 perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
2.26 ? 2% -1.1 1.16 ? 2% perf-profile.children.cycles-pp.skb_release_head_state
2.80 ? 2% -1.1 1.72 ? 2% perf-profile.children.cycles-pp.simple_copy_to_iter
2.20 ? 2% -1.1 1.13 ? 2% perf-profile.children.cycles-pp.unix_destruct_scm
1.99 ? 3% -1.0 0.96 ? 2% perf-profile.children.cycles-pp.sock_wfree
2.17 ? 2% -0.7 1.43 ? 5% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
1.45 -0.7 0.72 ? 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
2.20 ? 2% -0.6 1.55 ? 4% perf-profile.children.cycles-pp.__kmem_cache_free
1.48 ? 3% -0.6 0.84 perf-profile.children.cycles-pp._copy_from_iter
1.48 ? 3% -0.5 0.98 perf-profile.children.cycles-pp.get_obj_cgroup_from_current
1.41 ? 3% -0.5 0.91 ? 8% perf-profile.children.cycles-pp.mod_objcg_state
0.72 -0.5 0.25 ? 15% perf-profile.children.cycles-pp.__build_skb_around
2.06 -0.5 1.60 ? 2% perf-profile.children.cycles-pp.security_file_permission
0.96 ? 3% -0.4 0.54 perf-profile.children.cycles-pp.copyin
1.08 ? 2% -0.4 0.72 ? 2% perf-profile.children.cycles-pp.__might_resched
1.32 ? 3% -0.3 1.00 ? 2% perf-profile.children.cycles-pp.sock_recvmsg
1.75 -0.3 1.44 ? 3% perf-profile.children.cycles-pp.apparmor_file_permission
0.54 ? 6% -0.3 0.25 ? 6% perf-profile.children.cycles-pp.skb_unlink
0.68 -0.3 0.41 ? 3% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.68 -0.3 0.40 ? 3% perf-profile.children.cycles-pp.obj_cgroup_charge
0.53 ? 3% -0.3 0.27 perf-profile.children.cycles-pp.unix_write_space
1.84 ? 3% -0.3 1.58 ? 3% perf-profile.children.cycles-pp.aa_sk_perm
0.85 ? 3% -0.2 0.60 ? 4% perf-profile.children.cycles-pp.__check_heap_object
0.71 ? 2% -0.2 0.46 ? 3% perf-profile.children.cycles-pp.__get_obj_cgroup_from_memcg
0.71 ? 6% -0.2 0.48 ? 4% perf-profile.children.cycles-pp.skb_set_owner_w
1.16 ? 3% -0.2 0.93 ? 3% perf-profile.children.cycles-pp.security_socket_recvmsg
1.22 ? 2% -0.2 0.99 ? 4% perf-profile.children.cycles-pp.__fdget_pos
0.48 ? 3% -0.2 0.29 ? 3% perf-profile.children.cycles-pp.__might_fault
1.11 ? 2% -0.2 0.92 ? 5% perf-profile.children.cycles-pp.__fget_light
0.23 ? 5% -0.2 0.06 ? 15% perf-profile.children.cycles-pp.fsnotify_perm
0.99 ? 3% -0.2 0.82 ? 5% perf-profile.children.cycles-pp.__virt_addr_valid
0.62 ? 8% -0.2 0.46 ? 4% perf-profile.children.cycles-pp.skb_queue_tail
0.61 ? 2% -0.2 0.45 perf-profile.children.cycles-pp.__might_sleep
1.23 ? 2% -0.2 1.07 perf-profile.children.cycles-pp.security_socket_sendmsg
0.33 ? 2% -0.2 0.17 ? 2% perf-profile.children.cycles-pp.refill_obj_stock
0.41 ? 2% -0.2 0.25 ? 4% perf-profile.children.cycles-pp.kmalloc_slab
0.57 ? 2% -0.1 0.43 ? 3% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.42 -0.1 0.29 ? 4% perf-profile.children.cycles-pp.is_vmalloc_addr
0.52 ? 2% -0.1 0.40 ? 3% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.47 ? 2% -0.1 0.35 ? 4% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.54 ? 2% -0.1 0.42 perf-profile.children.cycles-pp.wait_for_unix_gc
0.47 ? 3% -0.1 0.35 ? 3% perf-profile.children.cycles-pp.hrtimer_interrupt
0.32 ? 5% -0.1 0.21 ? 6% perf-profile.children.cycles-pp.tick_sched_handle
0.21 ? 4% -0.1 0.10 ? 13% perf-profile.children.cycles-pp.task_tick_fair
0.41 ? 3% -0.1 0.30 ? 5% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.35 ? 6% -0.1 0.24 ? 6% perf-profile.children.cycles-pp.tick_sched_timer
0.31 ? 4% -0.1 0.21 ? 6% perf-profile.children.cycles-pp.update_process_times
0.26 ? 5% -0.1 0.16 ? 5% perf-profile.children.cycles-pp.kmalloc_size_roundup
0.24 ? 3% -0.1 0.15 ? 6% perf-profile.children.cycles-pp.scheduler_tick
0.30 ? 4% -0.1 0.21 ? 4% perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
0.20 ? 2% -0.1 0.12 ? 9% perf-profile.children.cycles-pp.should_failslab
0.50 -0.1 0.43 ? 2% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.24 ? 3% -0.1 0.18 ? 4% perf-profile.children.cycles-pp.rcu_all_qs
0.19 ? 5% -0.1 0.12 ? 3% perf-profile.children.cycles-pp.apparmor_socket_sendmsg
0.20 ? 3% -0.1 0.13 ? 5% perf-profile.children.cycles-pp.aa_file_perm
0.14 ? 6% -0.1 0.08 ? 19% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
0.19 ? 2% -0.1 0.12 ? 4% perf-profile.children.cycles-pp.apparmor_socket_recvmsg
0.36 -0.1 0.30 perf-profile.children.cycles-pp.security_socket_getpeersec_dgram
0.18 ? 11% -0.1 0.12 ? 5% perf-profile.children.cycles-pp.memcg_account_kmem
0.11 ? 18% -0.0 0.08 ? 7% perf-profile.children.cycles-pp.__mod_memcg_state
0.18 ? 2% -0.0 0.15 ? 2% perf-profile.children.cycles-pp.apparmor_socket_getpeersec_dgram
0.08 ? 4% -0.0 0.06 ? 8% perf-profile.children.cycles-pp.put_pid
0.00 +0.1 0.05 perf-profile.children.cycles-pp.default_send_IPI_single_phys
0.00 +0.1 0.07 ? 7% perf-profile.children.cycles-pp.invalidate_user_asid
0.00 +0.1 0.07 ? 12% perf-profile.children.cycles-pp.__bitmap_and
0.00 +0.1 0.08 ? 19% perf-profile.children.cycles-pp.read@plt
0.00 +0.1 0.08 ? 8% perf-profile.children.cycles-pp.sysvec_reschedule_ipi
0.00 +0.1 0.08 ? 4% perf-profile.children.cycles-pp.schedule_debug
0.00 +0.1 0.09 ? 5% perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
0.00 +0.1 0.10 ? 9% perf-profile.children.cycles-pp.place_entity
0.02 ? 99% +0.1 0.13 ? 5% perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
0.07 ? 12% +0.1 0.19 ? 4% perf-profile.children.cycles-pp.__list_add_valid
0.00 +0.1 0.12 ? 5% perf-profile.children.cycles-pp.native_irq_return_iret
0.00 +0.1 0.12 ? 8% perf-profile.children.cycles-pp.perf_trace_buf_alloc
0.00 +0.1 0.14 ? 10% perf-profile.children.cycles-pp.switch_ldt
0.00 +0.1 0.14 ? 6% perf-profile.children.cycles-pp.check_cfs_rq_runtime
0.00 +0.2 0.15 ? 16% perf-profile.children.cycles-pp.migrate_task_rq_fair
0.00 +0.2 0.16 ? 6% perf-profile.children.cycles-pp.__rdgsbase_inactive
0.00 +0.2 0.16 ? 6% perf-profile.children.cycles-pp.save_fpregs_to_fpstate
0.00 +0.2 0.16 ? 6% perf-profile.children.cycles-pp.perf_trace_buf_update
0.00 +0.2 0.16 ? 9% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.00 +0.2 0.18 ? 10% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.01 ?223% +0.2 0.19 ? 3% perf-profile.children.cycles-pp.rb_insert_color
0.00 +0.2 0.18 ? 3% perf-profile.children.cycles-pp.rb_next
0.00 +0.2 0.19 ? 8% perf-profile.children.cycles-pp.select_idle_cpu
0.00 +0.2 0.20 ? 4% perf-profile.children.cycles-pp.rcu_note_context_switch
0.00 +0.2 0.21 ? 16% perf-profile.children.cycles-pp.set_task_cpu
0.00 +0.2 0.22 ? 4% perf-profile.children.cycles-pp.resched_curr
0.08 ? 11% +0.2 0.32 ? 5% perf-profile.children.cycles-pp.task_h_load
0.00 +0.2 0.24 ? 3% perf-profile.children.cycles-pp.finish_wait
0.04 ? 45% +0.3 0.30 ? 4% perf-profile.children.cycles-pp.rb_erase
0.00 +0.3 0.26 ? 17% perf-profile.children.cycles-pp.perf_trace_sched_switch
0.66 +0.3 0.93 perf-profile.children.cycles-pp.mutex_lock
0.04 ? 45% +0.3 0.32 ? 7% perf-profile.children.cycles-pp.cpuacct_charge
0.00 +0.3 0.28 ? 4% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.20 ? 3% +0.3 0.47 ? 2% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.20 ? 5% +0.3 0.48 ? 4% perf-profile.children.cycles-pp.__list_del_entry_valid
0.02 ? 99% +0.3 0.31 ? 3% perf-profile.children.cycles-pp.native_sched_clock
0.00 +0.3 0.29 ? 3% perf-profile.children.cycles-pp.clear_buddies
0.06 ? 9% +0.3 0.35 ? 3% perf-profile.children.cycles-pp.sched_clock_cpu
0.00 +0.3 0.30 ? 8% perf-profile.children.cycles-pp.cr4_update_irqsoff
0.00 +0.3 0.33 ? 20% perf-profile.children.cycles-pp.available_idle_cpu
0.08 ? 12% +0.4 0.43 ? 4% perf-profile.children.cycles-pp.perf_tp_event
0.07 ? 14% +0.4 0.43 ? 5% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.12 ? 13% +0.4 0.49 ? 6% perf-profile.children.cycles-pp.wake_affine
0.11 ? 4% +0.4 0.48 ? 4% perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
0.06 ? 19% +0.4 0.44 ? 5% perf-profile.children.cycles-pp.finish_task_switch
0.08 ? 14% +0.4 0.48 ? 3% perf-profile.children.cycles-pp.__calc_delta
0.04 ? 45% +0.4 0.48 ? 2% perf-profile.children.cycles-pp.pick_next_entity
0.01 ?223% +0.5 0.48 ? 5% perf-profile.children.cycles-pp.set_next_buddy
0.09 ? 12% +0.6 0.67 ? 2% perf-profile.children.cycles-pp.update_min_vruntime
30.45 +0.6 31.04 perf-profile.children.cycles-pp.ksys_write
0.08 ? 12% +0.6 0.68 ? 3% perf-profile.children.cycles-pp.os_xsave
0.29 ? 6% +0.7 0.98 ? 3% perf-profile.children.cycles-pp.update_cfs_group
0.14 ? 6% +0.7 0.86 ? 2% perf-profile.children.cycles-pp.__update_load_avg_se
0.16 ? 9% +0.7 0.89 ? 2% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
0.08 ? 14% +0.8 0.84 ? 3% perf-profile.children.cycles-pp.check_preempt_wakeup
32.81 +0.8 33.57 perf-profile.children.cycles-pp.ksys_read
0.14 ? 13% +0.8 0.93 perf-profile.children.cycles-pp.update_rq_clock
0.14 ? 10% +0.8 0.94 ? 2% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.11 ? 12% +0.9 0.97 ? 3% perf-profile.children.cycles-pp.check_preempt_curr
0.22 ? 7% +0.9 1.08 ? 4% perf-profile.children.cycles-pp.prepare_to_wait
29.57 +0.9 30.44 perf-profile.children.cycles-pp.vfs_write
0.08 ? 17% +0.9 0.96 ? 23% perf-profile.children.cycles-pp.select_idle_sibling
32.01 +0.9 32.90 perf-profile.children.cycles-pp.vfs_read
0.14 ? 13% +0.9 1.04 ? 4% perf-profile.children.cycles-pp.__switch_to
29.92 +0.9 30.83 perf-profile.children.cycles-pp.sock_read_iter
0.07 ? 19% +0.9 0.99 ? 4% perf-profile.children.cycles-pp.put_prev_entity
0.14 ? 17% +1.0 1.13 ? 3% perf-profile.children.cycles-pp.___perf_sw_event
0.21 ? 13% +1.4 1.58 ? 3% perf-profile.children.cycles-pp.reweight_entity
0.22 ? 12% +1.4 1.63 ? 2% perf-profile.children.cycles-pp.__switch_to_asm
27.85 +1.4 29.28 perf-profile.children.cycles-pp.unix_stream_recvmsg
0.24 ? 11% +1.4 1.66 ? 5% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
27.44 +1.5 28.90 perf-profile.children.cycles-pp.unix_stream_read_generic
0.21 ? 10% +1.5 1.71 ? 2% perf-profile.children.cycles-pp.set_next_entity
27.44 +1.6 29.07 perf-profile.children.cycles-pp.sock_write_iter
0.28 ? 14% +1.7 1.99 ? 11% perf-profile.children.cycles-pp.select_task_rq_fair
0.30 ? 12% +1.8 2.14 ? 4% perf-profile.children.cycles-pp.switch_fpu_return
0.33 ? 14% +2.0 2.32 ? 10% perf-profile.children.cycles-pp.select_task_rq
26.50 +2.0 28.50 perf-profile.children.cycles-pp.sock_sendmsg
0.32 ? 13% +2.1 2.40 ? 3% perf-profile.children.cycles-pp.prepare_task_switch
25.13 +2.2 27.34 perf-profile.children.cycles-pp.unix_stream_sendmsg
2.10 +2.3 4.42 perf-profile.children.cycles-pp._raw_spin_lock
48.33 +2.5 50.78 perf-profile.children.cycles-pp.__libc_write
0.42 ? 10% +2.6 2.97 ? 2% perf-profile.children.cycles-pp.dequeue_entity
0.47 ? 10% +3.0 3.50 perf-profile.children.cycles-pp.enqueue_entity
0.52 ? 10% +3.1 3.61 ? 3% perf-profile.children.cycles-pp.update_load_avg
0.53 ? 11% +3.4 3.94 ? 4% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.54 ? 10% +3.5 4.00 ? 2% perf-profile.children.cycles-pp.update_curr
80.77 +3.5 84.30 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
0.56 ? 8% +3.8 4.40 ? 2% perf-profile.children.cycles-pp.pick_next_task_fair
9.67 +4.5 14.16 ? 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.82 ? 10% +4.9 5.69 ? 2% perf-profile.children.cycles-pp.dequeue_task_fair
0.81 ? 11% +5.2 6.00 perf-profile.children.cycles-pp.enqueue_task_fair
74.28 +5.7 79.93 perf-profile.children.cycles-pp.do_syscall_64
0.36 ? 23% +6.2 6.58 ? 5% perf-profile.children.cycles-pp.exit_to_user_mode_loop
0.95 ? 12% +6.2 7.17 perf-profile.children.cycles-pp.ttwu_do_activate
0.86 ? 13% +8.0 8.90 ? 4% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
2.01 ? 13% +11.9 13.94 ? 2% perf-profile.children.cycles-pp.try_to_wake_up
2.07 ? 13% +12.2 14.25 ? 2% perf-profile.children.cycles-pp.autoremove_wake_function
2.45 ? 10% +12.3 14.71 ? 2% perf-profile.children.cycles-pp.schedule_timeout
2.12 ? 13% +12.5 14.62 ? 2% perf-profile.children.cycles-pp.__wake_up_common
2.33 ? 12% +13.0 15.31 ? 2% perf-profile.children.cycles-pp.__wake_up_common_lock
2.61 ? 10% +13.2 15.80 ? 2% perf-profile.children.cycles-pp.sock_def_readable
2.40 ? 13% +14.7 17.08 ? 2% perf-profile.children.cycles-pp.unix_stream_data_wait
2.73 ? 11% +18.0 20.73 ? 2% perf-profile.children.cycles-pp.schedule
2.78 ? 11% +18.1 20.85 ? 2% perf-profile.children.cycles-pp.__schedule
11.23 -4.5 6.69 perf-profile.self.cycles-pp.syscall_return_via_sysret
5.73 ? 16% -3.9 1.88 ? 3% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
8.71 -3.4 5.30 ? 2% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
3.05 ? 2% -2.4 0.62 ? 12% perf-profile.self.cycles-pp.__slab_free
6.64 -2.2 4.45 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
3.56 ? 3% -2.0 1.57 ? 2% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
4.77 -1.9 2.84 perf-profile.self.cycles-pp.__entry_text_start
2.03 ? 2% -1.1 0.96 ? 7% perf-profile.self.cycles-pp.kmem_cache_free
1.40 -0.9 0.48 ? 6% perf-profile.self.cycles-pp.check_heap_object
0.99 ? 3% -0.8 0.21 ? 8% perf-profile.self.cycles-pp.skb_release_data
1.43 ? 3% -0.8 0.67 ? 3% perf-profile.self.cycles-pp.sock_wfree
0.85 ? 5% -0.8 0.09 ? 62% perf-profile.self.cycles-pp.___slab_alloc
1.98 -0.6 1.34 ? 3% perf-profile.self.cycles-pp.unix_stream_read_generic
1.60 ? 2% -0.5 1.09 ? 4% perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
1.24 ? 2% -0.5 0.78 ? 6% perf-profile.self.cycles-pp.mod_objcg_state
0.68 -0.4 0.24 ? 15% perf-profile.self.cycles-pp.__build_skb_around
0.94 -0.4 0.50 ? 4% perf-profile.self.cycles-pp.__alloc_skb
0.99 ? 3% -0.4 0.57 ? 3% perf-profile.self.cycles-pp.vfs_write
0.97 -0.4 0.56 ? 4% perf-profile.self.cycles-pp.kmem_cache_alloc_node
1.37 ? 2% -0.4 0.98 ? 4% perf-profile.self.cycles-pp.__kmem_cache_free
0.93 -0.4 0.56 ? 2% perf-profile.self.cycles-pp.sock_write_iter
0.87 -0.4 0.50 ? 4% perf-profile.self.cycles-pp.__kmem_cache_alloc_node
1.06 ? 2% -0.3 0.71 ? 2% perf-profile.self.cycles-pp.__might_resched
0.74 ? 2% -0.3 0.43 ? 3% perf-profile.self.cycles-pp.__cond_resched
0.73 ? 6% -0.3 0.48 ? 3% perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.84 ? 2% -0.2 0.59 ? 4% perf-profile.self.cycles-pp.__check_heap_object
1.54 -0.2 1.30 ? 3% perf-profile.self.cycles-pp.apparmor_file_permission
0.60 ? 2% -0.2 0.36 ? 2% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
1.13 ? 3% -0.2 0.91 ? 5% perf-profile.self.cycles-pp.aa_sk_perm
0.70 ? 6% -0.2 0.47 ? 4% perf-profile.self.cycles-pp.skb_set_owner_w
0.66 ? 2% -0.2 0.45 ? 3% perf-profile.self.cycles-pp.__get_obj_cgroup_from_memcg
1.43 ? 3% -0.2 1.22 perf-profile.self.cycles-pp.unix_stream_sendmsg
0.76 -0.2 0.56 perf-profile.self.cycles-pp.sock_read_iter
1.09 ? 2% -0.2 0.92 ? 5% perf-profile.self.cycles-pp.__fget_light
0.43 ? 2% -0.2 0.26 ? 3% perf-profile.self.cycles-pp.obj_cgroup_charge
0.23 ? 5% -0.2 0.06 ? 15% perf-profile.self.cycles-pp.fsnotify_perm
0.33 ? 3% -0.2 0.17 ? 2% perf-profile.self.cycles-pp.refill_obj_stock
0.31 ? 4% -0.2 0.15 ? 15% perf-profile.self.cycles-pp.security_file_permission
0.94 ? 3% -0.1 0.79 ? 5% perf-profile.self.cycles-pp.__virt_addr_valid
0.29 -0.1 0.14 ? 9% perf-profile.self.cycles-pp.__kmalloc_node_track_caller
0.54 -0.1 0.40 perf-profile.self.cycles-pp.__might_sleep
0.41 -0.1 0.28 ? 4% perf-profile.self.cycles-pp.is_vmalloc_addr
0.34 ? 2% -0.1 0.22 ? 4% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.27 ? 4% -0.1 0.15 ? 3% perf-profile.self.cycles-pp._copy_from_iter
0.24 ? 2% -0.1 0.12 ? 5% perf-profile.self.cycles-pp.ksys_write
0.15 ? 4% -0.1 0.06 ? 16% perf-profile.self.cycles-pp.sock_recvmsg
0.27 ? 4% -0.1 0.19 ? 4% perf-profile.self.cycles-pp.kmalloc_slab
0.35 -0.1 0.27 perf-profile.self.cycles-pp.__check_object_size
0.17 ? 2% -0.1 0.09 ? 6% perf-profile.self.cycles-pp.skb_copy_datagram_from_iter
0.22 ? 3% -0.1 0.14 ? 3% perf-profile.self.cycles-pp.ksys_read
0.28 ? 2% -0.1 0.21 ? 3% perf-profile.self.cycles-pp.consume_skb
0.20 ? 2% -0.1 0.13 ? 6% perf-profile.self.cycles-pp.kmalloc_reserve
0.18 ? 5% -0.1 0.11 ? 3% perf-profile.self.cycles-pp.apparmor_socket_sendmsg
0.26 ? 3% -0.1 0.19 ? 2% perf-profile.self.cycles-pp.sock_alloc_send_pskb
0.40 ? 2% -0.1 0.34 ? 3% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.16 ? 3% -0.1 0.09 ? 4% perf-profile.self.cycles-pp.alloc_skb_with_frags
0.15 ? 3% -0.1 0.09 ? 11% perf-profile.self.cycles-pp.sock_sendmsg
0.18 ? 4% -0.1 0.12 ? 7% perf-profile.self.cycles-pp.aa_file_perm
0.18 ? 2% -0.1 0.12 ? 5% perf-profile.self.cycles-pp.apparmor_socket_recvmsg
0.34 ? 2% -0.1 0.29 ? 4% perf-profile.self.cycles-pp.do_syscall_64
0.20 -0.1 0.14 ? 3% perf-profile.self.cycles-pp._copy_to_iter
0.12 ? 7% -0.1 0.07 ? 24% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
0.20 ? 2% -0.1 0.15 ? 4% perf-profile.self.cycles-pp.rcu_all_qs
0.12 ? 7% -0.0 0.08 ? 9% perf-profile.self.cycles-pp.kmalloc_size_roundup
0.11 ? 4% -0.0 0.06 ? 7% perf-profile.self.cycles-pp.__fdget_pos
0.42 ? 2% -0.0 0.37 ? 2% perf-profile.self.cycles-pp.unix_stream_recvmsg
0.64 ? 3% -0.0 0.60 ? 2% perf-profile.self.cycles-pp.__libc_write
0.08 -0.0 0.04 ? 44% perf-profile.self.cycles-pp.copyout
0.18 ? 2% -0.0 0.15 perf-profile.self.cycles-pp.security_socket_getpeersec_dgram
0.09 ? 5% -0.0 0.06 ? 20% perf-profile.self.cycles-pp.should_failslab
0.10 ? 7% -0.0 0.08 ? 6% perf-profile.self.cycles-pp.__might_fault
0.17 ? 2% -0.0 0.14 ? 3% perf-profile.self.cycles-pp.apparmor_socket_getpeersec_dgram
0.16 ? 3% -0.0 0.13 ? 6% perf-profile.self.cycles-pp.obj_cgroup_uncharge_pages
0.08 ? 6% -0.0 0.06 ? 9% perf-profile.self.cycles-pp.security_socket_sendmsg
0.28 ? 2% -0.0 0.26 ? 2% perf-profile.self.cycles-pp.unix_write_space
0.17 ? 2% -0.0 0.15 ? 3% perf-profile.self.cycles-pp.wait_for_unix_gc
0.12 ? 4% -0.0 0.10 ? 6% perf-profile.self.cycles-pp.__skb_datagram_iter
0.07 -0.0 0.06 perf-profile.self.cycles-pp.skb_copy_datagram_iter
0.08 ? 5% +0.0 0.10 ? 5% perf-profile.self.cycles-pp.scm_recv
0.17 ? 2% +0.0 0.19 ? 3% perf-profile.self.cycles-pp.__x64_sys_read
0.19 ? 3% +0.0 0.21 ? 4% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.00 +0.1 0.06 ? 9% perf-profile.self.cycles-pp.finish_wait
0.00 +0.1 0.06 ? 6% perf-profile.self.cycles-pp.cr4_update_irqsoff
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.perf_trace_buf_update
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.check_cfs_rq_runtime
0.00 +0.1 0.07 ? 7% perf-profile.self.cycles-pp.wake_affine
0.00 +0.1 0.07 ? 7% perf-profile.self.cycles-pp.invalidate_user_asid
0.00 +0.1 0.07 ? 15% perf-profile.self.cycles-pp.__bitmap_and
0.00 +0.1 0.07 ? 9% perf-profile.self.cycles-pp.asm_sysvec_reschedule_ipi
0.00 +0.1 0.08 ? 19% perf-profile.self.cycles-pp.read@plt
0.00 +0.1 0.08 ? 8% perf-profile.self.cycles-pp.schedule_debug
0.00 +0.1 0.08 ? 8% perf-profile.self.cycles-pp.perf_trace_buf_alloc
0.00 +0.1 0.09 ? 18% perf-profile.self.cycles-pp.migrate_task_rq_fair
0.00 +0.1 0.09 ? 5% perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
0.00 +0.1 0.10 ? 10% perf-profile.self.cycles-pp.place_entity
0.00 +0.1 0.10 ? 6% perf-profile.self.cycles-pp.__wake_up_common_lock
0.07 ? 10% +0.1 0.18 ? 4% perf-profile.self.cycles-pp.__list_add_valid
0.00 +0.1 0.12 ? 13% perf-profile.self.cycles-pp.select_idle_cpu
0.00 +0.1 0.12 ? 5% perf-profile.self.cycles-pp.native_irq_return_iret
0.00 +0.1 0.12 ? 15% perf-profile.self.cycles-pp.__cgroup_account_cputime
0.00 +0.1 0.13 ? 11% perf-profile.self.cycles-pp.switch_ldt
0.00 +0.1 0.14 ? 4% perf-profile.self.cycles-pp.check_preempt_curr
0.00 +0.1 0.15 ? 7% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.14 ? 3% +0.1 0.28 ? 2% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.00 +0.1 0.15 ? 6% perf-profile.self.cycles-pp.save_fpregs_to_fpstate
0.00 +0.2 0.15 ? 6% perf-profile.self.cycles-pp.__rdgsbase_inactive
0.02 ? 99% +0.2 0.19 ? 4% perf-profile.self.cycles-pp.ttwu_do_activate
0.01 ?223% +0.2 0.18 ? 4% perf-profile.self.cycles-pp.rb_insert_color
0.00 +0.2 0.17 ? 5% perf-profile.self.cycles-pp.autoremove_wake_function
0.00 +0.2 0.18 ? 2% perf-profile.self.cycles-pp.rb_next
0.00 +0.2 0.20 ? 4% perf-profile.self.cycles-pp.rcu_note_context_switch
0.28 ? 2% +0.2 0.48 ? 2% perf-profile.self.cycles-pp.mutex_lock
0.00 +0.2 0.20 ? 6% perf-profile.self.cycles-pp.exit_to_user_mode_loop
0.00 +0.2 0.20 ? 4% perf-profile.self.cycles-pp.resched_curr
0.07 ? 10% +0.2 0.30 ? 4% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.19 ? 4% +0.2 0.42 ? 5% perf-profile.self.cycles-pp.__list_del_entry_valid
0.04 ? 45% +0.2 0.28 ? 6% perf-profile.self.cycles-pp.perf_tp_event
0.00 +0.2 0.23 ? 6% perf-profile.self.cycles-pp.finish_task_switch
0.04 ? 45% +0.2 0.28 ? 4% perf-profile.self.cycles-pp.rb_erase
0.08 ? 11% +0.2 0.32 ? 5% perf-profile.self.cycles-pp.task_h_load
0.00 +0.3 0.26 ? 18% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.00 +0.3 0.27 ? 3% perf-profile.self.cycles-pp.clear_buddies
0.04 ? 71% +0.3 0.31 ? 7% perf-profile.self.cycles-pp.cpuacct_charge
0.02 ? 99% +0.3 0.30 ? 4% perf-profile.self.cycles-pp.native_sched_clock
0.00 +0.3 0.28 ? 4% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.05 ? 45% +0.3 0.33 ? 6% perf-profile.self.cycles-pp.select_task_rq
0.67 ? 3% +0.3 0.96 ? 7% perf-profile.self.cycles-pp.vfs_read
0.07 ? 11% +0.3 0.36 ? 3% perf-profile.self.cycles-pp.schedule_timeout
0.06 ? 16% +0.3 0.36 ? 6% perf-profile.self.cycles-pp.dequeue_entity
0.06 ? 8% +0.3 0.37 ? 3% perf-profile.self.cycles-pp.__wake_up_common
0.02 ?141% +0.3 0.33 ? 2% perf-profile.self.cycles-pp.schedule
0.00 +0.3 0.32 ? 20% perf-profile.self.cycles-pp.available_idle_cpu
0.01 ?223% +0.3 0.36 ? 6% perf-profile.self.cycles-pp.set_next_entity
0.04 ? 45% +0.4 0.40 ? 4% perf-profile.self.cycles-pp.check_preempt_wakeup
0.07 ? 14% +0.4 0.43 ? 5% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.08 ? 11% +0.4 0.44 ? 4% perf-profile.self.cycles-pp.prepare_to_wait
0.08 ? 14% +0.4 0.48 ? 4% perf-profile.self.cycles-pp.__calc_delta
0.05 ? 46% +0.4 0.45 ? 3% perf-profile.self.cycles-pp.switch_fpu_return
0.02 ?141% +0.4 0.42 ? 2% perf-profile.self.cycles-pp.pick_next_entity
0.77 +0.4 1.17 ? 5% perf-profile.self.cycles-pp.__libc_read
0.07 ? 15% +0.4 0.48 ? 3% perf-profile.self.cycles-pp.unix_stream_data_wait
0.02 ?142% +0.4 0.44 ? 20% perf-profile.self.cycles-pp.select_idle_sibling
0.01 ?223% +0.5 0.46 ? 5% perf-profile.self.cycles-pp.set_next_buddy
0.07 ? 17% +0.5 0.53 ? 3% perf-profile.self.cycles-pp.select_task_rq_fair
0.09 ? 13% +0.5 0.55 ? 2% perf-profile.self.cycles-pp.try_to_wake_up
0.09 ? 14% +0.5 0.57 ? 4% perf-profile.self.cycles-pp.reweight_entity
0.08 ? 16% +0.5 0.57 perf-profile.self.cycles-pp.update_rq_clock
0.09 ? 12% +0.5 0.59 ? 2% perf-profile.self.cycles-pp.dequeue_task_fair
0.09 ? 10% +0.6 0.64 perf-profile.self.cycles-pp.update_min_vruntime
0.02 ?141% +0.6 0.59 ? 3% perf-profile.self.cycles-pp.put_prev_entity
0.07 ? 16% +0.6 0.66 ? 2% perf-profile.self.cycles-pp.enqueue_task_fair
0.08 ? 12% +0.6 0.68 ? 3% perf-profile.self.cycles-pp.os_xsave
0.29 ? 6% +0.7 0.96 ? 3% perf-profile.self.cycles-pp.update_cfs_group
0.14 ? 9% +0.7 0.83 ? 2% perf-profile.self.cycles-pp.__update_load_avg_se
0.12 ? 8% +0.7 0.80 ? 5% perf-profile.self.cycles-pp.prepare_task_switch
1.33 ? 2% +0.8 2.09 ? 2% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.13 ? 11% +0.8 0.91 ? 2% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
1.74 +0.8 2.52 perf-profile.self.cycles-pp._raw_spin_lock
0.14 ? 10% +0.8 0.93 ? 2% perf-profile.self.cycles-pp.pick_next_task_fair
0.14 ? 15% +0.8 0.98 ? 4% perf-profile.self.cycles-pp.__switch_to
0.13 ? 16% +0.9 1.06 ? 3% perf-profile.self.cycles-pp.___perf_sw_event
0.17 ? 12% +1.2 1.34 perf-profile.self.cycles-pp.enqueue_entity
0.22 ? 12% +1.4 1.61 ? 2% perf-profile.self.cycles-pp.__switch_to_asm
0.24 ? 11% +1.4 1.66 ? 5% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
0.26 ? 12% +1.5 1.76 ? 3% perf-profile.self.cycles-pp.__schedule
0.23 ? 9% +1.5 1.76 ? 4% perf-profile.self.cycles-pp.update_load_avg
0.23 ? 9% +1.8 1.99 ? 2% perf-profile.self.cycles-pp.update_curr
0.52 ? 11% +3.3 3.81 ? 4% perf-profile.self.cycles-pp.switch_mm_irqs_off
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
sudo bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
sudo bin/lkp run generated-yaml-file
# if come across any failure that blocks the test,
# please remove ~/.lkp and /lkp dir to run from a clean state.
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
On 2023-03-09 at 11:46:50 +0800, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed a -7.9% regression of hackbench.throughput due to commit:
>
> commit: 829c1651e9c4a6f78398d3e67651cef9bb6b42cc ("sched/fair: sanitize vruntime of entity being placed")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: hackbench
> on test machine: 104 threads 2 sockets (Skylake) with 192G memory
> with following parameters:
>
> nr_threads: 100%
> iterations: 4
> mode: process
> ipc: socket
> cpufreq_governor: performance
>
In theory the patch deals with a corner case that the task sleeps for more than 1 minute,
but this is not the case for hackbench, and it did increase the preemption:
1.08e+08 ? 2% +648.7% 8.084e+08 ? 5% hackbench.time.involuntary_context_switches
It might be an issue described and fixed here:
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/ZAiFxWLSb9HDazSI@vingu-book/
I'll check if above patch helps.
thanks,
Chenyu