Greeting,
FYI, we noticed a -8.0% regression of will-it-scale.per_thread_ops due to commit:
commit: 286e04b8ed7a04279ae277f0f024430246ea5eec ("mm: memcg/slab: allocate obj_cgroups for non-root slab pages")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: will-it-scale
on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:
nr_task: 16
mode: thread
test: brk1
cpufreq_governor: performance
ucode: 0x5003006
test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale
In addition to that, the commit also has significant impact on the following tests:
+------------------+-------------------------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput -19.7% regression |
| test machine | 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters | cpufreq_governor=performance |
| | ipc=socket |
| | iterations=4 |
| | mode=threads |
| | nr_threads=100% |
| | ucode=0x5003006 |
+------------------+-------------------------------------------------------------------------------------+
| testcase: change | hackbench: boot-time.boot 10.6% regression |
| test machine | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory |
| test parameters | cpufreq_governor=performance |
| | ipc=socket |
| | iterations=4 |
| | mode=process |
| | nr_threads=100% |
| | ucode=0x5003003 |
+------------------+-------------------------------------------------------------------------------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
bin/lkp run generated-yaml-file
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/thread/16/debian-10.4-x86_64-20200603.cgz/lkp-csl-2ap2/brk1/will-it-scale/0x5003006
commit:
bf4f059954 ("mm: memcg/slab: obj_cgroup API")
286e04b8ed ("mm: memcg/slab: allocate obj_cgroups for non-root slab pages")
bf4f059954dcb221 286e04b8ed7a04279ae277f0f02
---------------- ---------------------------
%stddev %change %stddev
\ | \
1259202 -8.0% 1159095 ? 5% will-it-scale.16.threads
78699 -8.0% 72443 ? 5% will-it-scale.per_thread_ops
1259202 -8.0% 1159095 ? 5% will-it-scale.workload
0.11 ? 5% -0.0 0.10 ? 7% mpstat.cpu.all.soft%
168284 ? 14% -37.2% 105678 ? 43% numa-vmstat.node3.numa_other
4010690 +8.6% 4353788 ? 5% perf-stat.overall.path-length
10386 +1.4% 10527 proc-vmstat.nr_mapped
2118886 ? 7% -16.2% 1775301 ? 8% sched_debug.cfs_rq:/.min_vruntime.max
1879470 ? 9% -20.0% 1504476 ? 10% sched_debug.cfs_rq:/.spread0.max
4.09 ? 4% +9.6% 4.48 ? 2% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
292.83 ? 4% -67.9% 94.00 ?141% perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
2449 ? 3% -9.0% 2230 ? 2% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
4.07 ? 3% +9.7% 4.46 ? 2% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
41872 ? 4% -14.9% 35652 ? 16% softirqs.CPU30.SCHED
41359 ? 2% -12.7% 36103 ? 17% softirqs.CPU49.SCHED
42091 ? 3% -6.8% 39230 ? 3% softirqs.CPU62.SCHED
42051 ? 3% -6.9% 39166 ? 3% softirqs.CPU67.SCHED
0.01 ?223% +0.0 0.06 ? 13% perf-profile.children.cycles-pp.run_local_timers
0.21 ? 15% +0.1 0.33 ? 20% perf-profile.children.cycles-pp.anon_vma_clone
1.00 ? 9% -0.2 0.75 ? 13% perf-profile.self.cycles-pp.rwsem_optimistic_spin
0.07 ? 20% +0.1 0.13 ? 21% perf-profile.self.cycles-pp.anon_vma_clone
0.24 ? 9% +0.1 0.31 ? 12% perf-profile.self.cycles-pp.__vma_adjust
130.17 ? 22% +75.2% 228.00 ? 22% interrupts.CPU0.RES:Rescheduling_interrupts
5270 ? 21% -32.3% 3570 ? 30% interrupts.CPU109.NMI:Non-maskable_interrupts
5270 ? 21% -32.3% 3570 ? 30% interrupts.CPU109.PMI:Performance_monitoring_interrupts
117.83 ? 17% +19.2% 140.50 ? 7% interrupts.CPU134.NMI:Non-maskable_interrupts
117.83 ? 17% +19.2% 140.50 ? 7% interrupts.CPU134.PMI:Performance_monitoring_interrupts
861.00 +56.9% 1351 ? 42% interrupts.CPU188.CAL:Function_call_interrupts
66.67 ? 42% +59.2% 106.17 ? 27% interrupts.CPU23.NMI:Non-maskable_interrupts
66.67 ? 42% +59.2% 106.17 ? 27% interrupts.CPU23.PMI:Performance_monitoring_interrupts
260.00 ? 26% +45.8% 379.00 ? 21% interrupts.CPU95.NMI:Non-maskable_interrupts
260.00 ? 26% +45.8% 379.00 ? 21% interrupts.CPU95.PMI:Performance_monitoring_interrupts
23771 ? 12% -25.1% 17793 ? 19% interrupts.CPU98.CAL:Function_call_interrupts
will-it-scale.per_thread_ops
85000 +-------------------------------------------------------------------+
| .+ |
| + + + .+ ++ +. |
80000 |:.+ + +.+ :+ +.+++ :+ :+ + + :+ +.+ .+ : + +.+|
|+ :+: : ++.+ + + : : + + + : + + : + +.: :.+ |
| + : + : : : : + O + + + |
75000 |-+ :: :: : : O O |
| O::O + :: OO OO O |
70000 |-+OO + O O :: OO |
| O O OO :O |
| O O O O O + |
65000 |O+ |
| O |
| O |
60000 +-------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
***************************************************************************************************
lkp-csl-2ap4: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
gcc-9/performance/socket/4/x86_64-rhel-8.3/threads/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2ap4/hackbench/0x5003006
commit:
bf4f059954 ("mm: memcg/slab: obj_cgroup API")
286e04b8ed ("mm: memcg/slab: allocate obj_cgroups for non-root slab pages")
bf4f059954dcb221 286e04b8ed7a04279ae277f0f02
---------------- ---------------------------
%stddev %change %stddev
\ | \
200843 ? 5% -19.7% 161359 ? 4% hackbench.throughput
4.774e+09 ? 4% +6.5% 5.084e+09 ? 2% hackbench.time.involuntary_context_switches
4361 ? 7% +9.4% 4771 ? 5% hackbench.time.user_time
6.849e+09 ? 2% +6.2% 7.273e+09 ? 2% hackbench.time.voluntary_context_switches
0.00 ?115% +0.0 0.00 ? 50% mpstat.cpu.all.iowait%
0.03 ? 2% +0.0 0.04 ? 15% mpstat.cpu.all.soft%
55921 ? 31% +44.1% 80594 ? 10% numa-numastat.node2.other_node
86786 -52.4% 41350 ? 77% numa-numastat.node3.other_node
7930 ? 12% +28.7% 10208 ? 6% slabinfo.eventpoll_pwq.active_objs
7930 ? 12% +28.7% 10208 ? 6% slabinfo.eventpoll_pwq.num_objs
1.726e+08 ? 25% +38.1% 2.384e+08 ? 12% cpuidle.C1.time
11923928 ? 26% +40.4% 16738105 ? 10% cpuidle.C1.usage
8e+08 ? 35% +36.1% 1.089e+09 ? 3% cpuidle.C1E.time
3885779 ? 16% +30.1% 5054945 ? 8% cpuidle.C1E.usage
2995540 ? 27% +51.7% 4542950 ? 9% cpuidle.POLL.time
1099556 ? 26% +53.2% 1685053 ? 9% cpuidle.POLL.usage
5087256 ? 23% +74.8% 8894478 ? 25% turbostat.C1
0.06 ? 22% +0.1 0.11 ? 30% turbostat.C1%
1645987 ? 9% +69.4% 2787738 ? 26% turbostat.C1E
0.33 ? 32% +0.1 0.47 ? 9% turbostat.C1E%
7.906e+08 ? 2% +16.4% 9.202e+08 ? 6% turbostat.IRQ
61.67 ? 3% +4.3% 64.33 turbostat.PkgTmp
329726 ? 32% +113.5% 703843 ? 43% numa-meminfo.node0.FilePages
19643 ?108% +412.4% 100661 ? 60% numa-meminfo.node0.Inactive
19643 ?108% +412.4% 100661 ? 60% numa-meminfo.node0.Inactive(anon)
89287 ?130% +399.6% 446057 ? 70% numa-meminfo.node0.Shmem
42820 ? 18% -37.4% 26822 ? 16% numa-meminfo.node2.KReclaimable
42820 ? 18% -37.4% 26822 ? 16% numa-meminfo.node2.SReclaimable
153505 ? 43% -43.0% 87443 ? 7% numa-meminfo.node2.SUnreclaim
196325 ? 37% -41.8% 114266 ? 2% numa-meminfo.node2.Slab
80741 ? 26% +241.4% 275684 ? 47% numa-meminfo.node3.AnonPages.max
799.33 ? 28% +120.8% 1764 ? 40% numa-meminfo.node3.PageTables
4.072e+08 ? 2% -3.2% 3.942e+08 ? 3% perf-stat.i.cache-misses
1569742 ? 5% +6.4% 1670136 ? 2% perf-stat.i.cpu-migrations
1481 ? 2% +3.2% 1528 ? 3% perf-stat.i.cycles-between-cache-misses
0.00 ? 7% -0.0 0.00 ? 3% perf-stat.i.dTLB-store-miss-rate%
1191539 ? 11% -24.8% 896157 ? 11% perf-stat.i.dTLB-store-misses
64065533 ? 4% -4.4% 61233574 ? 4% perf-stat.i.node-store-misses
1124963 ? 12% -22.1% 876883 ? 14% perf-stat.i.node-stores
0.00 ? 8% -0.0 0.00 ? 5% perf-stat.overall.dTLB-store-miss-rate%
1189270 ? 11% -25.3% 888021 ? 12% perf-stat.ps.dTLB-store-misses
63319469 ? 3% -4.9% 60243321 ? 4% perf-stat.ps.node-store-misses
1130269 ? 12% -21.8% 884361 ? 15% perf-stat.ps.node-stores
1.032e+14 ? 2% +5.3% 1.087e+14 ? 2% perf-stat.total.instructions
82471 ? 32% +113.1% 175770 ? 43% numa-vmstat.node0.nr_file_pages
5078 ?109% +397.9% 25286 ? 60% numa-vmstat.node0.nr_inactive_anon
22361 ?130% +397.8% 111323 ? 70% numa-vmstat.node0.nr_shmem
5078 ?109% +397.9% 25286 ? 60% numa-vmstat.node0.nr_zone_inactive_anon
1257447 ? 13% -22.8% 971273 ? 11% numa-vmstat.node1.numa_local
84779 ? 36% +79.1% 151814 ? 5% numa-vmstat.node1.numa_other
10702 ? 18% -37.3% 6710 ? 16% numa-vmstat.node2.nr_slab_reclaimable
38337 ? 43% -43.0% 21857 ? 8% numa-vmstat.node2.nr_slab_unreclaimable
135697 ? 12% +17.9% 160016 ? 5% numa-vmstat.node2.numa_other
77071 ? 42% -41.7% 44935 ? 21% numa-vmstat.node3.nr_inactive_anon
199.00 ? 27% +121.4% 440.67 ? 39% numa-vmstat.node3.nr_page_table_pages
77070 ? 42% -41.7% 44935 ? 21% numa-vmstat.node3.nr_zone_inactive_anon
1317429 ? 6% +18.4% 1560077 ? 9% numa-vmstat.node3.numa_local
166193 -38.9% 101502 ? 50% numa-vmstat.node3.numa_other
51355399 ? 5% +17.9% 60523801 ? 9% sched_debug.cfs_rq:/.min_vruntime.avg
66657407 ? 6% +28.4% 85595963 ? 15% sched_debug.cfs_rq:/.min_vruntime.max
44178814 ? 7% +14.7% 50690341 ? 7% sched_debug.cfs_rq:/.min_vruntime.min
4396785 ? 33% +55.6% 6842280 ? 25% sched_debug.cfs_rq:/.min_vruntime.stddev
0.11 ? 3% -31.9% 0.07 ? 21% sched_debug.cfs_rq:/.nr_running.stddev
0.28 ? 45% +192.2% 0.83 ? 55% sched_debug.cfs_rq:/.removed.runnable_avg.avg
2.89 ? 48% +110.9% 6.10 ? 31% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
0.28 ? 45% +192.2% 0.83 ? 55% sched_debug.cfs_rq:/.removed.util_avg.avg
2.89 ? 48% +110.9% 6.10 ? 31% sched_debug.cfs_rq:/.removed.util_avg.stddev
-6080241 +143.1% -14782494 sched_debug.cfs_rq:/.spread0.avg
-12219782 +95.2% -23848569 sched_debug.cfs_rq:/.spread0.min
4043652 ? 38% +61.7% 6539680 ? 23% sched_debug.cfs_rq:/.spread0.stddev
478.33 ? 6% +15.6% 552.77 ? 4% sched_debug.cfs_rq:/.util_avg.min
161.41 ? 2% -14.0% 138.76 ? 2% sched_debug.cfs_rq:/.util_avg.stddev
300408 ? 4% +14.6% 344155 ? 8% sched_debug.cpu.clock.avg
306855 ? 4% +14.2% 350489 ? 8% sched_debug.cpu.clock.max
291967 ? 4% +15.1% 336065 ? 8% sched_debug.cpu.clock.min
284821 ? 4% +14.1% 324960 ? 7% sched_debug.cpu.clock_task.avg
295707 ? 4% +14.4% 338410 ? 8% sched_debug.cpu.clock_task.max
263320 ? 5% +12.3% 295775 ? 7% sched_debug.cpu.clock_task.min
2763 ? 6% -17.0% 2292 ? 9% sched_debug.cpu.curr->pid.stddev
1290116 ? 33% -29.3% 911513 ? 6% sched_debug.cpu.max_idle_balance_cost.max
137888 ? 71% -61.3% 53408 ? 13% sched_debug.cpu.max_idle_balance_cost.stddev
4.07 ? 2% -18.4% 3.32 ? 13% sched_debug.cpu.nr_running.stddev
27664104 ? 3% +9.5% 30279647 ? 2% sched_debug.cpu.nr_switches.avg
39994582 ? 5% +10.6% 44251966 ? 3% sched_debug.cpu.nr_switches.max
16105574 ? 3% +12.0% 18041900 sched_debug.cpu.nr_switches.min
4467021 ? 6% +24.3% 5553088 ? 7% sched_debug.cpu.nr_switches.stddev
1.99 ? 17% -69.7% 0.60 ?105% sched_debug.cpu.nr_uninterruptible.avg
-101.08 +19.4% -120.74 sched_debug.cpu.nr_uninterruptible.min
291784 ? 4% +15.1% 335862 ? 8% sched_debug.cpu_clk
290911 ? 4% +15.2% 334987 ? 8% sched_debug.ktime
292141 ? 4% +15.1% 336220 ? 8% sched_debug.sched_clk
0.18 ? 49% -75.5% 0.05 ?133% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
9.03 ?110% -91.6% 0.76 ?130% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
154.52 ?127% +4336.8% 6856 ?134% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
20267 ? 20% -66.3% 6840 ?108% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
23853 ? 13% +57.0% 37450 ? 17% perf-sched.total_wait_time.max.ms
133.73 ? 59% +87.1% 250.22 ? 34% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
42.92 ?141% +1554.6% 710.17 ? 73% perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
849.14 ? 13% +144.6% 2076 ? 17% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
38.67 ? 46% -79.3% 8.00 ?141% perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
71.00 ? 26% -56.3% 31.00 ? 77% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
405.33 ? 13% -21.8% 317.00 ? 10% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
20886 ? 20% +58.5% 33096 ? 27% perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
16704 ? 29% +122.2% 37109 ? 17% perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
0.30 ? 45% +2e+05% 588.96 ?141% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe
133.44 ? 59% +82.6% 243.63 ? 33% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
44.21 ?134% +1505.0% 709.59 ? 73% perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.02 ? 12% +180.4% 0.05 ? 45% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_read.walk_component.link_path_walk
0.00 ?141% +17900.0% 0.36 ?134% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.mutex_lock.perf_event_for_each_child._perf_ioctl
53.43 ? 16% +161.5% 139.70 ? 32% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.stop_one_cpu.__set_cpus_allowed_ptr.sched_setaffinity
0.01 ?141% +552.9% 0.07 ? 78% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.wp_page_copy.do_wp_page.__handle_mm_fault
545.69 ?117% -99.8% 1.04 ?139% perf-sched.wait_time.avg.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
776.66 ? 12% +161.0% 2026 ? 19% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.55 ? 61% +5.7e+05% 3131 ?141% perf-sched.wait_time.max.ms.do_task_dead.do_exit.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe
20886 ? 20% +54.6% 32295 ? 26% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
23853 ? 13% +41.7% 33811 ? 23% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
6067 ? 46% -73.3% 1618 ?133% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg
23050 ? 13% +38.6% 31938 ? 22% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
0.03 ? 81% +133.7% 0.07 ? 42% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.dput.terminate_walk.path_openat
0.00 ?141% +17900.0% 0.36 ?134% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.mutex_lock.perf_event_for_each_child._perf_ioctl
4272 ? 37% +76.3% 7531 ? 32% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.stop_one_cpu.__set_cpus_allowed_ptr.sched_setaffinity
0.03 ?141% +332.6% 0.14 ? 81% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.wp_page_copy.do_wp_page.__handle_mm_fault
570.36 ?140% -99.7% 1.65 ?141% perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
3612 ? 86% -99.9% 3.26 ?140% perf-sched.wait_time.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
21679 ? 11% +52.3% 33025 ? 23% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
16704 ? 29% +122.2% 37109 ? 17% perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
37.10 ? 3% -2.4 34.73 perf-profile.calltrace.cycles-pp.__libc_read
36.20 ? 3% -2.3 33.91 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
35.20 ? 3% -2.1 33.07 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
35.08 ? 3% -2.1 32.97 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
33.84 ? 3% -2.0 31.88 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
0.64 -0.2 0.40 ? 70% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.unix_stream_read_generic
1.10 ? 12% -0.2 0.89 ? 4% perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
1.08 ? 12% -0.2 0.87 ? 4% perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
0.95 ? 15% -0.2 0.79 ? 7% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_read
0.92 ? 15% -0.1 0.78 ? 6% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_read
4.98 ? 20% +0.8 5.80 perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
8.27 ? 2% +1.0 9.27 ? 9% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
8.67 +1.0 9.68 ? 10% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
37.18 ? 3% -2.4 34.79 perf-profile.children.cycles-pp.__libc_read
2.75 ? 92% -2.3 0.48 ? 39% perf-profile.children.cycles-pp.update_cfs_group
1.36 ? 12% -0.2 1.11 ? 6% perf-profile.children.cycles-pp.alloc_skb_with_frags
1.32 ? 12% -0.2 1.09 ? 6% perf-profile.children.cycles-pp.__alloc_skb
0.19 ? 66% -0.1 0.06 ? 7% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.66 ? 2% -0.1 0.59 ? 2% perf-profile.children.cycles-pp.cpuacct_charge
0.29 ? 8% -0.1 0.23 ? 8% perf-profile.children.cycles-pp._copy_from_iter
0.55 ? 5% -0.1 0.49 ? 2% perf-profile.children.cycles-pp.set_next_entity
0.33 ? 5% -0.1 0.27 perf-profile.children.cycles-pp.security_socket_sendmsg
0.38 ? 4% -0.1 0.32 ? 2% perf-profile.children.cycles-pp.fsnotify
0.27 ? 8% -0.1 0.22 ? 3% perf-profile.children.cycles-pp.entry_SYSCALL_64
0.48 ? 7% -0.0 0.43 ? 5% perf-profile.children.cycles-pp.___might_sleep
0.38 ? 3% -0.0 0.34 ? 5% perf-profile.children.cycles-pp.aa_sk_perm
0.24 ? 5% -0.0 0.19 ? 10% perf-profile.children.cycles-pp.sock_recvmsg
0.40 ? 6% -0.0 0.36 ? 4% perf-profile.children.cycles-pp.kmem_cache_alloc_node
0.21 ? 4% -0.0 0.17 ? 11% perf-profile.children.cycles-pp.security_socket_recvmsg
0.18 ? 9% -0.0 0.14 ? 5% perf-profile.children.cycles-pp.copyin
0.25 ? 6% -0.0 0.22 ? 5% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.15 ? 8% -0.0 0.13 ? 3% perf-profile.children.cycles-pp.__check_heap_object
0.09 ? 19% -0.0 0.06 ? 13% perf-profile.children.cycles-pp.__list_add_valid
0.14 ? 8% -0.0 0.12 ? 6% perf-profile.children.cycles-pp.__might_sleep
0.19 ? 6% -0.0 0.17 ? 4% perf-profile.children.cycles-pp.__list_del_entry_valid
0.15 ? 3% -0.0 0.13 ? 10% perf-profile.children.cycles-pp.fsnotify_parent
0.08 ? 5% -0.0 0.07 ? 7% perf-profile.children.cycles-pp.wait_for_unix_gc
0.04 ? 71% +0.0 0.07 perf-profile.children.cycles-pp.__module_text_address
0.02 ?141% +0.0 0.06 ? 8% perf-profile.children.cycles-pp.perf_swevent_event
0.31 ? 6% +0.0 0.35 ? 3% perf-profile.children.cycles-pp.task_tick_fair
0.02 ?141% +0.0 0.06 perf-profile.children.cycles-pp.__perf_event_account_interrupt
0.06 ? 71% +0.0 0.10 perf-profile.children.cycles-pp.is_module_text_address
0.34 ? 6% +0.1 0.40 ? 2% perf-profile.children.cycles-pp.scheduler_tick
0.15 ? 44% +0.1 0.23 ? 7% perf-profile.children.cycles-pp.perf_tp_event_match
0.41 ? 6% +0.1 0.49 ? 6% perf-profile.children.cycles-pp.update_process_times
0.41 ? 6% +0.1 0.49 ? 5% perf-profile.children.cycles-pp.tick_sched_handle
0.44 ? 6% +0.1 0.53 ? 6% perf-profile.children.cycles-pp.tick_sched_timer
0.21 ? 33% +0.1 0.32 ? 15% perf-profile.children.cycles-pp.in_task_stack
0.49 ? 4% +0.1 0.61 ? 8% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.30 ? 35% +0.1 0.45 ? 11% perf-profile.children.cycles-pp.get_stack_info
0.69 +0.3 0.94 ? 16% perf-profile.children.cycles-pp.hrtimer_interrupt
0.69 ? 2% +0.3 0.95 ? 16% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.76 +0.3 1.03 ? 16% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.86 ? 3% +0.3 1.13 ? 15% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
2.74 ? 92% -2.3 0.47 ? 39% perf-profile.self.cycles-pp.update_cfs_group
0.32 ? 52% -0.2 0.13 ? 6% perf-profile.self.cycles-pp.___perf_sw_event
0.37 ? 27% -0.1 0.26 ? 13% perf-profile.self.cycles-pp.__ksize
0.66 ? 2% -0.1 0.59 ? 2% perf-profile.self.cycles-pp.cpuacct_charge
0.32 ? 6% -0.1 0.25 ? 3% perf-profile.self.cycles-pp.common_file_perm
0.35 ? 3% -0.1 0.30 perf-profile.self.cycles-pp.fsnotify
0.14 ? 23% -0.0 0.10 ? 12% perf-profile.self.cycles-pp.mutex_unlock
0.16 ? 10% -0.0 0.13 ? 9% perf-profile.self.cycles-pp.aa_sk_perm
0.18 ? 4% -0.0 0.15 ? 5% perf-profile.self.cycles-pp.reweight_entity
0.16 ? 8% -0.0 0.13 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.13 ? 9% -0.0 0.10 ? 4% perf-profile.self.cycles-pp.__alloc_skb
0.14 -0.0 0.11 ? 8% perf-profile.self.cycles-pp.fsnotify_parent
0.12 ? 6% -0.0 0.09 ? 5% perf-profile.self.cycles-pp.__libc_read
0.24 ? 5% -0.0 0.22 ? 5% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.16 ? 7% -0.0 0.14 ? 3% perf-profile.self.cycles-pp.entry_SYSCALL_64
0.14 ? 8% -0.0 0.12 perf-profile.self.cycles-pp.check_preempt_wakeup
0.14 ? 8% -0.0 0.12 ? 6% perf-profile.self.cycles-pp.__check_heap_object
0.12 ? 10% -0.0 0.10 ? 8% perf-profile.self.cycles-pp.__might_sleep
0.08 ? 12% -0.0 0.06 ? 8% perf-profile.self.cycles-pp._copy_from_iter
0.14 ? 5% -0.0 0.12 ? 3% perf-profile.self.cycles-pp.enqueue_entity
0.08 ? 10% -0.0 0.06 ? 7% perf-profile.self.cycles-pp.unix_stream_recvmsg
0.07 ? 18% +0.0 0.09 ? 5% perf-profile.self.cycles-pp.autoremove_wake_function
0.05 ? 71% +0.0 0.09 ? 5% perf-profile.self.cycles-pp.perf_misc_flags
0.04 ? 71% +0.0 0.09 ? 10% perf-profile.self.cycles-pp.perf_swevent_overflow
0.34 ? 6% +0.0 0.38 ? 7% perf-profile.self.cycles-pp.vfs_write
0.07 ? 71% +0.0 0.12 ? 4% perf-profile.self.cycles-pp.perf_event_pid_type
0.00 +0.1 0.05 ? 8% perf-profile.self.cycles-pp.__perf_event_account_interrupt
0.14 ? 46% +0.1 0.20 ? 7% perf-profile.self.cycles-pp.perf_tp_event_match
0.16 ? 40% +0.1 0.24 ? 7% perf-profile.self.cycles-pp.__perf_event_header__init_id
0.41 ? 21% +0.1 0.51 ? 4% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.21 ? 33% +0.1 0.31 ? 17% perf-profile.self.cycles-pp.in_task_stack
0.90 ? 6% +0.1 1.04 ? 4% perf-profile.self.cycles-pp.perf_tp_event
0.26 ? 32% +0.2 0.42 ? 14% perf-profile.self.cycles-pp.__perf_event_overflow
0.32 ? 43% +0.2 0.49 ? 2% perf-profile.self.cycles-pp.perf_prepare_sample
0.58 ? 28% +0.2 0.83 ? 2% perf-profile.self.cycles-pp.perf_trace_sched_switch
11383 ? 69% +426.8% 59963 ? 45% interrupts.39:PCI-MSI.524290-edge.eth0-TxRx-1
5445 ? 40% +174.1% 14925 ? 59% interrupts.41:PCI-MSI.524292-edge.eth0-TxRx-3
2457907 ? 11% -22.6% 1901728 interrupts.CPU1.RES:Rescheduling_interrupts
2641257 -17.6% 2177600 ? 6% interrupts.CPU104.RES:Rescheduling_interrupts
11383 ? 69% +426.8% 59963 ? 45% interrupts.CPU11.39:PCI-MSI.524290-edge.eth0-TxRx-1
5160330 ? 8% -20.9% 4079569 ? 16% interrupts.CPU110.CAL:Function_call_interrupts
4480782 ? 10% +16.6% 5223363 ? 10% interrupts.CPU112.CAL:Function_call_interrupts
1993851 ? 17% +30.8% 2607702 ? 6% interrupts.CPU114.RES:Rescheduling_interrupts
1920162 ? 20% +43.8% 2760707 ? 14% interrupts.CPU115.RES:Rescheduling_interrupts
1838014 ? 19% +29.9% 2387620 ? 13% interrupts.CPU118.RES:Rescheduling_interrupts
5778 ? 35% +50.6% 8699 interrupts.CPU121.NMI:Non-maskable_interrupts
5778 ? 35% +50.6% 8699 interrupts.CPU121.PMI:Performance_monitoring_interrupts
5770 ? 35% +25.7% 7251 ? 28% interrupts.CPU124.NMI:Non-maskable_interrupts
5770 ? 35% +25.7% 7251 ? 28% interrupts.CPU124.PMI:Performance_monitoring_interrupts
5445 ? 40% +174.1% 14925 ? 59% interrupts.CPU13.41:PCI-MSI.524292-edge.eth0-TxRx-3
5188909 ? 3% +26.8% 6578819 ? 15% interrupts.CPU130.CAL:Function_call_interrupts
1741448 ? 13% +28.9% 2244772 ? 8% interrupts.CPU134.RES:Rescheduling_interrupts
4870246 ? 14% +23.2% 6002115 ? 13% interrupts.CPU135.CAL:Function_call_interrupts
2104688 ? 11% +22.1% 2570605 ? 9% interrupts.CPU138.RES:Rescheduling_interrupts
5567970 ? 5% -15.6% 4699013 ? 4% interrupts.CPU141.CAL:Function_call_interrupts
6426481 ? 17% -28.6% 4585671 ? 12% interrupts.CPU144.CAL:Function_call_interrupts
6230112 ? 12% -20.6% 4948034 ? 11% interrupts.CPU145.CAL:Function_call_interrupts
1567730 ? 26% +46.8% 2302159 ? 10% interrupts.CPU146.RES:Rescheduling_interrupts
1846928 ? 8% +12.3% 2074112 interrupts.CPU148.RES:Rescheduling_interrupts
1814816 ? 6% +11.5% 2023890 ? 5% interrupts.CPU149.RES:Rescheduling_interrupts
6912968 ? 13% -32.8% 4642666 ? 21% interrupts.CPU151.CAL:Function_call_interrupts
2319343 ? 9% -19.0% 1877846 ? 7% interrupts.CPU152.RES:Rescheduling_interrupts
1822993 ? 12% +43.2% 2610289 ? 16% interrupts.CPU153.RES:Rescheduling_interrupts
5838607 ? 5% -18.7% 4744873 ? 17% interrupts.CPU158.CAL:Function_call_interrupts
5926829 ? 10% -16.9% 4927464 ? 9% interrupts.CPU161.CAL:Function_call_interrupts
1682857 ? 19% +36.2% 2292749 ? 11% interrupts.CPU164.RES:Rescheduling_interrupts
4116921 ? 18% +66.9% 6869135 ? 32% interrupts.CPU171.CAL:Function_call_interrupts
4142580 +60.6% 6652336 ? 16% interrupts.CPU172.CAL:Function_call_interrupts
4502353 ? 18% +46.8% 6610734 ? 8% interrupts.CPU173.CAL:Function_call_interrupts
4659039 ? 14% +46.3% 6818440 ? 34% interrupts.CPU175.CAL:Function_call_interrupts
4447231 ? 9% +66.2% 7389442 ? 21% interrupts.CPU176.CAL:Function_call_interrupts
4255423 ? 13% +56.4% 6654582 ? 15% interrupts.CPU177.CAL:Function_call_interrupts
4319938 ? 10% +33.7% 5774236 ? 10% interrupts.CPU178.CAL:Function_call_interrupts
4203332 ? 9% +74.5% 7335115 ? 22% interrupts.CPU179.CAL:Function_call_interrupts
4614871 ? 10% +51.8% 7005269 ? 23% interrupts.CPU180.CAL:Function_call_interrupts
2181783 ? 3% -19.7% 1751755 ? 4% interrupts.CPU180.RES:Rescheduling_interrupts
4050520 ? 11% +75.9% 7123398 ? 15% interrupts.CPU182.CAL:Function_call_interrupts
1699459 ? 3% +32.3% 2249005 ? 16% interrupts.CPU182.RES:Rescheduling_interrupts
4990167 ? 5% +40.9% 7029004 ? 12% interrupts.CPU183.CAL:Function_call_interrupts
5150715 ? 20% +29.7% 6678194 ? 5% interrupts.CPU185.CAL:Function_call_interrupts
4503281 ? 8% +59.8% 7197423 ? 13% interrupts.CPU187.CAL:Function_call_interrupts
4827738 ? 14% +26.0% 6081651 ? 16% interrupts.CPU190.CAL:Function_call_interrupts
4726248 ? 8% -23.2% 3631004 ? 20% interrupts.CPU2.CAL:Function_call_interrupts
2067404 ? 9% -17.1% 1713473 ? 11% interrupts.CPU2.RES:Rescheduling_interrupts
1712460 ? 7% +19.9% 2053835 interrupts.CPU22.RES:Rescheduling_interrupts
3238643 ? 22% +27.6% 4131745 ? 12% interrupts.CPU29.CAL:Function_call_interrupts
1607899 ? 4% +28.7% 2069474 ? 14% interrupts.CPU29.RES:Rescheduling_interrupts
3601982 ? 8% +15.5% 4159215 ? 10% interrupts.CPU31.CAL:Function_call_interrupts
3801797 ? 4% +18.1% 4488433 ? 12% interrupts.CPU32.CAL:Function_call_interrupts
3374468 ? 10% +23.8% 4176507 ? 2% interrupts.CPU33.CAL:Function_call_interrupts
1753123 ? 4% +13.8% 1994342 ? 6% interrupts.CPU33.RES:Rescheduling_interrupts
1901796 +14.2% 2171182 ? 4% interrupts.CPU35.RES:Rescheduling_interrupts
2424585 ? 8% -16.6% 2022154 ? 2% interrupts.CPU45.RES:Rescheduling_interrupts
3750015 ? 14% +29.2% 4843627 ? 15% interrupts.CPU46.CAL:Function_call_interrupts
5556270 ? 21% -36.8% 3509724 ? 28% interrupts.CPU48.CAL:Function_call_interrupts
5711601 ? 15% -33.8% 3780223 ? 22% interrupts.CPU49.CAL:Function_call_interrupts
1613828 ? 18% +32.2% 2133584 ? 12% interrupts.CPU49.RES:Rescheduling_interrupts
1457715 ? 18% +43.2% 2087070 interrupts.CPU52.RES:Rescheduling_interrupts
1789983 ? 10% +23.7% 2214022 ? 16% interrupts.CPU54.RES:Rescheduling_interrupts
5662000 ? 26% -37.4% 3542511 ? 13% interrupts.CPU57.CAL:Function_call_interrupts
5269193 ? 12% -28.0% 3796389 ? 22% interrupts.CPU60.CAL:Function_call_interrupts
2215937 ? 10% -15.8% 1866080 ? 5% interrupts.CPU60.RES:Rescheduling_interrupts
4959625 ? 2% -22.4% 3846220 ? 17% interrupts.CPU63.CAL:Function_call_interrupts
2357629 ? 13% -27.5% 1708837 ? 18% interrupts.CPU63.RES:Rescheduling_interrupts
5291221 ? 10% -24.8% 3977451 ? 8% interrupts.CPU65.CAL:Function_call_interrupts
1630238 ? 26% +42.5% 2322835 ? 8% interrupts.CPU68.RES:Rescheduling_interrupts
5404145 ? 5% -25.2% 4044518 ? 18% interrupts.CPU69.CAL:Function_call_interrupts
4311698 ? 5% +43.0% 6164696 ? 13% interrupts.CPU72.CAL:Function_call_interrupts
3677831 ? 9% +70.6% 6273317 ? 29% interrupts.CPU73.CAL:Function_call_interrupts
3612623 ? 10% +54.6% 5583789 ? 22% interrupts.CPU74.CAL:Function_call_interrupts
3856325 ? 4% +51.3% 5834147 ? 15% interrupts.CPU75.CAL:Function_call_interrupts
1645387 ? 3% +27.5% 2098281 ? 14% interrupts.CPU75.RES:Rescheduling_interrupts
3739398 ? 11% +66.8% 6238004 ? 29% interrupts.CPU76.CAL:Function_call_interrupts
1598254 ? 19% +29.3% 2066964 ? 9% interrupts.CPU77.RES:Rescheduling_interrupts
3350555 ? 11% +52.9% 5124400 ? 15% interrupts.CPU80.CAL:Function_call_interrupts
3574937 ? 12% +46.1% 5224402 ? 8% interrupts.CPU81.CAL:Function_call_interrupts
3881925 ? 20% +38.7% 5385256 ? 23% interrupts.CPU82.CAL:Function_call_interrupts
3614411 ? 14% +65.7% 5990688 ? 16% interrupts.CPU83.CAL:Function_call_interrupts
3972826 ? 16% +31.4% 5218713 ? 6% interrupts.CPU84.CAL:Function_call_interrupts
4000216 ? 6% +52.0% 6079848 ? 24% interrupts.CPU85.CAL:Function_call_interrupts
3764192 ? 11% +42.2% 5353364 ? 17% interrupts.CPU86.CAL:Function_call_interrupts
3629453 ? 2% +54.7% 5613811 ? 20% interrupts.CPU87.CAL:Function_call_interrupts
3678632 ? 10% +70.2% 6260271 ? 26% interrupts.CPU88.CAL:Function_call_interrupts
3830790 ? 11% +63.4% 6260528 ? 25% interrupts.CPU90.CAL:Function_call_interrupts
5784 ? 35% +50.4% 8699 interrupts.CPU91.NMI:Non-maskable_interrupts
5784 ? 35% +50.4% 8699 interrupts.CPU91.PMI:Performance_monitoring_interrupts
3341401 ? 7% +57.7% 5268798 ? 8% interrupts.CPU92.CAL:Function_call_interrupts
3696549 ? 3% +86.3% 6885440 ? 27% interrupts.CPU93.CAL:Function_call_interrupts
5786 ? 35% +50.3% 8698 interrupts.CPU93.NMI:Non-maskable_interrupts
5786 ? 35% +50.3% 8698 interrupts.CPU93.PMI:Performance_monitoring_interrupts
4199119 ? 8% +39.0% 5835061 ? 10% interrupts.CPU94.CAL:Function_call_interrupts
3820035 ? 6% +57.8% 6027549 ? 28% interrupts.CPU95.CAL:Function_call_interrupts
5790 ? 35% +50.1% 8694 interrupts.CPU95.NMI:Non-maskable_interrupts
5790 ? 35% +50.1% 8694 interrupts.CPU95.PMI:Performance_monitoring_interrupts
1808381 ? 6% +26.4% 2286065 ? 11% interrupts.CPU95.RES:Rescheduling_interrupts
99695 ? 8% +27.7% 127348 ? 12% softirqs.CPU0.RCU
102168 ? 7% +26.6% 129395 ? 11% softirqs.CPU1.RCU
101438 ? 6% +25.1% 126885 ? 10% softirqs.CPU10.RCU
101352 ? 7% +24.3% 126007 ? 10% softirqs.CPU100.RCU
101061 ? 8% +24.9% 126249 ? 11% softirqs.CPU102.RCU
100479 ? 8% +25.1% 125672 ? 10% softirqs.CPU103.RCU
101891 ? 7% +23.5% 125847 ? 11% softirqs.CPU104.RCU
100756 ? 7% +24.3% 125247 ? 10% softirqs.CPU105.RCU
100557 ? 7% +23.9% 124600 ? 10% softirqs.CPU106.RCU
100714 ? 7% +25.7% 126571 ? 10% softirqs.CPU107.RCU
101251 ? 7% +24.3% 125831 ? 10% softirqs.CPU108.RCU
100308 ? 7% +25.5% 125861 ? 10% softirqs.CPU109.RCU
5798 ? 68% +429.4% 30699 ? 46% softirqs.CPU11.NET_RX
101135 ? 7% +27.1% 128567 ? 10% softirqs.CPU11.RCU
100161 ? 7% +24.2% 124410 ? 10% softirqs.CPU110.RCU
98850 ? 11% +27.5% 126012 ? 10% softirqs.CPU111.RCU
104203 ? 6% +26.4% 131712 ? 11% softirqs.CPU112.RCU
101400 ? 5% +29.1% 130884 ? 10% softirqs.CPU113.RCU
103374 ? 6% +20.8% 124837 ? 8% softirqs.CPU114.RCU
104312 ? 6% +24.8% 130182 ? 11% softirqs.CPU115.RCU
100093 ? 6% +28.3% 128418 ? 14% softirqs.CPU116.RCU
103618 ? 7% +25.7% 130214 ? 11% softirqs.CPU117.RCU
103544 ? 8% +26.1% 130595 ? 11% softirqs.CPU118.RCU
104245 ? 6% +26.6% 131973 ? 11% softirqs.CPU119.RCU
102211 ? 7% +25.7% 128480 ? 11% softirqs.CPU12.RCU
104976 ? 4% +28.1% 134429 ? 9% softirqs.CPU120.RCU
105246 ? 4% +28.3% 135026 ? 9% softirqs.CPU121.RCU
105073 ? 4% +27.3% 133799 ? 9% softirqs.CPU122.RCU
105312 ? 4% +26.5% 133244 ? 9% softirqs.CPU123.RCU
104797 ? 4% +27.6% 133755 ? 8% softirqs.CPU124.RCU
104474 ? 4% +27.4% 133097 ? 8% softirqs.CPU125.RCU
104492 ? 4% +29.2% 135051 ? 9% softirqs.CPU126.RCU
105827 ? 2% +26.9% 134277 ? 9% softirqs.CPU127.RCU
101849 ? 3% +27.4% 129782 ? 7% softirqs.CPU128.RCU
101012 ? 3% +28.8% 130053 ? 7% softirqs.CPU129.RCU
2774 ? 40% +174.4% 7614 ? 57% softirqs.CPU13.NET_RX
101485 ? 7% +25.3% 127149 ? 10% softirqs.CPU13.RCU
100533 ? 3% +27.2% 127897 ? 7% softirqs.CPU130.RCU
101081 ? 3% +27.8% 129222 ? 7% softirqs.CPU131.RCU
101450 ? 3% +28.4% 130217 ? 7% softirqs.CPU132.RCU
101700 ? 3% +27.7% 129857 ? 7% softirqs.CPU133.RCU
100851 ? 3% +27.5% 128605 ? 8% softirqs.CPU134.RCU
101043 ? 4% +27.1% 128377 ? 7% softirqs.CPU135.RCU
101163 ? 3% +27.8% 129269 ? 7% softirqs.CPU136.RCU
101350 ? 3% +28.8% 130556 ? 8% softirqs.CPU137.RCU
101569 ? 4% +28.2% 130187 ? 7% softirqs.CPU138.RCU
103024 ? 3% +26.8% 130598 ? 7% softirqs.CPU139.RCU
101217 ? 6% +26.6% 128113 ? 10% softirqs.CPU14.RCU
101171 ? 3% +28.0% 129543 ? 8% softirqs.CPU140.RCU
100983 ? 3% +27.6% 128904 ? 8% softirqs.CPU141.RCU
100777 ? 4% +27.7% 128651 ? 8% softirqs.CPU142.RCU
101548 ? 3% +26.1% 128069 ? 6% softirqs.CPU143.RCU
102292 ? 4% +28.7% 131622 ? 7% softirqs.CPU144.RCU
101796 ? 3% +29.3% 131646 ? 8% softirqs.CPU145.RCU
101098 ? 3% +28.6% 130051 ? 8% softirqs.CPU146.RCU
101267 ? 3% +28.2% 129816 ? 8% softirqs.CPU147.RCU
101847 ? 4% +27.7% 130042 ? 8% softirqs.CPU148.RCU
100743 ? 3% +28.3% 129222 ? 7% softirqs.CPU149.RCU
98982 ? 10% +28.7% 127411 ? 10% softirqs.CPU15.RCU
100636 ? 4% +29.0% 129827 ? 8% softirqs.CPU150.RCU
101113 ? 5% +30.0% 131420 ? 8% softirqs.CPU151.RCU
101505 ? 4% +29.0% 130930 ? 7% softirqs.CPU152.RCU
100566 ? 4% +30.3% 131071 ? 8% softirqs.CPU153.RCU
101747 ? 5% +27.9% 130127 ? 7% softirqs.CPU154.RCU
99882 ? 3% +28.1% 127990 ? 8% softirqs.CPU155.RCU
102357 ? 4% +28.3% 131323 ? 7% softirqs.CPU156.RCU
101490 ? 4% +29.4% 131298 ? 8% softirqs.CPU157.RCU
101391 ? 4% +28.5% 130291 ? 7% softirqs.CPU158.RCU
101583 ? 3% +28.3% 130345 ? 8% softirqs.CPU159.RCU
104519 ? 6% +25.9% 131595 ? 11% softirqs.CPU16.RCU
105238 ? 3% +30.4% 137272 ? 8% softirqs.CPU160.RCU
105282 ? 4% +29.8% 136612 ? 9% softirqs.CPU161.RCU
106084 ? 4% +28.8% 136592 ? 9% softirqs.CPU162.RCU
105736 ? 4% +27.8% 135088 ? 9% softirqs.CPU163.RCU
104681 ? 4% +30.2% 136271 ? 8% softirqs.CPU164.RCU
105572 ? 3% +27.8% 134963 ? 8% softirqs.CPU165.RCU
105612 ? 4% +29.6% 136902 ? 8% softirqs.CPU166.RCU
106257 ? 3% +28.6% 136631 ? 9% softirqs.CPU167.RCU
102940 ? 3% +18.9% 122351 ? 5% softirqs.CPU168.RCU
102433 ? 4% +19.3% 122208 ? 5% softirqs.CPU169.RCU
101963 ? 4% +28.2% 130705 ? 10% softirqs.CPU17.RCU
103129 ? 4% +19.5% 123216 ? 5% softirqs.CPU170.RCU
102988 ? 3% +19.9% 123495 ? 4% softirqs.CPU171.RCU
102638 ? 3% +19.8% 122960 ? 5% softirqs.CPU172.RCU
102750 ? 4% +20.0% 123326 ? 4% softirqs.CPU173.RCU
103262 ? 3% +19.5% 123363 ? 6% softirqs.CPU174.RCU
102636 ? 4% +19.1% 122231 ? 4% softirqs.CPU175.RCU
103507 ? 4% +14.2% 118205 ? 3% softirqs.CPU176.RCU
102388 ? 3% +14.7% 117390 ? 3% softirqs.CPU177.RCU
102423 ? 3% +15.3% 118067 ? 4% softirqs.CPU179.RCU
103346 ? 6% +23.0% 127133 ? 8% softirqs.CPU18.RCU
103193 ? 3% +15.3% 118961 ? 3% softirqs.CPU181.RCU
103182 ? 3% +14.9% 118546 ? 4% softirqs.CPU182.RCU
102714 ? 4% +15.2% 118323 ? 4% softirqs.CPU183.RCU
103227 ? 3% +14.8% 118493 ? 3% softirqs.CPU184.RCU
102946 ? 4% +15.3% 118676 ? 3% softirqs.CPU186.RCU
102318 ? 4% +15.6% 118299 ? 3% softirqs.CPU188.RCU
104098 ? 6% +24.7% 129778 ? 10% softirqs.CPU19.RCU
102541 ? 3% +15.0% 117893 ? 3% softirqs.CPU190.RCU
101042 ? 7% +28.0% 129341 ? 10% softirqs.CPU2.RCU
101819 ? 5% +26.2% 128473 ? 13% softirqs.CPU20.RCU
103710 ? 5% +26.2% 130919 ? 11% softirqs.CPU21.RCU
103462 ? 6% +26.2% 130556 ? 11% softirqs.CPU22.RCU
104698 ? 6% +23.6% 129449 ? 11% softirqs.CPU23.RCU
105912 ? 4% +28.9% 136526 ? 9% softirqs.CPU24.RCU
106915 ? 5% +28.0% 136862 ? 9% softirqs.CPU25.RCU
108194 ? 5% +25.2% 135427 ? 8% softirqs.CPU26.RCU
106435 ? 5% +26.4% 134565 ? 9% softirqs.CPU27.RCU
106432 ? 5% +27.1% 135282 ? 9% softirqs.CPU28.RCU
106762 ? 5% +26.4% 134947 ? 9% softirqs.CPU29.RCU
101073 ? 7% +26.0% 127315 ? 11% softirqs.CPU3.RCU
106558 ? 4% +28.3% 136717 ? 9% softirqs.CPU30.RCU
107555 ? 5% +26.0% 135530 ? 10% softirqs.CPU31.RCU
103366 ? 3% +28.3% 132635 ? 8% softirqs.CPU32.RCU
102621 ? 4% +28.4% 131744 ? 6% softirqs.CPU33.RCU
102563 ? 3% +26.9% 130144 ? 7% softirqs.CPU34.RCU
102718 ? 3% +26.7% 130101 ? 7% softirqs.CPU35.RCU
102256 ? 3% +28.2% 131054 ? 7% softirqs.CPU36.RCU
102832 ? 2% +28.2% 131813 ? 7% softirqs.CPU37.RCU
103243 ? 3% +26.9% 131040 ? 8% softirqs.CPU38.RCU
103107 ? 3% +27.8% 131771 ? 7% softirqs.CPU39.RCU
100677 ? 7% +26.5% 127350 ? 10% softirqs.CPU4.RCU
102675 ? 3% +27.9% 131327 ? 7% softirqs.CPU40.RCU
102932 ? 3% +27.3% 131012 ? 8% softirqs.CPU41.RCU
101673 ? 3% +27.3% 129440 ? 7% softirqs.CPU42.RCU
103102 ? 3% +28.0% 131926 ? 8% softirqs.CPU43.RCU
102486 ? 3% +27.9% 131040 ? 8% softirqs.CPU44.RCU
102649 ? 3% +26.9% 130231 ? 8% softirqs.CPU45.RCU
101143 ? 3% +28.5% 129932 ? 8% softirqs.CPU46.RCU
101886 ? 3% +25.0% 127395 ? 6% softirqs.CPU47.RCU
102679 ? 3% +28.8% 132211 ? 8% softirqs.CPU48.RCU
102383 ? 4% +29.1% 132151 ? 8% softirqs.CPU49.RCU
101739 ? 4% +29.5% 131737 ? 9% softirqs.CPU50.RCU
101851 ? 3% +28.2% 130554 ? 8% softirqs.CPU51.RCU
100790 ? 4% +28.6% 129635 ? 8% softirqs.CPU52.RCU
102025 ? 3% +29.5% 132075 ? 7% softirqs.CPU53.RCU
101675 ? 4% +29.0% 131203 ? 7% softirqs.CPU54.RCU
102229 ? 4% +28.5% 131337 ? 8% softirqs.CPU55.RCU
101345 ? 4% +28.8% 130535 ? 8% softirqs.CPU56.RCU
101302 ? 3% +29.9% 131631 ? 8% softirqs.CPU57.RCU
102543 ? 4% +26.3% 129467 ? 6% softirqs.CPU58.RCU
101876 ? 3% +28.0% 130376 ? 8% softirqs.CPU59.RCU
100954 ? 7% +26.2% 127397 ? 11% softirqs.CPU6.RCU
102808 ? 3% +28.6% 132232 ? 8% softirqs.CPU60.RCU
102230 ? 4% +28.0% 130855 ? 9% softirqs.CPU61.RCU
101560 ? 3% +27.6% 129585 ? 8% softirqs.CPU62.RCU
101792 ? 4% +28.0% 130286 ? 9% softirqs.CPU63.RCU
105773 ? 3% +30.0% 137481 ? 8% softirqs.CPU64.RCU
105212 ? 4% +29.7% 136426 ? 9% softirqs.CPU65.RCU
107146 ? 3% +29.6% 138847 ? 9% softirqs.CPU66.RCU
106932 ? 3% +30.1% 139150 ? 9% softirqs.CPU67.RCU
106536 ? 3% +29.8% 138241 ? 8% softirqs.CPU68.RCU
105525 ? 3% +30.4% 137607 ? 9% softirqs.CPU69.RCU
101492 ? 7% +25.4% 127288 ? 10% softirqs.CPU7.RCU
105803 ? 3% +29.1% 136623 ? 9% softirqs.CPU70.RCU
105655 ? 4% +30.5% 137861 ? 9% softirqs.CPU71.RCU
103655 ? 3% +21.4% 125865 ? 6% softirqs.CPU72.RCU
103901 ? 4% +20.2% 124850 ? 6% softirqs.CPU73.RCU
103789 ? 4% +21.1% 125669 ? 5% softirqs.CPU74.RCU
103170 ? 3% +21.9% 125769 ? 5% softirqs.CPU75.RCU
103156 ? 3% +21.3% 125098 ? 5% softirqs.CPU76.RCU
102168 ? 3% +22.5% 125114 ? 5% softirqs.CPU77.RCU
102852 ? 3% +20.8% 124219 ? 5% softirqs.CPU78.RCU
103777 ? 4% +20.8% 125375 ? 5% softirqs.CPU79.RCU
101313 ? 7% +26.6% 128252 ? 12% softirqs.CPU8.RCU
102620 ? 4% +15.7% 118694 ? 4% softirqs.CPU80.RCU
101557 ? 3% +15.8% 117652 ? 4% softirqs.CPU81.RCU
101501 ? 4% +16.0% 117701 ? 3% softirqs.CPU82.RCU
102729 ? 4% +15.2% 118338 ? 4% softirqs.CPU83.RCU
101540 ? 3% +16.5% 118250 ? 4% softirqs.CPU86.RCU
101607 ? 3% +15.5% 117332 ? 4% softirqs.CPU87.RCU
102737 ? 4% +15.2% 118336 ? 3% softirqs.CPU89.RCU
100455 ? 7% +23.6% 124154 ? 8% softirqs.CPU9.RCU
102382 ? 4% +15.5% 118301 ? 4% softirqs.CPU90.RCU
102257 ? 4% +17.7% 120341 ? 3% softirqs.CPU92.RCU
101809 ? 4% +15.9% 118027 ? 4% softirqs.CPU93.RCU
5986 ? 5% +36.4% 8162 ? 18% softirqs.CPU93.SCHED
102201 ? 3% +16.1% 118678 ? 4% softirqs.CPU94.RCU
102079 ? 4% +15.5% 117850 ? 3% softirqs.CPU95.RCU
99893 ? 7% +26.9% 126781 ? 10% softirqs.CPU96.RCU
101943 ? 8% +22.4% 124784 ? 10% softirqs.CPU97.RCU
100354 ? 7% +25.6% 126020 ? 11% softirqs.CPU98.RCU
101113 ? 7% +25.3% 126708 ? 11% softirqs.CPU99.RCU
16498 ? 32% +250.1% 57762 ? 20% softirqs.NET_RX
19715004 ? 4% +24.7% 24582787 ? 7% softirqs.RCU
1246365 ? 2% +12.5% 1401902 ? 4% softirqs.SCHED
126945 ? 2% +18.0% 149827 ? 5% softirqs.TIMER
***************************************************************************************************
lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-8.3/30%/debian-10.4-x86_64-20200603.cgz/300s/lkp-cfl-e1/shell1/unixbench/0xde
commit:
bf4f059954 ("mm: memcg/slab: obj_cgroup API")
286e04b8ed ("mm: memcg/slab: allocate obj_cgroups for non-root slab pages")
bf4f059954dcb221 286e04b8ed7a04279ae277f0f02
---------------- ---------------------------
%stddev %change %stddev
\ | \
1.372e+08 ? 38% -43.9% 77022150 ? 12% cpuidle.C1.time
1933835 ? 9% -12.9% 1683420 ? 9% cpuidle.C1.usage
49.39 +8.1% 53.40 ? 3% perf-stat.i.MPKI
2.46 +0.2 2.62 perf-stat.i.branch-miss-rate%
922.24 ? 9% +48.1% 1365 ? 36% sched_debug.cfs_rq:/.load_avg.max
257.37 ? 7% +46.4% 376.78 ? 35% sched_debug.cfs_rq:/.load_avg.stddev
1127 ? 5% -8.8% 1027 ? 5% sched_debug.cfs_rq:/.runnable_avg.max
1120 ? 5% -9.0% 1019 ? 5% sched_debug.cfs_rq:/.util_avg.max
0.00 ? 9% -23.0% 0.00 ? 20% sched_debug.cpu.next_balance.stddev
0.59 ? 3% +0.1 0.64 ? 2% perf-profile.calltrace.cycles-pp.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64
6.20 ? 6% +0.3 6.55 ? 6% perf-profile.calltrace.cycles-pp.asm_exc_page_fault
0.00 +0.5 0.52 perf-profile.calltrace.cycles-pp.__vma_adjust.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect
0.18 ? 6% -0.0 0.15 ? 5% perf-profile.children.cycles-pp.security_mmap_file
0.17 ? 6% -0.0 0.15 ? 11% perf-profile.children.cycles-pp.anon_vma_clone
0.12 ? 8% -0.0 0.10 ? 9% perf-profile.children.cycles-pp.__put_user_8
0.13 ? 7% -0.0 0.10 ? 4% perf-profile.children.cycles-pp.cp_new_stat
0.12 ? 9% -0.0 0.10 ? 4% perf-profile.children.cycles-pp.up_read
0.09 ? 5% -0.0 0.07 ? 6% perf-profile.children.cycles-pp.close@plt
0.24 ? 3% +0.0 0.26 ? 3% perf-profile.children.cycles-pp.open64
0.18 ? 5% +0.0 0.20 ? 6% perf-profile.children.cycles-pp.wmemchr
0.37 ? 5% +0.1 0.43 ? 2% perf-profile.children.cycles-pp.__close
0.96 ? 2% +0.1 1.04 ? 5% perf-profile.children.cycles-pp.__vma_adjust
0.12 ? 9% -0.0 0.09 ? 5% perf-profile.self.cycles-pp.up_read
0.22 ? 5% -0.0 0.20 ? 7% perf-profile.self.cycles-pp.mark_page_accessed
0.10 ? 10% -0.0 0.08 ? 10% perf-profile.self.cycles-pp.__put_user_8
0.10 ? 11% -0.0 0.08 ? 10% perf-profile.self.cycles-pp.do_fault
0.02 ?141% +0.0 0.06 ? 8% perf-profile.self.cycles-pp.acct_collect
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation
Thanks,
Oliver Sang