2023-07-19 14:18:49

by kernel test robot

[permalink] [raw]
Subject: [linus:master] [x86/fpu] b81fac906a: stress-ng.af-alg.ops_per_sec -34.6% regression



Hello,

kernel test robot noticed a -34.6% regression of stress-ng.af-alg.ops_per_sec on:


commit: b81fac906a8f9e682e513ddd95697ec7a20878d4 ("x86/fpu: Move FPU initialization into arch_cpu_finalize_init()")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

still regression on:
fix commit: fe3e0a13e597c1c8617814bf9b42ab732db5c26e
linus/master: ccff6d117d8dc8d8d86e8695a75e5f8b01e573bf

testcase: stress-ng
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:

nr_threads: 10%
disk: 1HDD
testtime: 60s
fs: ext4
class: os
test: af-alg
cpufreq_governor: performance


besides, we noticed there are below information in dmesg (attached kmsg.xz),
which we didn't see in dmesg for parent:

kern :info : [ 65.660392] AVX or AES-NI instructions are not detected.
user :notice: [ 65.661743] stress-ng: info: [4738] af-alg: 9 cryptographic algorithms are internal and may be unused

kern :info : [ 65.720205] AVX or AES-NI instructions are not detected.
kern :info : [ 65.820334] AVX2 instructions are not detected.
kern :info : [ 66.130644] AVX or AES-NI instructions are not detected.
kern :info : [ 66.167749] AVX or AES-NI instructions are not detected.
kern :info : [ 66.213178] AVX2 instructions are not detected.
kern :info : [ 66.318235] AVX or AES-NI instructions are not detected.
kern :info : [ 66.360766] AVX or AES-NI instructions are not detected.
kern :info : [ 66.437318] AVX2 instructions are not detected.
kern :info : [ 66.683666] AVX or AES-NI instructions are not detected.
user :notice: [ 66.713976] BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled

kern :info : [ 66.738790] AVX or AES-NI instructions are not detected.
kern :info : [ 66.822332] AVX2 instructions are not detected.
kern :info : [ 67.083360] AVX or AES-NI instructions are not detected.
kern :info : [ 67.122804] AVX or AES-NI instructions are not detected.



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
sudo bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
sudo bin/lkp run generated-yaml-file

# if come across any failure that blocks the test,
# please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
os/gcc-12/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp7/af-alg/stress-ng/60s

commit:
1703db2b90 ("x86/fpu: Mark init functions __init")
b81fac906a ("x86/fpu: Move FPU initialization into arch_cpu_finalize_init()")

1703db2b90c91b2e b81fac906a8f9e682e513ddd956
---------------- ---------------------------
%stddev %change %stddev
\ | \
4453936 +17.2% 5218922 cpuidle..usage
475.17 ? 9% -39.4% 288.00 ? 4% perf-c2c.HITM.local
68421 ? 6% -68.2% 21789 ?115% numa-meminfo.node1.AnonHugePages
397537 ? 11% -30.9% 274577 ? 25% numa-meminfo.node1.AnonPages.max
11275439 ? 8% -40.6% 6702239 ? 12% numa-numastat.node0.local_node
11307159 ? 8% -40.5% 6726387 ? 12% numa-numastat.node0.numa_hit
11307295 ? 8% -40.5% 6726432 ? 12% numa-vmstat.node0.numa_hit
11275576 ? 8% -40.6% 6702284 ? 12% numa-vmstat.node0.numa_local
90.28 +2.6% 92.63 iostat.cpu.idle
6.74 -23.4% 5.16 iostat.cpu.system
2.95 -26.4% 2.17 ? 2% iostat.cpu.user
0.12 -0.0 0.10 mpstat.cpu.all.soft%
5.78 -1.7 4.12 mpstat.cpu.all.sys%
3.02 -0.8 2.22 ? 2% mpstat.cpu.all.usr%
5.67 ? 8% -29.4% 4.00 vmstat.procs.r
27980 -29.5% 19716 vmstat.system.cs
69859 +18.8% 83015 vmstat.system.in
137449 -21.7% 107685 meminfo.Active
135863 -21.9% 106091 meminfo.Active(anon)
538488 ? 5% +52.9% 823250 ? 4% meminfo.DirectMap4k
78913 ? 2% -14.6% 67370 meminfo.Mapped
181011 -22.4% 140469 meminfo.Shmem
90999 -34.6% 59547 stress-ng.af-alg.ops
1516 -34.6% 992.18 stress-ng.af-alg.ops_per_sec
87.83 -32.8% 59.00 stress-ng.time.percent_of_cpu_this_job_got
52.14 -32.7% 35.10 stress-ng.time.system_time
175731 -34.6% 114964 stress-ng.time.voluntary_context_switches
317.50 -22.4% 246.33 turbostat.Avg_MHz
10.35 -2.1 8.23 turbostat.Busy%
3073 -2.4% 2999 turbostat.Bzy_MHz
14183 +20.3% 17060 ? 4% turbostat.C1
783741 ? 2% +124.3% 1758028 turbostat.C1E
7.98 ? 2% +8.2 16.21 turbostat.C1E%
4538585 +19.2% 5411831 turbostat.IRQ
30452 ? 2% -62.3% 11490 ? 9% turbostat.POLL
157.35 -3.5% 151.83 turbostat.PkgWatt
33967 -21.9% 26523 proc-vmstat.nr_active_anon
89639 -1.3% 88437 proc-vmstat.nr_anon_pages
748728 -1.3% 738629 proc-vmstat.nr_file_pages
101000 -4.0% 96980 proc-vmstat.nr_inactive_anon
19752 ? 2% -14.7% 16856 proc-vmstat.nr_mapped
45267 -22.4% 35122 proc-vmstat.nr_shmem
37642 -1.1% 37241 proc-vmstat.nr_slab_unreclaimable
33967 -21.9% 26523 proc-vmstat.nr_zone_active_anon
101000 -4.0% 96980 proc-vmstat.nr_zone_inactive_anon
21330465 -30.8% 14764467 proc-vmstat.numa_hit
21264274 -30.9% 14698273 proc-vmstat.numa_local
74767 ? 8% -26.1% 55256 ? 23% proc-vmstat.numa_pte_updates
77297 -21.4% 60741 proc-vmstat.pgactivate
21716615 -30.7% 15050540 proc-vmstat.pgalloc_normal
24651421 -31.0% 17003260 proc-vmstat.pgfault
21608074 -30.8% 14960607 proc-vmstat.pgfree
576512 +309.8% 2362624 proc-vmstat.unevictable_pgs_scanned
2.657e+09 -28.1% 1.91e+09 perf-stat.i.branch-instructions
55105518 -27.9% 39728424 perf-stat.i.branch-misses
5.80 ? 2% +1.3 7.13 ? 2% perf-stat.i.cache-miss-rate%
6856942 ? 3% -9.4% 6209194 ? 2% perf-stat.i.cache-misses
1.314e+08 -28.1% 94523874 perf-stat.i.cache-references
28932 -29.8% 20310 perf-stat.i.context-switches
1.16 +7.1% 1.24 perf-stat.i.cpi
1.969e+10 -22.9% 1.519e+10 perf-stat.i.cpu-cycles
649.36 ? 2% -37.1% 408.14 ? 2% perf-stat.i.cpu-migrations
3018 ? 2% -15.6% 2548 perf-stat.i.cycles-between-cache-misses
0.04 ? 2% +0.0 0.06 ? 3% perf-stat.i.dTLB-load-miss-rate%
1941644 +8.1% 2099200 ? 2% perf-stat.i.dTLB-load-misses
4.69e+09 -29.1% 3.326e+09 perf-stat.i.dTLB-loads
0.12 +0.0 0.13 perf-stat.i.dTLB-store-miss-rate%
2651571 -26.7% 1942815 perf-stat.i.dTLB-store-misses
2.113e+09 -29.5% 1.49e+09 perf-stat.i.dTLB-stores
1.698e+10 -27.7% 1.227e+10 perf-stat.i.instructions
0.87 -6.4% 0.81 perf-stat.i.ipc
2035 ? 3% -61.7% 778.70 ? 9% perf-stat.i.major-faults
0.31 -22.9% 0.24 perf-stat.i.metric.GHz
167.15 ? 2% -13.2% 145.14 ? 3% perf-stat.i.metric.K/sec
149.81 -28.9% 106.53 perf-stat.i.metric.M/sec
388617 -31.0% 268024 perf-stat.i.minor-faults
3246190 ? 7% -23.5% 2483951 ? 4% perf-stat.i.node-load-misses
32.23 ? 2% +6.0 38.26 ? 2% perf-stat.i.node-store-miss-rate%
886521 ? 3% +24.8% 1106439 ? 3% perf-stat.i.node-store-misses
1968660 ? 2% -8.4% 1802894 perf-stat.i.node-stores
390652 -31.2% 268802 perf-stat.i.page-faults
5.24 ? 2% +1.3 6.59 perf-stat.overall.cache-miss-rate%
1.16 +6.7% 1.24 perf-stat.overall.cpi
2863 ? 2% -14.8% 2440 perf-stat.overall.cycles-between-cache-misses
0.04 ? 2% +0.0 0.06 ? 3% perf-stat.overall.dTLB-load-miss-rate%
0.13 +0.0 0.13 perf-stat.overall.dTLB-store-miss-rate%
0.86 -6.3% 0.81 perf-stat.overall.ipc
30.85 ? 2% +7.0 37.89 ? 2% perf-stat.overall.node-store-miss-rate%
2.617e+09 -28.1% 1.881e+09 perf-stat.ps.branch-instructions
54225753 -27.9% 39076523 perf-stat.ps.branch-misses
6779427 ? 3% -9.6% 6127594 ? 2% perf-stat.ps.cache-misses
1.294e+08 -28.1% 93009393 perf-stat.ps.cache-references
28483 -29.8% 19996 perf-stat.ps.context-switches
1.94e+10 -22.9% 1.495e+10 perf-stat.ps.cpu-cycles
638.69 ? 2% -37.1% 401.75 ? 2% perf-stat.ps.cpu-migrations
1913060 +8.0% 2066590 ? 2% perf-stat.ps.dTLB-load-misses
4.62e+09 -29.1% 3.275e+09 perf-stat.ps.dTLB-loads
2610836 -26.8% 1912030 perf-stat.ps.dTLB-store-misses
2.081e+09 -29.5% 1.468e+09 perf-stat.ps.dTLB-stores
1.673e+10 -27.8% 1.208e+10 perf-stat.ps.instructions
2005 ? 3% -61.8% 766.73 ? 9% perf-stat.ps.major-faults
382594 -31.1% 263761 perf-stat.ps.minor-faults
3194487 ? 7% -23.5% 2443082 ? 4% perf-stat.ps.node-load-misses
872822 ? 3% +24.7% 1088214 ? 2% perf-stat.ps.node-store-misses
1956457 ? 2% -8.8% 1783576 perf-stat.ps.node-stores
384599 -31.2% 264527 perf-stat.ps.page-faults
1.046e+12 -27.3% 7.603e+11 perf-stat.total.instructions
0.00 ? 20% -72.7% 0.00 ?141% perf-sched.sch_delay.avg.ms.__cond_resched.async_synchronize_cookie_domain.call_usermodehelper_exec_async.ret_from_fork
0.00 ? 19% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.dput.step_into.path_lookupat.filename_lookup
0.00 ? 17% +84.6% 0.00 ? 20% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
0.01 ? 7% +375.3% 0.07 ? 11% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ? 20% +52.0% 0.01 ? 5% perf-sched.sch_delay.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read
0.01 ? 7% +29.3% 0.01 ? 3% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
0.00 ? 20% -72.7% 0.00 ?141% perf-sched.sch_delay.max.ms.__cond_resched.async_synchronize_cookie_domain.call_usermodehelper_exec_async.ret_from_fork
0.00 ? 11% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.dput.step_into.path_lookupat.filename_lookup
0.02 ? 29% -78.2% 0.00 ? 89% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.prepare_kernel_cred.call_usermodehelper_exec_async.ret_from_fork
0.01 ? 38% -55.6% 0.00 ? 14% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock_killable.pcpu_alloc.__percpu_counter_init.mm_init
0.02 ? 82% -64.2% 0.01 ? 60% perf-sched.sch_delay.max.ms.__cond_resched.unmap_vmas.unmap_region.do_vmi_align_munmap.do_vmi_munmap
0.02 ? 27% +67.6% 0.04 ? 19% perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.03 ? 44% -58.0% 0.01 ? 8% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
0.00 ?223% +900.0% 0.02 ? 29% perf-sched.sch_delay.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
0.02 ? 23% +72.0% 0.03 ? 20% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ? 31% -83.9% 0.00 ? 99% perf-sched.sch_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma
4.04 ? 46% +382.2% 19.47 ? 2% perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
0.01 ? 35% +223.4% 0.03 ? 13% perf-sched.sch_delay.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read
4.20 ? 17% +60.4% 6.74 ? 18% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
0.01 ? 6% +108.1% 0.01 ? 6% perf-sched.total_sch_delay.average.ms
5.58 ? 10% +249.1% 19.47 ? 2% perf-sched.total_sch_delay.max.ms
13.94 +47.6% 20.58 perf-sched.total_wait_and_delay.average.ms
79638 -29.4% 56254 perf-sched.total_wait_and_delay.count.ms
13.94 +47.6% 20.57 perf-sched.total_wait_time.average.ms
0.00 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
2.08 ? 2% +51.4% 3.15 ? 2% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
384.28 -30.6% 266.53 ? 2% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
298.54 ? 13% +63.3% 487.41 ? 18% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
368.03 ? 5% -80.2% 72.93 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
1.48 ? 8% -19.7% 1.19 ? 6% perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
1.82 +61.9% 2.95 ? 2% perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
78.21 ? 3% +32.4% 103.56 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
14.85 ? 2% +37.7% 20.44 perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
726.00 ? 8% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
12628 ? 2% -29.5% 8908 ? 2% perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
12520 ? 2% -29.7% 8797 ? 2% perf-sched.wait_and_delay.count.do_wait.kernel_wait.call_usermodehelper_exec_work.process_one_work
618.00 ? 3% -41.8% 359.83 ? 8% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
5408 ? 4% -62.3% 2039 ? 7% perf-sched.wait_and_delay.count.io_schedule.folio_wait_bit_common.filemap_fault.__do_fault
167.33 +55.0% 259.33 ? 2% perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
52.67 ? 8% -39.9% 31.67 ? 8% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
13.50 ? 8% +860.5% 129.67 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
1478 ? 23% -69.8% 446.50 ? 18% perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
12518 ? 2% -29.7% 8800 ? 2% perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
7535 -23.4% 5772 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
22238 -25.8% 16493 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
0.06 ?112% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
3.92 ? 29% -40.6% 2.33 ? 4% perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
5.35 ? 11% +3637.0% 200.02 ? 11% perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
5.03 +68.3% 8.46 ? 14% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
1.88 ? 12% +2136.1% 42.02 perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
2.08 ? 2% +51.4% 3.14 ? 2% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.01 ? 39% -70.6% 0.00 ? 56% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
0.56 ? 17% +48.9% 0.84 ? 9% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
0.10 ?223% +51536.6% 51.98 ? 5% perf-sched.wait_time.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
384.27 -30.6% 266.52 ? 2% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
298.54 ? 13% +63.3% 487.40 ? 18% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
368.02 ? 5% -80.2% 72.92 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
0.01 ? 37% -79.4% 0.00 ?159% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma
1.48 ? 8% -19.8% 1.18 ? 6% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
1.82 +62.1% 2.94 ? 2% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
4.08 +18.6% 4.84 ? 13% perf-sched.wait_time.avg.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
78.20 ? 3% +32.3% 103.49 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
1.89 ? 12% +2117.5% 42.02 perf-sched.wait_time.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read
14.84 ? 2% +37.7% 20.43 perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
0.02 ? 29% -68.5% 0.01 ? 31% perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.perf_event_mmap_event.perf_event_mmap
0.05 ?146% -87.6% 0.01 ? 21% perf-sched.wait_time.max.ms.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.[unknown]
0.01 ? 56% -69.8% 0.00 ? 70% perf-sched.wait_time.max.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open
0.01 ? 57% -84.4% 0.00 ?141% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.vm_area_dup.__split_vma.do_vmi_align_munmap
3.76 ? 12% +2857.8% 111.15 ? 11% perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
0.28 ?223% +41327.3% 116.13 ? 12% perf-sched.wait_time.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex
0.02 ? 38% -92.1% 0.00 ?141% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.mmap_region
0.03 ? 57% -94.4% 0.00 ?142% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma
3.92 ? 29% -40.7% 2.32 ? 4% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.crypto_larval_wait.crypto_alg_mod_lookup
5.12 ? 10% +3808.0% 200.02 ? 11% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec
4.09 +126.9% 9.29 ? 65% perf-sched.wait_time.max.ms.schedule_timeout.ext4_lazyinit_thread.part.0.kthread
5.02 +68.5% 8.45 ? 14% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
3.79 ? 12% +2833.2% 111.15 ? 11% perf-sched.wait_time.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read
24.78 ? 2% -6.4 18.42 ? 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
24.75 ? 2% -6.4 18.39 ? 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.72 ? 3% -3.2 7.54 ? 4% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.46 ? 3% -3.2 7.31 ? 4% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
9.77 ? 3% -3.0 6.75 ? 5% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
9.81 ? 3% -3.0 6.79 ? 4% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.15 -2.5 7.63 ? 3% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.15 -2.5 7.63 ? 3% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.15 -2.5 7.63 ? 3% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
17.43 ? 2% -2.4 14.98 perf-profile.calltrace.cycles-pp.asm_exc_page_fault
15.02 ? 3% -2.4 12.59 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
9.32 -2.4 6.94 ? 4% perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
9.27 -2.4 6.89 ? 4% perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
9.31 -2.4 6.93 ? 4% perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group
16.35 ? 2% -2.4 13.98 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
16.27 ? 2% -2.3 13.93 perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
12.31 ? 4% -2.2 10.08 perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
15.14 ? 2% -2.2 12.93 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
9.96 ? 8% -2.1 7.90 ? 3% perf-profile.calltrace.cycles-pp.jent_kcapi_random._rng_recvmsg.sock_recvmsg.sock_read_iter.vfs_read
9.96 ? 8% -2.1 7.90 ? 3% perf-profile.calltrace.cycles-pp.jent_read_entropy.jent_kcapi_random._rng_recvmsg.sock_recvmsg.sock_read_iter
9.96 ? 8% -2.1 7.90 ? 3% perf-profile.calltrace.cycles-pp.jent_gen_entropy.jent_read_entropy.jent_kcapi_random._rng_recvmsg.sock_recvmsg
9.95 ? 8% -2.1 7.89 ? 3% perf-profile.calltrace.cycles-pp.jent_measure_jitter.jent_gen_entropy.jent_read_entropy.jent_kcapi_random._rng_recvmsg
9.96 ? 8% -2.1 7.90 ? 3% perf-profile.calltrace.cycles-pp._rng_recvmsg.sock_recvmsg.sock_read_iter.vfs_read.ksys_read
6.08 ? 4% -1.9 4.14 ? 5% perf-profile.calltrace.cycles-pp.do_vmi_munmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
6.01 ? 4% -1.9 4.08 ? 5% perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.mmap_region.do_mmap.vm_mmap_pgoff
9.37 ? 5% -1.8 7.52 ? 3% perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
8.37 ? 8% -1.7 6.63 ? 3% perf-profile.calltrace.cycles-pp.jent_lfsr_time.jent_measure_jitter.jent_gen_entropy.jent_read_entropy.jent_kcapi_random
9.11 ? 5% -1.7 7.38 ? 3% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
4.25 ? 3% -1.3 2.94 ? 5% perf-profile.calltrace.cycles-pp.__split_vma.do_vmi_align_munmap.do_vmi_munmap.mmap_region.do_mmap
7.37 ? 3% -1.3 6.12 ? 3% perf-profile.calltrace.cycles-pp.next_uptodate_page.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
6.49 -1.2 5.32 ? 3% perf-profile.calltrace.cycles-pp.call_usermodehelper_exec_async.ret_from_fork
6.36 -1.2 5.20 ? 3% perf-profile.calltrace.cycles-pp.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
2.67 ? 5% -1.0 1.68 ? 10% perf-profile.calltrace.cycles-pp.vma_prepare.__split_vma.do_vmi_align_munmap.do_vmi_munmap.mmap_region
1.25 ? 12% -1.0 0.30 ?101% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.vma_prepare.__split_vma.do_vmi_align_munmap
1.35 ? 11% -0.9 0.43 ? 72% perf-profile.calltrace.cycles-pp.down_write.vma_prepare.__split_vma.do_vmi_align_munmap.do_vmi_munmap
4.68 ? 3% -0.8 3.83 ? 3% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
2.26 ? 3% -0.8 1.44 ? 7% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.__mmput.exit_mm.do_exit
4.44 ? 3% -0.8 3.63 ? 3% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
4.79 ? 2% -0.8 3.98 ? 3% perf-profile.calltrace.cycles-pp.bprm_execve.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
4.40 ? 3% -0.8 3.60 ? 3% perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
4.32 ? 3% -0.8 3.54 ? 3% perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
4.22 ? 2% -0.7 3.48 ? 3% perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.kernel_execve.call_usermodehelper_exec_async
4.24 ? 2% -0.7 3.50 ? 3% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
4.14 ? 2% -0.7 3.42 ? 3% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.kernel_execve
1.93 ? 3% -0.6 1.28 ? 6% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.__mmput.exit_mm.do_exit
1.89 ? 3% -0.6 1.25 ? 6% perf-profile.calltrace.cycles-pp.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput.exit_mm
1.41 ? 5% -0.6 0.78 ? 10% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.__mmput.exit_mm
1.32 ? 9% -0.6 0.74 ? 11% perf-profile.calltrace.cycles-pp.unmap_region.do_vmi_align_munmap.do_vmi_munmap.mmap_region.do_mmap
1.50 ? 4% -0.5 0.98 ? 7% perf-profile.calltrace.cycles-pp.release_pages.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput
1.57 ? 18% -0.5 1.10 ? 3% perf-profile.calltrace.cycles-pp.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.55 ? 18% -0.5 1.08 ? 3% perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
1.78 ? 2% -0.4 1.34 ? 3% perf-profile.calltrace.cycles-pp.call_usermodehelper_exec_work.process_one_work.worker_thread.kthread.ret_from_fork
2.92 -0.4 2.53 ? 2% perf-profile.calltrace.cycles-pp.do_cow_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
0.64 ? 4% -0.4 0.28 ?100% perf-profile.calltrace.cycles-pp.vma_prepare.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect
0.72 ? 5% -0.4 0.36 ? 70% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.elf_map.load_elf_interp
0.64 ? 7% -0.4 0.27 ?100% perf-profile.calltrace.cycles-pp.sendmsg
2.02 ? 2% -0.3 1.68 ? 3% perf-profile.calltrace.cycles-pp.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.02 ? 3% -0.3 1.68 ? 3% perf-profile.calltrace.cycles-pp.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.41 ? 3% -0.3 1.08 ? 3% perf-profile.calltrace.cycles-pp.kernel_clone.user_mode_thread.call_usermodehelper_exec_work.process_one_work.worker_thread
1.41 ? 3% -0.3 1.08 ? 3% perf-profile.calltrace.cycles-pp.user_mode_thread.call_usermodehelper_exec_work.process_one_work.worker_thread.kthread
0.59 ? 4% -0.3 0.27 ?100% perf-profile.calltrace.cycles-pp.getenv
1.49 -0.3 1.17 ? 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
1.50 -0.3 1.18 ? 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
1.51 -0.3 1.19 ? 3% perf-profile.calltrace.cycles-pp.__munmap
0.68 ? 5% -0.3 0.37 ? 71% perf-profile.calltrace.cycles-pp.vma_interval_tree_insert.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
1.54 ? 7% -0.3 1.23 ? 5% perf-profile.calltrace.cycles-pp.jent_memaccess.jent_measure_jitter.jent_gen_entropy.jent_read_entropy.jent_kcapi_random
1.47 ? 2% -0.3 1.16 ? 16% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
1.49 ? 2% -0.3 1.18 ? 16% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.75 ? 3% -0.3 1.45 ? 4% perf-profile.calltrace.cycles-pp.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.41 ? 2% -0.3 1.11 ? 3% perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
1.40 ? 2% -0.3 1.11 ? 3% perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
1.16 ? 4% -0.3 0.87 ? 5% perf-profile.calltrace.cycles-pp.load_elf_interp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
0.76 ? 5% -0.3 0.47 ? 44% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.elf_map.load_elf_interp.load_elf_binary.search_binary_handler
1.11 ? 5% -0.3 0.82 ? 9% perf-profile.calltrace.cycles-pp.alloc_bprm.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
0.74 ? 6% -0.3 0.46 ? 44% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.elf_map.load_elf_interp.load_elf_binary
1.07 ? 4% -0.3 0.78 ? 5% perf-profile.calltrace.cycles-pp.elf_map.load_elf_interp.load_elf_binary.search_binary_handler.exec_binprm
1.03 ? 5% -0.3 0.76 ? 10% perf-profile.calltrace.cycles-pp.mm_init.alloc_bprm.kernel_execve.call_usermodehelper_exec_async.ret_from_fork
1.39 ? 4% -0.3 1.13 ? 6% perf-profile.calltrace.cycles-pp.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64
1.12 ? 3% -0.3 0.86 ? 4% perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.user_mode_thread.call_usermodehelper_exec_work.process_one_work
2.00 ? 2% -0.2 1.77 ? 2% perf-profile.calltrace.cycles-pp._dl_addr
0.93 ? 4% -0.2 0.71 ? 4% perf-profile.calltrace.cycles-pp.unmap_region.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
0.68 ? 6% -0.2 0.47 ? 45% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
1.48 ? 5% -0.2 1.26 ? 6% perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
0.75 ? 3% -0.2 0.55 ? 3% perf-profile.calltrace.cycles-pp.dup_task_struct.copy_process.kernel_clone.user_mode_thread.call_usermodehelper_exec_work
1.09 ? 2% -0.2 0.90 ? 4% perf-profile.calltrace.cycles-pp.__mmap
1.07 ? 2% -0.2 0.88 ? 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
1.06 ? 2% -0.2 0.88 ? 4% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
1.07 ? 2% -0.2 0.88 ? 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
1.04 ? 3% -0.2 0.86 ? 4% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
0.86 ? 6% -0.2 0.68 ? 9% perf-profile.calltrace.cycles-pp.elf_map.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
1.08 ? 2% -0.2 0.90 ? 6% perf-profile.calltrace.cycles-pp.copy_page.do_cow_fault.do_fault.__handle_mm_fault.handle_mm_fault
0.92 ? 2% -0.2 0.75 ? 4% perf-profile.calltrace.cycles-pp.vma_complete.__split_vma.do_vmi_align_munmap.do_vmi_munmap.mmap_region
0.65 ? 5% -0.2 0.49 ? 45% perf-profile.calltrace.cycles-pp.filemap_fault.__do_fault.do_cow_fault.do_fault.__handle_mm_fault
0.72 ? 5% -0.1 0.58 ? 8% perf-profile.calltrace.cycles-pp.do_set_pte.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
0.76 ? 4% -0.1 0.62 ? 6% perf-profile.calltrace.cycles-pp.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
0.94 ? 3% -0.1 0.80 ? 3% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
0.95 ? 3% -0.1 0.81 ? 3% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
1.00 ? 5% -0.1 0.86 ? 8% perf-profile.calltrace.cycles-pp.vma_interval_tree_insert.vma_prepare.__split_vma.do_vmi_align_munmap.do_vmi_munmap
0.80 ? 3% -0.1 0.67 ? 5% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.96 ? 3% -0.1 0.82 ? 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64
0.78 ? 4% -0.1 0.65 ? 4% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.98 ? 3% -0.1 0.85 ? 3% perf-profile.calltrace.cycles-pp.open64
0.74 ? 4% -0.1 0.61 ? 5% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.95 ? 3% -0.1 0.82 ? 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
0.81 ? 4% -0.1 0.69 ? 4% perf-profile.calltrace.cycles-pp.__sys_bind.__x64_sys_bind.do_syscall_64.entry_SYSCALL_64_after_hwframe.bind
0.81 ? 4% -0.1 0.69 ? 5% perf-profile.calltrace.cycles-pp.__x64_sys_bind.do_syscall_64.entry_SYSCALL_64_after_hwframe.bind
0.82 ? 4% -0.1 0.70 ? 4% perf-profile.calltrace.cycles-pp.bind
0.74 ? 5% -0.1 0.62 ? 8% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.81 ? 5% -0.1 0.70 ? 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.bind
0.81 ? 4% -0.1 0.70 ? 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.bind
0.80 ? 5% -0.1 0.68 ? 5% perf-profile.calltrace.cycles-pp.alg_bind.__sys_bind.__x64_sys_bind.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.73 ? 4% -0.1 0.62 ? 8% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.41 ? 3% -0.1 1.31 ? 3% perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.67 ? 5% -0.1 0.58 ? 7% perf-profile.calltrace.cycles-pp.__do_fault.do_cow_fault.do_fault.__handle_mm_fault.handle_mm_fault
0.67 ? 3% -0.1 0.60 ? 6% perf-profile.calltrace.cycles-pp.mas_store_prealloc.vma_complete.__split_vma.do_vmi_align_munmap.do_vmi_munmap
0.71 ? 5% -0.1 0.66 ? 3% perf-profile.calltrace.cycles-pp.finish_fault.do_cow_fault.do_fault.__handle_mm_fault.handle_mm_fault
0.69 ? 10% +0.1 0.84 ? 3% perf-profile.calltrace.cycles-pp.__intel_pmu_enable_all.perf_adjust_freq_unthr_context.perf_event_task_tick.scheduler_tick.update_process_times
1.28 ? 7% +0.2 1.50 ? 3% perf-profile.calltrace.cycles-pp.perf_adjust_freq_unthr_context.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle
1.32 ? 7% +0.2 1.54 ? 3% perf-profile.calltrace.cycles-pp.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
2.00 ? 5% +0.3 2.28 ? 3% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
2.38 ? 5% +0.3 2.71 ? 4% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
2.41 ? 5% +0.3 2.74 ? 4% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
2.73 ? 5% +0.4 3.13 ? 3% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
3.18 ? 6% +0.5 3.67 ? 2% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +0.6 0.61 ? 6% perf-profile.calltrace.cycles-pp.verify_pkcs7_message_sig.verify_pkcs7_signature.mod_verify_sig.module_sig_check.load_module
0.00 +0.6 0.62 ? 6% perf-profile.calltrace.cycles-pp.module_sig_check.load_module.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.6 0.62 ? 6% perf-profile.calltrace.cycles-pp.mod_verify_sig.module_sig_check.load_module.__do_sys_finit_module.do_syscall_64
0.00 +0.6 0.62 ? 6% perf-profile.calltrace.cycles-pp.verify_pkcs7_signature.mod_verify_sig.module_sig_check.load_module.__do_sys_finit_module
0.00 +0.6 0.65 ? 3% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
3.58 ? 5% +0.7 4.23 ? 2% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
3.60 ? 5% +0.7 4.25 ? 2% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
5.61 ? 5% +0.9 6.50 ? 4% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
5.88 ? 5% +1.0 6.83 ? 4% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.00 +1.3 1.33 ? 7% perf-profile.calltrace.cycles-pp.load_module.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
0.00 +1.9 1.90 ? 11% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_lsr.serial8250_console_write.console_flush_all.console_unlock
0.00 +2.5 2.51 ? 4% perf-profile.calltrace.cycles-pp.wait_for_lsr.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit
6.02 ? 2% +2.8 8.78 ? 2% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.00 +3.2 3.18 ? 7% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit._printk
9.98 +3.4 13.35 perf-profile.calltrace.cycles-pp.ret_from_fork
0.00 +3.6 3.56 ? 8% perf-profile.calltrace.cycles-pp.do_one_initcall.do_init_module.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +3.8 3.82 ? 8% perf-profile.calltrace.cycles-pp.do_init_module.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
12.24 ? 3% +4.3 16.55 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
12.51 ? 3% +4.4 16.96 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
3.43 ? 3% +4.6 7.98 ? 3% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
13.69 ? 3% +4.7 18.35 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
13.69 ? 3% +4.7 18.35 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
13.67 ? 3% +4.7 18.33 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
2.68 ? 2% +4.7 7.34 ? 3% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
2.38 ? 3% +4.7 7.05 ? 3% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
0.00 +4.7 4.67 ? 3% perf-profile.calltrace.cycles-pp.memcpy_toio.drm_fb_memcpy.ast_primary_plane_helper_atomic_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm
13.10 ? 3% +4.7 17.80 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
13.93 ? 3% +4.7 18.65 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
0.00 +4.7 4.73 ? 3% perf-profile.calltrace.cycles-pp.ast_primary_plane_helper_atomic_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail.commit_tail
0.00 +4.7 4.73 ? 3% perf-profile.calltrace.cycles-pp.drm_fb_memcpy.ast_primary_plane_helper_atomic_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail
0.00 +4.7 4.73 ? 3% perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail.commit_tail.drm_atomic_helper_commit
0.00 +4.7 4.74 ? 3% perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_commit
0.00 +4.8 4.76 ? 3% perf-profile.calltrace.cycles-pp.commit_tail.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty
0.00 +4.8 4.76 ? 3% perf-profile.calltrace.cycles-pp.ast_mode_config_helper_atomic_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb
0.00 +4.8 4.79 ? 3% perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work
0.00 +4.8 4.80 ? 3% perf-profile.calltrace.cycles-pp.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work
0.00 +4.8 4.80 ? 3% perf-profile.calltrace.cycles-pp.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread
0.00 +5.2 5.18 ? 7% perf-profile.calltrace.cycles-pp.__do_sys_finit_module.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
0.00 +5.2 5.19 ? 7% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.syscall
0.00 +5.2 5.19 ? 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall
0.00 +5.2 5.19 ? 7% perf-profile.calltrace.cycles-pp.syscall
0.00 +5.4 5.38 ? 3% perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork
0.00 +5.4 5.38 ? 3% perf-profile.calltrace.cycles-pp.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread
49.25 ? 2% -4.4 44.89 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
49.19 ? 2% -4.3 44.84 perf-profile.children.cycles-pp.do_syscall_64
12.99 ? 2% -3.6 9.34 ? 4% perf-profile.children.cycles-pp.vm_mmap_pgoff
12.84 ? 2% -3.6 9.21 ? 4% perf-profile.children.cycles-pp.do_mmap
12.50 ? 2% -3.6 8.91 ? 4% perf-profile.children.cycles-pp.mmap_region
10.87 ? 3% -3.2 7.67 ? 4% perf-profile.children.cycles-pp.ksys_mmap_pgoff
21.46 ? 2% -2.9 18.51 perf-profile.children.cycles-pp.asm_exc_page_fault
20.13 ? 2% -2.8 17.30 perf-profile.children.cycles-pp.exc_page_fault
20.06 ? 2% -2.8 17.25 perf-profile.children.cycles-pp.do_user_addr_fault
18.96 ? 2% -2.7 16.28 ? 2% perf-profile.children.cycles-pp.handle_mm_fault
18.50 ? 2% -2.6 15.88 ? 2% perf-profile.children.cycles-pp.__handle_mm_fault
8.98 ? 3% -2.6 6.42 ? 3% perf-profile.children.cycles-pp.do_vmi_munmap
10.39 -2.6 7.84 ? 3% perf-profile.children.cycles-pp.do_group_exit
10.39 -2.5 7.84 ? 3% perf-profile.children.cycles-pp.__x64_sys_exit_group
10.37 -2.5 7.83 ? 3% perf-profile.children.cycles-pp.do_exit
8.82 ? 3% -2.5 6.30 ? 3% perf-profile.children.cycles-pp.do_vmi_align_munmap
4.66 ? 7% -2.5 2.19 ? 16% perf-profile.children.cycles-pp.down_write
15.37 ? 2% -2.4 12.93 ? 2% perf-profile.children.cycles-pp.do_fault
9.35 -2.4 6.96 ? 4% perf-profile.children.cycles-pp.exit_mm
9.32 -2.4 6.94 ? 4% perf-profile.children.cycles-pp.__mmput
9.29 -2.4 6.90 ? 3% perf-profile.children.cycles-pp.exit_mmap
3.93 ? 9% -2.3 1.64 ? 18% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
3.74 ? 9% -2.2 1.55 ? 19% perf-profile.children.cycles-pp.rwsem_optimistic_spin
10.28 ? 8% -2.1 8.16 ? 3% perf-profile.children.cycles-pp.jent_gen_entropy
10.27 ? 8% -2.1 8.16 ? 3% perf-profile.children.cycles-pp.jent_measure_jitter
9.96 ? 8% -2.1 7.90 ? 3% perf-profile.children.cycles-pp.jent_kcapi_random
9.96 ? 8% -2.1 7.90 ? 3% perf-profile.children.cycles-pp.jent_read_entropy
9.96 ? 8% -2.1 7.90 ? 3% perf-profile.children.cycles-pp._rng_recvmsg
12.31 ? 3% -2.0 10.28 ? 3% perf-profile.children.cycles-pp.do_read_fault
12.18 ? 3% -2.0 10.16 ? 3% perf-profile.children.cycles-pp.filemap_map_pages
8.64 ? 8% -1.8 6.85 ? 3% perf-profile.children.cycles-pp.jent_lfsr_time
10.05 ? 3% -1.6 8.42 ? 3% perf-profile.children.cycles-pp.next_uptodate_page
5.94 ? 2% -1.6 4.30 ? 5% perf-profile.children.cycles-pp.__split_vma
3.62 ? 4% -1.5 2.14 ? 9% perf-profile.children.cycles-pp.free_pgtables
3.81 ? 4% -1.3 2.49 ? 9% perf-profile.children.cycles-pp.vma_prepare
2.60 ? 6% -1.2 1.37 ? 13% perf-profile.children.cycles-pp.unlink_file_vma
2.24 ? 5% -1.2 1.03 ? 17% perf-profile.children.cycles-pp.rwsem_spin_on_owner
6.49 -1.2 5.32 ? 3% perf-profile.children.cycles-pp.call_usermodehelper_exec_async
6.36 -1.2 5.20 ? 3% perf-profile.children.cycles-pp.kernel_execve
2.98 ? 4% -1.0 2.00 ? 4% perf-profile.children.cycles-pp.unmap_region
5.35 ? 2% -1.0 4.39 ? 2% perf-profile.children.cycles-pp.unmap_vmas
5.05 ? 2% -0.9 4.15 ? 2% perf-profile.children.cycles-pp.unmap_page_range
4.97 ? 3% -0.9 4.08 ? 2% perf-profile.children.cycles-pp.zap_pmd_range
4.92 ? 2% -0.9 4.04 ? 2% perf-profile.children.cycles-pp.zap_pte_range
1.33 ? 19% -0.8 0.48 ? 28% perf-profile.children.cycles-pp.osq_lock
4.88 ? 2% -0.8 4.07 ? 3% perf-profile.children.cycles-pp.bprm_execve
2.60 -0.8 1.82 ? 4% perf-profile.children.cycles-pp.tlb_finish_mmu
4.28 ? 2% -0.7 3.56 ? 4% perf-profile.children.cycles-pp.exec_binprm
4.26 ? 2% -0.7 3.54 ? 4% perf-profile.children.cycles-pp.search_binary_handler
4.18 ? 2% -0.7 3.48 ? 4% perf-profile.children.cycles-pp.load_elf_binary
2.02 ? 2% -0.7 1.36 ? 6% perf-profile.children.cycles-pp.tlb_batch_pages_flush
2.78 ? 4% -0.5 2.24 ? 7% perf-profile.children.cycles-pp.vma_interval_tree_insert
1.62 ? 4% -0.5 1.09 ? 7% perf-profile.children.cycles-pp.release_pages
2.45 ? 3% -0.5 1.94 ? 3% perf-profile.children.cycles-pp.__vm_munmap
1.94 ? 5% -0.5 1.48 ? 6% perf-profile.children.cycles-pp.elf_map
1.78 ? 2% -0.4 1.34 ? 3% perf-profile.children.cycles-pp.call_usermodehelper_exec_work
3.03 -0.4 2.64 perf-profile.children.cycles-pp.do_cow_fault
1.91 ? 3% -0.4 1.53 ? 2% perf-profile.children.cycles-pp.__x64_sys_munmap
1.17 ? 3% -0.4 0.79 ? 7% perf-profile.children.cycles-pp.up_write
0.66 ? 13% -0.4 0.31 ? 12% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
2.02 ? 2% -0.3 1.68 ? 3% perf-profile.children.cycles-pp.__x64_sys_mprotect
2.02 ? 3% -0.3 1.68 ? 3% perf-profile.children.cycles-pp.do_mprotect_pkey
1.41 ? 3% -0.3 1.08 ? 3% perf-profile.children.cycles-pp.user_mode_thread
1.58 ? 4% -0.3 1.26 ? 3% perf-profile.children.cycles-pp.vma_complete
1.46 ? 2% -0.3 1.14 ? 3% perf-profile.children.cycles-pp.kernel_clone
1.52 ? 2% -0.3 1.20 ? 3% perf-profile.children.cycles-pp.__munmap
1.59 ? 7% -0.3 1.27 ? 5% perf-profile.children.cycles-pp.jent_memaccess
1.76 ? 3% -0.3 1.46 ? 4% perf-profile.children.cycles-pp.mprotect_fixup
1.12 ? 5% -0.3 0.83 ? 9% perf-profile.children.cycles-pp.alloc_bprm
1.17 ? 4% -0.3 0.88 ? 5% perf-profile.children.cycles-pp.load_elf_interp
1.96 ? 3% -0.3 1.68 perf-profile.children.cycles-pp.__x64_sys_openat
1.94 ? 3% -0.3 1.67 perf-profile.children.cycles-pp.do_sys_openat2
1.04 ? 5% -0.3 0.77 ? 10% perf-profile.children.cycles-pp.mm_init
2.01 ? 2% -0.3 1.74 perf-profile.children.cycles-pp.do_filp_open
1.99 ? 2% -0.3 1.72 perf-profile.children.cycles-pp.path_openat
1.16 ? 2% -0.3 0.90 ? 4% perf-profile.children.cycles-pp.copy_process
2.08 -0.2 1.84 ? 4% perf-profile.children.cycles-pp.mas_store_prealloc
2.02 ? 2% -0.2 1.78 ? 2% perf-profile.children.cycles-pp._dl_addr
0.82 ? 5% -0.2 0.57 ? 12% perf-profile.children.cycles-pp.pcpu_alloc
0.85 -0.2 0.61 ? 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.29 ? 2% -0.2 1.05 ? 6% perf-profile.children.cycles-pp.do_set_pte
2.10 ? 2% -0.2 1.87 ? 2% perf-profile.children.cycles-pp.__do_softirq
1.63 ? 4% -0.2 1.40 ? 5% perf-profile.children.cycles-pp.page_remove_rmap
1.18 ? 4% -0.2 0.95 perf-profile.children.cycles-pp.mas_alloc_nodes
0.65 ? 4% -0.2 0.43 ? 11% perf-profile.children.cycles-pp.__percpu_counter_init
0.68 ? 6% -0.2 0.46 ? 2% perf-profile.children.cycles-pp.vma_expand
1.08 ? 4% -0.2 0.86 perf-profile.children.cycles-pp.mas_preallocate
2.14 ? 2% -0.2 1.93 perf-profile.children.cycles-pp._raw_spin_lock
0.95 ? 2% -0.2 0.74 ? 6% perf-profile.children.cycles-pp.vma_interval_tree_remove
0.65 ? 3% -0.2 0.45 ? 2% perf-profile.children.cycles-pp.alloc_thread_stack_node
0.76 ? 3% -0.2 0.56 ? 2% perf-profile.children.cycles-pp.dup_task_struct
1.09 ? 2% -0.2 0.90 ? 5% perf-profile.children.cycles-pp.__mmap
0.41 ? 5% -0.2 0.22 ? 5% perf-profile.children.cycles-pp.folio_lruvec_lock_irqsave
1.11 ? 2% -0.2 0.93 ? 6% perf-profile.children.cycles-pp.copy_page
1.38 ? 2% -0.2 1.22 ? 2% perf-profile.children.cycles-pp.rcu_core
0.36 ? 7% -0.2 0.20 ? 10% perf-profile.children.cycles-pp.__get_vm_area_node
0.60 ? 3% -0.2 0.44 ? 3% perf-profile.children.cycles-pp.__vmalloc_node_range
1.26 ? 2% -0.2 1.10 ? 2% perf-profile.children.cycles-pp.rcu_do_batch
0.65 ? 4% -0.2 0.50 ? 4% perf-profile.children.cycles-pp.stress_mwc8
0.94 ? 5% -0.1 0.80 ? 3% perf-profile.children.cycles-pp.kmem_cache_alloc
0.31 ? 7% -0.1 0.17 ? 13% perf-profile.children.cycles-pp.alloc_vmap_area
0.37 ? 8% -0.1 0.23 ? 8% perf-profile.children.cycles-pp.delayed_vfree_work
1.79 ? 3% -0.1 1.65 ? 2% perf-profile.children.cycles-pp.__irq_exit_rcu
0.68 ? 6% -0.1 0.55 ? 8% perf-profile.children.cycles-pp.smpboot_thread_fn
0.99 ? 3% -0.1 0.85 ? 2% perf-profile.children.cycles-pp.open64
0.77 ? 4% -0.1 0.64 ? 6% perf-profile.children.cycles-pp.begin_new_exec
0.56 ? 5% -0.1 0.43 ? 8% perf-profile.children.cycles-pp.exec_mmap
0.51 ? 5% -0.1 0.38 ? 5% perf-profile.children.cycles-pp.__fput
0.66 ? 3% -0.1 0.53 ? 2% perf-profile.children.cycles-pp.flush_tlb_mm_range
0.41 ? 5% -0.1 0.28 ? 10% perf-profile.children.cycles-pp.free_swap_cache
0.64 ? 7% -0.1 0.52 ? 9% perf-profile.children.cycles-pp.sendmsg
0.58 ? 4% -0.1 0.45 ? 4% perf-profile.children.cycles-pp.task_work_run
0.53 ? 5% -0.1 0.41 ? 6% perf-profile.children.cycles-pp.folio_batch_move_lru
0.53 ? 8% -0.1 0.41 ? 5% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.56 ? 7% -0.1 0.44 ? 8% perf-profile.children.cycles-pp.___sys_sendmsg
0.42 ? 4% -0.1 0.30 ? 8% perf-profile.children.cycles-pp.free_pages_and_swap_cache
0.81 ? 4% -0.1 0.69 ? 5% perf-profile.children.cycles-pp.__x64_sys_bind
0.81 ? 4% -0.1 0.69 ? 5% perf-profile.children.cycles-pp.__sys_bind
0.52 ? 7% -0.1 0.40 ? 5% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
1.06 ? 2% -0.1 0.94 ? 4% perf-profile.children.cycles-pp.perf_event_mmap
0.80 ? 5% -0.1 0.68 ? 5% perf-profile.children.cycles-pp.alg_bind
1.02 ? 2% -0.1 0.91 ? 4% perf-profile.children.cycles-pp.perf_event_mmap_event
0.57 ? 5% -0.1 0.46 ? 4% perf-profile.children.cycles-pp.flush_tlb_func
0.82 ? 4% -0.1 0.71 ? 4% perf-profile.children.cycles-pp.bind
0.48 ? 9% -0.1 0.36 ? 9% perf-profile.children.cycles-pp.stress_rndbuf
0.32 ? 8% -0.1 0.21 ? 8% perf-profile.children.cycles-pp.remove_vm_area
0.75 ? 5% -0.1 0.64 ? 2% perf-profile.children.cycles-pp.kmem_cache_alloc_bulk
0.32 ? 5% -0.1 0.21 ? 21% perf-profile.children.cycles-pp.__rb_insert_augmented
0.36 ? 5% -0.1 0.25 ? 9% perf-profile.children.cycles-pp.kernel_wait
0.63 ? 4% -0.1 0.53 ? 5% perf-profile.children.cycles-pp.open_last_lookups
0.36 ? 5% -0.1 0.26 ? 8% perf-profile.children.cycles-pp.do_wait
0.78 ? 5% -0.1 0.68 ? 3% perf-profile.children.cycles-pp.mas_store_gfp
0.68 ? 6% -0.1 0.58 ? 4% perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
0.47 ? 7% -0.1 0.37 ? 6% perf-profile.children.cycles-pp.page_add_file_rmap
0.43 ? 7% -0.1 0.33 ? 5% perf-profile.children.cycles-pp.exit_to_user_mode_loop
0.18 ? 15% -0.1 0.08 ? 26% perf-profile.children.cycles-pp.__rb_erase_color
0.74 ? 5% -0.1 0.65 ? 6% perf-profile.children.cycles-pp.__do_fault
0.65 ? 3% -0.1 0.55 ? 5% perf-profile.children.cycles-pp.__slab_free
0.63 ? 3% -0.1 0.54 ? 5% perf-profile.children.cycles-pp.mas_destroy
0.69 ? 4% -0.1 0.59 ? 9% perf-profile.children.cycles-pp.filemap_fault
0.72 ? 3% -0.1 0.63 ? 2% perf-profile.children.cycles-pp.mas_wr_bnode
1.06 ? 3% -0.1 0.97 ? 4% perf-profile.children.cycles-pp.__schedule
0.46 ? 5% -0.1 0.36 ? 7% perf-profile.children.cycles-pp.filename_lookup
0.62 ? 3% -0.1 0.53 ? 6% perf-profile.children.cycles-pp.getenv
0.38 ? 3% -0.1 0.29 ? 5% perf-profile.children.cycles-pp.lru_add_drain_cpu
0.45 ? 5% -0.1 0.36 ? 6% perf-profile.children.cycles-pp.path_lookupat
0.51 ? 4% -0.1 0.42 ? 10% perf-profile.children.cycles-pp.perf_event_mmap_output
0.49 ? 5% -0.1 0.40 ? 10% perf-profile.children.cycles-pp.__mmdrop
0.38 ? 3% -0.1 0.29 ? 5% perf-profile.children.cycles-pp.lru_add_drain
0.51 ? 4% -0.1 0.42 ? 3% perf-profile.children.cycles-pp.lock_vma_under_rcu
0.69 ? 7% -0.1 0.60 ? 5% perf-profile.children.cycles-pp.mtree_range_walk
0.69 ? 5% -0.1 0.60 ? 6% perf-profile.children.cycles-pp.perf_iterate_sb
0.62 ? 2% -0.1 0.53 ? 7% perf-profile.children.cycles-pp.do_open
0.50 ? 3% -0.1 0.42 ? 4% perf-profile.children.cycles-pp.vfs_fstatat
0.48 ? 3% -0.1 0.40 ? 9% perf-profile.children.cycles-pp.do_dentry_open
0.41 ? 8% -0.1 0.33 ? 12% perf-profile.children.cycles-pp.___slab_alloc
0.47 ? 6% -0.1 0.39 ? 8% perf-profile.children.cycles-pp.__list_del_entry_valid
0.57 ? 7% -0.1 0.49 ? 6% perf-profile.children.cycles-pp.mas_walk
0.70 ? 5% -0.1 0.63 ? 5% perf-profile.children.cycles-pp.crypto_alloc_tfm_node
0.32 ? 4% -0.1 0.24 ? 11% perf-profile.children.cycles-pp.lru_add_fn
0.50 ? 4% -0.1 0.43 ? 11% perf-profile.children.cycles-pp.select_task_rq_fair
0.42 ? 5% -0.1 0.35 ? 5% perf-profile.children.cycles-pp.vfs_statx
0.52 ? 5% -0.1 0.44 ? 4% perf-profile.children.cycles-pp.native_flush_tlb_one_user
0.45 -0.1 0.38 ? 6% perf-profile.children.cycles-pp.rebalance_domains
0.11 ? 13% -0.1 0.04 ? 45% perf-profile.children.cycles-pp.free_vmap_area_noflush
0.30 ? 7% -0.1 0.23 ? 13% perf-profile.children.cycles-pp.getopt_long
0.27 -0.1 0.20 ? 7% perf-profile.children.cycles-pp.dput
0.28 ? 11% -0.1 0.22 ? 4% perf-profile.children.cycles-pp.getdents64
0.41 ? 3% -0.1 0.34 ? 6% perf-profile.children.cycles-pp.__xstat64
0.27 ? 12% -0.1 0.20 ? 4% perf-profile.children.cycles-pp.__x64_sys_getdents64
0.64 ? 6% -0.1 0.57 ? 6% perf-profile.children.cycles-pp.__filemap_get_folio
0.33 ? 9% -0.1 0.27 ? 4% perf-profile.children.cycles-pp.mas_find
0.25 ? 12% -0.1 0.18 ? 4% perf-profile.children.cycles-pp.__close_nocancel
0.44 ? 4% -0.1 0.38 ? 10% perf-profile.children.cycles-pp.find_idlest_cpu
0.39 ? 6% -0.1 0.33 ? 7% perf-profile.children.cycles-pp.walk_component
0.40 ? 6% -0.1 0.34 ? 8% perf-profile.children.cycles-pp.kmem_cache_free_bulk
0.25 ? 10% -0.1 0.19 ? 3% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
0.26 ? 11% -0.1 0.20 ? 2% perf-profile.children.cycles-pp.iterate_dir
0.16 ? 7% -0.1 0.10 ? 14% perf-profile.children.cycles-pp.vma_interval_tree_augment_rotate
0.28 ? 6% -0.1 0.22 ? 11% perf-profile.children.cycles-pp.wake_up_new_task
0.46 ? 3% -0.1 0.40 ? 3% perf-profile.children.cycles-pp.mas_split
0.38 ? 4% -0.1 0.33 ? 5% perf-profile.children.cycles-pp.__do_sys_newstat
0.34 ? 6% -0.1 0.28 ? 5% perf-profile.children.cycles-pp.crypto_create_tfm_node
0.22 ? 8% -0.1 0.16 ? 18% perf-profile.children.cycles-pp.__do_sys_newfstat
0.30 ? 6% -0.1 0.24 ? 4% perf-profile.children.cycles-pp.try_to_wake_up
0.42 ? 4% -0.1 0.36 ? 11% perf-profile.children.cycles-pp.find_idlest_group
0.36 ? 5% -0.1 0.31 ? 5% perf-profile.children.cycles-pp.step_into
0.17 ? 12% -0.1 0.12 ? 18% perf-profile.children.cycles-pp.vfs_fstat
0.36 ? 6% -0.1 0.31 ? 3% perf-profile.children.cycles-pp.__getrlimit
0.32 ? 6% -0.1 0.26 ? 6% perf-profile.children.cycles-pp.jent_kcapi_init
0.32 ? 6% -0.1 0.26 ? 6% perf-profile.children.cycles-pp.jent_entropy_collector_alloc
0.27 ? 5% -0.1 0.22 ? 8% perf-profile.children.cycles-pp.touch_atime
0.15 ? 8% -0.1 0.10 ? 11% perf-profile.children.cycles-pp.security_file_alloc
0.32 ? 6% -0.1 0.27 ? 6% perf-profile.children.cycles-pp.rng_bind
0.20 ? 10% -0.1 0.15 ? 8% perf-profile.children.cycles-pp.wait_task_zombie
0.22 ? 5% -0.1 0.17 ? 10% perf-profile.children.cycles-pp.folio_add_lru_vma
0.16 ? 8% -0.0 0.11 ? 12% perf-profile.children.cycles-pp.lockref_put_or_lock
0.74 ? 4% -0.0 0.69 ? 3% perf-profile.children.cycles-pp.finish_fault
0.26 ? 8% -0.0 0.22 ? 7% perf-profile.children.cycles-pp.xas_find
0.18 ? 12% -0.0 0.13 ? 8% perf-profile.children.cycles-pp.find_unlink_vmap_area
0.16 ? 3% -0.0 0.11 ? 17% perf-profile.children.cycles-pp.__wake_up_common
0.26 ? 8% -0.0 0.21 ? 11% perf-profile.children.cycles-pp.__percpu_counter_sum
0.18 ? 8% -0.0 0.14 ? 14% perf-profile.children.cycles-pp._compound_head
0.49 ? 3% -0.0 0.44 ? 4% perf-profile.children.cycles-pp.vma_alloc_folio
0.20 ? 9% -0.0 0.15 ? 8% perf-profile.children.cycles-pp.__list_add_valid
0.23 ? 6% -0.0 0.18 ? 8% perf-profile.children.cycles-pp.unmap_single_vma
0.18 ? 12% -0.0 0.13 ? 6% perf-profile.children.cycles-pp.put_cred_rcu
0.10 ? 8% -0.0 0.06 ? 52% perf-profile.children.cycles-pp.mem_cgroup_update_lru_size
0.32 ? 2% -0.0 0.28 ? 3% perf-profile.children.cycles-pp.__mod_lruvec_page_state
0.18 ? 8% -0.0 0.13 ? 14% perf-profile.children.cycles-pp.do_wp_page
0.15 ? 3% -0.0 0.10 ? 13% perf-profile.children.cycles-pp.percpu_counter_destroy
0.12 ? 11% -0.0 0.08 ? 12% perf-profile.children.cycles-pp.apparmor_file_alloc_security
0.11 ? 9% -0.0 0.07 ? 11% perf-profile.children.cycles-pp.__unfreeze_partials
0.08 ? 14% -0.0 0.04 ? 71% perf-profile.children.cycles-pp.security_inode_getattr
0.17 ? 10% -0.0 0.13 ? 9% perf-profile.children.cycles-pp.release_task
0.30 ? 6% -0.0 0.26 ? 9% perf-profile.children.cycles-pp.wmemchr
0.37 ? 3% -0.0 0.33 ? 5% perf-profile.children.cycles-pp.___perf_sw_event
0.28 ? 7% -0.0 0.24 ? 5% perf-profile.children.cycles-pp.vm_area_dup
0.17 ? 12% -0.0 0.13 ? 10% perf-profile.children.cycles-pp.try_to_unlazy
0.15 ? 11% -0.0 0.11 ? 12% perf-profile.children.cycles-pp.free_percpu
0.31 ? 4% -0.0 0.27 ? 5% perf-profile.children.cycles-pp.mas_next_entry
0.16 ? 5% -0.0 0.12 ? 12% perf-profile.children.cycles-pp.__fxstatat64
0.12 ? 15% -0.0 0.08 ? 12% perf-profile.children.cycles-pp.atime_needs_update
0.15 ? 7% -0.0 0.12 ? 9% perf-profile.children.cycles-pp.userfaultfd_unmap_prep
0.23 ? 7% -0.0 0.20 ? 7% perf-profile.children.cycles-pp.pick_link
0.16 ? 8% -0.0 0.13 ? 5% perf-profile.children.cycles-pp.dcache_readdir
0.16 ? 10% -0.0 0.12 ? 6% perf-profile.children.cycles-pp.__crypto_xor
0.17 ? 8% -0.0 0.14 ? 9% perf-profile.children.cycles-pp.__pthread_initialize_minimal_internal
0.12 ? 8% -0.0 0.08 ? 8% perf-profile.children.cycles-pp.__lookup_slow
0.22 ? 5% -0.0 0.20 ? 7% perf-profile.children.cycles-pp.d_path
0.14 ? 4% -0.0 0.11 ? 12% perf-profile.children.cycles-pp.__do_sys_newfstatat
0.11 ? 12% -0.0 0.08 ? 12% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.22 ? 6% -0.0 0.19 ? 6% perf-profile.children.cycles-pp.vm_area_alloc
0.09 ? 8% -0.0 0.06 ? 13% perf-profile.children.cycles-pp.d_alloc_parallel
0.12 ? 8% -0.0 0.09 ? 15% perf-profile.children.cycles-pp.af_alg_get_rsgl
0.14 ? 4% -0.0 0.12 ? 10% perf-profile.children.cycles-pp.__mod_lruvec_state
0.12 ? 8% -0.0 0.09 ? 11% perf-profile.children.cycles-pp.dequeue_task_fair
0.18 ? 4% -0.0 0.16 ? 5% perf-profile.children.cycles-pp.folio_add_new_anon_rmap
0.14 ? 6% -0.0 0.12 ? 11% perf-profile.children.cycles-pp.enqueue_task_fair
0.13 ? 5% -0.0 0.11 ? 5% perf-profile.children.cycles-pp.folio_test_hugetlb
0.10 ? 4% -0.0 0.08 ? 14% perf-profile.children.cycles-pp.folio_mark_accessed
0.07 ? 11% -0.0 0.05 ? 7% perf-profile.children.cycles-pp.dentry_kill
0.08 ? 6% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.check_heap_object
0.10 ? 6% +0.0 0.12 ? 9% perf-profile.children.cycles-pp.irqtime_account_irq
0.09 ? 9% +0.0 0.12 ? 9% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.06 ? 9% +0.0 0.09 ? 12% perf-profile.children.cycles-pp.ct_idle_exit
0.08 ? 12% +0.0 0.11 ? 9% perf-profile.children.cycles-pp.finish_task_switch
0.12 ? 12% +0.0 0.15 ? 7% perf-profile.children.cycles-pp.tick_nohz_next_event
0.16 ? 4% +0.0 0.20 ? 10% perf-profile.children.cycles-pp.native_sched_clock
0.17 ? 10% +0.0 0.20 ? 11% perf-profile.children.cycles-pp.clockevents_program_event
0.11 ? 9% +0.0 0.15 ? 8% perf-profile.children.cycles-pp.sched_clock_cpu
0.16 ? 10% +0.1 0.22 ? 5% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.17 ? 45% +0.1 0.23 ? 8% perf-profile.children.cycles-pp.xts_decrypt
0.00 +0.1 0.06 ? 14% perf-profile.children.cycles-pp.camellia_decrypt_cbc_2way
0.11 ? 10% +0.1 0.18 ? 7% perf-profile.children.cycles-pp.read_tsc
0.00 +0.1 0.08 ? 10% perf-profile.children.cycles-pp.trace_module_notify
0.01 ?223% +0.1 0.09 ? 16% perf-profile.children.cycles-pp.shmem_read_folio_gfp
0.00 +0.1 0.08 ? 14% perf-profile.children.cycles-pp.__serpent_enc_blk8_avx
0.00 +0.1 0.08 ? 14% perf-profile.children.cycles-pp.serpent_ecb_enc_8way_avx
0.00 +0.1 0.09 ? 15% perf-profile.children.cycles-pp.serpent_ecb_dec_8way_avx
0.00 +0.1 0.09 ? 17% perf-profile.children.cycles-pp.shash_digest_unaligned
0.00 +0.1 0.09 ? 17% perf-profile.children.cycles-pp.shash_update_unaligned
0.00 +0.1 0.09 ? 17% perf-profile.children.cycles-pp._sha256_update
0.00 +0.1 0.09 ? 17% perf-profile.children.cycles-pp.sha256_base_do_update
0.00 +0.1 0.09 ? 17% perf-profile.children.cycles-pp.sha256_ni_transform
0.01 ?223% +0.1 0.10 ? 15% perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
0.01 ?223% +0.1 0.10 ? 16% perf-profile.children.cycles-pp.drm_gem_get_pages
0.01 ?223% +0.1 0.10 ? 16% perf-profile.children.cycles-pp.drm_gem_shmem_get_pages
0.01 ?223% +0.1 0.10 ? 16% perf-profile.children.cycles-pp.drm_gem_shmem_get_pages_locked
0.00 +0.1 0.09 ? 15% perf-profile.children.cycles-pp.drm_client_buffer_vmap
0.01 ?223% +0.1 0.11 ? 14% perf-profile.children.cycles-pp.shmem_get_folio_gfp
0.00 +0.1 0.10 ? 11% perf-profile.children.cycles-pp.llist_reverse_order
0.00 +0.1 0.10 ? 11% perf-profile.children.cycles-pp.mpihelp_addmul_1
0.00 +0.1 0.10 ? 10% perf-profile.children.cycles-pp.mpih_sqr_n_basecase
0.00 +0.1 0.11 ? 9% perf-profile.children.cycles-pp.blocking_notifier_call_chain
0.00 +0.1 0.11 ? 9% perf-profile.children.cycles-pp.notifier_call_chain
0.00 +0.1 0.11 ? 19% perf-profile.children.cycles-pp.early_mod_check
0.01 ?223% +0.1 0.12 ? 16% perf-profile.children.cycles-pp.drm_gem_shmem_object_vmap
0.01 ?223% +0.1 0.12 ? 16% perf-profile.children.cycles-pp.drm_gem_shmem_vmap_locked
0.00 +0.1 0.11 ? 13% perf-profile.children.cycles-pp.pkcs7_verify
0.00 +0.1 0.11 ? 13% perf-profile.children.cycles-pp.pkcs7_digest
0.01 ?223% +0.1 0.12 ? 15% perf-profile.children.cycles-pp.drm_gem_vmap_unlocked
0.01 ?223% +0.1 0.12 ? 15% perf-profile.children.cycles-pp.drm_gem_vmap
0.00 +0.1 0.12 ? 12% perf-profile.children.cycles-pp.__camellia_enc_blk_2way
0.00 +0.1 0.12 ? 19% perf-profile.children.cycles-pp.mutex_spin_on_owner
0.38 ? 45% +0.1 0.50 ? 3% perf-profile.children.cycles-pp.ecb_encrypt
0.00 +0.1 0.12 ? 11% perf-profile.children.cycles-pp.__serpent_dec_blk8_avx
0.39 ? 44% +0.1 0.52 ? 5% perf-profile.children.cycles-pp.ecb_decrypt
0.00 +0.1 0.13 ? 19% perf-profile.children.cycles-pp.free_module
0.00 +0.1 0.13 ? 13% perf-profile.children.cycles-pp.io_serial_out
0.03 ?103% +0.1 0.16 ? 12% perf-profile.children.cycles-pp.__mutex_lock
0.00 +0.1 0.13 ? 12% perf-profile.children.cycles-pp.mpih_sqr_n
0.00 +0.1 0.14 ? 14% perf-profile.children.cycles-pp.native_flush_tlb_global
0.00 +0.1 0.15 ? 15% perf-profile.children.cycles-pp.__flush_tlb_all
0.00 +0.2 0.15 ? 15% perf-profile.children.cycles-pp.fast_imageblit
0.00 +0.2 0.16 ? 15% perf-profile.children.cycles-pp.sys_imageblit
0.88 ? 8% +0.2 1.04 ? 4% perf-profile.children.cycles-pp.__intel_pmu_enable_all
0.00 +0.2 0.18 ? 7% perf-profile.children.cycles-pp.camellia_dec_blk_2way
1.59 ? 7% +0.2 1.77 ? 3% perf-profile.children.cycles-pp.perf_event_task_tick
1.57 ? 7% +0.2 1.75 ? 3% perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context
0.46 ? 6% +0.2 0.66 ? 3% perf-profile.children.cycles-pp.menu_select
0.00 +0.2 0.20 ? 15% perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit
0.00 +0.2 0.20 ? 17% perf-profile.children.cycles-pp.flush_tlb_kernel_range
0.00 +0.2 0.22 ? 17% perf-profile.children.cycles-pp.fbcon_putcs
0.00 +0.2 0.22 ? 17% perf-profile.children.cycles-pp.bit_putcs
0.00 +0.2 0.24 ? 16% perf-profile.children.cycles-pp.fbcon_redraw
0.00 +0.2 0.24 ? 14% perf-profile.children.cycles-pp.lf
0.00 +0.2 0.24 ? 14% perf-profile.children.cycles-pp.con_scroll
0.00 +0.2 0.24 ? 14% perf-profile.children.cycles-pp.fbcon_scroll
0.00 +0.2 0.25 ? 15% perf-profile.children.cycles-pp.vt_console_print
0.00 +0.3 0.30 ? 10% perf-profile.children.cycles-pp.mpihelp_submul_1
0.00 +0.3 0.33 ? 8% perf-profile.children.cycles-pp.__sysvec_call_function
0.00 +0.3 0.33 ? 8% perf-profile.children.cycles-pp.mpihelp_divrem
0.00 +0.4 0.36 ? 8% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
3.94 ? 6% +0.4 4.32 ? 2% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.00 +0.4 0.39 ? 30% perf-profile.children.cycles-pp.set_memory_ro
0.00 +0.4 0.40 ? 31% perf-profile.children.cycles-pp.module_enable_ro
0.00 +0.4 0.42 ? 9% perf-profile.children.cycles-pp.sysvec_call_function
0.00 +0.4 0.42 ? 28% perf-profile.children.cycles-pp.change_page_attr_set_clr
0.25 ? 5% +0.4 0.69 ? 4% perf-profile.children.cycles-pp.__memcpy
0.00 +0.4 0.45 ? 5% perf-profile.children.cycles-pp.drm_fbdev_generic_damage_blit_real
0.00 +0.5 0.46 ? 23% perf-profile.children.cycles-pp.__purge_vmap_area_lazy
0.00 +0.5 0.48 ? 6% perf-profile.children.cycles-pp.wait_for_xmitr
0.00 +0.5 0.48 ? 6% perf-profile.children.cycles-pp.mpi_powm
0.00 +0.5 0.48 ? 6% perf-profile.children.cycles-pp.pkcs1pad_verify
0.00 +0.5 0.48 ? 6% perf-profile.children.cycles-pp.rsa_enc
0.00 +0.5 0.49 ? 24% perf-profile.children.cycles-pp.on_each_cpu_cond_mask
0.00 +0.5 0.50 ? 5% perf-profile.children.cycles-pp.public_key_verify_signature
0.00 +0.5 0.50 ? 23% perf-profile.children.cycles-pp.smp_call_function_many_cond
0.00 +0.5 0.50 ? 5% perf-profile.children.cycles-pp.pkcs7_validate_trust
0.00 +0.5 0.50 ? 5% perf-profile.children.cycles-pp.pkcs7_validate_trust_one
0.00 +0.5 0.51 ? 18% perf-profile.children.cycles-pp._vm_unmap_aliases
0.00 +0.5 0.52 ? 8% perf-profile.children.cycles-pp.asm_sysvec_call_function
4.44 ? 5% +0.5 4.97 ? 2% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
4.42 ? 5% +0.5 4.95 ? 2% perf-profile.children.cycles-pp.hrtimer_interrupt
0.00 +0.6 0.61 ? 6% perf-profile.children.cycles-pp.verify_pkcs7_message_sig
0.00 +0.6 0.62 ? 6% perf-profile.children.cycles-pp.module_sig_check
0.00 +0.6 0.62 ? 6% perf-profile.children.cycles-pp.mod_verify_sig
0.00 +0.6 0.62 ? 6% perf-profile.children.cycles-pp.verify_pkcs7_signature
6.73 ? 5% +0.7 7.45 ? 4% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.01 ?223% +0.8 0.76 ? 21% perf-profile.children.cycles-pp.delay_tsc
7.07 ? 5% +0.8 7.89 ? 3% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.00 +1.3 1.33 ? 7% perf-profile.children.cycles-pp.load_module
0.14 ? 23% +2.4 2.50 ? 11% perf-profile.children.cycles-pp.io_serial_in
6.13 ? 2% +2.8 8.90 ? 2% perf-profile.children.cycles-pp.intel_idle
0.18 ? 18% +3.1 3.26 ? 7% perf-profile.children.cycles-pp.wait_for_lsr
0.18 ? 18% +3.2 3.40 ? 7% perf-profile.children.cycles-pp.serial8250_console_write
9.98 +3.4 13.36 perf-profile.children.cycles-pp.ret_from_fork
0.19 ? 18% +3.5 3.66 ? 7% perf-profile.children.cycles-pp.console_flush_all
0.19 ? 18% +3.5 3.66 ? 7% perf-profile.children.cycles-pp.console_unlock
0.00 +3.6 3.56 ? 8% perf-profile.children.cycles-pp.do_one_initcall
0.19 ? 18% +3.6 3.80 ? 7% perf-profile.children.cycles-pp._printk
0.19 ? 18% +3.6 3.80 ? 7% perf-profile.children.cycles-pp.vprintk_emit
0.00 +3.8 3.82 ? 8% perf-profile.children.cycles-pp.do_init_module
12.69 ? 3% +4.5 17.19 perf-profile.children.cycles-pp.cpuidle_enter_state
12.72 ? 3% +4.5 17.24 perf-profile.children.cycles-pp.cpuidle_enter
3.43 ? 3% +4.6 7.98 ? 3% perf-profile.children.cycles-pp.kthread
13.69 ? 3% +4.7 18.35 perf-profile.children.cycles-pp.start_secondary
2.68 ? 2% +4.7 7.34 ? 3% perf-profile.children.cycles-pp.worker_thread
2.38 ? 3% +4.7 7.05 ? 3% perf-profile.children.cycles-pp.process_one_work
0.05 ? 48% +4.7 4.73 ? 3% perf-profile.children.cycles-pp.ast_primary_plane_helper_atomic_update
0.05 ? 48% +4.7 4.73 ? 3% perf-profile.children.cycles-pp.drm_fb_memcpy
0.05 ? 48% +4.7 4.73 ? 3% perf-profile.children.cycles-pp.memcpy_toio
0.05 ? 48% +4.7 4.73 ? 3% perf-profile.children.cycles-pp.drm_atomic_helper_commit_planes
0.06 ? 47% +4.7 4.74 ? 3% perf-profile.children.cycles-pp.drm_atomic_helper_commit_tail_rpm
0.06 ? 21% +4.7 4.76 ? 3% perf-profile.children.cycles-pp.commit_tail
0.06 ? 21% +4.7 4.76 ? 3% perf-profile.children.cycles-pp.ast_mode_config_helper_atomic_commit_tail
0.08 ? 32% +4.7 4.80 ? 3% perf-profile.children.cycles-pp.drm_atomic_commit
0.08 ? 32% +4.7 4.79 ? 3% perf-profile.children.cycles-pp.drm_atomic_helper_commit
13.93 ? 3% +4.7 18.65 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
13.93 ? 3% +4.7 18.65 perf-profile.children.cycles-pp.cpu_startup_entry
0.08 ? 32% +4.7 4.80 ? 3% perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
13.92 ? 3% +4.7 18.65 perf-profile.children.cycles-pp.do_idle
13.34 ? 3% +4.8 18.10 perf-profile.children.cycles-pp.cpuidle_idle_call
0.00 +5.2 5.18 ? 7% perf-profile.children.cycles-pp.__do_sys_finit_module
0.00 +5.2 5.19 ? 7% perf-profile.children.cycles-pp.syscall
0.11 ? 20% +5.3 5.38 ? 3% perf-profile.children.cycles-pp.drm_fb_helper_damage_work
0.11 ? 20% +5.3 5.38 ? 3% perf-profile.children.cycles-pp.drm_fbdev_generic_helper_fb_dirty
8.53 ? 8% -1.8 6.76 ? 3% perf-profile.self.cycles-pp.jent_lfsr_time
9.90 ? 3% -1.6 8.32 ? 3% perf-profile.self.cycles-pp.next_uptodate_page
2.19 ? 5% -1.2 1.02 ? 17% perf-profile.self.cycles-pp.rwsem_spin_on_owner
1.30 ? 18% -0.8 0.48 ? 29% perf-profile.self.cycles-pp.osq_lock
2.99 ? 2% -0.6 2.37 perf-profile.self.cycles-pp.zap_pte_range
2.75 ? 4% -0.5 2.20 ? 6% perf-profile.self.cycles-pp.vma_interval_tree_insert
1.14 ? 3% -0.4 0.78 ? 6% perf-profile.self.cycles-pp.up_write
0.66 ? 13% -0.4 0.30 ? 11% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
1.54 ? 8% -0.3 1.23 ? 6% perf-profile.self.cycles-pp.jent_memaccess
0.95 ? 4% -0.3 0.67 ? 7% perf-profile.self.cycles-pp.release_pages
1.72 ? 2% -0.2 1.50 ? 2% perf-profile.self.cycles-pp._dl_addr
1.47 ? 5% -0.2 1.25 ? 6% perf-profile.self.cycles-pp.page_remove_rmap
0.94 ? 2% -0.2 0.73 ? 5% perf-profile.self.cycles-pp.vma_interval_tree_remove
0.68 -0.2 0.49 ? 9% perf-profile.self.cycles-pp.down_write
1.09 ? 4% -0.2 0.92 ? 6% perf-profile.self.cycles-pp.copy_page
0.82 ? 6% -0.2 0.67 ? 4% perf-profile.self.cycles-pp.filemap_map_pages
0.37 ? 4% -0.1 0.25 ? 11% perf-profile.self.cycles-pp.free_swap_cache
0.50 ? 4% -0.1 0.39 perf-profile.self.cycles-pp.stress_mwc8
0.18 ? 10% -0.1 0.08 ? 18% perf-profile.self.cycles-pp.rwsem_down_write_slowpath
0.29 ? 4% -0.1 0.20 ? 20% perf-profile.self.cycles-pp.__rb_insert_augmented
0.28 ? 5% -0.1 0.18 ? 17% perf-profile.self.cycles-pp.pcpu_alloc
0.67 ? 7% -0.1 0.58 ? 3% perf-profile.self.cycles-pp.mtree_range_walk
0.63 ? 3% -0.1 0.54 ? 5% perf-profile.self.cycles-pp.__slab_free
0.46 ? 6% -0.1 0.37 ? 8% perf-profile.self.cycles-pp.__list_del_entry_valid
0.41 ? 6% -0.1 0.32 ? 6% perf-profile.self.cycles-pp.mmap_region
0.16 ? 14% -0.1 0.08 ? 26% perf-profile.self.cycles-pp.__rb_erase_color
0.40 ? 4% -0.1 0.32 ? 7% perf-profile.self.cycles-pp.page_add_file_rmap
0.44 ? 4% -0.1 0.37 ? 7% perf-profile.self.cycles-pp.__d_lookup_rcu
0.52 ? 5% -0.1 0.44 ? 4% perf-profile.self.cycles-pp.native_flush_tlb_one_user
0.51 ? 4% -0.1 0.44 ? 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.30 ? 12% -0.1 0.24 ? 14% perf-profile.self.cycles-pp.stress_rndbuf
0.16 ? 7% -0.1 0.10 ? 14% perf-profile.self.cycles-pp.vma_interval_tree_augment_rotate
0.22 ? 10% -0.1 0.16 ? 8% perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
0.27 ? 7% -0.1 0.22 ? 9% perf-profile.self.cycles-pp.___slab_alloc
0.27 ? 5% -0.1 0.22 ? 12% perf-profile.self.cycles-pp.perf_event_mmap_output
0.20 ? 10% -0.1 0.14 ? 5% perf-profile.self.cycles-pp.__list_add_valid
0.12 ? 11% -0.0 0.07 ? 13% perf-profile.self.cycles-pp.apparmor_file_alloc_security
0.22 ? 7% -0.0 0.17 ? 8% perf-profile.self.cycles-pp.__percpu_counter_sum
0.07 ? 17% -0.0 0.02 ? 99% perf-profile.self.cycles-pp.apparmor_file_open
0.43 ? 3% -0.0 0.38 ? 5% perf-profile.self.cycles-pp.kmem_cache_alloc
0.15 ? 9% -0.0 0.10 ? 17% perf-profile.self.cycles-pp.lockref_put_or_lock
0.22 ? 10% -0.0 0.17 ? 8% perf-profile.self.cycles-pp.lock_vma_under_rcu
0.22 ? 9% -0.0 0.17 ? 9% perf-profile.self.cycles-pp.do_dentry_open
0.10 ? 8% -0.0 0.05 ? 50% perf-profile.self.cycles-pp.mem_cgroup_update_lru_size
0.30 ? 6% -0.0 0.26 ? 5% perf-profile.self.cycles-pp.kmem_cache_free_bulk
0.12 ? 7% -0.0 0.08 ? 10% perf-profile.self.cycles-pp.filemap_read
0.16 ? 9% -0.0 0.12 ? 16% perf-profile.self.cycles-pp._compound_head
0.32 ? 5% -0.0 0.28 ? 6% perf-profile.self.cycles-pp.___perf_sw_event
0.22 ? 8% -0.0 0.18 ? 12% perf-profile.self.cycles-pp.unmap_single_vma
0.13 ? 14% -0.0 0.09 ? 19% perf-profile.self.cycles-pp.__fput
0.12 ? 8% -0.0 0.08 ? 20% perf-profile.self.cycles-pp.alloc_vmap_area
0.28 ? 6% -0.0 0.24 ? 4% perf-profile.self.cycles-pp.mas_wr_node_store
0.09 ? 12% -0.0 0.06 ? 46% perf-profile.self.cycles-pp.xas_start
0.10 ? 11% -0.0 0.07 ? 17% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.10 ? 13% -0.0 0.06 ? 14% perf-profile.self.cycles-pp.__schedule
0.14 ? 9% -0.0 0.12 ? 6% perf-profile.self.cycles-pp.__crypto_xor
0.07 ? 14% -0.0 0.04 ? 45% perf-profile.self.cycles-pp.mas_spanning_rebalance
0.10 ? 9% -0.0 0.08 ? 10% perf-profile.self.cycles-pp.folio_test_hugetlb
0.07 ? 16% +0.0 0.10 ? 11% perf-profile.self.cycles-pp.cpuidle_idle_call
0.16 ? 6% +0.0 0.19 ? 8% perf-profile.self.cycles-pp.native_sched_clock
0.11 ? 12% +0.1 0.18 ? 7% perf-profile.self.cycles-pp.read_tsc
0.00 +0.1 0.08 ? 9% perf-profile.self.cycles-pp.trace_module_notify
0.00 +0.1 0.08 ? 14% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.00 +0.1 0.08 ? 14% perf-profile.self.cycles-pp.__serpent_enc_blk8_avx
0.00 +0.1 0.09 ? 17% perf-profile.self.cycles-pp.sha256_ni_transform
0.00 +0.1 0.09 ? 15% perf-profile.self.cycles-pp.llist_reverse_order
0.00 +0.1 0.10 ? 10% perf-profile.self.cycles-pp.mpihelp_addmul_1
0.20 ? 6% +0.1 0.30 ? 3% perf-profile.self.cycles-pp.menu_select
0.00 +0.1 0.12 ? 12% perf-profile.self.cycles-pp.__camellia_enc_blk_2way
0.00 +0.1 0.12 ? 20% perf-profile.self.cycles-pp.mutex_spin_on_owner
0.00 +0.1 0.12 ? 11% perf-profile.self.cycles-pp.__serpent_dec_blk8_avx
0.00 +0.1 0.13 ? 13% perf-profile.self.cycles-pp.io_serial_out
0.00 +0.1 0.14 ? 23% perf-profile.self.cycles-pp.vprintk_emit
0.00 +0.1 0.14 ? 14% perf-profile.self.cycles-pp.native_flush_tlb_global
0.00 +0.2 0.15 ? 15% perf-profile.self.cycles-pp.fast_imageblit
0.88 ? 8% +0.2 1.04 ? 4% perf-profile.self.cycles-pp.__intel_pmu_enable_all
0.00 +0.2 0.18 ? 5% perf-profile.self.cycles-pp.camellia_dec_blk_2way
0.34 ? 5% +0.2 0.56 ? 3% perf-profile.self.cycles-pp.cpuidle_enter_state
0.00 +0.3 0.29 ? 10% perf-profile.self.cycles-pp.mpihelp_submul_1
0.24 ? 6% +0.4 0.68 ? 4% perf-profile.self.cycles-pp.__memcpy
0.00 +0.5 0.46 ? 25% perf-profile.self.cycles-pp.smp_call_function_many_cond
0.01 ?223% +0.8 0.76 ? 21% perf-profile.self.cycles-pp.delay_tsc
0.14 ? 23% +2.4 2.49 ? 11% perf-profile.self.cycles-pp.io_serial_in
6.13 ? 2% +2.8 8.90 ? 2% perf-profile.self.cycles-pp.intel_idle
0.05 ? 48% +4.7 4.71 ? 3% perf-profile.self.cycles-pp.memcpy_toio




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki



Attachments:
(No filename) (89.88 kB)
config-6.4.0-rc2-00017-gb81fac906a8f (161.15 kB)
job-script (8.87 kB)
job.yaml (6.34 kB)
reproduce (551.00 B)
kmsg.xz (38.54 kB)
Download all attachments

2023-07-31 12:46:29

by Feng Tang

[permalink] [raw]
Subject: Re: [linus:master] [x86/fpu] b81fac906a: stress-ng.af-alg.ops_per_sec -34.6% regression

On Wed, Jul 19, 2023 at 10:07:52PM +0800, kernel test robot wrote:
>
>
> Hello,
>
> kernel test robot noticed a -34.6% regression of stress-ng.af-alg.ops_per_sec on:
>
>
> commit: b81fac906a8f9e682e513ddd95697ec7a20878d4 ("x86/fpu: Move FPU initialization into arch_cpu_finalize_init()")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> still regression on:
> fix commit: fe3e0a13e597c1c8617814bf9b42ab732db5c26e
> linus/master: ccff6d117d8dc8d8d86e8695a75e5f8b01e573bf
>
> testcase: stress-ng
> test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> parameters:
>
> nr_threads: 10%
> disk: 1HDD
> testtime: 60s
> fs: ext4
> class: os
> test: af-alg
> cpufreq_governor: performance
>
>
> besides, we noticed there are below information in dmesg (attached kmsg.xz),
> which we didn't see in dmesg for parent:
>
> kern :info : [ 65.660392] AVX or AES-NI instructions are not detected.
> user :notice: [ 65.661743] stress-ng: info: [4738] af-alg: 9 cryptographic algorithms are internal and may be unused
>
> kern :info : [ 65.720205] AVX or AES-NI instructions are not detected.
> kern :info : [ 65.820334] AVX2 instructions are not detected.
> kern :info : [ 66.130644] AVX or AES-NI instructions are not detected.
> kern :info : [ 66.167749] AVX or AES-NI instructions are not detected.
> kern :info : [ 66.213178] AVX2 instructions are not detected.
> kern :info : [ 66.318235] AVX or AES-NI instructions are not detected.
> kern :info : [ 66.360766] AVX or AES-NI instructions are not detected.
> kern :info : [ 66.437318] AVX2 instructions are not detected.
> kern :info : [ 66.683666] AVX or AES-NI instructions are not detected.
> user :notice: [ 66.713976] BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
>
> kern :info : [ 66.738790] AVX or AES-NI instructions are not detected.
> kern :info : [ 66.822332] AVX2 instructions are not detected.
> kern :info : [ 67.083360] AVX or AES-NI instructions are not detected.
> kern :info : [ 67.122804] AVX or AES-NI instructions are not detected.

I think the regression is related with the commit, that commit
b81fac906a8f changes the FPU init order, and cause the 'X86_FEATURE_OSXSAVE'
missed the cpu capability setup, thus all later check of
boot_cpu_has(X86_FEATURE_OSXSAVE) will return false.

arch_cpu_finalize_init
identify_boot_cpu
identify_cpu
generic_identify
get_cpu_cap --> setup cpu capability
...
fpu__init_cpu
fpu__init_cpu_xstate
cr4_set_bits(X86_CR4_OSXSAVE);

Many security moduels in arch/x86 have dependency on X86_FEATURE_OSXSAVE,
and will fail to load later. While this 'af-alg' case involves secuirty
operation and tries to load those modules but fail, which cause the
regression.

Setting the 'OSXSVAE' feature bit after OSXSAVE enabling can restore
the regression, like the below RFC patch. pls help to review, thanks!

- Feng

---

From 62d05a24cae9198b0ed45a8857edbe4c9c4184f9 Mon Sep 17 00:00:00 2001
From: Feng Tang <[email protected]>
Date: Mon, 31 Jul 2023 16:45:34 +0800
Subject: [PATCH RFC] x86/fpu: set X86_FEATURE_OSXSAVE feature after enabling
OSXSAVE in CR4

Commit b81fac906a8f ("x86/fpu: Move FPU initialization into
arch_cpu_finalize_init()" optimized the FPU init order, and moves the
CR4_OSXSAVE enabling into a later flace.

arch_cpu_finalize_init
identify_boot_cpu
identify_cpu
generic_identify
get_cpu_cap --> setup cpu capability
...
fpu__init_cpu
fpu__init_cpu_xstate
cr4_set_bits(X86_CR4_OSXSAVE);

This makes 'X86_FEATURE_OSXSAVE' missed in cpu capability setup. Many
security module like 'camellia_aesni_avx_x86_64' depends on this feature
and will fail to load after the commit. 0Day caught a 34% regression
in stress-ng's 'af-alg' test case [1].

So set X86_FEATURE_OSXSAVE feature after OSXSAVE enabling to fix it.

[1]. https://lore.kernel.org/lkml/[email protected]/

Fixes: b81fac906a8f ("x86/fpu: Move FPU initialization into arch_cpu_finalize_init()")
Reported-by: kernel test robot <[email protected]>
Signed-off-by: Feng Tang <[email protected]>
---
arch/x86/kernel/fpu/xstate.c | 3 +++
1 file changed, 3 insertions(+)

diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c
index 0bab497c9436..8ebea0d522d2 100644
--- a/arch/x86/kernel/fpu/xstate.c
+++ b/arch/x86/kernel/fpu/xstate.c
@@ -173,6 +173,9 @@ void fpu__init_cpu_xstate(void)

cr4_set_bits(X86_CR4_OSXSAVE);

+ if (!boot_cpu_has(X86_FEATURE_OSXSAVE))
+ setup_force_cpu_cap(X86_FEATURE_OSXSAVE);
+
/*
* Must happen after CR4 setup and before xsetbv() to allow KVM
* lazy passthrough. Write independent of the dynamic state static
--
2.27.0