Greeting,
FYI, we noticed a -29.4% regression of vm-scalability.throughput due to commit:
commit: ecf8443e51a862b261313c2319ab4e4aed9e6b7e ("[PATCH v7 02/11] mm/mremap: Fix race between MOVE_PUD mremap and pageout")
url: https://github.com/0day-ci/linux/commits/Aneesh-Kumar-K-V/Speedup-mremap-on-ppc64/20210607-135424
base: https://git.kernel.org/cgit/linux/kernel/git/powerpc/linux.git next
in testcase: vm-scalability
on test machine: 192 threads 4 sockets Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
with following parameters:
runtime: 300
thp_enabled: never
thp_defrag: always
nr_task: 8
nr_ssd: 1
test: swap-w-seq-mt
cpufreq_governor: performance
ucode: 0x4003006
test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
bin/lkp run generated-yaml-file
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_ssd/nr_task/rootfs/runtime/tbox_group/test/testcase/thp_defrag/thp_enabled/ucode:
gcc-9/performance/x86_64-rhel-8.3/1/8/debian-10.4-x86_64-20200603.cgz/300/lkp-csl-2ap1/swap-w-seq-mt/vm-scalability/always/never/0x4003006
commit:
5f80ee2fc0 ("mm/mremap: Fix race between MOVE_PMD mremap and pageout")
ecf8443e51 ("mm/mremap: Fix race between MOVE_PUD mremap and pageout")
5f80ee2fc08b3613 ecf8443e51a862b261313c2319a
---------------- ---------------------------
%stddev %change %stddev
\ | \
7.93 ? 4% +14.6% 9.08 ? 5% vm-scalability.free_time
373210 -29.4% 263582 ? 3% vm-scalability.median
43.45 ? 5% -8.1 35.34 ? 7% vm-scalability.stddev%
2954757 -29.4% 2085715 ? 3% vm-scalability.throughput
220.03 +38.3% 304.37 ? 2% vm-scalability.time.elapsed_time
220.03 +38.3% 304.37 ? 2% vm-scalability.time.elapsed_time.max
32410 ? 2% +29.9% 42117 ? 10% vm-scalability.time.involuntary_context_switches
528.33 +13.1% 597.67 vm-scalability.time.percent_of_cpu_this_job_got
913.19 ? 2% +70.7% 1558 ? 3% vm-scalability.time.system_time
250.22 +4.7% 261.96 vm-scalability.time.user_time
34211 ? 11% -95.9% 1400 ? 38% vm-scalability.time.voluntary_context_switches
4.82 ? 3% +15.8% 5.57 ? 3% iostat.cpu.system
3.77e+08 ? 24% +46.4% 5.52e+08 ? 19% turbostat.IRQ
271.03 +30.2% 352.88 ? 2% uptime.boot
48496 +30.1% 63095 ? 2% uptime.idle
0.66 ? 12% -0.7 0.00 ? 57% mpstat.cpu.all.iowait%
0.06 ? 4% -0.0 0.06 ? 4% mpstat.cpu.all.soft%
3.09 ? 2% +0.9 4.03 ? 3% mpstat.cpu.all.sys%
0.60 -0.1 0.48 ? 4% mpstat.cpu.all.usr%
28255 ? 5% +77.2% 50067 ? 6% meminfo.Active
27866 ? 5% +78.2% 49655 ? 6% meminfo.Active(anon)
28180965 -20.0% 22548765 ? 3% meminfo.MemAvailable
28675444 -19.7% 23016466 ? 3% meminfo.MemFree
38693 ? 5% +56.1% 60412 ? 2% meminfo.Shmem
496106 ? 7% +2180.0% 11311305 ? 12% meminfo.SwapCached
60640 ? 3% -97.9% 1245 ? 46% meminfo.Writeback
87575 ? 3% -43.8% 49246 ? 8% slabinfo.dmaengine-unmap-16.active_objs
2123 ? 3% -44.5% 1179 ? 8% slabinfo.dmaengine-unmap-16.active_slabs
89210 ? 3% -44.5% 49532 ? 8% slabinfo.dmaengine-unmap-16.num_objs
2123 ? 3% -44.5% 1179 ? 8% slabinfo.dmaengine-unmap-16.num_slabs
4866 -20.4% 3874 ? 3% slabinfo.kmalloc-4k.active_objs
623.83 -20.9% 493.50 ? 3% slabinfo.kmalloc-4k.active_slabs
4992 -20.8% 3952 ? 3% slabinfo.kmalloc-4k.num_objs
623.83 -20.9% 493.50 ? 3% slabinfo.kmalloc-4k.num_slabs
374.17 ? 2% -47.8% 195.17 ? 53% vmstat.io.bi
1341570 -23.6% 1025527 ? 2% vmstat.io.bo
28897122 -21.2% 22757749 ? 2% vmstat.memory.free
1.238e+08 +8.1% 1.339e+08 ? 2% vmstat.memory.swpd
1.00 -100.0% 0.00 vmstat.procs.b
8.00 +29.2% 10.33 ? 7% vmstat.procs.r
364.17 ? 2% -48.3% 188.17 ? 55% vmstat.swap.si
1341561 -23.6% 1025520 ? 2% vmstat.swap.so
3328 ? 2% -19.9% 2666 vmstat.system.cs
1511984 +17.7% 1780129 vmstat.system.in
14215 ? 15% -98.5% 209.67 ? 84% numa-meminfo.node0.Writeback
308351 ?116% +1024.4% 3467064 ? 11% numa-meminfo.node1.FilePages
168771 ? 6% -15.1% 143367 ? 12% numa-meminfo.node1.KReclaimable
168771 ? 6% -15.1% 143367 ? 12% numa-meminfo.node1.SReclaimable
127034 ? 32% +2618.6% 3453577 ? 10% numa-meminfo.node1.SwapCached
14382 ? 9% -97.7% 332.00 ? 34% numa-meminfo.node1.Writeback
124583 ? 35% +2602.3% 3366554 ? 14% numa-meminfo.node2.FilePages
77286 ? 26% +55.8% 120430 ? 18% numa-meminfo.node2.KReclaimable
77286 ? 26% +55.8% 120430 ? 18% numa-meminfo.node2.SReclaimable
138348 ? 20% +28.1% 177225 ? 13% numa-meminfo.node2.Slab
113067 ? 23% +2874.7% 3363404 ? 14% numa-meminfo.node2.SwapCached
14263 ? 5% -97.4% 373.17 ? 37% numa-meminfo.node2.Writeback
13835 ? 13% +155.7% 35375 ? 18% numa-meminfo.node3.Active
13835 ? 13% +155.2% 35308 ? 18% numa-meminfo.node3.Active(anon)
261289 ?137% +1236.3% 3491534 ? 7% numa-meminfo.node3.FilePages
16541 ? 12% +130.9% 38192 ? 21% numa-meminfo.node3.Shmem
81973 ? 30% +3903.0% 3281379 ? 9% numa-meminfo.node3.SwapCached
16321 ? 6% -97.7% 372.33 ? 42% numa-meminfo.node3.Writeback
3643 ? 14% -98.5% 55.00 ?102% numa-vmstat.node0.nr_writeback
3619 ? 14% -98.6% 49.67 ?107% numa-vmstat.node0.nr_zone_write_pending
77012 ?116% +1039.9% 877855 ? 10% numa-vmstat.node1.nr_file_pages
42097 ? 6% -14.0% 36208 ? 10% numa-vmstat.node1.nr_slab_reclaimable
31679 ? 32% +2660.4% 874478 ? 10% numa-vmstat.node1.nr_swapcached
11014726 ? 2% +12.6% 12401236 ? 4% numa-vmstat.node1.nr_vmscan_write
3577 ? 10% -97.5% 89.83 ? 41% numa-vmstat.node1.nr_writeback
11011253 ? 2% +12.6% 12401174 ? 4% numa-vmstat.node1.nr_written
3502 ? 10% -97.7% 82.17 ? 49% numa-vmstat.node1.nr_zone_write_pending
31302 ? 35% +2617.2% 850555 ? 13% numa-vmstat.node2.nr_file_pages
19271 ? 26% +57.8% 30407 ? 17% numa-vmstat.node2.nr_slab_reclaimable
28421 ? 24% +2889.9% 849766 ? 12% numa-vmstat.node2.nr_swapcached
11041559 ? 3% +19.1% 13149436 ? 10% numa-vmstat.node2.nr_vmscan_write
3782 ? 4% -97.6% 92.00 ? 34% numa-vmstat.node2.nr_writeback
11037863 ? 3% +19.1% 13149374 ? 10% numa-vmstat.node2.nr_written
3724 ? 4% -97.9% 79.00 ? 39% numa-vmstat.node2.nr_zone_write_pending
3443 ? 14% +160.0% 8954 ? 20% numa-vmstat.node3.nr_active_anon
65304 ?137% +1252.7% 883364 ? 6% numa-vmstat.node3.nr_file_pages
4129 ? 12% +134.5% 9682 ? 23% numa-vmstat.node3.nr_shmem
20479 ? 30% +3956.2% 830690 ? 8% numa-vmstat.node3.nr_swapcached
10829696 ? 5% +14.0% 12346461 ? 4% numa-vmstat.node3.nr_vmscan_write
4124 ? 4% -97.6% 97.00 ? 39% numa-vmstat.node3.nr_writeback
10825671 ? 5% +14.0% 12346386 ? 4% numa-vmstat.node3.nr_written
3444 ? 14% +160.0% 8955 ? 20% numa-vmstat.node3.nr_zone_active_anon
4055 ? 4% -97.9% 83.67 ? 43% numa-vmstat.node3.nr_zone_write_pending
8412 ? 3% +28.0% 10768 ? 13% proc-vmstat.allocstall_movable
5996587 ? 46% +183.4% 16993010 ? 32% proc-vmstat.compact_free_scanned
3451053 ? 29% +75.7% 6064315 ? 15% proc-vmstat.compact_isolated
3259087 ? 27% +89.7% 6184085 ? 15% proc-vmstat.compact_migrate_scanned
6961 ? 5% +78.5% 12425 ? 6% proc-vmstat.nr_active_anon
40993758 -3.1% 39718983 proc-vmstat.nr_anon_pages
698799 -19.8% 560310 ? 3% proc-vmstat.nr_dirty_background_threshold
1399308 -19.8% 1121992 ? 3% proc-vmstat.nr_dirty_threshold
388625 ? 2% +695.9% 3092997 ? 11% proc-vmstat.nr_file_pages
7165507 -19.3% 5779541 ? 3% proc-vmstat.nr_free_pages
41119695 +3.5% 42542747 proc-vmstat.nr_inactive_anon
145036 +3.1% 149568 proc-vmstat.nr_page_table_pages
9675 ? 5% +56.0% 15094 ? 2% proc-vmstat.nr_shmem
123879 ? 8% +2178.6% 2822769 ? 12% proc-vmstat.nr_swapcached
38677277 +7.7% 41656286 ? 3% proc-vmstat.nr_vmscan_write
14819 ? 3% -97.9% 307.33 ? 50% proc-vmstat.nr_writeback
74789620 +5.5% 78882933 ? 2% proc-vmstat.nr_written
6962 ? 5% +78.5% 12426 ? 6% proc-vmstat.nr_zone_active_anon
41119448 +3.5% 42542544 proc-vmstat.nr_zone_inactive_anon
14582 ? 3% -98.1% 274.50 ? 47% proc-vmstat.nr_zone_write_pending
20967960 ? 16% +27.7% 26777776 ? 10% proc-vmstat.numa_pte_updates
16061 ? 13% +34.4% 21592 ? 7% proc-vmstat.pgactivate
1726355 ? 29% +75.7% 3033559 ? 15% proc-vmstat.pgmigrate_success
2.992e+08 +5.5% 3.155e+08 ? 2% proc-vmstat.pgpgout
48003 ? 2% +32.3% 63514 ? 3% proc-vmstat.pgreuse
74393768 -12.6% 65057070 ? 4% proc-vmstat.pgrotated
2.36e+08 +8.4% 2.558e+08 proc-vmstat.pgscan_anon
1.048e+08 ? 2% +24.6% 1.305e+08 ? 6% proc-vmstat.pgscan_direct
74792004 +5.5% 78884465 ? 2% proc-vmstat.pswpout
50067 ? 7% +17.0% 58567 ? 4% proc-vmstat.slabs_scanned
4.724e+09 -18.5% 3.85e+09 perf-stat.i.branch-instructions
3231 ? 2% -19.3% 2608 perf-stat.i.context-switches
1.86 ? 2% +47.3% 2.74 ? 5% perf-stat.i.cpi
3.281e+10 ? 2% +15.9% 3.803e+10 ? 4% perf-stat.i.cpu-cycles
206.24 -1.9% 202.23 perf-stat.i.cpu-migrations
642.02 ? 6% +14.2% 732.98 ? 8% perf-stat.i.cycles-between-cache-misses
4.851e+09 -17.3% 4.014e+09 perf-stat.i.dTLB-loads
3304908 -19.0% 2677570 ? 4% perf-stat.i.dTLB-store-misses
2.027e+09 -19.4% 1.633e+09 perf-stat.i.dTLB-stores
5484115 ? 3% -14.5% 4691022 ? 3% perf-stat.i.iTLB-load-misses
1.857e+10 -17.5% 1.533e+10 perf-stat.i.instructions
5181 ? 3% -16.9% 4306 ? 2% perf-stat.i.instructions-per-iTLB-miss
0.59 ? 3% -24.6% 0.44 ? 3% perf-stat.i.ipc
23.06 ? 6% -49.2% 11.72 ? 54% perf-stat.i.major-faults
0.17 ? 2% +15.9% 0.20 ? 4% perf-stat.i.metric.GHz
60.43 -18.1% 49.46 perf-stat.i.metric.M/sec
585912 -26.9% 428491 ? 2% perf-stat.i.minor-faults
87.62 +3.0 90.65 perf-stat.i.node-load-miss-rate%
1450164 ? 6% -29.1% 1027918 ? 3% perf-stat.i.node-loads
83.34 +4.1 87.48 perf-stat.i.node-store-miss-rate%
1130049 ? 10% -24.5% 853398 ? 11% perf-stat.i.node-stores
585935 -26.9% 428503 ? 2% perf-stat.i.page-faults
1.77 ? 2% +42.5% 2.52 ? 5% perf-stat.overall.cpi
560.05 ? 6% +22.0% 683.28 ? 11% perf-stat.overall.cycles-between-cache-misses
0.56 ? 2% -29.7% 0.40 ? 5% perf-stat.overall.ipc
91.72 +2.0 93.70 perf-stat.overall.node-load-miss-rate%
86.56 +3.1 89.65 perf-stat.overall.node-store-miss-rate%
7112 +13.7% 8086 perf-stat.overall.path-length
4.703e+09 -18.7% 3.821e+09 ? 2% perf-stat.ps.branch-instructions
3220 ? 2% -19.7% 2587 perf-stat.ps.context-switches
3.277e+10 ? 2% +17.3% 3.843e+10 ? 5% perf-stat.ps.cpu-cycles
205.01 -2.7% 199.55 ? 2% perf-stat.ps.cpu-migrations
4.834e+09 -17.5% 3.987e+09 perf-stat.ps.dTLB-loads
3283502 -19.6% 2639256 ? 4% perf-stat.ps.dTLB-store-misses
2.02e+09 -19.9% 1.618e+09 perf-stat.ps.dTLB-stores
5471166 ? 3% -15.0% 4652576 ? 2% perf-stat.ps.iTLB-load-misses
1.85e+10 -17.7% 1.523e+10 perf-stat.ps.instructions
22.79 ? 6% -49.2% 11.57 ? 54% perf-stat.ps.major-faults
581983 -27.5% 422213 ? 2% perf-stat.ps.minor-faults
1442428 ? 6% -29.8% 1013002 ? 3% perf-stat.ps.node-loads
1123088 ? 10% -25.3% 839225 ? 10% perf-stat.ps.node-stores
582006 -27.5% 422225 ? 2% perf-stat.ps.page-faults
4.085e+12 +13.6% 4.642e+12 perf-stat.total.instructions
0.01 ? 23% -49.2% 0.01 ? 13% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.55 ?221% -99.5% 0.00 perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.02 ? 32% -100.0% 0.00 perf-sched.sch_delay.avg.ms.kswapd.kthread.ret_from_fork
0.01 ? 62% -100.0% 0.00 perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
0.01 ? 17% -47.2% 0.00 ? 10% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
0.02 ? 21% -49.1% 0.01 ? 22% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.02 ? 37% -60.0% 0.01 ? 34% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
166.76 ?223% -100.0% 0.01 ? 10% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.03 ? 67% -62.5% 0.01 ? 51% perf-sched.sch_delay.max.ms.io_schedule.__lock_page_or_retry.do_swap_page.__handle_mm_fault
0.03 ?130% -77.5% 0.01 ? 61% perf-sched.sch_delay.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead
0.02 ? 39% -100.0% 0.00 perf-sched.sch_delay.max.ms.kswapd.kthread.ret_from_fork
0.09 ? 59% -85.5% 0.01 ? 10% perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
0.05 ? 77% -89.3% 0.01 ? 56% perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
0.02 ? 96% -100.0% 0.00 perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
0.02 ? 36% -62.8% 0.01 ? 21% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
1018 ? 14% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1018 ? 14% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
15.36 ? 84% -97.3% 0.42 ? 78% perf-sched.wait_and_delay.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
2042 ? 37% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
8.00 ? 28% -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
8.00 ? 28% -100.0% 0.00 perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
93.67 ?133% +261.4% 338.50 ? 12% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.__alloc_pages_slowpath.constprop.0
20.17 ?223% +547.1% 130.50 ? 10% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.mempool_alloc.bio_alloc_bioset.__swap_writepage
346.50 ? 74% +194.0% 1018 ? 10% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.shrink_node.do_try_to_free_pages.try_to_free_pages
9.00 ? 19% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
23.00 ? 5% +24.6% 28.67 ? 10% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
5996 ? 31% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
5996 ? 31% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
4492 ? 79% -99.9% 3.21 ? 71% perf-sched.wait_and_delay.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
6025 ? 32% -82.0% 1082 ? 4% perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
6024 ? 32% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
1018 ? 14% -99.9% 1.33 ? 15% perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
1018 ? 14% -99.8% 1.60 ? 57% perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
309.68 ?143% -100.0% 0.01 ? 27% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
224.46 ?171% -99.9% 0.13 ?110% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
378.18 ? 74% -99.8% 0.69 ? 48% perf-sched.wait_time.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead
15.36 ? 84% -97.3% 0.42 ? 78% perf-sched.wait_time.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
105.18 -100.0% 0.00 perf-sched.wait_time.avg.ms.kswapd.kthread.ret_from_fork
57.26 ? 71% -100.0% 0.00 perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
2.81 ? 22% -68.2% 0.89 ? 45% perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
754.86 ?100% -100.0% 0.00 perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
298.39 ?141% -100.0% 0.00 perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
2042 ? 37% -100.0% 0.37 ? 99% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.00 ? 10% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kswapd.kthread.ret_from_fork
5996 ? 31% -100.0% 2.66 ? 15% perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
5996 ? 31% -100.0% 2.56 ? 22% perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
1179 ?142% -100.0% 0.03 ? 89% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
818.62 ?149% -100.0% 0.13 ?110% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
4236 ? 54% -99.9% 3.55 ? 58% perf-sched.wait_time.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead
4492 ? 79% -99.9% 3.21 ? 71% perf-sched.wait_time.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
107.44 -100.0% 0.00 perf-sched.wait_time.max.ms.kswapd.kthread.ret_from_fork
6025 ? 32% -82.0% 1082 ? 4% perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
2367 ? 71% -100.0% 0.00 perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
5.00 -56.8% 2.16 ? 93% perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
1888 ?101% -100.0% 0.00 perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
1240 ?140% -100.0% 0.00 perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
6024 ? 32% -100.0% 0.74 ?107% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
0.01 ? 13% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kswapd.kthread.ret_from_fork
443.33 +38.2% 612.67 ? 2% interrupts.9:IO-APIC.9-fasteoi.acpi
2.506e+08 ? 2% +71.0% 4.285e+08 ? 2% interrupts.CAL:Function_call_interrupts
436770 +39.6% 609678 ? 3% interrupts.CPU0.LOC:Local_timer_interrupts
443.33 +38.2% 612.67 ? 2% interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
436473 +39.6% 609534 ? 3% interrupts.CPU1.LOC:Local_timer_interrupts
436112 +39.7% 609460 ? 3% interrupts.CPU10.LOC:Local_timer_interrupts
435885 +39.8% 609476 ? 3% interrupts.CPU100.LOC:Local_timer_interrupts
435912 +39.8% 609519 ? 3% interrupts.CPU101.LOC:Local_timer_interrupts
435983 +39.8% 609496 ? 3% interrupts.CPU102.LOC:Local_timer_interrupts
436229 +39.7% 609544 ? 3% interrupts.CPU103.LOC:Local_timer_interrupts
152.33 ? 49% +944.5% 1591 ? 79% interrupts.CPU103.NMI:Non-maskable_interrupts
152.33 ? 49% +944.5% 1591 ? 79% interrupts.CPU103.PMI:Performance_monitoring_interrupts
436056 +39.8% 609461 ? 3% interrupts.CPU104.LOC:Local_timer_interrupts
435861 +39.8% 609478 ? 3% interrupts.CPU105.LOC:Local_timer_interrupts
436021 +39.8% 609379 ? 3% interrupts.CPU106.LOC:Local_timer_interrupts
435843 +39.9% 609554 ? 3% interrupts.CPU107.LOC:Local_timer_interrupts
435999 +39.8% 609466 ? 3% interrupts.CPU108.LOC:Local_timer_interrupts
436372 +39.7% 609463 ? 3% interrupts.CPU109.LOC:Local_timer_interrupts
435946 +39.8% 609652 ? 3% interrupts.CPU11.LOC:Local_timer_interrupts
436096 +39.8% 609463 ? 3% interrupts.CPU110.LOC:Local_timer_interrupts
435856 +39.8% 609436 ? 3% interrupts.CPU111.LOC:Local_timer_interrupts
436180 +39.7% 609470 ? 3% interrupts.CPU112.LOC:Local_timer_interrupts
435902 +39.8% 609466 ? 3% interrupts.CPU113.LOC:Local_timer_interrupts
435823 +39.9% 609521 ? 3% interrupts.CPU114.LOC:Local_timer_interrupts
435831 +39.8% 609416 ? 3% interrupts.CPU115.LOC:Local_timer_interrupts
435716 +39.9% 609492 ? 3% interrupts.CPU116.LOC:Local_timer_interrupts
435934 +39.8% 609518 ? 3% interrupts.CPU117.LOC:Local_timer_interrupts
435946 +39.8% 609459 ? 3% interrupts.CPU118.LOC:Local_timer_interrupts
435735 +39.9% 609492 ? 3% interrupts.CPU119.LOC:Local_timer_interrupts
436018 +39.8% 609544 ? 3% interrupts.CPU12.LOC:Local_timer_interrupts
1730294 ? 54% +231.1% 5729689 ? 73% interrupts.CPU120.CAL:Function_call_interrupts
436513 +38.6% 605176 ? 4% interrupts.CPU120.LOC:Local_timer_interrupts
437125 +38.4% 605185 ? 4% interrupts.CPU121.LOC:Local_timer_interrupts
119.00 ? 49% +2582.6% 3192 ? 78% interrupts.CPU121.NMI:Non-maskable_interrupts
119.00 ? 49% +2582.6% 3192 ? 78% interrupts.CPU121.PMI:Performance_monitoring_interrupts
436783 +38.6% 605196 ? 4% interrupts.CPU122.LOC:Local_timer_interrupts
436134 +38.8% 605202 ? 4% interrupts.CPU123.LOC:Local_timer_interrupts
436402 +38.7% 605173 ? 4% interrupts.CPU124.LOC:Local_timer_interrupts
436347 +38.7% 605194 ? 4% interrupts.CPU125.LOC:Local_timer_interrupts
436510 +38.7% 605240 ? 4% interrupts.CPU126.LOC:Local_timer_interrupts
436805 +38.5% 605169 ? 4% interrupts.CPU127.LOC:Local_timer_interrupts
436659 +38.6% 605187 ? 4% interrupts.CPU128.LOC:Local_timer_interrupts
436231 +38.7% 605179 ? 4% interrupts.CPU129.LOC:Local_timer_interrupts
436500 +39.6% 609536 ? 3% interrupts.CPU13.LOC:Local_timer_interrupts
436424 +38.7% 605184 ? 4% interrupts.CPU130.LOC:Local_timer_interrupts
436297 +38.7% 605194 ? 4% interrupts.CPU131.LOC:Local_timer_interrupts
436562 +38.6% 605189 ? 4% interrupts.CPU132.LOC:Local_timer_interrupts
436922 +38.5% 605198 ? 4% interrupts.CPU133.LOC:Local_timer_interrupts
436581 +38.6% 605204 ? 4% interrupts.CPU134.LOC:Local_timer_interrupts
436103 +38.8% 605178 ? 4% interrupts.CPU135.LOC:Local_timer_interrupts
436818 +38.5% 605171 ? 4% interrupts.CPU136.LOC:Local_timer_interrupts
436327 +38.7% 605173 ? 4% interrupts.CPU137.LOC:Local_timer_interrupts
436190 +38.7% 605185 ? 4% interrupts.CPU138.LOC:Local_timer_interrupts
436439 +38.7% 605194 ? 4% interrupts.CPU139.LOC:Local_timer_interrupts
436084 +39.8% 609534 ? 3% interrupts.CPU14.LOC:Local_timer_interrupts
436238 +38.7% 605188 ? 4% interrupts.CPU140.LOC:Local_timer_interrupts
436692 +38.6% 605188 ? 4% interrupts.CPU141.LOC:Local_timer_interrupts
436428 +38.7% 605187 ? 4% interrupts.CPU142.LOC:Local_timer_interrupts
436618 +38.6% 605181 ? 4% interrupts.CPU143.LOC:Local_timer_interrupts
2395060 ? 21% +210.7% 7440320 ? 40% interrupts.CPU144.CAL:Function_call_interrupts
437076 ? 2% +38.3% 604600 ? 4% interrupts.CPU144.LOC:Local_timer_interrupts
4849080 ? 21% +113.2% 10339751 ? 39% interrupts.CPU144.TLB:TLB_shootdowns
437303 ? 2% +38.3% 604631 ? 4% interrupts.CPU145.LOC:Local_timer_interrupts
437337 ? 2% +38.3% 604650 ? 4% interrupts.CPU146.LOC:Local_timer_interrupts
1628946 ? 35% +144.0% 3974501 ? 26% interrupts.CPU147.CAL:Function_call_interrupts
436987 ? 2% +38.4% 604641 ? 4% interrupts.CPU147.LOC:Local_timer_interrupts
3313139 ? 38% +62.6% 5387495 ? 26% interrupts.CPU147.TLB:TLB_shootdowns
437013 ? 2% +38.4% 604647 ? 4% interrupts.CPU148.LOC:Local_timer_interrupts
437047 ? 2% +38.3% 604641 ? 4% interrupts.CPU149.LOC:Local_timer_interrupts
435764 +39.9% 609486 ? 3% interrupts.CPU15.LOC:Local_timer_interrupts
437104 ? 2% +38.3% 604641 ? 4% interrupts.CPU150.LOC:Local_timer_interrupts
437148 ? 2% +38.3% 604674 ? 4% interrupts.CPU151.LOC:Local_timer_interrupts
437106 ? 2% +38.3% 604644 ? 4% interrupts.CPU152.LOC:Local_timer_interrupts
437065 ? 2% +38.4% 604688 ? 4% interrupts.CPU153.LOC:Local_timer_interrupts
437182 ? 2% +38.3% 604648 ? 4% interrupts.CPU154.LOC:Local_timer_interrupts
437068 ? 2% +38.3% 604649 ? 4% interrupts.CPU155.LOC:Local_timer_interrupts
437126 ? 2% +38.3% 604638 ? 4% interrupts.CPU156.LOC:Local_timer_interrupts
437340 ? 2% +38.3% 604639 ? 4% interrupts.CPU157.LOC:Local_timer_interrupts
437229 ? 2% +38.3% 604631 ? 4% interrupts.CPU158.LOC:Local_timer_interrupts
437031 ? 2% +38.4% 604646 ? 4% interrupts.CPU159.LOC:Local_timer_interrupts
436217 +39.7% 609502 ? 3% interrupts.CPU16.LOC:Local_timer_interrupts
437281 ? 2% +38.3% 604671 ? 4% interrupts.CPU160.LOC:Local_timer_interrupts
437174 ? 2% +38.3% 604664 ? 4% interrupts.CPU161.LOC:Local_timer_interrupts
437093 ? 2% +38.3% 604644 ? 4% interrupts.CPU162.LOC:Local_timer_interrupts
437080 ? 2% +38.3% 604617 ? 4% interrupts.CPU163.LOC:Local_timer_interrupts
437014 ? 2% +38.4% 604623 ? 4% interrupts.CPU164.LOC:Local_timer_interrupts
437064 ? 2% +38.3% 604664 ? 4% interrupts.CPU165.LOC:Local_timer_interrupts
437060 ? 2% +38.3% 604645 ? 4% interrupts.CPU166.LOC:Local_timer_interrupts
437133 ? 2% +38.3% 604647 ? 4% interrupts.CPU167.LOC:Local_timer_interrupts
437691 +38.8% 607710 ? 3% interrupts.CPU168.LOC:Local_timer_interrupts
437778 +38.8% 607719 ? 3% interrupts.CPU169.LOC:Local_timer_interrupts
333.83 ?130% +630.9% 2440 ?113% interrupts.CPU169.NMI:Non-maskable_interrupts
333.83 ?130% +630.9% 2440 ?113% interrupts.CPU169.PMI:Performance_monitoring_interrupts
435893 +39.8% 609457 ? 3% interrupts.CPU17.LOC:Local_timer_interrupts
437845 +38.8% 607745 ? 3% interrupts.CPU170.LOC:Local_timer_interrupts
972.00 ?199% +287.4% 3765 ? 75% interrupts.CPU170.NMI:Non-maskable_interrupts
972.00 ?199% +287.4% 3765 ? 75% interrupts.CPU170.PMI:Performance_monitoring_interrupts
437428 +38.9% 607740 ? 3% interrupts.CPU171.LOC:Local_timer_interrupts
437494 +38.9% 607739 ? 3% interrupts.CPU172.LOC:Local_timer_interrupts
1590512 ? 27% +139.8% 3814104 ? 44% interrupts.CPU173.CAL:Function_call_interrupts
437559 +38.9% 607732 ? 3% interrupts.CPU173.LOC:Local_timer_interrupts
437721 +38.8% 607747 ? 3% interrupts.CPU174.LOC:Local_timer_interrupts
437316 +39.0% 607744 ? 3% interrupts.CPU175.LOC:Local_timer_interrupts
437481 +38.9% 607717 ? 3% interrupts.CPU176.LOC:Local_timer_interrupts
437671 +38.9% 607816 ? 3% interrupts.CPU177.LOC:Local_timer_interrupts
437549 +38.9% 607776 ? 3% interrupts.CPU178.LOC:Local_timer_interrupts
437791 +38.8% 607753 ? 3% interrupts.CPU179.LOC:Local_timer_interrupts
435869 +39.8% 609482 ? 3% interrupts.CPU18.LOC:Local_timer_interrupts
437851 +38.8% 607737 ? 3% interrupts.CPU180.LOC:Local_timer_interrupts
437762 +38.8% 607765 ? 3% interrupts.CPU181.LOC:Local_timer_interrupts
437506 +38.9% 607755 ? 3% interrupts.CPU182.LOC:Local_timer_interrupts
437825 +38.8% 607732 ? 3% interrupts.CPU183.LOC:Local_timer_interrupts
437482 +38.9% 607739 ? 3% interrupts.CPU184.LOC:Local_timer_interrupts
437502 +38.9% 607760 ? 3% interrupts.CPU185.LOC:Local_timer_interrupts
437904 +38.8% 607748 ? 3% interrupts.CPU186.LOC:Local_timer_interrupts
437601 +38.9% 607731 ? 3% interrupts.CPU187.LOC:Local_timer_interrupts
437425 +38.9% 607746 ? 3% interrupts.CPU188.LOC:Local_timer_interrupts
437660 +38.9% 607742 ? 3% interrupts.CPU189.LOC:Local_timer_interrupts
435885 +39.8% 609492 ? 3% interrupts.CPU19.LOC:Local_timer_interrupts
658863 ? 99% +449.3% 3618986 ? 71% interrupts.CPU190.CAL:Function_call_interrupts
437525 +38.9% 607724 ? 3% interrupts.CPU190.LOC:Local_timer_interrupts
437714 +38.9% 607775 ? 3% interrupts.CPU191.LOC:Local_timer_interrupts
436340 +39.7% 609513 ? 3% interrupts.CPU2.LOC:Local_timer_interrupts
435793 +39.9% 609516 ? 3% interrupts.CPU20.LOC:Local_timer_interrupts
435929 +39.8% 609516 ? 3% interrupts.CPU21.LOC:Local_timer_interrupts
435897 +39.8% 609589 ? 3% interrupts.CPU22.LOC:Local_timer_interrupts
436105 +39.7% 609457 ? 3% interrupts.CPU23.LOC:Local_timer_interrupts
436542 +38.6% 605219 ? 4% interrupts.CPU24.LOC:Local_timer_interrupts
437153 +38.4% 605215 ? 4% interrupts.CPU25.LOC:Local_timer_interrupts
436932 +38.5% 605126 ? 4% interrupts.CPU26.LOC:Local_timer_interrupts
436176 +38.8% 605266 ? 4% interrupts.CPU27.LOC:Local_timer_interrupts
436415 +38.7% 605223 ? 4% interrupts.CPU28.LOC:Local_timer_interrupts
436413 +38.7% 605249 ? 4% interrupts.CPU29.LOC:Local_timer_interrupts
435892 +39.8% 609496 ? 3% interrupts.CPU3.LOC:Local_timer_interrupts
1884 ? 93% -90.6% 177.17 ? 79% interrupts.CPU3.NMI:Non-maskable_interrupts
1884 ? 93% -90.6% 177.17 ? 79% interrupts.CPU3.PMI:Performance_monitoring_interrupts
3717304 ? 27% -51.0% 1822336 ? 39% interrupts.CPU3.TLB:TLB_shootdowns
436566 +38.6% 605260 ? 4% interrupts.CPU30.LOC:Local_timer_interrupts
436820 +38.5% 605209 ? 4% interrupts.CPU31.LOC:Local_timer_interrupts
436693 +38.6% 605210 ? 4% interrupts.CPU32.LOC:Local_timer_interrupts
436239 +38.7% 605207 ? 4% interrupts.CPU33.LOC:Local_timer_interrupts
436425 +38.7% 605210 ? 4% interrupts.CPU34.LOC:Local_timer_interrupts
436281 +38.7% 605214 ? 4% interrupts.CPU35.LOC:Local_timer_interrupts
436621 +38.6% 605203 ? 4% interrupts.CPU36.LOC:Local_timer_interrupts
436949 +38.5% 605220 ? 4% interrupts.CPU37.LOC:Local_timer_interrupts
436637 +38.6% 605224 ? 4% interrupts.CPU38.LOC:Local_timer_interrupts
436124 +38.8% 605199 ? 4% interrupts.CPU39.LOC:Local_timer_interrupts
435874 +39.8% 609500 ? 3% interrupts.CPU4.LOC:Local_timer_interrupts
436826 +38.5% 605202 ? 4% interrupts.CPU40.LOC:Local_timer_interrupts
436386 +38.7% 605204 ? 4% interrupts.CPU41.LOC:Local_timer_interrupts
436202 +38.7% 605125 ? 4% interrupts.CPU42.LOC:Local_timer_interrupts
436444 +38.7% 605206 ? 4% interrupts.CPU43.LOC:Local_timer_interrupts
436315 +38.7% 605211 ? 4% interrupts.CPU44.LOC:Local_timer_interrupts
436710 +38.6% 605216 ? 4% interrupts.CPU45.LOC:Local_timer_interrupts
436448 +38.7% 605224 ? 4% interrupts.CPU46.LOC:Local_timer_interrupts
436683 +38.6% 605205 ? 4% interrupts.CPU47.LOC:Local_timer_interrupts
437175 ? 2% +38.3% 604605 ? 4% interrupts.CPU48.LOC:Local_timer_interrupts
428230 ? 65% +243.5% 1470847 ? 61% interrupts.CPU49.CAL:Function_call_interrupts
437324 ? 2% +38.3% 604629 ? 4% interrupts.CPU49.LOC:Local_timer_interrupts
435982 +39.8% 609551 ? 3% interrupts.CPU5.LOC:Local_timer_interrupts
437347 ? 2% +38.3% 604679 ? 4% interrupts.CPU50.LOC:Local_timer_interrupts
436995 ? 2% +38.4% 604678 ? 4% interrupts.CPU51.LOC:Local_timer_interrupts
437070 ? 2% +38.3% 604662 ? 4% interrupts.CPU52.LOC:Local_timer_interrupts
437067 ? 2% +38.3% 604680 ? 4% interrupts.CPU53.LOC:Local_timer_interrupts
437130 ? 2% +38.3% 604671 ? 4% interrupts.CPU54.LOC:Local_timer_interrupts
437155 ? 2% +38.3% 604700 ? 4% interrupts.CPU55.LOC:Local_timer_interrupts
437141 ? 2% +38.3% 604682 ? 4% interrupts.CPU56.LOC:Local_timer_interrupts
437048 ? 2% +38.4% 604671 ? 4% interrupts.CPU57.LOC:Local_timer_interrupts
437189 ? 2% +38.3% 604701 ? 4% interrupts.CPU58.LOC:Local_timer_interrupts
437069 ? 2% +38.4% 604688 ? 4% interrupts.CPU59.LOC:Local_timer_interrupts
436032 +39.8% 609555 ? 3% interrupts.CPU6.LOC:Local_timer_interrupts
437190 ? 2% +38.3% 604670 ? 4% interrupts.CPU60.LOC:Local_timer_interrupts
437388 ? 2% +38.2% 604673 ? 4% interrupts.CPU61.LOC:Local_timer_interrupts
437227 ? 2% +38.3% 604666 ? 4% interrupts.CPU62.LOC:Local_timer_interrupts
437036 ? 2% +38.4% 604662 ? 4% interrupts.CPU63.LOC:Local_timer_interrupts
437304 ? 2% +38.3% 604662 ? 4% interrupts.CPU64.LOC:Local_timer_interrupts
437177 ? 2% +38.3% 604654 ? 4% interrupts.CPU65.LOC:Local_timer_interrupts
437090 ? 2% +38.3% 604659 ? 4% interrupts.CPU66.LOC:Local_timer_interrupts
437100 ? 2% +38.3% 604602 ? 4% interrupts.CPU67.LOC:Local_timer_interrupts
437019 ? 2% +38.4% 604618 ? 4% interrupts.CPU68.LOC:Local_timer_interrupts
437048 ? 2% +38.4% 604675 ? 4% interrupts.CPU69.LOC:Local_timer_interrupts
436254 +39.7% 609555 ? 3% interrupts.CPU7.LOC:Local_timer_interrupts
140.83 ? 25% +500.0% 845.00 ? 93% interrupts.CPU7.NMI:Non-maskable_interrupts
140.83 ? 25% +500.0% 845.00 ? 93% interrupts.CPU7.PMI:Performance_monitoring_interrupts
437047 ? 2% +38.4% 604663 ? 4% interrupts.CPU70.LOC:Local_timer_interrupts
437164 ? 2% +38.3% 604658 ? 4% interrupts.CPU71.LOC:Local_timer_interrupts
437731 +38.8% 607745 ? 3% interrupts.CPU72.LOC:Local_timer_interrupts
437796 +38.8% 607772 ? 3% interrupts.CPU73.LOC:Local_timer_interrupts
437891 +38.8% 607769 ? 3% interrupts.CPU74.LOC:Local_timer_interrupts
488.67 ?180% +653.0% 3679 ? 75% interrupts.CPU74.NMI:Non-maskable_interrupts
488.67 ?180% +653.0% 3679 ? 75% interrupts.CPU74.PMI:Performance_monitoring_interrupts
456635 ? 60% +233.5% 1522817 ? 41% interrupts.CPU75.CAL:Function_call_interrupts
437451 +38.9% 607764 ? 3% interrupts.CPU75.LOC:Local_timer_interrupts
437531 +38.9% 607754 ? 3% interrupts.CPU76.LOC:Local_timer_interrupts
437603 +38.9% 607768 ? 3% interrupts.CPU77.LOC:Local_timer_interrupts
437752 +38.8% 607791 ? 3% interrupts.CPU78.LOC:Local_timer_interrupts
437335 +39.0% 607782 ? 3% interrupts.CPU79.LOC:Local_timer_interrupts
436062 +39.8% 609477 ? 3% interrupts.CPU8.LOC:Local_timer_interrupts
437520 +38.9% 607783 ? 3% interrupts.CPU80.LOC:Local_timer_interrupts
437687 +38.9% 607803 ? 3% interrupts.CPU81.LOC:Local_timer_interrupts
437567 +38.9% 607820 ? 3% interrupts.CPU82.LOC:Local_timer_interrupts
437815 +38.8% 607773 ? 3% interrupts.CPU83.LOC:Local_timer_interrupts
437892 +38.8% 607756 ? 3% interrupts.CPU84.LOC:Local_timer_interrupts
437781 +38.8% 607806 ? 3% interrupts.CPU85.LOC:Local_timer_interrupts
437537 +38.9% 607769 ? 3% interrupts.CPU86.LOC:Local_timer_interrupts
437848 +38.8% 607766 ? 3% interrupts.CPU87.LOC:Local_timer_interrupts
437543 +38.9% 607809 ? 3% interrupts.CPU88.LOC:Local_timer_interrupts
437531 +38.9% 607776 ? 3% interrupts.CPU89.LOC:Local_timer_interrupts
435850 +39.8% 609496 ? 3% interrupts.CPU9.LOC:Local_timer_interrupts
437953 +38.8% 607759 ? 3% interrupts.CPU90.LOC:Local_timer_interrupts
437616 +38.9% 607766 ? 3% interrupts.CPU91.LOC:Local_timer_interrupts
437458 +38.9% 607762 ? 3% interrupts.CPU92.LOC:Local_timer_interrupts
437697 +38.9% 607787 ? 3% interrupts.CPU93.LOC:Local_timer_interrupts
437556 +38.9% 607804 ? 3% interrupts.CPU94.LOC:Local_timer_interrupts
437772 +38.8% 607836 ? 3% interrupts.CPU95.LOC:Local_timer_interrupts
436537 +39.6% 609467 ? 3% interrupts.CPU96.LOC:Local_timer_interrupts
436496 +39.6% 609495 ? 3% interrupts.CPU97.LOC:Local_timer_interrupts
436305 +39.7% 609468 ? 3% interrupts.CPU98.LOC:Local_timer_interrupts
435736 +39.9% 609454 ? 3% interrupts.CPU99.LOC:Local_timer_interrupts
83872531 +38.9% 1.165e+08 ? 4% interrupts.LOC:Local_timer_interrupts
0.00 +1.9e+104% 192.00 interrupts.MCP:Machine_check_polls
122140 ? 6% +15.2% 140671 ? 5% interrupts.NMI:Non-maskable_interrupts
122140 ? 6% +15.2% 140671 ? 5% interrupts.PMI:Performance_monitoring_interrupts
5.166e+08 ? 2% +15.1% 5.948e+08 ? 2% interrupts.TLB:TLB_shootdowns
17.91 ? 6% -7.3 10.57 ? 2% perf-profile.calltrace.cycles-pp.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
17.90 ? 6% -7.3 10.56 ? 2% perf-profile.calltrace.cycles-pp.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_lruvec
17.22 ? 6% -7.0 10.22 ? 2% perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list
16.87 ? 6% -6.8 10.07 ? 2% perf-profile.calltrace.cycles-pp.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list
40.74 ? 7% -4.6 36.13 ? 4% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
40.30 ? 8% -4.5 35.84 ? 4% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
40.30 ? 8% -4.5 35.84 ? 4% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
40.30 ? 8% -4.5 35.83 ? 4% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
38.05 ? 9% -4.4 33.60 ? 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
37.65 ? 9% -4.4 33.26 ? 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
4.33 ? 9% -3.2 1.13 ? 22% perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
5.57 ? 6% -2.7 2.85 ? 4% perf-profile.calltrace.cycles-pp.pageout.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
3.96 ? 5% -2.2 1.76 ? 7% perf-profile.calltrace.cycles-pp.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
3.72 ? 6% -1.8 1.90 ? 8% perf-profile.calltrace.cycles-pp.__swap_writepage.pageout.shrink_page_list.shrink_inactive_list.shrink_lruvec
3.91 ? 6% -1.8 2.13 ? 3% perf-profile.calltrace.cycles-pp.llist_add_batch.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty
1.83 ? 6% -1.4 0.41 ? 71% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty
3.04 ? 11% -0.9 2.17 ? 11% perf-profile.calltrace.cycles-pp.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
5.24 ? 7% -0.8 4.43 ? 4% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
3.40 ? 6% -0.8 2.62 ? 10% perf-profile.calltrace.cycles-pp.do_rw_once
4.81 ? 7% -0.7 4.06 ? 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
1.94 ? 11% -0.7 1.19 ? 9% perf-profile.calltrace.cycles-pp.__mem_cgroup_charge.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault
3.31 ? 8% -0.5 2.78 ? 5% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
3.29 ? 8% -0.5 2.76 ? 5% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.61 ? 5% -0.3 0.28 ?100% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
1.15 ? 17% -0.3 0.88 ? 8% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_slowpath.__alloc_pages.alloc_pages_vma.do_anonymous_page
0.73 ? 6% -0.1 0.62 ? 6% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.46 ? 44% +0.4 0.83 ? 17% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.do_access
1.58 ? 8% +0.5 2.07 ? 6% perf-profile.calltrace.cycles-pp.__default_send_IPI_dest_field.default_send_IPI_mask_sequence_phys.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush
0.00 +2.2 2.25 ? 4% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one
0.00 +2.4 2.38 ? 8% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.page_referenced_one
0.00 +3.4 3.37 ? 9% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath
0.00 +3.4 3.41 ? 9% perf-profile.calltrace.cycles-pp.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock
0.00 +3.7 3.72 ? 6% perf-profile.calltrace.cycles-pp.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk
5.56 ? 8% +10.7 16.22 ? 3% perf-profile.calltrace.cycles-pp.rmap_walk_anon.try_to_unmap.shrink_page_list.shrink_inactive_list.shrink_lruvec
5.84 ? 8% +10.8 16.68 ? 3% perf-profile.calltrace.cycles-pp.try_to_unmap.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
4.96 ? 8% +10.9 15.85 ? 3% perf-profile.calltrace.cycles-pp.try_to_unmap_one.rmap_walk_anon.try_to_unmap.shrink_page_list.shrink_inactive_list
0.10 ?223% +12.3 12.40 ? 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon
0.10 ?223% +12.5 12.62 ? 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon
0.34 ?109% +13.2 13.56 ? 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon.try_to_unmap
2.00 ? 12% +13.4 15.36 ? 3% perf-profile.calltrace.cycles-pp.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec
2.13 ? 12% +13.4 15.48 ? 3% perf-profile.calltrace.cycles-pp.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
0.32 ?105% +13.4 13.74 ? 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced
0.50 ? 77% +13.5 14.04 ? 3% perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon.try_to_unmap.shrink_page_list
0.51 ? 76% +13.9 14.39 ? 3% perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list
0.87 ? 22% +14.0 14.83 ? 3% perf-profile.calltrace.cycles-pp.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list
18.00 ? 6% -7.4 10.64 ? 2% perf-profile.children.cycles-pp.try_to_unmap_flush_dirty
17.99 ? 6% -7.4 10.64 ? 2% perf-profile.children.cycles-pp.arch_tlbbatch_flush
17.31 ? 6% -7.0 10.29 ? 2% perf-profile.children.cycles-pp.on_each_cpu_cond_mask
17.30 ? 6% -7.0 10.29 ? 3% perf-profile.children.cycles-pp.smp_call_function_many_cond
40.74 ? 7% -4.6 36.13 ? 4% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
40.74 ? 7% -4.6 36.13 ? 4% perf-profile.children.cycles-pp.cpu_startup_entry
40.74 ? 7% -4.6 36.13 ? 4% perf-profile.children.cycles-pp.do_idle
38.48 ? 8% -4.6 33.90 ? 4% perf-profile.children.cycles-pp.cpuidle_enter
38.46 ? 8% -4.6 33.89 ? 4% perf-profile.children.cycles-pp.cpuidle_enter_state
40.30 ? 8% -4.5 35.84 ? 4% perf-profile.children.cycles-pp.start_secondary
4.34 ? 9% -3.0 1.33 ? 8% perf-profile.children.cycles-pp.__remove_mapping
5.60 ? 6% -2.7 2.87 ? 4% perf-profile.children.cycles-pp.pageout
3.99 ? 5% -2.1 1.84 ? 3% perf-profile.children.cycles-pp.add_to_swap
4.12 ? 5% -1.8 2.29 ? 6% perf-profile.children.cycles-pp.llist_add_batch
3.74 ? 6% -1.8 1.98 ? 4% perf-profile.children.cycles-pp.__swap_writepage
1.78 ? 12% -1.4 0.38 ? 13% perf-profile.children.cycles-pp.mem_cgroup_swapout
1.64 ? 6% -1.1 0.55 ? 4% perf-profile.children.cycles-pp.get_swap_page
3.25 ? 11% -1.0 2.28 ? 11% perf-profile.children.cycles-pp.mem_cgroup_charge
1.57 ? 6% -0.9 0.66 ? 6% perf-profile.children.cycles-pp.swap_writepage
3.90 ? 6% -0.9 3.01 ? 11% perf-profile.children.cycles-pp.do_rw_once
5.60 ? 7% -0.8 4.78 ? 4% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
1.09 ? 19% -0.8 0.29 ? 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.35 ? 6% -0.8 0.57 ? 5% perf-profile.children.cycles-pp._swap_info_get
1.16 ? 5% -0.8 0.39 ? 8% perf-profile.children.cycles-pp.page_swap_info
5.12 ? 7% -0.7 4.37 ? 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
1.14 ? 5% -0.7 0.39 ? 8% perf-profile.children.cycles-pp.swap_set_page_dirty
2.02 ? 11% -0.7 1.30 ? 9% perf-profile.children.cycles-pp.__mem_cgroup_charge
0.88 ? 13% -0.7 0.19 ? 18% perf-profile.children.cycles-pp.page_counter_uncharge
2.00 ? 10% -0.6 1.36 ? 11% perf-profile.children.cycles-pp.get_page_from_freelist
0.79 ? 13% -0.6 0.16 ? 18% perf-profile.children.cycles-pp.page_counter_cancel
1.26 ? 2% -0.6 0.64 ? 7% perf-profile.children.cycles-pp.handle_irq_event
1.27 ? 2% -0.6 0.65 ? 7% perf-profile.children.cycles-pp.__common_interrupt
1.25 ? 2% -0.6 0.64 ? 7% perf-profile.children.cycles-pp.handle_irq_event_percpu
1.26 ? 2% -0.6 0.65 ? 7% perf-profile.children.cycles-pp.handle_edge_irq
1.25 ? 2% -0.6 0.64 ? 7% perf-profile.children.cycles-pp.__handle_irq_event_percpu
1.24 ? 2% -0.6 0.63 ? 7% perf-profile.children.cycles-pp.nvme_irq
1.28 ? 2% -0.6 0.67 ? 6% perf-profile.children.cycles-pp.common_interrupt
1.28 ? 2% -0.6 0.67 ? 6% perf-profile.children.cycles-pp.asm_common_interrupt
0.93 ? 5% -0.6 0.34 ? 5% perf-profile.children.cycles-pp.swap_page_sector
1.05 ? 7% -0.6 0.48 ? 4% perf-profile.children.cycles-pp.try_to_free_swap
1.02 ? 7% -0.6 0.46 ? 5% perf-profile.children.cycles-pp.page_swapcount
3.59 ? 8% -0.5 3.05 ? 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
3.56 ? 8% -0.5 3.04 ? 5% perf-profile.children.cycles-pp.hrtimer_interrupt
1.10 ? 2% -0.5 0.58 ? 8% perf-profile.children.cycles-pp.blk_update_request
1.11 ? 2% -0.5 0.59 ? 7% perf-profile.children.cycles-pp.blk_mq_end_request
0.70 ? 9% -0.5 0.20 ? 10% perf-profile.children.cycles-pp.swap_duplicate
0.70 ? 10% -0.5 0.19 ? 11% perf-profile.children.cycles-pp.__swap_duplicate
1.32 ? 5% -0.5 0.83 ? 2% perf-profile.children.cycles-pp.add_to_swap_cache
0.69 ? 9% -0.5 0.23 ? 5% perf-profile.children.cycles-pp.get_swap_pages
1.21 ? 11% -0.4 0.77 ? 12% perf-profile.children.cycles-pp.prep_new_page
1.13 ? 11% -0.4 0.71 ? 12% perf-profile.children.cycles-pp.clear_page_erms
0.93 ? 12% -0.4 0.52 ? 11% perf-profile.children.cycles-pp.try_charge
0.84 ? 5% -0.4 0.44 ? 4% perf-profile.children.cycles-pp.bio_alloc_bioset
1.12 ? 8% -0.4 0.74 ? 8% perf-profile.children.cycles-pp.submit_bio
0.45 ? 22% -0.4 0.06 ? 11% perf-profile.children.cycles-pp.swap_cgroup_record
1.11 ? 7% -0.4 0.72 ? 8% perf-profile.children.cycles-pp.submit_bio_noacct
0.66 ? 9% -0.4 0.29 ? 8% perf-profile.children.cycles-pp.put_swap_page
0.67 ? 6% -0.4 0.30 ? 7% perf-profile.children.cycles-pp.__delete_from_swap_cache
0.84 ? 8% -0.4 0.47 ? 8% perf-profile.children.cycles-pp.page_lock_anon_vma_read
0.57 ? 8% -0.4 0.20 ? 5% perf-profile.children.cycles-pp.scan_swap_map_slots
0.51 ? 3% -0.3 0.18 ? 11% perf-profile.children.cycles-pp.__frontswap_store
0.65 ? 3% -0.3 0.32 ? 9% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.72 ? 15% -0.3 0.39 ? 11% perf-profile.children.cycles-pp._find_next_bit
0.59 ? 8% -0.3 0.28 ? 7% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.80 ? 4% -0.3 0.48 ? 8% perf-profile.children.cycles-pp.end_swap_bio_write
0.79 ? 4% -0.3 0.48 ? 8% perf-profile.children.cycles-pp.end_page_writeback
0.61 ? 5% -0.3 0.31 ? 7% perf-profile.children.cycles-pp.native_flush_tlb_multi
0.38 ? 8% -0.3 0.09 ? 12% perf-profile.children.cycles-pp.mem_cgroup_id_get_online
0.72 ? 7% -0.3 0.43 ? 5% perf-profile.children.cycles-pp.blk_mq_submit_bio
0.64 ? 11% -0.3 0.35 ? 7% perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
0.47 ? 8% -0.3 0.21 ? 7% perf-profile.children.cycles-pp.sysvec_call_function_single
0.44 ? 7% -0.3 0.19 ? 7% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.52 ? 9% -0.2 0.27 ? 4% perf-profile.children.cycles-pp.cgroup_rstat_updated
0.56 ? 4% -0.2 0.33 ? 6% perf-profile.children.cycles-pp.bio_init
0.28 ? 13% -0.2 0.06 ? 13% perf-profile.children.cycles-pp.get_swap_device
0.36 ? 26% -0.2 0.15 ? 26% perf-profile.children.cycles-pp.cpumask_any_but
0.54 ? 5% -0.2 0.36 ? 4% perf-profile.children.cycles-pp.__mod_lruvec_page_state
0.27 ? 11% -0.2 0.09 ? 12% perf-profile.children.cycles-pp.scan_swap_map_try_ssd_cluster
0.44 ? 8% -0.2 0.26 ? 4% perf-profile.children.cycles-pp.isolate_lru_pages
0.38 ? 8% -0.2 0.20 ? 8% perf-profile.children.cycles-pp.__set_page_dirty_no_writeback
0.42 ? 7% -0.2 0.25 ? 10% perf-profile.children.cycles-pp.__list_del_entry_valid
0.27 ? 8% -0.2 0.11 ? 10% perf-profile.children.cycles-pp.mempool_alloc
0.43 ? 6% -0.2 0.26 ? 10% perf-profile.children.cycles-pp.rotate_reclaimable_page
0.30 ? 8% -0.2 0.14 ? 8% perf-profile.children.cycles-pp.blk_mq_flush_plug_list
0.29 ? 8% -0.2 0.13 ? 5% perf-profile.children.cycles-pp.blk_mq_sched_insert_requests
0.39 ? 10% -0.2 0.23 ? 4% perf-profile.children.cycles-pp.__test_set_page_writeback
0.30 ? 8% -0.2 0.14 ? 6% perf-profile.children.cycles-pp.blk_flush_plug_list
0.29 ? 9% -0.2 0.13 ? 6% perf-profile.children.cycles-pp.nvme_queue_rq
0.28 ? 8% -0.2 0.13 ? 5% perf-profile.children.cycles-pp.blk_mq_try_issue_list_directly
0.28 ? 8% -0.2 0.13 ? 6% perf-profile.children.cycles-pp.__blk_mq_try_issue_directly
0.28 ? 9% -0.1 0.13 ? 5% perf-profile.children.cycles-pp.blk_mq_request_issue_directly
0.48 ? 7% -0.1 0.33 ? 4% perf-profile.children.cycles-pp.cpumask_next
0.25 ? 8% -0.1 0.10 ? 9% perf-profile.children.cycles-pp.nvme_map_data
0.38 ? 5% -0.1 0.24 ? 9% perf-profile.children.cycles-pp.pagevec_lru_move_fn
0.40 ? 7% -0.1 0.26 ? 5% perf-profile.children.cycles-pp.rmqueue
0.21 ? 8% -0.1 0.08 ? 20% perf-profile.children.cycles-pp.kmem_cache_alloc
0.34 ? 7% -0.1 0.23 ? 3% perf-profile.children.cycles-pp.free_unref_page_list
0.34 ? 5% -0.1 0.23 ? 7% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.46 ? 11% -0.1 0.35 ? 9% perf-profile.children.cycles-pp.lru_cache_add
0.30 ? 13% -0.1 0.20 ? 10% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.23 ? 6% -0.1 0.12 ? 9% perf-profile.children.cycles-pp.rmqueue_bulk
0.42 ? 10% -0.1 0.32 ? 10% perf-profile.children.cycles-pp.__pagevec_lru_add
0.62 ? 5% -0.1 0.53 ? 7% perf-profile.children.cycles-pp.__softirqentry_text_start
0.12 ? 9% -0.1 0.03 ?100% perf-profile.children.cycles-pp.kmem_cache_free
0.21 ? 5% -0.1 0.13 ? 8% perf-profile.children.cycles-pp.xas_store
0.23 ? 7% -0.1 0.14 ? 14% perf-profile.children.cycles-pp.pagevec_move_tail_fn
0.22 ? 5% -0.1 0.14 ? 3% perf-profile.children.cycles-pp.__mod_node_page_state
0.24 ? 3% -0.1 0.16 ? 7% perf-profile.children.cycles-pp.xas_create
0.23 ? 10% -0.1 0.15 ? 4% perf-profile.children.cycles-pp.free_pcppages_bulk
0.33 ? 8% -0.1 0.25 ? 13% perf-profile.children.cycles-pp.submit_bio_checks
0.24 ? 8% -0.1 0.17 ? 9% perf-profile.children.cycles-pp.blk_attempt_plug_merge
0.21 ? 5% -0.1 0.14 ? 9% perf-profile.children.cycles-pp.test_clear_page_writeback
0.20 ? 6% -0.1 0.14 ? 8% perf-profile.children.cycles-pp.__mod_lruvec_state
0.18 ? 2% -0.1 0.11 ? 4% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.17 ? 8% -0.1 0.11 ? 11% perf-profile.children.cycles-pp.page_add_new_anon_rmap
0.09 ? 14% -0.1 0.03 ?101% perf-profile.children.cycles-pp.__count_memcg_events
0.19 ? 11% -0.1 0.13 ? 6% perf-profile.children.cycles-pp.page_mapping
0.20 ? 6% -0.1 0.14 ? 11% perf-profile.children.cycles-pp.load_balance
0.15 ? 10% -0.1 0.10 ? 5% perf-profile.children.cycles-pp.page_remove_rmap
0.16 ? 2% -0.1 0.10 ? 16% perf-profile.children.cycles-pp.update_sd_lb_stats
0.18 ? 11% -0.1 0.12 ? 10% perf-profile.children.cycles-pp.bio_attempt_back_merge
0.16 ? 4% -0.0 0.11 ? 14% perf-profile.children.cycles-pp.find_busiest_group
0.13 ? 8% -0.0 0.08 ? 8% perf-profile.children.cycles-pp.mutex_unlock
0.07 ? 9% -0.0 0.02 ? 99% perf-profile.children.cycles-pp.__mod_memcg_state
0.07 ? 7% -0.0 0.02 ? 99% perf-profile.children.cycles-pp.release_pages
0.10 ? 17% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.lock_page_lruvec_irqsave
0.06 ? 11% -0.0 0.02 ? 99% perf-profile.children.cycles-pp.mutex_lock
0.11 ? 6% -0.0 0.07 ? 8% perf-profile.children.cycles-pp.memset_erms
0.14 ? 10% -0.0 0.10 ? 19% perf-profile.children.cycles-pp.page_counter_try_charge
0.10 ? 11% -0.0 0.07 ? 11% perf-profile.children.cycles-pp.___might_sleep
0.10 ? 9% -0.0 0.07 ? 5% perf-profile.children.cycles-pp.__hrtimer_next_event_base
0.10 ? 10% -0.0 0.07 ? 8% perf-profile.children.cycles-pp.blk_throtl_bio
0.08 ? 15% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.rcu_idle_exit
0.08 ? 6% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.__blk_queue_split
0.11 ? 9% -0.0 0.08 ? 15% perf-profile.children.cycles-pp.__perf_sw_event
0.11 ? 7% -0.0 0.08 ? 8% perf-profile.children.cycles-pp.ll_back_merge_fn
0.09 ? 7% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.__cond_resched
0.12 ? 7% -0.0 0.09 ? 12% perf-profile.children.cycles-pp.update_rq_clock
0.08 ? 16% -0.0 0.05 ? 47% perf-profile.children.cycles-pp.bio_associate_blkg_from_css
0.07 ? 10% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.___perf_sw_event
0.10 ? 10% -0.0 0.08 ? 4% perf-profile.children.cycles-pp.hrtimer_update_next_event
0.09 ? 7% -0.0 0.07 ? 16% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.07 ? 7% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.update_irq_load_avg
0.09 ? 7% -0.0 0.07 ? 5% perf-profile.children.cycles-pp.hrtimer_next_event_without
0.08 ? 10% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.update_blocked_averages
1.01 ? 7% +0.2 1.20 ? 6% perf-profile.children.cycles-pp.native_flush_tlb_local
2.16 ? 7% +0.4 2.54 ? 2% perf-profile.children.cycles-pp.default_send_IPI_mask_sequence_phys
1.59 ? 8% +0.6 2.15 ? 3% perf-profile.children.cycles-pp.__default_send_IPI_dest_field
42.06 ? 5% +8.5 50.53 ? 2% perf-profile.children.cycles-pp.shrink_node
42.00 ? 5% +8.5 50.49 ? 2% perf-profile.children.cycles-pp.shrink_lruvec
41.98 ? 5% +8.5 50.48 ? 2% perf-profile.children.cycles-pp.shrink_inactive_list
40.95 ? 5% +8.9 49.83 ? 2% perf-profile.children.cycles-pp.shrink_page_list
5.08 ? 8% +11.0 16.05 ? 3% perf-profile.children.cycles-pp.try_to_unmap_one
5.86 ? 8% +11.0 16.84 ? 3% perf-profile.children.cycles-pp.try_to_unmap
2.15 ? 12% +13.5 15.64 ? 3% perf-profile.children.cycles-pp.page_referenced
1.30 ? 17% +13.7 15.00 ? 3% perf-profile.children.cycles-pp.page_referenced_one
7.62 ? 9% +24.3 31.90 ? 3% perf-profile.children.cycles-pp.rmap_walk_anon
1.58 ? 18% +24.6 26.23 ? 3% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
2.42 ? 19% +25.7 28.14 ? 3% perf-profile.children.cycles-pp._raw_spin_lock
1.80 ? 27% +27.0 28.78 ? 3% perf-profile.children.cycles-pp.page_vma_mapped_walk
8.26 ? 8% -4.0 4.26 ? 5% perf-profile.self.cycles-pp.smp_call_function_many_cond
3.24 ? 6% -1.1 2.10 ? 9% perf-profile.self.cycles-pp.do_rw_once
2.97 ? 8% -1.0 1.96 ? 10% perf-profile.self.cycles-pp.do_access
2.96 ? 6% -1.0 1.95 ? 6% perf-profile.self.cycles-pp.llist_add_batch
1.73 ? 9% -0.7 0.98 ? 8% perf-profile.self.cycles-pp.try_to_unmap_one
0.79 ? 13% -0.6 0.16 ? 18% perf-profile.self.cycles-pp.page_counter_cancel
1.10 ? 6% -0.6 0.50 ? 5% perf-profile.self.cycles-pp._swap_info_get
0.89 ? 6% -0.6 0.33 ? 8% perf-profile.self.cycles-pp.page_swap_info
0.73 ? 6% -0.5 0.27 ? 5% perf-profile.self.cycles-pp.swap_page_sector
0.98 ? 12% -0.4 0.53 ? 11% perf-profile.self.cycles-pp.clear_page_erms
0.59 ? 6% -0.4 0.17 ? 6% perf-profile.self.cycles-pp.get_swap_page
0.69 ? 13% -0.4 0.31 ? 12% perf-profile.self.cycles-pp.try_charge
2.74 ? 7% -0.3 2.40 ? 7% perf-profile.self.cycles-pp.flush_tlb_func
0.38 ? 8% -0.3 0.09 ? 10% perf-profile.self.cycles-pp.mem_cgroup_id_get_online
0.74 ? 7% -0.3 0.46 ? 4% perf-profile.self.cycles-pp.add_to_swap_cache
0.45 ? 9% -0.3 0.18 ? 10% perf-profile.self.cycles-pp.__delete_from_swap_cache
0.52 ? 10% -0.3 0.26 ? 5% perf-profile.self.cycles-pp.cgroup_rstat_updated
0.42 ? 10% -0.2 0.19 ? 12% perf-profile.self.cycles-pp.mem_cgroup_charge
0.55 ? 16% -0.2 0.33 ? 12% perf-profile.self.cycles-pp._find_next_bit
0.32 ? 4% -0.2 0.11 ? 9% perf-profile.self.cycles-pp.__frontswap_store
0.46 ? 4% -0.2 0.27 ? 8% perf-profile.self.cycles-pp.bio_init
0.32 ? 5% -0.2 0.16 ? 9% perf-profile.self.cycles-pp.native_flush_tlb_multi
0.40 ? 7% -0.2 0.24 ? 10% perf-profile.self.cycles-pp.__list_del_entry_valid
0.31 ? 16% -0.1 0.17 ? 21% perf-profile.self.cycles-pp.page_lock_anon_vma_read
0.19 ? 13% -0.1 0.07 ? 5% perf-profile.self.cycles-pp.scan_swap_map_slots
0.28 ? 9% -0.1 0.17 ? 11% perf-profile.self.cycles-pp.__set_page_dirty_no_writeback
0.36 ? 5% -0.1 0.25 ? 3% perf-profile.self.cycles-pp.shrink_page_list
0.33 ? 8% -0.1 0.22 ? 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.26 ? 8% -0.1 0.16 ? 4% perf-profile.self.cycles-pp.isolate_lru_pages
0.56 ? 6% -0.1 0.47 ? 4% perf-profile.self.cycles-pp.cpuidle_enter_state
0.29 ? 12% -0.1 0.21 ? 17% perf-profile.self.cycles-pp.get_page_from_freelist
0.16 ? 6% -0.1 0.08 ? 9% perf-profile.self.cycles-pp.end_page_writeback
0.29 ? 11% -0.1 0.21 ? 12% perf-profile.self.cycles-pp.__mem_cgroup_charge
0.20 ? 5% -0.1 0.13 ? 6% perf-profile.self.cycles-pp.__mod_node_page_state
0.22 ? 3% -0.1 0.15 ? 6% perf-profile.self.cycles-pp.xas_create
0.23 ? 8% -0.1 0.16 ? 7% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.13 ? 9% -0.1 0.06 ? 7% perf-profile.self.cycles-pp.rmqueue_bulk
0.17 ? 9% -0.1 0.10 ? 9% perf-profile.self.cycles-pp.__mod_lruvec_page_state
0.14 ? 8% -0.1 0.08 ? 7% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.15 ? 8% -0.1 0.09 ? 7% perf-profile.self.cycles-pp.__test_set_page_writeback
0.08 ? 9% -0.1 0.02 ? 99% perf-profile.self.cycles-pp.__swap_writepage
0.13 ? 12% -0.1 0.08 ? 9% perf-profile.self.cycles-pp.free_pcppages_bulk
0.11 ? 12% -0.0 0.06 ? 47% perf-profile.self.cycles-pp.page_counter_try_charge
0.15 ? 11% -0.0 0.11 ? 7% perf-profile.self.cycles-pp.page_mapping
0.12 ? 12% -0.0 0.08 ? 19% perf-profile.self.cycles-pp.__pagevec_lru_add
0.08 ? 8% -0.0 0.04 ? 45% perf-profile.self.cycles-pp.pagevec_lru_move_fn
0.12 ? 6% -0.0 0.08 ? 8% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.10 ? 10% -0.0 0.07 ? 16% perf-profile.self.cycles-pp.pagevec_move_tail_fn
0.10 ? 10% -0.0 0.06 ? 14% perf-profile.self.cycles-pp.mutex_unlock
0.08 ? 19% -0.0 0.04 ? 45% perf-profile.self.cycles-pp.___might_sleep
0.11 ? 8% -0.0 0.08 ? 12% perf-profile.self.cycles-pp.update_sd_lb_stats
0.09 ? 13% -0.0 0.06 ? 7% perf-profile.self.cycles-pp.__hrtimer_next_event_base
0.08 ? 7% -0.0 0.06 ? 8% perf-profile.self.cycles-pp.memset_erms
0.07 ? 11% -0.0 0.04 ? 45% perf-profile.self.cycles-pp.rotate_reclaimable_page
0.06 ? 7% -0.0 0.04 ? 45% perf-profile.self.cycles-pp.update_irq_load_avg
0.08 ? 4% -0.0 0.06 ? 7% perf-profile.self.cycles-pp.ll_back_merge_fn
0.16 ? 5% +0.1 0.21 ? 5% perf-profile.self.cycles-pp.asm_sysvec_call_function
1.01 ? 7% +0.2 1.20 ? 6% perf-profile.self.cycles-pp.native_flush_tlb_local
2.22 ? 6% +0.4 2.59 ? 4% perf-profile.self.cycles-pp.flush_smp_call_function_queue
1.28 ? 5% +0.4 1.72 perf-profile.self.cycles-pp._raw_spin_lock
1.59 ? 8% +0.6 2.15 ? 3% perf-profile.self.cycles-pp.__default_send_IPI_dest_field
0.34 ? 35% +0.6 0.93 ? 8% perf-profile.self.cycles-pp.page_vma_mapped_walk
1.35 ? 14% +20.2 21.54 ? 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
11952 ? 12% -27.8% 8625 ? 13% softirqs.CPU0.RCU
32374 ? 3% +30.1% 42116 ? 2% softirqs.CPU0.SCHED
10010 ? 15% -33.0% 6705 ? 17% softirqs.CPU1.RCU
29521 ? 3% +25.7% 37120 ? 8% softirqs.CPU1.SCHED
9560 ? 18% -35.2% 6197 ? 12% softirqs.CPU10.RCU
9641 ? 15% -42.8% 5512 ? 13% softirqs.CPU100.RCU
27690 ? 2% +31.8% 36488 ? 3% softirqs.CPU100.SCHED
9404 ? 16% -37.4% 5884 ? 14% softirqs.CPU101.RCU
26702 ? 4% +39.4% 37210 ? 4% softirqs.CPU101.SCHED
9636 ? 16% -40.6% 5728 ? 15% softirqs.CPU102.RCU
26856 ? 6% +37.4% 36889 ? 4% softirqs.CPU102.SCHED
9761 ? 16% -42.7% 5591 ? 14% softirqs.CPU103.RCU
26740 ? 3% +32.4% 35414 ? 8% softirqs.CPU103.SCHED
9279 ? 16% -34.8% 6049 ? 14% softirqs.CPU104.RCU
26363 ? 5% +36.1% 35872 ? 9% softirqs.CPU104.SCHED
9410 ? 16% -38.5% 5787 ? 15% softirqs.CPU105.RCU
26533 ? 9% +41.5% 37539 ? 3% softirqs.CPU105.SCHED
9764 ? 26% -39.6% 5894 ? 9% softirqs.CPU106.RCU
26912 ? 4% +38.8% 37354 ? 5% softirqs.CPU106.SCHED
9551 ? 18% -39.0% 5824 ? 12% softirqs.CPU107.RCU
27262 ? 5% +28.8% 35113 ? 4% softirqs.CPU107.SCHED
9607 ? 17% -37.9% 5968 ? 16% softirqs.CPU108.RCU
26904 ? 3% +34.2% 36105 ? 3% softirqs.CPU108.SCHED
9485 ? 18% -43.7% 5341 ? 7% softirqs.CPU109.RCU
26209 ? 6% +38.5% 36295 ? 2% softirqs.CPU109.SCHED
28056 ? 6% +36.9% 38396 ? 5% softirqs.CPU11.SCHED
9807 ? 14% -39.4% 5942 ? 11% softirqs.CPU110.RCU
26349 ? 4% +36.5% 35954 ? 5% softirqs.CPU110.SCHED
9271 ? 15% -40.6% 5506 ? 11% softirqs.CPU111.RCU
26559 ? 8% +36.5% 36243 ? 4% softirqs.CPU111.SCHED
9830 ? 20% -41.0% 5804 ? 14% softirqs.CPU112.RCU
26422 ? 5% +36.5% 36058 ? 7% softirqs.CPU112.SCHED
9658 ? 19% -37.0% 6088 ? 20% softirqs.CPU113.RCU
26912 ? 2% +38.2% 37204 ? 5% softirqs.CPU113.SCHED
9835 ? 17% -41.0% 5798 ? 13% softirqs.CPU114.RCU
27485 ? 2% +32.6% 36438 ? 7% softirqs.CPU114.SCHED
9878 ? 16% -38.1% 6114 ? 16% softirqs.CPU115.RCU
27028 ? 3% +34.2% 36262 ? 4% softirqs.CPU115.SCHED
9542 ? 17% -37.5% 5967 ? 11% softirqs.CPU116.RCU
26216 ? 7% +43.6% 37650 ? 4% softirqs.CPU116.SCHED
9457 ? 18% -39.8% 5691 ? 12% softirqs.CPU117.RCU
27605 ? 5% +34.0% 36982 ? 3% softirqs.CPU117.SCHED
9567 ? 17% -41.5% 5594 ? 15% softirqs.CPU118.RCU
28212 ? 2% +31.3% 37047 ? 5% softirqs.CPU118.SCHED
9858 ? 16% -39.7% 5943 ? 13% softirqs.CPU119.RCU
28244 ? 4% +32.8% 37516 ? 4% softirqs.CPU119.SCHED
10429 ? 16% -31.2% 7175 ? 16% softirqs.CPU12.RCU
27864 ? 3% +33.1% 37090 ? 11% softirqs.CPU12.SCHED
9672 ? 15% -38.5% 5944 ? 13% softirqs.CPU120.RCU
27469 ? 3% +25.6% 34512 ? 9% softirqs.CPU120.SCHED
9419 ? 17% -37.5% 5884 ? 16% softirqs.CPU121.RCU
26369 ? 7% +24.5% 32820 ? 7% softirqs.CPU121.SCHED
9627 ? 17% -41.6% 5626 ? 14% softirqs.CPU122.RCU
27261 ? 4% +37.2% 37408 ? 5% softirqs.CPU122.SCHED
9753 ? 17% -40.5% 5799 ? 12% softirqs.CPU123.RCU
27689 ? 2% +32.5% 36691 ? 7% softirqs.CPU123.SCHED
10024 ? 18% -40.7% 5942 ? 11% softirqs.CPU124.RCU
27738 ? 2% +31.2% 36403 ? 6% softirqs.CPU124.SCHED
9669 ? 15% -38.7% 5930 ? 14% softirqs.CPU125.RCU
27931 ? 2% +29.1% 36046 ? 7% softirqs.CPU125.SCHED
9931 ? 20% -41.1% 5849 ? 13% softirqs.CPU126.RCU
27307 ? 3% +33.4% 36414 ? 7% softirqs.CPU126.SCHED
9518 ? 14% -39.6% 5745 ? 17% softirqs.CPU127.RCU
27125 ? 5% +32.5% 35941 ? 9% softirqs.CPU127.SCHED
9714 ? 16% -39.9% 5837 ? 10% softirqs.CPU128.RCU
27767 ? 3% +32.7% 36842 ? 4% softirqs.CPU128.SCHED
9659 ? 14% -40.8% 5718 ? 10% softirqs.CPU129.RCU
27759 ? 3% +30.9% 36346 ? 4% softirqs.CPU129.SCHED
10789 ? 26% -46.1% 5820 ? 9% softirqs.CPU13.RCU
26247 ? 15% +45.6% 38218 ? 4% softirqs.CPU13.SCHED
9600 ? 15% -38.2% 5930 ? 16% softirqs.CPU130.RCU
27262 ? 4% +34.1% 36562 ? 6% softirqs.CPU130.SCHED
10393 ? 20% -45.2% 5699 ? 14% softirqs.CPU131.RCU
27624 ? 4% +33.4% 36844 ? 3% softirqs.CPU131.SCHED
9593 ? 14% -40.2% 5737 ? 14% softirqs.CPU132.RCU
26882 ? 9% +34.5% 36151 ? 2% softirqs.CPU132.SCHED
9524 ? 17% -39.6% 5749 ? 12% softirqs.CPU133.RCU
27165 ? 5% +31.0% 35586 ? 7% softirqs.CPU133.SCHED
9714 ? 12% -41.1% 5726 ? 14% softirqs.CPU134.RCU
27285 ? 3% +29.3% 35284 ? 5% softirqs.CPU134.SCHED
9561 ? 13% -38.3% 5901 ? 19% softirqs.CPU135.RCU
27567 ? 3% +28.9% 35546 ? 7% softirqs.CPU135.SCHED
9577 ? 14% -40.4% 5711 ? 12% softirqs.CPU136.RCU
28030 ? 3% +31.5% 36846 ? 5% softirqs.CPU136.SCHED
9717 ? 16% -40.6% 5770 ? 15% softirqs.CPU137.RCU
28225 ? 3% +31.7% 37183 ? 6% softirqs.CPU137.SCHED
9676 ? 14% -41.9% 5624 ? 13% softirqs.CPU138.RCU
27095 ? 3% +38.9% 37642 ? 6% softirqs.CPU138.SCHED
9678 ? 17% -40.0% 5805 ? 16% softirqs.CPU139.RCU
27639 ? 2% +29.6% 35811 ? 10% softirqs.CPU139.SCHED
9842 ? 16% -40.1% 5894 ? 11% softirqs.CPU14.RCU
25932 ? 13% +47.9% 38346 ? 3% softirqs.CPU14.SCHED
9769 ? 14% -40.6% 5807 ? 14% softirqs.CPU140.RCU
28046 ? 3% +29.1% 36200 ? 8% softirqs.CPU140.SCHED
9640 ? 15% -39.3% 5847 ? 15% softirqs.CPU141.RCU
9629 ? 13% -38.5% 5924 ? 12% softirqs.CPU142.RCU
27220 ? 3% +34.6% 36638 ? 5% softirqs.CPU142.SCHED
9691 ? 16% -40.7% 5750 ? 13% softirqs.CPU143.RCU
27774 ? 3% +31.3% 36455 ? 5% softirqs.CPU143.SCHED
9922 ? 17% -38.1% 6143 ? 12% softirqs.CPU144.RCU
26996 ? 4% +27.2% 34330 ? 4% softirqs.CPU144.SCHED
9626 ? 17% -40.9% 5693 ? 13% softirqs.CPU145.RCU
9475 ? 15% -41.8% 5519 ? 11% softirqs.CPU146.RCU
26918 ? 9% +35.7% 36521 ? 6% softirqs.CPU146.SCHED
10026 ? 13% -43.5% 5660 ? 14% softirqs.CPU147.RCU
27369 ? 3% +33.8% 36627 ? 5% softirqs.CPU147.SCHED
10079 ? 12% -42.3% 5813 ? 11% softirqs.CPU148.RCU
26659 ? 2% +41.8% 37791 ? 2% softirqs.CPU148.SCHED
9605 ? 14% -39.7% 5796 ? 12% softirqs.CPU149.RCU
27454 ? 2% +36.1% 37356 ? 7% softirqs.CPU149.SCHED
9534 ? 17% -36.3% 6071 ? 10% softirqs.CPU15.RCU
28826 ? 2% +33.7% 38554 ? 4% softirqs.CPU15.SCHED
9821 ? 14% -35.8% 6309 ? 29% softirqs.CPU150.RCU
27391 ? 4% +34.6% 36880 ? 6% softirqs.CPU150.SCHED
9559 ? 15% -41.6% 5585 ? 11% softirqs.CPU151.RCU
26371 ? 2% +37.9% 36354 ? 6% softirqs.CPU151.SCHED
9662 ? 14% -38.1% 5984 ? 16% softirqs.CPU152.RCU
26885 ? 4% +30.6% 35100 ? 10% softirqs.CPU152.SCHED
9520 ? 14% -40.5% 5664 ? 13% softirqs.CPU153.RCU
27614 ? 4% +33.9% 36988 ? 5% softirqs.CPU153.SCHED
9494 ? 14% -40.8% 5622 ? 13% softirqs.CPU154.RCU
27383 +31.7% 36070 ? 7% softirqs.CPU154.SCHED
9623 ? 13% -40.8% 5699 ? 12% softirqs.CPU155.RCU
28108 ? 2% +28.0% 35984 ? 8% softirqs.CPU155.SCHED
9297 ? 15% -38.2% 5747 ? 14% softirqs.CPU156.RCU
28227 ? 4% +28.6% 36312 ? 6% softirqs.CPU156.SCHED
9421 ? 13% -41.2% 5535 ? 15% softirqs.CPU157.RCU
26196 ? 11% +38.9% 36389 ? 6% softirqs.CPU157.SCHED
9572 ? 15% -42.3% 5518 ? 14% softirqs.CPU158.RCU
27901 ? 2% +31.3% 36647 ? 7% softirqs.CPU158.SCHED
9195 ? 17% -38.7% 5634 ? 17% softirqs.CPU159.RCU
27962 ? 2% +30.4% 36469 ? 6% softirqs.CPU159.SCHED
9822 ? 15% -32.6% 6617 ? 18% softirqs.CPU16.RCU
26952 ? 5% +39.9% 37703 ? 8% softirqs.CPU16.SCHED
9209 ? 20% -41.9% 5355 ? 13% softirqs.CPU160.RCU
27793 ? 2% +29.8% 36073 ? 9% softirqs.CPU160.SCHED
9543 ? 16% -40.5% 5678 ? 13% softirqs.CPU161.RCU
28608 ? 2% +28.7% 36806 ? 3% softirqs.CPU161.SCHED
9399 ? 17% -39.8% 5662 ? 12% softirqs.CPU162.RCU
28119 ? 2% +31.9% 37092 ? 8% softirqs.CPU162.SCHED
9334 ? 17% -40.3% 5568 ? 13% softirqs.CPU163.RCU
28083 +32.2% 37136 ? 5% softirqs.CPU163.SCHED
9558 ? 14% -43.1% 5438 ? 11% softirqs.CPU164.RCU
28339 ? 2% +30.8% 37066 ? 7% softirqs.CPU164.SCHED
9522 ? 15% -42.4% 5485 ? 12% softirqs.CPU165.RCU
28064 ? 3% +32.6% 37204 ? 7% softirqs.CPU165.SCHED
9509 ? 17% -41.0% 5609 ? 15% softirqs.CPU166.RCU
27120 ? 8% +33.2% 36135 ? 7% softirqs.CPU166.SCHED
9335 ? 16% -41.2% 5485 ? 11% softirqs.CPU167.RCU
28405 ? 2% +27.8% 36306 ? 3% softirqs.CPU167.SCHED
25148 ? 4% +38.7% 34890 ? 5% softirqs.CPU168.SCHED
8770 ? 17% -40.3% 5237 ? 10% softirqs.CPU169.RCU
10241 ? 25% -36.9% 6461 ? 16% softirqs.CPU17.RCU
26802 ? 6% +34.2% 35979 ? 10% softirqs.CPU17.SCHED
9208 ? 15% -42.2% 5318 ? 11% softirqs.CPU170.RCU
27330 ? 4% +23.0% 33604 ? 10% softirqs.CPU170.SCHED
8924 ? 17% -39.5% 5400 ? 9% softirqs.CPU171.RCU
28100 ? 3% +31.5% 36959 ? 4% softirqs.CPU171.SCHED
8924 ? 17% -38.0% 5529 ? 10% softirqs.CPU172.RCU
28259 ? 3% +31.1% 37040 ? 4% softirqs.CPU172.SCHED
8875 ? 16% -41.0% 5232 ? 13% softirqs.CPU173.RCU
28254 ? 2% +32.8% 37523 ? 4% softirqs.CPU173.SCHED
9176 ? 15% -42.1% 5317 ? 13% softirqs.CPU174.RCU
27499 ? 3% +38.5% 38081 ? 3% softirqs.CPU174.SCHED
8654 ? 15% -38.4% 5333 ? 13% softirqs.CPU175.RCU
27692 ? 2% +35.4% 37507 ? 3% softirqs.CPU175.SCHED
9225 ? 14% -41.6% 5389 ? 15% softirqs.CPU176.RCU
27375 ? 4% +34.9% 36923 ? 8% softirqs.CPU176.SCHED
9289 ? 15% -43.5% 5249 ? 13% softirqs.CPU177.RCU
28263 ? 3% +32.0% 37296 ? 6% softirqs.CPU177.SCHED
9471 ? 14% -41.8% 5510 ? 15% softirqs.CPU178.RCU
28202 ? 3% +29.6% 36560 ? 5% softirqs.CPU178.SCHED
9263 ? 13% -40.8% 5486 ? 15% softirqs.CPU179.RCU
27974 ? 4% +32.7% 37116 ? 4% softirqs.CPU179.SCHED
9755 ? 20% -38.9% 5961 ? 14% softirqs.CPU18.RCU
28714 ? 2% +33.8% 38408 ? 5% softirqs.CPU18.SCHED
9532 ? 17% -42.7% 5461 ? 17% softirqs.CPU180.RCU
28337 ? 2% +29.8% 36778 ? 3% softirqs.CPU180.SCHED
8762 ? 15% -39.3% 5319 ? 14% softirqs.CPU181.RCU
27484 ? 3% +33.9% 36801 ? 2% softirqs.CPU181.SCHED
9105 ? 14% -39.8% 5483 ? 13% softirqs.CPU182.RCU
27679 ? 4% +33.4% 36912 ? 4% softirqs.CPU182.SCHED
9146 ? 16% -41.3% 5367 ? 15% softirqs.CPU183.RCU
9449 ? 14% -38.4% 5822 ? 25% softirqs.CPU184.RCU
28179 ? 2% +34.7% 37964 ? 3% softirqs.CPU184.SCHED
9703 ? 17% -42.4% 5590 ? 21% softirqs.CPU185.RCU
28451 +35.1% 38440 ? 3% softirqs.CPU185.SCHED
8740 ? 17% -40.4% 5210 ? 11% softirqs.CPU186.RCU
27970 ? 3% +33.9% 37441 ? 3% softirqs.CPU186.SCHED
9322 ? 15% -42.1% 5401 ? 14% softirqs.CPU187.RCU
9517 ? 11% -44.2% 5313 ? 14% softirqs.CPU188.RCU
27852 ? 3% +34.4% 37436 ? 5% softirqs.CPU188.SCHED
9375 ? 12% -42.7% 5375 ? 16% softirqs.CPU189.RCU
27896 ? 3% +34.7% 37573 ? 4% softirqs.CPU189.SCHED
9548 ? 17% -38.0% 5921 ? 12% softirqs.CPU19.RCU
28340 ? 2% +34.4% 38104 ? 5% softirqs.CPU19.SCHED
9329 ? 14% -41.7% 5437 ? 16% softirqs.CPU190.RCU
28512 ? 3% +30.6% 37248 ? 3% softirqs.CPU190.SCHED
9956 ? 12% -38.6% 6112 ? 20% softirqs.CPU191.RCU
9920 ? 17% -35.0% 6447 ? 12% softirqs.CPU2.RCU
9792 ? 16% -38.1% 6058 ? 24% softirqs.CPU20.RCU
9552 ? 17% -37.0% 6016 ? 18% softirqs.CPU21.RCU
28148 +36.7% 38489 ? 6% softirqs.CPU21.SCHED
10016 ? 14% -39.9% 6023 ? 17% softirqs.CPU22.RCU
10216 ? 25% -42.6% 5863 ? 11% softirqs.CPU23.RCU
28979 ? 3% +32.4% 38368 ? 8% softirqs.CPU23.SCHED
10358 ? 15% -38.4% 6381 ? 11% softirqs.CPU24.RCU
26220 ? 4% +37.7% 36110 ? 8% softirqs.CPU24.SCHED
9726 ? 15% -35.9% 6235 ? 14% softirqs.CPU25.RCU
10261 ? 14% -41.7% 5986 ? 14% softirqs.CPU26.RCU
27998 ? 6% +36.6% 38241 ? 6% softirqs.CPU26.SCHED
9785 ? 14% -35.1% 6354 ? 13% softirqs.CPU27.RCU
28204 ? 4% +31.8% 37162 ? 7% softirqs.CPU27.SCHED
10151 ? 16% -39.1% 6183 ? 17% softirqs.CPU28.RCU
27983 ? 4% +35.9% 38026 ? 6% softirqs.CPU28.SCHED
10092 ? 14% -40.2% 6036 ? 15% softirqs.CPU29.RCU
28593 ? 3% +34.5% 38465 ? 4% softirqs.CPU29.SCHED
10086 ? 18% -34.8% 6575 ? 26% softirqs.CPU3.RCU
27482 ? 7% +40.4% 38591 ? 3% softirqs.CPU3.SCHED
9883 ? 16% -37.0% 6229 ? 15% softirqs.CPU30.RCU
29738 ? 7% +28.3% 38155 ? 4% softirqs.CPU30.SCHED
9925 ? 17% -38.8% 6077 ? 16% softirqs.CPU31.RCU
10305 ? 14% -39.1% 6275 ? 12% softirqs.CPU32.RCU
28109 ? 3% +34.8% 37891 ? 4% softirqs.CPU32.SCHED
10590 ? 19% -40.9% 6259 ? 17% softirqs.CPU33.RCU
28261 ? 2% +26.6% 35778 ? 7% softirqs.CPU33.SCHED
10001 ? 16% -38.6% 6142 ? 12% softirqs.CPU34.RCU
28560 +35.0% 38550 ? 3% softirqs.CPU34.SCHED
10201 ? 15% -38.8% 6247 ? 12% softirqs.CPU35.RCU
28600 ? 3% +33.1% 38059 ? 3% softirqs.CPU35.SCHED
10116 ? 13% -38.1% 6260 ? 13% softirqs.CPU36.RCU
10122 ? 15% -40.3% 6039 ? 14% softirqs.CPU37.RCU
28116 ? 3% +37.2% 38569 ? 3% softirqs.CPU37.SCHED
10111 ? 13% -39.1% 6158 ? 14% softirqs.CPU38.RCU
28486 +33.3% 37962 ? 4% softirqs.CPU38.SCHED
10005 ? 14% -38.3% 6174 ? 15% softirqs.CPU39.RCU
9741 ? 16% -41.7% 5676 ? 7% softirqs.CPU4.RCU
28352 ? 3% +35.1% 38298 ? 4% softirqs.CPU4.SCHED
10184 ? 12% -40.0% 6115 ? 13% softirqs.CPU40.RCU
28728 ? 2% +33.1% 38233 ? 4% softirqs.CPU40.SCHED
10194 ? 13% -39.6% 6162 ? 13% softirqs.CPU41.RCU
28822 +31.2% 37804 ? 5% softirqs.CPU41.SCHED
10618 ? 18% -41.7% 6195 ? 13% softirqs.CPU42.RCU
28518 +36.7% 38997 ? 3% softirqs.CPU42.SCHED
10139 ? 15% -38.7% 6218 ? 14% softirqs.CPU43.RCU
28442 ? 2% +31.5% 37398 ? 8% softirqs.CPU43.SCHED
10142 ? 13% -39.8% 6108 ? 11% softirqs.CPU44.RCU
10157 ? 14% -35.7% 6529 ? 19% softirqs.CPU45.RCU
28301 ? 2% +31.6% 37252 ? 8% softirqs.CPU45.SCHED
10304 ? 16% -40.1% 6174 ? 18% softirqs.CPU46.RCU
28728 ? 2% +34.4% 38599 ? 4% softirqs.CPU46.SCHED
10163 ? 14% -39.6% 6136 ? 14% softirqs.CPU47.RCU
28510 ? 5% +33.3% 37991 ? 3% softirqs.CPU47.SCHED
10751 ? 15% -39.5% 6504 ? 18% softirqs.CPU48.RCU
24789 ? 10% +37.8% 34152 ? 5% softirqs.CPU48.SCHED
10001 ? 16% -39.8% 6017 ? 12% softirqs.CPU49.RCU
27400 ? 4% +28.0% 35059 ? 10% softirqs.CPU49.SCHED
9899 ? 16% -37.8% 6155 ? 11% softirqs.CPU5.RCU
25043 ? 26% +52.9% 38283 ? 4% softirqs.CPU5.SCHED
9941 ? 15% -40.9% 5880 ? 13% softirqs.CPU50.RCU
27850 ? 5% +33.9% 37300 ? 5% softirqs.CPU50.SCHED
10580 ? 13% -42.0% 6131 ? 13% softirqs.CPU51.RCU
28440 ? 2% +34.7% 38294 ? 4% softirqs.CPU51.SCHED
10532 ? 11% -41.7% 6144 ? 12% softirqs.CPU52.RCU
27522 ? 3% +40.1% 38549 ? 3% softirqs.CPU52.SCHED
10169 ? 14% -39.9% 6116 ? 11% softirqs.CPU53.RCU
28015 ? 2% +38.0% 38675 ? 6% softirqs.CPU53.SCHED
27943 ? 3% +36.6% 38162 ? 5% softirqs.CPU54.SCHED
9711 ? 15% -39.8% 5848 ? 12% softirqs.CPU55.RCU
28057 ? 2% +32.3% 37105 ? 6% softirqs.CPU55.SCHED
9893 ? 14% -39.3% 6004 ? 11% softirqs.CPU56.RCU
28221 ? 2% +34.3% 37914 ? 6% softirqs.CPU56.SCHED
10168 ? 16% -40.7% 6028 ? 12% softirqs.CPU57.RCU
28823 ? 2% +32.2% 38113 ? 6% softirqs.CPU57.SCHED
10014 ? 14% -39.8% 6033 ? 14% softirqs.CPU58.RCU
28179 ? 2% +33.0% 37485 ? 6% softirqs.CPU58.SCHED
10303 ? 13% -41.0% 6083 ? 15% softirqs.CPU59.RCU
29026 +32.1% 38345 ? 4% softirqs.CPU59.SCHED
10178 ? 14% -34.1% 6709 ? 13% softirqs.CPU6.RCU
29174 ? 10% +32.6% 38682 ? 3% softirqs.CPU6.SCHED
9753 ? 15% -37.9% 6057 ? 13% softirqs.CPU60.RCU
28862 ? 2% +32.8% 38320 ? 4% softirqs.CPU60.SCHED
9816 ? 13% -40.9% 5805 ? 14% softirqs.CPU61.RCU
28572 ? 2% +35.2% 38638 ? 3% softirqs.CPU61.SCHED
10000 ? 14% -41.2% 5876 ? 13% softirqs.CPU62.RCU
28515 ? 2% +34.6% 38377 ? 4% softirqs.CPU62.SCHED
9722 ? 16% -35.3% 6293 ? 18% softirqs.CPU63.RCU
28871 +32.9% 38371 ? 3% softirqs.CPU63.SCHED
9335 ? 17% -39.8% 5620 ? 13% softirqs.CPU64.RCU
9671 ? 16% -38.8% 5918 ? 11% softirqs.CPU65.RCU
9635 ? 16% -38.9% 5884 ? 10% softirqs.CPU66.RCU
9620 ? 17% -40.7% 5702 ? 11% softirqs.CPU67.RCU
23386 ? 35% +66.2% 38859 ? 3% softirqs.CPU67.SCHED
9840 ? 16% -42.5% 5661 ? 11% softirqs.CPU68.RCU
28877 ? 3% +31.9% 38083 ? 7% softirqs.CPU68.SCHED
9621 ? 16% -40.2% 5750 ? 10% softirqs.CPU69.RCU
28471 ? 4% +34.2% 38218 ? 4% softirqs.CPU69.SCHED
9500 ? 19% -37.7% 5920 ? 11% softirqs.CPU7.RCU
27971 ? 2% +27.6% 35679 ? 12% softirqs.CPU7.SCHED
9950 ? 19% -42.6% 5713 ? 11% softirqs.CPU70.RCU
28378 ? 4% +36.4% 38698 ? 4% softirqs.CPU70.SCHED
9769 ? 17% -41.5% 5712 ? 11% softirqs.CPU71.RCU
28400 ? 3% +35.0% 38334 ? 2% softirqs.CPU71.SCHED
10186 ? 23% -36.0% 6516 ? 11% softirqs.CPU72.RCU
29528 ? 5% +19.4% 35245 ? 3% softirqs.CPU72.SCHED
9117 ? 16% -37.7% 5682 ? 11% softirqs.CPU73.RCU
8864 ? 16% -37.6% 5527 ? 10% softirqs.CPU74.RCU
9191 ? 15% -40.4% 5474 ? 10% softirqs.CPU75.RCU
28423 ? 4% +28.2% 36424 ? 8% softirqs.CPU75.SCHED
9557 ? 20% -44.1% 5341 ? 10% softirqs.CPU76.RCU
28240 ? 3% +38.6% 39129 ? 3% softirqs.CPU76.SCHED
9108 ? 14% -40.6% 5408 ? 11% softirqs.CPU77.RCU
28923 ? 2% +35.5% 39203 ? 2% softirqs.CPU77.SCHED
9485 ? 16% -41.3% 5571 ? 13% softirqs.CPU78.RCU
28339 ? 2% +38.9% 39369 ? 3% softirqs.CPU78.SCHED
8889 ? 15% -39.2% 5401 ? 12% softirqs.CPU79.RCU
28802 ? 2% +35.8% 39116 ? 3% softirqs.CPU79.SCHED
28293 ? 2% +32.1% 37372 ? 7% softirqs.CPU8.SCHED
9386 ? 13% -40.2% 5612 ? 14% softirqs.CPU80.RCU
28492 ? 3% +36.6% 38911 ? 2% softirqs.CPU80.SCHED
9460 ? 14% -41.5% 5535 ? 13% softirqs.CPU81.RCU
28808 +33.5% 38470 ? 5% softirqs.CPU81.SCHED
9799 ? 14% -41.7% 5717 ? 15% softirqs.CPU82.RCU
28638 ? 3% +33.8% 38305 ? 3% softirqs.CPU82.SCHED
9574 ? 13% -40.1% 5731 ? 14% softirqs.CPU83.RCU
28754 +34.1% 38564 ? 3% softirqs.CPU83.SCHED
9488 ? 18% -41.6% 5541 ? 15% softirqs.CPU84.RCU
28843 +34.3% 38733 ? 4% softirqs.CPU84.SCHED
9072 ? 15% -39.8% 5465 ? 13% softirqs.CPU85.RCU
28412 ? 3% +37.4% 39033 ? 2% softirqs.CPU85.SCHED
9680 ? 17% -41.2% 5696 ? 15% softirqs.CPU86.RCU
28536 ? 2% +35.4% 38644 ? 2% softirqs.CPU86.SCHED
9251 ? 16% -37.9% 5749 ? 13% softirqs.CPU87.RCU
28704 +34.3% 38561 ? 3% softirqs.CPU87.SCHED
9761 ? 14% -40.2% 5842 ? 16% softirqs.CPU88.RCU
28799 ? 2% +36.4% 39277 ? 4% softirqs.CPU88.SCHED
9979 ? 14% -42.2% 5770 ? 14% softirqs.CPU89.RCU
28862 ? 3% +36.8% 39497 ? 3% softirqs.CPU89.SCHED
9789 ? 16% -40.8% 5798 ? 12% softirqs.CPU9.RCU
27851 ? 5% +39.6% 38881 ? 5% softirqs.CPU9.SCHED
8905 ? 14% -39.5% 5388 ? 11% softirqs.CPU90.RCU
28731 ? 2% +36.1% 39093 ? 4% softirqs.CPU90.SCHED
9570 ? 13% -38.1% 5924 ? 18% softirqs.CPU91.RCU
9375 ? 12% -41.8% 5455 ? 14% softirqs.CPU92.RCU
28905 +34.1% 38760 ? 5% softirqs.CPU92.SCHED
9198 ? 15% -35.9% 5893 ? 17% softirqs.CPU93.RCU
9560 ? 13% -37.5% 5976 ? 13% softirqs.CPU94.RCU
29218 ? 2% +28.9% 37660 ? 9% softirqs.CPU94.SCHED
9535 ? 13% -40.6% 5667 ? 14% softirqs.CPU95.RCU
9324 ? 15% -35.0% 6063 ? 21% softirqs.CPU96.RCU
25757 ? 5% +25.0% 32203 ? 9% softirqs.CPU96.SCHED
9674 ? 14% -40.8% 5729 ? 18% softirqs.CPU97.RCU
9371 ? 15% -37.7% 5840 ? 13% softirqs.CPU98.RCU
26068 ? 4% +38.4% 36065 ? 8% softirqs.CPU98.SCHED
9497 ? 15% -41.0% 5602 ? 13% softirqs.CPU99.RCU
26556 ? 6% +36.5% 36239 ? 3% softirqs.CPU99.SCHED
1862214 ? 15% -39.5% 1127214 ? 12% softirqs.RCU
5342380 +32.5% 7080895 ? 3% softirqs.SCHED
42480 ? 3% +28.2% 54463 softirqs.TIMER
vm-scalability.time.system_time
1800 +--------------------------------------------------------------------+
1700 |-+ O O O |
| O O O OO O O OO O OO O O O |
1600 |-OO O O O O O O OO O O O OO O |
1500 |-+ O O OO O O O O|
| O O O |
1400 |-+ |
1300 |-+ |
1200 |-+ |
| |
1100 |-+ |
1000 |-+ |
|.++. +.+ .++.++. .+ +. + .++.+ .+ |
900 |-+ + + ++ +.+ ++.+ + + + + |
800 +--------------------------------------------------------------------+
vm-scalability.time.percent_of_cpu_this_job_got
640 +---------------------------------------------------------------------+
| O |
620 |-+ O |
600 |-O O OO OO O O O O O OO O O OO O OO O O |
| O O O OO OO OO O OO O |
580 |-+ O O O O O O O |
| O |
560 |-+ |
| |
540 |-+ + .+. +. |
520 |.+ +.+ .++.+ + + + + ++.+ + |
| + : + + + +. :+ +.+ .+ |
500 |-+ +: + + + + |
| + |
480 +---------------------------------------------------------------------+
vm-scalability.time.elapsed_time
340 +---------------------------------------------------------------------+
| O O O |
320 |-+ O O O O OO OO OO OO O |
| OO O O O OO O O O O O O O O O |
300 |-+ O O O O O O O OO O O |
| O O |
280 |-+ |
| |
260 |-+ |
| |
240 |-+ |
| +.++. +. .+ .+ + +. |
220 |.+ ++.++.++.+ ++ + +.+ + +.+.+ ++.+ |
| + |
200 +---------------------------------------------------------------------+
vm-scalability.time.elapsed_time.max
340 +---------------------------------------------------------------------+
| O O O |
320 |-+ O O O O OO OO OO OO O |
| OO O O O OO O O O O O O O O O |
300 |-+ O O O O O O O OO O O |
| O O |
280 |-+ |
| |
260 |-+ |
| |
240 |-+ |
| +.++. +. .+ .+ + +. |
220 |.+ ++.++.++.+ ++ + +.+ + +.+.+ ++.+ |
| + |
200 +---------------------------------------------------------------------+
vm-scalability.throughput
3.4e+06 +-----------------------------------------------------------------+
| |
3.2e+06 |-+ + |
| .+ + :+ .+ |
3e+06 |+. + ++.++.+ .++. + .+ +: +++. + + |
2.8e+06 |-+++.+ + + + + + |
| |
2.6e+06 |-+ |
| |
2.4e+06 |-+ |
2.2e+06 |-+ |
| O O O O OO O O O O O O O|
2e+06 |O+O O OO O OOO O OO O O O O O OO OO |
| O O O OO O O O OO O |
1.8e+06 +-----------------------------------------------------------------+
vm-scalability.median
420000 +------------------------------------------------------------------+
| + |
400000 |-+ : |
380000 |-+ .+ +.+ : : .++.+ |
|+. .++ +.++.++ .++. +. : :: +.+++ |
360000 |-+++ + + + + |
340000 |-+ |
| |
320000 |-+ |
300000 |-+ |
| |
280000 |-+ O O O |
260000 |-+ O OO OO O O O O OO OO O O OO O|
|O O OO O O O O O O OO O O OOO O O |
240000 +------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation
Thanks,
Oliver Sang
On Mon, Jun 14, 2021 at 7:39 AM kernel test robot <[email protected]> wrote:
>
> FYI, we noticed a -29.4% regression of vm-scalability.throughput due to commit:
> ecf8443e51a8 ("[PATCH v7 02/11] mm/mremap: Fix race between MOVE_PUD mremap and pageout")
Ouch.
I guess it's not a huge surprise, but that's a fairly large regression.
Probably because the pud lock is just one single lock ("No scalability
reason to split PUD locks yet").
What happens if pud_lockptr() were to do the same thing that pmd_lockptr() does?
Linus
On 6/14/21 8:25 PM, kernel test robot wrote:
>
>
> Greeting,
>
> FYI, we noticed a -29.4% regression of vm-scalability.throughput due to commit:
>
>
> commit: ecf8443e51a862b261313c2319ab4e4aed9e6b7e ("[PATCH v7 02/11] mm/mremap: Fix race between MOVE_PUD mremap and pageout")
> url: https://github.com/0day-ci/linux/commits/Aneesh-Kumar-K-V/Speedup-mremap-on-ppc64/20210607-135424
> base: https://git.kernel.org/cgit/linux/kernel/git/powerpc/linux.git next
>
>
We dropped that approach and is now using
https://lore.kernel.org/linux-mm/[email protected]
Instead of pud lock we are now using rmap lock with mremap.
Can you check with that series?
-aneesh