LinuxLists.cc - [lkp-robot] [MD] 5a409b4f56: aim7.jobs-per-min -27.5% regression

2018-06-04 06:46:08

Subject: [lkp-robot] [MD] 5a409b4f56: aim7.jobs-per-min -27.5% regression

Greeting,

FYI, we noticed a

commit: 5a409b4f56d50b2123
Details are as below:
--------------------------
========================== compiler/cpufreq_governor/ gcc-7/performance/4BRD_1
commit:
448ec638c6 ("md/raid5: 5a409b4f56 ("MD:
448ec638c6bcf369 ---------------- %stddev %change \ | 1640 -27.5% 2194 +37.9% 2194 +37.9% 50990311 107965 ? 4% -26.4% 49.14 +82.5% 7.123e+08 672282 16631387 ? 2% 9708009 33436649 4.16 -2.1 0.24 ? 2% +27.7 95.51 -25.6 0.09 +0.0 6051756 ? 3% 6055311 ? 3% 6481209 ? 3% 6485866 ? 3% 61404 -27.7% 2.60 ? 18% +11519.2% 304.10 -84.9% 400477 166461 -49.9% 78397 +27.0% 14427 +18.4% 1963 ? 5% +5.4% 101143 53684 ? 4% -18.1% 175580 39406 +26.2% 1767204 ? 10% 51634 ? 18% +979.3% 92259 ? 13% +551.7% 38969 +28.0% 1895204 ? 10% 49512 ? 19% +1003.1% 83323 ? 14% +593.1% 2.524e+09 50620790 3.965e+08 5987788 2.506e+08 2923498 5.327e+08 779874 ? 2% 6191357 204095 +1982.1% 9850 +26.3% 12908 ? 18% +979.3% 8876 +29.6% 3486319 ? 4% 3482713 ? 4% 9743 +28.1% 12377 ? 19% +1003.1% 9287 +30.0% 3678995 ? 4% 3497785 ? 4% 252.70 +100.2% 282.70 2978 ? 17% +52.5% 2978 ? 17% +52.5% 2078 +147.9% 2078 +148.1% 5538 ? 2% +26.2% 5586 ? 3% +27.1% 6878 +47.6% 6889 +47.5% 9843 ? 5% +1.6e+05% 161.90 ? 4% +1.5e+05% 10386 ? 4% +1.5e+05% 161.90 ? 4% +1.5e+05% 432.80 ? 12% 432.80 ? 12% 3956 -23.1% 4098 -19.8% 360.50 ? 15% 360.50 ? 15% 35373 ? 2% 19595 +27.1% 3607 +18.4% 490.30 ? 5% 13421 ? 4% -18.1% 18608 25286 +991.0% 35405 ? 2% 3607 +18.4% 18161 +29.8% 76941 ? 5% -36.8% 33878 ? 7% -35.5% 12568956 12560739 17938 ? 3% -33.5% 78296 ? 5% -36.0% 8848 ? 6% -38.2% 8874568 ? 8% 5435965 12863707 17938 ? 3% -33.5% 1.379e+13 0.30 +0.1 4.2e+10 15.99 +3.8 3.779e+10 2.364e+11 8.795e+08 4.44 -7.2% 2.508e+14 36915392 0.29 ? 2% +0.0 4.14e+10 1.417e+13 0.20 ? 4% -0.0 3.072e+09 ? 4% 1.535e+12 90.73 -11.7 8.291e+09 8.473e+08 5.646e+13 6810 -35.9% 0.23 +7.8% 5326672 1.873e+10 2.093e+10 39.38 -0.7 1.087e+10 1.673e+10 5326695 1875095 ? 7% 32868920 ? 6% 7267340 ? 5% 4278 ? 7% -54.7% 245.48 ? 2% 2692 ? 6% +126.0% 33.09 -73.0% 507.40 ? 4% +128.0% 1875095 ? 7% 32868921 ? 6% 7267341 ? 5% 35887197 37385506 34416296 1228844 ? 8% 0.83 -28.1% 2.07 ? 3% -24.6% 20.52 ? 4% -48.8% 35.96 ? 5% -42.2% 8.97 ? 11% -44.5% 6.40 ? 12% -45.5% 21.78 ? 7% +143.3% 328.86 ? 18% +303.4% 55.97 ? 17% +286.0% 0.10 ? 29% -82.4% 3.43 ? 25% -79.9% 0.56 ? 26% -80.7% 1228822 ? 8% 992.30 1485 -18.1% 515.45 ? 2% 201.54 248.73 ? 6% 222.78 ? 3% 77097 ? 4% +278.4% 181319 ? 6% +298.7% 19338 +392.3% 34877 ? 6% +303.5% 1107408 1107427 1107385 13.10 ? 9% -48.1% 1107408 1107427 1107385 13.10 ? 9% -48.1% 30.36 ? 7% +107.7% 381.48 ? 18% +269.8% 63.92 ? 18% +262.2% 31.34 ? 5% +118.4% 323.62 ? 17% +349.5% 53.23 ? 16% +350.3% 32.15 ? 3% +129.4% 285.20 ? 14% +420.8% 46.66 ? 12% +430.0% 33.02 ? 2% +133.2% 252.16 ? 10% +481.2% 40.74 ? 8% +503.2% 33.86 +131.5% 219.81 ? 8% +522.6% 35.45 ? 7% +554.2% 2600 ? 4% -30.5% 25309 ? 4% -19.5% 4534 ? 7% -21.2% 0.00 ? 2% -27.6% 1083917 1088142 1082048 3.53 ? 6% -73.0% 11.54 ? 3% -62.1% 3.10 ? 3% -66.8% 10764176 10976436 10547712 148628 ? 3% 11.13 ? 2% +24.5% 6420 ? 8% -48.7% -5500 -37.2% 3784 ? 6% -47.2% 10812670 11020646 10601390 144529 ? 3% 706116 771307 644658 49847 ? 6% -67.9% 9618827 8990451 418563 ? 25% 640964 -93.7% 679527 -92.1% 601661 -94.9% 24242 ? 21% -77.7% 1107383 1107383 0.00 -49.4% 0.03 -49.4% 0.00 -49.4% 0.00 -49.4% 0.03 -49.4% 0.00 -49.4% 0.01 ? 8% +79.9% 1107805 87.59 -87.6 87.57 -87.6 87.59 -87.5 87.51 -87.5 87.51 -87.5 87.50 -87.5 87.50 -87.5 82.37 -82.4 82.23 -82.2 87.79 -25.0 92.78 -13.0 93.08 -12.6 93.08 -12.6 93.11 -12.6 93.11 -12.6 93.14 -12.5 93.15 -12.5 3.40 ? 2% -1.4 3.33 ? 2% -1.4 1.12 ? 2% -0.7 1.16 ? 2% -0.6 0.00 +0.6 0.00 +0.6 0.00 +0.7 0.00 +0.7 0.00 +0.7 0.00 +0.8 0.00 +0.8 0.62 ? 3% +0.8 0.00 +0.8 0.63 ? 2% +0.8 0.62 ? 2% +0.8 3.92 ? 2% +0.9 3.92 ? 2% +0.9 0.69 ? 2% +0.9 0.00 +1.2 0.00 +1.2 0.00 +1.3 0.00 +1.3 0.00 +1.3 0.00 +1.3 0.00 +1.6 0.00 +1.7 0.00 +1.7 0.00 +1.9 0.00 +2.1 0.00 +2.1 0.00 +2.1 0.00 +2.2 2.24 ? 4% +2.2 0.00 +2.3 0.00 +2.3 0.00 +2.4 0.37 ? 65% +2.4 0.26 ?100% +2.5 0.26 ?100% +2.5 0.26 ?100% +2.6 0.10 ?200% +2.7 0.00 +2.7 1.20 ? 3% +3.1 0.63 ? 6% +3.8 0.63 ? 5% +3.8 0.63 ? 5% +3.8 1.26 ? 5% +5.3 1.27 ? 5% +5.3 1.30 ? 4% +8.4 1.33 ? 4% +8.9 2.28 ? 2% +9.1 1.59 ? 4% +10.4 1.59 ? 4% +10.4 1.59 ? 4% +10.4 1.63 ? 4% +10.8 0.00 +57.7 0.00 +57.7 0.05 ?299% +57.8 0.19 ?154% +62.5 0.19 ?154% +62.6 0.19 ?154% +62.6 0.20 ?154% +62.6 0.20 ?154% +62.6 0.20 ?154% +62.6 0.29 ?125% +62.8 0.29 ?126% +62.8 0.29 ?125% +62.8 0.62 ? 41% +62.9 88.51 -88.2 87.57 -87.2 87.59 -87.2 83.26 -83.2 88.85 -25.7 88.90 -25.7 88.83 -24.5 88.84 -24.5 89.38 -23.5 89.90 -13.4 92.79 -13.0 93.08 -12.6 93.09 -12.6 93.13 -12.5 93.13 -12.5 93.22 -12.4 93.22 -12.4 3.40 ? 2% -1.4 3.33 ? 2% -1.4 1.03 ? 7% -1.0 1.69 ? 2% -0.7 1.66 ? 2% -0.7 1.13 ? 2% -0.7 0.54 ? 8% -0.4 0.49 ? 9% -0.4 0.10 ? 8% -0.1 0.20 ? 5% -0.0 0.06 ? 10% +0.0 0.18 ? 5% +0.0 0.02 ?153% +0.1 0.03 ?100% +0.1 0.08 ? 5% +0.1 0.29 ? 4% +0.1 0.40 ? 4% +0.1 0.11 ? 11% +0.1 0.12 ? 8% +0.1 0.04 ? 50% +0.1 0.00 +0.1 0.00 +0.1 0.01 ?200% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.18 ? 4% +0.1 0.20 ? 6% +0.1 0.16 ? 10% +0.1 0.03 ?100% +0.1 0.06 ? 9% +0.1 0.08 ? 12% +0.1 0.03 ? 81% +0.1 0.17 ? 7% +0.1 0.11 ? 9% +0.1 0.16 ? 6% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.08 ? 10% +0.1 0.25 ? 4% +0.1 0.00 +0.1 0.08 ? 38% +0.1 0.18 ? 5% +0.1 0.00 +0.1 0.06 ? 11% +0.1 0.32 ? 5% +0.1 0.15 ? 10% +0.1 0.15 ? 11% +0.1 0.08 ? 17% +0.2 0.00 +0.2 0.12 ? 8% +0.2 0.10 ? 16% +0.2 0.10 ? 19% +0.2 0.10 ? 17% +0.2 0.10 ? 16% +0.2 0.11 ? 15% +0.2 0.09 ? 9% +0.2 0.10 ? 18% +0.2 0.10 ? 15% +0.2 0.09 ? 12% +0.2 0.09 ? 12% +0.2 0.09 ? 12% +0.2 0.09 ? 12% +0.2 0.09 ? 12% +0.2 0.00 +0.2 0.30 ? 4% +0.2 1.39 ? 2% +0.2 1.43 +0.2 0.14 ? 13% +0.2 0.00 +0.2 0.22 ? 6% +0.2 0.14 ? 13% +0.2 0.10 ? 8% +0.2 0.07 ? 10% +0.3 0.03 ? 83% +0.3 0.11 ? 5% +0.3 0.06 ? 36% +0.3 0.08 ? 9% +0.3 0.15 ? 12% +0.3 0.10 ? 11% +0.3 0.06 ? 12% +0.3 0.02 ?153% +0.3 0.17 ? 5% +0.3 0.19 ? 12% +0.3 0.18 ? 5% +0.3 0.18 ? 5% +0.3 0.21 ? 11% +0.4 0.26 ? 6% +0.4 1.20 ? 2% +0.4 0.28 ? 5% +0.4 0.00 +0.4 0.25 ? 8% +0.5 1.30 ? 2% +0.7 0.31 ? 8% +0.8 0.31 ? 8% +0.8 3.92 ? 2% +0.9 3.92 ? 2% +0.9 0.69 ? 2% +0.9 0.08 ? 13% +1.2 0.16 ? 9% +1.6 0.17 ? 9% +2.0 0.21 ? 6% +2.0 2.24 ? 4% +2.2 0.46 ? 6% +2.3 0.51 ? 6% +2.3 0.49 ? 6% +2.3 0.49 ? 6% +2.3 0.49 ? 6% +2.3 1.80 ? 3% +5.6 2.12 ? 4% +5.8 1.33 ? 4% +8.8 1.37 ? 4% +9.3 1.59 ? 4% +10.4 1.63 ? 4% +10.8 1.63 ? 4% +10.8 1.63 ? 4% +10.9 3.48 +12.2 1.36 ? 12% +57.8 0.43 ? 38% +62.4 0.55 ? 29% +62.5 0.55 ? 30% +62.5 0.55 ? 29% +62.6 0.66 ? 25% +62.9 0.39 ? 43% +63.6 5.43 ? 3% +64.2 89.86 -13.5 0.14 ? 8% -0.0 0.10 ? 12% -0.0 0.06 ? 7% +0.0 0.05 ? 12% +0.0 0.15 ? 6% +0.0 0.17 ? 4% +0.0 0.10 ? 11% +0.1 0.02 ?153% +0.1 0.02 ?152% +0.1 0.03 ?100% +0.1 0.08 ? 5% +0.1 0.01 ?200% +0.1 0.29 ? 4% +0.1 0.14 ? 7% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.34 ? 6% +0.1 0.18 ? 4% +0.1 0.16 ? 10% +0.1 0.06 ? 11% +0.1 0.08 ? 38% +0.1 0.18 ? 5% +0.1 0.00 +0.1 0.00 +0.2 0.00 +0.2 0.02 ?152% +0.3 1.33 ? 4% +8.8

1700 +-+---------------- |+ ++++++ :+ ++++ ++++ 1600 +-+ + +++ | | 1500 +-+ | 1400 +-+ | 1300 +-+ | O OO OO O 1200 +OO OOOOOOOOO | 1100 +-+---------------- [*] bisect-good sample
[O] bisect-bad sample

Disclaimer:
Results have been for informational design or configuration

Thanks,
Xiaolong
-27.5% regression of aim7.jobs-per-min due to commit:
34f338cb8465d65550cd85 ("MD: fix lock contention for flush bios")
el.org/cgit/linux/kernel/git/next/linux-next.git">https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
//sourceforge.net/projects/aimbench/files/aim-suite7/">https://sourceforge.net/projects/aimbench/files/aim-suite7/
------------------------------------------------------------------------>
===============================================================
disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
2G/xfs/x86_64-rhel-7.2/600/RAID1/debian-x86_64-2016-08-31.cgz/lkp-ivb-ep01/sync_disk_rw/aim7
Assigning NULL to sh->batch_head before testing bit R5_Overlap of a stripe")
fix lock contention for flush bios")
5a409b4f56d50b212334f338cb
--------------------------
%stddev
\
1189 aim7.jobs-per-min
3026 aim7.time.elapsed_time
3026 aim7.time.elapsed_time.max
-95.8% 2148266 aim7.time.involuntary_context_switches
79516 ? 2% aim7.time.minor_page_faults
89.66 ? 2% aim7.time.user_time
-35.7% 4.582e+08 aim7.time.voluntary_context_switches
+36.8% 919615 interrupts.CAL:Function_call_interrupts
-39.9% 9993075 ? 7% softirqs.RCU
+186.1% 27778773 softirqs.SCHED
+45.5% 48644912 softirqs.TIMER
2.01 mpstat.cpu.idle%
27.91 mpstat.cpu.iowait%
69.94 mpstat.cpu.sys%
0.13 mpstat.cpu.usr%
+59.0% 9623085 numa-numastat.node0.local_node
+59.0% 9626996 numa-numastat.node0.numa_hit
+48.4% 9616310 numa-numastat.node1.local_node
+48.3% 9620756 numa-numastat.node1.numa_hit
44424 vmstat.io.bo
302.10 vmstat.procs.b
45.80 ? 2% vmstat.procs.r
-43.5% 226094 vmstat.system.cs
83332 vmstat.system.in
99567 meminfo.Dirty
17082 meminfo.Inactive(anon)
2068 ? 4% meminfo.Mlocked
+991.0% 1103488 meminfo.SUnreclaim
43946 ? 3% meminfo.Shmem
+571.4% 1178829 meminfo.Slab
49717 numa-meminfo.node0.Dirty
+37.2% 2425487 ? 2% numa-meminfo.node0.MemUsed
557316 numa-meminfo.node0.SUnreclaim
601288 numa-meminfo.node0.Slab
49863 numa-meminfo.node1.Dirty
+24.7% 2363037 ? 3% numa-meminfo.node1.MemUsed
546165 numa-meminfo.node1.SUnreclaim
577534 numa-meminfo.node1.Slab
+894.5% 2.51e+10 cpuidle.C1.time
+316.5% 2.109e+08 cpuidle.C1.usage
+1871.1% 7.815e+09 cpuidle.C1E.time
+186.1% 17129412 cpuidle.C1E.usage
+97.5% 4.948e+08 ? 2% cpuidle.C3.time
-55.7% 1295033 cpuidle.C3.usage
+179.9% 1.491e+09 cpuidle.C6.time
+229.3% 2567769 cpuidle.C6.usage
+3333.6% 2.126e+08 cpuidle.POLL.time
4249504 cpuidle.POLL.usage
12444 numa-vmstat.node0.nr_dirty
139321 numa-vmstat.node0.nr_slab_unreclaimable
11505 numa-vmstat.node0.nr_zone_write_pending
+55.1% 5407021 numa-vmstat.node0.numa_hit
+55.1% 5403066 numa-vmstat.node0.numa_local
12479 numa-vmstat.node1.nr_dirty
136532 numa-vmstat.node1.nr_slab_unreclaimable
12074 numa-vmstat.node1.nr_zone_write_pending
+44.8% 5326772 numa-vmstat.node1.numa_hit
+47.1% 5145705 numa-vmstat.node1.numa_local
505.90 slabinfo.biovec-max.active_objs
+99.1% 562.90 slabinfo.biovec-max.num_objs
4543 ? 14% slabinfo.dmaengine-unmap-16.active_objs
4543 ? 14% slabinfo.dmaengine-unmap-16.num_objs
5153 ? 11% slabinfo.ip6_dst_cache.active_objs
5157 ? 11% slabinfo.ip6_dst_cache.num_objs
6990 ? 3% slabinfo.kmalloc-1024.active_objs
7097 ? 3% slabinfo.kmalloc-1024.num_objs
10151 ? 5% slabinfo.kmalloc-192.active_objs
10160 ? 5% slabinfo.kmalloc-192.num_objs
16002876 slabinfo.kmalloc-64.active_objs
250044 slabinfo.kmalloc-64.active_slabs
16002877 slabinfo.kmalloc-64.num_objs
250044 slabinfo.kmalloc-64.num_slabs
+45.2% 628.50 ? 6% slabinfo.nfs_read_data.active_objs
+45.2% 628.50 ? 6% slabinfo.nfs_read_data.num_objs
3041 slabinfo.pool_workqueue.active_objs
3286 slabinfo.pool_workqueue.num_objs
+56.6% 564.70 ? 11% slabinfo.secpath_cache.active_objs
+56.6% 564.70 ? 11% slabinfo.secpath_cache.num_objs
-8.3% 32432 proc-vmstat.nr_active_anon
24914 proc-vmstat.nr_dirty
4270 proc-vmstat.nr_inactive_anon
+5.4% 516.90 ? 4% proc-vmstat.nr_mlock
10986 ? 3% proc-vmstat.nr_shmem
+1.2% 18834 proc-vmstat.nr_slab_reclaimable
275882 proc-vmstat.nr_slab_unreclaimable
-8.3% 32465 proc-vmstat.nr_zone_active_anon
4270 proc-vmstat.nr_zone_inactive_anon
23572 proc-vmstat.nr_zone_write_pending
48622 ? 4% proc-vmstat.numa_hint_faults
21836 ? 5% proc-vmstat.numa_hint_faults_local
+53.3% 19272377 proc-vmstat.numa_hit
+53.4% 19264015 proc-vmstat.numa_local
11935 ? 2% proc-vmstat.numa_pages_migrated
50085 ? 4% proc-vmstat.numa_pte_updates
5466 ? 6% proc-vmstat.pgactivate
+368.7% 41590920 proc-vmstat.pgalloc_normal
+39.2% 7564148 proc-vmstat.pgfault
+255.1% 45683570 proc-vmstat.pgfree
11935 ? 2% proc-vmstat.pgmigrate_success
-40.8% 8.17e+12 perf-stat.branch-instructions
0.42 perf-stat.branch-miss-rate%
-17.6% 3.462e+10 perf-stat.branch-misses
19.74 perf-stat.cache-miss-rate%
-21.6% 2.963e+10 perf-stat.cache-misses
-36.5% 1.501e+11 perf-stat.cache-references
-22.2% 6.84e+08 perf-stat.context-switches
4.12 perf-stat.cpi
-44.5% 1.393e+14 perf-stat.cpu-cycles
+60.4% 59211221 perf-stat.cpu-migrations
0.34 ? 4% perf-stat.dTLB-load-miss-rate%
-30.2% 2.89e+10 ? 4% perf-stat.dTLB-load-misses
-40.1% 8.491e+12 perf-stat.dTLB-loads
0.18 ? 5% perf-stat.dTLB-store-miss-rate%
-28.0% 2.21e+09 ? 4% perf-stat.dTLB-store-misses
-20.2% 1.225e+12 perf-stat.dTLB-stores
79.07 perf-stat.iTLB-load-miss-rate%
-6.6% 7.743e+09 perf-stat.iTLB-load-misses
+141.8% 2.049e+09 ? 3% perf-stat.iTLB-loads
-40.2% 3.378e+13 perf-stat.instructions
4362 perf-stat.instructions-per-iTLB-miss
0.24 perf-stat.ipc
+39.2% 7413706 perf-stat.minor-faults
-29.9% 1.312e+10 perf-stat.node-load-misses
-29.2% 1.481e+10 perf-stat.node-loads
38.72 perf-stat.node-store-miss-rate%
-16.6% 9.069e+09 perf-stat.node-store-misses
-14.2% 1.435e+10 perf-stat.node-stores
+39.2% 7413708 perf-stat.page-faults
-54.8% 846645 ? 16% sched_debug.cfs_rq:/.MIN_vruntime.avg
-35.7% 21150379 ? 14% sched_debug.cfs_rq:/.MIN_vruntime.max
-44.7% 4015798 ? 14% sched_debug.cfs_rq:/.MIN_vruntime.stddev
1939 ? 11% sched_debug.cfs_rq:/.exec_clock.stddev
+65.3% 405.75 ? 7% sched_debug.cfs_rq:/.load_avg.avg
6087 ? 7% sched_debug.cfs_rq:/.load_avg.max
8.94 ? 7% sched_debug.cfs_rq:/.load_avg.min
1156 ? 7% sched_debug.cfs_rq:/.load_avg.stddev
-54.8% 846645 ? 16% sched_debug.cfs_rq:/.max_vruntime.avg
-35.7% 21150379 ? 14% sched_debug.cfs_rq:/.max_vruntime.max
-44.7% 4015798 ? 14% sched_debug.cfs_rq:/.max_vruntime.stddev
-13.2% 31149130 sched_debug.cfs_rq:/.min_vruntime.avg
-14.3% 32043914 sched_debug.cfs_rq:/.min_vruntime.max
-12.3% 30183927 sched_debug.cfs_rq:/.min_vruntime.min
-52.6% 582759 ? 4% sched_debug.cfs_rq:/.min_vruntime.stddev
0.60 ? 6% sched_debug.cfs_rq:/.nr_running.avg
1.56 ? 8% sched_debug.cfs_rq:/.nr_running.max
10.52 ? 3% sched_debug.cfs_rq:/.nr_spread_over.avg
20.77 ? 9% sched_debug.cfs_rq:/.nr_spread_over.max
4.98 ? 8% sched_debug.cfs_rq:/.nr_spread_over.min
3.49 ? 7% sched_debug.cfs_rq:/.nr_spread_over.stddev
53.00 ? 9% sched_debug.cfs_rq:/.runnable_load_avg.avg
1326 ? 14% sched_debug.cfs_rq:/.runnable_load_avg.max
216.07 ? 13% sched_debug.cfs_rq:/.runnable_load_avg.stddev
0.02 ? 50% sched_debug.cfs_rq:/.spread.avg
0.69 ? 50% sched_debug.cfs_rq:/.spread.max
0.11 ? 50% sched_debug.cfs_rq:/.spread.stddev
-52.6% 582732 ? 4% sched_debug.cfs_rq:/.spread0.stddev
-24.9% 745.56 ? 2% sched_debug.cfs_rq:/.util_avg.avg
1217 ? 2% sched_debug.cfs_rq:/.util_avg.max
-25.2% 385.73 ? 6% sched_debug.cfs_rq:/.util_avg.min
-14.9% 171.52 ? 3% sched_debug.cfs_rq:/.util_avg.stddev
-38.1% 154.02 ? 8% sched_debug.cfs_rq:/.util_est_enqueued.avg
-15.8% 187.58 ? 2% sched_debug.cfs_rq:/.util_est_enqueued.stddev
291767 ? 11% sched_debug.cpu.avg_idle.avg
722862 ? 3% sched_debug.cpu.avg_idle.max
95203 ? 17% sched_debug.cpu.avg_idle.min
140732 ? 6% sched_debug.cpu.avg_idle.stddev
+37.6% 1523823 sched_debug.cpu.clock.avg
+37.6% 1523834 sched_debug.cpu.clock.max
+37.6% 1523811 sched_debug.cpu.clock.min
6.80 ? 8% sched_debug.cpu.clock.stddev
+37.6% 1523823 sched_debug.cpu.clock_task.avg
+37.6% 1523834 sched_debug.cpu.clock_task.max
+37.6% 1523811 sched_debug.cpu.clock_task.min
6.80 ? 8% sched_debug.cpu.clock_task.stddev
63.06 ? 12% sched_debug.cpu.cpu_load[0].avg
1410 ? 18% sched_debug.cpu.cpu_load[0].max
231.50 ? 17% sched_debug.cpu.cpu_load[0].stddev
68.44 ? 9% sched_debug.cpu.cpu_load[1].avg
1454 ? 14% sched_debug.cpu.cpu_load[1].max
239.71 ? 13% sched_debug.cpu.cpu_load[1].stddev
73.74 ? 6% sched_debug.cpu.cpu_load[2].avg
1485 ? 9% sched_debug.cpu.cpu_load[2].max
247.32 ? 8% sched_debug.cpu.cpu_load[2].stddev
77.00 ? 3% sched_debug.cpu.cpu_load[3].avg
1465 ? 7% sched_debug.cpu.cpu_load[3].max
245.72 ? 6% sched_debug.cpu.cpu_load[3].stddev
78.38 ? 2% sched_debug.cpu.cpu_load[4].avg
1368 ? 5% sched_debug.cpu.cpu_load[4].max
231.90 ? 4% sched_debug.cpu.cpu_load[4].stddev
1807 ? 4% sched_debug.cpu.curr->pid.avg
20367 ? 4% sched_debug.cpu.curr->pid.max
3573 ? 5% sched_debug.cpu.curr->pid.stddev
0.00 ? 6% sched_debug.cpu.next_balance.stddev
+38.6% 1502777 sched_debug.cpu.nr_load_updates.avg
+38.6% 1508302 sched_debug.cpu.nr_load_updates.max
+38.7% 1501073 sched_debug.cpu.nr_load_updates.min
0.95 ? 6% sched_debug.cpu.nr_running.avg
4.37 ? 10% sched_debug.cpu.nr_running.max
1.03 ? 9% sched_debug.cpu.nr_running.stddev
-22.4% 8355047 sched_debug.cpu.nr_switches.avg
-22.2% 8545010 sched_debug.cpu.nr_switches.max
-22.8% 8143037 sched_debug.cpu.nr_switches.min
-22.7% 114880 ? 7% sched_debug.cpu.nr_switches.stddev
13.85 sched_debug.cpu.nr_uninterruptible.avg
3296 ? 11% sched_debug.cpu.nr_uninterruptible.max
-3455 sched_debug.cpu.nr_uninterruptible.min
1997 ? 4% sched_debug.cpu.nr_uninterruptible.stddev
-22.7% 8356821 sched_debug.cpu.sched_count.avg
-22.5% 8546277 sched_debug.cpu.sched_count.max
-23.2% 8144743 sched_debug.cpu.sched_count.min
-20.9% 114359 ? 7% sched_debug.cpu.sched_count.stddev
+259.0% 2534721 sched_debug.cpu.sched_goidle.avg
+232.4% 2564059 sched_debug.cpu.sched_goidle.max
+286.9% 2494236 sched_debug.cpu.sched_goidle.min
15979 ? 7% sched_debug.cpu.sched_goidle.stddev
-39.9% 5780369 sched_debug.cpu.ttwu_count.avg
-61.7% 3441265 ? 4% sched_debug.cpu.ttwu_count.min
+244.2% 1440565 ? 7% sched_debug.cpu.ttwu_count.stddev
40366 ? 2% sched_debug.cpu.ttwu_local.avg
53476 ? 4% sched_debug.cpu.ttwu_local.max
30636 ? 3% sched_debug.cpu.ttwu_local.min
5405 ? 9% sched_debug.cpu.ttwu_local.stddev
+37.6% 1523810 sched_debug.cpu_clk
+37.6% 1523810 sched_debug.ktime
0.00 ? 65% sched_debug.rt_rq:/.rt_nr_migratory.avg
0.01 ? 65% sched_debug.rt_rq:/.rt_nr_migratory.max
0.00 ? 65% sched_debug.rt_rq:/.rt_nr_migratory.stddev
0.00 ? 65% sched_debug.rt_rq:/.rt_nr_running.avg
0.01 ? 65% sched_debug.rt_rq:/.rt_nr_running.max
0.00 ? 65% sched_debug.rt_rq:/.rt_nr_running.stddev
0.01 ? 23% sched_debug.rt_rq:/.rt_time.avg
+37.6% 1524235 sched_debug.sched_clk
0.00 perf-profile.calltrace.cycles-pp.md_flush_request.raid1_make_request.md_handle_request.md_make_request.generic_make_request
0.00 perf-profile.calltrace.cycles-pp.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_write_iter.__vfs_write
0.05 ?299% perf-profile.calltrace.cycles-pp.blkdev_issue_flush.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
0.00 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync
0.00 perf-profile.calltrace.cycles-pp.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_write_iter
0.00 perf-profile.calltrace.cycles-pp.md_make_request.generic_make_request.submit_bio.submit_bio_wait.blkdev_issue_flush
0.00 perf-profile.calltrace.cycles-pp.md_handle_request.md_make_request.generic_make_request.submit_bio.submit_bio_wait
0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.md_flush_request.raid1_make_request.md_handle_request.md_make_request
0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.md_flush_request.raid1_make_request.md_handle_request
62.75 ? 8% perf-profile.calltrace.cycles-pp.raid1_make_request.md_handle_request.md_make_request.generic_make_request.submit_bio
79.76 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write.ksys_write
80.49 perf-profile.calltrace.cycles-pp.xfs_file_write_iter.__vfs_write.vfs_write.ksys_write.do_syscall_64
80.50 perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
80.56 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
80.56 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
80.64 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
80.65 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
1.97 ? 8% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
1.96 ? 9% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
0.42 ? 68% perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
0.60 ? 17% perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate
0.59 ? 15% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.raid1_write_request.raid1_make_request.md_handle_request
0.64 ? 15% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
0.65 ? 10% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle
0.68 ? 10% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
0.69 ? 10% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
0.79 ? 11% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.83 ? 7% perf-profile.calltrace.cycles-pp.__schedule.schedule.raid1_write_request.raid1_make_request.md_handle_request
1.45 ? 22% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait.__xfs_log_force_lsn
0.83 ? 7% perf-profile.calltrace.cycles-pp.schedule.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
1.46 ? 22% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
1.46 ? 22% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn
4.79 ? 6% perf-profile.calltrace.cycles-pp.ret_from_fork
4.79 ? 6% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
1.64 ? 23% perf-profile.calltrace.cycles-pp.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
1.17 ? 8% perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request
1.23 ? 18% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request.submit_flushes
1.27 ? 17% perf-profile.calltrace.cycles-pp.raid1_write_request.raid1_make_request.md_handle_request.submit_flushes.process_one_work
1.27 ? 17% perf-profile.calltrace.cycles-pp.md_handle_request.submit_flushes.process_one_work.worker_thread.kthread
1.27 ? 17% perf-profile.calltrace.cycles-pp.raid1_make_request.md_handle_request.submit_flushes.process_one_work.worker_thread
1.27 ? 17% perf-profile.calltrace.cycles-pp.submit_flushes.process_one_work.worker_thread.kthread.ret_from_fork
1.65 ? 14% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.raid_end_bio_io
1.71 ? 14% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.raid_end_bio_io.raid1_end_write_request
1.71 ? 14% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.raid_end_bio_io.raid1_end_write_request.brd_make_request
1.86 ? 13% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.raid_end_bio_io.raid1_end_write_request.brd_make_request.generic_make_request
2.10 ? 10% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn
2.10 ? 10% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
2.11 ? 10% perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
2.16 ? 10% perf-profile.calltrace.cycles-pp.raid_end_bio_io.raid1_end_write_request.brd_make_request.generic_make_request.flush_bio_list
4.44 ? 15% perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
2.25 ? 10% perf-profile.calltrace.cycles-pp.raid1_end_write_request.brd_make_request.generic_make_request.flush_bio_list.flush_pending_writes
2.30 ? 20% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.raid1_write_request.raid1_make_request.md_handle_request
2.35 ? 20% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
2.81 ? 7% perf-profile.calltrace.cycles-pp.md_thread.kthread.ret_from_fork
2.81 ? 7% perf-profile.calltrace.cycles-pp.raid1d.md_thread.kthread.ret_from_fork
2.81 ? 7% perf-profile.calltrace.cycles-pp.flush_pending_writes.raid1d.md_thread.kthread.ret_from_fork
2.81 ? 7% perf-profile.calltrace.cycles-pp.flush_bio_list.flush_pending_writes.raid1d.md_thread.kthread
2.76 ? 7% perf-profile.calltrace.cycles-pp.generic_make_request.flush_bio_list.flush_pending_writes.raid1d.md_thread
2.73 ? 7% perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.flush_bio_list.flush_pending_writes.raid1d
4.35 ? 15% perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write
4.38 ? 27% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync
4.39 ? 27% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
4.40 ? 27% perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write
6.55 ? 27% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
6.55 ? 27% perf-profile.calltrace.cycles-pp._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write
9.72 ? 9% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
10.26 ? 9% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
11.36 ? 27% perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
11.97 ? 9% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
11.98 ? 9% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
11.98 ? 9% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
12.47 ? 8% perf-profile.calltrace.cycles-pp.secondary_startup_64
57.66 ? 10% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.raid1_write_request.raid1_make_request
57.73 ? 10% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request
57.85 ? 9% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
62.73 ? 8% perf-profile.calltrace.cycles-pp.raid1_write_request.raid1_make_request.md_handle_request.md_make_request.generic_make_request
62.76 ? 8% perf-profile.calltrace.cycles-pp.md_handle_request.md_make_request.generic_make_request.submit_bio.xfs_submit_ioend
62.79 ? 8% perf-profile.calltrace.cycles-pp.md_make_request.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages
62.81 ? 8% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages
62.81 ? 8% perf-profile.calltrace.cycles-pp.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
62.82 ? 8% perf-profile.calltrace.cycles-pp.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
63.09 ? 8% perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
63.10 ? 8% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_write_iter
63.11 ? 8% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_write_iter.__vfs_write
63.52 ? 7% perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
0.26 ? 19% perf-profile.children.cycles-pp.md_flush_request
0.35 ? 19% perf-profile.children.cycles-pp.submit_bio_wait
0.39 ? 19% perf-profile.children.cycles-pp.blkdev_issue_flush
0.02 ?123% perf-profile.children.cycles-pp._raw_spin_lock_irq
63.11 ? 8% perf-profile.children.cycles-pp.md_make_request
63.17 ? 8% perf-profile.children.cycles-pp.submit_bio
64.31 ? 8% perf-profile.children.cycles-pp.raid1_make_request
64.33 ? 8% perf-profile.children.cycles-pp.md_handle_request
65.92 ? 7% perf-profile.children.cycles-pp.generic_make_request
76.51 ? 2% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
79.76 perf-profile.children.cycles-pp.xfs_file_fsync
80.49 perf-profile.children.cycles-pp.xfs_file_write_iter
80.54 perf-profile.children.cycles-pp.__vfs_write
80.60 perf-profile.children.cycles-pp.vfs_write
80.61 perf-profile.children.cycles-pp.ksys_write
80.83 perf-profile.children.cycles-pp.do_syscall_64
80.83 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
1.97 ? 8% perf-profile.children.cycles-pp.worker_thread
1.96 ? 9% perf-profile.children.cycles-pp.process_one_work
0.07 ? 37% perf-profile.children.cycles-pp.xlog_cil_force_lsn
0.96 ? 4% perf-profile.children.cycles-pp.reschedule_interrupt
0.94 ? 4% perf-profile.children.cycles-pp.scheduler_ipi
0.47 ? 11% perf-profile.children.cycles-pp.finish_wait
0.10 ? 38% perf-profile.children.cycles-pp.xlog_cil_push
0.09 ? 35% perf-profile.children.cycles-pp.xlog_write
0.04 ? 67% perf-profile.children.cycles-pp.flush_work
0.16 ? 11% perf-profile.children.cycles-pp.reweight_entity
0.10 ? 23% perf-profile.children.cycles-pp.brd_lookup_page
0.23 ? 13% perf-profile.children.cycles-pp.__update_load_avg_se
0.07 ? 16% perf-profile.children.cycles-pp.delay_tsc
0.08 ? 15% perf-profile.children.cycles-pp.find_next_bit
0.14 ? 14% perf-profile.children.cycles-pp.native_write_msr
0.36 ? 8% perf-profile.children.cycles-pp.__orc_find
0.46 ? 7% perf-profile.children.cycles-pp.dequeue_task_fair
0.18 ? 14% perf-profile.children.cycles-pp.__module_text_address
0.19 ? 13% perf-profile.children.cycles-pp.is_module_text_address
0.12 ? 19% perf-profile.children.cycles-pp.kmem_cache_alloc
0.08 ? 11% perf-profile.children.cycles-pp.clear_page_erms
0.08 ? 28% perf-profile.children.cycles-pp.__indirect_thunk_start
0.10 ? 25% perf-profile.children.cycles-pp.xfs_trans_alloc
0.09 ? 18% perf-profile.children.cycles-pp.md_wakeup_thread
0.09 ? 26% perf-profile.children.cycles-pp.rebalance_domains
0.09 ? 26% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.09 ? 20% perf-profile.children.cycles-pp.ktime_get
0.27 ? 12% perf-profile.children.cycles-pp.idle_cpu
0.30 ? 9% perf-profile.children.cycles-pp.unwind_get_return_address
0.25 ? 13% perf-profile.children.cycles-pp.__module_address
0.13 ? 8% perf-profile.children.cycles-pp.brd_insert_page
0.16 ? 14% perf-profile.children.cycles-pp.task_tick_fair
0.18 ? 24% perf-profile.children.cycles-pp.bio_alloc_bioset
0.14 ? 27% perf-profile.children.cycles-pp.generic_make_request_checks
0.28 ? 11% perf-profile.children.cycles-pp.__kernel_text_address
0.22 ? 15% perf-profile.children.cycles-pp.wake_up_page_bit
0.27 ? 10% perf-profile.children.cycles-pp.kernel_text_address
0.11 ? 11% perf-profile.children.cycles-pp.get_page_from_freelist
0.11 ? 19% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
0.11 ? 7% perf-profile.children.cycles-pp.__alloc_pages_nodemask
0.19 ? 22% perf-profile.children.cycles-pp.xfs_do_writepage
0.37 ? 10% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.12 ? 13% perf-profile.children.cycles-pp.switch_mm
0.20 ? 19% perf-profile.children.cycles-pp.io_serial_in
0.31 ? 7% perf-profile.children.cycles-pp.dequeue_entity
0.13 ? 26% perf-profile.children.cycles-pp.tick_nohz_next_event
0.19 ? 19% perf-profile.children.cycles-pp.mempool_alloc
0.45 ? 6% perf-profile.children.cycles-pp.orc_find
0.29 ? 19% perf-profile.children.cycles-pp.xfs_destroy_ioend
0.30 ? 18% perf-profile.children.cycles-pp.call_bio_endio
0.23 ? 25% perf-profile.children.cycles-pp.xlog_state_done_syncing
0.15 ? 22% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.27 ? 23% perf-profile.children.cycles-pp.write_cache_pages
0.26 ? 16% perf-profile.children.cycles-pp.wait_for_xmitr
0.25 ? 14% perf-profile.children.cycles-pp.serial8250_console_putchar
0.26 ? 13% perf-profile.children.cycles-pp.uart_console_write
0.26 ? 15% perf-profile.children.cycles-pp.serial8250_console_write
0.27 ? 15% perf-profile.children.cycles-pp.console_unlock
0.26 ? 12% perf-profile.children.cycles-pp.scheduler_tick
0.28 ? 15% perf-profile.children.cycles-pp.irq_work_run_list
0.28 ? 14% perf-profile.children.cycles-pp.xlog_state_do_callback
0.27 ? 16% perf-profile.children.cycles-pp.irq_work_run
0.27 ? 16% perf-profile.children.cycles-pp.printk
0.27 ? 16% perf-profile.children.cycles-pp.vprintk_emit
0.27 ? 17% perf-profile.children.cycles-pp.irq_work_interrupt
0.27 ? 17% perf-profile.children.cycles-pp.smp_irq_work_interrupt
0.18 ? 16% perf-profile.children.cycles-pp.poll_idle
0.49 ? 11% perf-profile.children.cycles-pp.update_load_avg
1.59 ? 6% perf-profile.children.cycles-pp.__save_stack_trace
1.65 ? 6% perf-profile.children.cycles-pp.save_stack_trace_tsk
0.36 ? 13% perf-profile.children.cycles-pp.update_process_times
0.23 ? 22% perf-profile.children.cycles-pp.find_busiest_group
0.45 ? 18% perf-profile.children.cycles-pp.brd_do_bvec
0.38 ? 14% perf-profile.children.cycles-pp.tick_sched_handle
0.34 ? 26% perf-profile.children.cycles-pp.xfs_log_commit_cil
0.33 ? 23% perf-profile.children.cycles-pp.io_schedule
0.29 ? 27% perf-profile.children.cycles-pp.__softirqentry_text_start
0.36 ? 25% perf-profile.children.cycles-pp.__xfs_trans_commit
0.31 ? 26% perf-profile.children.cycles-pp.irq_exit
0.35 ? 23% perf-profile.children.cycles-pp.wait_on_page_bit_common
0.42 ? 14% perf-profile.children.cycles-pp.tick_sched_timer
0.39 ? 22% perf-profile.children.cycles-pp.__filemap_fdatawait_range
0.37 ? 9% perf-profile.children.cycles-pp.schedule_idle
0.34 ? 17% perf-profile.children.cycles-pp.menu_select
0.49 ? 22% perf-profile.children.cycles-pp.xfs_vn_update_time
0.51 ? 18% perf-profile.children.cycles-pp.xlog_iodone
0.51 ? 22% perf-profile.children.cycles-pp.file_update_time
0.51 ? 21% perf-profile.children.cycles-pp.xfs_file_aio_write_checks
0.60 ? 15% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.69 ? 16% perf-profile.children.cycles-pp.pick_next_task_fair
1.64 ? 10% perf-profile.children.cycles-pp.schedule
0.72 ? 21% perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
0.44 ? 22% perf-profile.children.cycles-pp.load_balance
0.74 ? 15% perf-profile.children.cycles-pp.hrtimer_interrupt
2.00 ? 9% perf-profile.children.cycles-pp.__schedule
1.09 ? 16% perf-profile.children.cycles-pp.smp_apic_timer_interrupt
1.09 ? 16% perf-profile.children.cycles-pp.apic_timer_interrupt
4.79 ? 6% perf-profile.children.cycles-pp.ret_from_fork
4.79 ? 6% perf-profile.children.cycles-pp.kthread
1.64 ? 23% perf-profile.children.cycles-pp.xlog_wait
1.27 ? 17% perf-profile.children.cycles-pp.submit_flushes
1.74 ? 4% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
2.16 ? 10% perf-profile.children.cycles-pp.raid_end_bio_io
2.25 ? 10% perf-profile.children.cycles-pp.raid1_end_write_request
4.44 ? 15% perf-profile.children.cycles-pp.xfs_log_force_lsn
2.73 ? 7% perf-profile.children.cycles-pp.brd_make_request
2.81 ? 7% perf-profile.children.cycles-pp.md_thread
2.81 ? 7% perf-profile.children.cycles-pp.raid1d
2.81 ? 7% perf-profile.children.cycles-pp.flush_pending_writes
2.81 ? 7% perf-profile.children.cycles-pp.flush_bio_list
7.44 ? 27% perf-profile.children.cycles-pp._raw_spin_lock
7.97 ? 20% perf-profile.children.cycles-pp.remove_wait_queue
10.12 ? 8% perf-profile.children.cycles-pp.intel_idle
10.71 ? 8% perf-profile.children.cycles-pp.cpuidle_enter_state
11.98 ? 9% perf-profile.children.cycles-pp.start_secondary
12.47 ? 8% perf-profile.children.cycles-pp.secondary_startup_64
12.47 ? 8% perf-profile.children.cycles-pp.cpu_startup_entry
12.49 ? 8% perf-profile.children.cycles-pp.do_idle
15.72 ? 23% perf-profile.children.cycles-pp.__xfs_log_force_lsn
59.12 ? 10% perf-profile.children.cycles-pp.prepare_to_wait_event
62.82 ? 8% perf-profile.children.cycles-pp.xfs_submit_ioend
63.10 ? 8% perf-profile.children.cycles-pp.xfs_vm_writepages
63.10 ? 8% perf-profile.children.cycles-pp.do_writepages
63.11 ? 8% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
63.52 ? 7% perf-profile.children.cycles-pp.file_write_and_wait_range
64.02 ? 8% perf-profile.children.cycles-pp.raid1_write_request
69.64 ? 5% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
76.31 ? 2% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.09 ? 19% perf-profile.self.cycles-pp.md_flush_request
0.07 ? 21% perf-profile.self.cycles-pp.account_entity_enqueue
0.08 ? 12% perf-profile.self.cycles-pp.pick_next_task_fair
0.08 ? 18% perf-profile.self.cycles-pp.___perf_sw_event
0.18 ? 9% perf-profile.self.cycles-pp.__update_load_avg_se
0.22 ? 10% perf-profile.self.cycles-pp.__schedule
0.15 ? 11% perf-profile.self.cycles-pp._raw_spin_lock
0.07 ? 16% perf-profile.self.cycles-pp.delay_tsc
0.07 ? 23% perf-profile.self.cycles-pp.set_next_entity
0.08 ? 15% perf-profile.self.cycles-pp.find_next_bit
0.14 ? 14% perf-profile.self.cycles-pp.native_write_msr
0.07 ? 23% perf-profile.self.cycles-pp.kmem_cache_alloc
0.36 ? 8% perf-profile.self.cycles-pp.__orc_find
0.21 ? 12% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.08 ? 11% perf-profile.self.cycles-pp.clear_page_erms
0.08 ? 28% perf-profile.self.cycles-pp.__indirect_thunk_start
0.08 ? 20% perf-profile.self.cycles-pp.md_wakeup_thread
0.43 ? 12% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.27 ? 12% perf-profile.self.cycles-pp.idle_cpu
0.25 ? 13% perf-profile.self.cycles-pp.__module_address
0.17 ? 14% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.20 ? 19% perf-profile.self.cycles-pp.io_serial_in
0.32 ? 15% perf-profile.self.cycles-pp.update_load_avg
0.15 ? 17% perf-profile.self.cycles-pp.poll_idle
0.15 ? 16% perf-profile.self.cycles-pp.menu_select
0.18 ? 24% perf-profile.self.cycles-pp.find_busiest_group
0.35 ? 21% perf-profile.self.cycles-pp.raid1_write_request
10.12 ? 8% perf-profile.self.cycles-pp.intel_idle

aim7.jobs-per-min

--------------------------------------------------+
+++ ++++++ + + ++++++++++++ ++ ++|
+ +++++ ++.++ + ++ ++ + ++ |
|
|
|
|
|
|
|
|
O O |
OO OOOOOOOOOOOOOO OOOOOOOOO O |
|
--------------------------------------------------+

estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
may affect actual performance.

Attachments:

(No filename) (50.43 kB)
config-4.17.0-rc1-00189-g5a409b4 (167.13 kB)
job-script (7.69 kB)
job.yaml (5.37 kB)
reproduce (1.04 kB)
Download all attachments

2018-07-16 07:56:02

by Aaron Lu

[permalink] [raw]

Subject: Re: [LKP] [lkp-robot] [MD] 5a409b4f56: aim7.jobs-per-min -27.5% regression

Ping...
Any update on this?
Feel free to ask me for any additional data if you need.

Thanks,
Aaron

On Mon, Jun 04, 2018 at 02:42:03PM +0800, kernel test robot wrote:
>
> Greeting,
>
> FYI, we noticed a -27.5% regression of aim7.jobs-per-min due to commit:
>
>
> commit: 5a409b4f56d50b212334f338cb8465d65550cd85 ("MD: fix lock contention for flush bios")
> https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
>
> in testcase: aim7
> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
> with following parameters:
>
> disk: 4BRD_12G
> md: RAID1
> fs: xfs
> test: sync_disk_rw
> load: 600
> cpufreq_governor: performance
>
> test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
> test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
> =========================================================================================
> compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
> gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.2/600/RAID1/debian-x86_64-2016-08-31.cgz/lkp-ivb-ep01/sync_disk_rw/aim7
>
> commit:
> 448ec638c6 ("md/raid5: Assigning NULL to sh->batch_head before testing bit R5_Overlap of a stripe")
> 5a409b4f56 ("MD: fix lock contention for flush bios")
>
> 448ec638c6bcf369 5a409b4f56d50b212334f338cb
> ---------------- --------------------------
> %stddev %change %stddev
> \ | \
> 1640 -27.5% 1189 aim7.jobs-per-min
> 2194 +37.9% 3026 aim7.time.elapsed_time
> 2194 +37.9% 3026 aim7.time.elapsed_time.max
> 50990311 -95.8% 2148266 aim7.time.involuntary_context_switches
> 107965 ? 4% -26.4% 79516 ? 2% aim7.time.minor_page_faults
> 49.14 +82.5% 89.66 ? 2% aim7.time.user_time
> 7.123e+08 -35.7% 4.582e+08 aim7.time.voluntary_context_switches
> 672282 +36.8% 919615 interrupts.CAL:Function_call_interrupts
> 16631387 ? 2% -39.9% 9993075 ? 7% softirqs.RCU
> 9708009 +186.1% 27778773 softirqs.SCHED
> 33436649 +45.5% 48644912 softirqs.TIMER
> 4.16 -2.1 2.01 mpstat.cpu.idle%
> 0.24 ? 2% +27.7 27.91 mpstat.cpu.iowait%
> 95.51 -25.6 69.94 mpstat.cpu.sys%
> 0.09 +0.0 0.13 mpstat.cpu.usr%
> 6051756 ? 3% +59.0% 9623085 numa-numastat.node0.local_node
> 6055311 ? 3% +59.0% 9626996 numa-numastat.node0.numa_hit
> 6481209 ? 3% +48.4% 9616310 numa-numastat.node1.local_node
> 6485866 ? 3% +48.3% 9620756 numa-numastat.node1.numa_hit
> 61404 -27.7% 44424 vmstat.io.bo
> 2.60 ? 18% +11519.2% 302.10 vmstat.procs.b
> 304.10 -84.9% 45.80 ? 2% vmstat.procs.r
> 400477 -43.5% 226094 vmstat.system.cs
> 166461 -49.9% 83332 vmstat.system.in
> 78397 +27.0% 99567 meminfo.Dirty
> 14427 +18.4% 17082 meminfo.Inactive(anon)
> 1963 ? 5% +5.4% 2068 ? 4% meminfo.Mlocked
> 101143 +991.0% 1103488 meminfo.SUnreclaim
> 53684 ? 4% -18.1% 43946 ? 3% meminfo.Shmem
> 175580 +571.4% 1178829 meminfo.Slab
> 39406 +26.2% 49717 numa-meminfo.node0.Dirty
> 1767204 ? 10% +37.2% 2425487 ? 2% numa-meminfo.node0.MemUsed
> 51634 ? 18% +979.3% 557316 numa-meminfo.node0.SUnreclaim
> 92259 ? 13% +551.7% 601288 numa-meminfo.node0.Slab
> 38969 +28.0% 49863 numa-meminfo.node1.Dirty
> 1895204 ? 10% +24.7% 2363037 ? 3% numa-meminfo.node1.MemUsed
> 49512 ? 19% +1003.1% 546165 numa-meminfo.node1.SUnreclaim
> 83323 ? 14% +593.1% 577534 numa-meminfo.node1.Slab
> 2.524e+09 +894.5% 2.51e+10 cpuidle.C1.time
> 50620790 +316.5% 2.109e+08 cpuidle.C1.usage
> 3.965e+08 +1871.1% 7.815e+09 cpuidle.C1E.time
> 5987788 +186.1% 17129412 cpuidle.C1E.usage
> 2.506e+08 +97.5% 4.948e+08 ? 2% cpuidle.C3.time
> 2923498 -55.7% 1295033 cpuidle.C3.usage
> 5.327e+08 +179.9% 1.491e+09 cpuidle.C6.time
> 779874 ? 2% +229.3% 2567769 cpuidle.C6.usage
> 6191357 +3333.6% 2.126e+08 cpuidle.POLL.time
> 204095 +1982.1% 4249504 cpuidle.POLL.usage
> 9850 +26.3% 12444 numa-vmstat.node0.nr_dirty
> 12908 ? 18% +979.3% 139321 numa-vmstat.node0.nr_slab_unreclaimable
> 8876 +29.6% 11505 numa-vmstat.node0.nr_zone_write_pending
> 3486319 ? 4% +55.1% 5407021 numa-vmstat.node0.numa_hit
> 3482713 ? 4% +55.1% 5403066 numa-vmstat.node0.numa_local
> 9743 +28.1% 12479 numa-vmstat.node1.nr_dirty
> 12377 ? 19% +1003.1% 136532 numa-vmstat.node1.nr_slab_unreclaimable
> 9287 +30.0% 12074 numa-vmstat.node1.nr_zone_write_pending
> 3678995 ? 4% +44.8% 5326772 numa-vmstat.node1.numa_hit
> 3497785 ? 4% +47.1% 5145705 numa-vmstat.node1.numa_local
> 252.70 +100.2% 505.90 slabinfo.biovec-max.active_objs
> 282.70 +99.1% 562.90 slabinfo.biovec-max.num_objs
> 2978 ? 17% +52.5% 4543 ? 14% slabinfo.dmaengine-unmap-16.active_objs
> 2978 ? 17% +52.5% 4543 ? 14% slabinfo.dmaengine-unmap-16.num_objs
> 2078 +147.9% 5153 ? 11% slabinfo.ip6_dst_cache.active_objs
> 2078 +148.1% 5157 ? 11% slabinfo.ip6_dst_cache.num_objs
> 5538 ? 2% +26.2% 6990 ? 3% slabinfo.kmalloc-1024.active_objs
> 5586 ? 3% +27.1% 7097 ? 3% slabinfo.kmalloc-1024.num_objs
> 6878 +47.6% 10151 ? 5% slabinfo.kmalloc-192.active_objs
> 6889 +47.5% 10160 ? 5% slabinfo.kmalloc-192.num_objs
> 9843 ? 5% +1.6e+05% 16002876 slabinfo.kmalloc-64.active_objs
> 161.90 ? 4% +1.5e+05% 250044 slabinfo.kmalloc-64.active_slabs
> 10386 ? 4% +1.5e+05% 16002877 slabinfo.kmalloc-64.num_objs
> 161.90 ? 4% +1.5e+05% 250044 slabinfo.kmalloc-64.num_slabs
> 432.80 ? 12% +45.2% 628.50 ? 6% slabinfo.nfs_read_data.active_objs
> 432.80 ? 12% +45.2% 628.50 ? 6% slabinfo.nfs_read_data.num_objs
> 3956 -23.1% 3041 slabinfo.pool_workqueue.active_objs
> 4098 -19.8% 3286 slabinfo.pool_workqueue.num_objs
> 360.50 ? 15% +56.6% 564.70 ? 11% slabinfo.secpath_cache.active_objs
> 360.50 ? 15% +56.6% 564.70 ? 11% slabinfo.secpath_cache.num_objs
> 35373 ? 2% -8.3% 32432 proc-vmstat.nr_active_anon
> 19595 +27.1% 24914 proc-vmstat.nr_dirty
> 3607 +18.4% 4270 proc-vmstat.nr_inactive_anon
> 490.30 ? 5% +5.4% 516.90 ? 4% proc-vmstat.nr_mlock
> 13421 ? 4% -18.1% 10986 ? 3% proc-vmstat.nr_shmem
> 18608 +1.2% 18834 proc-vmstat.nr_slab_reclaimable
> 25286 +991.0% 275882 proc-vmstat.nr_slab_unreclaimable
> 35405 ? 2% -8.3% 32465 proc-vmstat.nr_zone_active_anon
> 3607 +18.4% 4270 proc-vmstat.nr_zone_inactive_anon
> 18161 +29.8% 23572 proc-vmstat.nr_zone_write_pending
> 76941 ? 5% -36.8% 48622 ? 4% proc-vmstat.numa_hint_faults
> 33878 ? 7% -35.5% 21836 ? 5% proc-vmstat.numa_hint_faults_local
> 12568956 +53.3% 19272377 proc-vmstat.numa_hit
> 12560739 +53.4% 19264015 proc-vmstat.numa_local
> 17938 ? 3% -33.5% 11935 ? 2% proc-vmstat.numa_pages_migrated
> 78296 ? 5% -36.0% 50085 ? 4% proc-vmstat.numa_pte_updates
> 8848 ? 6% -38.2% 5466 ? 6% proc-vmstat.pgactivate
> 8874568 ? 8% +368.7% 41590920 proc-vmstat.pgalloc_normal
> 5435965 +39.2% 7564148 proc-vmstat.pgfault
> 12863707 +255.1% 45683570 proc-vmstat.pgfree
> 17938 ? 3% -33.5% 11935 ? 2% proc-vmstat.pgmigrate_success
> 1.379e+13 -40.8% 8.17e+12 perf-stat.branch-instructions
> 0.30 +0.1 0.42 perf-stat.branch-miss-rate%
> 4.2e+10 -17.6% 3.462e+10 perf-stat.branch-misses
> 15.99 +3.8 19.74 perf-stat.cache-miss-rate%
> 3.779e+10 -21.6% 2.963e+10 perf-stat.cache-misses
> 2.364e+11 -36.5% 1.501e+11 perf-stat.cache-references
> 8.795e+08 -22.2% 6.84e+08 perf-stat.context-switches
> 4.44 -7.2% 4.12 perf-stat.cpi
> 2.508e+14 -44.5% 1.393e+14 perf-stat.cpu-cycles
> 36915392 +60.4% 59211221 perf-stat.cpu-migrations
> 0.29 ? 2% +0.0 0.34 ? 4% perf-stat.dTLB-load-miss-rate%
> 4.14e+10 -30.2% 2.89e+10 ? 4% perf-stat.dTLB-load-misses
> 1.417e+13 -40.1% 8.491e+12 perf-stat.dTLB-loads
> 0.20 ? 4% -0.0 0.18 ? 5% perf-stat.dTLB-store-miss-rate%
> 3.072e+09 ? 4% -28.0% 2.21e+09 ? 4% perf-stat.dTLB-store-misses
> 1.535e+12 -20.2% 1.225e+12 perf-stat.dTLB-stores
> 90.73 -11.7 79.07 perf-stat.iTLB-load-miss-rate%
> 8.291e+09 -6.6% 7.743e+09 perf-stat.iTLB-load-misses
> 8.473e+08 +141.8% 2.049e+09 ? 3% perf-stat.iTLB-loads
> 5.646e+13 -40.2% 3.378e+13 perf-stat.instructions
> 6810 -35.9% 4362 perf-stat.instructions-per-iTLB-miss
> 0.23 +7.8% 0.24 perf-stat.ipc
> 5326672 +39.2% 7413706 perf-stat.minor-faults
> 1.873e+10 -29.9% 1.312e+10 perf-stat.node-load-misses
> 2.093e+10 -29.2% 1.481e+10 perf-stat.node-loads
> 39.38 -0.7 38.72 perf-stat.node-store-miss-rate%
> 1.087e+10 -16.6% 9.069e+09 perf-stat.node-store-misses
> 1.673e+10 -14.2% 1.435e+10 perf-stat.node-stores
> 5326695 +39.2% 7413708 perf-stat.page-faults
> 1875095 ? 7% -54.8% 846645 ? 16% sched_debug.cfs_rq:/.MIN_vruntime.avg
> 32868920 ? 6% -35.7% 21150379 ? 14% sched_debug.cfs_rq:/.MIN_vruntime.max
> 7267340 ? 5% -44.7% 4015798 ? 14% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> 4278 ? 7% -54.7% 1939 ? 11% sched_debug.cfs_rq:/.exec_clock.stddev
> 245.48 ? 2% +65.3% 405.75 ? 7% sched_debug.cfs_rq:/.load_avg.avg
> 2692 ? 6% +126.0% 6087 ? 7% sched_debug.cfs_rq:/.load_avg.max
> 33.09 -73.0% 8.94 ? 7% sched_debug.cfs_rq:/.load_avg.min
> 507.40 ? 4% +128.0% 1156 ? 7% sched_debug.cfs_rq:/.load_avg.stddev
> 1875095 ? 7% -54.8% 846645 ? 16% sched_debug.cfs_rq:/.max_vruntime.avg
> 32868921 ? 6% -35.7% 21150379 ? 14% sched_debug.cfs_rq:/.max_vruntime.max
> 7267341 ? 5% -44.7% 4015798 ? 14% sched_debug.cfs_rq:/.max_vruntime.stddev
> 35887197 -13.2% 31149130 sched_debug.cfs_rq:/.min_vruntime.avg
> 37385506 -14.3% 32043914 sched_debug.cfs_rq:/.min_vruntime.max
> 34416296 -12.3% 30183927 sched_debug.cfs_rq:/.min_vruntime.min
> 1228844 ? 8% -52.6% 582759 ? 4% sched_debug.cfs_rq:/.min_vruntime.stddev
> 0.83 -28.1% 0.60 ? 6% sched_debug.cfs_rq:/.nr_running.avg
> 2.07 ? 3% -24.6% 1.56 ? 8% sched_debug.cfs_rq:/.nr_running.max
> 20.52 ? 4% -48.8% 10.52 ? 3% sched_debug.cfs_rq:/.nr_spread_over.avg
> 35.96 ? 5% -42.2% 20.77 ? 9% sched_debug.cfs_rq:/.nr_spread_over.max
> 8.97 ? 11% -44.5% 4.98 ? 8% sched_debug.cfs_rq:/.nr_spread_over.min
> 6.40 ? 12% -45.5% 3.49 ? 7% sched_debug.cfs_rq:/.nr_spread_over.stddev
> 21.78 ? 7% +143.3% 53.00 ? 9% sched_debug.cfs_rq:/.runnable_load_avg.avg
> 328.86 ? 18% +303.4% 1326 ? 14% sched_debug.cfs_rq:/.runnable_load_avg.max
> 55.97 ? 17% +286.0% 216.07 ? 13% sched_debug.cfs_rq:/.runnable_load_avg.stddev
> 0.10 ? 29% -82.4% 0.02 ? 50% sched_debug.cfs_rq:/.spread.avg
> 3.43 ? 25% -79.9% 0.69 ? 50% sched_debug.cfs_rq:/.spread.max
> 0.56 ? 26% -80.7% 0.11 ? 50% sched_debug.cfs_rq:/.spread.stddev
> 1228822 ? 8% -52.6% 582732 ? 4% sched_debug.cfs_rq:/.spread0.stddev
> 992.30 -24.9% 745.56 ? 2% sched_debug.cfs_rq:/.util_avg.avg
> 1485 -18.1% 1217 ? 2% sched_debug.cfs_rq:/.util_avg.max
> 515.45 ? 2% -25.2% 385.73 ? 6% sched_debug.cfs_rq:/.util_avg.min
> 201.54 -14.9% 171.52 ? 3% sched_debug.cfs_rq:/.util_avg.stddev
> 248.73 ? 6% -38.1% 154.02 ? 8% sched_debug.cfs_rq:/.util_est_enqueued.avg
> 222.78 ? 3% -15.8% 187.58 ? 2% sched_debug.cfs_rq:/.util_est_enqueued.stddev
> 77097 ? 4% +278.4% 291767 ? 11% sched_debug.cpu.avg_idle.avg
> 181319 ? 6% +298.7% 722862 ? 3% sched_debug.cpu.avg_idle.max
> 19338 +392.3% 95203 ? 17% sched_debug.cpu.avg_idle.min
> 34877 ? 6% +303.5% 140732 ? 6% sched_debug.cpu.avg_idle.stddev
> 1107408 +37.6% 1523823 sched_debug.cpu.clock.avg
> 1107427 +37.6% 1523834 sched_debug.cpu.clock.max
> 1107385 +37.6% 1523811 sched_debug.cpu.clock.min
> 13.10 ? 9% -48.1% 6.80 ? 8% sched_debug.cpu.clock.stddev
> 1107408 +37.6% 1523823 sched_debug.cpu.clock_task.avg
> 1107427 +37.6% 1523834 sched_debug.cpu.clock_task.max
> 1107385 +37.6% 1523811 sched_debug.cpu.clock_task.min
> 13.10 ? 9% -48.1% 6.80 ? 8% sched_debug.cpu.clock_task.stddev
> 30.36 ? 7% +107.7% 63.06 ? 12% sched_debug.cpu.cpu_load[0].avg
> 381.48 ? 18% +269.8% 1410 ? 18% sched_debug.cpu.cpu_load[0].max
> 63.92 ? 18% +262.2% 231.50 ? 17% sched_debug.cpu.cpu_load[0].stddev
> 31.34 ? 5% +118.4% 68.44 ? 9% sched_debug.cpu.cpu_load[1].avg
> 323.62 ? 17% +349.5% 1454 ? 14% sched_debug.cpu.cpu_load[1].max
> 53.23 ? 16% +350.3% 239.71 ? 13% sched_debug.cpu.cpu_load[1].stddev
> 32.15 ? 3% +129.4% 73.74 ? 6% sched_debug.cpu.cpu_load[2].avg
> 285.20 ? 14% +420.8% 1485 ? 9% sched_debug.cpu.cpu_load[2].max
> 46.66 ? 12% +430.0% 247.32 ? 8% sched_debug.cpu.cpu_load[2].stddev
> 33.02 ? 2% +133.2% 77.00 ? 3% sched_debug.cpu.cpu_load[3].avg
> 252.16 ? 10% +481.2% 1465 ? 7% sched_debug.cpu.cpu_load[3].max
> 40.74 ? 8% +503.2% 245.72 ? 6% sched_debug.cpu.cpu_load[3].stddev
> 33.86 +131.5% 78.38 ? 2% sched_debug.cpu.cpu_load[4].avg
> 219.81 ? 8% +522.6% 1368 ? 5% sched_debug.cpu.cpu_load[4].max
> 35.45 ? 7% +554.2% 231.90 ? 4% sched_debug.cpu.cpu_load[4].stddev
> 2600 ? 4% -30.5% 1807 ? 4% sched_debug.cpu.curr->pid.avg
> 25309 ? 4% -19.5% 20367 ? 4% sched_debug.cpu.curr->pid.max
> 4534 ? 7% -21.2% 3573 ? 5% sched_debug.cpu.curr->pid.stddev
> 0.00 ? 2% -27.6% 0.00 ? 6% sched_debug.cpu.next_balance.stddev
> 1083917 +38.6% 1502777 sched_debug.cpu.nr_load_updates.avg
> 1088142 +38.6% 1508302 sched_debug.cpu.nr_load_updates.max
> 1082048 +38.7% 1501073 sched_debug.cpu.nr_load_updates.min
> 3.53 ? 6% -73.0% 0.95 ? 6% sched_debug.cpu.nr_running.avg
> 11.54 ? 3% -62.1% 4.37 ? 10% sched_debug.cpu.nr_running.max
> 3.10 ? 3% -66.8% 1.03 ? 9% sched_debug.cpu.nr_running.stddev
> 10764176 -22.4% 8355047 sched_debug.cpu.nr_switches.avg
> 10976436 -22.2% 8545010 sched_debug.cpu.nr_switches.max
> 10547712 -22.8% 8143037 sched_debug.cpu.nr_switches.min
> 148628 ? 3% -22.7% 114880 ? 7% sched_debug.cpu.nr_switches.stddev
> 11.13 ? 2% +24.5% 13.85 sched_debug.cpu.nr_uninterruptible.avg
> 6420 ? 8% -48.7% 3296 ? 11% sched_debug.cpu.nr_uninterruptible.max
> -5500 -37.2% -3455 sched_debug.cpu.nr_uninterruptible.min
> 3784 ? 6% -47.2% 1997 ? 4% sched_debug.cpu.nr_uninterruptible.stddev
> 10812670 -22.7% 8356821 sched_debug.cpu.sched_count.avg
> 11020646 -22.5% 8546277 sched_debug.cpu.sched_count.max
> 10601390 -23.2% 8144743 sched_debug.cpu.sched_count.min
> 144529 ? 3% -20.9% 114359 ? 7% sched_debug.cpu.sched_count.stddev
> 706116 +259.0% 2534721 sched_debug.cpu.sched_goidle.avg
> 771307 +232.4% 2564059 sched_debug.cpu.sched_goidle.max
> 644658 +286.9% 2494236 sched_debug.cpu.sched_goidle.min
> 49847 ? 6% -67.9% 15979 ? 7% sched_debug.cpu.sched_goidle.stddev
> 9618827 -39.9% 5780369 sched_debug.cpu.ttwu_count.avg
> 8990451 -61.7% 3441265 ? 4% sched_debug.cpu.ttwu_count.min
> 418563 ? 25% +244.2% 1440565 ? 7% sched_debug.cpu.ttwu_count.stddev
> 640964 -93.7% 40366 ? 2% sched_debug.cpu.ttwu_local.avg
> 679527 -92.1% 53476 ? 4% sched_debug.cpu.ttwu_local.max
> 601661 -94.9% 30636 ? 3% sched_debug.cpu.ttwu_local.min
> 24242 ? 21% -77.7% 5405 ? 9% sched_debug.cpu.ttwu_local.stddev
> 1107383 +37.6% 1523810 sched_debug.cpu_clk
> 1107383 +37.6% 1523810 sched_debug.ktime
> 0.00 -49.4% 0.00 ? 65% sched_debug.rt_rq:/.rt_nr_migratory.avg
> 0.03 -49.4% 0.01 ? 65% sched_debug.rt_rq:/.rt_nr_migratory.max
> 0.00 -49.4% 0.00 ? 65% sched_debug.rt_rq:/.rt_nr_migratory.stddev
> 0.00 -49.4% 0.00 ? 65% sched_debug.rt_rq:/.rt_nr_running.avg
> 0.03 -49.4% 0.01 ? 65% sched_debug.rt_rq:/.rt_nr_running.max
> 0.00 -49.4% 0.00 ? 65% sched_debug.rt_rq:/.rt_nr_running.stddev
> 0.01 ? 8% +79.9% 0.01 ? 23% sched_debug.rt_rq:/.rt_time.avg
> 1107805 +37.6% 1524235 sched_debug.sched_clk
> 87.59 -87.6 0.00 perf-profile.calltrace.cycles-pp.md_flush_request.raid1_make_request.md_handle_request.md_make_request.generic_make_request
> 87.57 -87.6 0.00 perf-profile.calltrace.cycles-pp.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> 87.59 -87.5 0.05 ?299% perf-profile.calltrace.cycles-pp.blkdev_issue_flush.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
> 87.51 -87.5 0.00 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync
> 87.51 -87.5 0.00 perf-profile.calltrace.cycles-pp.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_write_iter
> 87.50 -87.5 0.00 perf-profile.calltrace.cycles-pp.md_make_request.generic_make_request.submit_bio.submit_bio_wait.blkdev_issue_flush
> 87.50 -87.5 0.00 perf-profile.calltrace.cycles-pp.md_handle_request.md_make_request.generic_make_request.submit_bio.submit_bio_wait
> 82.37 -82.4 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.md_flush_request.raid1_make_request.md_handle_request.md_make_request
> 82.23 -82.2 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.md_flush_request.raid1_make_request.md_handle_request
> 87.79 -25.0 62.75 ? 8% perf-profile.calltrace.cycles-pp.raid1_make_request.md_handle_request.md_make_request.generic_make_request.submit_bio
> 92.78 -13.0 79.76 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write.ksys_write
> 93.08 -12.6 80.49 perf-profile.calltrace.cycles-pp.xfs_file_write_iter.__vfs_write.vfs_write.ksys_write.do_syscall_64
> 93.08 -12.6 80.50 perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 93.11 -12.6 80.56 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 93.11 -12.6 80.56 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 93.14 -12.5 80.64 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 93.15 -12.5 80.65 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 3.40 ? 2% -1.4 1.97 ? 8% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
> 3.33 ? 2% -1.4 1.96 ? 9% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
> 1.12 ? 2% -0.7 0.42 ? 68% perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
> 1.16 ? 2% -0.6 0.60 ? 17% perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate
> 0.00 +0.6 0.59 ? 15% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.raid1_write_request.raid1_make_request.md_handle_request
> 0.00 +0.6 0.64 ? 15% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
> 0.00 +0.7 0.65 ? 10% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle
> 0.00 +0.7 0.68 ? 10% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
> 0.00 +0.7 0.69 ? 10% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
> 0.00 +0.8 0.79 ? 11% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 0.00 +0.8 0.83 ? 7% perf-profile.calltrace.cycles-pp.__schedule.schedule.raid1_write_request.raid1_make_request.md_handle_request
> 0.62 ? 3% +0.8 1.45 ? 22% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait.__xfs_log_force_lsn
> 0.00 +0.8 0.83 ? 7% perf-profile.calltrace.cycles-pp.schedule.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
> 0.63 ? 2% +0.8 1.46 ? 22% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 0.62 ? 2% +0.8 1.46 ? 22% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn
> 3.92 ? 2% +0.9 4.79 ? 6% perf-profile.calltrace.cycles-pp.ret_from_fork
> 3.92 ? 2% +0.9 4.79 ? 6% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
> 0.69 ? 2% +0.9 1.64 ? 23% perf-profile.calltrace.cycles-pp.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
> 0.00 +1.2 1.17 ? 8% perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request
> 0.00 +1.2 1.23 ? 18% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request.submit_flushes
> 0.00 +1.3 1.27 ? 17% perf-profile.calltrace.cycles-pp.raid1_write_request.raid1_make_request.md_handle_request.submit_flushes.process_one_work
> 0.00 +1.3 1.27 ? 17% perf-profile.calltrace.cycles-pp.md_handle_request.submit_flushes.process_one_work.worker_thread.kthread
> 0.00 +1.3 1.27 ? 17% perf-profile.calltrace.cycles-pp.raid1_make_request.md_handle_request.submit_flushes.process_one_work.worker_thread
> 0.00 +1.3 1.27 ? 17% perf-profile.calltrace.cycles-pp.submit_flushes.process_one_work.worker_thread.kthread.ret_from_fork
> 0.00 +1.6 1.65 ? 14% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.raid_end_bio_io
> 0.00 +1.7 1.71 ? 14% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.raid_end_bio_io.raid1_end_write_request
> 0.00 +1.7 1.71 ? 14% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.raid_end_bio_io.raid1_end_write_request.brd_make_request
> 0.00 +1.9 1.86 ? 13% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.raid_end_bio_io.raid1_end_write_request.brd_make_request.generic_make_request
> 0.00 +2.1 2.10 ? 10% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn
> 0.00 +2.1 2.10 ? 10% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 0.00 +2.1 2.11 ? 10% perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
> 0.00 +2.2 2.16 ? 10% perf-profile.calltrace.cycles-pp.raid_end_bio_io.raid1_end_write_request.brd_make_request.generic_make_request.flush_bio_list
> 2.24 ? 4% +2.2 4.44 ? 15% perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
> 0.00 +2.3 2.25 ? 10% perf-profile.calltrace.cycles-pp.raid1_end_write_request.brd_make_request.generic_make_request.flush_bio_list.flush_pending_writes
> 0.00 +2.3 2.30 ? 20% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.raid1_write_request.raid1_make_request.md_handle_request
> 0.00 +2.4 2.35 ? 20% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
> 0.37 ? 65% +2.4 2.81 ? 7% perf-profile.calltrace.cycles-pp.md_thread.kthread.ret_from_fork
> 0.26 ?100% +2.5 2.81 ? 7% perf-profile.calltrace.cycles-pp.raid1d.md_thread.kthread.ret_from_fork
> 0.26 ?100% +2.5 2.81 ? 7% perf-profile.calltrace.cycles-pp.flush_pending_writes.raid1d.md_thread.kthread.ret_from_fork
> 0.26 ?100% +2.6 2.81 ? 7% perf-profile.calltrace.cycles-pp.flush_bio_list.flush_pending_writes.raid1d.md_thread.kthread
> 0.10 ?200% +2.7 2.76 ? 7% perf-profile.calltrace.cycles-pp.generic_make_request.flush_bio_list.flush_pending_writes.raid1d.md_thread
> 0.00 +2.7 2.73 ? 7% perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.flush_bio_list.flush_pending_writes.raid1d
> 1.20 ? 3% +3.1 4.35 ? 15% perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> 0.63 ? 6% +3.8 4.38 ? 27% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync
> 0.63 ? 5% +3.8 4.39 ? 27% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
> 0.63 ? 5% +3.8 4.40 ? 27% perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> 1.26 ? 5% +5.3 6.55 ? 27% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
> 1.27 ? 5% +5.3 6.55 ? 27% perf-profile.calltrace.cycles-pp._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> 1.30 ? 4% +8.4 9.72 ? 9% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
> 1.33 ? 4% +8.9 10.26 ? 9% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 2.28 ? 2% +9.1 11.36 ? 27% perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
> 1.59 ? 4% +10.4 11.97 ? 9% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 1.59 ? 4% +10.4 11.98 ? 9% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
> 1.59 ? 4% +10.4 11.98 ? 9% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
> 1.63 ? 4% +10.8 12.47 ? 8% perf-profile.calltrace.cycles-pp.secondary_startup_64
> 0.00 +57.7 57.66 ? 10% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.raid1_write_request.raid1_make_request
> 0.00 +57.7 57.73 ? 10% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request
> 0.05 ?299% +57.8 57.85 ? 9% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
> 0.19 ?154% +62.5 62.73 ? 8% perf-profile.calltrace.cycles-pp.raid1_write_request.raid1_make_request.md_handle_request.md_make_request.generic_make_request
> 0.19 ?154% +62.6 62.76 ? 8% perf-profile.calltrace.cycles-pp.md_handle_request.md_make_request.generic_make_request.submit_bio.xfs_submit_ioend
> 0.19 ?154% +62.6 62.79 ? 8% perf-profile.calltrace.cycles-pp.md_make_request.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages
> 0.20 ?154% +62.6 62.81 ? 8% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages
> 0.20 ?154% +62.6 62.81 ? 8% perf-profile.calltrace.cycles-pp.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> 0.20 ?154% +62.6 62.82 ? 8% perf-profile.calltrace.cycles-pp.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> 0.29 ?125% +62.8 63.09 ? 8% perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
> 0.29 ?126% +62.8 63.10 ? 8% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_write_iter
> 0.29 ?125% +62.8 63.11 ? 8% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> 0.62 ? 41% +62.9 63.52 ? 7% perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
> 88.51 -88.2 0.26 ? 19% perf-profile.children.cycles-pp.md_flush_request
> 87.57 -87.2 0.35 ? 19% perf-profile.children.cycles-pp.submit_bio_wait
> 87.59 -87.2 0.39 ? 19% perf-profile.children.cycles-pp.blkdev_issue_flush
> 83.26 -83.2 0.02 ?123% perf-profile.children.cycles-pp._raw_spin_lock_irq
> 88.85 -25.7 63.11 ? 8% perf-profile.children.cycles-pp.md_make_request
> 88.90 -25.7 63.17 ? 8% perf-profile.children.cycles-pp.submit_bio
> 88.83 -24.5 64.31 ? 8% perf-profile.children.cycles-pp.raid1_make_request
> 88.84 -24.5 64.33 ? 8% perf-profile.children.cycles-pp.md_handle_request
> 89.38 -23.5 65.92 ? 7% perf-profile.children.cycles-pp.generic_make_request
> 89.90 -13.4 76.51 ? 2% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
> 92.79 -13.0 79.76 perf-profile.children.cycles-pp.xfs_file_fsync
> 93.08 -12.6 80.49 perf-profile.children.cycles-pp.xfs_file_write_iter
> 93.09 -12.6 80.54 perf-profile.children.cycles-pp.__vfs_write
> 93.13 -12.5 80.60 perf-profile.children.cycles-pp.vfs_write
> 93.13 -12.5 80.61 perf-profile.children.cycles-pp.ksys_write
> 93.22 -12.4 80.83 perf-profile.children.cycles-pp.do_syscall_64
> 93.22 -12.4 80.83 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 3.40 ? 2% -1.4 1.97 ? 8% perf-profile.children.cycles-pp.worker_thread
> 3.33 ? 2% -1.4 1.96 ? 9% perf-profile.children.cycles-pp.process_one_work
> 1.03 ? 7% -1.0 0.07 ? 37% perf-profile.children.cycles-pp.xlog_cil_force_lsn
> 1.69 ? 2% -0.7 0.96 ? 4% perf-profile.children.cycles-pp.reschedule_interrupt
> 1.66 ? 2% -0.7 0.94 ? 4% perf-profile.children.cycles-pp.scheduler_ipi
> 1.13 ? 2% -0.7 0.47 ? 11% perf-profile.children.cycles-pp.finish_wait
> 0.54 ? 8% -0.4 0.10 ? 38% perf-profile.children.cycles-pp.xlog_cil_push
> 0.49 ? 9% -0.4 0.09 ? 35% perf-profile.children.cycles-pp.xlog_write
> 0.10 ? 8% -0.1 0.04 ? 67% perf-profile.children.cycles-pp.flush_work
> 0.20 ? 5% -0.0 0.16 ? 11% perf-profile.children.cycles-pp.reweight_entity
> 0.06 ? 10% +0.0 0.10 ? 23% perf-profile.children.cycles-pp.brd_lookup_page
> 0.18 ? 5% +0.0 0.23 ? 13% perf-profile.children.cycles-pp.__update_load_avg_se
> 0.02 ?153% +0.1 0.07 ? 16% perf-profile.children.cycles-pp.delay_tsc
> 0.03 ?100% +0.1 0.08 ? 15% perf-profile.children.cycles-pp.find_next_bit
> 0.08 ? 5% +0.1 0.14 ? 14% perf-profile.children.cycles-pp.native_write_msr
> 0.29 ? 4% +0.1 0.36 ? 8% perf-profile.children.cycles-pp.__orc_find
> 0.40 ? 4% +0.1 0.46 ? 7% perf-profile.children.cycles-pp.dequeue_task_fair
> 0.11 ? 11% +0.1 0.18 ? 14% perf-profile.children.cycles-pp.__module_text_address
> 0.12 ? 8% +0.1 0.19 ? 13% perf-profile.children.cycles-pp.is_module_text_address
> 0.04 ? 50% +0.1 0.12 ? 19% perf-profile.children.cycles-pp.kmem_cache_alloc
> 0.00 +0.1 0.08 ? 11% perf-profile.children.cycles-pp.clear_page_erms
> 0.00 +0.1 0.08 ? 28% perf-profile.children.cycles-pp.__indirect_thunk_start
> 0.01 ?200% +0.1 0.10 ? 25% perf-profile.children.cycles-pp.xfs_trans_alloc
> 0.00 +0.1 0.09 ? 18% perf-profile.children.cycles-pp.md_wakeup_thread
> 0.00 +0.1 0.09 ? 26% perf-profile.children.cycles-pp.rebalance_domains
> 0.00 +0.1 0.09 ? 26% perf-profile.children.cycles-pp.get_next_timer_interrupt
> 0.00 +0.1 0.09 ? 20% perf-profile.children.cycles-pp.ktime_get
> 0.18 ? 4% +0.1 0.27 ? 12% perf-profile.children.cycles-pp.idle_cpu
> 0.20 ? 6% +0.1 0.30 ? 9% perf-profile.children.cycles-pp.unwind_get_return_address
> 0.16 ? 10% +0.1 0.25 ? 13% perf-profile.children.cycles-pp.__module_address
> 0.03 ?100% +0.1 0.13 ? 8% perf-profile.children.cycles-pp.brd_insert_page
> 0.06 ? 9% +0.1 0.16 ? 14% perf-profile.children.cycles-pp.task_tick_fair
> 0.08 ? 12% +0.1 0.18 ? 24% perf-profile.children.cycles-pp.bio_alloc_bioset
> 0.03 ? 81% +0.1 0.14 ? 27% perf-profile.children.cycles-pp.generic_make_request_checks
> 0.17 ? 7% +0.1 0.28 ? 11% perf-profile.children.cycles-pp.__kernel_text_address
> 0.11 ? 9% +0.1 0.22 ? 15% perf-profile.children.cycles-pp.wake_up_page_bit
> 0.16 ? 6% +0.1 0.27 ? 10% perf-profile.children.cycles-pp.kernel_text_address
> 0.00 +0.1 0.11 ? 11% perf-profile.children.cycles-pp.get_page_from_freelist
> 0.00 +0.1 0.11 ? 19% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
> 0.00 +0.1 0.11 ? 7% perf-profile.children.cycles-pp.__alloc_pages_nodemask
> 0.08 ? 10% +0.1 0.19 ? 22% perf-profile.children.cycles-pp.xfs_do_writepage
> 0.25 ? 4% +0.1 0.37 ? 10% perf-profile.children.cycles-pp.switch_mm_irqs_off
> 0.00 +0.1 0.12 ? 13% perf-profile.children.cycles-pp.switch_mm
> 0.08 ? 38% +0.1 0.20 ? 19% perf-profile.children.cycles-pp.io_serial_in
> 0.18 ? 5% +0.1 0.31 ? 7% perf-profile.children.cycles-pp.dequeue_entity
> 0.00 +0.1 0.13 ? 26% perf-profile.children.cycles-pp.tick_nohz_next_event
> 0.06 ? 11% +0.1 0.19 ? 19% perf-profile.children.cycles-pp.mempool_alloc
> 0.32 ? 5% +0.1 0.45 ? 6% perf-profile.children.cycles-pp.orc_find
> 0.15 ? 10% +0.1 0.29 ? 19% perf-profile.children.cycles-pp.xfs_destroy_ioend
> 0.15 ? 11% +0.1 0.30 ? 18% perf-profile.children.cycles-pp.call_bio_endio
> 0.08 ? 17% +0.2 0.23 ? 25% perf-profile.children.cycles-pp.xlog_state_done_syncing
> 0.00 +0.2 0.15 ? 22% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
> 0.12 ? 8% +0.2 0.27 ? 23% perf-profile.children.cycles-pp.write_cache_pages
> 0.10 ? 16% +0.2 0.26 ? 16% perf-profile.children.cycles-pp.wait_for_xmitr
> 0.10 ? 19% +0.2 0.25 ? 14% perf-profile.children.cycles-pp.serial8250_console_putchar
> 0.10 ? 17% +0.2 0.26 ? 13% perf-profile.children.cycles-pp.uart_console_write
> 0.10 ? 16% +0.2 0.26 ? 15% perf-profile.children.cycles-pp.serial8250_console_write
> 0.11 ? 15% +0.2 0.27 ? 15% perf-profile.children.cycles-pp.console_unlock
> 0.09 ? 9% +0.2 0.26 ? 12% perf-profile.children.cycles-pp.scheduler_tick
> 0.10 ? 18% +0.2 0.28 ? 15% perf-profile.children.cycles-pp.irq_work_run_list
> 0.10 ? 15% +0.2 0.28 ? 14% perf-profile.children.cycles-pp.xlog_state_do_callback
> 0.09 ? 12% +0.2 0.27 ? 16% perf-profile.children.cycles-pp.irq_work_run
> 0.09 ? 12% +0.2 0.27 ? 16% perf-profile.children.cycles-pp.printk
> 0.09 ? 12% +0.2 0.27 ? 16% perf-profile.children.cycles-pp.vprintk_emit
> 0.09 ? 12% +0.2 0.27 ? 17% perf-profile.children.cycles-pp.irq_work_interrupt
> 0.09 ? 12% +0.2 0.27 ? 17% perf-profile.children.cycles-pp.smp_irq_work_interrupt
> 0.00 +0.2 0.18 ? 16% perf-profile.children.cycles-pp.poll_idle
> 0.30 ? 4% +0.2 0.49 ? 11% perf-profile.children.cycles-pp.update_load_avg
> 1.39 ? 2% +0.2 1.59 ? 6% perf-profile.children.cycles-pp.__save_stack_trace
> 1.43 +0.2 1.65 ? 6% perf-profile.children.cycles-pp.save_stack_trace_tsk
> 0.14 ? 13% +0.2 0.36 ? 13% perf-profile.children.cycles-pp.update_process_times
> 0.00 +0.2 0.23 ? 22% perf-profile.children.cycles-pp.find_busiest_group
> 0.22 ? 6% +0.2 0.45 ? 18% perf-profile.children.cycles-pp.brd_do_bvec
> 0.14 ? 13% +0.2 0.38 ? 14% perf-profile.children.cycles-pp.tick_sched_handle
> 0.10 ? 8% +0.2 0.34 ? 26% perf-profile.children.cycles-pp.xfs_log_commit_cil
> 0.07 ? 10% +0.3 0.33 ? 23% perf-profile.children.cycles-pp.io_schedule
> 0.03 ? 83% +0.3 0.29 ? 27% perf-profile.children.cycles-pp.__softirqentry_text_start
> 0.11 ? 5% +0.3 0.36 ? 25% perf-profile.children.cycles-pp.__xfs_trans_commit
> 0.06 ? 36% +0.3 0.31 ? 26% perf-profile.children.cycles-pp.irq_exit
> 0.08 ? 9% +0.3 0.35 ? 23% perf-profile.children.cycles-pp.wait_on_page_bit_common
> 0.15 ? 12% +0.3 0.42 ? 14% perf-profile.children.cycles-pp.tick_sched_timer
> 0.10 ? 11% +0.3 0.39 ? 22% perf-profile.children.cycles-pp.__filemap_fdatawait_range
> 0.06 ? 12% +0.3 0.37 ? 9% perf-profile.children.cycles-pp.schedule_idle
> 0.02 ?153% +0.3 0.34 ? 17% perf-profile.children.cycles-pp.menu_select
> 0.17 ? 5% +0.3 0.49 ? 22% perf-profile.children.cycles-pp.xfs_vn_update_time
> 0.19 ? 12% +0.3 0.51 ? 18% perf-profile.children.cycles-pp.xlog_iodone
> 0.18 ? 5% +0.3 0.51 ? 22% perf-profile.children.cycles-pp.file_update_time
> 0.18 ? 5% +0.3 0.51 ? 21% perf-profile.children.cycles-pp.xfs_file_aio_write_checks
> 0.21 ? 11% +0.4 0.60 ? 15% perf-profile.children.cycles-pp.__hrtimer_run_queues
> 0.26 ? 6% +0.4 0.69 ? 16% perf-profile.children.cycles-pp.pick_next_task_fair
> 1.20 ? 2% +0.4 1.64 ? 10% perf-profile.children.cycles-pp.schedule
> 0.28 ? 5% +0.4 0.72 ? 21% perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
> 0.00 +0.4 0.44 ? 22% perf-profile.children.cycles-pp.load_balance
> 0.25 ? 8% +0.5 0.74 ? 15% perf-profile.children.cycles-pp.hrtimer_interrupt
> 1.30 ? 2% +0.7 2.00 ? 9% perf-profile.children.cycles-pp.__schedule
> 0.31 ? 8% +0.8 1.09 ? 16% perf-profile.children.cycles-pp.smp_apic_timer_interrupt
> 0.31 ? 8% +0.8 1.09 ? 16% perf-profile.children.cycles-pp.apic_timer_interrupt
> 3.92 ? 2% +0.9 4.79 ? 6% perf-profile.children.cycles-pp.ret_from_fork
> 3.92 ? 2% +0.9 4.79 ? 6% perf-profile.children.cycles-pp.kthread
> 0.69 ? 2% +0.9 1.64 ? 23% perf-profile.children.cycles-pp.xlog_wait
> 0.08 ? 13% +1.2 1.27 ? 17% perf-profile.children.cycles-pp.submit_flushes
> 0.16 ? 9% +1.6 1.74 ? 4% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> 0.17 ? 9% +2.0 2.16 ? 10% perf-profile.children.cycles-pp.raid_end_bio_io
> 0.21 ? 6% +2.0 2.25 ? 10% perf-profile.children.cycles-pp.raid1_end_write_request
> 2.24 ? 4% +2.2 4.44 ? 15% perf-profile.children.cycles-pp.xfs_log_force_lsn
> 0.46 ? 6% +2.3 2.73 ? 7% perf-profile.children.cycles-pp.brd_make_request
> 0.51 ? 6% +2.3 2.81 ? 7% perf-profile.children.cycles-pp.md_thread
> 0.49 ? 6% +2.3 2.81 ? 7% perf-profile.children.cycles-pp.raid1d
> 0.49 ? 6% +2.3 2.81 ? 7% perf-profile.children.cycles-pp.flush_pending_writes
> 0.49 ? 6% +2.3 2.81 ? 7% perf-profile.children.cycles-pp.flush_bio_list
> 1.80 ? 3% +5.6 7.44 ? 27% perf-profile.children.cycles-pp._raw_spin_lock
> 2.12 ? 4% +5.8 7.97 ? 20% perf-profile.children.cycles-pp.remove_wait_queue
> 1.33 ? 4% +8.8 10.12 ? 8% perf-profile.children.cycles-pp.intel_idle
> 1.37 ? 4% +9.3 10.71 ? 8% perf-profile.children.cycles-pp.cpuidle_enter_state
> 1.59 ? 4% +10.4 11.98 ? 9% perf-profile.children.cycles-pp.start_secondary
> 1.63 ? 4% +10.8 12.47 ? 8% perf-profile.children.cycles-pp.secondary_startup_64
> 1.63 ? 4% +10.8 12.47 ? 8% perf-profile.children.cycles-pp.cpu_startup_entry
> 1.63 ? 4% +10.9 12.49 ? 8% perf-profile.children.cycles-pp.do_idle
> 3.48 +12.2 15.72 ? 23% perf-profile.children.cycles-pp.__xfs_log_force_lsn
> 1.36 ? 12% +57.8 59.12 ? 10% perf-profile.children.cycles-pp.prepare_to_wait_event
> 0.43 ? 38% +62.4 62.82 ? 8% perf-profile.children.cycles-pp.xfs_submit_ioend
> 0.55 ? 29% +62.5 63.10 ? 8% perf-profile.children.cycles-pp.xfs_vm_writepages
> 0.55 ? 30% +62.5 63.10 ? 8% perf-profile.children.cycles-pp.do_writepages
> 0.55 ? 29% +62.6 63.11 ? 8% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
> 0.66 ? 25% +62.9 63.52 ? 7% perf-profile.children.cycles-pp.file_write_and_wait_range
> 0.39 ? 43% +63.6 64.02 ? 8% perf-profile.children.cycles-pp.raid1_write_request
> 5.43 ? 3% +64.2 69.64 ? 5% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 89.86 -13.5 76.31 ? 2% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
> 0.14 ? 8% -0.0 0.09 ? 19% perf-profile.self.cycles-pp.md_flush_request
> 0.10 ? 12% -0.0 0.07 ? 21% perf-profile.self.cycles-pp.account_entity_enqueue
> 0.06 ? 7% +0.0 0.08 ? 12% perf-profile.self.cycles-pp.pick_next_task_fair
> 0.05 ? 12% +0.0 0.08 ? 18% perf-profile.self.cycles-pp.___perf_sw_event
> 0.15 ? 6% +0.0 0.18 ? 9% perf-profile.self.cycles-pp.__update_load_avg_se
> 0.17 ? 4% +0.0 0.22 ? 10% perf-profile.self.cycles-pp.__schedule
> 0.10 ? 11% +0.1 0.15 ? 11% perf-profile.self.cycles-pp._raw_spin_lock
> 0.02 ?153% +0.1 0.07 ? 16% perf-profile.self.cycles-pp.delay_tsc
> 0.02 ?152% +0.1 0.07 ? 23% perf-profile.self.cycles-pp.set_next_entity
> 0.03 ?100% +0.1 0.08 ? 15% perf-profile.self.cycles-pp.find_next_bit
> 0.08 ? 5% +0.1 0.14 ? 14% perf-profile.self.cycles-pp.native_write_msr
> 0.01 ?200% +0.1 0.07 ? 23% perf-profile.self.cycles-pp.kmem_cache_alloc
> 0.29 ? 4% +0.1 0.36 ? 8% perf-profile.self.cycles-pp.__orc_find
> 0.14 ? 7% +0.1 0.21 ? 12% perf-profile.self.cycles-pp.switch_mm_irqs_off
> 0.00 +0.1 0.08 ? 11% perf-profile.self.cycles-pp.clear_page_erms
> 0.00 +0.1 0.08 ? 28% perf-profile.self.cycles-pp.__indirect_thunk_start
> 0.00 +0.1 0.08 ? 20% perf-profile.self.cycles-pp.md_wakeup_thread
> 0.34 ? 6% +0.1 0.43 ? 12% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.18 ? 4% +0.1 0.27 ? 12% perf-profile.self.cycles-pp.idle_cpu
> 0.16 ? 10% +0.1 0.25 ? 13% perf-profile.self.cycles-pp.__module_address
> 0.06 ? 11% +0.1 0.17 ? 14% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> 0.08 ? 38% +0.1 0.20 ? 19% perf-profile.self.cycles-pp.io_serial_in
> 0.18 ? 5% +0.1 0.32 ? 15% perf-profile.self.cycles-pp.update_load_avg
> 0.00 +0.1 0.15 ? 17% perf-profile.self.cycles-pp.poll_idle
> 0.00 +0.2 0.15 ? 16% perf-profile.self.cycles-pp.menu_select
> 0.00 +0.2 0.18 ? 24% perf-profile.self.cycles-pp.find_busiest_group
> 0.02 ?152% +0.3 0.35 ? 21% perf-profile.self.cycles-pp.raid1_write_request
> 1.33 ? 4% +8.8 10.12 ? 8% perf-profile.self.cycles-pp.intel_idle
>
>
>
> aim7.jobs-per-min
>
> 1700 +-+------------------------------------------------------------------+
> |+ ++++++ :+ ++++ ++++ +++ ++++++ + + ++++++++++++ ++ ++|
> 1600 +-+ + +++ + +++++ ++.++ + ++ ++ + ++ |
> | |
> | |
> 1500 +-+ |
> | |
> 1400 +-+ |
> | |
> 1300 +-+ |
> | |
> O OO OO O O O |
> 1200 +OO OOOOOOOOO OO OOOOOOOOOOOOOO OOOOOOOOO O |
> | |
> 1100 +-+------------------------------------------------------------------+
>
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Xiaolong

2018-07-16 08:03:23

by Xiao Ni

[permalink] [raw]

Subject: Re: [LKP] [lkp-robot] [MD] 5a409b4f56: aim7.jobs-per-min -27.5% regression

Hi Aaron

I have no update for this yet. I'll have a look this week and give response later.

Regards
Xiao

----- Original Message -----
> From: "Aaron Lu" <[email protected]>
> To: "Xiao Ni" <[email protected]>
> Cc: "kernel test robot" <[email protected]>, "Stephen Rothwell" <[email protected]>, [email protected], "LKML"
> <[email protected]>, "Shaohua Li" <[email protected]>, "Ming Lei" <[email protected]>
> Sent: Monday, July 16, 2018 3:54:30 PM
> Subject: Re: [LKP] [lkp-robot] [MD] 5a409b4f56: aim7.jobs-per-min -27.5% regression
>
> Ping...
> Any update on this?
> Feel free to ask me for any additional data if you need.
>
> Thanks,
> Aaron
>
> On Mon, Jun 04, 2018 at 02:42:03PM +0800, kernel test robot wrote:
> >
> > Greeting,
> >
> > FYI, we noticed a -27.5% regression of aim7.jobs-per-min due to commit:
> >
> >
> > commit: 5a409b4f56d50b212334f338cb8465d65550cd85 ("MD: fix lock contention
> > for flush bios")
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
> >
> > in testcase: aim7
> > on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with
> > 384G memory
> > with following parameters:
> >
> > disk: 4BRD_12G
> > md: RAID1
> > fs: xfs
> > test: sync_disk_rw
> > load: 600
> > cpufreq_governor: performance
> >
> > test-description: AIM7 is a traditional UNIX system level benchmark suite
> > which is used to test and measure the performance of multiuser system.
> > test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
> >
> >
> > Details are as below:
> > -------------------------------------------------------------------------------------------------->
> >
> > =========================================================================================
> > compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
> > gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.2/600/RAID1/debian-x86_64-2016-08-31.cgz/lkp-ivb-ep01/sync_disk_rw/aim7
> >
> > commit:
> > 448ec638c6 ("md/raid5: Assigning NULL to sh->batch_head before testing
> > bit R5_Overlap of a stripe")
> > 5a409b4f56 ("MD: fix lock contention for flush bios")
> >
> > 448ec638c6bcf369 5a409b4f56d50b212334f338cb
> > ---------------- --------------------------
> > %stddev %change %stddev
> > \ | \
> > 1640 -27.5% 1189 aim7.jobs-per-min
> > 2194 +37.9% 3026 aim7.time.elapsed_time
> > 2194 +37.9% 3026 aim7.time.elapsed_time.max
> > 50990311 -95.8% 2148266
> > aim7.time.involuntary_context_switches
> > 107965 ± 4% -26.4% 79516 ± 2% aim7.time.minor_page_faults
> > 49.14 +82.5% 89.66 ± 2% aim7.time.user_time
> > 7.123e+08 -35.7% 4.582e+08
> > aim7.time.voluntary_context_switches
> > 672282 +36.8% 919615
> > interrupts.CAL:Function_call_interrupts
> > 16631387 ± 2% -39.9% 9993075 ± 7% softirqs.RCU
> > 9708009 +186.1% 27778773 softirqs.SCHED
> > 33436649 +45.5% 48644912 softirqs.TIMER
> > 4.16 -2.1 2.01 mpstat.cpu.idle%
> > 0.24 ± 2% +27.7 27.91 mpstat.cpu.iowait%
> > 95.51 -25.6 69.94 mpstat.cpu.sys%
> > 0.09 +0.0 0.13 mpstat.cpu.usr%
> > 6051756 ± 3% +59.0% 9623085
> > numa-numastat.node0.local_node
> > 6055311 ± 3% +59.0% 9626996 numa-numastat.node0.numa_hit
> > 6481209 ± 3% +48.4% 9616310
> > numa-numastat.node1.local_node
> > 6485866 ± 3% +48.3% 9620756 numa-numastat.node1.numa_hit
> > 61404 -27.7% 44424 vmstat.io.bo
> > 2.60 ± 18% +11519.2% 302.10 vmstat.procs.b
> > 304.10 -84.9% 45.80 ± 2% vmstat.procs.r
> > 400477 -43.5% 226094 vmstat.system.cs
> > 166461 -49.9% 83332 vmstat.system.in
> > 78397 +27.0% 99567 meminfo.Dirty
> > 14427 +18.4% 17082 meminfo.Inactive(anon)
> > 1963 ± 5% +5.4% 2068 ± 4% meminfo.Mlocked
> > 101143 +991.0% 1103488 meminfo.SUnreclaim
> > 53684 ± 4% -18.1% 43946 ± 3% meminfo.Shmem
> > 175580 +571.4% 1178829 meminfo.Slab
> > 39406 +26.2% 49717 numa-meminfo.node0.Dirty
> > 1767204 ± 10% +37.2% 2425487 ± 2% numa-meminfo.node0.MemUsed
> > 51634 ± 18% +979.3% 557316 numa-meminfo.node0.SUnreclaim
> > 92259 ± 13% +551.7% 601288 numa-meminfo.node0.Slab
> > 38969 +28.0% 49863 numa-meminfo.node1.Dirty
> > 1895204 ± 10% +24.7% 2363037 ± 3% numa-meminfo.node1.MemUsed
> > 49512 ± 19% +1003.1% 546165 numa-meminfo.node1.SUnreclaim
> > 83323 ± 14% +593.1% 577534 numa-meminfo.node1.Slab
> > 2.524e+09 +894.5% 2.51e+10 cpuidle.C1.time
> > 50620790 +316.5% 2.109e+08 cpuidle.C1.usage
> > 3.965e+08 +1871.1% 7.815e+09 cpuidle.C1E.time
> > 5987788 +186.1% 17129412 cpuidle.C1E.usage
> > 2.506e+08 +97.5% 4.948e+08 ± 2% cpuidle.C3.time
> > 2923498 -55.7% 1295033 cpuidle.C3.usage
> > 5.327e+08 +179.9% 1.491e+09 cpuidle.C6.time
> > 779874 ± 2% +229.3% 2567769 cpuidle.C6.usage
> > 6191357 +3333.6% 2.126e+08 cpuidle.POLL.time
> > 204095 +1982.1% 4249504 cpuidle.POLL.usage
> > 9850 +26.3% 12444 numa-vmstat.node0.nr_dirty
> > 12908 ± 18% +979.3% 139321
> > numa-vmstat.node0.nr_slab_unreclaimable
> > 8876 +29.6% 11505
> > numa-vmstat.node0.nr_zone_write_pending
> > 3486319 ± 4% +55.1% 5407021 numa-vmstat.node0.numa_hit
> > 3482713 ± 4% +55.1% 5403066 numa-vmstat.node0.numa_local
> > 9743 +28.1% 12479 numa-vmstat.node1.nr_dirty
> > 12377 ± 19% +1003.1% 136532
> > numa-vmstat.node1.nr_slab_unreclaimable
> > 9287 +30.0% 12074
> > numa-vmstat.node1.nr_zone_write_pending
> > 3678995 ± 4% +44.8% 5326772 numa-vmstat.node1.numa_hit
> > 3497785 ± 4% +47.1% 5145705 numa-vmstat.node1.numa_local
> > 252.70 +100.2% 505.90
> > slabinfo.biovec-max.active_objs
> > 282.70 +99.1% 562.90 slabinfo.biovec-max.num_objs
> > 2978 ± 17% +52.5% 4543 ± 14%
> > slabinfo.dmaengine-unmap-16.active_objs
> > 2978 ± 17% +52.5% 4543 ± 14%
> > slabinfo.dmaengine-unmap-16.num_objs
> > 2078 +147.9% 5153 ± 11%
> > slabinfo.ip6_dst_cache.active_objs
> > 2078 +148.1% 5157 ± 11%
> > slabinfo.ip6_dst_cache.num_objs
> > 5538 ± 2% +26.2% 6990 ± 3%
> > slabinfo.kmalloc-1024.active_objs
> > 5586 ± 3% +27.1% 7097 ± 3%
> > slabinfo.kmalloc-1024.num_objs
> > 6878 +47.6% 10151 ± 5%
> > slabinfo.kmalloc-192.active_objs
> > 6889 +47.5% 10160 ± 5% slabinfo.kmalloc-192.num_objs
> > 9843 ± 5% +1.6e+05% 16002876
> > slabinfo.kmalloc-64.active_objs
> > 161.90 ± 4% +1.5e+05% 250044
> > slabinfo.kmalloc-64.active_slabs
> > 10386 ± 4% +1.5e+05% 16002877 slabinfo.kmalloc-64.num_objs
> > 161.90 ± 4% +1.5e+05% 250044 slabinfo.kmalloc-64.num_slabs
> > 432.80 ± 12% +45.2% 628.50 ± 6%
> > slabinfo.nfs_read_data.active_objs
> > 432.80 ± 12% +45.2% 628.50 ± 6%
> > slabinfo.nfs_read_data.num_objs
> > 3956 -23.1% 3041
> > slabinfo.pool_workqueue.active_objs
> > 4098 -19.8% 3286
> > slabinfo.pool_workqueue.num_objs
> > 360.50 ± 15% +56.6% 564.70 ± 11%
> > slabinfo.secpath_cache.active_objs
> > 360.50 ± 15% +56.6% 564.70 ± 11%
> > slabinfo.secpath_cache.num_objs
> > 35373 ± 2% -8.3% 32432 proc-vmstat.nr_active_anon
> > 19595 +27.1% 24914 proc-vmstat.nr_dirty
> > 3607 +18.4% 4270 proc-vmstat.nr_inactive_anon
> > 490.30 ± 5% +5.4% 516.90 ± 4% proc-vmstat.nr_mlock
> > 13421 ± 4% -18.1% 10986 ± 3% proc-vmstat.nr_shmem
> > 18608 +1.2% 18834
> > proc-vmstat.nr_slab_reclaimable
> > 25286 +991.0% 275882
> > proc-vmstat.nr_slab_unreclaimable
> > 35405 ± 2% -8.3% 32465
> > proc-vmstat.nr_zone_active_anon
> > 3607 +18.4% 4270
> > proc-vmstat.nr_zone_inactive_anon
> > 18161 +29.8% 23572
> > proc-vmstat.nr_zone_write_pending
> > 76941 ± 5% -36.8% 48622 ± 4% proc-vmstat.numa_hint_faults
> > 33878 ± 7% -35.5% 21836 ± 5%
> > proc-vmstat.numa_hint_faults_local
> > 12568956 +53.3% 19272377 proc-vmstat.numa_hit
> > 12560739 +53.4% 19264015 proc-vmstat.numa_local
> > 17938 ± 3% -33.5% 11935 ± 2%
> > proc-vmstat.numa_pages_migrated
> > 78296 ± 5% -36.0% 50085 ± 4% proc-vmstat.numa_pte_updates
> > 8848 ± 6% -38.2% 5466 ± 6% proc-vmstat.pgactivate
> > 8874568 ± 8% +368.7% 41590920 proc-vmstat.pgalloc_normal
> > 5435965 +39.2% 7564148 proc-vmstat.pgfault
> > 12863707 +255.1% 45683570 proc-vmstat.pgfree
> > 17938 ± 3% -33.5% 11935 ± 2% proc-vmstat.pgmigrate_success
> > 1.379e+13 -40.8% 8.17e+12 perf-stat.branch-instructions
> > 0.30 +0.1 0.42 perf-stat.branch-miss-rate%
> > 4.2e+10 -17.6% 3.462e+10 perf-stat.branch-misses
> > 15.99 +3.8 19.74 perf-stat.cache-miss-rate%
> > 3.779e+10 -21.6% 2.963e+10 perf-stat.cache-misses
> > 2.364e+11 -36.5% 1.501e+11 perf-stat.cache-references
> > 8.795e+08 -22.2% 6.84e+08 perf-stat.context-switches
> > 4.44 -7.2% 4.12 perf-stat.cpi
> > 2.508e+14 -44.5% 1.393e+14 perf-stat.cpu-cycles
> > 36915392 +60.4% 59211221 perf-stat.cpu-migrations
> > 0.29 ± 2% +0.0 0.34 ± 4%
> > perf-stat.dTLB-load-miss-rate%
> > 4.14e+10 -30.2% 2.89e+10 ± 4% perf-stat.dTLB-load-misses
> > 1.417e+13 -40.1% 8.491e+12 perf-stat.dTLB-loads
> > 0.20 ± 4% -0.0 0.18 ± 5%
> > perf-stat.dTLB-store-miss-rate%
> > 3.072e+09 ± 4% -28.0% 2.21e+09 ± 4% perf-stat.dTLB-store-misses
> > 1.535e+12 -20.2% 1.225e+12 perf-stat.dTLB-stores
> > 90.73 -11.7 79.07
> > perf-stat.iTLB-load-miss-rate%
> > 8.291e+09 -6.6% 7.743e+09 perf-stat.iTLB-load-misses
> > 8.473e+08 +141.8% 2.049e+09 ± 3% perf-stat.iTLB-loads
> > 5.646e+13 -40.2% 3.378e+13 perf-stat.instructions
> > 6810 -35.9% 4362
> > perf-stat.instructions-per-iTLB-miss
> > 0.23 +7.8% 0.24 perf-stat.ipc
> > 5326672 +39.2% 7413706 perf-stat.minor-faults
> > 1.873e+10 -29.9% 1.312e+10 perf-stat.node-load-misses
> > 2.093e+10 -29.2% 1.481e+10 perf-stat.node-loads
> > 39.38 -0.7 38.72
> > perf-stat.node-store-miss-rate%
> > 1.087e+10 -16.6% 9.069e+09 perf-stat.node-store-misses
> > 1.673e+10 -14.2% 1.435e+10 perf-stat.node-stores
> > 5326695 +39.2% 7413708 perf-stat.page-faults
> > 1875095 ± 7% -54.8% 846645 ± 16%
> > sched_debug.cfs_rq:/.MIN_vruntime.avg
> > 32868920 ± 6% -35.7% 21150379 ± 14%
> > sched_debug.cfs_rq:/.MIN_vruntime.max
> > 7267340 ± 5% -44.7% 4015798 ± 14%
> > sched_debug.cfs_rq:/.MIN_vruntime.stddev
> > 4278 ± 7% -54.7% 1939 ± 11%
> > sched_debug.cfs_rq:/.exec_clock.stddev
> > 245.48 ± 2% +65.3% 405.75 ± 7%
> > sched_debug.cfs_rq:/.load_avg.avg
> > 2692 ± 6% +126.0% 6087 ± 7%
> > sched_debug.cfs_rq:/.load_avg.max
> > 33.09 -73.0% 8.94 ± 7%
> > sched_debug.cfs_rq:/.load_avg.min
> > 507.40 ± 4% +128.0% 1156 ± 7%
> > sched_debug.cfs_rq:/.load_avg.stddev
> > 1875095 ± 7% -54.8% 846645 ± 16%
> > sched_debug.cfs_rq:/.max_vruntime.avg
> > 32868921 ± 6% -35.7% 21150379 ± 14%
> > sched_debug.cfs_rq:/.max_vruntime.max
> > 7267341 ± 5% -44.7% 4015798 ± 14%
> > sched_debug.cfs_rq:/.max_vruntime.stddev
> > 35887197 -13.2% 31149130
> > sched_debug.cfs_rq:/.min_vruntime.avg
> > 37385506 -14.3% 32043914
> > sched_debug.cfs_rq:/.min_vruntime.max
> > 34416296 -12.3% 30183927
> > sched_debug.cfs_rq:/.min_vruntime.min
> > 1228844 ± 8% -52.6% 582759 ± 4%
> > sched_debug.cfs_rq:/.min_vruntime.stddev
> > 0.83 -28.1% 0.60 ± 6%
> > sched_debug.cfs_rq:/.nr_running.avg
> > 2.07 ± 3% -24.6% 1.56 ± 8%
> > sched_debug.cfs_rq:/.nr_running.max
> > 20.52 ± 4% -48.8% 10.52 ± 3%
> > sched_debug.cfs_rq:/.nr_spread_over.avg
> > 35.96 ± 5% -42.2% 20.77 ± 9%
> > sched_debug.cfs_rq:/.nr_spread_over.max
> > 8.97 ± 11% -44.5% 4.98 ± 8%
> > sched_debug.cfs_rq:/.nr_spread_over.min
> > 6.40 ± 12% -45.5% 3.49 ± 7%
> > sched_debug.cfs_rq:/.nr_spread_over.stddev
> > 21.78 ± 7% +143.3% 53.00 ± 9%
> > sched_debug.cfs_rq:/.runnable_load_avg.avg
> > 328.86 ± 18% +303.4% 1326 ± 14%
> > sched_debug.cfs_rq:/.runnable_load_avg.max
> > 55.97 ± 17% +286.0% 216.07 ± 13%
> > sched_debug.cfs_rq:/.runnable_load_avg.stddev
> > 0.10 ± 29% -82.4% 0.02 ± 50%
> > sched_debug.cfs_rq:/.spread.avg
> > 3.43 ± 25% -79.9% 0.69 ± 50%
> > sched_debug.cfs_rq:/.spread.max
> > 0.56 ± 26% -80.7% 0.11 ± 50%
> > sched_debug.cfs_rq:/.spread.stddev
> > 1228822 ± 8% -52.6% 582732 ± 4%
> > sched_debug.cfs_rq:/.spread0.stddev
> > 992.30 -24.9% 745.56 ± 2%
> > sched_debug.cfs_rq:/.util_avg.avg
> > 1485 -18.1% 1217 ± 2%
> > sched_debug.cfs_rq:/.util_avg.max
> > 515.45 ± 2% -25.2% 385.73 ± 6%
> > sched_debug.cfs_rq:/.util_avg.min
> > 201.54 -14.9% 171.52 ± 3%
> > sched_debug.cfs_rq:/.util_avg.stddev
> > 248.73 ± 6% -38.1% 154.02 ± 8%
> > sched_debug.cfs_rq:/.util_est_enqueued.avg
> > 222.78 ± 3% -15.8% 187.58 ± 2%
> > sched_debug.cfs_rq:/.util_est_enqueued.stddev
> > 77097 ± 4% +278.4% 291767 ± 11% sched_debug.cpu.avg_idle.avg
> > 181319 ± 6% +298.7% 722862 ± 3% sched_debug.cpu.avg_idle.max
> > 19338 +392.3% 95203 ± 17% sched_debug.cpu.avg_idle.min
> > 34877 ± 6% +303.5% 140732 ± 6%
> > sched_debug.cpu.avg_idle.stddev
> > 1107408 +37.6% 1523823 sched_debug.cpu.clock.avg
> > 1107427 +37.6% 1523834 sched_debug.cpu.clock.max
> > 1107385 +37.6% 1523811 sched_debug.cpu.clock.min
> > 13.10 ± 9% -48.1% 6.80 ± 8% sched_debug.cpu.clock.stddev
> > 1107408 +37.6% 1523823
> > sched_debug.cpu.clock_task.avg
> > 1107427 +37.6% 1523834
> > sched_debug.cpu.clock_task.max
> > 1107385 +37.6% 1523811
> > sched_debug.cpu.clock_task.min
> > 13.10 ± 9% -48.1% 6.80 ± 8%
> > sched_debug.cpu.clock_task.stddev
> > 30.36 ± 7% +107.7% 63.06 ± 12%
> > sched_debug.cpu.cpu_load[0].avg
> > 381.48 ± 18% +269.8% 1410 ± 18%
> > sched_debug.cpu.cpu_load[0].max
> > 63.92 ± 18% +262.2% 231.50 ± 17%
> > sched_debug.cpu.cpu_load[0].stddev
> > 31.34 ± 5% +118.4% 68.44 ± 9%
> > sched_debug.cpu.cpu_load[1].avg
> > 323.62 ± 17% +349.5% 1454 ± 14%
> > sched_debug.cpu.cpu_load[1].max
> > 53.23 ± 16% +350.3% 239.71 ± 13%
> > sched_debug.cpu.cpu_load[1].stddev
> > 32.15 ± 3% +129.4% 73.74 ± 6%
> > sched_debug.cpu.cpu_load[2].avg
> > 285.20 ± 14% +420.8% 1485 ± 9%
> > sched_debug.cpu.cpu_load[2].max
> > 46.66 ± 12% +430.0% 247.32 ± 8%
> > sched_debug.cpu.cpu_load[2].stddev
> > 33.02 ± 2% +133.2% 77.00 ± 3%
> > sched_debug.cpu.cpu_load[3].avg
> > 252.16 ± 10% +481.2% 1465 ± 7%
> > sched_debug.cpu.cpu_load[3].max
> > 40.74 ± 8% +503.2% 245.72 ± 6%
> > sched_debug.cpu.cpu_load[3].stddev
> > 33.86 +131.5% 78.38 ± 2%
> > sched_debug.cpu.cpu_load[4].avg
> > 219.81 ± 8% +522.6% 1368 ± 5%
> > sched_debug.cpu.cpu_load[4].max
> > 35.45 ± 7% +554.2% 231.90 ± 4%
> > sched_debug.cpu.cpu_load[4].stddev
> > 2600 ± 4% -30.5% 1807 ± 4% sched_debug.cpu.curr->pid.avg
> > 25309 ± 4% -19.5% 20367 ± 4% sched_debug.cpu.curr->pid.max
> > 4534 ± 7% -21.2% 3573 ± 5%
> > sched_debug.cpu.curr->pid.stddev
> > 0.00 ± 2% -27.6% 0.00 ± 6%
> > sched_debug.cpu.next_balance.stddev
> > 1083917 +38.6% 1502777
> > sched_debug.cpu.nr_load_updates.avg
> > 1088142 +38.6% 1508302
> > sched_debug.cpu.nr_load_updates.max
> > 1082048 +38.7% 1501073
> > sched_debug.cpu.nr_load_updates.min
> > 3.53 ± 6% -73.0% 0.95 ± 6%
> > sched_debug.cpu.nr_running.avg
> > 11.54 ± 3% -62.1% 4.37 ± 10%
> > sched_debug.cpu.nr_running.max
> > 3.10 ± 3% -66.8% 1.03 ± 9%
> > sched_debug.cpu.nr_running.stddev
> > 10764176 -22.4% 8355047
> > sched_debug.cpu.nr_switches.avg
> > 10976436 -22.2% 8545010
> > sched_debug.cpu.nr_switches.max
> > 10547712 -22.8% 8143037
> > sched_debug.cpu.nr_switches.min
> > 148628 ± 3% -22.7% 114880 ± 7%
> > sched_debug.cpu.nr_switches.stddev
> > 11.13 ± 2% +24.5% 13.85
> > sched_debug.cpu.nr_uninterruptible.avg
> > 6420 ± 8% -48.7% 3296 ± 11%
> > sched_debug.cpu.nr_uninterruptible.max
> > -5500 -37.2% -3455
> > sched_debug.cpu.nr_uninterruptible.min
> > 3784 ± 6% -47.2% 1997 ± 4%
> > sched_debug.cpu.nr_uninterruptible.stddev
> > 10812670 -22.7% 8356821
> > sched_debug.cpu.sched_count.avg
> > 11020646 -22.5% 8546277
> > sched_debug.cpu.sched_count.max
> > 10601390 -23.2% 8144743
> > sched_debug.cpu.sched_count.min
> > 144529 ± 3% -20.9% 114359 ± 7%
> > sched_debug.cpu.sched_count.stddev
> > 706116 +259.0% 2534721
> > sched_debug.cpu.sched_goidle.avg
> > 771307 +232.4% 2564059
> > sched_debug.cpu.sched_goidle.max
> > 644658 +286.9% 2494236
> > sched_debug.cpu.sched_goidle.min
> > 49847 ± 6% -67.9% 15979 ± 7%
> > sched_debug.cpu.sched_goidle.stddev
> > 9618827 -39.9% 5780369
> > sched_debug.cpu.ttwu_count.avg
> > 8990451 -61.7% 3441265 ± 4%
> > sched_debug.cpu.ttwu_count.min
> > 418563 ± 25% +244.2% 1440565 ± 7%
> > sched_debug.cpu.ttwu_count.stddev
> > 640964 -93.7% 40366 ± 2%
> > sched_debug.cpu.ttwu_local.avg
> > 679527 -92.1% 53476 ± 4%
> > sched_debug.cpu.ttwu_local.max
> > 601661 -94.9% 30636 ± 3%
> > sched_debug.cpu.ttwu_local.min
> > 24242 ± 21% -77.7% 5405 ± 9%
> > sched_debug.cpu.ttwu_local.stddev
> > 1107383 +37.6% 1523810 sched_debug.cpu_clk
> > 1107383 +37.6% 1523810 sched_debug.ktime
> > 0.00 -49.4% 0.00 ± 65%
> > sched_debug.rt_rq:/.rt_nr_migratory.avg
> > 0.03 -49.4% 0.01 ± 65%
> > sched_debug.rt_rq:/.rt_nr_migratory.max
> > 0.00 -49.4% 0.00 ± 65%
> > sched_debug.rt_rq:/.rt_nr_migratory.stddev
> > 0.00 -49.4% 0.00 ± 65%
> > sched_debug.rt_rq:/.rt_nr_running.avg
> > 0.03 -49.4% 0.01 ± 65%
> > sched_debug.rt_rq:/.rt_nr_running.max
> > 0.00 -49.4% 0.00 ± 65%
> > sched_debug.rt_rq:/.rt_nr_running.stddev
> > 0.01 ± 8% +79.9% 0.01 ± 23%
> > sched_debug.rt_rq:/.rt_time.avg
> > 1107805 +37.6% 1524235 sched_debug.sched_clk
> > 87.59 -87.6 0.00
> > perf-profile.calltrace.cycles-pp.md_flush_request.raid1_make_request.md_handle_request.md_make_request.generic_make_request
> > 87.57 -87.6 0.00
> > perf-profile.calltrace.cycles-pp.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> > 87.59 -87.5 0.05 ±299%
> > perf-profile.calltrace.cycles-pp.blkdev_issue_flush.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
> > 87.51 -87.5 0.00
> > perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync
> > 87.51 -87.5 0.00
> > perf-profile.calltrace.cycles-pp.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_write_iter
> > 87.50 -87.5 0.00
> > perf-profile.calltrace.cycles-pp.md_make_request.generic_make_request.submit_bio.submit_bio_wait.blkdev_issue_flush
> > 87.50 -87.5 0.00
> > perf-profile.calltrace.cycles-pp.md_handle_request.md_make_request.generic_make_request.submit_bio.submit_bio_wait
> > 82.37 -82.4 0.00
> > perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.md_flush_request.raid1_make_request.md_handle_request.md_make_request
> > 82.23 -82.2 0.00
> > perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.md_flush_request.raid1_make_request.md_handle_request
> > 87.79 -25.0 62.75 ± 8%
> > perf-profile.calltrace.cycles-pp.raid1_make_request.md_handle_request.md_make_request.generic_make_request.submit_bio
> > 92.78 -13.0 79.76
> > perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write.ksys_write
> > 93.08 -12.6 80.49
> > perf-profile.calltrace.cycles-pp.xfs_file_write_iter.__vfs_write.vfs_write.ksys_write.do_syscall_64
> > 93.08 -12.6 80.50
> > perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> > 93.11 -12.6 80.56
> > perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> > 93.11 -12.6 80.56
> > perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> > 93.14 -12.5 80.64
> > perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> > 93.15 -12.5 80.65
> > perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> > 3.40 ± 2% -1.4 1.97 ± 8%
> > perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
> > 3.33 ± 2% -1.4 1.96 ± 9%
> > perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
> > 1.12 ± 2% -0.7 0.42 ± 68%
> > perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
> > 1.16 ± 2% -0.6 0.60 ± 17%
> > perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate
> > 0.00 +0.6 0.59 ± 15%
> > perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.raid1_write_request.raid1_make_request.md_handle_request
> > 0.00 +0.6 0.64 ± 15%
> > perf-profile.calltrace.cycles-pp.__wake_up_common_lock.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
> > 0.00 +0.7 0.65 ± 10%
> > perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle
> > 0.00 +0.7 0.68 ± 10%
> > perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
> > 0.00 +0.7 0.69 ± 10%
> > perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
> > 0.00 +0.8 0.79 ± 11%
> > perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> > 0.00 +0.8 0.83 ± 7%
> > perf-profile.calltrace.cycles-pp.__schedule.schedule.raid1_write_request.raid1_make_request.md_handle_request
> > 0.62 ± 3% +0.8 1.45 ± 22%
> > perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait.__xfs_log_force_lsn
> > 0.00 +0.8 0.83 ± 7%
> > perf-profile.calltrace.cycles-pp.schedule.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
> > 0.63 ± 2% +0.8 1.46 ± 22%
> > perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> > 0.62 ± 2% +0.8 1.46 ± 22%
> > perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn
> > 3.92 ± 2% +0.9 4.79 ± 6%
> > perf-profile.calltrace.cycles-pp.ret_from_fork
> > 3.92 ± 2% +0.9 4.79 ± 6%
> > perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
> > 0.69 ± 2% +0.9 1.64 ± 23%
> > perf-profile.calltrace.cycles-pp.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
> > 0.00 +1.2 1.17 ± 8%
> > perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request
> > 0.00 +1.2 1.23 ± 18%
> > perf-profile.calltrace.cycles-pp.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request.submit_flushes
> > 0.00 +1.3 1.27 ± 17%
> > perf-profile.calltrace.cycles-pp.raid1_write_request.raid1_make_request.md_handle_request.submit_flushes.process_one_work
> > 0.00 +1.3 1.27 ± 17%
> > perf-profile.calltrace.cycles-pp.md_handle_request.submit_flushes.process_one_work.worker_thread.kthread
> > 0.00 +1.3 1.27 ± 17%
> > perf-profile.calltrace.cycles-pp.raid1_make_request.md_handle_request.submit_flushes.process_one_work.worker_thread
> > 0.00 +1.3 1.27 ± 17%
> > perf-profile.calltrace.cycles-pp.submit_flushes.process_one_work.worker_thread.kthread.ret_from_fork
> > 0.00 +1.6 1.65 ± 14%
> > perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.raid_end_bio_io
> > 0.00 +1.7 1.71 ± 14%
> > perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.raid_end_bio_io.raid1_end_write_request
> > 0.00 +1.7 1.71 ± 14%
> > perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.raid_end_bio_io.raid1_end_write_request.brd_make_request
> > 0.00 +1.9 1.86 ± 13%
> > perf-profile.calltrace.cycles-pp.__wake_up_common_lock.raid_end_bio_io.raid1_end_write_request.brd_make_request.generic_make_request
> > 0.00 +2.1 2.10 ± 10%
> > perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn
> > 0.00 +2.1 2.10 ± 10%
> > perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> > 0.00 +2.1 2.11 ± 10%
> > perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
> > 0.00 +2.2 2.16 ± 10%
> > perf-profile.calltrace.cycles-pp.raid_end_bio_io.raid1_end_write_request.brd_make_request.generic_make_request.flush_bio_list
> > 2.24 ± 4% +2.2 4.44 ± 15%
> > perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
> > 0.00 +2.3 2.25 ± 10%
> > perf-profile.calltrace.cycles-pp.raid1_end_write_request.brd_make_request.generic_make_request.flush_bio_list.flush_pending_writes
> > 0.00 +2.3 2.30 ± 20%
> > perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.raid1_write_request.raid1_make_request.md_handle_request
> > 0.00 +2.4 2.35 ± 20%
> > perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
> > 0.37 ± 65% +2.4 2.81 ± 7%
> > perf-profile.calltrace.cycles-pp.md_thread.kthread.ret_from_fork
> > 0.26 ±100% +2.5 2.81 ± 7%
> > perf-profile.calltrace.cycles-pp.raid1d.md_thread.kthread.ret_from_fork
> > 0.26 ±100% +2.5 2.81 ± 7%
> > perf-profile.calltrace.cycles-pp.flush_pending_writes.raid1d.md_thread.kthread.ret_from_fork
> > 0.26 ±100% +2.6 2.81 ± 7%
> > perf-profile.calltrace.cycles-pp.flush_bio_list.flush_pending_writes.raid1d.md_thread.kthread
> > 0.10 ±200% +2.7 2.76 ± 7%
> > perf-profile.calltrace.cycles-pp.generic_make_request.flush_bio_list.flush_pending_writes.raid1d.md_thread
> > 0.00 +2.7 2.73 ± 7%
> > perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.flush_bio_list.flush_pending_writes.raid1d
> > 1.20 ± 3% +3.1 4.35 ± 15%
> > perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> > 0.63 ± 6% +3.8 4.38 ± 27%
> > perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync
> > 0.63 ± 5% +3.8 4.39 ± 27%
> > perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
> > 0.63 ± 5% +3.8 4.40 ± 27%
> > perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> > 1.26 ± 5% +5.3 6.55 ± 27%
> > perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter
> > 1.27 ± 5% +5.3 6.55 ± 27%
> > perf-profile.calltrace.cycles-pp._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> > 1.30 ± 4% +8.4 9.72 ± 9%
> > perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
> > 1.33 ± 4% +8.9 10.26 ± 9%
> > perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> > 2.28 ± 2% +9.1 11.36 ± 27%
> > perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
> > 1.59 ± 4% +10.4 11.97 ± 9%
> > perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> > 1.59 ± 4% +10.4 11.98 ± 9%
> > perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
> > 1.59 ± 4% +10.4 11.98 ± 9%
> > perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
> > 1.63 ± 4% +10.8 12.47 ± 8%
> > perf-profile.calltrace.cycles-pp.secondary_startup_64
> > 0.00 +57.7 57.66 ± 10%
> > perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.raid1_write_request.raid1_make_request
> > 0.00 +57.7 57.73 ± 10%
> > perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request
> > 0.05 ±299% +57.8 57.85 ± 9%
> > perf-profile.calltrace.cycles-pp.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
> > 0.19 ±154% +62.5 62.73 ± 8%
> > perf-profile.calltrace.cycles-pp.raid1_write_request.raid1_make_request.md_handle_request.md_make_request.generic_make_request
> > 0.19 ±154% +62.6 62.76 ± 8%
> > perf-profile.calltrace.cycles-pp.md_handle_request.md_make_request.generic_make_request.submit_bio.xfs_submit_ioend
> > 0.19 ±154% +62.6 62.79 ± 8%
> > perf-profile.calltrace.cycles-pp.md_make_request.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages
> > 0.20 ±154% +62.6 62.81 ± 8%
> > perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages
> > 0.20 ±154% +62.6 62.81 ± 8%
> > perf-profile.calltrace.cycles-pp.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> > 0.20 ±154% +62.6 62.82 ± 8%
> > perf-profile.calltrace.cycles-pp.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> > 0.29 ±125% +62.8 63.09 ± 8%
> > perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
> > 0.29 ±126% +62.8 63.10 ± 8%
> > perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_write_iter
> > 0.29 ±125% +62.8 63.11 ± 8%
> > perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_write_iter.__vfs_write
> > 0.62 ± 41% +62.9 63.52 ± 7%
> > perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_write_iter.__vfs_write.vfs_write
> > 88.51 -88.2 0.26 ± 19%
> > perf-profile.children.cycles-pp.md_flush_request
> > 87.57 -87.2 0.35 ± 19%
> > perf-profile.children.cycles-pp.submit_bio_wait
> > 87.59 -87.2 0.39 ± 19%
> > perf-profile.children.cycles-pp.blkdev_issue_flush
> > 83.26 -83.2 0.02 ±123%
> > perf-profile.children.cycles-pp._raw_spin_lock_irq
> > 88.85 -25.7 63.11 ± 8%
> > perf-profile.children.cycles-pp.md_make_request
> > 88.90 -25.7 63.17 ± 8%
> > perf-profile.children.cycles-pp.submit_bio
> > 88.83 -24.5 64.31 ± 8%
> > perf-profile.children.cycles-pp.raid1_make_request
> > 88.84 -24.5 64.33 ± 8%
> > perf-profile.children.cycles-pp.md_handle_request
> > 89.38 -23.5 65.92 ± 7%
> > perf-profile.children.cycles-pp.generic_make_request
> > 89.90 -13.4 76.51 ± 2%
> > perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
> > 92.79 -13.0 79.76
> > perf-profile.children.cycles-pp.xfs_file_fsync
> > 93.08 -12.6 80.49
> > perf-profile.children.cycles-pp.xfs_file_write_iter
> > 93.09 -12.6 80.54
> > perf-profile.children.cycles-pp.__vfs_write
> > 93.13 -12.5 80.60
> > perf-profile.children.cycles-pp.vfs_write
> > 93.13 -12.5 80.61
> > perf-profile.children.cycles-pp.ksys_write
> > 93.22 -12.4 80.83
> > perf-profile.children.cycles-pp.do_syscall_64
> > 93.22 -12.4 80.83
> > perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> > 3.40 ± 2% -1.4 1.97 ± 8%
> > perf-profile.children.cycles-pp.worker_thread
> > 3.33 ± 2% -1.4 1.96 ± 9%
> > perf-profile.children.cycles-pp.process_one_work
> > 1.03 ± 7% -1.0 0.07 ± 37%
> > perf-profile.children.cycles-pp.xlog_cil_force_lsn
> > 1.69 ± 2% -0.7 0.96 ± 4%
> > perf-profile.children.cycles-pp.reschedule_interrupt
> > 1.66 ± 2% -0.7 0.94 ± 4%
> > perf-profile.children.cycles-pp.scheduler_ipi
> > 1.13 ± 2% -0.7 0.47 ± 11%
> > perf-profile.children.cycles-pp.finish_wait
> > 0.54 ± 8% -0.4 0.10 ± 38%
> > perf-profile.children.cycles-pp.xlog_cil_push
> > 0.49 ± 9% -0.4 0.09 ± 35%
> > perf-profile.children.cycles-pp.xlog_write
> > 0.10 ± 8% -0.1 0.04 ± 67%
> > perf-profile.children.cycles-pp.flush_work
> > 0.20 ± 5% -0.0 0.16 ± 11%
> > perf-profile.children.cycles-pp.reweight_entity
> > 0.06 ± 10% +0.0 0.10 ± 23%
> > perf-profile.children.cycles-pp.brd_lookup_page
> > 0.18 ± 5% +0.0 0.23 ± 13%
> > perf-profile.children.cycles-pp.__update_load_avg_se
> > 0.02 ±153% +0.1 0.07 ± 16%
> > perf-profile.children.cycles-pp.delay_tsc
> > 0.03 ±100% +0.1 0.08 ± 15%
> > perf-profile.children.cycles-pp.find_next_bit
> > 0.08 ± 5% +0.1 0.14 ± 14%
> > perf-profile.children.cycles-pp.native_write_msr
> > 0.29 ± 4% +0.1 0.36 ± 8%
> > perf-profile.children.cycles-pp.__orc_find
> > 0.40 ± 4% +0.1 0.46 ± 7%
> > perf-profile.children.cycles-pp.dequeue_task_fair
> > 0.11 ± 11% +0.1 0.18 ± 14%
> > perf-profile.children.cycles-pp.__module_text_address
> > 0.12 ± 8% +0.1 0.19 ± 13%
> > perf-profile.children.cycles-pp.is_module_text_address
> > 0.04 ± 50% +0.1 0.12 ± 19%
> > perf-profile.children.cycles-pp.kmem_cache_alloc
> > 0.00 +0.1 0.08 ± 11%
> > perf-profile.children.cycles-pp.clear_page_erms
> > 0.00 +0.1 0.08 ± 28%
> > perf-profile.children.cycles-pp.__indirect_thunk_start
> > 0.01 ±200% +0.1 0.10 ± 25%
> > perf-profile.children.cycles-pp.xfs_trans_alloc
> > 0.00 +0.1 0.09 ± 18%
> > perf-profile.children.cycles-pp.md_wakeup_thread
> > 0.00 +0.1 0.09 ± 26%
> > perf-profile.children.cycles-pp.rebalance_domains
> > 0.00 +0.1 0.09 ± 26%
> > perf-profile.children.cycles-pp.get_next_timer_interrupt
> > 0.00 +0.1 0.09 ± 20%
> > perf-profile.children.cycles-pp.ktime_get
> > 0.18 ± 4% +0.1 0.27 ± 12%
> > perf-profile.children.cycles-pp.idle_cpu
> > 0.20 ± 6% +0.1 0.30 ± 9%
> > perf-profile.children.cycles-pp.unwind_get_return_address
> > 0.16 ± 10% +0.1 0.25 ± 13%
> > perf-profile.children.cycles-pp.__module_address
> > 0.03 ±100% +0.1 0.13 ± 8%
> > perf-profile.children.cycles-pp.brd_insert_page
> > 0.06 ± 9% +0.1 0.16 ± 14%
> > perf-profile.children.cycles-pp.task_tick_fair
> > 0.08 ± 12% +0.1 0.18 ± 24%
> > perf-profile.children.cycles-pp.bio_alloc_bioset
> > 0.03 ± 81% +0.1 0.14 ± 27%
> > perf-profile.children.cycles-pp.generic_make_request_checks
> > 0.17 ± 7% +0.1 0.28 ± 11%
> > perf-profile.children.cycles-pp.__kernel_text_address
> > 0.11 ± 9% +0.1 0.22 ± 15%
> > perf-profile.children.cycles-pp.wake_up_page_bit
> > 0.16 ± 6% +0.1 0.27 ± 10%
> > perf-profile.children.cycles-pp.kernel_text_address
> > 0.00 +0.1 0.11 ± 11%
> > perf-profile.children.cycles-pp.get_page_from_freelist
> > 0.00 +0.1 0.11 ± 19%
> > perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
> > 0.00 +0.1 0.11 ± 7%
> > perf-profile.children.cycles-pp.__alloc_pages_nodemask
> > 0.08 ± 10% +0.1 0.19 ± 22%
> > perf-profile.children.cycles-pp.xfs_do_writepage
> > 0.25 ± 4% +0.1 0.37 ± 10%
> > perf-profile.children.cycles-pp.switch_mm_irqs_off
> > 0.00 +0.1 0.12 ± 13%
> > perf-profile.children.cycles-pp.switch_mm
> > 0.08 ± 38% +0.1 0.20 ± 19%
> > perf-profile.children.cycles-pp.io_serial_in
> > 0.18 ± 5% +0.1 0.31 ± 7%
> > perf-profile.children.cycles-pp.dequeue_entity
> > 0.00 +0.1 0.13 ± 26%
> > perf-profile.children.cycles-pp.tick_nohz_next_event
> > 0.06 ± 11% +0.1 0.19 ± 19%
> > perf-profile.children.cycles-pp.mempool_alloc
> > 0.32 ± 5% +0.1 0.45 ± 6%
> > perf-profile.children.cycles-pp.orc_find
> > 0.15 ± 10% +0.1 0.29 ± 19%
> > perf-profile.children.cycles-pp.xfs_destroy_ioend
> > 0.15 ± 11% +0.1 0.30 ± 18%
> > perf-profile.children.cycles-pp.call_bio_endio
> > 0.08 ± 17% +0.2 0.23 ± 25%
> > perf-profile.children.cycles-pp.xlog_state_done_syncing
> > 0.00 +0.2 0.15 ± 22%
> > perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
> > 0.12 ± 8% +0.2 0.27 ± 23%
> > perf-profile.children.cycles-pp.write_cache_pages
> > 0.10 ± 16% +0.2 0.26 ± 16%
> > perf-profile.children.cycles-pp.wait_for_xmitr
> > 0.10 ± 19% +0.2 0.25 ± 14%
> > perf-profile.children.cycles-pp.serial8250_console_putchar
> > 0.10 ± 17% +0.2 0.26 ± 13%
> > perf-profile.children.cycles-pp.uart_console_write
> > 0.10 ± 16% +0.2 0.26 ± 15%
> > perf-profile.children.cycles-pp.serial8250_console_write
> > 0.11 ± 15% +0.2 0.27 ± 15%
> > perf-profile.children.cycles-pp.console_unlock
> > 0.09 ± 9% +0.2 0.26 ± 12%
> > perf-profile.children.cycles-pp.scheduler_tick
> > 0.10 ± 18% +0.2 0.28 ± 15%
> > perf-profile.children.cycles-pp.irq_work_run_list
> > 0.10 ± 15% +0.2 0.28 ± 14%
> > perf-profile.children.cycles-pp.xlog_state_do_callback
> > 0.09 ± 12% +0.2 0.27 ± 16%
> > perf-profile.children.cycles-pp.irq_work_run
> > 0.09 ± 12% +0.2 0.27 ± 16%
> > perf-profile.children.cycles-pp.printk
> > 0.09 ± 12% +0.2 0.27 ± 16%
> > perf-profile.children.cycles-pp.vprintk_emit
> > 0.09 ± 12% +0.2 0.27 ± 17%
> > perf-profile.children.cycles-pp.irq_work_interrupt
> > 0.09 ± 12% +0.2 0.27 ± 17%
> > perf-profile.children.cycles-pp.smp_irq_work_interrupt
> > 0.00 +0.2 0.18 ± 16%
> > perf-profile.children.cycles-pp.poll_idle
> > 0.30 ± 4% +0.2 0.49 ± 11%
> > perf-profile.children.cycles-pp.update_load_avg
> > 1.39 ± 2% +0.2 1.59 ± 6%
> > perf-profile.children.cycles-pp.__save_stack_trace
> > 1.43 +0.2 1.65 ± 6%
> > perf-profile.children.cycles-pp.save_stack_trace_tsk
> > 0.14 ± 13% +0.2 0.36 ± 13%
> > perf-profile.children.cycles-pp.update_process_times
> > 0.00 +0.2 0.23 ± 22%
> > perf-profile.children.cycles-pp.find_busiest_group
> > 0.22 ± 6% +0.2 0.45 ± 18%
> > perf-profile.children.cycles-pp.brd_do_bvec
> > 0.14 ± 13% +0.2 0.38 ± 14%
> > perf-profile.children.cycles-pp.tick_sched_handle
> > 0.10 ± 8% +0.2 0.34 ± 26%
> > perf-profile.children.cycles-pp.xfs_log_commit_cil
> > 0.07 ± 10% +0.3 0.33 ± 23%
> > perf-profile.children.cycles-pp.io_schedule
> > 0.03 ± 83% +0.3 0.29 ± 27%
> > perf-profile.children.cycles-pp.__softirqentry_text_start
> > 0.11 ± 5% +0.3 0.36 ± 25%
> > perf-profile.children.cycles-pp.__xfs_trans_commit
> > 0.06 ± 36% +0.3 0.31 ± 26%
> > perf-profile.children.cycles-pp.irq_exit
> > 0.08 ± 9% +0.3 0.35 ± 23%
> > perf-profile.children.cycles-pp.wait_on_page_bit_common
> > 0.15 ± 12% +0.3 0.42 ± 14%
> > perf-profile.children.cycles-pp.tick_sched_timer
> > 0.10 ± 11% +0.3 0.39 ± 22%
> > perf-profile.children.cycles-pp.__filemap_fdatawait_range
> > 0.06 ± 12% +0.3 0.37 ± 9%
> > perf-profile.children.cycles-pp.schedule_idle
> > 0.02 ±153% +0.3 0.34 ± 17%
> > perf-profile.children.cycles-pp.menu_select
> > 0.17 ± 5% +0.3 0.49 ± 22%
> > perf-profile.children.cycles-pp.xfs_vn_update_time
> > 0.19 ± 12% +0.3 0.51 ± 18%
> > perf-profile.children.cycles-pp.xlog_iodone
> > 0.18 ± 5% +0.3 0.51 ± 22%
> > perf-profile.children.cycles-pp.file_update_time
> > 0.18 ± 5% +0.3 0.51 ± 21%
> > perf-profile.children.cycles-pp.xfs_file_aio_write_checks
> > 0.21 ± 11% +0.4 0.60 ± 15%
> > perf-profile.children.cycles-pp.__hrtimer_run_queues
> > 0.26 ± 6% +0.4 0.69 ± 16%
> > perf-profile.children.cycles-pp.pick_next_task_fair
> > 1.20 ± 2% +0.4 1.64 ± 10%
> > perf-profile.children.cycles-pp.schedule
> > 0.28 ± 5% +0.4 0.72 ± 21%
> > perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
> > 0.00 +0.4 0.44 ± 22%
> > perf-profile.children.cycles-pp.load_balance
> > 0.25 ± 8% +0.5 0.74 ± 15%
> > perf-profile.children.cycles-pp.hrtimer_interrupt
> > 1.30 ± 2% +0.7 2.00 ± 9%
> > perf-profile.children.cycles-pp.__schedule
> > 0.31 ± 8% +0.8 1.09 ± 16%
> > perf-profile.children.cycles-pp.smp_apic_timer_interrupt
> > 0.31 ± 8% +0.8 1.09 ± 16%
> > perf-profile.children.cycles-pp.apic_timer_interrupt
> > 3.92 ± 2% +0.9 4.79 ± 6%
> > perf-profile.children.cycles-pp.ret_from_fork
> > 3.92 ± 2% +0.9 4.79 ± 6%
> > perf-profile.children.cycles-pp.kthread
> > 0.69 ± 2% +0.9 1.64 ± 23%
> > perf-profile.children.cycles-pp.xlog_wait
> > 0.08 ± 13% +1.2 1.27 ± 17%
> > perf-profile.children.cycles-pp.submit_flushes
> > 0.16 ± 9% +1.6 1.74 ± 4%
> > perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> > 0.17 ± 9% +2.0 2.16 ± 10%
> > perf-profile.children.cycles-pp.raid_end_bio_io
> > 0.21 ± 6% +2.0 2.25 ± 10%
> > perf-profile.children.cycles-pp.raid1_end_write_request
> > 2.24 ± 4% +2.2 4.44 ± 15%
> > perf-profile.children.cycles-pp.xfs_log_force_lsn
> > 0.46 ± 6% +2.3 2.73 ± 7%
> > perf-profile.children.cycles-pp.brd_make_request
> > 0.51 ± 6% +2.3 2.81 ± 7%
> > perf-profile.children.cycles-pp.md_thread
> > 0.49 ± 6% +2.3 2.81 ± 7%
> > perf-profile.children.cycles-pp.raid1d
> > 0.49 ± 6% +2.3 2.81 ± 7%
> > perf-profile.children.cycles-pp.flush_pending_writes
> > 0.49 ± 6% +2.3 2.81 ± 7%
> > perf-profile.children.cycles-pp.flush_bio_list
> > 1.80 ± 3% +5.6 7.44 ± 27%
> > perf-profile.children.cycles-pp._raw_spin_lock
> > 2.12 ± 4% +5.8 7.97 ± 20%
> > perf-profile.children.cycles-pp.remove_wait_queue
> > 1.33 ± 4% +8.8 10.12 ± 8%
> > perf-profile.children.cycles-pp.intel_idle
> > 1.37 ± 4% +9.3 10.71 ± 8%
> > perf-profile.children.cycles-pp.cpuidle_enter_state
> > 1.59 ± 4% +10.4 11.98 ± 9%
> > perf-profile.children.cycles-pp.start_secondary
> > 1.63 ± 4% +10.8 12.47 ± 8%
> > perf-profile.children.cycles-pp.secondary_startup_64
> > 1.63 ± 4% +10.8 12.47 ± 8%
> > perf-profile.children.cycles-pp.cpu_startup_entry
> > 1.63 ± 4% +10.9 12.49 ± 8%
> > perf-profile.children.cycles-pp.do_idle
> > 3.48 +12.2 15.72 ± 23%
> > perf-profile.children.cycles-pp.__xfs_log_force_lsn
> > 1.36 ± 12% +57.8 59.12 ± 10%
> > perf-profile.children.cycles-pp.prepare_to_wait_event
> > 0.43 ± 38% +62.4 62.82 ± 8%
> > perf-profile.children.cycles-pp.xfs_submit_ioend
> > 0.55 ± 29% +62.5 63.10 ± 8%
> > perf-profile.children.cycles-pp.xfs_vm_writepages
> > 0.55 ± 30% +62.5 63.10 ± 8%
> > perf-profile.children.cycles-pp.do_writepages
> > 0.55 ± 29% +62.6 63.11 ± 8%
> > perf-profile.children.cycles-pp.__filemap_fdatawrite_range
> > 0.66 ± 25% +62.9 63.52 ± 7%
> > perf-profile.children.cycles-pp.file_write_and_wait_range
> > 0.39 ± 43% +63.6 64.02 ± 8%
> > perf-profile.children.cycles-pp.raid1_write_request
> > 5.43 ± 3% +64.2 69.64 ± 5%
> > perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> > 89.86 -13.5 76.31 ± 2%
> > perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
> > 0.14 ± 8% -0.0 0.09 ± 19%
> > perf-profile.self.cycles-pp.md_flush_request
> > 0.10 ± 12% -0.0 0.07 ± 21%
> > perf-profile.self.cycles-pp.account_entity_enqueue
> > 0.06 ± 7% +0.0 0.08 ± 12%
> > perf-profile.self.cycles-pp.pick_next_task_fair
> > 0.05 ± 12% +0.0 0.08 ± 18%
> > perf-profile.self.cycles-pp.___perf_sw_event
> > 0.15 ± 6% +0.0 0.18 ± 9%
> > perf-profile.self.cycles-pp.__update_load_avg_se
> > 0.17 ± 4% +0.0 0.22 ± 10%
> > perf-profile.self.cycles-pp.__schedule
> > 0.10 ± 11% +0.1 0.15 ± 11%
> > perf-profile.self.cycles-pp._raw_spin_lock
> > 0.02 ±153% +0.1 0.07 ± 16%
> > perf-profile.self.cycles-pp.delay_tsc
> > 0.02 ±152% +0.1 0.07 ± 23%
> > perf-profile.self.cycles-pp.set_next_entity
> > 0.03 ±100% +0.1 0.08 ± 15%
> > perf-profile.self.cycles-pp.find_next_bit
> > 0.08 ± 5% +0.1 0.14 ± 14%
> > perf-profile.self.cycles-pp.native_write_msr
> > 0.01 ±200% +0.1 0.07 ± 23%
> > perf-profile.self.cycles-pp.kmem_cache_alloc
> > 0.29 ± 4% +0.1 0.36 ± 8%
> > perf-profile.self.cycles-pp.__orc_find
> > 0.14 ± 7% +0.1 0.21 ± 12%
> > perf-profile.self.cycles-pp.switch_mm_irqs_off
> > 0.00 +0.1 0.08 ± 11%
> > perf-profile.self.cycles-pp.clear_page_erms
> > 0.00 +0.1 0.08 ± 28%
> > perf-profile.self.cycles-pp.__indirect_thunk_start
> > 0.00 +0.1 0.08 ± 20%
> > perf-profile.self.cycles-pp.md_wakeup_thread
> > 0.34 ± 6% +0.1 0.43 ± 12%
> > perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> > 0.18 ± 4% +0.1 0.27 ± 12%
> > perf-profile.self.cycles-pp.idle_cpu
> > 0.16 ± 10% +0.1 0.25 ± 13%
> > perf-profile.self.cycles-pp.__module_address
> > 0.06 ± 11% +0.1 0.17 ± 14%
> > perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> > 0.08 ± 38% +0.1 0.20 ± 19%
> > perf-profile.self.cycles-pp.io_serial_in
> > 0.18 ± 5% +0.1 0.32 ± 15%
> > perf-profile.self.cycles-pp.update_load_avg
> > 0.00 +0.1 0.15 ± 17%
> > perf-profile.self.cycles-pp.poll_idle
> > 0.00 +0.2 0.15 ± 16%
> > perf-profile.self.cycles-pp.menu_select
> > 0.00 +0.2 0.18 ± 24%
> > perf-profile.self.cycles-pp.find_busiest_group
> > 0.02 ±152% +0.3 0.35 ± 21%
> > perf-profile.self.cycles-pp.raid1_write_request
> > 1.33 ± 4% +8.8 10.12 ± 8%
> > perf-profile.self.cycles-pp.intel_idle
> >
> >
> >
> > aim7.jobs-per-min
> >
> > 1700
> > +-+------------------------------------------------------------------+
> > |+ ++++++ :+ ++++ ++++ +++ ++++++ + + ++++++++++++ ++
> > |++|
> > 1600 +-+ + +++ + +++++ ++.++ + ++ ++ + ++
> > |
> > | |
> > | |
> > 1500 +-+
> > |
> > | |
> > 1400 +-+
> > |
> > | |
> > 1300 +-+
> > |
> > | |
> > O OO OO O O O
> > |
> > 1200 +OO OOOOOOOOO OO OOOOOOOOOOOOOO OOOOOOOOO O
> > |
> > | |
> > 1100
> > +-+------------------------------------------------------------------+
> >
> >
> >
> > [*] bisect-good sample
> > [O] bisect-bad sample
> >
> >
> > Disclaimer:
> > Results have been estimated based on internal Intel analysis and are
> > provided
> > for informational purposes only. Any difference in system hardware or
> > software
> > design or configuration may affect actual performance.
> >
> >
> > Thanks,
> > Xiaolong
>

2018-07-16 08:05:57

by Aaron Lu

[permalink] [raw]

Subject: Re: [LKP] [lkp-robot] [MD] 5a409b4f56: aim7.jobs-per-min -27.5% regression

On Mon, Jul 16, 2018 at 04:01:44AM -0400, Xiao Ni wrote:
> Hi Aaron
>
> I have no update for this yet. I'll have a look this week and give response later.

Good to hear this, thanks for your time.