Greeting,
FYI, we noticed a 16.0% improvement of fsmark.app_overhead due to commit:
commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC: Convert socket page send code to use iov_iter()")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: fsmark
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
with following parameters:
iterations: 1x
nr_threads: 64t
disk: 1BRD_48G
fs: xfs
fs2: nfsv4
filesize: 4M
test_size: 40G
sync_method: fsyncBeforeClose
cpufreq_governor: performance
test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-ep01/40G/fsmark
commit:
e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use iov_iter_kvec()")
0472e47660 ("SUNRPC: Convert socket page send code to use iov_iter()")
e791f8e9380d945e 0472e476604998c127f3c80d291
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:4 50% 2:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
%stddev %change %stddev
\ | \
15118573 ± 2% +16.0% 17538083 fsmark.app_overhead
510.93 -22.7% 395.12 fsmark.files_per_sec
24.90 +22.8% 30.57 fsmark.time.elapsed_time
24.90 +22.8% 30.57 fsmark.time.elapsed_time.max
288.00 ± 2% -27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got
70.03 ± 2% -11.3% 62.14 fsmark.time.system_time
4391964 -16.7% 3658341 meminfo.max_used_kB
6.10 ± 4% +1.9 7.97 ± 3% mpstat.cpu.all.iowait%
0.27 -0.0 0.24 ± 3% mpstat.cpu.all.soft%
13668070 ± 40% +118.0% 29801846 ± 19% numa-numastat.node0.local_node
13677774 ± 40% +117.9% 29810258 ± 19% numa-numastat.node0.numa_hit
5.70 ± 3% +32.1% 7.53 ± 3% iostat.cpu.iowait
16.42 ± 2% -5.8% 15.47 iostat.cpu.system
2.57 -4.1% 2.46 iostat.cpu.user
1406781 ± 2% -15.5% 1188498 vmstat.io.bo
251792 ± 3% -16.6% 209928 vmstat.system.cs
84841 -1.9% 83239 vmstat.system.in
97374502 ± 20% +66.1% 1.617e+08 ± 17% cpuidle.C1E.time
573934 ± 19% +44.6% 829662 ± 26% cpuidle.C1E.usage
5.892e+08 ± 8% +15.3% 6.796e+08 ± 2% cpuidle.C6.time
1968016 ± 3% -15.1% 1670867 ± 3% cpuidle.POLL.time
106420 ± 47% +86.2% 198108 ± 35% numa-meminfo.node0.Active
106037 ± 48% +86.2% 197395 ± 35% numa-meminfo.node0.Active(anon)
105052 ± 48% +86.6% 196037 ± 35% numa-meminfo.node0.AnonPages
212876 ± 24% -41.5% 124572 ± 56% numa-meminfo.node1.Active
211801 ± 24% -41.5% 123822 ± 56% numa-meminfo.node1.Active(anon)
208559 ± 24% -42.2% 120547 ± 57% numa-meminfo.node1.AnonPages
9955 +1.6% 10116 proc-vmstat.nr_kernel_stack
452.25 ± 59% +280.9% 1722 ±100% proc-vmstat.numa_hint_faults_local
33817303 +55.0% 52421773 ± 5% proc-vmstat.numa_hit
33804286 +55.0% 52408807 ± 5% proc-vmstat.numa_local
33923002 +81.8% 61663426 ± 5% proc-vmstat.pgalloc_normal
184765 +9.3% 201985 proc-vmstat.pgfault
12840986 +216.0% 40581327 ± 7% proc-vmstat.pgfree
31447 ± 11% -26.1% 23253 ± 13% sched_debug.cfs_rq:/.min_vruntime.max
4241 ± 3% -12.2% 3724 ± 11% sched_debug.cfs_rq:/.min_vruntime.stddev
20631 ± 11% -36.7% 13069 ± 29% sched_debug.cfs_rq:/.spread0.max
4238 ± 4% -12.1% 3724 ± 11% sched_debug.cfs_rq:/.spread0.stddev
497105 ± 19% -16.0% 417777 ± 4% sched_debug.cpu.avg_idle.avg
21199 ± 10% -12.0% 18650 ± 3% sched_debug.cpu.nr_load_updates.max
2229 ± 10% -15.0% 1895 ± 4% sched_debug.cpu.nr_load_updates.stddev
4.86 ± 5% -23.6% 3.72 ± 5% sched_debug.cpu.nr_uninterruptible.stddev
524.75 ± 2% -10.7% 468.50 turbostat.Avg_MHz
5.26 ± 41% -1.6 3.66 ± 2% turbostat.C1%
573633 ± 19% +44.6% 829267 ± 26% turbostat.C1E
8.53 ± 20% +3.4 11.88 ± 17% turbostat.C1E%
76.75 -6.4% 71.86 turbostat.CorWatt
2534071 ± 2% +16.2% 2943968 turbostat.IRQ
2.89 ± 24% +52.9% 4.42 ± 22% turbostat.Pkg%pc2
104.28 -4.6% 99.51 turbostat.PkgWatt
2289 +18.8% 2720 turbostat.SMI
26496 ± 47% +86.2% 49347 ± 35% numa-vmstat.node0.nr_active_anon
26251 ± 48% +86.7% 49008 ± 35% numa-vmstat.node0.nr_anon_pages
26496 ± 47% +86.2% 49347 ± 35% numa-vmstat.node0.nr_zone_active_anon
8272770 ± 35% +99.6% 16515977 ± 17% numa-vmstat.node0.numa_hit
8199681 ± 36% +100.9% 16474282 ± 17% numa-vmstat.node0.numa_local
52953 ± 24% -41.5% 30956 ± 56% numa-vmstat.node1.nr_active_anon
52144 ± 24% -42.2% 30136 ± 57% numa-vmstat.node1.nr_anon_pages
2899 ± 2% -19.4% 2336 ± 10% numa-vmstat.node1.nr_writeback
52953 ± 24% -41.5% 30956 ± 56% numa-vmstat.node1.nr_zone_active_anon
4549 ± 8% -18.9% 3689 ± 7% numa-vmstat.node1.nr_zone_write_pending
43.25 -1.6 41.69 perf-stat.i.cache-miss-rate%
286377 ± 2% -19.1% 231819 ± 2% perf-stat.i.context-switches
2.258e+10 ± 4% -16.2% 1.893e+10 ± 10% perf-stat.i.cpu-cycles
4084 -34.0% 2696 ± 2% perf-stat.i.cpu-migrations
21563675 ± 24% -30.8% 14929445 ± 5% perf-stat.i.dTLB-load-misses
1996341 ± 16% -25.7% 1482826 ± 6% perf-stat.i.dTLB-store-misses
2.313e+09 ± 4% -11.7% 2.042e+09 ± 6% perf-stat.i.dTLB-stores
75.93 -1.3 74.66 perf-stat.i.iTLB-load-miss-rate%
1561381 ± 3% -13.4% 1351731 perf-stat.i.iTLB-load-misses
549643 ± 4% -10.1% 494255 ± 3% perf-stat.i.iTLB-loads
6423 -10.0% 5779 perf-stat.i.minor-faults
31.72 ± 2% +4.4 36.14 perf-stat.i.node-load-miss-rate%
23492890 +23.5% 29020305 ± 3% perf-stat.i.node-load-misses
15.13 ± 8% +7.9 23.02 ± 3% perf-stat.i.node-store-miss-rate%
11517615 +71.3% 19734054 ± 2% perf-stat.i.node-store-misses
6423 -10.0% 5779 perf-stat.i.page-faults
16.88 ± 14% +33.4% 22.52 ± 23% perf-stat.overall.MPKI
46.67 -2.4 44.31 perf-stat.overall.cache-miss-rate%
31.96 +4.8 36.72 perf-stat.overall.node-load-miss-rate%
13.85 +8.0 21.87 perf-stat.overall.node-store-miss-rate%
2.266e+08 +2.8% 2.331e+08 ± 2% perf-stat.ps.cache-references
275634 ± 2% -18.5% 224718 ± 2% perf-stat.ps.context-switches
2.174e+10 ± 4% -15.6% 1.835e+10 ± 10% perf-stat.ps.cpu-cycles
3931 -33.5% 2614 ± 2% perf-stat.ps.cpu-migrations
20746040 ± 24% -30.2% 14476157 ± 5% perf-stat.ps.dTLB-load-misses
1921077 ± 16% -25.2% 1437750 ± 6% perf-stat.ps.dTLB-store-misses
2.227e+09 ± 4% -11.1% 1.979e+09 ± 6% perf-stat.ps.dTLB-stores
1503433 ± 3% -12.8% 1310741 perf-stat.ps.iTLB-load-misses
529058 ± 4% -9.4% 479204 ± 3% perf-stat.ps.iTLB-loads
6200 -9.4% 5620 perf-stat.ps.minor-faults
22613159 +24.4% 28133123 ± 3% perf-stat.ps.node-load-misses
11085254 +72.6% 19131576 ± 2% perf-stat.ps.node-store-misses
6200 -9.4% 5620 perf-stat.ps.page-faults
7008 ± 13% +55.2% 10876 ± 31% softirqs.CPU1.NET_RX
6509 ± 4% +51.8% 9883 ± 27% softirqs.CPU1.RCU
7294 ± 13% +36.7% 9974 ± 8% softirqs.CPU10.RCU
7800 ± 14% +85.5% 14469 ± 23% softirqs.CPU12.NET_RX
5697 ± 43% +77.5% 10110 ± 24% softirqs.CPU13.NET_RX
15944 ± 9% +14.6% 18278 ± 12% softirqs.CPU14.TIMER
6064 ± 19% +68.6% 10223 ± 31% softirqs.CPU15.NET_RX
7796 ± 14% +80.3% 14059 ± 25% softirqs.CPU16.NET_RX
15934 ± 10% +22.1% 19452 ± 11% softirqs.CPU18.TIMER
6725 ± 7% +40.0% 9413 ± 18% softirqs.CPU19.NET_RX
5710 ± 3% +53.4% 8756 ± 12% softirqs.CPU20.RCU
7018 ± 14% +65.5% 11616 ± 40% softirqs.CPU21.NET_RX
6389 ± 18% +66.9% 10666 ± 31% softirqs.CPU23.NET_RX
7259 ± 7% +36.1% 9881 ± 6% softirqs.CPU24.RCU
6491 ± 20% +58.5% 10289 ± 33% softirqs.CPU25.NET_RX
7090 ± 10% +58.7% 11256 ± 29% softirqs.CPU27.NET_RX
6333 ± 27% +70.3% 10786 ± 27% softirqs.CPU29.NET_RX
5813 ± 20% +84.2% 10706 ± 36% softirqs.CPU3.NET_RX
7041 ± 23% +105.7% 14483 ± 21% softirqs.CPU30.NET_RX
6654 ± 7% +64.1% 10918 ± 29% softirqs.CPU31.NET_RX
18019 ± 10% +11.1% 20016 ± 7% softirqs.CPU31.TIMER
4666 ± 8% +104.0% 9518 ± 23% softirqs.CPU32.RCU
15721 ± 16% +35.9% 21371 ± 11% softirqs.CPU32.TIMER
15684 ± 13% +40.0% 21959 ± 18% softirqs.CPU34.TIMER
6489 ± 15% +69.7% 11013 ± 37% softirqs.CPU35.NET_RX
7930 ± 7% +82.1% 14442 ± 29% softirqs.CPU36.NET_RX
15744 ± 14% +24.3% 19563 ± 6% softirqs.CPU36.TIMER
7028 ± 13% +29.3% 9085 ± 11% softirqs.CPU39.NET_RX
7491 ± 22% +100.4% 15011 ± 24% softirqs.CPU4.NET_RX
6119 ± 13% +58.7% 9710 ± 37% softirqs.CPU5.NET_RX
6980 ± 8% +47.8% 10318 ± 42% softirqs.CPU7.NET_RX
285674 +65.7% 473395 softirqs.NET_RX
267950 ± 5% +21.1% 324597 softirqs.RCU
238298 ± 2% +15.1% 274371 ± 2% softirqs.SCHED
689305 ± 3% +14.1% 786236 ± 3% softirqs.TIMER
56196 ± 2% +19.9% 67389 interrupts.CPU0.LOC:Local_timer_interrupts
55971 ± 2% +19.7% 67005 interrupts.CPU1.LOC:Local_timer_interrupts
5265 ± 17% -32.2% 3568 ± 29% interrupts.CPU1.RES:Rescheduling_interrupts
56163 ± 2% +19.2% 66960 interrupts.CPU10.LOC:Local_timer_interrupts
56178 ± 2% +19.5% 67139 interrupts.CPU11.LOC:Local_timer_interrupts
4669 ± 15% -30.3% 3253 ± 29% interrupts.CPU11.RES:Rescheduling_interrupts
56081 +19.7% 67142 interrupts.CPU12.LOC:Local_timer_interrupts
55999 ± 2% +19.5% 66893 interrupts.CPU13.LOC:Local_timer_interrupts
4746 ± 23% -30.0% 3324 ± 29% interrupts.CPU13.RES:Rescheduling_interrupts
55991 ± 2% +19.5% 66898 interrupts.CPU14.LOC:Local_timer_interrupts
55855 ± 2% +20.0% 67031 interrupts.CPU15.LOC:Local_timer_interrupts
55943 ± 2% +19.9% 67087 interrupts.CPU16.LOC:Local_timer_interrupts
56164 +18.9% 66802 interrupts.CPU17.LOC:Local_timer_interrupts
4579 ± 20% -45.6% 2492 ± 16% interrupts.CPU17.RES:Rescheduling_interrupts
55913 ± 2% +19.6% 66860 interrupts.CPU18.LOC:Local_timer_interrupts
55820 ± 2% +19.7% 66822 interrupts.CPU19.LOC:Local_timer_interrupts
4733 ± 17% -35.5% 3052 ± 21% interrupts.CPU19.RES:Rescheduling_interrupts
55989 ± 2% +20.0% 67177 interrupts.CPU2.LOC:Local_timer_interrupts
55891 ± 2% +20.3% 67258 interrupts.CPU20.LOC:Local_timer_interrupts
55966 ± 2% +19.6% 66921 interrupts.CPU21.LOC:Local_timer_interrupts
4920 ± 21% -33.4% 3278 ± 31% interrupts.CPU21.RES:Rescheduling_interrupts
55945 ± 2% +19.9% 67098 interrupts.CPU22.LOC:Local_timer_interrupts
55945 ± 2% +19.9% 67073 interrupts.CPU23.LOC:Local_timer_interrupts
4972 ± 24% -34.1% 3277 ± 36% interrupts.CPU23.RES:Rescheduling_interrupts
56093 ± 2% +19.8% 67185 interrupts.CPU24.LOC:Local_timer_interrupts
55910 ± 2% +19.7% 66914 interrupts.CPU25.LOC:Local_timer_interrupts
4660 ± 25% -28.9% 3313 ± 34% interrupts.CPU25.RES:Rescheduling_interrupts
56105 ± 2% +18.8% 66631 interrupts.CPU26.LOC:Local_timer_interrupts
55928 ± 2% +19.5% 66827 interrupts.CPU27.LOC:Local_timer_interrupts
55934 ± 2% +19.3% 66740 interrupts.CPU28.LOC:Local_timer_interrupts
55918 ± 2% +19.5% 66812 interrupts.CPU29.LOC:Local_timer_interrupts
4825 ± 22% -30.3% 3362 ± 30% interrupts.CPU29.RES:Rescheduling_interrupts
55920 ± 2% +19.8% 67004 interrupts.CPU3.LOC:Local_timer_interrupts
56007 ± 2% +19.5% 66917 interrupts.CPU30.LOC:Local_timer_interrupts
56238 +18.8% 66838 interrupts.CPU31.LOC:Local_timer_interrupts
4859 ± 21% -33.0% 3257 ± 36% interrupts.CPU31.RES:Rescheduling_interrupts
56099 +19.3% 66921 interrupts.CPU32.LOC:Local_timer_interrupts
55963 ± 2% +19.5% 66866 interrupts.CPU33.LOC:Local_timer_interrupts
55992 ± 2% +19.2% 66767 interrupts.CPU34.LOC:Local_timer_interrupts
55964 ± 2% +19.6% 66939 interrupts.CPU35.LOC:Local_timer_interrupts
4797 ± 26% -33.6% 3185 ± 38% interrupts.CPU35.RES:Rescheduling_interrupts
55770 ± 2% +20.2% 67016 interrupts.CPU36.LOC:Local_timer_interrupts
55889 +20.0% 67095 interrupts.CPU37.LOC:Local_timer_interrupts
56140 ± 2% +18.9% 66743 interrupts.CPU38.LOC:Local_timer_interrupts
56040 ± 2% +19.2% 66828 interrupts.CPU39.LOC:Local_timer_interrupts
5126 ± 23% -40.5% 3052 ± 11% interrupts.CPU39.RES:Rescheduling_interrupts
56022 ± 2% +19.9% 67149 interrupts.CPU4.LOC:Local_timer_interrupts
55850 ± 2% +20.1% 67066 interrupts.CPU5.LOC:Local_timer_interrupts
4741 ± 23% -31.9% 3227 ± 29% interrupts.CPU5.RES:Rescheduling_interrupts
55912 +19.5% 66798 interrupts.CPU6.LOC:Local_timer_interrupts
56062 ± 2% +19.6% 67052 interrupts.CPU7.LOC:Local_timer_interrupts
56118 +19.6% 67142 interrupts.CPU8.LOC:Local_timer_interrupts
56110 ± 2% +19.6% 67121 interrupts.CPU9.LOC:Local_timer_interrupts
2216 ± 43% -69.4% 677.50 ±112% interrupts.CPU9.NMI:Non-maskable_interrupts
2216 ± 43% -69.4% 677.50 ±112% interrupts.CPU9.PMI:Performance_monitoring_interrupts
2240009 ± 2% +19.6% 2678940 interrupts.LOC:Local_timer_interrupts
168296 ± 3% -10.1% 151314 interrupts.RES:Rescheduling_interrupts
20.37 -3.3 17.04 ± 5% perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
20.38 -3.3 17.04 ± 5% perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
20.27 -3.3 16.95 ± 5% perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
20.21 -3.3 16.91 ± 5% perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
26.02 -3.1 22.91 ± 4% perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
9.04 ± 2% -1.9 7.18 ± 7% perf-profile.calltrace.cycles-pp.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
9.02 ± 2% -1.9 7.17 ± 7% perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound
9.02 ± 2% -1.9 7.16 ± 7% perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write
9.03 ± 2% -1.9 7.18 ± 7% perf-profile.calltrace.cycles-pp.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch
9.07 ± 2% -1.9 7.22 ± 7% perf-profile.calltrace.cycles-pp.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
10.50 ± 2% -1.4 9.06 ± 4% perf-profile.calltrace.cycles-pp.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
10.51 ± 2% -1.4 9.08 ± 4% perf-profile.calltrace.cycles-pp.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
10.45 ± 2% -1.4 9.02 ± 4% perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch
9.82 ± 3% -1.4 8.45 ± 3% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound
9.82 ± 3% -1.4 8.45 ± 3% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit
9.82 ± 3% -1.4 8.45 ± 3% perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
10.61 ± 5% -1.1 9.46 ± 4% perf-profile.calltrace.cycles-pp.write
7.54 -1.1 6.41 ± 6% perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write
7.54 -1.1 6.41 ± 6% perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
10.27 ± 5% -1.1 9.15 ± 4% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
7.49 -1.1 6.37 ± 6% perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev
10.23 ± 5% -1.1 9.11 ± 4% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
10.35 ± 5% -1.1 9.23 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
10.08 ± 5% -1.1 8.96 ± 4% perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.33 ± 5% -1.1 9.22 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
10.05 ± 5% -1.1 8.95 ± 4% perf-profile.calltrace.cycles-pp.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64
9.80 ± 5% -1.1 8.73 ± 4% perf-profile.calltrace.cycles-pp.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_write
5.31 ± 5% -1.0 4.27 ± 5% perf-profile.calltrace.cycles-pp.rpc_free_task.rpc_async_release.process_one_work.worker_thread.kthread
5.31 ± 5% -1.0 4.27 ± 5% perf-profile.calltrace.cycles-pp.rpc_async_release.process_one_work.worker_thread.kthread.ret_from_fork
4.38 ± 5% -1.0 3.35 ± 2% perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv
4.38 ± 6% -1.0 3.36 ± 2% perf-profile.calltrace.cycles-pp.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd
4.39 ± 6% -1.0 3.40 ± 2% perf-profile.calltrace.cycles-pp.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd.kthread
3.53 ± 5% -0.9 2.62 ± 3% perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom
3.52 ± 5% -0.9 2.62 ± 3% perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.svc_recvfrom
5.01 ± 6% -0.9 4.14 ± 3% perf-profile.calltrace.cycles-pp.fsync
4.09 ± 7% -0.8 3.24 ± 5% perf-profile.calltrace.cycles-pp.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work.worker_thread
4.90 ± 6% -0.8 4.05 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
4.90 ± 6% -0.8 4.05 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.fsync
4.90 ± 6% -0.8 4.05 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
4.90 ± 6% -0.8 4.05 ± 3% perf-profile.calltrace.cycles-pp.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.90 ± 6% -0.8 4.05 ± 3% perf-profile.calltrace.cycles-pp.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
5.84 ± 2% -0.8 5.01 ± 4% perf-profile.calltrace.cycles-pp.write_cache_pages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
1.15 ± 7% -0.8 0.36 ±100% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.xfs_ilock
4.38 ± 7% -0.8 3.59 ± 3% perf-profile.calltrace.cycles-pp.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64
5.17 ± 2% -0.8 4.39 ± 5% perf-profile.calltrace.cycles-pp.xfs_do_writepage.write_cache_pages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
4.89 ± 4% -0.7 4.18 ± 5% perf-profile.calltrace.cycles-pp.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request.submit_bio
1.38 ± 9% -0.7 0.68 ± 22% perf-profile.calltrace.cycles-pp.down_write.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write
1.38 ± 9% -0.7 0.68 ± 22% perf-profile.calltrace.cycles-pp.call_rwsem_down_write_failed.down_write.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev
1.38 ± 9% -0.7 0.68 ± 22% perf-profile.calltrace.cycles-pp.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.xfs_ilock.xfs_file_buffered_aio_write
1.38 ± 9% -0.7 0.68 ± 22% perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
4.01 ± 3% -0.6 3.40 ± 5% perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.xfs_add_to_ioend.xfs_do_writepage
4.01 ± 3% -0.6 3.40 ± 5% perf-profile.calltrace.cycles-pp.submit_bio.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages.xfs_vm_writepages
4.01 ± 3% -0.6 3.40 ± 5% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages
3.99 ± 3% -0.6 3.38 ± 5% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.xfs_add_to_ioend
4.11 ± 3% -0.6 3.51 ± 4% perf-profile.calltrace.cycles-pp.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages.xfs_vm_writepages.do_writepages
3.97 ± 5% -0.5 3.43 ± 4% perf-profile.calltrace.cycles-pp.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
3.97 ± 5% -0.5 3.43 ± 4% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages
3.98 ± 5% -0.5 3.44 ± 4% perf-profile.calltrace.cycles-pp.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
3.96 ± 5% -0.5 3.43 ± 4% perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages
3.92 ± 5% -0.5 3.40 ± 4% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.xfs_submit_ioend
2.89 ± 6% -0.5 2.38 ± 6% perf-profile.calltrace.cycles-pp.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
2.77 ± 10% -0.5 2.28 ± 2% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync
2.77 ± 10% -0.5 2.28 ± 2% perf-profile.calltrace.cycles-pp.nfs_writepages.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file_fsync
2.77 ± 10% -0.5 2.28 ± 2% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync
2.74 ± 10% -0.5 2.25 ± 2% perf-profile.calltrace.cycles-pp.write_cache_pages.nfs_writepages.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range
3.12 ± 7% -0.5 2.64 ± 4% perf-profile.calltrace.cycles-pp.nfs_write_begin.generic_perform_write.nfs_file_write.__vfs_write.vfs_write
2.93 ± 7% -0.5 2.47 ± 3% perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write.__vfs_write
3.37 ± 4% -0.5 2.91 ± 4% perf-profile.calltrace.cycles-pp.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.vfs_write
2.51 ± 4% -0.4 2.06 ± 7% perf-profile.calltrace.cycles-pp.clear_page_erms.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec
3.12 ± 5% -0.4 2.68 ± 7% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request
2.91 ± 5% -0.4 2.48 ± 3% perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
3.02 ± 5% -0.4 2.59 ± 7% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request
2.59 ± 4% -0.4 2.18 ± 5% perf-profile.calltrace.cycles-pp.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write
2.84 ± 7% -0.4 2.43 ± 3% perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write
3.27 -0.4 2.86 ± 6% perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
3.20 -0.4 2.79 ± 6% perf-profile.calltrace.cycles-pp.memcpy_erms.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
2.85 ± 5% -0.4 2.44 ± 3% perf-profile.calltrace.cycles-pp.memcpy_erms._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
2.50 ± 4% -0.4 2.14 ± 7% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
2.41 ± 4% -0.4 2.06 ± 7% perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
2.02 ± 11% -0.3 1.67 ± 3% perf-profile.calltrace.cycles-pp.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writepages.__filemap_fdatawrite_range
0.60 ± 8% -0.3 0.26 ±100% perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
1.96 ± 11% -0.3 1.62 ± 3% perf-profile.calltrace.cycles-pp.nfs_do_writepage.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writepages
1.55 ± 2% -0.3 1.22 ± 5% perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
2.35 ± 4% -0.3 2.03 ± 7% perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
1.60 ± 2% -0.3 1.32 ± 4% perf-profile.calltrace.cycles-pp.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync
1.49 ± 6% -0.2 1.24 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request
1.03 ± 6% -0.2 0.80 ± 7% perf-profile.calltrace.cycles-pp.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
0.97 ± 7% -0.2 0.74 ± 9% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
1.45 ± 6% -0.2 1.23 ± 4% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write
0.97 ± 4% -0.2 0.77 ± 3% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
1.22 ± 6% -0.2 1.03 ± 6% perf-profile.calltrace.cycles-pp.nfs_commit_release_pages.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work
1.22 ± 6% -0.2 1.03 ± 5% perf-profile.calltrace.cycles-pp.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work.worker_thread
1.25 ± 5% -0.2 1.06 ± 8% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
1.06 ± 2% -0.2 0.87 ± 4% perf-profile.calltrace.cycles-pp.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync
1.27 ± 5% -0.2 1.11 ± 6% perf-profile.calltrace.cycles-pp.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
0.78 ± 8% -0.2 0.63 ± 6% perf-profile.calltrace.cycles-pp.__set_page_dirty_nobuffers.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write
1.11 ± 7% -0.1 0.99 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
0.68 -0.1 0.57 ± 5% perf-profile.calltrace.cycles-pp.nfs_create_request.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write
1.04 ± 7% -0.1 0.93 ± 5% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion
1.02 ± 8% -0.1 0.92 ± 4% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback
0.71 ± 2% -0.1 0.60 ± 8% perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range
0.72 ± 2% -0.1 0.61 ± 8% perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_and_wait_range
0.72 -0.1 0.62 ± 7% perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_fsync
0.93 ± 5% -0.1 0.83 ± 5% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write
0.73 -0.1 0.67 ± 5% perf-profile.calltrace.cycles-pp.end_page_writeback.xfs_destroy_ioend.process_one_work.worker_thread.kthread
4.86 ± 6% +0.5 5.39 perf-profile.calltrace.cycles-pp.svc_recv.nfsd.kthread.ret_from_fork
0.00 +0.5 0.54 ± 3% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.svc_recv.nfsd.kthread
0.00 +0.6 0.59 ± 6% perf-profile.calltrace.cycles-pp.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
0.00 +0.6 0.63 ± 6% perf-profile.calltrace.cycles-pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
0.00 +0.7 0.65 ± 4% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.svc_recv.nfsd.kthread.ret_from_fork
0.00 +0.7 0.74 ± 8% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
0.00 +0.9 0.86 ± 8% perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages
9.98 ± 4% +2.5 12.44 ± 4% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
9.61 ± 4% +2.5 12.15 ± 4% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
47.02 ± 3% +2.7 49.68 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64
0.00 +3.5 3.54 ± 3% perf-profile.calltrace.cycles-pp.memcpy_erms.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg
0.00 +3.6 3.56 ± 3% perf-profile.calltrace.cycles-pp.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
0.00 +3.7 3.67 ± 3% perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages
2.29 ± 6% +3.8 6.05 ± 3% perf-profile.calltrace.cycles-pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_from_fork
2.28 ± 6% +3.8 6.05 ± 3% perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.kthread
1.86 ± 3% +3.8 5.64 ± 4% perf-profile.calltrace.cycles-pp.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread
1.85 ± 3% +3.8 5.64 ± 4% perf-profile.calltrace.cycles-pp.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work
1.80 ± 4% +3.8 5.60 ± 4% perf-profile.calltrace.cycles-pp.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule
1.79 ± 3% +3.8 5.60 ± 4% perf-profile.calltrace.cycles-pp.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute
0.00 +5.5 5.47 ± 4% perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request
0.00 +5.6 5.56 ± 4% perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit
0.00 +5.6 5.57 ± 4% perf-profile.calltrace.cycles-pp.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit
20.37 -3.3 17.04 ± 5% perf-profile.children.cycles-pp.svc_process_common
20.38 -3.3 17.04 ± 5% perf-profile.children.cycles-pp.svc_process
20.27 -3.3 16.95 ± 5% perf-profile.children.cycles-pp.nfsd_dispatch
20.21 -3.3 16.91 ± 5% perf-profile.children.cycles-pp.nfsd4_proc_compound
26.02 -3.1 22.91 ± 4% perf-profile.children.cycles-pp.nfsd
15.88 ± 5% -1.9 13.95 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
15.87 ± 5% -1.9 13.95 ± 4% perf-profile.children.cycles-pp.do_syscall_64
12.61 ± 3% -1.9 10.73 ± 3% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
12.60 ± 3% -1.9 10.73 ± 3% perf-profile.children.cycles-pp.do_writepages
9.04 ± 2% -1.9 7.18 ± 7% perf-profile.children.cycles-pp.nfsd_vfs_write
9.02 ± 2% -1.9 7.17 ± 7% perf-profile.children.cycles-pp.do_iter_readv_writev
9.02 ± 2% -1.9 7.16 ± 7% perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
9.03 ± 2% -1.9 7.18 ± 7% perf-profile.children.cycles-pp.do_iter_write
9.07 ± 2% -1.9 7.22 ± 7% perf-profile.children.cycles-pp.nfsd4_write
2.21 ± 4% -1.8 0.41 ± 4% perf-profile.children.cycles-pp.inet_sendpage
2.17 ± 4% -1.8 0.40 ± 5% perf-profile.children.cycles-pp.tcp_sendpage
10.50 ± 2% -1.4 9.06 ± 4% perf-profile.children.cycles-pp.xfs_file_fsync
10.51 ± 2% -1.4 9.08 ± 4% perf-profile.children.cycles-pp.nfsd_commit
10.45 ± 2% -1.4 9.02 ± 4% perf-profile.children.cycles-pp.file_write_and_wait_range
9.82 ± 3% -1.4 8.45 ± 3% perf-profile.children.cycles-pp.xfs_vm_writepages
8.59 ± 4% -1.3 7.27 ± 4% perf-profile.children.cycles-pp.write_cache_pages
8.02 ± 4% -1.2 6.87 ± 4% perf-profile.children.cycles-pp.submit_bio
8.02 ± 4% -1.2 6.87 ± 4% perf-profile.children.cycles-pp.generic_make_request
8.00 ± 4% -1.1 6.85 ± 4% perf-profile.children.cycles-pp.brd_make_request
7.94 ± 4% -1.1 6.79 ± 4% perf-profile.children.cycles-pp.brd_do_bvec
10.63 ± 5% -1.1 9.49 ± 4% perf-profile.children.cycles-pp.write
7.55 -1.1 6.41 ± 5% perf-profile.children.cycles-pp.iomap_apply
7.54 -1.1 6.41 ± 6% perf-profile.children.cycles-pp.iomap_file_buffered_write
10.28 ± 5% -1.1 9.16 ± 4% perf-profile.children.cycles-pp.ksys_write
7.50 -1.1 6.37 ± 6% perf-profile.children.cycles-pp.iomap_write_actor
4.68 ± 5% -1.1 3.56 ± 2% perf-profile.children.cycles-pp.inet_recvmsg
4.68 ± 5% -1.1 3.56 ± 2% perf-profile.children.cycles-pp.tcp_recvmsg
10.24 ± 5% -1.1 9.12 ± 4% perf-profile.children.cycles-pp.vfs_write
10.08 ± 5% -1.1 8.98 ± 4% perf-profile.children.cycles-pp.__vfs_write
10.05 ± 5% -1.1 8.95 ± 4% perf-profile.children.cycles-pp.nfs_file_write
5.51 ± 5% -1.1 4.43 ± 5% perf-profile.children.cycles-pp.rpc_free_task
9.82 ± 5% -1.1 8.74 ± 4% perf-profile.children.cycles-pp.generic_perform_write
1.39 ± 5% -1.1 0.32 ± 4% perf-profile.children.cycles-pp.tcp_sendpage_locked
1.36 ± 5% -1.1 0.31 ± 5% perf-profile.children.cycles-pp.do_tcp_sendpages
5.31 ± 5% -1.0 4.27 ± 5% perf-profile.children.cycles-pp.rpc_async_release
4.39 ± 6% -1.0 3.40 ± 2% perf-profile.children.cycles-pp.svc_recvfrom
3.55 ± 5% -0.9 2.64 ± 2% perf-profile.children.cycles-pp.__skb_datagram_iter
3.55 ± 5% -0.9 2.64 ± 2% perf-profile.children.cycles-pp.skb_copy_datagram_iter
5.01 ± 6% -0.9 4.15 ± 3% perf-profile.children.cycles-pp.fsync
4.09 ± 7% -0.9 3.24 ± 5% perf-profile.children.cycles-pp.nfs_write_completion
4.90 ± 6% -0.8 4.05 ± 3% perf-profile.children.cycles-pp.__x64_sys_fsync
4.91 ± 6% -0.8 4.07 ± 3% perf-profile.children.cycles-pp.nfs_file_fsync
4.90 ± 6% -0.8 4.05 ± 3% perf-profile.children.cycles-pp.do_fsync
5.36 ± 4% -0.8 4.53 ± 5% perf-profile.children.cycles-pp.grab_cache_page_write_begin
4.39 ± 7% -0.8 3.60 ± 2% perf-profile.children.cycles-pp.filemap_write_and_wait_range
5.18 ± 2% -0.8 4.40 ± 5% perf-profile.children.cycles-pp.xfs_do_writepage
5.21 ± 4% -0.7 4.47 ± 5% perf-profile.children.cycles-pp.pagecache_get_page
4.90 ± 4% -0.7 4.19 ± 5% perf-profile.children.cycles-pp.brd_insert_page
1.38 ± 9% -0.7 0.68 ± 22% perf-profile.children.cycles-pp.xfs_ilock
1.41 ± 8% -0.7 0.71 ± 22% perf-profile.children.cycles-pp.call_rwsem_down_write_failed
1.41 ± 8% -0.7 0.71 ± 22% perf-profile.children.cycles-pp.rwsem_down_write_failed
1.44 ± 7% -0.7 0.74 ± 21% perf-profile.children.cycles-pp.down_write
4.12 ± 3% -0.6 3.52 ± 4% perf-profile.children.cycles-pp.xfs_add_to_ioend
1.18 ± 6% -0.6 0.61 ± 21% perf-profile.children.cycles-pp.rwsem_spin_on_owner
3.98 ± 5% -0.5 3.44 ± 4% perf-profile.children.cycles-pp.xfs_submit_ioend
6.25 -0.5 5.72 ± 4% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
2.89 ± 6% -0.5 2.38 ± 6% perf-profile.children.cycles-pp.nfs_end_page_writeback
2.78 ± 10% -0.5 2.29 ± 2% perf-profile.children.cycles-pp.nfs_writepages
3.12 ± 7% -0.5 2.64 ± 4% perf-profile.children.cycles-pp.nfs_write_begin
3.37 ± 4% -0.5 2.91 ± 4% perf-profile.children.cycles-pp.nfs_write_end
2.54 ± 4% -0.5 2.08 ± 7% perf-profile.children.cycles-pp.clear_page_erms
2.92 ± 5% -0.4 2.48 ± 3% perf-profile.children.cycles-pp._copy_to_iter
2.60 ± 4% -0.4 2.18 ± 5% perf-profile.children.cycles-pp.nfs_updatepage
2.71 ± 4% -0.4 2.29 ± 6% perf-profile.children.cycles-pp.add_to_page_cache_lru
0.75 ± 3% -0.4 0.37 ± 6% perf-profile.children.cycles-pp.release_sock
2.51 ± 4% -0.4 2.15 ± 7% perf-profile.children.cycles-pp.iomap_write_begin
2.02 ± 10% -0.3 1.67 ± 3% perf-profile.children.cycles-pp.nfs_writepages_callback
2.23 ± 2% -0.3 1.89 ± 4% perf-profile.children.cycles-pp.__filemap_fdatawait_range
0.82 ± 7% -0.3 0.47 ± 7% perf-profile.children.cycles-pp.__tcp_push_pending_frames
1.96 ± 11% -0.3 1.63 ± 3% perf-profile.children.cycles-pp.nfs_do_writepage
1.55 ± 2% -0.3 1.22 ± 5% perf-profile.children.cycles-pp.iomap_write_end
0.78 ± 7% -0.3 0.48 ± 6% perf-profile.children.cycles-pp.svc_send
1.78 ± 3% -0.3 1.48 ± 4% perf-profile.children.cycles-pp.end_page_writeback
1.73 ± 3% -0.3 1.45 ± 4% perf-profile.children.cycles-pp.test_clear_page_writeback
0.90 ± 5% -0.3 0.63 ± 8% perf-profile.children.cycles-pp.xas_load
0.35 ± 14% -0.2 0.10 ± 14% perf-profile.children.cycles-pp.simple_copy_to_iter
1.78 ± 4% -0.2 1.54 ± 7% perf-profile.children.cycles-pp.__wake_up_common
0.43 ± 6% -0.2 0.19 ± 7% perf-profile.children.cycles-pp.lock_sock_nested
1.41 ± 5% -0.2 1.18 ± 4% perf-profile.children.cycles-pp.nfs_commit_release
1.41 ± 5% -0.2 1.18 ± 4% perf-profile.children.cycles-pp.nfs_commit_release_pages
1.95 ± 4% -0.2 1.72 ± 5% perf-profile.children.cycles-pp.try_to_wake_up
0.64 ± 8% -0.2 0.41 ± 4% perf-profile.children.cycles-pp.svc_tcp_sendto
0.63 ± 8% -0.2 0.41 ± 5% perf-profile.children.cycles-pp.svc_sendto
0.62 ± 8% -0.2 0.41 ± 5% perf-profile.children.cycles-pp.svc_send_common
0.62 ± 8% -0.2 0.41 ± 5% perf-profile.children.cycles-pp.kernel_sendpage
1.65 ± 5% -0.2 1.44 ± 7% perf-profile.children.cycles-pp.autoremove_wake_function
1.34 ± 3% -0.2 1.14 ± 3% perf-profile.children.cycles-pp.wait_on_page_bit_common
0.98 ± 4% -0.2 0.79 ± 2% perf-profile.children.cycles-pp.poll_idle
0.26 ± 8% -0.2 0.08 ± 13% perf-profile.children.cycles-pp._raw_spin_lock_bh
1.59 ± 5% -0.2 1.42 ± 5% perf-profile.children.cycles-pp.schedule
0.79 ± 5% -0.2 0.62 ± 4% perf-profile.children.cycles-pp.tcp_v6_do_rcv
0.64 ± 5% -0.2 0.47 ± 8% perf-profile.children.cycles-pp.tcp_v6_rcv
0.34 ± 12% -0.2 0.18 ± 20% perf-profile.children.cycles-pp.xas_start
0.24 ± 26% -0.2 0.08 ± 40% perf-profile.children.cycles-pp.osq_lock
0.76 ± 5% -0.2 0.61 ± 5% perf-profile.children.cycles-pp.tcp_rcv_established
0.49 ± 26% -0.2 0.34 ± 14% perf-profile.children.cycles-pp.nfs_request_add_commit_list
0.66 ± 5% -0.1 0.52 ± 8% perf-profile.children.cycles-pp.ip6_input
0.66 ± 4% -0.1 0.51 ± 8% perf-profile.children.cycles-pp.ip6_input_finish
0.65 ± 4% -0.1 0.51 ± 9% perf-profile.children.cycles-pp.ip6_protocol_deliver_rcu
0.79 ± 8% -0.1 0.64 ± 7% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
0.83 ± 6% -0.1 0.69 ± 8% perf-profile.children.cycles-pp.__nfs_commit_inode
0.42 ± 4% -0.1 0.29 ± 5% perf-profile.children.cycles-pp.xs_stream_data_receive_workfn
0.42 ± 4% -0.1 0.29 ± 5% perf-profile.children.cycles-pp.xs_stream_data_receive
0.70 ± 4% -0.1 0.56 ± 6% perf-profile.children.cycles-pp.percpu_counter_add_batch
0.71 ± 4% -0.1 0.58 ± 8% perf-profile.children.cycles-pp.ipv6_rcv
0.86 ± 6% -0.1 0.73 ± 6% perf-profile.children.cycles-pp.__local_bh_enable_ip
0.78 ± 5% -0.1 0.65 ± 8% perf-profile.children.cycles-pp.process_backlog
0.45 ± 14% -0.1 0.32 ± 14% perf-profile.children.cycles-pp.__mutex_lock
1.06 ± 4% -0.1 0.93 ± 7% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.86 ± 6% -0.1 0.73 ± 9% perf-profile.children.cycles-pp.pagevec_lru_move_fn
0.75 ± 5% -0.1 0.63 ± 8% perf-profile.children.cycles-pp.__netif_receive_skb_one_core
0.61 ± 8% -0.1 0.49 ± 9% perf-profile.children.cycles-pp.iomap_set_page_dirty
0.94 ± 5% -0.1 0.81 ± 8% perf-profile.children.cycles-pp.__lru_cache_add
0.82 ± 6% -0.1 0.70 ± 2% perf-profile.children.cycles-pp.clear_page_dirty_for_io
0.80 ± 6% -0.1 0.68 ± 7% perf-profile.children.cycles-pp.net_rx_action
0.94 ± 2% -0.1 0.82 ± 3% perf-profile.children.cycles-pp.io_schedule
0.81 ± 7% -0.1 0.70 ± 7% perf-profile.children.cycles-pp.do_softirq_own_stack
0.69 -0.1 0.58 ± 5% perf-profile.children.cycles-pp.nfs_create_request
0.62 ± 4% -0.1 0.51 ± 5% perf-profile.children.cycles-pp.__pagevec_lru_add_fn
0.32 ± 4% -0.1 0.21 ± 11% perf-profile.children.cycles-pp.xs_sock_recvmsg
0.49 ± 12% -0.1 0.38 ± 7% perf-profile.children.cycles-pp.sched_ttwu_pending
0.82 ± 6% -0.1 0.72 ± 7% perf-profile.children.cycles-pp.do_softirq
0.32 ± 10% -0.1 0.22 ± 7% perf-profile.children.cycles-pp.xfs_map_blocks
0.40 ± 13% -0.1 0.30 ± 12% perf-profile.children.cycles-pp.mutex_spin_on_owner
0.47 ± 2% -0.1 0.38 ± 5% perf-profile.children.cycles-pp.kmem_cache_alloc
0.26 ± 4% -0.1 0.17 ± 8% perf-profile.children.cycles-pp.__wake_up_common_lock
0.52 ± 6% -0.1 0.43 ± 4% perf-profile.children.cycles-pp.account_page_dirtied
0.22 ± 16% -0.1 0.13 ± 9% perf-profile.children.cycles-pp.__check_object_size
0.50 ± 8% -0.1 0.42 ± 7% perf-profile.children.cycles-pp.iomap_set_range_uptodate
0.41 ± 3% -0.1 0.33 ± 14% perf-profile.children.cycles-pp.__set_page_dirty
0.31 ± 11% -0.1 0.24 ± 11% perf-profile.children.cycles-pp.nfs_io_completion_release
0.41 ± 8% -0.1 0.33 ± 7% perf-profile.children.cycles-pp.nfs_inode_remove_request
0.25 ± 9% -0.1 0.18 ± 8% perf-profile.children.cycles-pp.xfs_iomap_write_allocate
0.33 ± 8% -0.1 0.26 ± 4% perf-profile.children.cycles-pp.___might_sleep
0.25 ± 5% -0.1 0.18 ± 15% perf-profile.children.cycles-pp.__xa_set_mark
0.31 ± 9% -0.1 0.24 ± 5% perf-profile.children.cycles-pp.nfs_scan_commit_list
0.25 ± 5% -0.1 0.18 ± 8% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.45 ± 6% -0.1 0.39 ± 4% perf-profile.children.cycles-pp.nfs_lock_and_join_requests
0.50 ± 4% -0.1 0.45 ± 5% perf-profile.children.cycles-pp.nfs_page_group_destroy
0.19 ± 11% -0.1 0.14 ± 8% perf-profile.children.cycles-pp.xfs_bmap_btalloc
0.21 ± 2% -0.1 0.15 ± 14% perf-profile.children.cycles-pp.__lock_sock
0.17 ± 8% -0.1 0.12 ± 7% perf-profile.children.cycles-pp.xfs_alloc_ag_vextent
0.18 ± 9% -0.1 0.13 ± 6% perf-profile.children.cycles-pp.xfs_alloc_vextent
0.20 ± 12% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.xfs_bmapi_write
0.30 ± 9% -0.0 0.25 ± 5% perf-profile.children.cycles-pp.select_task_rq_fair
0.16 ± 8% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.xfs_alloc_ag_vextent_near
0.16 ± 15% -0.0 0.12 ± 17% perf-profile.children.cycles-pp.__generic_write_end
0.48 ± 2% -0.0 0.44 ± 7% perf-profile.children.cycles-pp.dequeue_task_fair
0.31 ± 2% -0.0 0.27 ± 10% perf-profile.children.cycles-pp.release_pages
0.20 ± 4% -0.0 0.17 ± 10% perf-profile.children.cycles-pp.nfs_initiate_commit
0.17 ± 10% -0.0 0.13 ± 5% perf-profile.children.cycles-pp.__might_sleep
0.29 ± 6% -0.0 0.26 ± 4% perf-profile.children.cycles-pp.dec_zone_page_state
0.23 ± 7% -0.0 0.20 ± 7% perf-profile.children.cycles-pp.__pagevec_release
0.19 ± 9% -0.0 0.15 ± 9% perf-profile.children.cycles-pp.nfs_get_lock_context
0.17 ± 13% -0.0 0.13 ± 8% perf-profile.children.cycles-pp.check_preempt_curr
0.09 ± 17% -0.0 0.06 ± 20% perf-profile.children.cycles-pp.memset_erms
0.14 ± 8% -0.0 0.11 ± 6% perf-profile.children.cycles-pp.nfs_request_remove_commit_list
0.07 ± 10% -0.0 0.04 ± 58% perf-profile.children.cycles-pp.xfs_defer_finish_noroll
0.15 ± 5% -0.0 0.12 ± 6% perf-profile.children.cycles-pp.__nfs_find_lock_context
0.19 ± 4% -0.0 0.16 ± 11% perf-profile.children.cycles-pp.mem_cgroup_try_charge
0.17 ± 8% -0.0 0.15 ± 10% perf-profile.children.cycles-pp.vfs_create
0.16 ± 2% -0.0 0.14 ± 8% perf-profile.children.cycles-pp.__fprop_inc_percpu_max
0.16 ± 8% -0.0 0.13 ± 8% perf-profile.children.cycles-pp.xfs_create
0.10 ± 4% -0.0 0.07 ± 17% perf-profile.children.cycles-pp.xfs_file_aio_write_checks
0.11 ± 10% -0.0 0.08 ± 15% perf-profile.children.cycles-pp.nfs_pageio_doio
0.11 ± 10% -0.0 0.08 ± 15% perf-profile.children.cycles-pp.nfs_generic_pg_pgios
0.19 ± 4% -0.0 0.17 ± 7% perf-profile.children.cycles-pp.update_rq_clock
0.15 ± 4% -0.0 0.13 ± 5% perf-profile.children.cycles-pp.__xfs_trans_commit
0.08 ± 15% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
0.06 ± 15% +0.0 0.07 ± 14% perf-profile.children.cycles-pp.selinux_ip_postroute
0.08 ± 16% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.nf_hook_slow
0.04 ± 60% +0.0 0.08 ± 19% perf-profile.children.cycles-pp.__inc_numa_state
0.03 ±100% +0.0 0.07 ± 12% perf-profile.children.cycles-pp.get_task_policy
0.41 ± 6% +0.0 0.46 perf-profile.children.cycles-pp.__release_sock
0.14 ± 11% +0.1 0.19 ± 8% perf-profile.children.cycles-pp.__list_add_valid
0.09 ± 17% +0.1 0.14 ± 10% perf-profile.children.cycles-pp.svc_xprt_do_enqueue
0.14 ± 10% +0.1 0.20 ± 2% perf-profile.children.cycles-pp.tcp_clean_rtx_queue
0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.free_unref_page_commit
0.10 ± 11% +0.1 0.17 ± 6% perf-profile.children.cycles-pp.iov_iter_advance
0.17 ± 9% +0.1 0.24 ± 2% perf-profile.children.cycles-pp.tcp_ack
0.11 ± 21% +0.1 0.18 ± 2% perf-profile.children.cycles-pp.schedule_timeout
0.00 +0.1 0.08 ± 17% perf-profile.children.cycles-pp.free_one_page
0.23 ± 12% +0.1 0.32 ± 4% perf-profile.children.cycles-pp.__kfree_skb
0.10 ± 4% +0.1 0.19 ± 6% perf-profile.children.cycles-pp.xas_create
0.12 ± 17% +0.1 0.24 ± 4% perf-profile.children.cycles-pp.skb_release_data
0.00 +0.1 0.14 ± 5% perf-profile.children.cycles-pp.__free_pages_ok
0.00 +0.2 0.21 ± 9% perf-profile.children.cycles-pp.skb_page_frag_refill
0.00 +0.2 0.22 ± 9% perf-profile.children.cycles-pp.sk_page_frag_refill
0.00 +0.2 0.25 ± 5% perf-profile.children.cycles-pp.__sk_flush_backlog
0.03 ±100% +0.4 0.39 ± 7% perf-profile.children.cycles-pp.free_pcppages_bulk
0.07 ± 29% +0.5 0.54 ± 6% perf-profile.children.cycles-pp.free_unref_page
4.86 ± 6% +0.5 5.40 ± 2% perf-profile.children.cycles-pp.svc_recv
9.98 ± 4% +2.5 12.44 ± 4% perf-profile.children.cycles-pp.worker_thread
9.61 ± 4% +2.5 12.15 ± 4% perf-profile.children.cycles-pp.process_one_work
47.02 ± 3% +2.7 49.68 ± 4% perf-profile.children.cycles-pp.secondary_startup_64
47.02 ± 3% +2.7 49.68 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry
47.02 ± 3% +2.7 49.69 ± 4% perf-profile.children.cycles-pp.do_idle
6.12 ± 2% +2.8 8.88 ± 3% perf-profile.children.cycles-pp.memcpy_erms
0.00 +3.6 3.58 ± 3% perf-profile.children.cycles-pp.memcpy_from_page
0.00 +3.7 3.68 ± 3% perf-profile.children.cycles-pp._copy_from_iter_full
2.43 ± 6% +3.8 6.20 ± 3% perf-profile.children.cycles-pp.__rpc_execute
2.29 ± 6% +3.8 6.05 ± 3% perf-profile.children.cycles-pp.rpc_async_schedule
1.93 ± 3% +3.8 5.71 ± 4% perf-profile.children.cycles-pp.call_transmit
1.92 ± 4% +3.8 5.71 ± 4% perf-profile.children.cycles-pp.xprt_transmit
1.87 ± 4% +3.8 5.67 ± 4% perf-profile.children.cycles-pp.xs_tcp_send_request
1.86 ± 4% +3.8 5.67 ± 4% perf-profile.children.cycles-pp.xs_sendpages
0.21 ± 7% +5.3 5.54 ± 4% perf-profile.children.cycles-pp.tcp_sendmsg_locked
0.24 ± 8% +5.4 5.63 ± 4% perf-profile.children.cycles-pp.tcp_sendmsg
0.25 ± 9% +5.4 5.64 ± 4% perf-profile.children.cycles-pp.sock_sendmsg
1.17 ± 7% -0.6 0.61 ± 21% perf-profile.self.cycles-pp.rwsem_spin_on_owner
2.50 ± 4% -0.5 2.05 ± 7% perf-profile.self.cycles-pp.clear_page_erms
2.46 ± 4% -0.4 2.06 ± 5% perf-profile.self.cycles-pp.brd_do_bvec
0.22 ± 10% -0.2 0.03 ±100% perf-profile.self.cycles-pp.__skb_datagram_iter
1.53 ± 6% -0.2 1.34 ± 3% perf-profile.self.cycles-pp._raw_spin_lock
0.91 ± 4% -0.2 0.74 ± 2% perf-profile.self.cycles-pp.poll_idle
0.24 ± 25% -0.2 0.08 ± 40% perf-profile.self.cycles-pp.osq_lock
0.74 ± 14% -0.2 0.59 ± 10% perf-profile.self.cycles-pp.nfs_do_writepage
0.32 ± 13% -0.1 0.17 ± 18% perf-profile.self.cycles-pp.xas_start
0.97 ± 3% -0.1 0.84 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.57 ± 6% -0.1 0.45 ± 9% perf-profile.self.cycles-pp.xas_load
0.55 ± 10% -0.1 0.43 ± 7% perf-profile.self.cycles-pp.__schedule
0.14 ± 12% -0.1 0.03 ±100% perf-profile.self.cycles-pp._raw_spin_lock_bh
0.53 ± 8% -0.1 0.43 ± 9% perf-profile.self.cycles-pp.percpu_counter_add_batch
0.40 ± 13% -0.1 0.30 ± 12% perf-profile.self.cycles-pp.mutex_spin_on_owner
0.33 ± 8% -0.1 0.24 ± 12% perf-profile.self.cycles-pp.nfs_end_page_writeback
0.74 ± 6% -0.1 0.65 ± 8% perf-profile.self.cycles-pp.nfs_updatepage
0.40 ± 8% -0.1 0.31 ± 11% perf-profile.self.cycles-pp.__test_set_page_writeback
0.50 ± 8% -0.1 0.41 ± 7% perf-profile.self.cycles-pp.iomap_set_range_uptodate
0.56 ± 5% -0.1 0.48 ± 8% perf-profile.self.cycles-pp.try_to_wake_up
0.38 ± 8% -0.1 0.30 ± 5% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.28 ± 9% -0.1 0.20 ± 6% perf-profile.self.cycles-pp.iomap_write_end
0.45 ± 7% -0.1 0.38 ± 6% perf-profile.self.cycles-pp.__pagevec_lru_add_fn
0.15 ± 14% -0.1 0.08 ± 27% perf-profile.self.cycles-pp.add_to_page_cache_lru
0.43 ± 6% -0.1 0.36 perf-profile.self.cycles-pp.clear_page_dirty_for_io
0.51 ± 8% -0.1 0.45 ± 7% perf-profile.self.cycles-pp.test_clear_page_writeback
0.20 ± 9% -0.1 0.14 ± 11% perf-profile.self.cycles-pp.xas_store
0.31 ± 8% -0.1 0.25 ± 4% perf-profile.self.cycles-pp.___might_sleep
0.15 ± 20% -0.1 0.09 ± 12% perf-profile.self.cycles-pp.__check_object_size
0.22 ± 6% -0.1 0.17 ± 10% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.08 ± 11% -0.0 0.03 ±100% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.28 ± 4% -0.0 0.24 ± 10% perf-profile.self.cycles-pp.release_pages
0.27 ± 8% -0.0 0.23 ± 2% perf-profile.self.cycles-pp.dec_zone_page_state
0.19 ± 6% -0.0 0.15 ± 12% perf-profile.self.cycles-pp.kmem_cache_alloc
0.15 ± 10% -0.0 0.11 ± 7% perf-profile.self.cycles-pp.__might_sleep
0.25 ± 10% -0.0 0.21 ± 6% perf-profile.self.cycles-pp.wait_on_page_bit_common
0.07 ± 12% -0.0 0.03 ±100% perf-profile.self.cycles-pp.___slab_alloc
0.23 ± 7% -0.0 0.20 ± 10% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.18 ± 5% -0.0 0.15 ± 2% perf-profile.self.cycles-pp.select_task_rq_fair
0.14 ± 5% -0.0 0.11 ± 17% perf-profile.self.cycles-pp.alloc_pages_current
0.09 ± 16% -0.0 0.06 ± 20% perf-profile.self.cycles-pp.memset_erms
0.10 ± 8% -0.0 0.07 perf-profile.self.cycles-pp._cond_resched
0.15 ± 4% -0.0 0.12 ± 6% perf-profile.self.cycles-pp.__nfs_find_lock_context
0.11 ± 13% -0.0 0.09 ± 14% perf-profile.self.cycles-pp.nfs_request_add_commit_list_locked
0.09 ± 4% -0.0 0.07 perf-profile.self.cycles-pp.nfs_request_remove_commit_list
0.08 ± 10% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
0.08 ± 10% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.nfs_request_add_commit_list
0.10 ± 5% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.___perf_sw_event
0.08 ± 13% +0.0 0.11 ± 10% perf-profile.self.cycles-pp.mem_cgroup_commit_charge
0.03 ±100% +0.0 0.07 ± 12% perf-profile.self.cycles-pp.run_timer_softirq
0.03 ±100% +0.0 0.07 ± 12% perf-profile.self.cycles-pp.get_task_policy
0.03 ±100% +0.0 0.07 ± 17% perf-profile.self.cycles-pp.__inc_numa_state
0.03 ±102% +0.1 0.08 ± 8% perf-profile.self.cycles-pp.svc_recv
0.11 ± 13% +0.1 0.17 ± 12% perf-profile.self.cycles-pp.__list_add_valid
0.09 ± 14% +0.1 0.16 ± 9% perf-profile.self.cycles-pp.iov_iter_advance
0.06 ± 13% +0.1 0.15 ± 8% perf-profile.self.cycles-pp.xas_create
0.00 +0.1 0.15 ± 14% perf-profile.self.cycles-pp.tcp_sendmsg_locked
0.00 +0.2 0.20 ± 11% perf-profile.self.cycles-pp.free_pcppages_bulk
0.04 ± 60% +0.2 0.28 ± 7% perf-profile.self.cycles-pp.svc_tcp_recvfrom
1.25 +0.3 1.52 ± 5% perf-profile.self.cycles-pp.get_page_from_freelist
6.04 ± 2% +2.7 8.74 ± 3% perf-profile.self.cycles-pp.memcpy_erms
fsmark.time.percent_of_cpu_this_job_got
320 +-+-------------------------------------------------------------------+
| |
300 +-++. + |
|. +.. .+. +.. +.. +. .. + .+.|
280 +-+ +.+. +.. : + .+..+. .+. .+. .+.+.. .. + +. |
| : + + +. +. +. .+ |
260 +-+ + + |
| |
240 +-+ |
| |
220 +-+ |
| O O O O O O O |
200 O-+ O O O O O O O O |
| O O O O O O O O O O |
180 +-+-------------------------------------------------------------------+
fsmark.time.elapsed_time
32 +-+O-------------------------------------------------------------------+
O O O O O O O O O O O O |
31 +-+ O O O O O O O O O O O O |
30 +-+ O |
| |
29 +-+ |
28 +-+ |
| |
27 +-+ |
26 +-+ .+ |
| + + +.+.. .+. + .+. .+. |
25 +-+ .. + .+ + : +..+.+..+. .. + +. +. +..|
24 +-+ .+.+..+.+ +. + + : .. + |
| +.+. + + |
23 +-+--------------------------------------------------------------------+
fsmark.time.elapsed_time.max
32 +-+O-------------------------------------------------------------------+
O O O O O O O O O O O O |
31 +-+ O O O O O O O O O O O O |
30 +-+ O |
| |
29 +-+ |
28 +-+ |
| |
27 +-+ |
26 +-+ .+ |
| + + +.+.. .+. + .+. .+. |
25 +-+ .. + .+ + : +..+.+..+. .. + +. +. +..|
24 +-+ .+.+..+.+ +. + + : .. + |
| +.+. + + |
23 +-+--------------------------------------------------------------------+
fsmark.files_per_sec
520 +-+-------------------------------------------------------------------+
|.. +.+. + +.. .+.. : .+. .+.. .+.+..+ |
500 +-+ +.. + + : +.+..+.+. +..+ .+.+. |
480 +-+ + + +.+. |
| |
460 +-+ |
| |
440 +-+ |
| |
420 +-+ |
400 +-+ O |
| O O O O O O O O O |
380 O-+ O O O O O O O O O |
| O O O O O O |
360 +-+-------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
Hi Trond,
On 5/20/2019 1:54 PM, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed a 16.0% improvement of fsmark.app_overhead due to commit:
>
>
> commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC: Convert socket page send code to use iov_iter()")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: fsmark
> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
> with following parameters:
>
> iterations: 1x
> nr_threads: 64t
> disk: 1BRD_48G
> fs: xfs
> fs2: nfsv4
> filesize: 4M
> test_size: 40G
> sync_method: fsyncBeforeClose
> cpufreq_governor: performance
>
> test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
> test-url: https://sourceforge.net/projects/fsmark/
>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
> =========================================================================================
> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
> gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-ep01/40G/fsmark
>
> commit:
> e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use iov_iter_kvec()")
> 0472e47660 ("SUNRPC: Convert socket page send code to use iov_iter()")
>
> e791f8e9380d945e 0472e476604998c127f3c80d291
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> :4 50% 2:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
> %stddev %change %stddev
> \ | \
> 15118573 ? 2% +16.0% 17538083 fsmark.app_overhead
> 510.93 -22.7% 395.12 fsmark.files_per_sec
> 24.90 +22.8% 30.57 fsmark.time.elapsed_time
> 24.90 +22.8% 30.57 fsmark.time.elapsed_time.max
> 288.00 ? 2% -27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got
> 70.03 ? 2% -11.3% 62.14 fsmark.time.system_time
> 4391964 -16.7% 3658341 meminfo.max_used_kB
> 6.10 ? 4% +1.9 7.97 ? 3% mpstat.cpu.all.iowait%
> 0.27 -0.0 0.24 ? 3% mpstat.cpu.all.soft%
> 13668070 ? 40% +118.0% 29801846 ? 19% numa-numastat.node0.local_node
> 13677774 ? 40% +117.9% 29810258 ? 19% numa-numastat.node0.numa_hit
> 5.70 ? 3% +32.1% 7.53 ? 3% iostat.cpu.iowait
> 16.42 ? 2% -5.8% 15.47 iostat.cpu.system
> 2.57 -4.1% 2.46 iostat.cpu.user
> 1406781 ? 2% -15.5% 1188498 vmstat.io.bo
> 251792 ? 3% -16.6% 209928 vmstat.system.cs
> 84841 -1.9% 83239 vmstat.system.in
> 97374502 ? 20% +66.1% 1.617e+08 ? 17% cpuidle.C1E.time
> 573934 ? 19% +44.6% 829662 ? 26% cpuidle.C1E.usage
> 5.892e+08 ? 8% +15.3% 6.796e+08 ? 2% cpuidle.C6.time
> 1968016 ? 3% -15.1% 1670867 ? 3% cpuidle.POLL.time
> 106420 ? 47% +86.2% 198108 ? 35% numa-meminfo.node0.Active
> 106037 ? 48% +86.2% 197395 ? 35% numa-meminfo.node0.Active(anon)
> 105052 ? 48% +86.6% 196037 ? 35% numa-meminfo.node0.AnonPages
> 212876 ? 24% -41.5% 124572 ? 56% numa-meminfo.node1.Active
> 211801 ? 24% -41.5% 123822 ? 56% numa-meminfo.node1.Active(anon)
> 208559 ? 24% -42.2% 120547 ? 57% numa-meminfo.node1.AnonPages
> 9955 +1.6% 10116 proc-vmstat.nr_kernel_stack
> 452.25 ? 59% +280.9% 1722 ?100% proc-vmstat.numa_hint_faults_local
> 33817303 +55.0% 52421773 ? 5% proc-vmstat.numa_hit
> 33804286 +55.0% 52408807 ? 5% proc-vmstat.numa_local
> 33923002 +81.8% 61663426 ? 5% proc-vmstat.pgalloc_normal
> 184765 +9.3% 201985 proc-vmstat.pgfault
> 12840986 +216.0% 40581327 ? 7% proc-vmstat.pgfree
> 31447 ? 11% -26.1% 23253 ? 13% sched_debug.cfs_rq:/.min_vruntime.max
> 4241 ? 3% -12.2% 3724 ? 11% sched_debug.cfs_rq:/.min_vruntime.stddev
> 20631 ? 11% -36.7% 13069 ? 29% sched_debug.cfs_rq:/.spread0.max
> 4238 ? 4% -12.1% 3724 ? 11% sched_debug.cfs_rq:/.spread0.stddev
> 497105 ? 19% -16.0% 417777 ? 4% sched_debug.cpu.avg_idle.avg
> 21199 ? 10% -12.0% 18650 ? 3% sched_debug.cpu.nr_load_updates.max
> 2229 ? 10% -15.0% 1895 ? 4% sched_debug.cpu.nr_load_updates.stddev
> 4.86 ? 5% -23.6% 3.72 ? 5% sched_debug.cpu.nr_uninterruptible.stddev
> 524.75 ? 2% -10.7% 468.50 turbostat.Avg_MHz
> 5.26 ? 41% -1.6 3.66 ? 2% turbostat.C1%
> 573633 ? 19% +44.6% 829267 ? 26% turbostat.C1E
> 8.53 ? 20% +3.4 11.88 ? 17% turbostat.C1E%
> 76.75 -6.4% 71.86 turbostat.CorWatt
> 2534071 ? 2% +16.2% 2943968 turbostat.IRQ
> 2.89 ? 24% +52.9% 4.42 ? 22% turbostat.Pkg%pc2
> 104.28 -4.6% 99.51 turbostat.PkgWatt
> 2289 +18.8% 2720 turbostat.SMI
> 26496 ? 47% +86.2% 49347 ? 35% numa-vmstat.node0.nr_active_anon
> 26251 ? 48% +86.7% 49008 ? 35% numa-vmstat.node0.nr_anon_pages
> 26496 ? 47% +86.2% 49347 ? 35% numa-vmstat.node0.nr_zone_active_anon
> 8272770 ? 35% +99.6% 16515977 ? 17% numa-vmstat.node0.numa_hit
> 8199681 ? 36% +100.9% 16474282 ? 17% numa-vmstat.node0.numa_local
> 52953 ? 24% -41.5% 30956 ? 56% numa-vmstat.node1.nr_active_anon
> 52144 ? 24% -42.2% 30136 ? 57% numa-vmstat.node1.nr_anon_pages
> 2899 ? 2% -19.4% 2336 ? 10% numa-vmstat.node1.nr_writeback
> 52953 ? 24% -41.5% 30956 ? 56% numa-vmstat.node1.nr_zone_active_anon
> 4549 ? 8% -18.9% 3689 ? 7% numa-vmstat.node1.nr_zone_write_pending
> 43.25 -1.6 41.69 perf-stat.i.cache-miss-rate%
> 286377 ? 2% -19.1% 231819 ? 2% perf-stat.i.context-switches
> 2.258e+10 ? 4% -16.2% 1.893e+10 ? 10% perf-stat.i.cpu-cycles
> 4084 -34.0% 2696 ? 2% perf-stat.i.cpu-migrations
> 21563675 ? 24% -30.8% 14929445 ? 5% perf-stat.i.dTLB-load-misses
> 1996341 ? 16% -25.7% 1482826 ? 6% perf-stat.i.dTLB-store-misses
> 2.313e+09 ? 4% -11.7% 2.042e+09 ? 6% perf-stat.i.dTLB-stores
> 75.93 -1.3 74.66 perf-stat.i.iTLB-load-miss-rate%
> 1561381 ? 3% -13.4% 1351731 perf-stat.i.iTLB-load-misses
> 549643 ? 4% -10.1% 494255 ? 3% perf-stat.i.iTLB-loads
> 6423 -10.0% 5779 perf-stat.i.minor-faults
> 31.72 ? 2% +4.4 36.14 perf-stat.i.node-load-miss-rate%
> 23492890 +23.5% 29020305 ? 3% perf-stat.i.node-load-misses
> 15.13 ? 8% +7.9 23.02 ? 3% perf-stat.i.node-store-miss-rate%
> 11517615 +71.3% 19734054 ? 2% perf-stat.i.node-store-misses
> 6423 -10.0% 5779 perf-stat.i.page-faults
> 16.88 ? 14% +33.4% 22.52 ? 23% perf-stat.overall.MPKI
> 46.67 -2.4 44.31 perf-stat.overall.cache-miss-rate%
> 31.96 +4.8 36.72 perf-stat.overall.node-load-miss-rate%
> 13.85 +8.0 21.87 perf-stat.overall.node-store-miss-rate%
> 2.266e+08 +2.8% 2.331e+08 ? 2% perf-stat.ps.cache-references
> 275634 ? 2% -18.5% 224718 ? 2% perf-stat.ps.context-switches
> 2.174e+10 ? 4% -15.6% 1.835e+10 ? 10% perf-stat.ps.cpu-cycles
> 3931 -33.5% 2614 ? 2% perf-stat.ps.cpu-migrations
> 20746040 ? 24% -30.2% 14476157 ? 5% perf-stat.ps.dTLB-load-misses
> 1921077 ? 16% -25.2% 1437750 ? 6% perf-stat.ps.dTLB-store-misses
> 2.227e+09 ? 4% -11.1% 1.979e+09 ? 6% perf-stat.ps.dTLB-stores
> 1503433 ? 3% -12.8% 1310741 perf-stat.ps.iTLB-load-misses
> 529058 ? 4% -9.4% 479204 ? 3% perf-stat.ps.iTLB-loads
> 6200 -9.4% 5620 perf-stat.ps.minor-faults
> 22613159 +24.4% 28133123 ? 3% perf-stat.ps.node-load-misses
> 11085254 +72.6% 19131576 ? 2% perf-stat.ps.node-store-misses
> 6200 -9.4% 5620 perf-stat.ps.page-faults
> 7008 ? 13% +55.2% 10876 ? 31% softirqs.CPU1.NET_RX
> 6509 ? 4% +51.8% 9883 ? 27% softirqs.CPU1.RCU
> 7294 ? 13% +36.7% 9974 ? 8% softirqs.CPU10.RCU
> 7800 ? 14% +85.5% 14469 ? 23% softirqs.CPU12.NET_RX
> 5697 ? 43% +77.5% 10110 ? 24% softirqs.CPU13.NET_RX
> 15944 ? 9% +14.6% 18278 ? 12% softirqs.CPU14.TIMER
> 6064 ? 19% +68.6% 10223 ? 31% softirqs.CPU15.NET_RX
> 7796 ? 14% +80.3% 14059 ? 25% softirqs.CPU16.NET_RX
> 15934 ? 10% +22.1% 19452 ? 11% softirqs.CPU18.TIMER
> 6725 ? 7% +40.0% 9413 ? 18% softirqs.CPU19.NET_RX
> 5710 ? 3% +53.4% 8756 ? 12% softirqs.CPU20.RCU
> 7018 ? 14% +65.5% 11616 ? 40% softirqs.CPU21.NET_RX
> 6389 ? 18% +66.9% 10666 ? 31% softirqs.CPU23.NET_RX
> 7259 ? 7% +36.1% 9881 ? 6% softirqs.CPU24.RCU
> 6491 ? 20% +58.5% 10289 ? 33% softirqs.CPU25.NET_RX
> 7090 ? 10% +58.7% 11256 ? 29% softirqs.CPU27.NET_RX
> 6333 ? 27% +70.3% 10786 ? 27% softirqs.CPU29.NET_RX
> 5813 ? 20% +84.2% 10706 ? 36% softirqs.CPU3.NET_RX
> 7041 ? 23% +105.7% 14483 ? 21% softirqs.CPU30.NET_RX
> 6654 ? 7% +64.1% 10918 ? 29% softirqs.CPU31.NET_RX
> 18019 ? 10% +11.1% 20016 ? 7% softirqs.CPU31.TIMER
> 4666 ? 8% +104.0% 9518 ? 23% softirqs.CPU32.RCU
> 15721 ? 16% +35.9% 21371 ? 11% softirqs.CPU32.TIMER
> 15684 ? 13% +40.0% 21959 ? 18% softirqs.CPU34.TIMER
> 6489 ? 15% +69.7% 11013 ? 37% softirqs.CPU35.NET_RX
> 7930 ? 7% +82.1% 14442 ? 29% softirqs.CPU36.NET_RX
> 15744 ? 14% +24.3% 19563 ? 6% softirqs.CPU36.TIMER
> 7028 ? 13% +29.3% 9085 ? 11% softirqs.CPU39.NET_RX
> 7491 ? 22% +100.4% 15011 ? 24% softirqs.CPU4.NET_RX
> 6119 ? 13% +58.7% 9710 ? 37% softirqs.CPU5.NET_RX
> 6980 ? 8% +47.8% 10318 ? 42% softirqs.CPU7.NET_RX
> 285674 +65.7% 473395 softirqs.NET_RX
> 267950 ? 5% +21.1% 324597 softirqs.RCU
> 238298 ? 2% +15.1% 274371 ? 2% softirqs.SCHED
> 689305 ? 3% +14.1% 786236 ? 3% softirqs.TIMER
> 56196 ? 2% +19.9% 67389 interrupts.CPU0.LOC:Local_timer_interrupts
> 55971 ? 2% +19.7% 67005 interrupts.CPU1.LOC:Local_timer_interrupts
> 5265 ? 17% -32.2% 3568 ? 29% interrupts.CPU1.RES:Rescheduling_interrupts
> 56163 ? 2% +19.2% 66960 interrupts.CPU10.LOC:Local_timer_interrupts
> 56178 ? 2% +19.5% 67139 interrupts.CPU11.LOC:Local_timer_interrupts
> 4669 ? 15% -30.3% 3253 ? 29% interrupts.CPU11.RES:Rescheduling_interrupts
> 56081 +19.7% 67142 interrupts.CPU12.LOC:Local_timer_interrupts
> 55999 ? 2% +19.5% 66893 interrupts.CPU13.LOC:Local_timer_interrupts
> 4746 ? 23% -30.0% 3324 ? 29% interrupts.CPU13.RES:Rescheduling_interrupts
> 55991 ? 2% +19.5% 66898 interrupts.CPU14.LOC:Local_timer_interrupts
> 55855 ? 2% +20.0% 67031 interrupts.CPU15.LOC:Local_timer_interrupts
> 55943 ? 2% +19.9% 67087 interrupts.CPU16.LOC:Local_timer_interrupts
> 56164 +18.9% 66802 interrupts.CPU17.LOC:Local_timer_interrupts
> 4579 ? 20% -45.6% 2492 ? 16% interrupts.CPU17.RES:Rescheduling_interrupts
> 55913 ? 2% +19.6% 66860 interrupts.CPU18.LOC:Local_timer_interrupts
> 55820 ? 2% +19.7% 66822 interrupts.CPU19.LOC:Local_timer_interrupts
> 4733 ? 17% -35.5% 3052 ? 21% interrupts.CPU19.RES:Rescheduling_interrupts
> 55989 ? 2% +20.0% 67177 interrupts.CPU2.LOC:Local_timer_interrupts
> 55891 ? 2% +20.3% 67258 interrupts.CPU20.LOC:Local_timer_interrupts
> 55966 ? 2% +19.6% 66921 interrupts.CPU21.LOC:Local_timer_interrupts
> 4920 ? 21% -33.4% 3278 ? 31% interrupts.CPU21.RES:Rescheduling_interrupts
> 55945 ? 2% +19.9% 67098 interrupts.CPU22.LOC:Local_timer_interrupts
> 55945 ? 2% +19.9% 67073 interrupts.CPU23.LOC:Local_timer_interrupts
> 4972 ? 24% -34.1% 3277 ? 36% interrupts.CPU23.RES:Rescheduling_interrupts
> 56093 ? 2% +19.8% 67185 interrupts.CPU24.LOC:Local_timer_interrupts
> 55910 ? 2% +19.7% 66914 interrupts.CPU25.LOC:Local_timer_interrupts
> 4660 ? 25% -28.9% 3313 ? 34% interrupts.CPU25.RES:Rescheduling_interrupts
> 56105 ? 2% +18.8% 66631 interrupts.CPU26.LOC:Local_timer_interrupts
> 55928 ? 2% +19.5% 66827 interrupts.CPU27.LOC:Local_timer_interrupts
> 55934 ? 2% +19.3% 66740 interrupts.CPU28.LOC:Local_timer_interrupts
> 55918 ? 2% +19.5% 66812 interrupts.CPU29.LOC:Local_timer_interrupts
> 4825 ? 22% -30.3% 3362 ? 30% interrupts.CPU29.RES:Rescheduling_interrupts
> 55920 ? 2% +19.8% 67004 interrupts.CPU3.LOC:Local_timer_interrupts
> 56007 ? 2% +19.5% 66917 interrupts.CPU30.LOC:Local_timer_interrupts
> 56238 +18.8% 66838 interrupts.CPU31.LOC:Local_timer_interrupts
> 4859 ? 21% -33.0% 3257 ? 36% interrupts.CPU31.RES:Rescheduling_interrupts
> 56099 +19.3% 66921 interrupts.CPU32.LOC:Local_timer_interrupts
> 55963 ? 2% +19.5% 66866 interrupts.CPU33.LOC:Local_timer_interrupts
> 55992 ? 2% +19.2% 66767 interrupts.CPU34.LOC:Local_timer_interrupts
> 55964 ? 2% +19.6% 66939 interrupts.CPU35.LOC:Local_timer_interrupts
> 4797 ? 26% -33.6% 3185 ? 38% interrupts.CPU35.RES:Rescheduling_interrupts
> 55770 ? 2% +20.2% 67016 interrupts.CPU36.LOC:Local_timer_interrupts
> 55889 +20.0% 67095 interrupts.CPU37.LOC:Local_timer_interrupts
> 56140 ? 2% +18.9% 66743 interrupts.CPU38.LOC:Local_timer_interrupts
> 56040 ? 2% +19.2% 66828 interrupts.CPU39.LOC:Local_timer_interrupts
> 5126 ? 23% -40.5% 3052 ? 11% interrupts.CPU39.RES:Rescheduling_interrupts
> 56022 ? 2% +19.9% 67149 interrupts.CPU4.LOC:Local_timer_interrupts
> 55850 ? 2% +20.1% 67066 interrupts.CPU5.LOC:Local_timer_interrupts
> 4741 ? 23% -31.9% 3227 ? 29% interrupts.CPU5.RES:Rescheduling_interrupts
> 55912 +19.5% 66798 interrupts.CPU6.LOC:Local_timer_interrupts
> 56062 ? 2% +19.6% 67052 interrupts.CPU7.LOC:Local_timer_interrupts
> 56118 +19.6% 67142 interrupts.CPU8.LOC:Local_timer_interrupts
> 56110 ? 2% +19.6% 67121 interrupts.CPU9.LOC:Local_timer_interrupts
> 2216 ? 43% -69.4% 677.50 ?112% interrupts.CPU9.NMI:Non-maskable_interrupts
> 2216 ? 43% -69.4% 677.50 ?112% interrupts.CPU9.PMI:Performance_monitoring_interrupts
> 2240009 ? 2% +19.6% 2678940 interrupts.LOC:Local_timer_interrupts
> 168296 ? 3% -10.1% 151314 interrupts.RES:Rescheduling_interrupts
> 20.37 -3.3 17.04 ? 5% perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
> 20.38 -3.3 17.04 ? 5% perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
> 20.27 -3.3 16.95 ? 5% perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
> 20.21 -3.3 16.91 ? 5% perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
> 26.02 -3.1 22.91 ? 4% perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
> 9.04 ? 2% -1.9 7.18 ? 7% perf-profile.calltrace.cycles-pp.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
> 9.02 ? 2% -1.9 7.17 ? 7% perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound
> 9.02 ? 2% -1.9 7.16 ? 7% perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write
> 9.03 ? 2% -1.9 7.18 ? 7% perf-profile.calltrace.cycles-pp.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch
> 9.07 ? 2% -1.9 7.22 ? 7% perf-profile.calltrace.cycles-pp.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
> 10.50 ? 2% -1.4 9.06 ? 4% perf-profile.calltrace.cycles-pp.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
> 10.51 ? 2% -1.4 9.08 ? 4% perf-profile.calltrace.cycles-pp.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
> 10.45 ? 2% -1.4 9.02 ? 4% perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch
> 9.82 ? 3% -1.4 8.45 ? 3% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound
> 9.82 ? 3% -1.4 8.45 ? 3% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit
> 9.82 ? 3% -1.4 8.45 ? 3% perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
> 10.61 ? 5% -1.1 9.46 ? 4% perf-profile.calltrace.cycles-pp.write
> 7.54 -1.1 6.41 ? 6% perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write
> 7.54 -1.1 6.41 ? 6% perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
> 10.27 ? 5% -1.1 9.15 ? 4% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 7.49 -1.1 6.37 ? 6% perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev
> 10.23 ? 5% -1.1 9.11 ? 4% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 10.35 ? 5% -1.1 9.23 ? 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
> 10.08 ? 5% -1.1 8.96 ? 4% perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 10.33 ? 5% -1.1 9.22 ? 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 10.05 ? 5% -1.1 8.95 ? 4% perf-profile.calltrace.cycles-pp.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64
> 9.80 ? 5% -1.1 8.73 ? 4% perf-profile.calltrace.cycles-pp.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_write
> 5.31 ? 5% -1.0 4.27 ? 5% perf-profile.calltrace.cycles-pp.rpc_free_task.rpc_async_release.process_one_work.worker_thread.kthread
> 5.31 ? 5% -1.0 4.27 ? 5% perf-profile.calltrace.cycles-pp.rpc_async_release.process_one_work.worker_thread.kthread.ret_from_fork
> 4.38 ? 5% -1.0 3.35 ? 2% perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv
> 4.38 ? 6% -1.0 3.36 ? 2% perf-profile.calltrace.cycles-pp.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd
> 4.39 ? 6% -1.0 3.40 ? 2% perf-profile.calltrace.cycles-pp.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd.kthread
> 3.53 ? 5% -0.9 2.62 ? 3% perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom
> 3.52 ? 5% -0.9 2.62 ? 3% perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.svc_recvfrom
> 5.01 ? 6% -0.9 4.14 ? 3% perf-profile.calltrace.cycles-pp.fsync
> 4.09 ? 7% -0.8 3.24 ? 5% perf-profile.calltrace.cycles-pp.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work.worker_thread
> 4.90 ? 6% -0.8 4.05 ? 3% perf-profile.calltrace.cycles-pp.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
> 4.90 ? 6% -0.8 4.05 ? 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.fsync
> 4.90 ? 6% -0.8 4.05 ? 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
> 4.90 ? 6% -0.8 4.05 ? 3% perf-profile.calltrace.cycles-pp.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 4.90 ? 6% -0.8 4.05 ? 3% perf-profile.calltrace.cycles-pp.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
> 5.84 ? 2% -0.8 5.01 ? 4% perf-profile.calltrace.cycles-pp.write_cache_pages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> 1.15 ? 7% -0.8 0.36 ?100% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.xfs_ilock
> 4.38 ? 7% -0.8 3.59 ? 3% perf-profile.calltrace.cycles-pp.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64
> 5.17 ? 2% -0.8 4.39 ? 5% perf-profile.calltrace.cycles-pp.xfs_do_writepage.write_cache_pages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> 4.89 ? 4% -0.7 4.18 ? 5% perf-profile.calltrace.cycles-pp.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request.submit_bio
> 1.38 ? 9% -0.7 0.68 ? 22% perf-profile.calltrace.cycles-pp.down_write.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write
> 1.38 ? 9% -0.7 0.68 ? 22% perf-profile.calltrace.cycles-pp.call_rwsem_down_write_failed.down_write.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev
> 1.38 ? 9% -0.7 0.68 ? 22% perf-profile.calltrace.cycles-pp.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.xfs_ilock.xfs_file_buffered_aio_write
> 1.38 ? 9% -0.7 0.68 ? 22% perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
> 4.01 ? 3% -0.6 3.40 ? 5% perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.xfs_add_to_ioend.xfs_do_writepage
> 4.01 ? 3% -0.6 3.40 ? 5% perf-profile.calltrace.cycles-pp.submit_bio.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages.xfs_vm_writepages
> 4.01 ? 3% -0.6 3.40 ? 5% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages
> 3.99 ? 3% -0.6 3.38 ? 5% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.xfs_add_to_ioend
> 4.11 ? 3% -0.6 3.51 ? 4% perf-profile.calltrace.cycles-pp.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages.xfs_vm_writepages.do_writepages
> 3.97 ? 5% -0.5 3.43 ? 4% perf-profile.calltrace.cycles-pp.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> 3.97 ? 5% -0.5 3.43 ? 4% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages
> 3.98 ? 5% -0.5 3.44 ? 4% perf-profile.calltrace.cycles-pp.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> 3.96 ? 5% -0.5 3.43 ? 4% perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages
> 3.92 ? 5% -0.5 3.40 ? 4% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.xfs_submit_ioend
> 2.89 ? 6% -0.5 2.38 ? 6% perf-profile.calltrace.cycles-pp.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
> 2.77 ? 10% -0.5 2.28 ? 2% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync
> 2.77 ? 10% -0.5 2.28 ? 2% perf-profile.calltrace.cycles-pp.nfs_writepages.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file_fsync
> 2.77 ? 10% -0.5 2.28 ? 2% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync
> 2.74 ? 10% -0.5 2.25 ? 2% perf-profile.calltrace.cycles-pp.write_cache_pages.nfs_writepages.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range
> 3.12 ? 7% -0.5 2.64 ? 4% perf-profile.calltrace.cycles-pp.nfs_write_begin.generic_perform_write.nfs_file_write.__vfs_write.vfs_write
> 2.93 ? 7% -0.5 2.47 ? 3% perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write.__vfs_write
> 3.37 ? 4% -0.5 2.91 ? 4% perf-profile.calltrace.cycles-pp.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.vfs_write
> 2.51 ? 4% -0.4 2.06 ? 7% perf-profile.calltrace.cycles-pp.clear_page_erms.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec
> 3.12 ? 5% -0.4 2.68 ? 7% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request
> 2.91 ? 5% -0.4 2.48 ? 3% perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
> 3.02 ? 5% -0.4 2.59 ? 7% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request
> 2.59 ? 4% -0.4 2.18 ? 5% perf-profile.calltrace.cycles-pp.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write
> 2.84 ? 7% -0.4 2.43 ? 3% perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write
> 3.27 -0.4 2.86 ? 6% perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
> 3.20 -0.4 2.79 ? 6% perf-profile.calltrace.cycles-pp.memcpy_erms.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
> 2.85 ? 5% -0.4 2.44 ? 3% perf-profile.calltrace.cycles-pp.memcpy_erms._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
> 2.50 ? 4% -0.4 2.14 ? 7% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
> 2.41 ? 4% -0.4 2.06 ? 7% perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
> 2.02 ? 11% -0.3 1.67 ? 3% perf-profile.calltrace.cycles-pp.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writepages.__filemap_fdatawrite_range
> 0.60 ? 8% -0.3 0.26 ?100% perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
> 1.96 ? 11% -0.3 1.62 ? 3% perf-profile.calltrace.cycles-pp.nfs_do_writepage.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writepages
> 1.55 ? 2% -0.3 1.22 ? 5% perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
> 2.35 ? 4% -0.3 2.03 ? 7% perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
> 1.60 ? 2% -0.3 1.32 ? 4% perf-profile.calltrace.cycles-pp.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync
> 1.49 ? 6% -0.2 1.24 ? 6% perf-profile.calltrace.cycles-pp._raw_spin_lock.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request
> 1.03 ? 6% -0.2 0.80 ? 7% perf-profile.calltrace.cycles-pp.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
> 0.97 ? 7% -0.2 0.74 ? 9% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
> 1.45 ? 6% -0.2 1.23 ? 4% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write
> 0.97 ? 4% -0.2 0.77 ? 3% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
> 1.22 ? 6% -0.2 1.03 ? 6% perf-profile.calltrace.cycles-pp.nfs_commit_release_pages.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work
> 1.22 ? 6% -0.2 1.03 ? 5% perf-profile.calltrace.cycles-pp.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work.worker_thread
> 1.25 ? 5% -0.2 1.06 ? 8% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
> 1.06 ? 2% -0.2 0.87 ? 4% perf-profile.calltrace.cycles-pp.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync
> 1.27 ? 5% -0.2 1.11 ? 6% perf-profile.calltrace.cycles-pp.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
> 0.78 ? 8% -0.2 0.63 ? 6% perf-profile.calltrace.cycles-pp.__set_page_dirty_nobuffers.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write
> 1.11 ? 7% -0.1 0.99 ? 6% perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
> 0.68 -0.1 0.57 ? 5% perf-profile.calltrace.cycles-pp.nfs_create_request.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write
> 1.04 ? 7% -0.1 0.93 ? 5% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion
> 1.02 ? 8% -0.1 0.92 ? 4% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback
> 0.71 ? 2% -0.1 0.60 ? 8% perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range
> 0.72 ? 2% -0.1 0.61 ? 8% perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_and_wait_range
> 0.72 -0.1 0.62 ? 7% perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_fsync
> 0.93 ? 5% -0.1 0.83 ? 5% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write
> 0.73 -0.1 0.67 ? 5% perf-profile.calltrace.cycles-pp.end_page_writeback.xfs_destroy_ioend.process_one_work.worker_thread.kthread
> 4.86 ? 6% +0.5 5.39 perf-profile.calltrace.cycles-pp.svc_recv.nfsd.kthread.ret_from_fork
> 0.00 +0.5 0.54 ? 3% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.svc_recv.nfsd.kthread
> 0.00 +0.6 0.59 ? 6% perf-profile.calltrace.cycles-pp.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
> 0.00 +0.6 0.63 ? 6% perf-profile.calltrace.cycles-pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
> 0.00 +0.7 0.65 ? 4% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.svc_recv.nfsd.kthread.ret_from_fork
> 0.00 +0.7 0.74 ? 8% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
> 0.00 +0.9 0.86 ? 8% perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages
> 9.98 ? 4% +2.5 12.44 ? 4% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
> 9.61 ? 4% +2.5 12.15 ? 4% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
> 47.02 ? 3% +2.7 49.68 ? 4% perf-profile.calltrace.cycles-pp.secondary_startup_64
> 0.00 +3.5 3.54 ? 3% perf-profile.calltrace.cycles-pp.memcpy_erms.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg
> 0.00 +3.6 3.56 ? 3% perf-profile.calltrace.cycles-pp.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
> 0.00 +3.7 3.67 ? 3% perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages
> 2.29 ? 6% +3.8 6.05 ? 3% perf-profile.calltrace.cycles-pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_from_fork
> 2.28 ? 6% +3.8 6.05 ? 3% perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.kthread
> 1.86 ? 3% +3.8 5.64 ? 4% perf-profile.calltrace.cycles-pp.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread
> 1.85 ? 3% +3.8 5.64 ? 4% perf-profile.calltrace.cycles-pp.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work
> 1.80 ? 4% +3.8 5.60 ? 4% perf-profile.calltrace.cycles-pp.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule
> 1.79 ? 3% +3.8 5.60 ? 4% perf-profile.calltrace.cycles-pp.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute
> 0.00 +5.5 5.47 ? 4% perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request
> 0.00 +5.6 5.56 ? 4% perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit
> 0.00 +5.6 5.57 ? 4% perf-profile.calltrace.cycles-pp.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit
> 20.37 -3.3 17.04 ? 5% perf-profile.children.cycles-pp.svc_process_common
> 20.38 -3.3 17.04 ? 5% perf-profile.children.cycles-pp.svc_process
> 20.27 -3.3 16.95 ? 5% perf-profile.children.cycles-pp.nfsd_dispatch
> 20.21 -3.3 16.91 ? 5% perf-profile.children.cycles-pp.nfsd4_proc_compound
> 26.02 -3.1 22.91 ? 4% perf-profile.children.cycles-pp.nfsd
> 15.88 ? 5% -1.9 13.95 ? 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 15.87 ? 5% -1.9 13.95 ? 4% perf-profile.children.cycles-pp.do_syscall_64
> 12.61 ? 3% -1.9 10.73 ? 3% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
> 12.60 ? 3% -1.9 10.73 ? 3% perf-profile.children.cycles-pp.do_writepages
> 9.04 ? 2% -1.9 7.18 ? 7% perf-profile.children.cycles-pp.nfsd_vfs_write
> 9.02 ? 2% -1.9 7.17 ? 7% perf-profile.children.cycles-pp.do_iter_readv_writev
> 9.02 ? 2% -1.9 7.16 ? 7% perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
> 9.03 ? 2% -1.9 7.18 ? 7% perf-profile.children.cycles-pp.do_iter_write
> 9.07 ? 2% -1.9 7.22 ? 7% perf-profile.children.cycles-pp.nfsd4_write
> 2.21 ? 4% -1.8 0.41 ? 4% perf-profile.children.cycles-pp.inet_sendpage
> 2.17 ? 4% -1.8 0.40 ? 5% perf-profile.children.cycles-pp.tcp_sendpage
> 10.50 ? 2% -1.4 9.06 ? 4% perf-profile.children.cycles-pp.xfs_file_fsync
> 10.51 ? 2% -1.4 9.08 ? 4% perf-profile.children.cycles-pp.nfsd_commit
> 10.45 ? 2% -1.4 9.02 ? 4% perf-profile.children.cycles-pp.file_write_and_wait_range
> 9.82 ? 3% -1.4 8.45 ? 3% perf-profile.children.cycles-pp.xfs_vm_writepages
> 8.59 ? 4% -1.3 7.27 ? 4% perf-profile.children.cycles-pp.write_cache_pages
> 8.02 ? 4% -1.2 6.87 ? 4% perf-profile.children.cycles-pp.submit_bio
> 8.02 ? 4% -1.2 6.87 ? 4% perf-profile.children.cycles-pp.generic_make_request
> 8.00 ? 4% -1.1 6.85 ? 4% perf-profile.children.cycles-pp.brd_make_request
> 7.94 ? 4% -1.1 6.79 ? 4% perf-profile.children.cycles-pp.brd_do_bvec
> 10.63 ? 5% -1.1 9.49 ? 4% perf-profile.children.cycles-pp.write
> 7.55 -1.1 6.41 ? 5% perf-profile.children.cycles-pp.iomap_apply
> 7.54 -1.1 6.41 ? 6% perf-profile.children.cycles-pp.iomap_file_buffered_write
> 10.28 ? 5% -1.1 9.16 ? 4% perf-profile.children.cycles-pp.ksys_write
> 7.50 -1.1 6.37 ? 6% perf-profile.children.cycles-pp.iomap_write_actor
> 4.68 ? 5% -1.1 3.56 ? 2% perf-profile.children.cycles-pp.inet_recvmsg
> 4.68 ? 5% -1.1 3.56 ? 2% perf-profile.children.cycles-pp.tcp_recvmsg
> 10.24 ? 5% -1.1 9.12 ? 4% perf-profile.children.cycles-pp.vfs_write
> 10.08 ? 5% -1.1 8.98 ? 4% perf-profile.children.cycles-pp.__vfs_write
> 10.05 ? 5% -1.1 8.95 ? 4% perf-profile.children.cycles-pp.nfs_file_write
> 5.51 ? 5% -1.1 4.43 ? 5% perf-profile.children.cycles-pp.rpc_free_task
> 9.82 ? 5% -1.1 8.74 ? 4% perf-profile.children.cycles-pp.generic_perform_write
> 1.39 ? 5% -1.1 0.32 ? 4% perf-profile.children.cycles-pp.tcp_sendpage_locked
> 1.36 ? 5% -1.1 0.31 ? 5% perf-profile.children.cycles-pp.do_tcp_sendpages
> 5.31 ? 5% -1.0 4.27 ? 5% perf-profile.children.cycles-pp.rpc_async_release
> 4.39 ? 6% -1.0 3.40 ? 2% perf-profile.children.cycles-pp.svc_recvfrom
> 3.55 ? 5% -0.9 2.64 ? 2% perf-profile.children.cycles-pp.__skb_datagram_iter
> 3.55 ? 5% -0.9 2.64 ? 2% perf-profile.children.cycles-pp.skb_copy_datagram_iter
> 5.01 ? 6% -0.9 4.15 ? 3% perf-profile.children.cycles-pp.fsync
> 4.09 ? 7% -0.9 3.24 ? 5% perf-profile.children.cycles-pp.nfs_write_completion
> 4.90 ? 6% -0.8 4.05 ? 3% perf-profile.children.cycles-pp.__x64_sys_fsync
> 4.91 ? 6% -0.8 4.07 ? 3% perf-profile.children.cycles-pp.nfs_file_fsync
> 4.90 ? 6% -0.8 4.05 ? 3% perf-profile.children.cycles-pp.do_fsync
> 5.36 ? 4% -0.8 4.53 ? 5% perf-profile.children.cycles-pp.grab_cache_page_write_begin
> 4.39 ? 7% -0.8 3.60 ? 2% perf-profile.children.cycles-pp.filemap_write_and_wait_range
> 5.18 ? 2% -0.8 4.40 ? 5% perf-profile.children.cycles-pp.xfs_do_writepage
> 5.21 ? 4% -0.7 4.47 ? 5% perf-profile.children.cycles-pp.pagecache_get_page
> 4.90 ? 4% -0.7 4.19 ? 5% perf-profile.children.cycles-pp.brd_insert_page
> 1.38 ? 9% -0.7 0.68 ? 22% perf-profile.children.cycles-pp.xfs_ilock
> 1.41 ? 8% -0.7 0.71 ? 22% perf-profile.children.cycles-pp.call_rwsem_down_write_failed
> 1.41 ? 8% -0.7 0.71 ? 22% perf-profile.children.cycles-pp.rwsem_down_write_failed
> 1.44 ? 7% -0.7 0.74 ? 21% perf-profile.children.cycles-pp.down_write
> 4.12 ? 3% -0.6 3.52 ? 4% perf-profile.children.cycles-pp.xfs_add_to_ioend
> 1.18 ? 6% -0.6 0.61 ? 21% perf-profile.children.cycles-pp.rwsem_spin_on_owner
> 3.98 ? 5% -0.5 3.44 ? 4% perf-profile.children.cycles-pp.xfs_submit_ioend
> 6.25 -0.5 5.72 ? 4% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
> 2.89 ? 6% -0.5 2.38 ? 6% perf-profile.children.cycles-pp.nfs_end_page_writeback
> 2.78 ? 10% -0.5 2.29 ? 2% perf-profile.children.cycles-pp.nfs_writepages
> 3.12 ? 7% -0.5 2.64 ? 4% perf-profile.children.cycles-pp.nfs_write_begin
> 3.37 ? 4% -0.5 2.91 ? 4% perf-profile.children.cycles-pp.nfs_write_end
> 2.54 ? 4% -0.5 2.08 ? 7% perf-profile.children.cycles-pp.clear_page_erms
> 2.92 ? 5% -0.4 2.48 ? 3% perf-profile.children.cycles-pp._copy_to_iter
> 2.60 ? 4% -0.4 2.18 ? 5% perf-profile.children.cycles-pp.nfs_updatepage
> 2.71 ? 4% -0.4 2.29 ? 6% perf-profile.children.cycles-pp.add_to_page_cache_lru
> 0.75 ? 3% -0.4 0.37 ? 6% perf-profile.children.cycles-pp.release_sock
> 2.51 ? 4% -0.4 2.15 ? 7% perf-profile.children.cycles-pp.iomap_write_begin
> 2.02 ? 10% -0.3 1.67 ? 3% perf-profile.children.cycles-pp.nfs_writepages_callback
> 2.23 ? 2% -0.3 1.89 ? 4% perf-profile.children.cycles-pp.__filemap_fdatawait_range
> 0.82 ? 7% -0.3 0.47 ? 7% perf-profile.children.cycles-pp.__tcp_push_pending_frames
> 1.96 ? 11% -0.3 1.63 ? 3% perf-profile.children.cycles-pp.nfs_do_writepage
> 1.55 ? 2% -0.3 1.22 ? 5% perf-profile.children.cycles-pp.iomap_write_end
> 0.78 ? 7% -0.3 0.48 ? 6% perf-profile.children.cycles-pp.svc_send
> 1.78 ? 3% -0.3 1.48 ? 4% perf-profile.children.cycles-pp.end_page_writeback
> 1.73 ? 3% -0.3 1.45 ? 4% perf-profile.children.cycles-pp.test_clear_page_writeback
> 0.90 ? 5% -0.3 0.63 ? 8% perf-profile.children.cycles-pp.xas_load
> 0.35 ? 14% -0.2 0.10 ? 14% perf-profile.children.cycles-pp.simple_copy_to_iter
> 1.78 ? 4% -0.2 1.54 ? 7% perf-profile.children.cycles-pp.__wake_up_common
> 0.43 ? 6% -0.2 0.19 ? 7% perf-profile.children.cycles-pp.lock_sock_nested
> 1.41 ? 5% -0.2 1.18 ? 4% perf-profile.children.cycles-pp.nfs_commit_release
> 1.41 ? 5% -0.2 1.18 ? 4% perf-profile.children.cycles-pp.nfs_commit_release_pages
> 1.95 ? 4% -0.2 1.72 ? 5% perf-profile.children.cycles-pp.try_to_wake_up
> 0.64 ? 8% -0.2 0.41 ? 4% perf-profile.children.cycles-pp.svc_tcp_sendto
> 0.63 ? 8% -0.2 0.41 ? 5% perf-profile.children.cycles-pp.svc_sendto
> 0.62 ? 8% -0.2 0.41 ? 5% perf-profile.children.cycles-pp.svc_send_common
> 0.62 ? 8% -0.2 0.41 ? 5% perf-profile.children.cycles-pp.kernel_sendpage
> 1.65 ? 5% -0.2 1.44 ? 7% perf-profile.children.cycles-pp.autoremove_wake_function
> 1.34 ? 3% -0.2 1.14 ? 3% perf-profile.children.cycles-pp.wait_on_page_bit_common
> 0.98 ? 4% -0.2 0.79 ? 2% perf-profile.children.cycles-pp.poll_idle
> 0.26 ? 8% -0.2 0.08 ? 13% perf-profile.children.cycles-pp._raw_spin_lock_bh
> 1.59 ? 5% -0.2 1.42 ? 5% perf-profile.children.cycles-pp.schedule
> 0.79 ? 5% -0.2 0.62 ? 4% perf-profile.children.cycles-pp.tcp_v6_do_rcv
> 0.64 ? 5% -0.2 0.47 ? 8% perf-profile.children.cycles-pp.tcp_v6_rcv
> 0.34 ? 12% -0.2 0.18 ? 20% perf-profile.children.cycles-pp.xas_start
> 0.24 ? 26% -0.2 0.08 ? 40% perf-profile.children.cycles-pp.osq_lock
> 0.76 ? 5% -0.2 0.61 ? 5% perf-profile.children.cycles-pp.tcp_rcv_established
> 0.49 ? 26% -0.2 0.34 ? 14% perf-profile.children.cycles-pp.nfs_request_add_commit_list
> 0.66 ? 5% -0.1 0.52 ? 8% perf-profile.children.cycles-pp.ip6_input
> 0.66 ? 4% -0.1 0.51 ? 8% perf-profile.children.cycles-pp.ip6_input_finish
> 0.65 ? 4% -0.1 0.51 ? 9% perf-profile.children.cycles-pp.ip6_protocol_deliver_rcu
> 0.79 ? 8% -0.1 0.64 ? 7% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
> 0.83 ? 6% -0.1 0.69 ? 8% perf-profile.children.cycles-pp.__nfs_commit_inode
> 0.42 ? 4% -0.1 0.29 ? 5% perf-profile.children.cycles-pp.xs_stream_data_receive_workfn
> 0.42 ? 4% -0.1 0.29 ? 5% perf-profile.children.cycles-pp.xs_stream_data_receive
> 0.70 ? 4% -0.1 0.56 ? 6% perf-profile.children.cycles-pp.percpu_counter_add_batch
> 0.71 ? 4% -0.1 0.58 ? 8% perf-profile.children.cycles-pp.ipv6_rcv
> 0.86 ? 6% -0.1 0.73 ? 6% perf-profile.children.cycles-pp.__local_bh_enable_ip
> 0.78 ? 5% -0.1 0.65 ? 8% perf-profile.children.cycles-pp.process_backlog
> 0.45 ? 14% -0.1 0.32 ? 14% perf-profile.children.cycles-pp.__mutex_lock
> 1.06 ? 4% -0.1 0.93 ? 7% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.86 ? 6% -0.1 0.73 ? 9% perf-profile.children.cycles-pp.pagevec_lru_move_fn
> 0.75 ? 5% -0.1 0.63 ? 8% perf-profile.children.cycles-pp.__netif_receive_skb_one_core
> 0.61 ? 8% -0.1 0.49 ? 9% perf-profile.children.cycles-pp.iomap_set_page_dirty
> 0.94 ? 5% -0.1 0.81 ? 8% perf-profile.children.cycles-pp.__lru_cache_add
> 0.82 ? 6% -0.1 0.70 ? 2% perf-profile.children.cycles-pp.clear_page_dirty_for_io
> 0.80 ? 6% -0.1 0.68 ? 7% perf-profile.children.cycles-pp.net_rx_action
> 0.94 ? 2% -0.1 0.82 ? 3% perf-profile.children.cycles-pp.io_schedule
> 0.81 ? 7% -0.1 0.70 ? 7% perf-profile.children.cycles-pp.do_softirq_own_stack
> 0.69 -0.1 0.58 ? 5% perf-profile.children.cycles-pp.nfs_create_request
> 0.62 ? 4% -0.1 0.51 ? 5% perf-profile.children.cycles-pp.__pagevec_lru_add_fn
> 0.32 ? 4% -0.1 0.21 ? 11% perf-profile.children.cycles-pp.xs_sock_recvmsg
> 0.49 ? 12% -0.1 0.38 ? 7% perf-profile.children.cycles-pp.sched_ttwu_pending
> 0.82 ? 6% -0.1 0.72 ? 7% perf-profile.children.cycles-pp.do_softirq
> 0.32 ? 10% -0.1 0.22 ? 7% perf-profile.children.cycles-pp.xfs_map_blocks
> 0.40 ? 13% -0.1 0.30 ? 12% perf-profile.children.cycles-pp.mutex_spin_on_owner
> 0.47 ? 2% -0.1 0.38 ? 5% perf-profile.children.cycles-pp.kmem_cache_alloc
> 0.26 ? 4% -0.1 0.17 ? 8% perf-profile.children.cycles-pp.__wake_up_common_lock
> 0.52 ? 6% -0.1 0.43 ? 4% perf-profile.children.cycles-pp.account_page_dirtied
> 0.22 ? 16% -0.1 0.13 ? 9% perf-profile.children.cycles-pp.__check_object_size
> 0.50 ? 8% -0.1 0.42 ? 7% perf-profile.children.cycles-pp.iomap_set_range_uptodate
> 0.41 ? 3% -0.1 0.33 ? 14% perf-profile.children.cycles-pp.__set_page_dirty
> 0.31 ? 11% -0.1 0.24 ? 11% perf-profile.children.cycles-pp.nfs_io_completion_release
> 0.41 ? 8% -0.1 0.33 ? 7% perf-profile.children.cycles-pp.nfs_inode_remove_request
> 0.25 ? 9% -0.1 0.18 ? 8% perf-profile.children.cycles-pp.xfs_iomap_write_allocate
> 0.33 ? 8% -0.1 0.26 ? 4% perf-profile.children.cycles-pp.___might_sleep
> 0.25 ? 5% -0.1 0.18 ? 15% perf-profile.children.cycles-pp.__xa_set_mark
> 0.31 ? 9% -0.1 0.24 ? 5% perf-profile.children.cycles-pp.nfs_scan_commit_list
> 0.25 ? 5% -0.1 0.18 ? 8% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
> 0.45 ? 6% -0.1 0.39 ? 4% perf-profile.children.cycles-pp.nfs_lock_and_join_requests
> 0.50 ? 4% -0.1 0.45 ? 5% perf-profile.children.cycles-pp.nfs_page_group_destroy
> 0.19 ? 11% -0.1 0.14 ? 8% perf-profile.children.cycles-pp.xfs_bmap_btalloc
> 0.21 ? 2% -0.1 0.15 ? 14% perf-profile.children.cycles-pp.__lock_sock
> 0.17 ? 8% -0.1 0.12 ? 7% perf-profile.children.cycles-pp.xfs_alloc_ag_vextent
> 0.18 ? 9% -0.1 0.13 ? 6% perf-profile.children.cycles-pp.xfs_alloc_vextent
> 0.20 ? 12% -0.1 0.15 ? 5% perf-profile.children.cycles-pp.xfs_bmapi_write
> 0.30 ? 9% -0.0 0.25 ? 5% perf-profile.children.cycles-pp.select_task_rq_fair
> 0.16 ? 8% -0.0 0.11 ? 7% perf-profile.children.cycles-pp.xfs_alloc_ag_vextent_near
> 0.16 ? 15% -0.0 0.12 ? 17% perf-profile.children.cycles-pp.__generic_write_end
> 0.48 ? 2% -0.0 0.44 ? 7% perf-profile.children.cycles-pp.dequeue_task_fair
> 0.31 ? 2% -0.0 0.27 ? 10% perf-profile.children.cycles-pp.release_pages
> 0.20 ? 4% -0.0 0.17 ? 10% perf-profile.children.cycles-pp.nfs_initiate_commit
> 0.17 ? 10% -0.0 0.13 ? 5% perf-profile.children.cycles-pp.__might_sleep
> 0.29 ? 6% -0.0 0.26 ? 4% perf-profile.children.cycles-pp.dec_zone_page_state
> 0.23 ? 7% -0.0 0.20 ? 7% perf-profile.children.cycles-pp.__pagevec_release
> 0.19 ? 9% -0.0 0.15 ? 9% perf-profile.children.cycles-pp.nfs_get_lock_context
> 0.17 ? 13% -0.0 0.13 ? 8% perf-profile.children.cycles-pp.check_preempt_curr
> 0.09 ? 17% -0.0 0.06 ? 20% perf-profile.children.cycles-pp.memset_erms
> 0.14 ? 8% -0.0 0.11 ? 6% perf-profile.children.cycles-pp.nfs_request_remove_commit_list
> 0.07 ? 10% -0.0 0.04 ? 58% perf-profile.children.cycles-pp.xfs_defer_finish_noroll
> 0.15 ? 5% -0.0 0.12 ? 6% perf-profile.children.cycles-pp.__nfs_find_lock_context
> 0.19 ? 4% -0.0 0.16 ? 11% perf-profile.children.cycles-pp.mem_cgroup_try_charge
> 0.17 ? 8% -0.0 0.15 ? 10% perf-profile.children.cycles-pp.vfs_create
> 0.16 ? 2% -0.0 0.14 ? 8% perf-profile.children.cycles-pp.__fprop_inc_percpu_max
> 0.16 ? 8% -0.0 0.13 ? 8% perf-profile.children.cycles-pp.xfs_create
> 0.10 ? 4% -0.0 0.07 ? 17% perf-profile.children.cycles-pp.xfs_file_aio_write_checks
> 0.11 ? 10% -0.0 0.08 ? 15% perf-profile.children.cycles-pp.nfs_pageio_doio
> 0.11 ? 10% -0.0 0.08 ? 15% perf-profile.children.cycles-pp.nfs_generic_pg_pgios
> 0.19 ? 4% -0.0 0.17 ? 7% perf-profile.children.cycles-pp.update_rq_clock
> 0.15 ? 4% -0.0 0.13 ? 5% perf-profile.children.cycles-pp.__xfs_trans_commit
> 0.08 ? 15% -0.0 0.06 ? 6% perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
> 0.06 ? 15% +0.0 0.07 ? 14% perf-profile.children.cycles-pp.selinux_ip_postroute
> 0.08 ? 16% +0.0 0.10 ? 10% perf-profile.children.cycles-pp.nf_hook_slow
> 0.04 ? 60% +0.0 0.08 ? 19% perf-profile.children.cycles-pp.__inc_numa_state
> 0.03 ?100% +0.0 0.07 ? 12% perf-profile.children.cycles-pp.get_task_policy
> 0.41 ? 6% +0.0 0.46 perf-profile.children.cycles-pp.__release_sock
> 0.14 ? 11% +0.1 0.19 ? 8% perf-profile.children.cycles-pp.__list_add_valid
> 0.09 ? 17% +0.1 0.14 ? 10% perf-profile.children.cycles-pp.svc_xprt_do_enqueue
> 0.14 ? 10% +0.1 0.20 ? 2% perf-profile.children.cycles-pp.tcp_clean_rtx_queue
> 0.00 +0.1 0.06 ? 11% perf-profile.children.cycles-pp.free_unref_page_commit
> 0.10 ? 11% +0.1 0.17 ? 6% perf-profile.children.cycles-pp.iov_iter_advance
> 0.17 ? 9% +0.1 0.24 ? 2% perf-profile.children.cycles-pp.tcp_ack
> 0.11 ? 21% +0.1 0.18 ? 2% perf-profile.children.cycles-pp.schedule_timeout
> 0.00 +0.1 0.08 ? 17% perf-profile.children.cycles-pp.free_one_page
> 0.23 ? 12% +0.1 0.32 ? 4% perf-profile.children.cycles-pp.__kfree_skb
> 0.10 ? 4% +0.1 0.19 ? 6% perf-profile.children.cycles-pp.xas_create
> 0.12 ? 17% +0.1 0.24 ? 4% perf-profile.children.cycles-pp.skb_release_data
> 0.00 +0.1 0.14 ? 5% perf-profile.children.cycles-pp.__free_pages_ok
> 0.00 +0.2 0.21 ? 9% perf-profile.children.cycles-pp.skb_page_frag_refill
> 0.00 +0.2 0.22 ? 9% perf-profile.children.cycles-pp.sk_page_frag_refill
> 0.00 +0.2 0.25 ? 5% perf-profile.children.cycles-pp.__sk_flush_backlog
> 0.03 ?100% +0.4 0.39 ? 7% perf-profile.children.cycles-pp.free_pcppages_bulk
> 0.07 ? 29% +0.5 0.54 ? 6% perf-profile.children.cycles-pp.free_unref_page
> 4.86 ? 6% +0.5 5.40 ? 2% perf-profile.children.cycles-pp.svc_recv
> 9.98 ? 4% +2.5 12.44 ? 4% perf-profile.children.cycles-pp.worker_thread
> 9.61 ? 4% +2.5 12.15 ? 4% perf-profile.children.cycles-pp.process_one_work
> 47.02 ? 3% +2.7 49.68 ? 4% perf-profile.children.cycles-pp.secondary_startup_64
> 47.02 ? 3% +2.7 49.68 ? 4% perf-profile.children.cycles-pp.cpu_startup_entry
> 47.02 ? 3% +2.7 49.69 ? 4% perf-profile.children.cycles-pp.do_idle
> 6.12 ? 2% +2.8 8.88 ? 3% perf-profile.children.cycles-pp.memcpy_erms
> 0.00 +3.6 3.58 ? 3% perf-profile.children.cycles-pp.memcpy_from_page
> 0.00 +3.7 3.68 ? 3% perf-profile.children.cycles-pp._copy_from_iter_full
> 2.43 ? 6% +3.8 6.20 ? 3% perf-profile.children.cycles-pp.__rpc_execute
> 2.29 ? 6% +3.8 6.05 ? 3% perf-profile.children.cycles-pp.rpc_async_schedule
> 1.93 ? 3% +3.8 5.71 ? 4% perf-profile.children.cycles-pp.call_transmit
> 1.92 ? 4% +3.8 5.71 ? 4% perf-profile.children.cycles-pp.xprt_transmit
> 1.87 ? 4% +3.8 5.67 ? 4% perf-profile.children.cycles-pp.xs_tcp_send_request
> 1.86 ? 4% +3.8 5.67 ? 4% perf-profile.children.cycles-pp.xs_sendpages
> 0.21 ? 7% +5.3 5.54 ? 4% perf-profile.children.cycles-pp.tcp_sendmsg_locked
> 0.24 ? 8% +5.4 5.63 ? 4% perf-profile.children.cycles-pp.tcp_sendmsg
> 0.25 ? 9% +5.4 5.64 ? 4% perf-profile.children.cycles-pp.sock_sendmsg
> 1.17 ? 7% -0.6 0.61 ? 21% perf-profile.self.cycles-pp.rwsem_spin_on_owner
> 2.50 ? 4% -0.5 2.05 ? 7% perf-profile.self.cycles-pp.clear_page_erms
> 2.46 ? 4% -0.4 2.06 ? 5% perf-profile.self.cycles-pp.brd_do_bvec
> 0.22 ? 10% -0.2 0.03 ?100% perf-profile.self.cycles-pp.__skb_datagram_iter
> 1.53 ? 6% -0.2 1.34 ? 3% perf-profile.self.cycles-pp._raw_spin_lock
> 0.91 ? 4% -0.2 0.74 ? 2% perf-profile.self.cycles-pp.poll_idle
> 0.24 ? 25% -0.2 0.08 ? 40% perf-profile.self.cycles-pp.osq_lock
> 0.74 ? 14% -0.2 0.59 ? 10% perf-profile.self.cycles-pp.nfs_do_writepage
> 0.32 ? 13% -0.1 0.17 ? 18% perf-profile.self.cycles-pp.xas_start
> 0.97 ? 3% -0.1 0.84 ? 8% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.57 ? 6% -0.1 0.45 ? 9% perf-profile.self.cycles-pp.xas_load
> 0.55 ? 10% -0.1 0.43 ? 7% perf-profile.self.cycles-pp.__schedule
> 0.14 ? 12% -0.1 0.03 ?100% perf-profile.self.cycles-pp._raw_spin_lock_bh
> 0.53 ? 8% -0.1 0.43 ? 9% perf-profile.self.cycles-pp.percpu_counter_add_batch
> 0.40 ? 13% -0.1 0.30 ? 12% perf-profile.self.cycles-pp.mutex_spin_on_owner
> 0.33 ? 8% -0.1 0.24 ? 12% perf-profile.self.cycles-pp.nfs_end_page_writeback
> 0.74 ? 6% -0.1 0.65 ? 8% perf-profile.self.cycles-pp.nfs_updatepage
> 0.40 ? 8% -0.1 0.31 ? 11% perf-profile.self.cycles-pp.__test_set_page_writeback
> 0.50 ? 8% -0.1 0.41 ? 7% perf-profile.self.cycles-pp.iomap_set_range_uptodate
> 0.56 ? 5% -0.1 0.48 ? 8% perf-profile.self.cycles-pp.try_to_wake_up
> 0.38 ? 8% -0.1 0.30 ? 5% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> 0.28 ? 9% -0.1 0.20 ? 6% perf-profile.self.cycles-pp.iomap_write_end
> 0.45 ? 7% -0.1 0.38 ? 6% perf-profile.self.cycles-pp.__pagevec_lru_add_fn
> 0.15 ? 14% -0.1 0.08 ? 27% perf-profile.self.cycles-pp.add_to_page_cache_lru
> 0.43 ? 6% -0.1 0.36 perf-profile.self.cycles-pp.clear_page_dirty_for_io
> 0.51 ? 8% -0.1 0.45 ? 7% perf-profile.self.cycles-pp.test_clear_page_writeback
> 0.20 ? 9% -0.1 0.14 ? 11% perf-profile.self.cycles-pp.xas_store
> 0.31 ? 8% -0.1 0.25 ? 4% perf-profile.self.cycles-pp.___might_sleep
> 0.15 ? 20% -0.1 0.09 ? 12% perf-profile.self.cycles-pp.__check_object_size
> 0.22 ? 6% -0.1 0.17 ? 10% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
> 0.08 ? 11% -0.0 0.03 ?100% perf-profile.self.cycles-pp.switch_mm_irqs_off
> 0.28 ? 4% -0.0 0.24 ? 10% perf-profile.self.cycles-pp.release_pages
> 0.27 ? 8% -0.0 0.23 ? 2% perf-profile.self.cycles-pp.dec_zone_page_state
> 0.19 ? 6% -0.0 0.15 ? 12% perf-profile.self.cycles-pp.kmem_cache_alloc
> 0.15 ? 10% -0.0 0.11 ? 7% perf-profile.self.cycles-pp.__might_sleep
> 0.25 ? 10% -0.0 0.21 ? 6% perf-profile.self.cycles-pp.wait_on_page_bit_common
> 0.07 ? 12% -0.0 0.03 ?100% perf-profile.self.cycles-pp.___slab_alloc
> 0.23 ? 7% -0.0 0.20 ? 10% perf-profile.self.cycles-pp._raw_spin_lock_irq
> 0.18 ? 5% -0.0 0.15 ? 2% perf-profile.self.cycles-pp.select_task_rq_fair
> 0.14 ? 5% -0.0 0.11 ? 17% perf-profile.self.cycles-pp.alloc_pages_current
> 0.09 ? 16% -0.0 0.06 ? 20% perf-profile.self.cycles-pp.memset_erms
> 0.10 ? 8% -0.0 0.07 perf-profile.self.cycles-pp._cond_resched
> 0.15 ? 4% -0.0 0.12 ? 6% perf-profile.self.cycles-pp.__nfs_find_lock_context
> 0.11 ? 13% -0.0 0.09 ? 14% perf-profile.self.cycles-pp.nfs_request_add_commit_list_locked
> 0.09 ? 4% -0.0 0.07 perf-profile.self.cycles-pp.nfs_request_remove_commit_list
> 0.08 ? 10% -0.0 0.06 ? 11% perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
> 0.08 ? 10% -0.0 0.06 ? 11% perf-profile.self.cycles-pp.nfs_request_add_commit_list
> 0.10 ? 5% -0.0 0.08 ? 5% perf-profile.self.cycles-pp.___perf_sw_event
> 0.08 ? 13% +0.0 0.11 ? 10% perf-profile.self.cycles-pp.mem_cgroup_commit_charge
> 0.03 ?100% +0.0 0.07 ? 12% perf-profile.self.cycles-pp.run_timer_softirq
> 0.03 ?100% +0.0 0.07 ? 12% perf-profile.self.cycles-pp.get_task_policy
> 0.03 ?100% +0.0 0.07 ? 17% perf-profile.self.cycles-pp.__inc_numa_state
> 0.03 ?102% +0.1 0.08 ? 8% perf-profile.self.cycles-pp.svc_recv
> 0.11 ? 13% +0.1 0.17 ? 12% perf-profile.self.cycles-pp.__list_add_valid
> 0.09 ? 14% +0.1 0.16 ? 9% perf-profile.self.cycles-pp.iov_iter_advance
> 0.06 ? 13% +0.1 0.15 ? 8% perf-profile.self.cycles-pp.xas_create
> 0.00 +0.1 0.15 ? 14% perf-profile.self.cycles-pp.tcp_sendmsg_locked
> 0.00 +0.2 0.20 ? 11% perf-profile.self.cycles-pp.free_pcppages_bulk
> 0.04 ? 60% +0.2 0.28 ? 7% perf-profile.self.cycles-pp.svc_tcp_recvfrom
> 1.25 +0.3 1.52 ? 5% perf-profile.self.cycles-pp.get_page_from_freelist
> 6.04 ? 2% +2.7 8.74 ? 3% perf-profile.self.cycles-pp.memcpy_erms
>
>
>
> fsmark.time.percent_of_cpu_this_job_got
>
> 320 +-+-------------------------------------------------------------------+
> | |
> 300 +-++. + |
> |. +.. .+. +.. +.. +. .. + .+.|
> 280 +-+ +.+. +.. : + .+..+. .+. .+. .+.+.. .. + +. |
> | : + + +. +. +. .+ |
> 260 +-+ + + |
> | |
> 240 +-+ |
> | |
> 220 +-+ |
> | O O O O O O O |
> 200 O-+ O O O O O O O O |
> | O O O O O O O O O O |
> 180 +-+-------------------------------------------------------------------+
>
>
> fsmark.time.elapsed_time
>
> 32 +-+O-------------------------------------------------------------------+
> O O O O O O O O O O O O |
> 31 +-+ O O O O O O O O O O O O |
> 30 +-+ O |
> | |
> 29 +-+ |
> 28 +-+ |
> | |
> 27 +-+ |
> 26 +-+ .+ |
> | + + +.+.. .+. + .+. .+. |
> 25 +-+ .. + .+ + : +..+.+..+. .. + +. +. +..|
> 24 +-+ .+.+..+.+ +. + + : .. + |
> | +.+. + + |
> 23 +-+--------------------------------------------------------------------+
>
>
> fsmark.time.elapsed_time.max
>
> 32 +-+O-------------------------------------------------------------------+
> O O O O O O O O O O O O |
> 31 +-+ O O O O O O O O O O O O |
> 30 +-+ O |
> | |
> 29 +-+ |
> 28 +-+ |
> | |
> 27 +-+ |
> 26 +-+ .+ |
> | + + +.+.. .+. + .+. .+. |
> 25 +-+ .. + .+ + : +..+.+..+. .. + +. +. +..|
> 24 +-+ .+.+..+.+ +. + + : .. + |
> | +.+. + + |
> 23 +-+--------------------------------------------------------------------+
>
>
> fsmark.files_per_sec
>
> 520 +-+-------------------------------------------------------------------+
> |.. +.+. + +.. .+.. : .+. .+.. .+.+..+ |
> 500 +-+ +.. + + : +.+..+.+. +..+ .+.+. |
> 480 +-+ + + +.+. |
> | |
> 460 +-+ |
> | |
> 440 +-+ |
> | |
> 420 +-+ |
> 400 +-+ O |
> | O O O O O O O O O |
> 380 O-+ O O O O O O O O O |
> | O O O O O O |
> 360 +-+-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Rong Chen
>
>
> _______________________________________________
> LKP mailing list
> [email protected]
> https://lists.01.org/mailman/listinfo/lkp
>
Do you have time to take a look at this regression?
--
Zhengjun Xing
Hi Xing,
On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote:
> Hi Trond,
>
> On 5/20/2019 1:54 PM, kernel test robot wrote:
> > Greeting,
> >
> > FYI, we noticed a 16.0% improvement of fsmark.app_overhead due to
> > commit:
> >
> >
> > commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC: Convert
> > socket page send code to use iov_iter()")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
> > master
> >
> > in testcase: fsmark
> > on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @
> > 3.00GHz with 384G memory
> > with following parameters:
> >
> > iterations: 1x
> > nr_threads: 64t
> > disk: 1BRD_48G
> > fs: xfs
> > fs2: nfsv4
> > filesize: 4M
> > test_size: 40G
> > sync_method: fsyncBeforeClose
> > cpufreq_governor: performance
> >
> > test-description: The fsmark is a file system benchmark to test
> > synchronous write workloads, for example, mail servers workload.
> > test-url: https://sourceforge.net/projects/fsmark/
> >
> >
> >
> > Details are as below:
> > -----------------------------------------------------------------
> > --------------------------------->
> >
> >
> > To reproduce:
> >
> > git clone https://github.com/intel/lkp-tests.git
> > cd lkp-tests
> > bin/lkp install job.yaml # job file is attached in this
> > email
> > bin/lkp run job.yaml
> >
> > ===================================================================
> > ======================
> > compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/n
> > r_threads/rootfs/sync_method/tbox_group/test_size/testcase:
> > gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-
> > 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-
> > ep01/40G/fsmark
> >
> > commit:
> > e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use
> > iov_iter_kvec()")
> > 0472e47660 ("SUNRPC: Convert socket page send code to use
> > iov_iter()")
> >
> > e791f8e9380d945e 0472e476604998c127f3c80d291
> > ---------------- ---------------------------
> > fail:runs %reproduction fail:runs
> > | | |
> > :4 50% 2:4 dmesg.WARNING:at#for
> > _ip_interrupt_entry/0x
> > %stddev %change %stddev
> > \ | \
> > 15118573 ± 2% +16.0% 17538083 fsmark.app_overhead
> > 510.93 -22.7% 395.12 fsmark.files_per_sec
> > 24.90 +22.8% 30.57 fsmark.time.elapsed_
> > time
> > 24.90 +22.8% 30.57 fsmark.time.elapsed_
> > time.max
> > 288.00 ± 2% -
> > 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got
> > 70.03 ± 2% -
> > 11.3% 62.14 fsmark.time.system_time
> > 4391964 -16.7% 3658341 meminfo.max_used_kB
> > 6.10 ± 4% +1.9 7.97
> > ± 3% mpstat.cpu.all.iowait%
> > 0.27 -0.0 0.24 ± 3% mpstat.cpu.all.soft%
> > 13668070 ± 40% +118.0% 29801846 ± 19% numa-
> > numastat.node0.local_node
> > 13677774 ± 40% +117.9% 29810258 ± 19% numa-
> > numastat.node0.numa_hit
> > 5.70 ± 3% +32.1% 7.53 ± 3% iostat.cpu.iowait
> > 16.42 ± 2% -5.8% 15.47 iostat.cpu.system
> > 2.57 -4.1% 2.46 iostat.cpu.user
> > 1406781 ± 2% -15.5% 1188498 vmstat.io.bo
> > 251792 ± 3% -16.6% 209928 vmstat.system.cs
> > 84841 -1.9% 83239 vmstat.system.in
> > 97374502 ± 20% +66.1% 1.617e+08 ± 17% cpuidle.C1E.time
> > 573934 ± 19% +44.6% 829662 ± 26% cpuidle.C1E.usage
> > 5.892e+08 ± 8% +15.3% 6.796e+08 ± 2% cpuidle.C6.time
> > 1968016 ± 3% -15.1% 1670867 ± 3% cpuidle.POLL.time
> > 106420 ± 47% +86.2% 198108 ± 35% numa-
> > meminfo.node0.Active
> > 106037 ± 48% +86.2% 197395 ± 35% numa-
> > meminfo.node0.Active(anon)
> > 105052 ± 48% +86.6% 196037 ± 35% numa-
> > meminfo.node0.AnonPages
> > 212876 ± 24% -41.5% 124572 ± 56% numa-
> > meminfo.node1.Active
> > 211801 ± 24% -41.5% 123822 ± 56% numa-
> > meminfo.node1.Active(anon)
> > 208559 ± 24% -42.2% 120547 ± 57% numa-
> > meminfo.node1.AnonPages
> > 9955 +1.6% 10116 proc-
> > vmstat.nr_kernel_stack
> > 452.25 ± 59% +280.9% 1722 ±100% proc-
> > vmstat.numa_hint_faults_local
> > 33817303 +55.0% 52421773 ± 5% proc-vmstat.numa_hit
> > 33804286 +55.0% 52408807 ± 5% proc-
> > vmstat.numa_local
> > 33923002 +81.8% 61663426 ± 5% proc-
> > vmstat.pgalloc_normal
> > 184765 +9.3% 201985 proc-vmstat.pgfault
> > 12840986 +216.0% 40581327 ± 7% proc-vmstat.pgfree
> > 31447 ± 11% -26.1% 23253 ±
> > 13% sched_debug.cfs_rq:/.min_vruntime.max
> > 4241 ± 3% -12.2% 3724 ±
> > 11% sched_debug.cfs_rq:/.min_vruntime.stddev
> > 20631 ± 11% -36.7% 13069 ±
> > 29% sched_debug.cfs_rq:/.spread0.max
> > 4238 ± 4% -12.1% 3724 ±
> > 11% sched_debug.cfs_rq:/.spread0.stddev
> > 497105 ± 19% -16.0% 417777
> > ± 4% sched_debug.cpu.avg_idle.avg
> > 21199 ± 10% -12.0% 18650
> > ± 3% sched_debug.cpu.nr_load_updates.max
> > 2229 ± 10% -15.0% 1895
> > ± 4% sched_debug.cpu.nr_load_updates.stddev
> > 4.86 ± 5% -23.6% 3.72
> > ± 5% sched_debug.cpu.nr_uninterruptible.stddev
> > 524.75 ± 2% -10.7% 468.50 turbostat.Avg_MHz
> > 5.26 ± 41% -1.6 3.66 ± 2% turbostat.C1%
> > 573633 ± 19% +44.6% 829267 ± 26% turbostat.C1E
> > 8.53 ± 20% +3.4 11.88 ± 17% turbostat.C1E%
> > 76.75 -6.4% 71.86 turbostat.CorWatt
> > 2534071 ± 2% +16.2% 2943968 turbostat.IRQ
> > 2.89 ± 24% +52.9% 4.42 ± 22% turbostat.Pkg%pc2
> > 104.28 -4.6% 99.51 turbostat.PkgWatt
> > 2289 +18.8% 2720 turbostat.SMI
> > 26496 ± 47% +86.2% 49347 ± 35% numa-
> > vmstat.node0.nr_active_anon
> > 26251 ± 48% +86.7% 49008 ± 35% numa-
> > vmstat.node0.nr_anon_pages
> > 26496 ± 47% +86.2% 49347 ± 35% numa-
> > vmstat.node0.nr_zone_active_anon
> > 8272770 ± 35% +99.6% 16515977 ± 17% numa-
> > vmstat.node0.numa_hit
> > 8199681 ± 36% +100.9% 16474282 ± 17% numa-
> > vmstat.node0.numa_local
> > 52953 ± 24% -41.5% 30956 ± 56% numa-
> > vmstat.node1.nr_active_anon
> > 52144 ± 24% -42.2% 30136 ± 57% numa-
> > vmstat.node1.nr_anon_pages
> > 2899 ± 2% -19.4% 2336 ± 10% numa-
> > vmstat.node1.nr_writeback
> > 52953 ± 24% -41.5% 30956 ± 56% numa-
> > vmstat.node1.nr_zone_active_anon
> > 4549 ± 8% -18.9% 3689 ± 7% numa-
> > vmstat.node1.nr_zone_write_pending
> > 43.25 -1.6 41.69 perf-stat.i.cache-
> > miss-rate%
> > 286377 ± 2% -19.1% 231819 ± 2% perf-stat.i.context-
> > switches
> > 2.258e+10 ± 4% -16.2% 1.893e+10 ± 10% perf-stat.i.cpu-
> > cycles
> > 4084 -34.0% 2696 ± 2% perf-stat.i.cpu-
> > migrations
> > 21563675 ± 24% -30.8% 14929445 ± 5% perf-stat.i.dTLB-
> > load-misses
> > 1996341 ± 16% -25.7% 1482826 ± 6% perf-stat.i.dTLB-
> > store-misses
> > 2.313e+09 ± 4% -11.7% 2.042e+09 ± 6% perf-stat.i.dTLB-
> > stores
> > 75.93 -1.3 74.66 perf-stat.i.iTLB-
> > load-miss-rate%
> > 1561381 ± 3% -13.4% 1351731 perf-stat.i.iTLB-
> > load-misses
> > 549643 ± 4% -10.1% 494255 ± 3% perf-stat.i.iTLB-
> > loads
> > 6423 -10.0% 5779 perf-stat.i.minor-
> > faults
> > 31.72 ± 2% +4.4 36.14 perf-stat.i.node-
> > load-miss-rate%
> > 23492890 +23.5% 29020305 ± 3% perf-stat.i.node-
> > load-misses
> > 15.13 ± 8% +7.9 23.02 ± 3% perf-stat.i.node-
> > store-miss-rate%
> > 11517615 +71.3% 19734054 ± 2% perf-stat.i.node-
> > store-misses
> > 6423 -10.0% 5779 perf-stat.i.page-
> > faults
> > 16.88 ± 14% +33.4% 22.52 ± 23% perf-
> > stat.overall.MPKI
> > 46.67 -2.4 44.31 perf-
> > stat.overall.cache-miss-rate%
> > 31.96 +4.8 36.72 perf-
> > stat.overall.node-load-miss-rate%
> > 13.85 +8.0 21.87 perf-
> > stat.overall.node-store-miss-rate%
> > 2.266e+08 +2.8% 2.331e+08 ± 2% perf-stat.ps.cache-
> > references
> > 275634 ± 2% -18.5% 224718 ± 2% perf-
> > stat.ps.context-switches
> > 2.174e+10 ± 4% -15.6% 1.835e+10 ± 10% perf-stat.ps.cpu-
> > cycles
> > 3931 -33.5% 2614 ± 2% perf-stat.ps.cpu-
> > migrations
> > 20746040 ± 24% -30.2% 14476157 ± 5% perf-stat.ps.dTLB-
> > load-misses
> > 1921077 ± 16% -25.2% 1437750 ± 6% perf-stat.ps.dTLB-
> > store-misses
> > 2.227e+09 ± 4% -11.1% 1.979e+09 ± 6% perf-stat.ps.dTLB-
> > stores
> > 1503433 ± 3% -12.8% 1310741 perf-stat.ps.iTLB-
> > load-misses
> > 529058 ± 4% -9.4% 479204 ± 3% perf-stat.ps.iTLB-
> > loads
> > 6200 -9.4% 5620 perf-stat.ps.minor-
> > faults
> > 22613159 +24.4% 28133123 ± 3% perf-stat.ps.node-
> > load-misses
> > 11085254 +72.6% 19131576 ± 2% perf-stat.ps.node-
> > store-misses
> > 6200 -9.4% 5620 perf-stat.ps.page-
> > faults
> > 7008 ± 13% +55.2% 10876 ± 31% softirqs.CPU1.NET_RX
> > 6509 ± 4% +51.8% 9883 ± 27% softirqs.CPU1.RCU
> > 7294 ± 13% +36.7% 9974 ± 8% softirqs.CPU10.RCU
> > 7800 ± 14% +85.5% 14469 ±
> > 23% softirqs.CPU12.NET_RX
> > 5697 ± 43% +77.5% 10110 ±
> > 24% softirqs.CPU13.NET_RX
> > 15944 ± 9% +14.6% 18278 ± 12% softirqs.CPU14.TIMER
> > 6064 ± 19% +68.6% 10223 ±
> > 31% softirqs.CPU15.NET_RX
> > 7796 ± 14% +80.3% 14059 ±
> > 25% softirqs.CPU16.NET_RX
> > 15934 ± 10% +22.1% 19452 ± 11% softirqs.CPU18.TIMER
> > 6725 ± 7% +40.0% 9413 ±
> > 18% softirqs.CPU19.NET_RX
> > 5710 ± 3% +53.4% 8756 ± 12% softirqs.CPU20.RCU
> > 7018 ± 14% +65.5% 11616 ±
> > 40% softirqs.CPU21.NET_RX
> > 6389 ± 18% +66.9% 10666 ±
> > 31% softirqs.CPU23.NET_RX
> > 7259 ± 7% +36.1% 9881 ± 6% softirqs.CPU24.RCU
> > 6491 ± 20% +58.5% 10289 ±
> > 33% softirqs.CPU25.NET_RX
> > 7090 ± 10% +58.7% 11256 ±
> > 29% softirqs.CPU27.NET_RX
> > 6333 ± 27% +70.3% 10786 ±
> > 27% softirqs.CPU29.NET_RX
> > 5813 ± 20% +84.2% 10706 ± 36% softirqs.CPU3.NET_RX
> > 7041 ± 23% +105.7% 14483 ±
> > 21% softirqs.CPU30.NET_RX
> > 6654 ± 7% +64.1% 10918 ±
> > 29% softirqs.CPU31.NET_RX
> > 18019 ± 10% +11.1% 20016 ± 7% softirqs.CPU31.TIMER
> > 4666 ± 8% +104.0% 9518 ± 23% softirqs.CPU32.RCU
> > 15721 ± 16% +35.9% 21371 ± 11% softirqs.CPU32.TIMER
> > 15684 ± 13% +40.0% 21959 ± 18% softirqs.CPU34.TIMER
> > 6489 ± 15% +69.7% 11013 ±
> > 37% softirqs.CPU35.NET_RX
> > 7930 ± 7% +82.1% 14442 ±
> > 29% softirqs.CPU36.NET_RX
> > 15744 ± 14% +24.3% 19563 ± 6% softirqs.CPU36.TIMER
> > 7028 ± 13% +29.3% 9085 ±
> > 11% softirqs.CPU39.NET_RX
> > 7491 ± 22% +100.4% 15011 ± 24% softirqs.CPU4.NET_RX
> > 6119 ± 13% +58.7% 9710 ± 37% softirqs.CPU5.NET_RX
> > 6980 ± 8% +47.8% 10318 ± 42% softirqs.CPU7.NET_RX
> > 285674 +65.7% 473395 softirqs.NET_RX
> > 267950 ± 5% +21.1% 324597 softirqs.RCU
> > 238298 ± 2% +15.1% 274371 ± 2% softirqs.SCHED
> > 689305 ± 3% +14.1% 786236 ± 3% softirqs.TIMER
> > 56196
> > ± 2% +19.9% 67389 interrupts.CPU0.LOC:Local_timer_
> > interrupts
> > 55971
> > ± 2% +19.7% 67005 interrupts.CPU1.LOC:Local_timer_
> > interrupts
> > 5265 ± 17% -32.2% 3568 ±
> > 29% interrupts.CPU1.RES:Rescheduling_interrupts
> > 56163
> > ± 2% +19.2% 66960 interrupts.CPU10.LOC:Local_timer
> > _interrupts
> > 56178
> > ± 2% +19.5% 67139 interrupts.CPU11.LOC:Local_timer
> > _interrupts
> > 4669 ± 15% -30.3% 3253 ±
> > 29% interrupts.CPU11.RES:Rescheduling_interrupts
> > 56081 +19.7% 67142 interrupts.CPU12.LOC
> > :Local_timer_interrupts
> > 55999
> > ± 2% +19.5% 66893 interrupts.CPU13.LOC:Local_timer
> > _interrupts
> > 4746 ± 23% -30.0% 3324 ±
> > 29% interrupts.CPU13.RES:Rescheduling_interrupts
> > 55991
> > ± 2% +19.5% 66898 interrupts.CPU14.LOC:Local_timer
> > _interrupts
> > 55855
> > ± 2% +20.0% 67031 interrupts.CPU15.LOC:Local_timer
> > _interrupts
> > 55943
> > ± 2% +19.9% 67087 interrupts.CPU16.LOC:Local_timer
> > _interrupts
> > 56164 +18.9% 66802 interrupts.CPU17.LOC
> > :Local_timer_interrupts
> > 4579 ± 20% -45.6% 2492 ±
> > 16% interrupts.CPU17.RES:Rescheduling_interrupts
> > 55913
> > ± 2% +19.6% 66860 interrupts.CPU18.LOC:Local_timer
> > _interrupts
> > 55820
> > ± 2% +19.7% 66822 interrupts.CPU19.LOC:Local_timer
> > _interrupts
> > 4733 ± 17% -35.5% 3052 ±
> > 21% interrupts.CPU19.RES:Rescheduling_interrupts
> > 55989
> > ± 2% +20.0% 67177 interrupts.CPU2.LOC:Local_timer_
> > interrupts
> > 55891
> > ± 2% +20.3% 67258 interrupts.CPU20.LOC:Local_timer
> > _interrupts
> > 55966
> > ± 2% +19.6% 66921 interrupts.CPU21.LOC:Local_timer
> > _interrupts
> > 4920 ± 21% -33.4% 3278 ±
> > 31% interrupts.CPU21.RES:Rescheduling_interrupts
> > 55945
> > ± 2% +19.9% 67098 interrupts.CPU22.LOC:Local_timer
> > _interrupts
> > 55945
> > ± 2% +19.9% 67073 interrupts.CPU23.LOC:Local_timer
> > _interrupts
> > 4972 ± 24% -34.1% 3277 ±
> > 36% interrupts.CPU23.RES:Rescheduling_interrupts
> > 56093
> > ± 2% +19.8% 67185 interrupts.CPU24.LOC:Local_timer
> > _interrupts
> > 55910
> > ± 2% +19.7% 66914 interrupts.CPU25.LOC:Local_timer
> > _interrupts
> > 4660 ± 25% -28.9% 3313 ±
> > 34% interrupts.CPU25.RES:Rescheduling_interrupts
> > 56105
> > ± 2% +18.8% 66631 interrupts.CPU26.LOC:Local_timer
> > _interrupts
> > 55928
> > ± 2% +19.5% 66827 interrupts.CPU27.LOC:Local_timer
> > _interrupts
> > 55934
> > ± 2% +19.3% 66740 interrupts.CPU28.LOC:Local_timer
> > _interrupts
> > 55918
> > ± 2% +19.5% 66812 interrupts.CPU29.LOC:Local_timer
> > _interrupts
> > 4825 ± 22% -30.3% 3362 ±
> > 30% interrupts.CPU29.RES:Rescheduling_interrupts
> > 55920
> > ± 2% +19.8% 67004 interrupts.CPU3.LOC:Local_timer_
> > interrupts
> > 56007
> > ± 2% +19.5% 66917 interrupts.CPU30.LOC:Local_timer
> > _interrupts
> > 56238 +18.8% 66838 interrupts.CPU31.LOC
> > :Local_timer_interrupts
> > 4859 ± 21% -33.0% 3257 ±
> > 36% interrupts.CPU31.RES:Rescheduling_interrupts
> > 56099 +19.3% 66921 interrupts.CPU32.LOC
> > :Local_timer_interrupts
> > 55963
> > ± 2% +19.5% 66866 interrupts.CPU33.LOC:Local_timer
> > _interrupts
> > 55992
> > ± 2% +19.2% 66767 interrupts.CPU34.LOC:Local_timer
> > _interrupts
> > 55964
> > ± 2% +19.6% 66939 interrupts.CPU35.LOC:Local_timer
> > _interrupts
> > 4797 ± 26% -33.6% 3185 ±
> > 38% interrupts.CPU35.RES:Rescheduling_interrupts
> > 55770
> > ± 2% +20.2% 67016 interrupts.CPU36.LOC:Local_timer
> > _interrupts
> > 55889 +20.0% 67095 interrupts.CPU37.LOC
> > :Local_timer_interrupts
> > 56140
> > ± 2% +18.9% 66743 interrupts.CPU38.LOC:Local_timer
> > _interrupts
> > 56040
> > ± 2% +19.2% 66828 interrupts.CPU39.LOC:Local_timer
> > _interrupts
> > 5126 ± 23% -40.5% 3052 ±
> > 11% interrupts.CPU39.RES:Rescheduling_interrupts
> > 56022
> > ± 2% +19.9% 67149 interrupts.CPU4.LOC:Local_timer_
> > interrupts
> > 55850
> > ± 2% +20.1% 67066 interrupts.CPU5.LOC:Local_timer_
> > interrupts
> > 4741 ± 23% -31.9% 3227 ±
> > 29% interrupts.CPU5.RES:Rescheduling_interrupts
> > 55912 +19.5% 66798 interrupts.CPU6.LOC:
> > Local_timer_interrupts
> > 56062
> > ± 2% +19.6% 67052 interrupts.CPU7.LOC:Local_timer_
> > interrupts
> > 56118 +19.6% 67142 interrupts.CPU8.LOC:
> > Local_timer_interrupts
> > 56110
> > ± 2% +19.6% 67121 interrupts.CPU9.LOC:Local_timer_
> > interrupts
> > 2216 ± 43% -69.4% 677.50
> > ±112% interrupts.CPU9.NMI:Non-maskable_interrupts
> > 2216 ± 43% -69.4% 677.50
> > ±112% interrupts.CPU9.PMI:Performance_monitoring_interrupts
> > 2240009
> > ± 2% +19.6% 2678940 interrupts.LOC:Local_timer_inter
> > rupts
> > 168296 ± 3% -
> > 10.1% 151314 interrupts.RES:Rescheduling_interrupts
> > 20.37 -3.3 17.04 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
> > 20.38 -3.3 17.04 ± 5% perf-
> > profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
> > 20.27 -3.3 16.95 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
> > 20.21 -3.3 16.91 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
> > .nfsd
> > 26.02 -3.1 22.91 ± 4% perf-
> > profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
> > 9.04 ± 2% -1.9 7.18 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc
> > _process_common
> > 9.02 ± 2% -1.9 7.17 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write.nf
> > sd4_proc_compound
> > 9.02 ± 2% -1.9 7.16 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.n
> > fsd_vfs_write.nfsd4_write
> > 9.03 ± 2% -1.9 7.18 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfs
> > d_dispatch
> > 9.07 ± 2% -1.9 7.22 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
> > .svc_process
> > 10.50 ± 2% -1.4 9.06 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc
> > _process_common
> > 10.51 ± 2% -1.4 9.08 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
> > .svc_process
> > 10.45 ± 2% -1.4 9.02 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_
> > compound.nfsd_dispatch
> > 9.82 ± 3% -1.4 8.45 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fs
> > ync.nfsd_commit.nfsd4_proc_compound
> > 9.82 ± 3% -1.4 8.45 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_ran
> > ge.xfs_file_fsync.nfsd_commit
> > 9.82 ± 3% -1.4 8.45 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_
> > write_and_wait_range.xfs_file_fsync
> > 10.61 ± 5% -1.1 9.46 ± 4% perf-
> > profile.calltrace.cycles-pp.write
> > 7.54 -1.1 6.41 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_writ
> > e.do_iter_readv_writev.do_iter_write
> > 7.54 -1.1 6.41 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_re
> > adv_writev.do_iter_write.nfsd_vfs_write
> > 10.27 ± 5% -1.1 9.15 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> > 7.49 -1.1 6.37 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file
> > _buffered_aio_write.do_iter_readv_writev
> > 10.23 ± 5% -1.1 9.11 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwfram
> > e.write
> > 10.35 ± 5% -1.1 9.23 ± 4% perf-
> > profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
> > 10.08 ± 5% -1.1 8.96 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_
> > after_hwframe
> > 10.33 ± 5% -1.1 9.22 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> > 10.05 ± 5% -1.1 8.95 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64
> > 9.80 ± 5% -1.1 8.73 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_
> > write
> > 5.31 ± 5% -1.0 4.27 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.rpc_free_task.rpc_async_release.process_one_work.worker_thread.k
> > thread
> > 5.31 ± 5% -1.0 4.27 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.rpc_async_release.process_one_work.worker_thread.kthread.ret_fro
> > m_fork
> > 4.38 ± 5% -1.0 3.35 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.tcp_recvmsg.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv
> > 4.38 ± 6% -1.0 3.36 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd
> > 4.39 ± 6% -1.0 3.40 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd.kthread
> > 3.53 ± 5% -0.9 2.62 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.svc_recvfrom.svc
> > _tcp_recvfrom
> > 3.52 ± 5% -0.9 2.62 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recv
> > msg.svc_recvfrom
> > 5.01 ± 6% -0.9 4.14 ± 3% perf-
> > profile.calltrace.cycles-pp.fsync
> > 4.09 ± 7% -0.8 3.24 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.nfs_write_completion.rpc_free_task.rpc_async_release.process_one
> > _work.worker_thread
> > 4.90 ± 6% -0.8 4.05 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsy
> > nc
> > 4.90 ± 6% -0.8 4.05 ± 3% perf-
> > profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.fsync
> > 4.90 ± 6% -0.8 4.05 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
> > 4.90 ± 6% -0.8 4.05 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSC
> > ALL_64_after_hwframe
> > 4.90 ± 6% -0.8 4.05 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hw
> > frame.fsync
> > 5.84 ± 2% -0.8 5.01 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.write_cache_pages.xfs_vm_writepages.do_writepages.__filemap_fdat
> > awrite_range.file_write_and_wait_range
> > 1.15 ± 7% -0.8 0.36 ±100% perf-
> > profile.calltrace.cycles-
> > pp.rwsem_spin_on_owner.rwsem_down_write_failed.call_rwsem_down_writ
> > e_failed.down_write.xfs_ilock
> > 4.38 ± 7% -0.8 3.59 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_f
> > sync.do_syscall_64
> > 5.17 ± 2% -0.8 4.39 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.xfs_do_writepage.write_cache_pages.xfs_vm_writepages.do_writepag
> > es.__filemap_fdatawrite_range
> > 4.89 ± 4% -0.7 4.18 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_reques
> > t.submit_bio
> > 1.38 ± 9% -0.7 0.68 ± 22% perf-
> > profile.calltrace.cycles-
> > pp.down_write.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_w
> > ritev.do_iter_write
> > 1.38 ± 9% -0.7 0.68 ± 22% perf-
> > profile.calltrace.cycles-
> > pp.call_rwsem_down_write_failed.down_write.xfs_ilock.xfs_file_buffe
> > red_aio_write.do_iter_readv_writev
> > 1.38 ± 9% -0.7 0.68 ± 22% perf-
> > profile.calltrace.cycles-
> > pp.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.
> > xfs_ilock.xfs_file_buffered_aio_write
> > 1.38 ± 9% -0.7 0.68 ± 22% perf-
> > profile.calltrace.cycles-
> > pp.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev.do_it
> > er_write.nfsd_vfs_write
> > 4.01 ± 3% -0.6 3.40 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.brd_make_request.generic_make_request.submit_bio.xfs_add_to_ioen
> > d.xfs_do_writepage
> > 4.01 ± 3% -0.6 3.40 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.submit_bio.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages.x
> > fs_vm_writepages
> > 4.01 ± 3% -0.6 3.40 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.generic_make_request.submit_bio.xfs_add_to_ioend.xfs_do_writepag
> > e.write_cache_pages
> > 3.99 ± 3% -0.6 3.38 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.xfs
> > _add_to_ioend
> > 4.11 ± 3% -0.6 3.51 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages.xfs_vm_write
> > pages.do_writepages
> > 3.97 ± 5% -0.5 3.43 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__fi
> > lemap_fdatawrite_range
> > 3.97 ± 5% -0.5 3.43 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepag
> > es.do_writepages
> > 3.98 ± 5% -0.5 3.44 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdata
> > write_range.file_write_and_wait_range
> > 3.96 ± 5% -0.5 3.43 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.brd_make_request.generic_make_request.submit_bio.xfs_submit_ioen
> > d.xfs_vm_writepages
> > 3.92 ± 5% -0.5 3.40 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.xfs
> > _submit_ioend
> > 2.89 ± 6% -0.5 2.38 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_as
> > ync_release.process_one_work
> > 2.77 ± 10% -0.5 2.28 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file
> > _fsync.do_fsync.__x64_sys_fsync
> > 2.77 ± 10% -0.5 2.28 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.nfs_writepages.do_writepages.__filemap_fdatawrite_range.filemap_
> > write_and_wait_range.nfs_file_fsync
> > 2.77 ± 10% -0.5 2.28 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_
> > range.nfs_file_fsync.do_fsync
> > 2.74 ± 10% -0.5 2.25 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.write_cache_pages.nfs_writepages.do_writepages.__filemap_fdatawr
> > ite_range.filemap_write_and_wait_range
> > 3.12 ± 7% -0.5 2.64 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.nfs_write_begin.generic_perform_write.nfs_file_write.__vfs_write
> > .vfs_write
> > 2.93 ± 7% -0.5 2.47 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.grab_cache_page_write_begin.nfs_write_begin.generic_perform_writ
> > e.nfs_file_write.__vfs_write
> > 3.37 ± 4% -0.5 2.91 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.v
> > fs_write
> > 2.51 ± 4% -0.4 2.06 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.clear_page_erms.get_page_from_freelist.__alloc_pages_nodemask.br
> > d_insert_page.brd_do_bvec
> > 3.12 ± 5% -0.4 2.68 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_requ
> > est.generic_make_request
> > 2.91 ± 5% -0.4 2.48 ± 3% perf-
> > profile.calltrace.cycles-
> > pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_rec
> > vmsg.inet_recvmsg
> > 3.02 ± 5% -0.4 2.59 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.br
> > d_do_bvec.brd_make_request
> > 2.59 ± 4% -0.4 2.18 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_writ
> > e.__vfs_write
> > 2.84 ± 7% -0.4 2.43 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.g
> > eneric_perform_write.nfs_file_write
> > 3.27 -0.4 2.86 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iom
> > ap_file_buffered_write.xfs_file_buffered_aio_write
> > 3.20 -0.4 2.79 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.memcpy_erms.iov_iter_copy_from_user_atomic.iomap_write_actor.iom
> > ap_apply.iomap_file_buffered_write
> > 2.85 ± 5% -0.4 2.44 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.memcpy_erms._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_
> > iter.tcp_recvmsg
> > 2.50 ± 4% -0.4 2.14 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffe
> > red_write.xfs_file_buffered_aio_write
> > 2.41 ± 4% -0.4 2.06 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.
> > iomap_apply.iomap_file_buffered_write
> > 2.02 ± 11% -0.3 1.67 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writ
> > epages.__filemap_fdatawrite_range
> > 0.60 ± 8% -0.3 0.26 ±100% perf-
> > profile.calltrace.cycles-
> > pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_app
> > ly.iomap_file_buffered_write
> > 1.96 ± 11% -0.3 1.62 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.nfs_do_writepage.nfs_writepages_callback.write_cache_pages.nfs_w
> > ritepages.do_writepages
> > 1.55 ± 2% -0.3 1.22 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffere
> > d_write.xfs_file_buffered_aio_write
> > 2.35 ± 4% -0.3 2.03 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
> > .iomap_write_actor.iomap_apply
> > 1.60 ± 2% -0.3 1.32 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_
> > fsync.do_fsync.__x64_sys_fsync
> > 1.49 ± 6% -0.2 1.24 ± 6% perf-
> > profile.calltrace.cycles-
> > pp._raw_spin_lock.brd_insert_page.brd_do_bvec.brd_make_request.gene
> > ric_make_request
> > 1.03 ± 6% -0.2 0.80 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.r
> > pc_free_task.rpc_async_release
> > 0.97 ± 7% -0.2 0.74 ± 9% perf-
> > profile.calltrace.cycles-
> > pp.test_clear_page_writeback.end_page_writeback.nfs_end_page_writeb
> > ack.nfs_write_completion.rpc_free_task
> > 1.45 ± 6% -0.2 1.23 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_b
> > egin.nfs_write_begin.generic_perform_write
> > 0.97 ± 4% -0.2 0.77 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_se
> > condary
> > 1.22 ± 6% -0.2 1.03 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.nfs_commit_release_pages.nfs_commit_release.rpc_free_task.rpc_as
> > ync_release.process_one_work
> > 1.22 ± 6% -0.2 1.03 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_w
> > ork.worker_thread
> > 1.25 ± 5% -0.2 1.06 ± 8% perf-
> > profile.calltrace.cycles-
> > pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_b
> > egin.iomap_write_begin.iomap_write_actor
> > 1.06 ± 2% -0.2 0.87 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_
> > and_wait_range.nfs_file_fsync.do_fsync
> > 1.27 ± 5% -0.2 1.11 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc
> > _free_task.rpc_async_release
> > 0.78 ± 8% -0.2 0.63 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.__set_page_dirty_nobuffers.nfs_updatepage.nfs_write_end.generic_
> > perform_write.nfs_file_write
> > 1.11 ± 7% -0.1 0.99 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_wri
> > te_completion.rpc_free_task
> > 0.68 -0.1 0.57 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.nfs_create_request.nfs_updatepage.nfs_write_end.generic_perform_
> > write.nfs_file_write
> > 1.04 ± 7% -0.1 0.93 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.nfs_e
> > nd_page_writeback.nfs_write_completion
> > 1.02 ± 8% -0.1 0.92 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up
> > _page_bit.nfs_end_page_writeback
> > 0.71 ± 2% -0.1 0.60 ± 8% perf-
> > profile.calltrace.cycles-
> > pp.__schedule.schedule.io_schedule.wait_on_page_bit_common.__filema
> > p_fdatawait_range
> > 0.72 ± 2% -0.1 0.61 ± 8% perf-
> > profile.calltrace.cycles-
> > pp.schedule.io_schedule.wait_on_page_bit_common.__filemap_fdatawait
> > _range.filemap_write_and_wait_range
> > 0.72 -0.1 0.62 ± 7% perf-
> > profile.calltrace.cycles-
> > pp.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range.fi
> > lemap_write_and_wait_range.nfs_file_fsync
> > 0.93 ± 5% -0.1 0.83 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_
> > begin.nfs_write_begin.generic_perform_write
> > 0.73 -0.1 0.67 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.end_page_writeback.xfs_destroy_ioend.process_one_work.worker_thr
> > ead.kthread
> > 4.86 ± 6% +0.5 5.39 perf-
> > profile.calltrace.cycles-pp.svc_recv.nfsd.kthread.ret_from_fork
> > 0.00 +0.5 0.54 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.get_page_from_freelist.__alloc_pages_nodemask.svc_recv.nfsd.kthr
> > ead
> > 0.00 +0.6 0.59 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_se
> > ndmsg_locked
> > 0.00 +0.6 0.63 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_loc
> > ked.tcp_sendmsg
> > 0.00 +0.7 0.65 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.__alloc_pages_nodemask.svc_recv.nfsd.kthread.ret_from_fork
> > 0.00 +0.7 0.74 ± 8% perf-
> > profile.calltrace.cycles-
> > pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
> > .sock_sendmsg
> > 0.00 +0.9 0.86 ± 8% perf-
> > profile.calltrace.cycles-
> > pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_se
> > ndpages
> > 9.98 ± 4% +2.5 12.44 ± 4% perf-
> > profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
> > 9.61 ± 4% +2.5 12.15 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.process_one_work.worker_thread.kthread.ret_from_fork
> > 47.02 ± 3% +2.7 49.68 ± 4% perf-
> > profile.calltrace.cycles-pp.secondary_startup_64
> > 0.00 +3.5 3.54 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.memcpy_erms.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_lo
> > cked.tcp_sendmsg
> > 0.00 +3.6 3.56 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_locked.tcp_sen
> > dmsg.sock_sendmsg
> > 0.00 +3.7 3.67 ± 3% perf-
> > profile.calltrace.cycles-
> > pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
> > .xs_sendpages
> > 2.29 ± 6% +3.8 6.05 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_fr
> > om_fork
> > 2.28 ± 6% +3.8 6.05 ± 3% perf-
> > profile.calltrace.cycles-
> > pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.
> > kthread
> > 1.86 ± 3% +3.8 5.64 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work.
> > worker_thread
> > 1.85 ± 3% +3.8 5.64 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule.pro
> > cess_one_work
> > 1.80 ± 4% +3.8 5.60 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute.rp
> > c_async_schedule
> > 1.79 ± 3% +3.8 5.60 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit.__r
> > pc_execute
> > 0.00 +5.5 5.47 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_
> > send_request
> > 0.00 +5.6 5.56 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_t
> > ransmit
> > 0.00 +5.6 5.57 ± 4% perf-
> > profile.calltrace.cycles-
> > pp.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit.call
> > _transmit
> > 20.37 -3.3 17.04 ± 5% perf-
> > profile.children.cycles-pp.svc_process_common
> > 20.38 -3.3 17.04 ± 5% perf-
> > profile.children.cycles-pp.svc_process
> > 20.27 -3.3 16.95 ± 5% perf-
> > profile.children.cycles-pp.nfsd_dispatch
> > 20.21 -3.3 16.91 ± 5% perf-
> > profile.children.cycles-pp.nfsd4_proc_compound
> > 26.02 -3.1 22.91 ± 4% perf-
> > profile.children.cycles-pp.nfsd
> > 15.88 ± 5% -1.9 13.95 ± 3% perf-
> > profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> > 15.87 ± 5% -1.9 13.95 ± 4% perf-
> > profile.children.cycles-pp.do_syscall_64
> > 12.61 ± 3% -1.9 10.73 ± 3% perf-
> > profile.children.cycles-pp.__filemap_fdatawrite_range
> > 12.60 ± 3% -1.9 10.73 ± 3% perf-
> > profile.children.cycles-pp.do_writepages
> > 9.04 ± 2% -1.9 7.18 ± 7% perf-
> > profile.children.cycles-pp.nfsd_vfs_write
> > 9.02 ± 2% -1.9 7.17 ± 7% perf-
> > profile.children.cycles-pp.do_iter_readv_writev
> > 9.02 ± 2% -1.9 7.16 ± 7% perf-
> > profile.children.cycles-pp.xfs_file_buffered_aio_write
> > 9.03 ± 2% -1.9 7.18 ± 7% perf-
> > profile.children.cycles-pp.do_iter_write
> > 9.07 ± 2% -1.9 7.22 ± 7% perf-
> > profile.children.cycles-pp.nfsd4_write
> > 2.21 ± 4% -1.8 0.41 ± 4% perf-
> > profile.children.cycles-pp.inet_sendpage
> > 2.17 ± 4% -1.8 0.40 ± 5% perf-
> > profile.children.cycles-pp.tcp_sendpage
> > 10.50 ± 2% -1.4 9.06 ± 4% perf-
> > profile.children.cycles-pp.xfs_file_fsync
> > 10.51 ± 2% -1.4 9.08 ± 4% perf-
> > profile.children.cycles-pp.nfsd_commit
> > 10.45 ± 2% -1.4 9.02 ± 4% perf-
> > profile.children.cycles-pp.file_write_and_wait_range
> > 9.82 ± 3% -1.4 8.45 ± 3% perf-
> > profile.children.cycles-pp.xfs_vm_writepages
> > 8.59 ± 4% -1.3 7.27 ± 4% perf-
> > profile.children.cycles-pp.write_cache_pages
> > 8.02 ± 4% -1.2 6.87 ± 4% perf-
> > profile.children.cycles-pp.submit_bio
> > 8.02 ± 4% -1.2 6.87 ± 4% perf-
> > profile.children.cycles-pp.generic_make_request
> > 8.00 ± 4% -1.1 6.85 ± 4% perf-
> > profile.children.cycles-pp.brd_make_request
> > 7.94 ± 4% -1.1 6.79 ± 4% perf-
> > profile.children.cycles-pp.brd_do_bvec
> > 10.63 ± 5% -1.1 9.49 ± 4% perf-
> > profile.children.cycles-pp.write
> > 7.55 -1.1 6.41 ± 5% perf-
> > profile.children.cycles-pp.iomap_apply
> > 7.54 -1.1 6.41 ± 6% perf-
> > profile.children.cycles-pp.iomap_file_buffered_write
> > 10.28 ± 5% -1.1 9.16 ± 4% perf-
> > profile.children.cycles-pp.ksys_write
> > 7.50 -1.1 6.37 ± 6% perf-
> > profile.children.cycles-pp.iomap_write_actor
> > 4.68 ± 5% -1.1 3.56 ± 2% perf-
> > profile.children.cycles-pp.inet_recvmsg
> > 4.68 ± 5% -1.1 3.56 ± 2% perf-
> > profile.children.cycles-pp.tcp_recvmsg
> > 10.24 ± 5% -1.1 9.12 ± 4% perf-
> > profile.children.cycles-pp.vfs_write
> > 10.08 ± 5% -1.1 8.98 ± 4% perf-
> > profile.children.cycles-pp.__vfs_write
> > 10.05 ± 5% -1.1 8.95 ± 4% perf-
> > profile.children.cycles-pp.nfs_file_write
> > 5.51 ± 5% -1.1 4.43 ± 5% perf-
> > profile.children.cycles-pp.rpc_free_task
> > 9.82 ± 5% -1.1 8.74 ± 4% perf-
> > profile.children.cycles-pp.generic_perform_write
> > 1.39 ± 5% -1.1 0.32 ± 4% perf-
> > profile.children.cycles-pp.tcp_sendpage_locked
> > 1.36 ± 5% -1.1 0.31 ± 5% perf-
> > profile.children.cycles-pp.do_tcp_sendpages
> > 5.31 ± 5% -1.0 4.27 ± 5% perf-
> > profile.children.cycles-pp.rpc_async_release
> > 4.39 ± 6% -1.0 3.40 ± 2% perf-
> > profile.children.cycles-pp.svc_recvfrom
> > 3.55 ± 5% -0.9 2.64 ± 2% perf-
> > profile.children.cycles-pp.__skb_datagram_iter
> > 3.55 ± 5% -0.9 2.64 ± 2% perf-
> > profile.children.cycles-pp.skb_copy_datagram_iter
> > 5.01 ± 6% -0.9 4.15 ± 3% perf-
> > profile.children.cycles-pp.fsync
> > 4.09 ± 7% -0.9 3.24 ± 5% perf-
> > profile.children.cycles-pp.nfs_write_completion
> > 4.90 ± 6% -0.8 4.05 ± 3% perf-
> > profile.children.cycles-pp.__x64_sys_fsync
> > 4.91 ± 6% -0.8 4.07 ± 3% perf-
> > profile.children.cycles-pp.nfs_file_fsync
> > 4.90 ± 6% -0.8 4.05 ± 3% perf-
> > profile.children.cycles-pp.do_fsync
> > 5.36 ± 4% -0.8 4.53 ± 5% perf-
> > profile.children.cycles-pp.grab_cache_page_write_begin
> > 4.39 ± 7% -0.8 3.60 ± 2% perf-
> > profile.children.cycles-pp.filemap_write_and_wait_range
> > 5.18 ± 2% -0.8 4.40 ± 5% perf-
> > profile.children.cycles-pp.xfs_do_writepage
> > 5.21 ± 4% -0.7 4.47 ± 5% perf-
> > profile.children.cycles-pp.pagecache_get_page
> > 4.90 ± 4% -0.7 4.19 ± 5% perf-
> > profile.children.cycles-pp.brd_insert_page
> > 1.38 ± 9% -0.7 0.68 ± 22% perf-
> > profile.children.cycles-pp.xfs_ilock
> > 1.41 ± 8% -0.7 0.71 ± 22% perf-
> > profile.children.cycles-pp.call_rwsem_down_write_failed
> > 1.41 ± 8% -0.7 0.71 ± 22% perf-
> > profile.children.cycles-pp.rwsem_down_write_failed
> > 1.44 ± 7% -0.7 0.74 ± 21% perf-
> > profile.children.cycles-pp.down_write
> > 4.12 ± 3% -0.6 3.52 ± 4% perf-
> > profile.children.cycles-pp.xfs_add_to_ioend
> > 1.18 ± 6% -0.6 0.61 ± 21% perf-
> > profile.children.cycles-pp.rwsem_spin_on_owner
> > 3.98 ± 5% -0.5 3.44 ± 4% perf-
> > profile.children.cycles-pp.xfs_submit_ioend
> > 6.25 -0.5 5.72 ± 4% perf-
> > profile.children.cycles-pp.iov_iter_copy_from_user_atomic
> > 2.89 ± 6% -0.5 2.38 ± 6% perf-
> > profile.children.cycles-pp.nfs_end_page_writeback
> > 2.78 ± 10% -0.5 2.29 ± 2% perf-
> > profile.children.cycles-pp.nfs_writepages
> > 3.12 ± 7% -0.5 2.64 ± 4% perf-
> > profile.children.cycles-pp.nfs_write_begin
> > 3.37 ± 4% -0.5 2.91 ± 4% perf-
> > profile.children.cycles-pp.nfs_write_end
> > 2.54 ± 4% -0.5 2.08 ± 7% perf-
> > profile.children.cycles-pp.clear_page_erms
> > 2.92 ± 5% -0.4 2.48 ± 3% perf-
> > profile.children.cycles-pp._copy_to_iter
> > 2.60 ± 4% -0.4 2.18 ± 5% perf-
> > profile.children.cycles-pp.nfs_updatepage
> > 2.71 ± 4% -0.4 2.29 ± 6% perf-
> > profile.children.cycles-pp.add_to_page_cache_lru
> > 0.75 ± 3% -0.4 0.37 ± 6% perf-
> > profile.children.cycles-pp.release_sock
> > 2.51 ± 4% -0.4 2.15 ± 7% perf-
> > profile.children.cycles-pp.iomap_write_begin
> > 2.02 ± 10% -0.3 1.67 ± 3% perf-
> > profile.children.cycles-pp.nfs_writepages_callback
> > 2.23 ± 2% -0.3 1.89 ± 4% perf-
> > profile.children.cycles-pp.__filemap_fdatawait_range
> > 0.82 ± 7% -0.3 0.47 ± 7% perf-
> > profile.children.cycles-pp.__tcp_push_pending_frames
> > 1.96 ± 11% -0.3 1.63 ± 3% perf-
> > profile.children.cycles-pp.nfs_do_writepage
> > 1.55 ± 2% -0.3 1.22 ± 5% perf-
> > profile.children.cycles-pp.iomap_write_end
> > 0.78 ± 7% -0.3 0.48 ± 6% perf-
> > profile.children.cycles-pp.svc_send
> > 1.78 ± 3% -0.3 1.48 ± 4% perf-
> > profile.children.cycles-pp.end_page_writeback
> > 1.73 ± 3% -0.3 1.45 ± 4% perf-
> > profile.children.cycles-pp.test_clear_page_writeback
> > 0.90 ± 5% -0.3 0.63 ± 8% perf-
> > profile.children.cycles-pp.xas_load
> > 0.35 ± 14% -0.2 0.10 ± 14% perf-
> > profile.children.cycles-pp.simple_copy_to_iter
> > 1.78 ± 4% -0.2 1.54 ± 7% perf-
> > profile.children.cycles-pp.__wake_up_common
> > 0.43 ± 6% -0.2 0.19 ± 7% perf-
> > profile.children.cycles-pp.lock_sock_nested
> > 1.41 ± 5% -0.2 1.18 ± 4% perf-
> > profile.children.cycles-pp.nfs_commit_release
> > 1.41 ± 5% -0.2 1.18 ± 4% perf-
> > profile.children.cycles-pp.nfs_commit_release_pages
> > 1.95 ± 4% -0.2 1.72 ± 5% perf-
> > profile.children.cycles-pp.try_to_wake_up
> > 0.64 ± 8% -0.2 0.41 ± 4% perf-
> > profile.children.cycles-pp.svc_tcp_sendto
> > 0.63 ± 8% -0.2 0.41 ± 5% perf-
> > profile.children.cycles-pp.svc_sendto
> > 0.62 ± 8% -0.2 0.41 ± 5% perf-
> > profile.children.cycles-pp.svc_send_common
> > 0.62 ± 8% -0.2 0.41 ± 5% perf-
> > profile.children.cycles-pp.kernel_sendpage
> > 1.65 ± 5% -0.2 1.44 ± 7% perf-
> > profile.children.cycles-pp.autoremove_wake_function
> > 1.34 ± 3% -0.2 1.14 ± 3% perf-
> > profile.children.cycles-pp.wait_on_page_bit_common
> > 0.98 ± 4% -0.2 0.79 ± 2% perf-
> > profile.children.cycles-pp.poll_idle
> > 0.26 ± 8% -0.2 0.08 ± 13% perf-
> > profile.children.cycles-pp._raw_spin_lock_bh
> > 1.59 ± 5% -0.2 1.42 ± 5% perf-
> > profile.children.cycles-pp.schedule
> > 0.79 ± 5% -0.2 0.62 ± 4% perf-
> > profile.children.cycles-pp.tcp_v6_do_rcv
> > 0.64 ± 5% -0.2 0.47 ± 8% perf-
> > profile.children.cycles-pp.tcp_v6_rcv
> > 0.34 ± 12% -0.2 0.18 ± 20% perf-
> > profile.children.cycles-pp.xas_start
> > 0.24 ± 26% -0.2 0.08 ± 40% perf-
> > profile.children.cycles-pp.osq_lock
> > 0.76 ± 5% -0.2 0.61 ± 5% perf-
> > profile.children.cycles-pp.tcp_rcv_established
> > 0.49 ± 26% -0.2 0.34 ± 14% perf-
> > profile.children.cycles-pp.nfs_request_add_commit_list
> > 0.66 ± 5% -0.1 0.52 ± 8% perf-
> > profile.children.cycles-pp.ip6_input
> > 0.66 ± 4% -0.1 0.51 ± 8% perf-
> > profile.children.cycles-pp.ip6_input_finish
> > 0.65 ± 4% -0.1 0.51 ± 9% perf-
> > profile.children.cycles-pp.ip6_protocol_deliver_rcu
> > 0.79 ± 8% -0.1 0.64 ± 7% perf-
> > profile.children.cycles-pp.__set_page_dirty_nobuffers
> > 0.83 ± 6% -0.1 0.69 ± 8% perf-
> > profile.children.cycles-pp.__nfs_commit_inode
> > 0.42 ± 4% -0.1 0.29 ± 5% perf-
> > profile.children.cycles-pp.xs_stream_data_receive_workfn
> > 0.42 ± 4% -0.1 0.29 ± 5% perf-
> > profile.children.cycles-pp.xs_stream_data_receive
> > 0.70 ± 4% -0.1 0.56 ± 6% perf-
> > profile.children.cycles-pp.percpu_counter_add_batch
> > 0.71 ± 4% -0.1 0.58 ± 8% perf-
> > profile.children.cycles-pp.ipv6_rcv
> > 0.86 ± 6% -0.1 0.73 ± 6% perf-
> > profile.children.cycles-pp.__local_bh_enable_ip
> > 0.78 ± 5% -0.1 0.65 ± 8% perf-
> > profile.children.cycles-pp.process_backlog
> > 0.45 ± 14% -0.1 0.32 ± 14% perf-
> > profile.children.cycles-pp.__mutex_lock
> > 1.06 ± 4% -0.1 0.93 ± 7% perf-
> > profile.children.cycles-pp._raw_spin_lock_irqsave
> > 0.86 ± 6% -0.1 0.73 ± 9% perf-
> > profile.children.cycles-pp.pagevec_lru_move_fn
> > 0.75 ± 5% -0.1 0.63 ± 8% perf-
> > profile.children.cycles-pp.__netif_receive_skb_one_core
> > 0.61 ± 8% -0.1 0.49 ± 9% perf-
> > profile.children.cycles-pp.iomap_set_page_dirty
> > 0.94 ± 5% -0.1 0.81 ± 8% perf-
> > profile.children.cycles-pp.__lru_cache_add
> > 0.82 ± 6% -0.1 0.70 ± 2% perf-
> > profile.children.cycles-pp.clear_page_dirty_for_io
> > 0.80 ± 6% -0.1 0.68 ± 7% perf-
> > profile.children.cycles-pp.net_rx_action
> > 0.94 ± 2% -0.1 0.82 ± 3% perf-
> > profile.children.cycles-pp.io_schedule
> > 0.81 ± 7% -0.1 0.70 ± 7% perf-
> > profile.children.cycles-pp.do_softirq_own_stack
> > 0.69 -0.1 0.58 ± 5% perf-
> > profile.children.cycles-pp.nfs_create_request
> > 0.62 ± 4% -0.1 0.51 ± 5% perf-
> > profile.children.cycles-pp.__pagevec_lru_add_fn
> > 0.32 ± 4% -0.1 0.21 ± 11% perf-
> > profile.children.cycles-pp.xs_sock_recvmsg
> > 0.49 ± 12% -0.1 0.38 ± 7% perf-
> > profile.children.cycles-pp.sched_ttwu_pending
> > 0.82 ± 6% -0.1 0.72 ± 7% perf-
> > profile.children.cycles-pp.do_softirq
> > 0.32 ± 10% -0.1 0.22 ± 7% perf-
> > profile.children.cycles-pp.xfs_map_blocks
> > 0.40 ± 13% -0.1 0.30 ± 12% perf-
> > profile.children.cycles-pp.mutex_spin_on_owner
> > 0.47 ± 2% -0.1 0.38 ± 5% perf-
> > profile.children.cycles-pp.kmem_cache_alloc
> > 0.26 ± 4% -0.1 0.17 ± 8% perf-
> > profile.children.cycles-pp.__wake_up_common_lock
> > 0.52 ± 6% -0.1 0.43 ± 4% perf-
> > profile.children.cycles-pp.account_page_dirtied
> > 0.22 ± 16% -0.1 0.13 ± 9% perf-
> > profile.children.cycles-pp.__check_object_size
> > 0.50 ± 8% -0.1 0.42 ± 7% perf-
> > profile.children.cycles-pp.iomap_set_range_uptodate
> > 0.41 ± 3% -0.1 0.33 ± 14% perf-
> > profile.children.cycles-pp.__set_page_dirty
> > 0.31 ± 11% -0.1 0.24 ± 11% perf-
> > profile.children.cycles-pp.nfs_io_completion_release
> > 0.41 ± 8% -0.1 0.33 ± 7% perf-
> > profile.children.cycles-pp.nfs_inode_remove_request
> > 0.25 ± 9% -0.1 0.18 ± 8% perf-
> > profile.children.cycles-pp.xfs_iomap_write_allocate
> > 0.33 ± 8% -0.1 0.26 ± 4% perf-
> > profile.children.cycles-pp.___might_sleep
> > 0.25 ± 5% -0.1 0.18 ± 15% perf-
> > profile.children.cycles-pp.__xa_set_mark
> > 0.31 ± 9% -0.1 0.24 ± 5% perf-
> > profile.children.cycles-pp.nfs_scan_commit_list
> > 0.25 ± 5% -0.1 0.18 ± 8% perf-
> > profile.children.cycles-pp.__x86_indirect_thunk_rax
> > 0.45 ± 6% -0.1 0.39 ± 4% perf-
> > profile.children.cycles-pp.nfs_lock_and_join_requests
> > 0.50 ± 4% -0.1 0.45 ± 5% perf-
> > profile.children.cycles-pp.nfs_page_group_destroy
> > 0.19 ± 11% -0.1 0.14 ± 8% perf-
> > profile.children.cycles-pp.xfs_bmap_btalloc
> > 0.21 ± 2% -0.1 0.15 ± 14% perf-
> > profile.children.cycles-pp.__lock_sock
> > 0.17 ± 8% -0.1 0.12 ± 7% perf-
> > profile.children.cycles-pp.xfs_alloc_ag_vextent
> > 0.18 ± 9% -0.1 0.13 ± 6% perf-
> > profile.children.cycles-pp.xfs_alloc_vextent
> > 0.20 ± 12% -0.1 0.15 ± 5% perf-
> > profile.children.cycles-pp.xfs_bmapi_write
> > 0.30 ± 9% -0.0 0.25 ± 5% perf-
> > profile.children.cycles-pp.select_task_rq_fair
> > 0.16 ± 8% -0.0 0.11 ± 7% perf-
> > profile.children.cycles-pp.xfs_alloc_ag_vextent_near
> > 0.16 ± 15% -0.0 0.12 ± 17% perf-
> > profile.children.cycles-pp.__generic_write_end
> > 0.48 ± 2% -0.0 0.44 ± 7% perf-
> > profile.children.cycles-pp.dequeue_task_fair
> > 0.31 ± 2% -0.0 0.27 ± 10% perf-
> > profile.children.cycles-pp.release_pages
> > 0.20 ± 4% -0.0 0.17 ± 10% perf-
> > profile.children.cycles-pp.nfs_initiate_commit
> > 0.17 ± 10% -0.0 0.13 ± 5% perf-
> > profile.children.cycles-pp.__might_sleep
> > 0.29 ± 6% -0.0 0.26 ± 4% perf-
> > profile.children.cycles-pp.dec_zone_page_state
> > 0.23 ± 7% -0.0 0.20 ± 7% perf-
> > profile.children.cycles-pp.__pagevec_release
> > 0.19 ± 9% -0.0 0.15 ± 9% perf-
> > profile.children.cycles-pp.nfs_get_lock_context
> > 0.17 ± 13% -0.0 0.13 ± 8% perf-
> > profile.children.cycles-pp.check_preempt_curr
> > 0.09 ± 17% -0.0 0.06 ± 20% perf-
> > profile.children.cycles-pp.memset_erms
> > 0.14 ± 8% -0.0 0.11 ± 6% perf-
> > profile.children.cycles-pp.nfs_request_remove_commit_list
> > 0.07 ± 10% -0.0 0.04 ± 58% perf-
> > profile.children.cycles-pp.xfs_defer_finish_noroll
> > 0.15 ± 5% -0.0 0.12 ± 6% perf-
> > profile.children.cycles-pp.__nfs_find_lock_context
> > 0.19 ± 4% -0.0 0.16 ± 11% perf-
> > profile.children.cycles-pp.mem_cgroup_try_charge
> > 0.17 ± 8% -0.0 0.15 ± 10% perf-
> > profile.children.cycles-pp.vfs_create
> > 0.16 ± 2% -0.0 0.14 ± 8% perf-
> > profile.children.cycles-pp.__fprop_inc_percpu_max
> > 0.16 ± 8% -0.0 0.13 ± 8% perf-
> > profile.children.cycles-pp.xfs_create
> > 0.10 ± 4% -0.0 0.07 ± 17% perf-
> > profile.children.cycles-pp.xfs_file_aio_write_checks
> > 0.11 ± 10% -0.0 0.08 ± 15% perf-
> > profile.children.cycles-pp.nfs_pageio_doio
> > 0.11 ± 10% -0.0 0.08 ± 15% perf-
> > profile.children.cycles-pp.nfs_generic_pg_pgios
> > 0.19 ± 4% -0.0 0.17 ± 7% perf-
> > profile.children.cycles-pp.update_rq_clock
> > 0.15 ± 4% -0.0 0.13 ± 5% perf-
> > profile.children.cycles-pp.__xfs_trans_commit
> > 0.08 ± 15% -0.0 0.06 ± 6% perf-
> > profile.children.cycles-pp.get_mem_cgroup_from_mm
> > 0.06 ± 15% +0.0 0.07 ± 14% perf-
> > profile.children.cycles-pp.selinux_ip_postroute
> > 0.08 ± 16% +0.0 0.10 ± 10% perf-
> > profile.children.cycles-pp.nf_hook_slow
> > 0.04 ± 60% +0.0 0.08 ± 19% perf-
> > profile.children.cycles-pp.__inc_numa_state
> > 0.03 ±100% +0.0 0.07 ± 12% perf-
> > profile.children.cycles-pp.get_task_policy
> > 0.41 ± 6% +0.0 0.46 perf-
> > profile.children.cycles-pp.__release_sock
> > 0.14 ± 11% +0.1 0.19 ± 8% perf-
> > profile.children.cycles-pp.__list_add_valid
> > 0.09 ± 17% +0.1 0.14 ± 10% perf-
> > profile.children.cycles-pp.svc_xprt_do_enqueue
> > 0.14 ± 10% +0.1 0.20 ± 2% perf-
> > profile.children.cycles-pp.tcp_clean_rtx_queue
> > 0.00 +0.1 0.06 ± 11% perf-
> > profile.children.cycles-pp.free_unref_page_commit
> > 0.10 ± 11% +0.1 0.17 ± 6% perf-
> > profile.children.cycles-pp.iov_iter_advance
> > 0.17 ± 9% +0.1 0.24 ± 2% perf-
> > profile.children.cycles-pp.tcp_ack
> > 0.11 ± 21% +0.1 0.18 ± 2% perf-
> > profile.children.cycles-pp.schedule_timeout
> > 0.00 +0.1 0.08 ± 17% perf-
> > profile.children.cycles-pp.free_one_page
> > 0.23 ± 12% +0.1 0.32 ± 4% perf-
> > profile.children.cycles-pp.__kfree_skb
> > 0.10 ± 4% +0.1 0.19 ± 6% perf-
> > profile.children.cycles-pp.xas_create
> > 0.12 ± 17% +0.1 0.24 ± 4% perf-
> > profile.children.cycles-pp.skb_release_data
> > 0.00 +0.1 0.14 ± 5% perf-
> > profile.children.cycles-pp.__free_pages_ok
> > 0.00 +0.2 0.21 ± 9% perf-
> > profile.children.cycles-pp.skb_page_frag_refill
> > 0.00 +0.2 0.22 ± 9% perf-
> > profile.children.cycles-pp.sk_page_frag_refill
> > 0.00 +0.2 0.25 ± 5% perf-
> > profile.children.cycles-pp.__sk_flush_backlog
> > 0.03 ±100% +0.4 0.39 ± 7% perf-
> > profile.children.cycles-pp.free_pcppages_bulk
> > 0.07 ± 29% +0.5 0.54 ± 6% perf-
> > profile.children.cycles-pp.free_unref_page
> > 4.86 ± 6% +0.5 5.40 ± 2% perf-
> > profile.children.cycles-pp.svc_recv
> > 9.98 ± 4% +2.5 12.44 ± 4% perf-
> > profile.children.cycles-pp.worker_thread
> > 9.61 ± 4% +2.5 12.15 ± 4% perf-
> > profile.children.cycles-pp.process_one_work
> > 47.02 ± 3% +2.7 49.68 ± 4% perf-
> > profile.children.cycles-pp.secondary_startup_64
> > 47.02 ± 3% +2.7 49.68 ± 4% perf-
> > profile.children.cycles-pp.cpu_startup_entry
> > 47.02 ± 3% +2.7 49.69 ± 4% perf-
> > profile.children.cycles-pp.do_idle
> > 6.12 ± 2% +2.8 8.88 ± 3% perf-
> > profile.children.cycles-pp.memcpy_erms
> > 0.00 +3.6 3.58 ± 3% perf-
> > profile.children.cycles-pp.memcpy_from_page
> > 0.00 +3.7 3.68 ± 3% perf-
> > profile.children.cycles-pp._copy_from_iter_full
> > 2.43 ± 6% +3.8 6.20 ± 3% perf-
> > profile.children.cycles-pp.__rpc_execute
> > 2.29 ± 6% +3.8 6.05 ± 3% perf-
> > profile.children.cycles-pp.rpc_async_schedule
> > 1.93 ± 3% +3.8 5.71 ± 4% perf-
> > profile.children.cycles-pp.call_transmit
> > 1.92 ± 4% +3.8 5.71 ± 4% perf-
> > profile.children.cycles-pp.xprt_transmit
> > 1.87 ± 4% +3.8 5.67 ± 4% perf-
> > profile.children.cycles-pp.xs_tcp_send_request
> > 1.86 ± 4% +3.8 5.67 ± 4% perf-
> > profile.children.cycles-pp.xs_sendpages
> > 0.21 ± 7% +5.3 5.54 ± 4% perf-
> > profile.children.cycles-pp.tcp_sendmsg_locked
> > 0.24 ± 8% +5.4 5.63 ± 4% perf-
> > profile.children.cycles-pp.tcp_sendmsg
> > 0.25 ± 9% +5.4 5.64 ± 4% perf-
> > profile.children.cycles-pp.sock_sendmsg
> > 1.17 ± 7% -0.6 0.61 ± 21% perf-
> > profile.self.cycles-pp.rwsem_spin_on_owner
> > 2.50 ± 4% -0.5 2.05 ± 7% perf-
> > profile.self.cycles-pp.clear_page_erms
> > 2.46 ± 4% -0.4 2.06 ± 5% perf-
> > profile.self.cycles-pp.brd_do_bvec
> > 0.22 ± 10% -0.2 0.03 ±100% perf-
> > profile.self.cycles-pp.__skb_datagram_iter
> > 1.53 ± 6% -0.2 1.34 ± 3% perf-
> > profile.self.cycles-pp._raw_spin_lock
> > 0.91 ± 4% -0.2 0.74 ± 2% perf-
> > profile.self.cycles-pp.poll_idle
> > 0.24 ± 25% -0.2 0.08 ± 40% perf-
> > profile.self.cycles-pp.osq_lock
> > 0.74 ± 14% -0.2 0.59 ± 10% perf-
> > profile.self.cycles-pp.nfs_do_writepage
> > 0.32 ± 13% -0.1 0.17 ± 18% perf-
> > profile.self.cycles-pp.xas_start
> > 0.97 ± 3% -0.1 0.84 ± 8% perf-
> > profile.self.cycles-pp._raw_spin_lock_irqsave
> > 0.57 ± 6% -0.1 0.45 ± 9% perf-
> > profile.self.cycles-pp.xas_load
> > 0.55 ± 10% -0.1 0.43 ± 7% perf-
> > profile.self.cycles-pp.__schedule
> > 0.14 ± 12% -0.1 0.03 ±100% perf-
> > profile.self.cycles-pp._raw_spin_lock_bh
> > 0.53 ± 8% -0.1 0.43 ± 9% perf-
> > profile.self.cycles-pp.percpu_counter_add_batch
> > 0.40 ± 13% -0.1 0.30 ± 12% perf-
> > profile.self.cycles-pp.mutex_spin_on_owner
> > 0.33 ± 8% -0.1 0.24 ± 12% perf-
> > profile.self.cycles-pp.nfs_end_page_writeback
> > 0.74 ± 6% -0.1 0.65 ± 8% perf-
> > profile.self.cycles-pp.nfs_updatepage
> > 0.40 ± 8% -0.1 0.31 ± 11% perf-
> > profile.self.cycles-pp.__test_set_page_writeback
> > 0.50 ± 8% -0.1 0.41 ± 7% perf-
> > profile.self.cycles-pp.iomap_set_range_uptodate
> > 0.56 ± 5% -0.1 0.48 ± 8% perf-
> > profile.self.cycles-pp.try_to_wake_up
> > 0.38 ± 8% -0.1 0.30 ± 5% perf-
> > profile.self.cycles-pp._raw_spin_unlock_irqrestore
> > 0.28 ± 9% -0.1 0.20 ± 6% perf-
> > profile.self.cycles-pp.iomap_write_end
> > 0.45 ± 7% -0.1 0.38 ± 6% perf-
> > profile.self.cycles-pp.__pagevec_lru_add_fn
> > 0.15 ± 14% -0.1 0.08 ± 27% perf-
> > profile.self.cycles-pp.add_to_page_cache_lru
> > 0.43 ± 6% -0.1 0.36 perf-
> > profile.self.cycles-pp.clear_page_dirty_for_io
> > 0.51 ± 8% -0.1 0.45 ± 7% perf-
> > profile.self.cycles-pp.test_clear_page_writeback
> > 0.20 ± 9% -0.1 0.14 ± 11% perf-
> > profile.self.cycles-pp.xas_store
> > 0.31 ± 8% -0.1 0.25 ± 4% perf-
> > profile.self.cycles-pp.___might_sleep
> > 0.15 ± 20% -0.1 0.09 ± 12% perf-
> > profile.self.cycles-pp.__check_object_size
> > 0.22 ± 6% -0.1 0.17 ± 10% perf-
> > profile.self.cycles-pp.__x86_indirect_thunk_rax
> > 0.08 ± 11% -0.0 0.03 ±100% perf-
> > profile.self.cycles-pp.switch_mm_irqs_off
> > 0.28 ± 4% -0.0 0.24 ± 10% perf-
> > profile.self.cycles-pp.release_pages
> > 0.27 ± 8% -0.0 0.23 ± 2% perf-
> > profile.self.cycles-pp.dec_zone_page_state
> > 0.19 ± 6% -0.0 0.15 ± 12% perf-
> > profile.self.cycles-pp.kmem_cache_alloc
> > 0.15 ± 10% -0.0 0.11 ± 7% perf-
> > profile.self.cycles-pp.__might_sleep
> > 0.25 ± 10% -0.0 0.21 ± 6% perf-
> > profile.self.cycles-pp.wait_on_page_bit_common
> > 0.07 ± 12% -0.0 0.03 ±100% perf-
> > profile.self.cycles-pp.___slab_alloc
> > 0.23 ± 7% -0.0 0.20 ± 10% perf-
> > profile.self.cycles-pp._raw_spin_lock_irq
> > 0.18 ± 5% -0.0 0.15 ± 2% perf-
> > profile.self.cycles-pp.select_task_rq_fair
> > 0.14 ± 5% -0.0 0.11 ± 17% perf-
> > profile.self.cycles-pp.alloc_pages_current
> > 0.09 ± 16% -0.0 0.06 ± 20% perf-
> > profile.self.cycles-pp.memset_erms
> > 0.10 ± 8% -0.0 0.07 perf-
> > profile.self.cycles-pp._cond_resched
> > 0.15 ± 4% -0.0 0.12 ± 6% perf-
> > profile.self.cycles-pp.__nfs_find_lock_context
> > 0.11 ± 13% -0.0 0.09 ± 14% perf-
> > profile.self.cycles-pp.nfs_request_add_commit_list_locked
> > 0.09 ± 4% -0.0 0.07 perf-
> > profile.self.cycles-pp.nfs_request_remove_commit_list
> > 0.08 ± 10% -0.0 0.06 ± 11% perf-
> > profile.self.cycles-pp.get_mem_cgroup_from_mm
> > 0.08 ± 10% -0.0 0.06 ± 11% perf-
> > profile.self.cycles-pp.nfs_request_add_commit_list
> > 0.10 ± 5% -0.0 0.08 ± 5% perf-
> > profile.self.cycles-pp.___perf_sw_event
> > 0.08 ± 13% +0.0 0.11 ± 10% perf-
> > profile.self.cycles-pp.mem_cgroup_commit_charge
> > 0.03 ±100% +0.0 0.07 ± 12% perf-
> > profile.self.cycles-pp.run_timer_softirq
> > 0.03 ±100% +0.0 0.07 ± 12% perf-
> > profile.self.cycles-pp.get_task_policy
> > 0.03 ±100% +0.0 0.07 ± 17% perf-
> > profile.self.cycles-pp.__inc_numa_state
> > 0.03 ±102% +0.1 0.08 ± 8% perf-
> > profile.self.cycles-pp.svc_recv
> > 0.11 ± 13% +0.1 0.17 ± 12% perf-
> > profile.self.cycles-pp.__list_add_valid
> > 0.09 ± 14% +0.1 0.16 ± 9% perf-
> > profile.self.cycles-pp.iov_iter_advance
> > 0.06 ± 13% +0.1 0.15 ± 8% perf-
> > profile.self.cycles-pp.xas_create
> > 0.00 +0.1 0.15 ± 14% perf-
> > profile.self.cycles-pp.tcp_sendmsg_locked
> > 0.00 +0.2 0.20 ± 11% perf-
> > profile.self.cycles-pp.free_pcppages_bulk
> > 0.04 ± 60% +0.2 0.28 ± 7% perf-
> > profile.self.cycles-pp.svc_tcp_recvfrom
> > 1.25 +0.3 1.52 ± 5% perf-
> > profile.self.cycles-pp.get_page_from_freelist
> > 6.04 ± 2% +2.7 8.74 ± 3% perf-
> > profile.self.cycles-pp.memcpy_erms
> >
> >
> >
> >
> > fsmark.time.percent_of_cpu_this_job_got
> >
> >
> > 320 +-+---------------------------------------------------------
> > ----------+
> > |
> > |
> > 300 +-
> > ++. +
> > |
> > |. +.. .+. +.. +.. +.
> > .. + .+.|
> > 280 +-+ +.+. +..
> > : + .+..+. .+. .+. .+.+.. .. + +. |
> > | : + + +. +. +. .+
> > |
> > 260 +-
> > + + +
> > |
> > |
> > |
> > 240 +-
> > +
> > |
> > |
> > |
> > 220 +-
> > +
> > |
> > | O O O O O
> > O O |
> > 200 O-+ O O O O O O
> > O O |
> > | O O O O O O O O
> > O O |
> > 180 +-+---------------------------------------------------------
> > ----------+
> >
> >
> >
> >
> >
> > fsmark.time.elapsed_time
> >
> >
> > 32 +-+O---------------------------------------------------------
> > ----------+
> > O O O O O O O O O O O
> > O |
> > 31 +-+ O O O O O O O O O
> > O O O |
> > 30 +-
> > + O
> > |
> > |
> > |
> > 29 +-
> > +
> > |
> > 28 +-
> > +
> > |
> > |
> > |
> > 27 +-
> > +
> > |
> > 26 +-
> > + .+
> > |
> > | + + +.+..
> > .+. + .+. .+. |
> > 25 +-+ .. + .+ + : +..+.+..+.
> > .. + +. +. +..|
> > 24 +-+ .+.+..+.+ +. + + :
> > .. + |
> > | +.+. + +
> > |
> > 23 +-+--------------------------------------------------------
> > ------------+
> >
> >
> >
> >
> >
> > fsmark.time.elapsed_time.max
> >
> >
> > 32 +-+O---------------------------------------------------------
> > ----------+
> > O O O O O O O O O O O
> > O |
> > 31 +-+ O O O O O O O O O
> > O O O |
> > 30 +-
> > + O
> > |
> > |
> > |
> > 29 +-
> > +
> > |
> > 28 +-
> > +
> > |
> > |
> > |
> > 27 +-
> > +
> > |
> > 26 +-
> > + .+
> > |
> > | + + +.+..
> > .+. + .+. .+. |
> > 25 +-+ .. + .+ + : +..+.+..+.
> > .. + +. +. +..|
> > 24 +-+ .+.+..+.+ +. + + :
> > .. + |
> > | +.+. + +
> > |
> > 23 +-+--------------------------------------------------------
> > ------------+
> >
> >
> >
> >
> >
> > fsmark.files_per_sec
> >
> >
> > 520 +-+---------------------------------------------------------
> > ----------+
> > |.. +.+. + +.. .+..
> > : .+. .+.. .+.+..+ |
> > 500 +-+ +..
> > + + : +.+..+.+. +..+ .+.+. |
> > 480 +-
> > + + + +.+.
> > |
> > |
> > |
> > 460 +-
> > +
> > |
> > |
> > |
> > 440 +-
> > +
> > |
> > |
> > |
> > 420 +-
> > +
> > |
> > 400 +-
> > + O
> > |
> > | O O O O O O O
> > O O |
> > 380 O-+ O O O O O O O
> > O O |
> > | O O O O O
> > O |
> > 360 +-+---------------------------------------------------------
> > ----------+
> >
> >
> >
> >
> > [*] bisect-good sample
> > [O] bisect-bad sample
> >
> >
> >
> > Disclaimer:
> > Results have been estimated based on internal Intel analysis and
> > are provided
> > for informational purposes only. Any difference in system hardware
> > or software
> > design or configuration may affect actual performance.
> >
> >
> > Thanks,
> > Rong Chen
> >
> >
> > _______________________________________________
> > LKP mailing list
> > [email protected]
> > https://lists.01.org/mailman/listinfo/lkp
> >
>
> Do you have time to take a look at this regression?
From your stats, it looks to me as if the problem is increased NUMA
overhead. Pretty much everything else appears to be the same or
actually performing better than previously. Am I interpreting that
correctly?
If my interpretation above is correct, then I'm not seeing where this
patch would be introducing new NUMA regressions. It is just converting
from using one method of doing socket I/O to another. Could it perhaps
be a memory artefact due to your running the NFS client and server on
the same machine?
Apologies for pushing back a little, but I just don't have the
hardware available to test NUMA configurations, so I'm relying on
external testing for the above kind of scenario.
Thanks
Trond
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
On 5/30/2019 10:00 AM, Trond Myklebust wrote:
> Hi Xing,
>
> On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote:
>> Hi Trond,
>>
>> On 5/20/2019 1:54 PM, kernel test robot wrote:
>>> Greeting,
>>>
>>> FYI, we noticed a 16.0% improvement of fsmark.app_overhead due to
>>> commit:
>>>
>>>
>>> commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC: Convert
>>> socket page send code to use iov_iter()")
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
>>> master
>>>
>>> in testcase: fsmark
>>> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @
>>> 3.00GHz with 384G memory
>>> with following parameters:
>>>
>>> iterations: 1x
>>> nr_threads: 64t
>>> disk: 1BRD_48G
>>> fs: xfs
>>> fs2: nfsv4
>>> filesize: 4M
>>> test_size: 40G
>>> sync_method: fsyncBeforeClose
>>> cpufreq_governor: performance
>>>
>>> test-description: The fsmark is a file system benchmark to test
>>> synchronous write workloads, for example, mail servers workload.
>>> test-url: https://sourceforge.net/projects/fsmark/
>>>
>>>
>>>
>>> Details are as below:
>>> -----------------------------------------------------------------
>>> --------------------------------->
>>>
>>>
>>> To reproduce:
>>>
>>> git clone https://github.com/intel/lkp-tests.git
>>> cd lkp-tests
>>> bin/lkp install job.yaml # job file is attached in this
>>> email
>>> bin/lkp run job.yaml
>>>
>>> ===================================================================
>>> ======================
>>> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/n
>>> r_threads/rootfs/sync_method/tbox_group/test_size/testcase:
>>> gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-
>>> 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-
>>> ep01/40G/fsmark
>>>
>>> commit:
>>> e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use
>>> iov_iter_kvec()")
>>> 0472e47660 ("SUNRPC: Convert socket page send code to use
>>> iov_iter()")
>>>
>>> e791f8e9380d945e 0472e476604998c127f3c80d291
>>> ---------------- ---------------------------
>>> fail:runs %reproduction fail:runs
>>> | | |
>>> :4 50% 2:4 dmesg.WARNING:at#for
>>> _ip_interrupt_entry/0x
>>> %stddev %change %stddev
>>> \ | \
>>> 15118573 ± 2% +16.0% 17538083 fsmark.app_overhead
>>> 510.93 -22.7% 395.12 fsmark.files_per_sec
>>> 24.90 +22.8% 30.57 fsmark.time.elapsed_
>>> time
>>> 24.90 +22.8% 30.57 fsmark.time.elapsed_
>>> time.max
>>> 288.00 ± 2% -
>>> 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got
>>> 70.03 ± 2% -
>>> 11.3% 62.14 fsmark.time.system_time
>>>
>>
>> Do you have time to take a look at this regression?
>
> From your stats, it looks to me as if the problem is increased NUMA
> overhead. Pretty much everything else appears to be the same or
> actually performing better than previously. Am I interpreting that
> correctly?
The real regression is the throughput(fsmark.files_per_sec) is decreased
by 22.7%.
>
> If my interpretation above is correct, then I'm not seeing where this
> patch would be introducing new NUMA regressions. It is just converting
> from using one method of doing socket I/O to another. Could it perhaps
> be a memory artefact due to your running the NFS client and server on
> the same machine?
>
> Apologies for pushing back a little, but I just don't have the
> hardware available to test NUMA configurations, so I'm relying on
> external testing for the above kind of scenario.
>
Thanks for looking at this. If you need more information, please let me
know.
> Thanks
> Trond
>
--
Zhengjun Xing
On Thu, 2019-05-30 at 15:20 +0800, Xing Zhengjun wrote:
>
> On 5/30/2019 10:00 AM, Trond Myklebust wrote:
> > Hi Xing,
> >
> > On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote:
> > > Hi Trond,
> > >
> > > On 5/20/2019 1:54 PM, kernel test robot wrote:
> > > > Greeting,
> > > >
> > > > FYI, we noticed a 16.0% improvement of fsmark.app_overhead due
> > > > to
> > > > commit:
> > > >
> > > >
> > > > commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC:
> > > > Convert
> > > > socket page send code to use iov_iter()")
> > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
> > > > master
> > > >
> > > > in testcase: fsmark
> > > > on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @
> > > > 3.00GHz with 384G memory
> > > > with following parameters:
> > > >
> > > > iterations: 1x
> > > > nr_threads: 64t
> > > > disk: 1BRD_48G
> > > > fs: xfs
> > > > fs2: nfsv4
> > > > filesize: 4M
> > > > test_size: 40G
> > > > sync_method: fsyncBeforeClose
> > > > cpufreq_governor: performance
> > > >
> > > > test-description: The fsmark is a file system benchmark to test
> > > > synchronous write workloads, for example, mail servers
> > > > workload.
> > > > test-url: https://sourceforge.net/projects/fsmark/
> > > >
> > > >
> > > >
> > > > Details are as below:
> > > > -------------------------------------------------------------
> > > > ----
> > > > --------------------------------->
> > > >
> > > >
> > > > To reproduce:
> > > >
> > > > git clone https://github.com/intel/lkp-tests.git
> > > > cd lkp-tests
> > > > bin/lkp install job.yaml # job file is attached in
> > > > this
> > > > email
> > > > bin/lkp run job.yaml
> > > >
> > > > ===============================================================
> > > > ====
> > > > ======================
> > > > compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconf
> > > > ig/n
> > > > r_threads/rootfs/sync_method/tbox_group/test_size/testcase:
> > > > gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-
> > > > 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-
> > > > ep01/40G/fsmark
> > > >
> > > > commit:
> > > > e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use
> > > > iov_iter_kvec()")
> > > > 0472e47660 ("SUNRPC: Convert socket page send code to use
> > > > iov_iter()")
> > > >
> > > > e791f8e9380d945e 0472e476604998c127f3c80d291
> > > > ---------------- ---------------------------
> > > > fail:runs %reproduction fail:runs
> > > > | | |
> > > > :4 50% 2:4 dmesg.WARNING:a
> > > > t#for
> > > > _ip_interrupt_entry/0x
> > > > %stddev %change %stddev
> > > > \ | \
> > > > 15118573
> > > > ± 2% +16.0% 17538083 fsmark.app_overhead
> > > > 510.93 -
> > > > 22.7% 395.12 fsmark.files_per_sec
> > > > 24.90 +22.8% 30.57 fsmark.time.ela
> > > > psed_
> > > > time
> > > > 24.90 +22.8% 30.57 fsmark.time.ela
> > > > psed_
> > > > time.max
> > > > 288.00 ± 2% -
> > > > 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got
> > > > 70.03 ± 2% -
> > > > 11.3% 62.14 fsmark.time.system_time
> > > >
> > >
> > > Do you have time to take a look at this regression?
> >
> > From your stats, it looks to me as if the problem is increased
> > NUMA
> > overhead. Pretty much everything else appears to be the same or
> > actually performing better than previously. Am I interpreting that
> > correctly?
> The real regression is the throughput(fsmark.files_per_sec) is
> decreased
> by 22.7%.
Understood, but I'm trying to make sense of why. I'm not able to
reproduce this, so I have to rely on your performance stats to
understand where the 22.7% regression is coming from. As far as I can
see, the only numbers in the stats you published that are showing a
performance regression (other than the fsmark number itself), are the
NUMA numbers. Is that a correct interpretation?
> > If my interpretation above is correct, then I'm not seeing where
> > this
> > patch would be introducing new NUMA regressions. It is just
> > converting
> > from using one method of doing socket I/O to another. Could it
> > perhaps
> > be a memory artefact due to your running the NFS client and server
> > on
> > the same machine?
> >
> > Apologies for pushing back a little, but I just don't have the
> > hardware available to test NUMA configurations, so I'm relying on
> > external testing for the above kind of scenario.
> >
> Thanks for looking at this. If you need more information, please let
> me
> know.
> > Thanks
> > Trond
> >
--
Trond Myklebust
CTO, Hammerspace Inc
4300 El Camino Real, Suite 105
Los Altos, CA 94022
http://www.hammer.space
On 5/31/2019 3:10 AM, Trond Myklebust wrote:
> On Thu, 2019-05-30 at 15:20 +0800, Xing Zhengjun wrote:
>>
>> On 5/30/2019 10:00 AM, Trond Myklebust wrote:
>>> Hi Xing,
>>>
>>> On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote:
>>>> Hi Trond,
>>>>
>>>> On 5/20/2019 1:54 PM, kernel test robot wrote:
>>>>> Greeting,
>>>>>
>>>>> FYI, we noticed a 16.0% improvement of fsmark.app_overhead due
>>>>> to
>>>>> commit:
>>>>>
>>>>>
>>>>> commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC:
>>>>> Convert
>>>>> socket page send code to use iov_iter()")
>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
>>>>> master
>>>>>
>>>>> in testcase: fsmark
>>>>> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @
>>>>> 3.00GHz with 384G memory
>>>>> with following parameters:
>>>>>
>>>>> iterations: 1x
>>>>> nr_threads: 64t
>>>>> disk: 1BRD_48G
>>>>> fs: xfs
>>>>> fs2: nfsv4
>>>>> filesize: 4M
>>>>> test_size: 40G
>>>>> sync_method: fsyncBeforeClose
>>>>> cpufreq_governor: performance
>>>>>
>>>>> test-description: The fsmark is a file system benchmark to test
>>>>> synchronous write workloads, for example, mail servers
>>>>> workload.
>>>>> test-url: https://sourceforge.net/projects/fsmark/
>>>>>
>>>>>
>>>>>
>>>>> Details are as below:
>>>>> -------------------------------------------------------------
>>>>> ----
>>>>> --------------------------------->
>>>>>
>>>>>
>>>>> To reproduce:
>>>>>
>>>>> git clone https://github.com/intel/lkp-tests.git
>>>>> cd lkp-tests
>>>>> bin/lkp install job.yaml # job file is attached in
>>>>> this
>>>>> email
>>>>> bin/lkp run job.yaml
>>>>>
>>>>> ===============================================================
>>>>> ====
>>>>> ======================
>>>>> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconf
>>>>> ig/n
>>>>> r_threads/rootfs/sync_method/tbox_group/test_size/testcase:
>>>>> gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-
>>>>> 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-
>>>>> ep01/40G/fsmark
>>>>>
>>>>> commit:
>>>>> e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use
>>>>> iov_iter_kvec()")
>>>>> 0472e47660 ("SUNRPC: Convert socket page send code to use
>>>>> iov_iter()")
>>>>>
>>>>> e791f8e9380d945e 0472e476604998c127f3c80d291
>>>>> ---------------- ---------------------------
>>>>> fail:runs %reproduction fail:runs
>>>>> | | |
>>>>> :4 50% 2:4 dmesg.WARNING:a
>>>>> t#for
>>>>> _ip_interrupt_entry/0x
>>>>> %stddev %change %stddev
>>>>> \ | \
>>>>> 15118573
>>>>> ± 2% +16.0% 17538083 fsmark.app_overhead
>>>>> 510.93 -
>>>>> 22.7% 395.12 fsmark.files_per_sec
>>>>> 24.90 +22.8% 30.57 fsmark.time.ela
>>>>> psed_
>>>>> time
>>>>> 24.90 +22.8% 30.57 fsmark.time.ela
>>>>> psed_
>>>>> time.max
>>>>> 288.00 ± 2% -
>>>>> 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got
>>>>> 70.03 ± 2% -
>>>>> 11.3% 62.14 fsmark.time.system_time
>>>>>
>>>>
>>>> Do you have time to take a look at this regression?
>>>
>>> From your stats, it looks to me as if the problem is increased
>>> NUMA
>>> overhead. Pretty much everything else appears to be the same or
>>> actually performing better than previously. Am I interpreting that
>>> correctly?
>> The real regression is the throughput(fsmark.files_per_sec) is
>> decreased
>> by 22.7%.
>
> Understood, but I'm trying to make sense of why. I'm not able to
> reproduce this, so I have to rely on your performance stats to
> understand where the 22.7% regression is coming from. As far as I can
> see, the only numbers in the stats you published that are showing a
> performance regression (other than the fsmark number itself), are the
> NUMA numbers. Is that a correct interpretation?
>
We re-test the case yesterday, the test result almost is the same.
we will do more test and also check the test case itself, if you need
more information, please let me know, thanks.
>>> If my interpretation above is correct, then I'm not seeing where
>>> this
>>> patch would be introducing new NUMA regressions. It is just
>>> converting
>>> from using one method of doing socket I/O to another. Could it
>>> perhaps
>>> be a memory artefact due to your running the NFS client and server
>>> on
>>> the same machine?
>>>
>>> Apologies for pushing back a little, but I just don't have the
>>> hardware available to test NUMA configurations, so I'm relying on
>>> external testing for the above kind of scenario.
>>>
>> Thanks for looking at this. If you need more information, please let
>> me
>> know.
>>> Thanks
>>> Trond
>>>
--
Zhengjun Xing
Hi Trond,
I retest, it still can be reproduced. I test with the following
parameters, only change "nr_threads", the test results are as the
following. From the test results, more threads in the test, more
regression will happen. Could you help to check? Thanks.
In testcase: fsmark
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
with 384G memory
with following parameters:
iterations: 20x
nr_threads: 1t
disk: 1BRD_48G
fs: xfs
fs2: nfsv4
filesize: 4M
test_size: 80G
sync_method: fsyncBeforeClose
cpufreq_governor: performance
test-description: The fsmark is a file system benchmark to test
synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/
commit:
e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use iov_iter_kvec()")
0472e47660 ("SUNRPC: Convert socket page send code to use iov_iter()")
e791f8e9380d945e 0472e476604998c127f3c80d291
---------------- ---------------------------
%stddev %change %stddev
\ | \
59.74 -0.7% 59.32 fsmark.files_per_sec
(nr_threads= 1)
114.06 -8.1% 104.83 fsmark.files_per_sec
(nr_threads= 2)
184.53 -13.1% 160.29 fsmark.files_per_sec
(nr_threads= 4)
257.05 -15.5% 217.22 fsmark.files_per_sec
(nr_threads= 8)
306.08 -15.5% 258.68 fsmark.files_per_sec
(nr_threads=16)
498.34 -22.7% 385.33 fsmark.files_per_sec
(nr_threads=32)
527.29 -22.6% 407.96 fsmark.files_per_sec
(nr_threads=64)
On 5/31/2019 11:27 AM, Xing Zhengjun wrote:
>
>
> On 5/31/2019 3:10 AM, Trond Myklebust wrote:
>> On Thu, 2019-05-30 at 15:20 +0800, Xing Zhengjun wrote:
>>>
>>> On 5/30/2019 10:00 AM, Trond Myklebust wrote:
>>>> Hi Xing,
>>>>
>>>> On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote:
>>>>> Hi Trond,
>>>>>
>>>>> On 5/20/2019 1:54 PM, kernel test robot wrote:
>>>>>> Greeting,
>>>>>>
>>>>>> FYI, we noticed a 16.0% improvement of fsmark.app_overhead due
>>>>>> to
>>>>>> commit:
>>>>>>
>>>>>>
>>>>>> commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC:
>>>>>> Convert
>>>>>> socket page send code to use iov_iter()")
>>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
>>>>>> master
>>>>>>
>>>>>> in testcase: fsmark
>>>>>> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @
>>>>>> 3.00GHz with 384G memory
>>>>>> with following parameters:
>>>>>>
>>>>>> iterations: 1x
>>>>>> nr_threads: 64t
>>>>>> disk: 1BRD_48G
>>>>>> fs: xfs
>>>>>> fs2: nfsv4
>>>>>> filesize: 4M
>>>>>> test_size: 40G
>>>>>> sync_method: fsyncBeforeClose
>>>>>> cpufreq_governor: performance
>>>>>>
>>>>>> test-description: The fsmark is a file system benchmark to test
>>>>>> synchronous write workloads, for example, mail servers
>>>>>> workload.
>>>>>> test-url: https://sourceforge.net/projects/fsmark/
>>>>>>
>>>>>>
>>>>>>
>>>>>> Details are as below:
>>>>>> -------------------------------------------------------------
>>>>>> ----
>>>>>> --------------------------------->
>>>>>>
>>>>>>
>>>>>> To reproduce:
>>>>>>
>>>>>> git clone https://github.com/intel/lkp-tests.git
>>>>>> cd lkp-tests
>>>>>> bin/lkp install job.yaml # job file is attached in
>>>>>> this
>>>>>> email
>>>>>> bin/lkp run job.yaml
>>>>>>
>>>>>> ===============================================================
>>>>>> ====
>>>>>> ======================
>>>>>> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconf
>>>>>> ig/n
>>>>>> r_threads/rootfs/sync_method/tbox_group/test_size/testcase:
>>>>>> gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-
>>>>>> 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-
>>>>>> ep01/40G/fsmark
>>>>>>
>>>>>> commit:
>>>>>> e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use
>>>>>> iov_iter_kvec()")
>>>>>> 0472e47660 ("SUNRPC: Convert socket page send code to use
>>>>>> iov_iter()")
>>>>>>
>>>>>> e791f8e9380d945e 0472e476604998c127f3c80d291
>>>>>> ---------------- ---------------------------
>>>>>> fail:runs %reproduction fail:runs
>>>>>> | | |
>>>>>> :4 50% 2:4 dmesg.WARNING:a
>>>>>> t#for
>>>>>> _ip_interrupt_entry/0x
>>>>>> %stddev %change %stddev
>>>>>> \ | \
>>>>>> 15118573
>>>>>> ± 2% +16.0% 17538083 fsmark.app_overhead
>>>>>> 510.93 -
>>>>>> 22.7% 395.12 fsmark.files_per_sec
>>>>>> 24.90 +22.8% 30.57 fsmark.time.ela
>>>>>> psed_
>>>>>> time
>>>>>> 24.90 +22.8% 30.57 fsmark.time.ela
>>>>>> psed_
>>>>>> time.max
>>>>>> 288.00 ± 2% -
>>>>>> 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got
>>>>>> 70.03 ± 2% -
>>>>>> 11.3% 62.14 fsmark.time.system_time
>>>>>>
>>>>>
>>>>> Do you have time to take a look at this regression?
>>>>
>>>> From your stats, it looks to me as if the problem is increased
>>>> NUMA
>>>> overhead. Pretty much everything else appears to be the same or
>>>> actually performing better than previously. Am I interpreting that
>>>> correctly?
>>> The real regression is the throughput(fsmark.files_per_sec) is
>>> decreased
>>> by 22.7%.
>>
>> Understood, but I'm trying to make sense of why. I'm not able to
>> reproduce this, so I have to rely on your performance stats to
>> understand where the 22.7% regression is coming from. As far as I can
>> see, the only numbers in the stats you published that are showing a
>> performance regression (other than the fsmark number itself), are the
>> NUMA numbers. Is that a correct interpretation?
>>
> We re-test the case yesterday, the test result almost is the same.
> we will do more test and also check the test case itself, if you need
> more information, please let me know, thanks.
>
>>>> If my interpretation above is correct, then I'm not seeing where
>>>> this
>>>> patch would be introducing new NUMA regressions. It is just
>>>> converting
>>>> from using one method of doing socket I/O to another. Could it
>>>> perhaps
>>>> be a memory artefact due to your running the NFS client and server
>>>> on
>>>> the same machine?
>>>>
>>>> Apologies for pushing back a little, but I just don't have the
>>>> hardware available to test NUMA configurations, so I'm relying on
>>>> external testing for the above kind of scenario.
>>>>
>>> Thanks for looking at this. If you need more information, please let
>>> me
>>> know.
>>>> Thanks
>>>> Trond
>>>>
>
--
Zhengjun Xing