On Thu, 2021-05-27 at 09:27 +0800, Xing, Zhengjun wrote:
> Hi Rik,
>
> Do you have time to look at this? Thanks.
Hello,
I will try to take a look at this on Friday.
However, even if I manage to reproduce it on one of
the systems I have access to, I'm still not sure how
exactly we would root cause the issue.
Is it due to
select_idle_sibling() doing a little bit
more work?
Is it because we invoke test_idle_cores() a little
earlier, widening the race window with CPUs going idle,
causing select_idle_cpu to do a lot more work?
Is it a locality thing where random placement on any
core in the LLC is somehow better than placement on
the same core as "prev" when there is no idle core?
Is it tbench running
faster when the woken up task is
placed on the runqueue behind the current task on the
"target" cpu, even though that CPU isn't currently
idle, because tbench happens to go to sleep fast?
In other words, I'm
not quite sure whether this is
a tbench (and other similar benchmark) specific thing,
or a kernel thing, or what instrumentation we would
want in select_idle_sibling / select_idle_cpu for us
to root cause issues like this more easily in the
future...
> On 5/19/2021 4:22 PM, kernel test robot wrote:
> > Greeting,
> >
> > FYI, we noticed a -29.1% regression of tbench.throughput-MB/sec due
> > to commit:
> >
> >
> > commit: c722f35b513f807629603bbf24640b1a48be21b5 ("sched/fair:
> > Bring back select_idle_smt(), but differently")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
> > master
> >
> >
> > in testcase: tbench
> > on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252
> > CPU @ 2.10GHz with 128G memory
> > with following parameters:
> >
> > nr_threads: 100%
> > cluster: cs-localhost
> > cpufreq_governor: performance
> > ucode: 0x5003006
> >
> >
> >
> >
> > If you fix the issue, kindly add following tag
> > Reported-by: kernel test robot <[email protected]>
> >
> >
> > Details are as below:
> > -----------------------------------------------------------------
> > --------------------------------->
> >
> >
> > To reproduce:
> >
> > git clone https://github.com/intel/lkp-tests.git
> > cd lkp-tests
> > bin/lkp install job.yaml # job file is
> > attached in this email
> > bin/lkp split-job --compatible job.yaml # generate the
> > yaml file for lkp run
> > bin/lkp run generated-yaml-file
> >
> > ===================================================================
> > ======================
> > cluster/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_gr
> > oup/testcase/ucode:
> > cs-localhost/gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-
> > x86_64-20200603.cgz/lkp-csl-2sp8/tbench/0x5003006
> >
> > commit:
> > 6db12ee045 ("psi: allow unprivileged users with CAP_SYS_RESOURCE
> > to write psi files")
> > c722f35b51 ("sched/fair: Bring back select_idle_smt(), but
> > differently")
> >
> > 6db12ee0456d0e36 c722f35b513f807629603bbf246
> > ---------------- ---------------------------
> > %stddev %change %stddev
> > \ | \
> > 15027 -29.1% 10651 ± 5% tbench.throughput-
> > MB/sec
> > 1.893e+09 ± 2% -87.5% 2.36e+08 ±
> > 73% tbench.time.involuntary_context_switches
> > 51693 ± 21% +65.7% 85645
> > ± 7% tbench.time.minor_page_faults
> > 5200 -45.9% 2815
> > ± 9% tbench.time.percent_of_cpu_this_job_got
> > 32467 -48.9% 16595 ±
> > 10% tbench.time.system_time
> > 4990 -26.1% 3686 ± 6% tbench.time.user_time
> > 1.387e+08 ± 25% +760.7% 1.194e+09
> > ± 8% tbench.time.voluntary_context_switches
> > 3954 ± 8% +307.9% 16131 ± 8% uptime.idle
> > 93556034 ± 4% -24.5% 70597092 ± 5% numa-
> > numastat.node0.local_node
> > 93594173 ± 4% -24.5% 70643519 ± 5% numa-
> > numastat.node0.numa_hit
> > 1.011e+08 ± 2% -29.7% 71069877 ± 6% numa-
> > numastat.node1.local_node
> > 1.012e+08 ± 2% -29.7% 71110181 ± 6% numa-
> > numastat.node1.numa_hit
> > 47900004 ± 3% -21.4% 37635451 ± 6% numa-
> > vmstat.node0.numa_hit
> > 47817433 ± 3% -21.5% 37543255 ± 6% numa-
> > vmstat.node0.numa_local
> > 51712415 ± 2% -26.3% 38110581 ± 8% numa-
> > vmstat.node1.numa_hit
> > 51561304 ± 2% -26.4% 37969236 ± 8% numa-
> > vmstat.node1.numa_local
> > 2.17 ± 31% +861.5% 20.83 ± 9% vmstat.cpu.id
> > 88.67 -19.5% 71.33 ± 2% vmstat.cpu.sy
> > 132.50 -16.2% 111.00 vmstat.procs.r
> > 197679 +6.4% 210333 vmstat.system.in
> > 1.445e+09 ± 24% +838.9% 1.357e+10 ± 10% cpuidle.C1.time
> > 1.585e+08 ± 24% +803.8% 1.433e+09 ± 8% cpuidle.C1.usage
> > 238575 ± 17% +101.0% 479497 ± 9% cpuidle.C1E.usage
> > 16003682 ± 25% +636.0% 1.178e+08 ± 7% cpuidle.POLL.time
> > 1428997 ± 25% +572.1% 9604232 ± 8% cpuidle.POLL.usage
> > 22.00 +14.7 36.65 ± 4% mpstat.cpu.all.idle%
> > 0.81 +0.1 0.90 mpstat.cpu.all.irq%
> > 15.57 +4.6 20.20 ± 2% mpstat.cpu.all.soft%
> > 54.87 -18.2 36.68 ± 5% mpstat.cpu.all.sys%
> > 6.75 -1.2 5.57 ± 3% mpstat.cpu.all.usr%
> > 1.948e+08 -27.3% 1.417e+08 ± 4% proc-vmstat.numa_hit
> > 1.947e+08 -27.3% 1.416e+08 ± 4% proc-
> > vmstat.numa_local
> > 1.516e+09 -28.0% 1.092e+09 ± 5% proc-
> > vmstat.pgalloc_normal
> > 2867654 ± 2% +5.3% 3018753 proc-vmstat.pgfault
> > 1.512e+09 -28.1% 1.087e+09 ± 5% proc-vmstat.pgfree
> > 2663 -15.7% 2244 turbostat.Avg_MHz
> > 97.59 -15.8 81.80 ± 2% turbostat.Busy%
> > 1.585e+08 ± 24% +803.8% 1.433e+09 ± 8% turbostat.C1
> > 2.09 ± 24% +17.5 19.58 ± 10% turbostat.C1%
> > 237770 ± 17% +101.5% 479071 ± 9% turbostat.C1E
> > 2.38 ± 17% +663.9% 18.18 ± 10% turbostat.CPU%c1
> > 74.39 -6.2% 69.77 turbostat.RAMWatt
> > 0.01 ± 59% +886.2% 0.10 ± 48% perf-
> > sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_us
> > er_mode.entry_SYSCALL_64_after_hwframe.[unknown]
> > 0.00 ± 62% +2215.4% 0.05 ± 40% perf-
> > sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__rel
> > ease_sock.release_sock.tcp_sendmsg
> > 0.01 ± 24% +378.9% 0.05 ± 22% perf-
> > sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_
> > cache_alloc_node.__alloc_skb.sk_stream_alloc_skb
> > 0.05 ± 24% +47.3% 0.07 ± 12% perf-
> > sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do
> > _open.isra
> > 2347 ± 20% +28.2% 3010 perf-
> > sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
> > 1.41 ± 25% -34.8% 0.92 ± 19% perf-
> > sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.aa_sk
> > _perm.security_socket_sendmsg.sock_sendmsg
> > 790.43 ± 10% +30.2% 1029 perf-
> > sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
> > 215752 ± 29% -94.4% 12122 ± 46% perf-
> > sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_t
> > o_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
> > 2004 ± 12% -37.7% 1248 ± 6% perf-
> > sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
> > 0.06 ± 23% +762.8% 0.53 ± 42% perf-
> > sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_us
> > er_mode.entry_SYSCALL_64_after_hwframe.[unknown]
> > 0.02 ± 8% +421.8% 0.10 ± 32% perf-
> > sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__rel
> > ease_sock.release_sock.tcp_sendmsg
> > 0.03 ± 12% +191.3% 0.09 ± 18% perf-
> > sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_
> > cache_alloc_node.__alloc_skb.sk_stream_alloc_skb
> > 685.22 ± 7% +17.4% 804.65 ± 12% perf-
> > sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_
> > fork
> > 785.50 ± 10% +30.3% 1023 perf-
> > sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
> > 1.18 ± 35% -33.6% 0.78 ± 18% perf-
> > sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_u
> > ser_mode.asm_sysvec_apic_timer_interrupt.[unknown]
> > 1.58 ± 15% -34.9% 1.03 ± 14% perf-
> > sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.aa_sk
> > _perm.security_socket_sendmsg.sock_sendmsg
> > 51178 ± 93% +1451.5% 794049
> > ± 9% sched_debug.cfs_rq:/.MIN_vruntime.avg
> > 2407970 ± 59% +466.9% 13650279
> > ± 4% sched_debug.cfs_rq:/.MIN_vruntime.max
> > 330425 ± 71% +821.3% 3044331
> > ± 6% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> > 6186 ± 5% -91.9% 499.13
> > ±165% sched_debug.cfs_rq:/.load.min
> > 4.01 ± 7% -39.0% 2.45 ±
> > 15% sched_debug.cfs_rq:/.load_avg.min
> > 51178 ± 93% +1451.5% 794049
> > ± 9% sched_debug.cfs_rq:/.max_vruntime.avg
> > 2407973 ± 59% +466.9% 13650279
> > ± 4% sched_debug.cfs_rq:/.max_vruntime.max
> > 330425 ± 71% +821.3% 3044331
> > ± 6% sched_debug.cfs_rq:/.max_vruntime.stddev
> > 23445823 -41.9% 13622977 ±
> > 11% sched_debug.cfs_rq:/.min_vruntime.avg
> > 25062917 ± 2% -32.7% 16866064
> > ± 8% sched_debug.cfs_rq:/.min_vruntime.max
> > 22902926 ± 2% -42.8% 13102629 ±
> > 11% sched_debug.cfs_rq:/.min_vruntime.min
> > 0.92 -19.0% 0.75
> > ± 3% sched_debug.cfs_rq:/.nr_running.avg
> > 0.74 ± 12% -89.7% 0.08
> > ±115% sched_debug.cfs_rq:/.nr_running.min
> > 0.07 ± 32% +432.3% 0.38 ±
> > 10% sched_debug.cfs_rq:/.nr_running.stddev
> > 1096 ± 2% -
> > 18.1% 898.07 sched_debug.cfs_rq:/.runnable_avg.avg
> > 203.00 ± 4% +8.1% 219.36
> > ± 2% sched_debug.cfs_rq:/.runnable_avg.stddev
> > 996603 ± 48% +225.1% 3240056 ±
> > 25% sched_debug.cfs_rq:/.spread0.max
> > 855.14 -35.7% 549.90
> > ± 6% sched_debug.cfs_rq:/.util_avg.avg
> > 1230 ± 6% -21.1% 971.21
> > ± 6% sched_debug.cfs_rq:/.util_avg.max
> > 142.21 ± 3% -38.9% 86.89 ±
> > 17% sched_debug.cfs_rq:/.util_avg.stddev
> > 630.23 ± 5% -86.6% 84.51 ±
> > 65% sched_debug.cfs_rq:/.util_est_enqueued.avg
> > 1148 ± 5% -57.8% 484.29 ±
> > 17% sched_debug.cfs_rq:/.util_est_enqueued.max
> > 162.67 ± 21% -92.9% 11.56
> > ±138% sched_debug.cfs_rq:/.util_est_enqueued.min
> > 189.55 ± 5% -60.4% 75.05 ±
> > 31% sched_debug.cfs_rq:/.util_est_enqueued.stddev
> > 13.27 ± 7% -56.2% 5.81 ±
> > 16% sched_debug.cpu.clock.stddev
> > 2627 -22.2% 2042 ± 3% sched_debug.cpu.curr-
> > >pid.avg
> > 1954 ± 10% -92.0% 157.14 ±137% sched_debug.cpu.curr-
> > >pid.min
> > 963.90 ± 6% +58.2% 1525 ± 4% sched_debug.cpu.curr-
> > >pid.stddev
> > 0.00 ± 7% -47.7% 0.00 ±
> > 15% sched_debug.cpu.next_balance.stddev
> > 1.30 -14.2% 1.12
> > ± 2% sched_debug.cpu.nr_running.avg
> > 2.50 ± 6% +79.0% 4.47
> > ± 6% sched_debug.cpu.nr_running.max
> > 0.76 ± 10% -89.8% 0.08
> > ±115% sched_debug.cpu.nr_running.min
> > 0.51 ± 3% +73.6% 0.89
> > ± 6% sched_debug.cpu.nr_running.stddev
> > 8.53 ± 3% +154.8% 21.73 ± 6% perf-stat.i.MPKI
> > 2.178e+10 -13.8% 1.878e+10 ± 3% perf-stat.i.branch-
> > instructions
> > 1.69 +0.2 1.90 perf-stat.i.branch-
> > miss-rate%
> > 3.675e+08 -3.5% 3.545e+08 perf-stat.i.branch-
> > misses
> > 18.30 ± 7% -13.5 4.80 ± 36% perf-stat.i.cache-
> > miss-rate%
> > 1.546e+08 ± 9% -48.7% 79255656 ± 18% perf-stat.i.cache-
> > misses
> > 9.224e+08 ± 2% +113.6% 1.971e+09 ± 4% perf-stat.i.cache-
> > references
> > 2.553e+11 -14.5% 2.183e+11 perf-stat.i.cpu-
> > cycles
> > 240694 ± 23% +737.9% 2016725 ± 7% perf-stat.i.cpu-
> > migrations
> > 1700 ± 10% +78.2% 3029 ± 11% perf-stat.i.cycles-
> > between-cache-misses
> > 0.02 ± 19% +0.2 0.18 ± 7% perf-stat.i.dTLB-
> > load-miss-rate%
> > 6572698 ± 19% +582.9% 44886023 ± 7% perf-stat.i.dTLB-
> > load-misses
> > 3.024e+10 -14.7% 2.58e+10 ± 3% perf-stat.i.dTLB-
> > loads
> > 0.01 ± 16% +0.1 0.06 ± 8% perf-stat.i.dTLB-
> > store-miss-rate%
> > 1539288 ± 16% +464.4% 8687418 ± 8% perf-stat.i.dTLB-
> > store-misses
> > 1.709e+10 -14.7% 1.457e+10 ± 3% perf-stat.i.dTLB-
> > stores
> > 73.36 -12.9 60.46 ± 2% perf-stat.i.iTLB-
> > load-miss-rate%
> > 1.847e+08 ± 2% -10.9% 1.645e+08 ± 2% perf-stat.i.iTLB-
> > load-misses
> > 67816199 ± 2% +59.3% 1.08e+08 ± 3% perf-stat.i.iTLB-
> > loads
> > 1.073e+11 -13.7% 9.264e+10 ± 3% perf-
> > stat.i.instructions
> > 2.66 -14.5% 2.27 perf-
> > stat.i.metric.GHz
> > 1188 ± 4% -48.1% 616.51 ± 13% perf-
> > stat.i.metric.K/sec
> > 729.51 -12.6% 637.79 ± 2% perf-
> > stat.i.metric.M/sec
> > 3886 ± 2% +5.4% 4097 perf-stat.i.minor-
> > faults
> > 31516592 ± 8% -29.4% 22240687 ± 13% perf-stat.i.node-
> > load-misses
> > 5444441 ± 5% -16.3% 4559306 ± 4% perf-stat.i.node-
> > loads
> > 96.59 -4.9 91.73 perf-stat.i.node-
> > store-miss-rate%
> > 432854 ± 9% +96.5% 850464 ± 17% perf-stat.i.node-
> > stores
> > 3887 ± 2% +5.4% 4098 perf-stat.i.page-
> > faults
> > 8.60 ± 3% +147.7% 21.30 ± 6% perf-
> > stat.overall.MPKI
> > 1.69 +0.2 1.89 perf-
> > stat.overall.branch-miss-rate%
> > 16.76 ± 9% -12.7 4.07 ± 23% perf-
> > stat.overall.cache-miss-rate%
> > 1666 ± 9% +69.9% 2830 ± 14% perf-
> > stat.overall.cycles-between-cache-misses
> > 0.02 ± 18% +0.2 0.17 ± 8% perf-
> > stat.overall.dTLB-load-miss-rate%
> > 0.01 ± 16% +0.1 0.06 ± 9% perf-
> > stat.overall.dTLB-store-miss-rate%
> > 73.13 -12.8 60.37 ± 2% perf-
> > stat.overall.iTLB-load-miss-rate%
> > 95.70 -4.9 90.84 perf-
> > stat.overall.node-store-miss-rate%
> > 2.175e+10 -13.7% 1.876e+10 ± 3% perf-stat.ps.branch-
> > instructions
> > 3.67e+08 -3.5% 3.54e+08 perf-stat.ps.branch-
> > misses
> > 1.543e+08 ± 9% -48.7% 79211813 ± 18% perf-stat.ps.cache-
> > misses
> > 9.214e+08 ± 2% +113.5% 1.967e+09 ± 4% perf-stat.ps.cache-
> > references
> > 2.549e+11 -14.5% 2.18e+11 perf-stat.ps.cpu-
> > cycles
> > 240897 ± 23% +735.5% 2012779 ± 7% perf-stat.ps.cpu-
> > migrations
> > 6575045 ± 19% +581.3% 44797852 ± 7% perf-stat.ps.dTLB-
> > load-misses
> > 3.02e+10 -14.7% 2.577e+10 ± 3% perf-stat.ps.dTLB-
> > loads
> > 1539214 ± 16% +463.3% 8670426 ± 8% perf-stat.ps.dTLB-
> > store-misses
> > 1.707e+10 -14.7% 1.456e+10 ± 3% perf-stat.ps.dTLB-
> > stores
> > 1.844e+08 ± 2% -10.9% 1.643e+08 ± 2% perf-stat.ps.iTLB-
> > load-misses
> > 67743849 ± 2% +59.2% 1.079e+08 ± 3% perf-stat.ps.iTLB-
> > loads
> > 1.072e+11 -13.7% 9.253e+10 ± 3% perf-
> > stat.ps.instructions
> > 3886 ± 2% +5.5% 4098 perf-stat.ps.minor-
> > faults
> > 31470314 ± 8% -29.4% 22222434 ± 13% perf-stat.ps.node-
> > load-misses
> > 5442735 ± 5% -16.2% 4560730 ± 3% perf-stat.ps.node-
> > loads
> > 432399 ± 9% +96.4% 849120 ± 17% perf-stat.ps.node-
> > stores
> > 3886 ± 2% +5.5% 4098 perf-stat.ps.page-
> > faults
> > 7.73e+13 -13.7% 6.672e+13 ± 3% perf-
> > stat.total.instructions
> > 0.00 +4.1e+103% 40.83 ±117% interrupts.102:PCI-
> > MSI.31981635-edge.i40e-eth0-TxRx-66
> > 163522 ± 5% +659.4% 1241719 ±
> > 40% interrupts.CAL:Function_call_interrupts
> > 2717 ± 28% +275.9% 10215 ±
> > 52% interrupts.CPU0.CAL:Function_call_interrupts
> > 30618 ± 41% +290.0% 119405
> > ± 9% interrupts.CPU0.RES:Rescheduling_interrupts
> > 1686 ± 25% +483.4% 9836 ±
> > 44% interrupts.CPU1.CAL:Function_call_interrupts
> > 26633 ± 50% +342.3% 117798 ±
> > 10% interrupts.CPU1.RES:Rescheduling_interrupts
> > 1633 ± 25% +563.1% 10833 ±
> > 50% interrupts.CPU10.CAL:Function_call_interrupts
> > 27398 ± 51% +319.3% 114895 ±
> > 10% interrupts.CPU10.RES:Rescheduling_interrupts
> > 1857 ± 21% +446.8% 10154 ±
> > 50% interrupts.CPU11.CAL:Function_call_interrupts
> > 26206 ± 53% +312.9% 108207
> > ± 8% interrupts.CPU11.RES:Rescheduling_interrupts
> > 1423 ± 15% +614.7% 10176 ±
> > 52% interrupts.CPU12.CAL:Function_call_interrupts
> > 26880 ± 58% +322.7% 113610
> > ± 8% interrupts.CPU12.RES:Rescheduling_interrupts
> > 1457 ± 18% +585.6% 9988 ±
> > 52% interrupts.CPU13.CAL:Function_call_interrupts
> > 26741 ± 57% +330.3% 115064 ±
> > 10% interrupts.CPU13.RES:Rescheduling_interrupts
> > 1440 ± 8% +626.7% 10466 ±
> > 52% interrupts.CPU14.CAL:Function_call_interrupts
> > 27280 ± 58% +321.3% 114920 ±
> > 10% interrupts.CPU14.RES:Rescheduling_interrupts
> > 1547 ± 13% +550.6% 10065 ±
> > 51% interrupts.CPU15.CAL:Function_call_interrupts
> > 24652 ± 60% +366.8% 115077 ±
> > 10% interrupts.CPU15.RES:Rescheduling_interrupts
> > 1613 ± 14% +518.7% 9981 ±
> > 52% interrupts.CPU16.CAL:Function_call_interrupts
> > 26360 ± 60% +338.3% 115544
> > ± 9% interrupts.CPU16.RES:Rescheduling_interrupts
> > 1783 ± 22% +476.1% 10273 ±
> > 50% interrupts.CPU17.CAL:Function_call_interrupts
> > 29465 ± 57% +290.9% 115190 ±
> > 10% interrupts.CPU17.RES:Rescheduling_interrupts
> > 1692 ± 15% +539.1% 10816 ±
> > 49% interrupts.CPU18.CAL:Function_call_interrupts
> > 25242 ± 56% +355.0% 114843
> > ± 8% interrupts.CPU18.RES:Rescheduling_interrupts
> > 1546 ± 14% +576.6% 10463 ±
> > 48% interrupts.CPU19.CAL:Function_call_interrupts
> > 27541 ± 53% +318.0% 115135 ±
> > 10% interrupts.CPU19.RES:Rescheduling_interrupts
> > 1539 ± 20% +597.8% 10744 ±
> > 50% interrupts.CPU2.CAL:Function_call_interrupts
> > 26770 ± 52% +333.8% 116123 ±
> > 10% interrupts.CPU2.RES:Rescheduling_interrupts
> > 1904 ± 25% +432.4% 10138 ±
> > 52% interrupts.CPU20.CAL:Function_call_interrupts
> > 25460 ± 61% +353.1% 115361
> > ± 8% interrupts.CPU20.RES:Rescheduling_interrupts
> > 1762 ± 23% +469.0% 10028 ±
> > 47% interrupts.CPU21.CAL:Function_call_interrupts
> > 26522 ± 54% +331.6% 114472
> > ± 8% interrupts.CPU21.RES:Rescheduling_interrupts
> > 1684 ± 19% +506.5% 10214 ±
> > 49% interrupts.CPU22.CAL:Function_call_interrupts
> > 26964 ± 52% +320.4% 113356
> > ± 9% interrupts.CPU22.RES:Rescheduling_interrupts
> > 1664 ± 16% +552.7% 10863 ±
> > 47% interrupts.CPU23.CAL:Function_call_interrupts
> > 25881 ± 53% +331.9% 111768
> > ± 9% interrupts.CPU23.RES:Rescheduling_interrupts
> > 2767 ± 70% +456.6% 15402 ±
> > 58% interrupts.CPU24.CAL:Function_call_interrupts
> > 34637 ± 23% +245.6% 119696
> > ± 7% interrupts.CPU24.RES:Rescheduling_interrupts
> > 1644 ± 19% +790.7% 14649 ±
> > 62% interrupts.CPU25.CAL:Function_call_interrupts
> > 31698 ± 34% +271.3% 117692
> > ± 9% interrupts.CPU25.RES:Rescheduling_interrupts
> > 1600 ± 20% +885.1% 15766 ±
> > 63% interrupts.CPU26.CAL:Function_call_interrupts
> > 32053 ± 35% +263.6% 116533
> > ± 8% interrupts.CPU26.RES:Rescheduling_interrupts
> > 1906 ± 40% +773.1% 16646 ±
> > 65% interrupts.CPU27.CAL:Function_call_interrupts
> > 32311 ± 37% +257.4% 115489
> > ± 8% interrupts.CPU27.RES:Rescheduling_interrupts
> > 1847 ± 15% +729.0% 15312 ±
> > 64% interrupts.CPU28.CAL:Function_call_interrupts
> > 30973 ± 38% +271.9% 115189
> > ± 8% interrupts.CPU28.RES:Rescheduling_interrupts
> > 1637 ± 24% +825.0% 15146 ±
> > 64% interrupts.CPU29.CAL:Function_call_interrupts
> > 32830 ± 34% +250.8% 115184
> > ± 6% interrupts.CPU29.RES:Rescheduling_interrupts
> > 1433 ± 14% +603.8% 10090 ±
> > 50% interrupts.CPU3.CAL:Function_call_interrupts
> > 26983 ± 49% +320.2% 113392 ±
> > 10% interrupts.CPU3.RES:Rescheduling_interrupts
> > 1597 ± 14% +863.9% 15400 ±
> > 62% interrupts.CPU30.CAL:Function_call_interrupts
> > 31376 ± 38% +260.7% 113167
> > ± 8% interrupts.CPU30.RES:Rescheduling_interrupts
> > 1648 ± 16% +828.7% 15304 ±
> > 62% interrupts.CPU31.CAL:Function_call_interrupts
> > 32435 ± 33% +258.4% 116241
> > ± 8% interrupts.CPU31.RES:Rescheduling_interrupts
> > 2081 ± 28% +635.8% 15314 ±
> > 64% interrupts.CPU32.CAL:Function_call_interrupts
> > 30678 ± 34% +276.2% 115409
> > ± 8% interrupts.CPU32.RES:Rescheduling_interrupts
> > 1508 ± 19% +951.6% 15860 ±
> > 65% interrupts.CPU33.CAL:Function_call_interrupts
> > 32920 ± 39% +251.5% 115730
> > ± 8% interrupts.CPU33.RES:Rescheduling_interrupts
> > 1529 ± 22% +936.7% 15857 ±
> > 65% interrupts.CPU34.CAL:Function_call_interrupts
> > 31214 ± 27% +266.4% 114381
> > ± 8% interrupts.CPU34.RES:Rescheduling_interrupts
> > 1543 ± 19% +933.2% 15950 ±
> > 60% interrupts.CPU35.CAL:Function_call_interrupts
> > 30887 ± 33% +269.9% 114244
> > ± 7% interrupts.CPU35.RES:Rescheduling_interrupts
> > 1566 ± 17% +890.1% 15505 ±
> > 64% interrupts.CPU36.CAL:Function_call_interrupts
> > 31545 ± 41% +254.5% 111828
> > ± 9% interrupts.CPU36.RES:Rescheduling_interrupts
> > 1518 ± 17% +889.4% 15021 ±
> > 61% interrupts.CPU37.CAL:Function_call_interrupts
> > 31749 ± 38% +266.0% 116188
> > ± 9% interrupts.CPU37.RES:Rescheduling_interrupts
> > 1507 ± 18% +946.1% 15770 ±
> > 62% interrupts.CPU38.CAL:Function_call_interrupts
> > 32670 ± 36% +257.7% 116863
> > ± 7% interrupts.CPU38.RES:Rescheduling_interrupts
> > 1662 ± 22% +818.7% 15277 ±
> > 63% interrupts.CPU39.CAL:Function_call_interrupts
> > 32667 ± 37% +255.3% 116081
> > ± 9% interrupts.CPU39.RES:Rescheduling_interrupts
> > 1908 ± 49% +447.9% 10454 ±
> > 49% interrupts.CPU4.CAL:Function_call_interrupts
> > 26121 ± 56% +341.3% 115279
> > ± 7% interrupts.CPU4.RES:Rescheduling_interrupts
> > 1599 ± 30% +883.5% 15734 ±
> > 66% interrupts.CPU40.CAL:Function_call_interrupts
> > 31759 ± 37% +259.8% 114285
> > ± 8% interrupts.CPU40.RES:Rescheduling_interrupts
> > 1535 ± 19% +889.5% 15194 ±
> > 62% interrupts.CPU41.CAL:Function_call_interrupts
> > 31350 ± 37% +263.1% 113826
> > ± 9% interrupts.CPU41.RES:Rescheduling_interrupts
> > 1565 ± 19% +959.8% 16588 ±
> > 63% interrupts.CPU42.CAL:Function_call_interrupts
> > 30781 ± 33% +269.8% 113842
> > ± 7% interrupts.CPU42.RES:Rescheduling_interrupts
> > 1581 ± 11% +929.0% 16272 ±
> > 63% interrupts.CPU43.CAL:Function_call_interrupts
> > 31679 ± 33% +268.0% 116568
> > ± 7% interrupts.CPU43.RES:Rescheduling_interrupts
> > 1520 ± 22% +914.5% 15422 ±
> > 65% interrupts.CPU44.CAL:Function_call_interrupts
> > 31967 ± 41% +264.6% 116553
> > ± 9% interrupts.CPU44.RES:Rescheduling_interrupts
> > 1503 ± 16% +931.3% 15503 ±
> > 63% interrupts.CPU45.CAL:Function_call_interrupts
> > 30829 ± 32% +272.3% 114770
> > ± 8% interrupts.CPU45.RES:Rescheduling_interrupts
> > 1492 ± 15% +948.3% 15643 ±
> > 64% interrupts.CPU46.CAL:Function_call_interrupts
> > 32136 ± 37% +256.1% 114426
> > ± 8% interrupts.CPU46.RES:Rescheduling_interrupts
> > 6530 ± 4% +186.1% 18685 ±
> > 49% interrupts.CPU47.CAL:Function_call_interrupts
> > 32843 ± 36% +239.2% 111397
> > ± 8% interrupts.CPU47.RES:Rescheduling_interrupts
> > 1501 ± 21% +552.3% 9795 ±
> > 49% interrupts.CPU48.CAL:Function_call_interrupts
> > 27770 ± 49% +306.1% 112779 ±
> > 10% interrupts.CPU48.RES:Rescheduling_interrupts
> > 1716 ± 12% +479.8% 9951 ±
> > 47% interrupts.CPU49.CAL:Function_call_interrupts
> > 27441 ± 50% +306.8% 111620 ±
> > 12% interrupts.CPU49.RES:Rescheduling_interrupts
> > 1536 ± 20% +561.4% 10164 ±
> > 45% interrupts.CPU5.CAL:Function_call_interrupts
> > 26043 ± 58% +336.2% 113601
> > ± 8% interrupts.CPU5.RES:Rescheduling_interrupts
> > 1466 ± 15% +652.1% 11028 ±
> > 52% interrupts.CPU50.CAL:Function_call_interrupts
> > 25157 ± 58% +346.9% 112432 ±
> > 11% interrupts.CPU50.RES:Rescheduling_interrupts
> > 1480 ± 14% +607.5% 10472 ±
> > 49% interrupts.CPU51.CAL:Function_call_interrupts
> > 25280 ± 53% +326.7% 107877 ±
> > 10% interrupts.CPU51.RES:Rescheduling_interrupts
> > 1718 ± 28% +513.8% 10546 ±
> > 49% interrupts.CPU52.CAL:Function_call_interrupts
> > 25402 ± 61% +343.7% 112698
> > ± 9% interrupts.CPU52.RES:Rescheduling_interrupts
> > 1471 ± 18% +584.4% 10067 ±
> > 51% interrupts.CPU53.CAL:Function_call_interrupts
> > 26502 ± 51% +317.7% 110702
> > ± 9% interrupts.CPU53.RES:Rescheduling_interrupts
> > 1520 ± 10% +577.5% 10301 ±
> > 49% interrupts.CPU54.CAL:Function_call_interrupts
> > 26868 ± 53% +311.4% 110540 ±
> > 11% interrupts.CPU54.RES:Rescheduling_interrupts
> > 1378 ± 15% +629.6% 10059 ±
> > 48% interrupts.CPU55.CAL:Function_call_interrupts
> > 26538 ± 55% +328.1% 113615
> > ± 9% interrupts.CPU55.RES:Rescheduling_interrupts
> > 1432 ± 12% +613.7% 10226 ±
> > 50% interrupts.CPU56.CAL:Function_call_interrupts
> > 27378 ± 54% +318.3% 114536
> > ± 9% interrupts.CPU56.RES:Rescheduling_interrupts
> > 1578 ± 23% +551.4% 10283 ±
> > 51% interrupts.CPU57.CAL:Function_call_interrupts
> > 25737 ± 62% +337.5% 112612 ±
> > 10% interrupts.CPU57.RES:Rescheduling_interrupts
> > 1483 ± 22% +630.0% 10829 ±
> > 52% interrupts.CPU58.CAL:Function_call_interrupts
> > 27343 ± 53% +315.8% 113705
> > ± 9% interrupts.CPU58.RES:Rescheduling_interrupts
> > 1460 ± 18% +602.5% 10261 ±
> > 51% interrupts.CPU59.CAL:Function_call_interrupts
> > 25911 ± 58% +298.8% 103334
> > ± 7% interrupts.CPU59.RES:Rescheduling_interrupts
> > 1569 ± 8% +530.0% 9889 ±
> > 50% interrupts.CPU6.CAL:Function_call_interrupts
> > 27162 ± 52% +310.3% 111462 ±
> > 10% interrupts.CPU6.RES:Rescheduling_interrupts
> > 1587 ± 31% +530.4% 10005 ±
> > 51% interrupts.CPU60.CAL:Function_call_interrupts
> > 26060 ± 56% +330.3% 112129
> > ± 9% interrupts.CPU60.RES:Rescheduling_interrupts
> > 1414 ± 16% +608.1% 10012 ±
> > 49% interrupts.CPU61.CAL:Function_call_interrupts
> > 25916 ± 59% +345.2% 115375 ±
> > 10% interrupts.CPU61.RES:Rescheduling_interrupts
> > 1501 ± 18% +601.8% 10534 ±
> > 53% interrupts.CPU62.CAL:Function_call_interrupts
> > 27209 ± 60% +317.1% 113499
> > ± 9% interrupts.CPU62.RES:Rescheduling_interrupts
> > 1411 ± 18% +612.3% 10052 ±
> > 51% interrupts.CPU63.CAL:Function_call_interrupts
> > 25584 ± 59% +345.4% 113944
> > ± 9% interrupts.CPU63.RES:Rescheduling_interrupts
> > 1890 ± 17% +437.8% 10169 ±
> > 50% interrupts.CPU64.CAL:Function_call_interrupts
> > 26884 ± 60% +322.0% 113444 ±
> > 10% interrupts.CPU64.RES:Rescheduling_interrupts
> > 1729 ± 25% +475.6% 9955 ±
> > 48% interrupts.CPU65.CAL:Function_call_interrupts
> > 26952 ± 54% +321.5% 113608 ±
> > 10% interrupts.CPU65.RES:Rescheduling_interrupts
> > 1897 ± 16% +481.0% 11020 ±
> > 46% interrupts.CPU66.CAL:Function_call_interrupts
> > 25947 ± 58% +327.7% 110971
> > ± 9% interrupts.CPU66.RES:Rescheduling_interrupts
> > 1761 ± 18% +467.0% 9987 ±
> > 50% interrupts.CPU67.CAL:Function_call_interrupts
> > 27578 ± 52% +307.5% 112380 ±
> > 10% interrupts.CPU67.RES:Rescheduling_interrupts
> > 1784 ± 18% +451.1% 9835 ±
> > 49% interrupts.CPU68.CAL:Function_call_interrupts
> > 25578 ± 60% +340.7% 112722 ±
> > 10% interrupts.CPU68.RES:Rescheduling_interrupts
> > 1837 ± 17% +436.5% 9857 ±
> > 49% interrupts.CPU69.CAL:Function_call_interrupts
> > 26266 ± 59% +329.5% 112809
> > ± 9% interrupts.CPU69.RES:Rescheduling_interrupts
> > 1465 ± 15% +583.8% 10020 ±
> > 49% interrupts.CPU7.CAL:Function_call_interrupts
> > 25720 ± 57% +351.1% 116026
> > ± 8% interrupts.CPU7.RES:Rescheduling_interrupts
> > 1810 ± 21% +425.7% 9515 ±
> > 44% interrupts.CPU70.CAL:Function_call_interrupts
> > 27634 ± 51% +304.4% 111767 ±
> > 10% interrupts.CPU70.RES:Rescheduling_interrupts
> > 1891 ± 12% +415.1% 9740 ±
> > 48% interrupts.CPU71.CAL:Function_call_interrupts
> > 26205 ± 55% +323.3% 110934
> > ± 9% interrupts.CPU71.RES:Rescheduling_interrupts
> > 2946 ± 93% +429.1% 15592 ±
> > 63% interrupts.CPU72.CAL:Function_call_interrupts
> > 33476 ± 29% +241.2% 114214
> > ± 8% interrupts.CPU72.RES:Rescheduling_interrupts
> > 1797 ± 28% +730.7% 14931 ±
> > 64% interrupts.CPU73.CAL:Function_call_interrupts
> > 31188 ± 31% +269.5% 115229
> > ± 9% interrupts.CPU73.RES:Rescheduling_interrupts
> > 1649 ± 23% +918.0% 16794 ±
> > 64% interrupts.CPU74.CAL:Function_call_interrupts
> > 30456 ± 37% +274.1% 113927
> > ± 9% interrupts.CPU74.RES:Rescheduling_interrupts
> > 1875 ± 31% +785.5% 16610 ±
> > 62% interrupts.CPU75.CAL:Function_call_interrupts
> > 31503 ± 38% +259.7% 113310
> > ± 8% interrupts.CPU75.RES:Rescheduling_interrupts
> > 1833 ± 17% +766.5% 15885 ±
> > 64% interrupts.CPU76.CAL:Function_call_interrupts
> > 29718 ± 38% +278.7% 112554
> > ± 9% interrupts.CPU76.RES:Rescheduling_interrupts
> > 1650 ± 15% +857.9% 15811 ±
> > 66% interrupts.CPU77.CAL:Function_call_interrupts
> > 31625 ± 29% +252.2% 111391
> > ± 7% interrupts.CPU77.RES:Rescheduling_interrupts
> > 1740 ± 16% +798.0% 15632 ±
> > 62% interrupts.CPU78.CAL:Function_call_interrupts
> > 32549 ± 34% +239.6% 110549
> > ± 9% interrupts.CPU78.RES:Rescheduling_interrupts
> > 1675 ± 19% +830.7% 15595 ±
> > 64% interrupts.CPU79.CAL:Function_call_interrupts
> > 31016 ± 36% +266.4% 113656
> > ± 8% interrupts.CPU79.RES:Rescheduling_interrupts
> > 1437 ± 11% +635.4% 10570 ±
> > 49% interrupts.CPU8.CAL:Function_call_interrupts
> > 27602 ± 60% +322.8% 116699 ±
> > 10% interrupts.CPU8.RES:Rescheduling_interrupts
> > 1683 ± 27% +802.2% 15187 ±
> > 65% interrupts.CPU80.CAL:Function_call_interrupts
> > 31149 ± 43% +268.5% 114795
> > ± 8% interrupts.CPU80.RES:Rescheduling_interrupts
> > 1583 ± 24% +911.9% 16020 ±
> > 66% interrupts.CPU81.CAL:Function_call_interrupts
> > 31699 ± 38% +260.9% 114418
> > ± 8% interrupts.CPU81.RES:Rescheduling_interrupts
> > 1752 ± 36% +827.9% 16262 ±
> > 66% interrupts.CPU82.CAL:Function_call_interrupts
> > 29360 ± 36% +288.9% 114167
> > ± 8% interrupts.CPU82.RES:Rescheduling_interrupts
> > 1558 ± 13% +932.4% 16090 ±
> > 64% interrupts.CPU83.CAL:Function_call_interrupts
> > 30230 ± 36% +270.2% 111904
> > ± 8% interrupts.CPU83.RES:Rescheduling_interrupts
> > 1455 ± 17% +958.3% 15407 ±
> > 64% interrupts.CPU84.CAL:Function_call_interrupts
> > 30911 ± 34% +254.7% 109645 ±
> > 10% interrupts.CPU84.RES:Rescheduling_interrupts
> > 1603 ± 10% +827.8% 14881 ±
> > 62% interrupts.CPU85.CAL:Function_call_interrupts
> > 31085 ± 37% +265.4% 113578
> > ± 8% interrupts.CPU85.RES:Rescheduling_interrupts
> > 1575 ± 17% +904.7% 15827 ±
> > 64% interrupts.CPU86.CAL:Function_call_interrupts
> > 33065 ± 38% +248.7% 115301
> > ± 8% interrupts.CPU86.RES:Rescheduling_interrupts
> > 1498 ± 18% +920.9% 15295 ±
> > 64% interrupts.CPU87.CAL:Function_call_interrupts
> > 30477 ± 42% +274.6% 114181
> > ± 9% interrupts.CPU87.RES:Rescheduling_interrupts
> > 1418 ± 17% +960.6% 15044 ±
> > 64% interrupts.CPU88.CAL:Function_call_interrupts
> > 31735 ± 32% +255.3% 112741
> > ± 9% interrupts.CPU88.RES:Rescheduling_interrupts
> > 1459 ± 21% +984.3% 15826 ±
> > 65% interrupts.CPU89.CAL:Function_call_interrupts
> > 31037 ± 41% +260.3% 111826
> > ± 9% interrupts.CPU89.RES:Rescheduling_interrupts
> > 1505 ± 18% +562.6% 9975 ±
> > 47% interrupts.CPU9.CAL:Function_call_interrupts
> > 27015 ± 64% +327.1% 115392
> > ± 9% interrupts.CPU9.RES:Rescheduling_interrupts
> > 1592 ± 24% +935.1% 16487 ±
> > 66% interrupts.CPU90.CAL:Function_call_interrupts
> > 31256 ± 37% +257.9% 111877
> > ± 8% interrupts.CPU90.RES:Rescheduling_interrupts
> > 1576 ± 19% +882.7% 15489 ±
> > 66% interrupts.CPU91.CAL:Function_call_interrupts
> > 31276 ± 37% +266.3% 114573
> > ± 8% interrupts.CPU91.RES:Rescheduling_interrupts
> > 1706 ± 18% +786.3% 15121 ±
> > 66% interrupts.CPU92.CAL:Function_call_interrupts
> > 31400 ± 43% +260.4% 113170
> > ± 8% interrupts.CPU92.RES:Rescheduling_interrupts
> > 1347 ± 18% +1012.1% 14981 ±
> > 66% interrupts.CPU93.CAL:Function_call_interrupts
> > 29869 ± 35% +277.4% 112727
> > ± 8% interrupts.CPU93.RES:Rescheduling_interrupts
> > 1372 ± 12% +937.5% 14243 ±
> > 67% interrupts.CPU94.CAL:Function_call_interrupts
> > 30286 ± 40% +271.0% 112366
> > ± 7% interrupts.CPU94.RES:Rescheduling_interrupts
> > 1739 ± 27% +735.8% 14538 ±
> > 64% interrupts.CPU95.CAL:Function_call_interrupts
> > 30315 ± 39% +266.8% 111198
> > ± 8% interrupts.CPU95.RES:Rescheduling_interrupts
> > 2791109 ± 14% +291.3% 10921566
> > ± 8% interrupts.RES:Rescheduling_interrupts
> > 1.73 ± 27% -1.5 0.26 ±100% perf-
> > profile.calltrace.cycles-
> > pp.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64.e
> > ntry_SYSCALL_64_after_hwframe
> > 2.22 ± 22% -1.2 0.97 perf-
> > profile.calltrace.cycles-pp.kill
> > 2.08 ± 23% -1.2 0.84 perf-
> > profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill
> > 2.03 ± 24% -1.2 0.80 perf-
> > profile.calltrace.cycles-
> > pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
> > 1.99 ± 24% -1.2 0.76 perf-
> > profile.calltrace.cycles-
> > pp.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
> > 1.85 ± 26% -1.2 0.62 perf-
> > profile.calltrace.cycles-
> > pp.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_6
> > 4_after_hwframe.kill
> > 45.78 -0.9 44.84 perf-
> > profile.calltrace.cycles-pp.__send
> > 45.45 -0.9 44.56 perf-
> > profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send
> > 44.92 -0.6 44.37 perf-
> > profile.calltrace.cycles-
> > pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
> > 44.85 -0.5 44.31 perf-
> > profile.calltrace.cycles-
> > pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__
> > send
> > 44.78 -0.5 44.25 perf-
> > profile.calltrace.cycles-
> > pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_aft
> > er_hwframe.__send
> > 44.54 -0.5 44.03 perf-
> > profile.calltrace.cycles-
> > pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_S
> > YSCALL_64_after_hwframe
> > 1.47 ± 4% -0.2 1.29 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__t
> > cp_transmit_skb
> > 2.48 -0.1 2.33 perf-
> > profile.calltrace.cycles-
> > pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol
> > _deliver_rcu
> > 1.66 -0.1 1.57 perf-
> > profile.calltrace.cycles-
> > pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tc
> > p_v4_rcv
> > 0.74 ± 3% -0.1 0.64 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_re
> > cvfrom
> > 0.62 ± 5% -0.1 0.54 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.skb_release_all.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_
> > recvmsg
> > 0.61 ± 4% -0.1 0.53 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.skb_release_head_state.skb_release_all.__kfree_skb.tcp_recvmsg_l
> > ocked.tcp_recvmsg
> > 0.88 ± 3% -0.1 0.79 perf-
> > profile.calltrace.cycles-
> > pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
> > .__ip_queue_xmit
> > 0.80 ± 4% -0.1 0.72 perf-
> > profile.calltrace.cycles-
> > pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_out
> > put2.ip_output
> > 1.27 ± 3% -0.1 1.20 perf-
> > profile.calltrace.cycles-
> > pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
> > .__sys_sendto
> > 1.14 ± 3% -0.1 1.08 perf-
> > profile.calltrace.cycles-
> > pp.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_
> > sendmsg
> > 0.96 ± 2% -0.1 0.90 perf-
> > profile.calltrace.cycles-
> > pp.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.
> > __sys_sendto
> > 1.59 -0.1 1.53 perf-
> > profile.calltrace.cycles-
> > pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvm
> > sg.__sys_recvfrom
> > 1.57 -0.1 1.51 perf-
> > profile.calltrace.cycles-
> > pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tc
> > p_recvmsg.inet_recvmsg
> > 1.06 ± 3% -0.1 1.01 perf-
> > profile.calltrace.cycles-
> > pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.tcp_s
> > endmsg_locked.tcp_sendmsg
> > 0.90 ± 2% -0.1 0.85 perf-
> > profile.calltrace.cycles-
> > pp.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.s
> > ock_sendmsg
> > 0.95 -0.0 0.91 perf-
> > profile.calltrace.cycles-
> > pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_rec
> > vmsg_locked.tcp_recvmsg
> > 1.76 ± 3% +0.1 1.82 perf-
> > profile.calltrace.cycles-
> > pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__sc
> > hedule.schedule_idle
> > 1.81 ± 3% +0.1 1.88 perf-
> > profile.calltrace.cycles-
> > pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle.d
> > o_idle
> > 0.61 ± 2% +0.1 0.68 perf-
> > profile.calltrace.cycles-
> > pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_
> > startup_64_no_verify
> > 1.94 ± 3% +0.1 2.01 perf-
> > profile.calltrace.cycles-
> > pp.perf_trace_sched_switch.__schedule.schedule_idle.do_idle.cpu_sta
> > rtup_entry
> > 0.58 ± 2% +0.1 0.67 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup
> > _entry
> > 0.89 ± 6% +0.2 1.07 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activat
> > e.try_to_wake_up
> > 0.92 ± 3% +0.2 1.10 ± 5% perf-
> > profile.calltrace.cycles-
> > pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.sche
> > dule
> > 2.55 +0.2 2.78 perf-
> > profile.calltrace.cycles-
> > pp.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_co
> > mmon_lock.sock_def_readable
> > 6.25 ± 3% +0.2 6.48 perf-
> > profile.calltrace.cycles-
> > pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
> > 2.10 ± 3% +0.2 2.34 perf-
> > profile.calltrace.cycles-
> > pp._raw_spin_lock.try_to_wake_up.__wake_up_common.__wake_up_common_
> > lock.sock_def_readable
> > 1.96 ± 3% +0.2 2.20 perf-
> > profile.calltrace.cycles-
> > pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up._
> > _wake_up_common.__wake_up_common_lock
> > 3.94 ± 2% +0.2 4.19 perf-
> > profile.calltrace.cycles-
> > pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
> > .__wake_up_common
> > 4.83 ± 2% +0.3 5.11 perf-
> > profile.calltrace.cycles-
> > pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_comm
> > on.__wake_up_common_lock
> > 4.86 ± 2% +0.3 5.15 perf-
> > profile.calltrace.cycles-
> > pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_commo
> > n_lock.sock_def_readable
> > 5.42 ± 2% +0.3 5.72 perf-
> > profile.calltrace.cycles-
> > pp.set_task_cpu.try_to_wake_up.__wake_up_common.__wake_up_common_lo
> > ck.sock_def_readable
> > 3.86 ± 2% +0.5 4.31 perf-
> > profile.calltrace.cycles-
> > pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_seconda
> > ry
> > 3.90 ± 2% +0.5 4.36 perf-
> > profile.calltrace.cycles-
> > pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondar
> > y_startup_64_no_verify
> > 0.18 ±141% +0.5 0.66 ± 5% perf-
> > profile.calltrace.cycles-
> > pp._raw_spin_lock.__schedule.schedule_idle.do_idle.cpu_startup_entr
> > y
> > 0.00 +0.5 0.53 ± 2% perf-
> > profile.calltrace.cycles-
> > pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_
> > idle
> > 7.74 ± 2% +0.6 8.32 perf-
> > profile.calltrace.cycles-
> > pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_ti
> > meout
> > 33.75 +0.6 34.34 perf-
> > profile.calltrace.cycles-
> > pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_fin
> > ish_output2.ip_output
> > 0.00 +0.6 0.59 ± 6% perf-
> > profile.calltrace.cycles-
> > pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.sched
> > ule_idle.do_idle
> > 23.16 +0.6 23.75 perf-
> > profile.calltrace.cycles-
> > pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.ent
> > ry_SYSCALL_64_after_hwframe
> > 23.06 +0.6 23.66 perf-
> > profile.calltrace.cycles-
> > pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_sy
> > scall_64
> > 8.59 ± 2% +0.6 9.21 perf-
> > profile.calltrace.cycles-
> > pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woke
> > n
> > 22.39 +0.6 23.03 perf-
> > profile.calltrace.cycles-
> > pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64
> > _sys_recvfrom
> > 33.98 +0.7 34.64 perf-
> > profile.calltrace.cycles-
> > pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
> > .__tcp_transmit_skb
> > 33.89 +0.7 34.56 perf-
> > profile.calltrace.cycles-
> > pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip
> > _queue_xmit
> > 16.58 ± 2% +0.7 17.32 perf-
> > profile.calltrace.cycles-
> > pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
> > 31.96 +0.7 32.70 perf-
> > profile.calltrace.cycles-
> > pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll.
> > net_rx_action
> > 16.76 ± 2% +0.8 17.51 perf-
> > profile.calltrace.cycles-
> > pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_lo
> > cked
> > 16.91 ± 2% +0.8 17.66 perf-
> > profile.calltrace.cycles-
> > pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_
> > recvmsg
> > 17.24 ± 2% +0.8 18.00 perf-
> > profile.calltrace.cycles-
> > pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recv
> > msg
> > 18.29 +0.8 19.07 perf-
> > profile.calltrace.cycles-
> > pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_r
> > ecvfrom
> > 31.40 +0.9 32.29 perf-
> > profile.calltrace.cycles-
> > pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_bac
> > klog.__napi_poll
> > 31.33 +0.9 32.22 perf-
> > profile.calltrace.cycles-
> > pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_
> > skb_one_core.process_backlog
> > 31.29 +0.9 32.18 perf-
> > profile.calltrace.cycles-
> > pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
> > .ip_rcv.__netif_receive_skb_one_core
> > 31.15 +0.9 32.05 perf-
> > profile.calltrace.cycles-
> > pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_lo
> > cal_deliver.ip_rcv
> > 29.63 +1.0 30.61 perf-
> > profile.calltrace.cycles-
> > pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_delive
> > r_finish.ip_local_deliver
> > 29.48 +1.0 30.49 perf-
> > profile.calltrace.cycles-
> > pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver
> > _rcu.ip_local_deliver_finish
> > 0.00 +1.1 1.05 perf-
> > profile.calltrace.cycles-
> > pp.available_idle_cpu.select_idle_cpu.select_idle_sibling.select_ta
> > sk_rq_fair.try_to_wake_up
> > 24.86 ± 2% +1.2 26.07 perf-
> > profile.calltrace.cycles-
> > pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv
> > _established.tcp_v4_do_rcv
> > 24.61 ± 2% +1.2 25.82 perf-
> > profile.calltrace.cycles-
> > pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_r
> > eadable.tcp_rcv_established
> > 25.65 ± 2% +1.2 26.87 perf-
> > profile.calltrace.cycles-
> > pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.i
> > p_protocol_deliver_rcu
> > 25.34 ± 2% +1.2 26.58 perf-
> > profile.calltrace.cycles-
> > pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_
> > v4_do_rcv.tcp_v4_rcv
> > 14.20 ± 2% +1.6 15.78 perf-
> > profile.calltrace.cycles-
> > pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup
> > _entry
> > 14.62 ± 2% +1.6 16.26 perf-
> > profile.calltrace.cycles-
> > pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.star
> > t_secondary
> > 14.64 ± 2% +1.6 16.28 perf-
> > profile.calltrace.cycles-
> > pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondar
> > y_startup_64_no_verify
> > 0.00 +2.0 2.03 perf-
> > profile.calltrace.cycles-
> > pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_w
> > ake_up.__wake_up_common
> > 19.87 ± 2% +2.3 22.16 perf-
> > profile.calltrace.cycles-
> > pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_n
> > o_verify
> > 19.89 ± 2% +2.3 22.19 perf-
> > profile.calltrace.cycles-
> > pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> > 19.89 ± 2% +2.3 22.19 perf-
> > profile.calltrace.cycles-
> > pp.start_secondary.secondary_startup_64_no_verify
> > 20.12 ± 2% +2.3 22.45 perf-
> > profile.calltrace.cycles-pp.secondary_startup_64_no_verify
> > 0.00 +2.4 2.36 perf-
> > profile.calltrace.cycles-
> > pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.__wake_up
> > _common.__wake_up_common_lock
> > 74.35 -2.1 72.23 perf-
> > profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> > 73.04 -1.8 71.27 perf-
> > profile.children.cycles-pp.do_syscall_64
> > 2.25 ± 22% -1.3 0.99 perf-
> > profile.children.cycles-pp.kill
> > 1.99 ± 24% -1.2 0.76 perf-
> > profile.children.cycles-pp.__x64_sys_kill
> > 1.85 ± 26% -1.2 0.62 perf-
> > profile.children.cycles-pp.kill_something_info
> > 1.73 ± 27% -1.2 0.50 perf-
> > profile.children.cycles-pp.kill_pid_info
> > 1.71 ± 28% -1.2 0.48 perf-
> > profile.children.cycles-pp.group_send_sig_info
> > 1.59 ± 30% -1.2 0.38 perf-
> > profile.children.cycles-pp.security_task_kill
> > 1.58 ± 30% -1.2 0.37 perf-
> > profile.children.cycles-pp.apparmor_task_kill
> > 45.86 -1.0 44.90 perf-
> > profile.children.cycles-pp.__send
> > 1.28 ± 21% -0.9 0.39 ± 2% perf-
> > profile.children.cycles-pp.aa_sk_perm
> > 0.85 ± 30% -0.6 0.24 ± 3% perf-
> > profile.children.cycles-pp.aa_get_task_label
> > 0.91 ± 19% -0.6 0.32 ± 3% perf-
> > profile.children.cycles-pp.security_socket_recvmsg
> > 0.93 ± 19% -0.6 0.34 ± 3% perf-
> > profile.children.cycles-pp.sock_recvmsg
> > 44.86 -0.5 44.31 perf-
> > profile.children.cycles-pp.__x64_sys_sendto
> > 44.79 -0.5 44.25 perf-
> > profile.children.cycles-pp.__sys_sendto
> > 44.54 -0.5 44.03 perf-
> > profile.children.cycles-pp.sock_sendmsg
> > 1.09 ± 8% -0.3 0.75 perf-
> > profile.children.cycles-pp.syscall_exit_to_user_mode
> > 0.91 ± 9% -0.3 0.59 perf-
> > profile.children.cycles-pp.exit_to_user_mode_prepare
> > 0.44 ± 22% -0.3 0.14 ± 2% perf-
> > profile.children.cycles-pp.security_socket_sendmsg
> > 1.53 ± 4% -0.2 1.35 ± 2% perf-
> > profile.children.cycles-pp.__dev_queue_xmit
> > 2.59 -0.1 2.44 perf-
> > profile.children.cycles-pp.tcp_ack
> > 1.46 ± 2% -0.1 1.34 perf-
> > profile.children.cycles-pp.__kfree_skb
> > 1.74 ± 2% -0.1 1.64 perf-
> > profile.children.cycles-pp.tcp_clean_rtx_queue
> > 0.72 ± 4% -0.1 0.62 ± 2% perf-
> > profile.children.cycles-pp.skb_release_all
> > 0.70 ± 4% -0.1 0.61 ± 2% perf-
> > profile.children.cycles-pp.skb_release_head_state
> > 0.82 ± 4% -0.1 0.74 perf-
> > profile.children.cycles-pp.loopback_xmit
> > 0.90 ± 4% -0.1 0.82 perf-
> > profile.children.cycles-pp.dev_hard_start_xmit
> > 1.28 ± 3% -0.1 1.20 perf-
> > profile.children.cycles-pp._copy_from_iter_full
> > 0.69 ± 4% -0.1 0.62 ± 2% perf-
> > profile.children.cycles-pp._raw_spin_lock_bh
> > 0.54 ± 6% -0.1 0.46 ± 2% perf-
> > profile.children.cycles-pp.switch_mm_irqs_off
> > 0.53 ± 4% -0.1 0.45 ± 2% perf-
> > profile.children.cycles-pp.dst_release
> > 1.58 -0.1 1.51 perf-
> > profile.children.cycles-pp.__skb_datagram_iter
> > 0.96 ± 2% -0.1 0.90 perf-
> > profile.children.cycles-pp.sk_stream_alloc_skb
> > 1.60 -0.1 1.53 perf-
> > profile.children.cycles-pp.skb_copy_datagram_iter
> > 0.40 ± 3% -0.1 0.34 ± 4% perf-
> > profile.children.cycles-pp.tcp_send_mss
> > 0.36 ± 4% -0.1 0.31 ± 3% perf-
> > profile.children.cycles-pp.tcp_current_mss
> > 0.91 ± 2% -0.1 0.86 perf-
> > profile.children.cycles-pp.__alloc_skb
> > 0.23 ± 5% -0.0 0.18 ± 4% perf-
> > profile.children.cycles-pp.irqtime_account_irq
> > 0.27 ± 5% -0.0 0.23 perf-
> > profile.children.cycles-pp.load_new_mm_cr3
> > 0.17 ± 2% -0.0 0.13 ± 4% perf-
> > profile.children.cycles-pp.tcp_ack_update_rtt
> > 0.53 -0.0 0.49 ± 2% perf-
> > profile.children.cycles-pp.sched_clock_cpu
> > 0.61 ± 4% -0.0 0.58 ± 3% perf-
> > profile.children.cycles-pp.release_sock
> > 0.33 ± 8% -0.0 0.29 ± 2% perf-
> > profile.children.cycles-pp.next_token
> > 0.95 -0.0 0.92 perf-
> > profile.children.cycles-pp._copy_to_iter
> > 0.49 ± 2% -0.0 0.46 ± 2% perf-
> > profile.children.cycles-pp.switch_fpu_return
> > 0.41 ± 3% -0.0 0.38 ± 3% perf-
> > profile.children.cycles-pp.tcp_event_new_data_sent
> > 0.39 ± 3% -0.0 0.36 ± 2% perf-
> > profile.children.cycles-pp.netif_rx
> > 0.31 ± 2% -0.0 0.28 perf-
> > profile.children.cycles-pp.reweight_entity
> > 0.21 ± 4% -0.0 0.18 ± 4% perf-
> > profile.children.cycles-pp.kmem_cache_alloc_node
> > 0.33 ± 2% -0.0 0.30 ± 3% perf-
> > profile.children.cycles-pp.__entry_text_start
> > 0.20 ± 4% -0.0 0.17 ± 3% perf-
> > profile.children.cycles-pp.enqueue_to_backlog
> > 0.38 -0.0 0.35 ± 2% perf-
> > profile.children.cycles-pp.__check_object_size
> > 0.38 ± 3% -0.0 0.35 perf-
> > profile.children.cycles-pp.netif_rx_internal
> > 0.32 ± 3% -0.0 0.29 perf-
> > profile.children.cycles-pp.syscall_return_via_sysret
> > 0.16 ± 4% -0.0 0.13 ± 5% perf-
> > profile.children.cycles-pp.ip_local_out
> > 0.28 ± 3% -0.0 0.25 ± 2% perf-
> > profile.children.cycles-pp.ttwu_do_wakeup
> > 0.13 ± 5% -0.0 0.11 ± 6% perf-
> > profile.children.cycles-pp.__virt_addr_valid
> > 0.13 ± 3% -0.0 0.11 ± 6% perf-
> > profile.children.cycles-pp.nf_hook_slow
> > 0.31 -0.0 0.29 ± 2% perf-
> > profile.children.cycles-pp.simple_copy_to_iter
> > 0.11 ± 6% -0.0 0.09 ± 5% perf-
> > profile.children.cycles-pp.import_single_range
> > 0.11 ± 9% -0.0 0.09 ± 4% perf-
> > profile.children.cycles-pp.sock_put
> > 0.26 ± 2% -0.0 0.24 ± 3% perf-
> > profile.children.cycles-pp.check_preempt_curr
> > 0.09 ± 5% -0.0 0.07 ± 7% perf-
> > profile.children.cycles-pp.__might_fault
> > 0.13 ± 5% -0.0 0.11 ± 4% perf-
> > profile.children.cycles-pp.copy_fpregs_to_fpstate
> > 0.17 ± 2% -0.0 0.15 ± 3% perf-
> > profile.children.cycles-pp.tcp_rearm_rto
> > 0.13 ± 6% -0.0 0.11 ± 5% perf-
> > profile.children.cycles-pp.__cond_resched
> > 0.10 ± 7% -0.0 0.09 ± 4% perf-
> > profile.children.cycles-pp.tcp_event_data_recv
> > 0.11 ± 6% -0.0 0.09 ± 5% perf-
> > profile.children.cycles-pp.check_kill_permission
> > 0.23 ± 3% -0.0 0.22 perf-
> > profile.children.cycles-pp.__ksize
> > 0.10 ± 4% -0.0 0.08 ± 5% perf-
> > profile.children.cycles-pp.tcp_update_pacing_rate
> > 0.07 ± 7% -0.0 0.05 ± 8% perf-
> > profile.children.cycles-pp.server
> > 0.08 ± 4% +0.0 0.09 ± 4% perf-
> > profile.children.cycles-pp.attach_entity_load_avg
> > 0.07 ± 5% +0.0 0.09 ± 5% perf-
> > profile.children.cycles-pp.is_module_text_address
> > 0.10 ± 8% +0.0 0.12 ± 5% perf-
> > profile.children.cycles-pp.set_next_task_idle
> > 0.09 ± 5% +0.0 0.11 ± 4% perf-
> > profile.children.cycles-pp.__update_idle_core
> > 0.29 +0.0 0.32 ± 2% perf-
> > profile.children.cycles-pp.tcp_cleanup_rbuf
> > 0.30 +0.0 0.33 perf-
> > profile.children.cycles-pp.kmem_cache_free
> > 0.13 ± 5% +0.0 0.15 ± 3% perf-
> > profile.children.cycles-pp.perf_instruction_pointer
> > 0.08 ± 7% +0.0 0.10 ± 6% perf-
> > profile.children.cycles-pp.cpus_share_cache
> > 0.10 ± 4% +0.0 0.13 ± 5% perf-
> > profile.children.cycles-pp.pick_next_task_idle
> > 0.20 ± 3% +0.0 0.23 ± 2% perf-
> > profile.children.cycles-pp.tick_nohz_idle_exit
> > 0.55 ± 2% +0.0 0.58 perf-
> > profile.children.cycles-pp.update_rq_clock
> > 0.32 ± 4% +0.0 0.34 ± 2% perf-
> > profile.children.cycles-pp.tick_nohz_get_sleep_length
> > 0.23 ± 2% +0.0 0.26 ± 2% perf-
> > profile.children.cycles-pp._find_next_bit
> > 0.19 ± 5% +0.0 0.22 ± 3% perf-
> > profile.children.cycles-pp.tick_nohz_next_event
> > 0.49 ± 2% +0.0 0.52 ± 2% perf-
> > profile.children.cycles-pp.finish_task_switch
> > 0.33 ± 3% +0.0 0.37 perf-
> > profile.children.cycles-pp.migrate_task_rq_fair
> > 0.22 ± 5% +0.0 0.26 ± 4% perf-
> > profile.children.cycles-pp.start_kernel
> > 0.19 ± 2% +0.0 0.23 ± 2% perf-
> > profile.children.cycles-pp.cpumask_next
> > 0.29 ± 4% +0.0 0.34 ± 2% perf-
> > profile.children.cycles-pp.__slab_free
> > 0.60 +0.0 0.65 ± 2% perf-
> > profile.children.cycles-pp.set_next_entity
> > 0.90 +0.0 0.94 perf-
> > profile.children.cycles-pp._raw_spin_lock_irqsave
> > 0.62 ± 2% +0.1 0.69 perf-
> > profile.children.cycles-pp.menu_select
> > 2.01 +0.1 2.10 perf-
> > profile.children.cycles-pp.select_idle_cpu
> > 1.16 +0.1 1.27 perf-
> > profile.children.cycles-pp.available_idle_cpu
> > 2.62 +0.2 2.86 perf-
> > profile.children.cycles-pp.select_task_rq_fair
> > 4.07 ± 2% +0.2 4.32 perf-
> > profile.children.cycles-pp.enqueue_entity
> > 5.00 ± 2% +0.3 5.28 perf-
> > profile.children.cycles-pp.ttwu_do_activate
> > 4.97 ± 2% +0.3 5.26 perf-
> > profile.children.cycles-pp.enqueue_task_fair
> > 5.60 ± 3% +0.3 5.90 perf-
> > profile.children.cycles-pp.set_task_cpu
> > 2.54 ± 4% +0.4 2.91 ± 4% perf-
> > profile.children.cycles-pp.update_load_avg
> > 3.37 ± 2% +0.4 3.76 perf-
> > profile.children.cycles-pp._raw_spin_lock
> > 2.61 ± 3% +0.4 3.02 ± 2% perf-
> > profile.children.cycles-pp.native_queued_spin_lock_slowpath
> > 1.46 ± 8% +0.4 1.88 ± 8% perf-
> > profile.children.cycles-pp.update_cfs_group
> > 3.95 ± 2% +0.5 4.42 perf-
> > profile.children.cycles-pp.schedule_idle
> > 17.05 +0.5 17.54 perf-
> > profile.children.cycles-pp.schedule
> > 7.76 ± 2% +0.6 8.34 perf-
> > profile.children.cycles-pp.dequeue_entity
> > 23.16 +0.6 23.76 perf-
> > profile.children.cycles-pp.inet_recvmsg
> > 23.07 +0.6 23.67 perf-
> > profile.children.cycles-pp.tcp_recvmsg
> > 8.61 ± 2% +0.6 9.23 perf-
> > profile.children.cycles-pp.dequeue_task_fair
> > 22.41 +0.6 23.05 perf-
> > profile.children.cycles-pp.tcp_recvmsg_locked
> > 34.18 +0.7 34.86 perf-
> > profile.children.cycles-pp.__local_bh_enable_ip
> > 33.99 +0.7 34.68 perf-
> > profile.children.cycles-pp.do_softirq
> > 34.00 +0.7 34.69 perf-
> > profile.children.cycles-pp.__softirqentry_text_start
> > 33.40 +0.7 34.14 perf-
> > profile.children.cycles-pp.net_rx_action
> > 33.18 +0.8 33.94 perf-
> > profile.children.cycles-pp.__napi_poll
> > 16.91 ± 2% +0.8 17.66 perf-
> > profile.children.cycles-pp.schedule_timeout
> > 17.24 ± 2% +0.8 18.00 perf-
> > profile.children.cycles-pp.wait_woken
> > 33.09 +0.8 33.86 perf-
> > profile.children.cycles-pp.process_backlog
> > 18.30 +0.8 19.07 perf-
> > profile.children.cycles-pp.sk_wait_data
> > 32.81 +0.8 33.61 perf-
> > profile.children.cycles-pp.__netif_receive_skb_one_core
> > 32.47 +0.8 33.30 perf-
> > profile.children.cycles-pp.ip_rcv
> > 31.91 +0.9 32.81 perf-
> > profile.children.cycles-pp.ip_local_deliver_finish
> > 31.97 +0.9 32.87 perf-
> > profile.children.cycles-pp.ip_local_deliver
> > 31.86 +0.9 32.77 perf-
> > profile.children.cycles-pp.ip_protocol_deliver_rcu
> > 31.72 +0.9 32.63 perf-
> > profile.children.cycles-pp.tcp_v4_rcv
> > 20.81 +0.9 21.74 perf-
> > profile.children.cycles-pp.__schedule
> > 30.18 +1.0 31.16 perf-
> > profile.children.cycles-pp.tcp_v4_do_rcv
> > 30.03 +1.0 31.04 perf-
> > profile.children.cycles-pp.tcp_rcv_established
> > 25.54 ± 2% +1.2 26.76 perf-
> > profile.children.cycles-pp.__wake_up_common
> > 25.29 ± 2% +1.2 26.52 perf-
> > profile.children.cycles-pp.try_to_wake_up
> > 26.34 ± 2% +1.2 27.58 perf-
> > profile.children.cycles-pp.sock_def_readable
> > 26.04 ± 2% +1.2 27.29 perf-
> > profile.children.cycles-pp.__wake_up_common_lock
> > 14.36 ± 2% +1.6 15.96 perf-
> > profile.children.cycles-pp.intel_idle
> > 14.80 ± 2% +1.7 16.46 perf-
> > profile.children.cycles-pp.cpuidle_enter_state
> > 14.80 ± 2% +1.7 16.47 perf-
> > profile.children.cycles-pp.cpuidle_enter
> > 19.89 ± 2% +2.3 22.19 perf-
> > profile.children.cycles-pp.start_secondary
> > 20.10 ± 2% +2.3 22.43 perf-
> > profile.children.cycles-pp.do_idle
> > 20.12 ± 2% +2.3 22.45 perf-
> > profile.children.cycles-pp.secondary_startup_64_no_verify
> > 20.12 ± 2% +2.3 22.45 perf-
> > profile.children.cycles-pp.cpu_startup_entry
> > 0.00 +2.4 2.43 perf-
> > profile.children.cycles-pp.select_idle_sibling
> > 1.18 ± 22% -0.9 0.30 ± 2% perf-
> > profile.self.cycles-pp.aa_sk_perm
> > 0.84 ± 30% -0.6 0.23 ± 4% perf-
> > profile.self.cycles-pp.aa_get_task_label
> > 0.66 ± 31% -0.6 0.10 ± 4% perf-
> > profile.self.cycles-pp.apparmor_task_kill
> > 0.61 ± 5% -0.1 0.55 ± 3% perf-
> > profile.self.cycles-pp._raw_spin_lock_bh
> > 0.52 ± 4% -0.1 0.45 ± 2% perf-
> > profile.self.cycles-pp.dst_release
> > 0.33 ± 2% -0.1 0.26 ± 2% perf-
> > profile.self.cycles-pp.select_task_rq_fair
> > 0.43 ± 6% -0.1 0.38 ± 2% perf-
> > profile.self.cycles-pp.__tcp_transmit_skb
> > 0.70 ± 3% -0.1 0.65 ± 2% perf-
> > profile.self.cycles-pp.tcp_sendmsg_locked
> > 0.27 ± 5% -0.0 0.23 perf-
> > profile.self.cycles-pp.load_new_mm_cr3
> > 0.49 ± 2% -0.0 0.45 ± 2% perf-
> > profile.self.cycles-pp.switch_fpu_return
> > 0.16 ± 4% -0.0 0.12 ± 3% perf-
> > profile.self.cycles-pp.tcp_ack_update_rtt
> > 0.33 ± 3% -0.0 0.30 ± 3% perf-
> > profile.self.cycles-pp.tcp_clean_rtx_queue
> > 0.18 ± 6% -0.0 0.15 ± 2% perf-
> > profile.self.cycles-pp.__ip_queue_xmit
> > 0.34 ± 2% -0.0 0.30 ± 3% perf-
> > profile.self.cycles-pp.tcp_write_xmit
> > 0.25 ± 7% -0.0 0.22 ± 6% perf-
> > profile.self.cycles-pp.switch_mm_irqs_off
> > 0.13 ± 7% -0.0 0.10 ± 12% perf-
> > profile.self.cycles-pp.tcp_current_mss
> > 0.25 ± 5% -0.0 0.22 ± 3% perf-
> > profile.self.cycles-pp.ip_finish_output2
> > 0.10 ± 7% -0.0 0.07 ± 8% perf-
> > profile.self.cycles-pp.exit_to_user_mode_prepare
> > 0.13 ± 5% -0.0 0.10 ± 3% perf-
> > profile.self.cycles-pp.do_user_addr_fault
> > 0.13 ± 5% -0.0 0.11 ± 6% perf-
> > profile.self.cycles-pp.__virt_addr_valid
> > 0.21 ± 6% -0.0 0.18 ± 3% perf-
> > profile.self.cycles-pp.tcp_wfree
> > 0.32 ± 3% -0.0 0.29 ± 2% perf-
> > profile.self.cycles-pp.syscall_return_via_sysret
> > 0.23 ± 4% -0.0 0.21 ± 2% perf-
> > profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
> > 0.11 ± 8% -0.0 0.09 perf-
> > profile.self.cycles-pp.import_single_range
> > 0.15 ± 3% -0.0 0.14 ± 3% perf-
> > profile.self.cycles-pp.tcp_rearm_rto
> > 0.09 ± 8% -0.0 0.07 ± 5% perf-
> > profile.self.cycles-pp.sched_clock_cpu
> > 0.22 -0.0 0.20 ± 3% perf-
> > profile.self.cycles-pp.__softirqentry_text_start
> > 0.11 ± 5% -0.0 0.09 ± 5% perf-
> > profile.self.cycles-pp.__netif_receive_skb_one_core
> > 0.07 ± 8% -0.0 0.05 ± 8% perf-
> > profile.self.cycles-pp.tcp_v4_do_rcv
> > 0.10 ± 4% -0.0 0.08 ± 7% perf-
> > profile.self.cycles-pp.perf_callchain
> > 0.13 ± 5% -0.0 0.11 ± 3% perf-
> > profile.self.cycles-pp.copy_fpregs_to_fpstate
> > 0.13 ± 5% -0.0 0.11 ± 4% perf-
> > profile.self.cycles-pp.kmem_cache_alloc_node
> > 0.09 -0.0 0.08 ± 6% perf-
> > profile.self.cycles-pp.__napi_poll
> > 0.08 ± 6% -0.0 0.06 ± 6% perf-
> > profile.self.cycles-pp.__x64_sys_sendto
> > 0.12 ± 3% -0.0 0.11 ± 3% perf-
> > profile.self.cycles-pp.__sys_sendto
> > 0.11 ± 6% -0.0 0.09 ± 5% perf-
> > profile.self.cycles-pp.__x64_sys_recvfrom
> > 0.14 ± 3% -0.0 0.12 ± 3% perf-
> > profile.self.cycles-pp.__kmalloc_node_track_caller
> > 0.07 ± 7% -0.0 0.05 ± 8% perf-
> > profile.self.cycles-pp.ftrace_ops_trampoline
> > 0.09 ± 5% -0.0 0.08 ± 5% perf-
> > profile.self.cycles-pp.tcp_update_pacing_rate
> > 0.06 ± 8% +0.0 0.07 perf-
> > profile.self.cycles-pp.cpuidle_enter_state
> > 0.06 +0.0 0.07 ± 6% perf-
> > profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
> > 0.08 ± 7% +0.0 0.10 ± 4% perf-
> > profile.self.cycles-pp.cpus_share_cache
> > 0.05 ± 8% +0.0 0.07 ± 5% perf-
> > profile.self.cycles-pp.perf_instruction_pointer
> > 0.28 +0.0 0.30 ± 3% perf-
> > profile.self.cycles-pp.tcp_cleanup_rbuf
> > 0.18 ± 5% +0.0 0.20 perf-
> > profile.self.cycles-pp.migrate_task_rq_fair
> > 0.24 ± 3% +0.0 0.27 ± 2% perf-
> > profile.self.cycles-pp.menu_select
> > 0.41 ± 3% +0.0 0.44 ± 2% perf-
> > profile.self.cycles-pp.update_rq_clock
> > 0.22 ± 4% +0.0 0.26 ± 2% perf-
> > profile.self.cycles-pp._find_next_bit
> > 0.16 ± 2% +0.0 0.20 ± 3% perf-
> > profile.self.cycles-pp.do_idle
> > 0.29 ± 3% +0.0 0.34 ± 2% perf-
> > profile.self.cycles-pp.__slab_free
> > 0.87 +0.0 0.92 perf-
> > profile.self.cycles-pp._raw_spin_lock_irqsave
> > 1.24 +0.0 1.29 ± 2% perf-
> > profile.self.cycles-pp.__schedule
> > 1.15 +0.1 1.26 perf-
> > profile.self.cycles-pp.available_idle_cpu
> > 0.00 +0.1 0.14 ± 3% perf-
> > profile.self.cycles-pp.select_idle_sibling
> > 0.43 ± 7% +0.2 0.61 ± 8% perf-
> > profile.self.cycles-pp.set_task_cpu
> > 1.78 ± 4% +0.3 2.12 ± 5% perf-
> > profile.self.cycles-pp.update_load_avg
> > 2.61 ± 3% +0.4 3.01 ± 2% perf-
> > profile.self.cycles-pp.native_queued_spin_lock_slowpath
> > 1.46 ± 8% +0.4 1.88 ± 8% perf-
> > profile.self.cycles-pp.update_cfs_group
> > 14.36 ± 2% +1.6 15.96 perf-
> > profile.self.cycles-pp.intel_idle
> > 20074513 ± 7% -24.9% 15075828 ± 7% softirqs.CPU0.NET_RX
> > 38385 ± 66% +476.2% 221177 ± 9% softirqs.CPU0.SCHED
> > 32352 ± 12% -78.4% 6999 ± 45% softirqs.CPU0.TIMER
> > 20359880 ± 5% -26.9% 14879840 ± 7% softirqs.CPU1.NET_RX
> > 36748 ± 67% +492.7% 217810 ± 10% softirqs.CPU1.SCHED
> > 31252 ± 13% -85.5% 4524 ± 59% softirqs.CPU1.TIMER
> > 20405448 ± 5% -25.5% 15210284 ± 7% softirqs.CPU10.NET_RX
> > 36443 ± 69% +499.5% 218467 ± 9% softirqs.CPU10.SCHED
> > 30609 ± 11% -84.5% 4741 ± 63% softirqs.CPU10.TIMER
> > 20394573 ± 4% -29.0% 14484845 ± 10% softirqs.CPU11.NET_RX
> > 36636 ± 70% +464.1% 206683 ± 7% softirqs.CPU11.SCHED
> > 30678 ± 11% -84.8% 4653 ± 67% softirqs.CPU11.TIMER
> > 20405829 ± 4% -26.1% 15089688 ± 7% softirqs.CPU12.NET_RX
> > 37061 ± 70% +494.1% 220185 ± 8% softirqs.CPU12.SCHED
> > 31218 ± 10% -84.2% 4919 ± 66% softirqs.CPU12.TIMER
> > 20444043 ± 5% -26.2% 15095203 ± 7% softirqs.CPU13.NET_RX
> > 36503 ± 70% +497.6% 218132 ± 9% softirqs.CPU13.SCHED
> > 30651 ± 11% -84.6% 4710 ± 62% softirqs.CPU13.TIMER
> > 20419827 ± 5% -25.6% 15186252 ± 7% softirqs.CPU14.NET_RX
> > 36704 ± 70% +494.3% 218149 ± 9% softirqs.CPU14.SCHED
> > 30681 ± 11% -84.3% 4806 ± 65% softirqs.CPU14.TIMER
> > 20510872 ± 5% -25.7% 15232936 ± 7% softirqs.CPU15.NET_RX
> > 36871 ± 69% +493.3% 218761 ± 9% softirqs.CPU15.SCHED
> > 31009 ± 12% -84.5% 4793 ± 63% softirqs.CPU15.TIMER
> > 20418436 ± 5% -25.4% 15224178 ± 7% softirqs.CPU16.NET_RX
> > 36854 ± 69% +495.1% 219327 ± 9% softirqs.CPU16.SCHED
> > 31013 ± 11% -84.7% 4758 ± 64% softirqs.CPU16.TIMER
> > 20372676 ± 4% -25.9% 15095272 ± 7% softirqs.CPU17.NET_RX
> > 36811 ± 70% +494.0% 218669 ± 9% softirqs.CPU17.SCHED
> > 31390 ± 13% -77.9% 6929 ± 67% softirqs.CPU17.TIMER
> > 20416587 ± 5% -26.0% 15118161 ± 7% softirqs.CPU18.NET_RX
> > 36394 ± 70% +502.6% 219293 ± 9% softirqs.CPU18.SCHED
> > 30912 ± 11% -84.5% 4784 ± 63% softirqs.CPU18.TIMER
> > 20432227 ± 4% -25.8% 15168258 ± 7% softirqs.CPU19.NET_RX
> > 36602 ± 70% +497.8% 218796 ± 9% softirqs.CPU19.SCHED
> > 30843 ± 11% -82.9% 5259 ± 56% softirqs.CPU19.TIMER
> > 20304573 ± 6% -26.0% 15034216 ± 7% softirqs.CPU2.NET_RX
> > 36968 ± 69% +495.1% 220012 ± 9% softirqs.CPU2.SCHED
> > 30526 ± 12% -84.4% 4764 ± 62% softirqs.CPU2.TIMER
> > 20444514 ± 5% -25.5% 15240754 ± 7% softirqs.CPU20.NET_RX
> > 36759 ± 70% +495.9% 219056 ± 9% softirqs.CPU20.SCHED
> > 30846 ± 11% -84.3% 4831 ± 61% softirqs.CPU20.TIMER
> > 20253081 ± 6% -25.1% 15174237 ± 7% softirqs.CPU21.NET_RX
> > 36699 ± 70% +495.2% 218452 ± 8% softirqs.CPU21.SCHED
> > 30719 ± 12% -83.5% 5060 ± 65% softirqs.CPU21.TIMER
> > 20460332 ± 4% -25.8% 15186744 ± 7% softirqs.CPU22.NET_RX
> > 36652 ± 70% +497.1% 218854 ± 9% softirqs.CPU22.SCHED
> > 30899 ± 11% -84.4% 4811 ± 63% softirqs.CPU22.TIMER
> > 20389322 ± 5% -26.5% 14979439 ± 8% softirqs.CPU23.NET_RX
> > 36717 ± 70% +492.5% 217548 ± 8% softirqs.CPU23.SCHED
> > 30864 ± 11% -83.1% 5216 ± 60% softirqs.CPU23.TIMER
> > 21350836 ± 3% -30.0% 14945967 ± 6% softirqs.CPU24.NET_RX
> > 47117 ± 46% +359.7% 216593 ± 8% softirqs.CPU24.SCHED
> > 31954 ± 20% -84.0% 5123 ± 66% softirqs.CPU24.TIMER
> > 21494795 ± 3% -28.6% 15347973 ± 6% softirqs.CPU25.NET_RX
> > 46755 ± 49% +362.6% 216302 ± 8% softirqs.CPU25.SCHED
> > 30392 ± 9% -82.7% 5245 ± 47% softirqs.CPU25.TIMER
> > 21529925 ± 3% -28.8% 15338064 ± 6% softirqs.CPU26.NET_RX
> > 46989 ± 48% +360.6% 216437 ± 8% softirqs.CPU26.SCHED
> > 30126 ± 9% -84.3% 4715 ± 60% softirqs.CPU26.TIMER
> > 21609177 ± 3% -28.9% 15374089 ± 6% softirqs.CPU27.NET_RX
> > 47211 ± 48% +358.1% 216275 ± 8% softirqs.CPU27.SCHED
> > 30806 ± 9% -84.4% 4806 ± 59% softirqs.CPU27.TIMER
> > 21434518 ± 3% -28.0% 15426523 ± 6% softirqs.CPU28.NET_RX
> > 46072 ± 49% +369.3% 216211 ± 8% softirqs.CPU28.SCHED
> > 29926 ± 10% -83.7% 4871 ± 60% softirqs.CPU28.TIMER
> > 21436007 ± 3% -28.3% 15360927 ± 6% softirqs.CPU29.NET_RX
> > 46751 ± 48% +363.2% 216561 ± 8% softirqs.CPU29.SCHED
> > 29983 ± 9% -83.7% 4876 ± 58% softirqs.CPU29.TIMER
> > 20556029 ± 5% -27.1% 14991636 ± 9% softirqs.CPU3.NET_RX
> > 36617 ± 69% +485.7% 214485 ± 9% softirqs.CPU3.SCHED
> > 30777 ± 11% -82.7% 5311 ± 54% softirqs.CPU3.TIMER
> > 21514774 ± 3% -29.0% 15275932 ± 6% softirqs.CPU30.NET_RX
> > 46934 ± 48% +361.6% 216667 ± 8% softirqs.CPU30.SCHED
> > 30153 ± 9% -84.1% 4799 ± 58% softirqs.CPU30.TIMER
> > 21317864 ± 4% -28.1% 15318620 ± 6% softirqs.CPU31.NET_RX
> > 46511 ± 49% +365.4% 216450 ± 8% softirqs.CPU31.SCHED
> > 29862 ± 10% -83.9% 4819 ± 60% softirqs.CPU31.TIMER
> > 21191794 ± 5% -27.4% 15375714 ± 6% softirqs.CPU32.NET_RX
> > 45002 ± 47% +380.8% 216377 ± 8% softirqs.CPU32.SCHED
> > 30188 ± 13% -84.0% 4825 ± 60% softirqs.CPU32.TIMER
> > 21594693 ± 3% -28.5% 15432125 ± 6% softirqs.CPU33.NET_RX
> > 46875 ± 48% +361.2% 216187 ± 8% softirqs.CPU33.SCHED
> > 30026 ± 10% -84.0% 4803 ± 60% softirqs.CPU33.TIMER
> > 21459214 ± 3% -28.2% 15402703 ± 6% softirqs.CPU34.NET_RX
> > 45836 ± 47% +371.2% 215962 ± 8% softirqs.CPU34.SCHED
> > 29899 ± 10% -84.1% 4764 ± 58% softirqs.CPU34.TIMER
> > 21334265 ± 4% -28.2% 15311475 ± 6% softirqs.CPU35.NET_RX
> > 46008 ± 47% +369.7% 216108 ± 8% softirqs.CPU35.SCHED
> > 30033 ± 9% -83.8% 4872 ± 61% softirqs.CPU35.TIMER
> > 21434454 ± 3% -29.8% 15051817 ± 6% softirqs.CPU36.NET_RX
> > 47102 ± 48% +353.6% 213647 ± 8% softirqs.CPU36.SCHED
> > 30159 ± 9% -84.3% 4724 ± 59% softirqs.CPU36.TIMER
> > 21446302 ± 3% -28.5% 15324037 ± 6% softirqs.CPU37.NET_RX
> > 46540 ± 48% +365.2% 216506 ± 8% softirqs.CPU37.SCHED
> > 29955 ± 9% -84.3% 4688 ± 60% softirqs.CPU37.TIMER
> > 21408080 ± 4% -28.1% 15398099 ± 6% softirqs.CPU38.NET_RX
> > 46514 ± 47% +364.5% 216055 ± 8% softirqs.CPU38.SCHED
> > 30224 ± 10% -84.2% 4763 ± 59% softirqs.CPU38.TIMER
> > 21488127 ± 3% -28.2% 15418075 ± 6% softirqs.CPU39.NET_RX
> > 46334 ± 49% +366.6% 216186 ± 8% softirqs.CPU39.SCHED
> > 29912 ± 10% -84.1% 4757 ± 59% softirqs.CPU39.TIMER
> > 20494991 ± 5% -25.8% 15203629 ± 7% softirqs.CPU4.NET_RX
> > 36606 ± 70% +500.9% 219950 ± 8% softirqs.CPU4.SCHED
> > 30735 ± 11% -83.9% 4963 ± 64% softirqs.CPU4.TIMER
> > 21463890 ± 3% -28.3% 15380723 ± 6% softirqs.CPU40.NET_RX
> > 46863 ± 48% +361.7% 216368 ± 8% softirqs.CPU40.SCHED
> > 30014 ± 9% -84.1% 4763 ± 59% softirqs.CPU40.TIMER
> > 21482678 ± 3% -28.7% 15323146 ± 6% softirqs.CPU41.NET_RX
> > 46944 ± 48% +361.1% 216451 ± 8% softirqs.CPU41.SCHED
> > 30167 ± 9% -83.9% 4864 ± 59% softirqs.CPU41.TIMER
> > 21294399 ± 4% -28.3% 15278181 ± 6% softirqs.CPU42.NET_RX
> > 45452 ± 47% +376.3% 216501 ± 8% softirqs.CPU42.SCHED
> > 29866 ± 10% -82.6% 5185 ± 54% softirqs.CPU42.TIMER
> > 21365880 ± 3% -28.1% 15358507 ± 6% softirqs.CPU43.NET_RX
> > 46603 ± 49% +364.4% 216428 ± 8% softirqs.CPU43.SCHED
> > 30073 ± 9% -84.0% 4809 ± 59% softirqs.CPU43.TIMER
> > 21520046 ± 3% -28.2% 15449623 ± 6% softirqs.CPU44.NET_RX
> > 46912 ± 48% +361.0% 216249 ± 8% softirqs.CPU44.SCHED
> > 29983 ± 10% -83.8% 4865 ± 58% softirqs.CPU44.TIMER
> > 21519719 ± 3% -28.4% 15408388 ± 6% softirqs.CPU45.NET_RX
> > 46866 ± 48% +361.2% 216163 ± 8% softirqs.CPU45.SCHED
> > 29935 ± 9% -83.0% 5082 ± 55% softirqs.CPU45.TIMER
> > 21419897 ± 3% -28.3% 15362156 ± 6% softirqs.CPU46.NET_RX
> > 46939 ± 48% +361.6% 216649 ± 8% softirqs.CPU46.SCHED
> > 29931 ± 10% -83.4% 4981 ± 56% softirqs.CPU46.TIMER
> > 21420535 ± 3% -28.7% 15271884 ± 6% softirqs.CPU47.NET_RX
> > 45846 ± 47% +370.1% 215519 ± 8% softirqs.CPU47.SCHED
> > 29858 ± 10% -84.1% 4740 ± 60% softirqs.CPU47.TIMER
> > 20312914 ± 5% -25.7% 15093130 ± 7% softirqs.CPU48.NET_RX
> > 36150 ± 69% +505.3% 218836 ± 9% softirqs.CPU48.SCHED
> > 30533 ± 12% -84.7% 4657 ± 65% softirqs.CPU48.TIMER
> > 20470745 ± 4% -28.8% 14574011 ± 8% softirqs.CPU49.NET_RX
> > 36483 ± 70% +488.7% 214762 ± 11% softirqs.CPU49.SCHED
> > 30893 ± 11% -85.0% 4630 ± 54% softirqs.CPU49.TIMER
> > 20156700 ± 7% -24.7% 15172321 ± 7% softirqs.CPU5.NET_RX
> > 35518 ± 68% +517.8% 219449 ± 9% softirqs.CPU5.SCHED
> > 30265 ± 13% -83.0% 5157 ± 53% softirqs.CPU5.TIMER
> > 20423756 ± 5% -26.0% 15104088 ± 7% softirqs.CPU50.NET_RX
> > 36375 ± 70% +499.0% 217904 ± 9% softirqs.CPU50.SCHED
> > 30692 ± 11% -83.8% 4974 ± 56% softirqs.CPU50.TIMER
> > 20528259 ± 5% -27.9% 14802498 ± 11% softirqs.CPU51.NET_RX
> > 36471 ± 71% +481.1% 211940 ± 9% softirqs.CPU51.SCHED
> > 30830 ± 11% -84.8% 4699 ± 64% softirqs.CPU51.TIMER
> > 20406992 ± 4% -25.5% 15212005 ± 7% softirqs.CPU52.NET_RX
> > 36584 ± 70% +498.1% 218812 ± 9% softirqs.CPU52.SCHED
> > 30640 ± 11% -83.2% 5158 ± 60% softirqs.CPU52.TIMER
> > 20215189 ± 6% -25.1% 15148729 ± 7% softirqs.CPU53.NET_RX
> > 35623 ± 68% +515.2% 219165 ± 9% softirqs.CPU53.SCHED
> > 30368 ± 13% -82.7% 5247 ± 54% softirqs.CPU53.TIMER
> > 20363531 ± 5% -25.9% 15088299 ± 7% softirqs.CPU54.NET_RX
> > 36829 ± 70% +495.7% 219407 ± 9% softirqs.CPU54.SCHED
> > 30604 ± 12% -84.6% 4718 ± 63% softirqs.CPU54.TIMER
> > 20446122 ± 5% -26.0% 15123333 ± 7% softirqs.CPU55.NET_RX
> > 36648 ± 70% +496.4% 218572 ± 9% softirqs.CPU55.SCHED
> > 30692 ± 11% -84.3% 4807 ± 61% softirqs.CPU55.TIMER
> > 20460809 ± 5% -26.2% 15103440 ± 6% softirqs.CPU56.NET_RX
> > 36546 ± 71% +499.2% 218984 ± 9% softirqs.CPU56.SCHED
> > 30796 ± 11% -85.0% 4612 ± 60% softirqs.CPU56.TIMER
> > 20579289 ± 5% -25.7% 15290133 ± 7% softirqs.CPU57.NET_RX
> > 36635 ± 70% +497.9% 219057 ± 9% softirqs.CPU57.SCHED
> > 30824 ± 11% -84.1% 4887 ± 61% softirqs.CPU57.TIMER
> > 20340209 ± 5% -25.1% 15226056 ± 7% softirqs.CPU58.NET_RX
> > 36661 ± 70% +497.4% 219034 ± 9% softirqs.CPU58.SCHED
> > 30643 ± 12% -83.8% 4958 ± 67% softirqs.CPU58.TIMER
> > 20407456 ± 5% -31.2% 14032788 ± 14% softirqs.CPU59.NET_RX
> > 36581 ± 70% +445.7% 199631 ± 7% softirqs.CPU59.SCHED
> > 30705 ± 11% -84.8% 4662 ± 66% softirqs.CPU59.TIMER
> > 20391632 ± 5% -26.0% 15081271 ± 7% softirqs.CPU6.NET_RX
> > 36990 ± 70% +492.1% 219031 ± 9% softirqs.CPU6.SCHED
> > 31051 ± 10% -84.9% 4685 ± 66% softirqs.CPU6.TIMER
> > 20313212 ± 5% -25.8% 15082404 ± 7% softirqs.CPU60.NET_RX
> > 36711 ± 70% +497.9% 219487 ± 9% softirqs.CPU60.SCHED
> > 30611 ± 12% -77.9% 6758 ± 62% softirqs.CPU60.TIMER
> > 20370636 ± 5% -25.5% 15182632 ± 7% softirqs.CPU61.NET_RX
> > 36456 ± 69% +499.3% 218484 ± 9% softirqs.CPU61.SCHED
> > 30655 ± 11% -83.6% 5031 ± 56% softirqs.CPU61.TIMER
> > 20447745 ± 5% -25.8% 15175825 ± 7% softirqs.CPU62.NET_RX
> > 36878 ± 70% +491.8% 218266 ± 9% softirqs.CPU62.SCHED
> > 30914 ± 12% -83.8% 5002 ± 63% softirqs.CPU62.TIMER
> > 20585734 ± 4% -26.3% 15163240 ± 7% softirqs.CPU63.NET_RX
> > 36766 ± 69% +493.6% 218247 ± 9% softirqs.CPU63.SCHED
> > 30783 ± 11% -84.3% 4848 ± 63% softirqs.CPU63.TIMER
> > 20529909 ± 5% -26.0% 15201926 ± 7% softirqs.CPU64.NET_RX
> > 36996 ± 69% +492.3% 219138 ± 9% softirqs.CPU64.SCHED
> > 31589 ± 12% -84.3% 4944 ± 59% softirqs.CPU64.TIMER
> > 20469049 ± 5% -26.2% 15105347 ± 7% softirqs.CPU65.NET_RX
> > 36816 ± 70% +495.3% 219168 ± 9% softirqs.CPU65.SCHED
> > 31163 ± 11% -84.5% 4843 ± 62% softirqs.CPU65.TIMER
> > 20451275 ± 5% -26.1% 15117422 ± 7% softirqs.CPU66.NET_RX
> > 36621 ± 70% +498.4% 219158 ± 9% softirqs.CPU66.SCHED
> > 31143 ± 11% -84.5% 4812 ± 63% softirqs.CPU66.TIMER
> > 20535600 ± 4% -26.1% 15183175 ± 7% softirqs.CPU67.NET_RX
> > 36716 ± 70% +496.0% 218827 ± 9% softirqs.CPU67.SCHED
> > 31123 ± 11% -84.5% 4811 ± 62% softirqs.CPU67.TIMER
> > 20606503 ± 4% -25.9% 15263709 ± 7% softirqs.CPU68.NET_RX
> > 36843 ± 71% +493.6% 218696 ± 9% softirqs.CPU68.SCHED
> > 31375 ± 10% -84.5% 4850 ± 61% softirqs.CPU68.TIMER
> > 20565997 ± 4% -26.1% 15207162 ± 7% softirqs.CPU69.NET_RX
> > 36903 ± 70% +493.4% 218980 ± 9% softirqs.CPU69.SCHED
> > 31377 ± 10% -84.6% 4840 ± 63% softirqs.CPU69.TIMER
> > 20398774 ± 5% -26.3% 15029635 ± 7% softirqs.CPU7.NET_RX
> > 36601 ± 69% +495.4% 217915 ± 9% softirqs.CPU7.SCHED
> > 30791 ± 11% -84.9% 4640 ± 66% softirqs.CPU7.TIMER
> > 20433298 ± 4% -26.2% 15079571 ± 7% softirqs.CPU70.NET_RX
> > 36471 ± 70% +499.8% 218755 ± 9% softirqs.CPU70.SCHED
> > 31125 ± 11% -84.6% 4791 ± 61% softirqs.CPU70.TIMER
> > 20514519 ± 4% -26.6% 15064099 ± 7% softirqs.CPU71.NET_RX
> > 36787 ± 70% +493.4% 218283 ± 9% softirqs.CPU71.SCHED
> > 31230 ± 11% -84.3% 4914 ± 64% softirqs.CPU71.TIMER
> > 21343935 ± 3% -29.8% 14976558 ± 6% softirqs.CPU72.NET_RX
> > 47439 ± 47% +356.6% 216628 ± 8% softirqs.CPU72.SCHED
> > 33558 ± 25% -86.3% 4611 ± 62% softirqs.CPU72.TIMER
> > 21417052 ± 3% -28.3% 15349903 ± 6% softirqs.CPU73.NET_RX
> > 46794 ± 48% +362.8% 216549 ± 8% softirqs.CPU73.SCHED
> > 30149 ± 10% -83.8% 4893 ± 57% softirqs.CPU73.TIMER
> > 21398667 ± 3% -28.4% 15321598 ± 6% softirqs.CPU74.NET_RX
> > 117019 ± 29% -36.5% 74286 ± 24% softirqs.CPU74.RCU
> > 46710 ± 47% +363.0% 216256 ± 8% softirqs.CPU74.SCHED
> > 30250 ± 10% -84.3% 4756 ± 61% softirqs.CPU74.TIMER
> > 21158934 ± 6% -27.3% 15387291 ± 6% softirqs.CPU75.NET_RX
> > 116793 ± 29% -36.3% 74379 ± 24% softirqs.CPU75.RCU
> > 46862 ± 48% +361.5% 216265 ± 8% softirqs.CPU75.SCHED
> > 29925 ± 11% -83.8% 4847 ± 57% softirqs.CPU75.TIMER
> > 21327908 ± 4% -27.8% 15394469 ± 6% softirqs.CPU76.NET_RX
> > 45803 ± 47% +372.3% 216338 ± 8% softirqs.CPU76.SCHED
> > 30109 ± 10% -83.7% 4920 ± 57% softirqs.CPU76.TIMER
> > 21477994 ± 4% -28.6% 15330910 ± 6% softirqs.CPU77.NET_RX
> > 46487 ± 47% +365.6% 216466 ± 8% softirqs.CPU77.SCHED
> > 30216 ± 10% -84.4% 4726 ± 59% softirqs.CPU77.TIMER
> > 21460057 ± 4% -29.1% 15225183 ± 6% softirqs.CPU78.NET_RX
> > 46884 ± 48% +362.1% 216676 ± 8% softirqs.CPU78.SCHED
> > 30421 ± 10% -84.4% 4730 ± 60% softirqs.CPU78.TIMER
> > 21541321 ± 3% -28.9% 15319139 ± 6% softirqs.CPU79.NET_RX
> > 46875 ± 48% +362.6% 216857 ± 8% softirqs.CPU79.SCHED
> > 30230 ± 9% -82.5% 5291 ± 76% softirqs.CPU79.TIMER
> > 20493021 ± 4% -26.0% 15160469 ± 7% softirqs.CPU8.NET_RX
> > 36579 ± 70% +498.1% 218781 ± 9% softirqs.CPU8.SCHED
> > 30682 ± 11% -84.4% 4790 ± 59% softirqs.CPU8.TIMER
> > 21172364 ± 7% -27.3% 15402317 ± 6% softirqs.CPU80.NET_RX
> > 46254 ± 47% +368.0% 216468 ± 8% softirqs.CPU80.SCHED
> > 29465 ± 12% -83.2% 4953 ± 56% softirqs.CPU80.TIMER
> > 21567300 ± 3% -28.5% 15430307 ± 6% softirqs.CPU81.NET_RX
> > 46695 ± 48% +362.7% 216072 ± 8% softirqs.CPU81.SCHED
> > 30093 ± 9% -84.0% 4801 ± 60% softirqs.CPU81.TIMER
> > 21491215 ± 3% -28.3% 15417714 ± 6% softirqs.CPU82.NET_RX
> > 46133 ± 47% +368.7% 216210 ± 8% softirqs.CPU82.SCHED
> > 29921 ± 10% -76.8% 6934 ± 67% softirqs.CPU82.TIMER
> > 21460124 ± 3% -28.5% 15348172 ± 6% softirqs.CPU83.NET_RX
> > 46491 ± 48% +365.6% 216476 ± 8% softirqs.CPU83.SCHED
> > 29970 ± 10% -83.7% 4888 ± 56% softirqs.CPU83.TIMER
> > 21445385 ± 4% -30.8% 14839411 ± 9% softirqs.CPU84.NET_RX
> > 46941 ± 48% +350.7% 211545 ± 9% softirqs.CPU84.SCHED
> > 30037 ± 10% -84.3% 4718 ± 60% softirqs.CPU84.TIMER
> > 21551635 ± 3% -28.9% 15333781 ± 6% softirqs.CPU85.NET_RX
> > 46814 ± 48% +362.1% 216323 ± 8% softirqs.CPU85.SCHED
> > 29996 ± 10% -83.9% 4817 ± 59% softirqs.CPU85.TIMER
> > 21556404 ± 3% -28.6% 15382397 ± 6% softirqs.CPU86.NET_RX
> > 46934 ± 48% +360.7% 216231 ± 8% softirqs.CPU86.SCHED
> > 30053 ± 9% -83.7% 4900 ± 55% softirqs.CPU86.TIMER
> > 21549266 ± 3% -28.5% 15417462 ± 6% softirqs.CPU87.NET_RX
> > 46429 ± 49% +366.0% 216351 ± 8% softirqs.CPU87.SCHED
> > 29937 ± 10% -83.9% 4806 ± 58% softirqs.CPU87.TIMER
> > 21537823 ± 3% -28.7% 15361608 ± 6% softirqs.CPU88.NET_RX
> > 46887 ± 48% +360.1% 215706 ± 8% softirqs.CPU88.SCHED
> > 29981 ± 9% -84.1% 4781 ± 59% softirqs.CPU88.TIMER
> > 21559950 ± 3% -29.0% 15300783 ± 6% softirqs.CPU89.NET_RX
> > 46878 ± 48% +361.6% 216403 ± 8% softirqs.CPU89.SCHED
> > 30017 ± 9% -84.1% 4783 ± 59% softirqs.CPU89.TIMER
> > 20571529 ± 5% -25.7% 15286193 ± 7% softirqs.CPU9.NET_RX
> > 36585 ± 70% +498.6% 219007 ± 9% softirqs.CPU9.SCHED
> > 30724 ± 11% -83.7% 5009 ± 58% softirqs.CPU9.TIMER
> > 21116816 ± 5% -27.5% 15303522 ± 6% softirqs.CPU90.NET_RX
> > 46158 ± 48% +368.6% 216275 ± 8% softirqs.CPU90.SCHED
> > 29583 ± 11% -83.6% 4842 ± 58% softirqs.CPU90.TIMER
> > 21247319 ± 4% -27.6% 15382623 ± 6% softirqs.CPU91.NET_RX
> > 46803 ± 48% +362.0% 216238 ± 8% softirqs.CPU91.SCHED
> > 29690 ± 10% -83.6% 4856 ± 58% softirqs.CPU91.TIMER
> > 21660224 ± 3% -28.6% 15460344 ± 6% softirqs.CPU92.NET_RX
> > 46935 ± 48% +359.9% 215852 ± 8% softirqs.CPU92.SCHED
> > 30143 ± 9% -84.0% 4814 ± 59% softirqs.CPU92.TIMER
> > 21653985 ± 3% -28.9% 15405532 ± 6% softirqs.CPU93.NET_RX
> > 46972 ± 48% +359.5% 215821 ± 8% softirqs.CPU93.SCHED
> > 30034 ± 10% -83.5% 4964 ± 59% softirqs.CPU93.TIMER
> > 21602751 ± 3% -28.8% 15385232 ± 6% softirqs.CPU94.NET_RX
> > 46892 ± 48% +362.2% 216728 ± 8% softirqs.CPU94.SCHED
> > 29959 ± 10% -83.6% 4921 ± 58% softirqs.CPU94.TIMER
> > 21277877 ± 4% -28.4% 15224899 ± 6% softirqs.CPU95.NET_RX
> > 45607 ± 48% +373.5% 215952 ± 8% softirqs.CPU95.SCHED
> > 29711 ± 10% -83.0% 5065 ± 55% softirqs.CPU95.TIMER
> > 2.009e+09 -27.4% 1.46e+09 ± 5% softirqs.NET_RX
> > 3996567 ± 14% +421.3% 20835185 ± 8% softirqs.SCHED
> > 2929735 ± 3% -83.8% 474988 ± 53% softirqs.TIMER
> >
> >
> >
> >
> > tbench.throughput-
> > MB_sec
> >
> >
> > 16000 +--------------------------------------------------------
> > -----------+
> > | .+. .+.
> > |
> > 15000 |.+.+..+. .+.+.+..+ +.+.+..+.+ +. .+..+.
> > .+.+. .+.+.+.+..+.+.|
> > | + + + +..+
> > |
> > 14000 |-
> > + |
> >
> > |
> > |
> > 13000 |-
> > + |
> >
> > |
> > |
> > 12000 |-
> > + |
> >
> > | O O O O
> > O O |
> > 11000 |-+ O O
> > O O O O |
> > | O O O O O O O O O O O O
> > O O O |
> > 10000 |-+
> > O O O |
> >
> > |
> > |
> > 9000 +--------------------------------------------------------
> > -----------+
> >
> >
> >
> >
> >
> > tbench.time.user_time
> >
> >
> >
> > 5200 +-----------------------------------------------------------
> > ---------+
> > 5000 |-+.+.. .+.+ +.. .+.+.. + + + : .+.
> > .+.. .|
> > |.+ +.+. .+..+ + + +.+ +. .+ +
> > .+ : .+ + +.+ |
> > 4800 |-
> > + + + + + +.+.
> > |
> > 4600 |-
> > +
> > |
> > |
> > |
> > 4400 |-
> > +
> > |
> > 4200 |-
> > +
> > |
> > 4000 |-+ O O O
> > O O |
> > |
> > O O O O O
> > |
> > 3800 |-+ O O O
> > O |
> > 3600 |-+ O O O O O O
> > O O |
> > | O O O
> > O O |
> > 3400 |-
> > + O O O
> > |
> > 3200 +-----------------------------------------------------------
> > ---------+
> >
> >
> >
> >
> >
> > tbench.time.system_time
> >
> >
> >
> > 34000 +--------------------------------------------------------
> > -----------+
> > 32000 |.+ .+ : .+.. .+.+.
> > .+. .+. .+ +.+..+.+.|
> > | +.+..+ :
> > .+..+.+.+.+ + + +..+ +.+.+..+ |
> > 30000 |-
> > + + |
> >
> > 28000 |-
> > + |
> >
> > |
> > |
> > 26000 |-
> > + |
> >
> > 24000 |-
> > + |
> >
> > 22000 |-
> > + |
> >
> > |
> > |
> > 20000 |-
> > + O O O |
> >
> > 18000 |-
> > O O O O O O |
> >
> > | O O O O O
> > O O O O O |
> > 16000 |-+ O O O O
> > O O O O O O |
> > 14000 +--------------------------------------------------------
> > -----------+
> >
> >
> >
> >
> >
> > tbench.time.percent_of_cpu_this_job_got
> >
> >
> >
> > 5500 +-----------------------------------------------------------
> > ---------+
> > |. +.+ .+. .+.+..
> > .+ .+ .+.+. .+.+.|
> > 5000 |-+. + + .+.+.+.+.+. + + + .+. +
> > .+.+.+. +.+. |
> > | +..+ +. + +
> > |
> > |
> > |
> > 4500 |-
> > +
> > |
> > |
> > |
> > 4000 |-
> > +
> > |
> > |
> > |
> > 3500 |-
> > +
> > |
> > |
> > O |
> > | O O O
> > O O |
> > 3000 |-O O O O
> > O O O |
> > | O O O O O O O O O O O O
> > O O O |
> > 2500 +-----------------------------------------------------------
> > ---------+
> >
> >
> >
> >
> >
> > tbench.time.voluntary_context_switches
> >
> >
> >
> > 1.4e+09 +------------------------------------------------------
> > -----------+
> > | O
> > O O O O O |
> > 1.2e+09 |-O O O O O O O O O O O O
> > O O O O |
> > | O O O O O
> > |
> > 1e+09 |-
> > + O O O |
> >
> > |
> > |
> > 8e+08 |-
> > + |
> >
> > |
> > |
> > 6e+08 |-
> > + |
> >
> > |
> > |
> > 4e+08 |-
> > + |
> >
> > | .+.+.. +. +. +.
> > |
> > 2e+08 |.+ + +.+.+.+.+.. .+. .+. + +..
> > + +.+.+. .+..+. .|
> > | +.+ + +.+ + + +.+
> > +.+ |
> > 0 +------------------------------------------------------
> > -----------+
> >
> >
> >
> >
> >
> > tbench.time.involuntary_context_switches
> >
> >
> >
> > 2e+09 +------------------------------------------------------
> > -----------+
> > 1.8e+09 |.+. .+ + .+.+.+.+.+..+.+.+ +.+. .+..+.
> > .+. .+ +..+.+.+.|
> > | +.+. + + + +.+
> > |
> > 1.6e+09 |-
> > + |
> >
> > 1.4e+09 |-
> > + |
> >
> > |
> > |
> > 1.2e+09 |-
> > + |
> >
> > 1e+09 |-
> > + |
> >
> > 8e+08 |-
> > + |
> >
> > |
> > |
> > 6e+08 |-
> > + O O O |
> >
> > 4e+08 |-
> > O O O O O O |
> >
> > | O O O O O
> > O O O |
> > 2e+08 |-+ O O O O O O O O O
> > O O O |
> > 0 +------------------------------------------------------
> > -----------+
> >
> >
> >
> >
> > [*] bisect-good sample
> > [O] bisect-bad sample
> >
> >
> >
> > Disclaimer:
> > Results have been estimated based on internal Intel analysis and
> > are provided
> > for informational purposes only. Any difference in system hardware
> > or software
> > design or configuration may affect actual performance.
> >
> >
> > ---
> > 0DAY/LKP+ Test Infrastructure Open Source
> > Technology Center
> > https://lists.01.org/hyperkitty/list/[email protected] Intel
> > Corporation
> >
> > Thanks,
> > Oliver Sang
> >
> >
> >
> > _______________________________________________
> > LKP mailing list -- [email protected]
> > To unsubscribe send an email to [email protected]
>
--
All Rights Reversed.
Hi Rik,
Do you have time to look at this? I re-test it in v5.13 and v5.14,
the regression still existed. Thanks.
On 5/27/2021 10:00 AM, Rik van Riel wrote:
> Hello,
>
> I will try to take a look at this on Friday.
>
> However, even if I manage to reproduce it on one of
> the systems I have access to, I'm still not sure how
> exactly we would root cause the issue.
>
> Is it due to
> select_idle_sibling() doing a little bit
> more work?
>
> Is it because we invoke test_idle_cores() a little
> earlier, widening the race window with CPUs going idle,
> causing select_idle_cpu to do a lot more work?
>
> Is it a locality thing where random placement on any
> core in the LLC is somehow better than placement on
> the same core as "prev" when there is no idle core?
>
> Is it tbench running
> faster when the woken up task is
> placed on the runqueue behind the current task on the
> "target" cpu, even though that CPU isn't currently
> idle, because tbench happens to go to sleep fast?
>
> In other words, I'm
> not quite sure whether this is
> a tbench (and other similar benchmark) specific thing,
> or a kernel thing, or what instrumentation we would
> want in select_idle_sibling / select_idle_cpu for us
> to root cause issues like this more easily in the
> future...
--
Zhengjun Xing