[Problem Statement]
Searching for an idle cpu/core in a LLC domain is time consuming
when the LLC domain is relatively busy. Even worse, it is possible
that after iterating the whole LLC domain, it might still
fail to find an idle cpu/core.
The following histogram is the time(usec) spent in select_idle_cpu(),
when running 224 instance of netperf:
@usec:
[1] 2 | |
[2, 4) 1 | |
[4, 8) 4 | |
[8, 16) 948 | |
[16, 32) 18420 |@@@@@@@@@@@@@@@@@@ |
[32, 64) 40391 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[64, 128) 49345 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[128, 256) 52539 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[256, 512) 45178 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[512, 1K) 28838 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[1K, 2K) 10825 |@@@@@@@@@@ |
[2K, 4K) 1352 |@ |
[4K, 8K) 24 |
The time spent in select_idle_cpu() is visible to some workloads and might
bring negative impact.
[Symptom analysis]
The patch [1] from Mel Gorman has been applied to track the efficiency
of select_idle_sibling. Copy the indicators here:
SIS Search Efficiency(se_eff%):
A ratio expressed as a percentage of runqueues scanned versus
idle CPUs found. A 100% efficiency indicates that the target,
prev or recent CPU of a task was idle at wakeup. The lower the
efficiency, the more runqueues were scanned before an idle CPU
was found.
SIS Domain Search Efficiency(dom_eff%):
Similar, except only for the slower SIS
patch.
SIS Fast Success Rate(fast_rate%):
Percentage of SIS that used target, prev or
recent CPUs.
SIS Success rate(success_rate%):
Percentage of scans that found an idle CPU.
netperf, hackbench, schbench, tbench
were tested with 25% 50% 75% 100% 125% 150% 175% 200% of cpu number
respectively. Each test lasts for 60 seconds, and repeats 3 times.
The system would reboot into a fresh environment for each benchmark testing.
Test on vanilla kernel:
schedstat_parse.py -f netperf_vanilla.log
case load se_eff% dom_eff% fast_rate% success_rate%
TCP_RR thread-28 99.978 18.535 99.995 100.000
TCP_RR thread-56 99.397 5.671 99.964 100.000
TCP_RR thread-84 21.721 6.818 73.632 100.000
TCP_RR thread-112 12.500 5.533 59.000 100.000
TCP_RR thread-140 8.524 4.535 49.020 100.000
TCP_RR thread-168 6.438 3.945 40.309 99.999
TCP_RR thread-196 5.397 3.718 32.320 99.982
TCP_RR thread-224 4.874 3.661 25.775 99.767
UDP_RR thread-28 99.988 17.704 99.997 100.000
UDP_RR thread-56 99.528 5.977 99.970 100.000
UDP_RR thread-84 24.219 6.992 76.479 100.000
UDP_RR thread-112 13.907 5.706 62.538 100.000
UDP_RR thread-140 9.408 4.699 52.519 100.000
UDP_RR thread-168 7.095 4.077 44.352 100.000
UDP_RR thread-196 5.757 3.775 35.764 99.991
UDP_RR thread-224 5.124 3.704 28.748 99.860
schedstat_parse.py -f schbench_vanilla.log
(mthread-n has 1/4 * n * 112 tasks)
case load se_eff% dom_eff% fast_rate% success_rate%
normal mthread-1 99.152 6.400 99.941 100.000
normal mthread-2 97.844 4.003 99.908 100.000
normal mthread-3 96.395 2.118 99.917 99.998
normal mthread-4 55.288 1.451 98.615 99.804
normal mthread-5 7.004 1.870 45.597 61.036
normal mthread-6 3.354 1.346 20.777 34.230
normal mthread-7 2.183 1.028 11.257 21.055
normal mthread-8 1.653 0.825 7.849 15.549
schedstat_parse.py -f hackbench_vanilla.log
(group-n has 1/4 * n * 112 tasks)
case load se_eff% dom_eff% fast_rate% success_rate%
process-pipe group-1 99.991 7.692 99.999 100.000
process-pipe group-2 99.934 4.615 99.997 100.000
process-pipe group-3 99.597 3.198 99.987 100.000
process-pipe group-4 98.378 2.464 99.958 100.000
process-pipe group-5 27.474 3.653 89.811 99.800
process-pipe group-6 20.201 4.098 82.763 99.570
process-pipe group-7 16.423 4.156 77.398 99.316
process-pipe group-8 13.165 3.920 72.232 98.828
process-sockets group-1 99.977 5.882 99.999 100.000
process-sockets group-2 99.927 5.505 99.996 100.000
process-sockets group-3 99.397 3.250 99.980 100.000
process-sockets group-4 79.680 4.258 98.864 99.998
process-sockets group-5 7.673 2.503 63.659 92.115
process-sockets group-6 4.642 1.584 58.946 88.048
process-sockets group-7 3.493 1.379 49.816 81.164
process-sockets group-8 3.015 1.407 40.845 75.500
threads-pipe group-1 99.997 0.000 100.000 100.000
threads-pipe group-2 99.894 2.932 99.997 100.000
threads-pipe group-3 99.611 4.117 99.983 100.000
threads-pipe group-4 97.703 2.624 99.937 100.000
threads-pipe group-5 22.919 3.623 87.150 99.764
threads-pipe group-6 18.016 4.038 80.491 99.557
threads-pipe group-7 14.663 3.991 75.239 99.247
threads-pipe group-8 12.242 3.808 70.651 98.644
threads-sockets group-1 99.990 6.667 99.999 100.000
threads-sockets group-2 99.940 5.114 99.997 100.000
threads-sockets group-3 99.469 4.115 99.977 100.000
threads-sockets group-4 87.528 4.038 99.400 100.000
threads-sockets group-5 6.942 2.398 59.244 88.337
threads-sockets group-6 4.359 1.954 49.448 87.860
threads-sockets group-7 2.845 1.345 41.198 77.102
threads-sockets group-8 2.871 1.404 38.512 74.312
schedstat_parse.py -f tbench_vanilla.log
case load se_eff% dom_eff% fast_rate% success_rate%
loopback thread-28 99.976 18.369 99.995 100.000
loopback thread-56 99.222 7.799 99.934 100.000
loopback thread-84 19.723 6.819 70.215 100.000
loopback thread-112 11.283 5.371 55.371 99.999
loopback thread-140 0.000 0.000 0.000 0.000
loopback thread-168 0.000 0.000 0.000 0.000
loopback thread-196 0.000 0.000 0.000 0.000
loopback thread-224 0.000 0.000 0.000 0.000
According to the test on vanilla kernel, if the system is busy,
the SIS Search Efficiency(se_eff%) drops significantly. Although some
benchmarks would finally find an idle cpu(success_rate% = 100%), it is
doubtful whether it is worth it to search the whole LLC domain.
[Proposal]
It would be ideal to have a crystal ball to predict the success rate
of finding an idle cpu/core in the LLC. If it is doomed to fail,
there is no need to search in the LLC domain. There are many potential
metrics which could be used to predict the success rate. And the
metric should be carefully chosen that, it should help reduce the
unnecessary cpu runqueue scan, but meanwhile not give up the opportunity
to find an idle cpu.
Choose average cpu utilization as the candidate, since the util_avg is
a metric of accumulated historic activity, which seems to be more accurate
than instantaneous metrics(such as rq->nr_running) on calculating the probability
of find an idle cpu. Only when the average cpu utilization has reaches
85% of the total cpu capacity, this domain is regarded as overloaded.
The reason to choose 85% is that, this is the threshold of an overloaded
LLC sched group(imbalance_pct = 117, threshold = 100 / 117 * 100%).
Furthermore, to minimize the overhead of calculating the metrics in
select_idle_cpu(), borrow the statistics from periodic load balance.
[Benchmark result]
In summary, patched kernel has better performance than vanilla
in some microbenchmarks, and no significant negative impact to other
microbenchmarks. Patched kernel also benefits a known OLTP benchmark,
which operates with system about 95% busy.
The following is the benchmark result comparison between
baseline:vanilla and compare:patched kernel. Positive compare%
indicates better performance.
OLTP(95% busy)
baseline compare
1.00 +1.41%
hackbench(Throughput)
(group-n has 1/4 * n * 112 tasks)
=========
case load baseline(std%) compare%( std%)
process-pipe group-1 1.00 ( 0.32) -0.03 ( 0.22)
process-pipe group-2 1.00 ( 0.60) +1.12 ( 0.46)
process-pipe group-3 1.00 ( 0.03) +1.44 ( 0.63)
process-pipe group-4 1.00 ( 0.44) +1.52 ( 0.66)
process-pipe group-5 1.00 ( 0.13) +0.02 ( 0.45)
process-pipe group-6 1.00 ( 0.65) -0.13 ( 0.36)
process-pipe group-7 1.00 ( 0.12) -0.35 ( 0.15)
process-pipe group-8 1.00 ( 0.05) -0.72 ( 0.09)
process-sockets group-1 1.00 ( 0.08) -0.75 ( 0.61)
process-sockets group-2 1.00 ( 0.32) +0.15 ( 0.28)
process-sockets group-3 1.00 ( 0.19) +0.33 ( 0.22)
process-sockets group-4 1.00 ( 0.40) +0.47 ( 0.22)
process-sockets group-5 1.00 ( 1.96) +6.09 ( 0.36)
process-sockets group-6 1.00 ( 0.42) -1.18 ( 0.64)
process-sockets group-7 1.00 ( 0.12) -0.57 ( 0.12)
process-sockets group-8 1.00 ( 0.14) +0.33 ( 0.29)
threads-pipe group-1 1.00 ( 0.25) -1.08 ( 0.30)
threads-pipe group-2 1.00 ( 0.59) +0.54 ( 0.60)
threads-pipe group-3 1.00 ( 0.98) -0.47 ( 0.74)
threads-pipe group-4 1.00 ( 0.24) +0.10 ( 0.14)
threads-pipe group-5 1.00 ( 0.15) -0.03 ( 0.28)
threads-pipe group-6 1.00 ( 0.28) -0.50 ( 0.18)
threads-pipe group-7 1.00 ( 0.11) -0.63 ( 0.04)
threads-pipe group-8 1.00 ( 0.08) -0.43 ( 0.04)
threads-sockets group-1 1.00 ( 0.49) -1.30 ( 0.16)
threads-sockets group-2 1.00 ( 0.19) -0.18 ( 0.09)
threads-sockets group-3 1.00 ( 0.28) +0.50 ( 0.04)
threads-sockets group-4 1.00 ( 1.36) +1.50 ( 0.78)
threads-sockets group-5 1.00 ( 0.33) +1.33 ( 0.50)
threads-sockets group-6 1.00 ( 0.21) +0.36 ( 0.16)
threads-sockets group-7 1.00 ( 0.09) -0.36 ( 0.15)
threads-sockets group-8 1.00 ( 0.12) +1.07 ( 0.03)
netperf(Throughput)
=======
case load baseline(std%) compare%( std%)
TCP_RR thread-28 1.00 ( 0.29) -0.23 ( 0.42)
TCP_RR thread-56 1.00 ( 1.03) +0.42 ( 1.00)
TCP_RR thread-84 1.00 ( 0.80) +0.39 ( 0.84)
TCP_RR thread-112 1.00 ( 0.43) +0.45 ( 0.35)
TCP_RR thread-140 1.00 ( 0.26) +0.58 ( 0.32)
TCP_RR thread-168 1.00 ( 0.21) +0.69 ( 0.23)
TCP_RR thread-196 1.00 ( 0.20) +0.77 ( 0.18)
TCP_RR thread-224 1.00 ( 0.18) +164.89 ( 44.90)
UDP_RR thread-28 1.00 ( 0.38) -0.48 ( 0.34)
UDP_RR thread-56 1.00 ( 0.59) +1.07 ( 0.62)
UDP_RR thread-84 1.00 ( 1.04) +0.44 ( 1.03)
UDP_RR thread-112 1.00 ( 14.96) +2.53 ( 32.65)
UDP_RR thread-140 1.00 ( 14.83) +0.96 ( 18.83)
UDP_RR thread-168 1.00 ( 13.12) +0.76 ( 8.64)
UDP_RR thread-196 1.00 ( 9.67) +1.32 ( 18.53)
UDP_RR thread-224 1.00 ( 14.10) +126.46 ( 42.27)
tbench(Throughput)
======
case load baseline(std%) compare%( std%)
loopback thread-28 1.00 ( 0.12) +0.25 ( 0.08)
loopback thread-56 1.00 ( 0.10) +0.15 ( 0.11)
loopback thread-84 1.00 ( 0.11) +0.09 ( 0.09)
loopback thread-112 1.00 ( 0.08) -0.11 ( 0.09)
loopback thread-140 1.00 ( 0.17) +2.54 ( 0.24)
loopback thread-168 1.00 ( 0.04) +2.44 ( 0.16)
loopback thread-196 1.00 ( 0.10) +2.41 ( 0.10)
loopback thread-224 1.00 ( 0.17) +2.30 ( 0.10)
schbench(Latency)
(mthread-n has 1/4 * n * 112 tasks)
========
case load baseline(std%) compare%( std%)
normal mthread-1 1.00 ( 1.01) +0.57 ( 1.06)
normal mthread-2 1.00 ( 20.99) +14.65 ( 4.67)
normal mthread-3 1.00 ( 3.03) -0.12 ( 3.41)
normal mthread-4 1.00 ( 0.76) -4.71 ( 21.04)
normal mthread-5 1.00 ( 1.80) -4.90 ( 1.39)
normal mthread-6 1.00 ( 0.08) -0.59 ( 1.56)
normal mthread-7 1.00 ( 2.68) -1.70 ( 3.53)
normal mthread-8 1.00 ( 2.19) -0.50 ( 3.46)
According to the results above, when the workloads is 2 times of the cpu
number(224), the throughput of netperf improves a lot(164.89% and 126.46%).
It might be interesting to look into the reason why this patch benefits
netperf significantly. Further investigation has shown that, the system
utilization is around 90% on vanilla kernel, while 100% on patched kernel.
It is still under investigation why vanilla kernel could not reach 100%.
For the latency result from schbench, there seems to be some regressions.
And consider the standard deviation ratio, only when the running task is
125% of the cpu number, there is a valid regression.
[Limitation]
A corner case needs to be covered that, it is possible that there is unbalanced
tasks among CPUs due to cpu affinity. For example, suppose the LLC domain is
composed of 8 cpus, and 10 tasks are bound to cpu0~cpu6, while cpu7 is idle:
cpu0 cpu1 cpu2 cpu3 cpu4 cpu5 cpu6 cpu7
util_avg 1024 1024 1024 1024 1024 1024 1024 0
Since the total average util is higher than 85% of cpu capacity
(7 * 1024 / 8 * 1024 = 87.5%), this LLC domain is regarded as overloaded and
will not be scanned, leaving the idle cpu7 undetected.
A possible workaround to mitigate this problem is that, the idle cpu's capacity
can be subtracted from the total cpu util. In the example above, the adjusted
total average util becomes 1024 * 7 - 1024, and the util ratio is 6/8 = 75%,
as it is lower than 85%. the idle cpu searching will continue. But currently
I don't have better solution in mind to deal with it gracefully.
Link: https://lore.kernel.org/lkml/[email protected]/ #1
Suggested-by: Tim Chen <[email protected]>
Signed-off-by: Chen Yu <[email protected]>
---
include/linux/sched/topology.h | 1 +
kernel/sched/fair.c | 30 ++++++++++++++++++++++++++++++
2 files changed, 31 insertions(+)
diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h
index 8054641c0a7b..ec0cc3fe5b0a 100644
--- a/include/linux/sched/topology.h
+++ b/include/linux/sched/topology.h
@@ -81,6 +81,7 @@ struct sched_domain_shared {
atomic_t ref;
atomic_t nr_busy_cpus;
int has_idle_cores;
+ bool overloaded;
};
struct sched_domain {
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 5146163bfabb..1a58befe892d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6273,6 +6273,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
int i, cpu, idle_cpu = -1, nr = INT_MAX;
struct rq *this_rq = this_rq();
int this = smp_processor_id();
+ struct sched_domain_shared *sd_share;
struct sched_domain *this_sd;
u64 time = 0;
@@ -6280,6 +6281,10 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
if (!this_sd)
return -1;
+ sd_share = rcu_dereference(per_cpu(sd_llc_shared, target));
+ if (sd_share && READ_ONCE(sd_share->overloaded))
+ return -1;
+
cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr);
if (sched_feat(SIS_PROP) && !has_idle_core) {
@@ -9212,6 +9217,7 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd
struct sg_lb_stats *local = &sds->local_stat;
struct sg_lb_stats tmp_sgs;
int sg_status = 0;
+ unsigned long sum_util = 0;
do {
struct sg_lb_stats *sgs = &tmp_sgs;
@@ -9242,6 +9248,7 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd
/* Now, start updating sd_lb_stats */
sds->total_load += sgs->group_load;
sds->total_capacity += sgs->group_capacity;
+ sum_util += sgs->group_util;
sg = sg->next;
} while (sg != env->sd->groups);
@@ -9268,6 +9275,29 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd
WRITE_ONCE(rd->overutilized, SG_OVERUTILIZED);
trace_sched_overutilized_tp(rd, SG_OVERUTILIZED);
}
+
+ /*
+ * Check if the LLC domain is overloaded. The overload hint
+ * could be used to skip the LLC domain idle cpu search in
+ * select_idle_cpu(). The update of this hint occurs during
+ * periodic load balancing, rather than frequent newidle balance.
+ */
+ if (env->idle != CPU_NEWLY_IDLE &&
+ env->sd->span_weight == per_cpu(sd_llc_size, env->dst_cpu)) {
+ struct sched_domain_shared *sd_share =
+ rcu_dereference(per_cpu(sd_llc_shared, env->dst_cpu));
+
+ if (!sd_share)
+ return;
+
+ /*
+ * Derived from group_is_overloaded(). The default imbalance_pct
+ * is 117 on LLC domain, which means the threshold of average
+ * utilization is 85%.
+ */
+ WRITE_ONCE(sd_share->overloaded, (sds->total_capacity * 100) <
+ (sum_util * env->sd->imbalance_pct));
+ }
}
#define NUMA_IMBALANCE_MIN 2
--
2.25.1
Greeting,
FYI, we noticed a 46.9% improvement of stress-ng.dccp.ops_per_sec due to commit:
commit: e9accc2386f31d149ea339e8be401ad36361c9be ("[PATCH][RFC] sched: Stop searching for idle cpu if the LLC domain is overloaded")
url: https://github.com/0day-ci/linux/commits/Chen-Yu/sched-Stop-searching-for-idle-cpu-if-the-LLC-domain-is-overloaded/20220207-135940
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 6275d291a763090ed1ff8aed7376aebd9f9dd6f5
patch link: https://lore.kernel.org/lkml/[email protected]
in testcase: stress-ng
on test machine: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with 128G memory
with following parameters:
nr_threads: 100%
testtime: 60s
class: network
test: dccp
cpufreq_governor: performance
ucode: 0xd000280
In addition to that, the commit also has significant impact on the following tests:
+------------------+---------------------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_Mbps 16.8% improvement |
| test machine | 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G memory |
| test parameters | cluster=cs-localhost |
| | cpufreq_governor=performance |
| | ip=ipv4 |
| | nr_threads=50% |
| | runtime=300s |
| | send_size=5K |
| | test=TCP_SENDFILE |
| | ucode=0xd000280 |
+------------------+---------------------------------------------------------------------------------+
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
sudo bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
sudo bin/lkp run generated-yaml-file
# if come across any failure that blocks the test,
# please remove ~/.lkp and /lkp dir to run from a clean state.
=========================================================================================
class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
network/gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-icl-2sp6/dccp/stress-ng/60s/0xd000280
commit:
6275d291a7 ("Merge irq/core into tip/master")
e9accc2386 ("sched: Stop searching for idle cpu if the LLC domain is overloaded")
6275d291a763090e e9accc2386f31d149ea339e8be4
---------------- ---------------------------
%stddev %change %stddev
\ | \
90045 +46.9% 132301 stress-ng.dccp.messages_per_sec
10977133 +46.9% 16128324 stress-ng.dccp.ops
182946 +46.9% 268798 stress-ng.dccp.ops_per_sec
1962636 ? 9% +50620.0% 9.955e+08 stress-ng.time.involuntary_context_switches
23828 -13.0% 20733 stress-ng.time.minor_page_faults
321.72 ? 2% -27.0% 234.84 stress-ng.time.user_time
2.002e+08 +408.6% 1.018e+09 stress-ng.time.voluntary_context_switches
1.235e+09 -52.4% 5.879e+08 ? 11% cpuidle..time
1.61e+08 ? 2% -85.6% 23133750 ? 21% cpuidle..usage
47363 ? 10% +26.7% 60009 ? 17% numa-meminfo.node1.Mapped
145038 ? 3% +10.0% 159495 ? 3% numa-meminfo.node1.Shmem
14.71 -8.2 6.49 ? 16% mpstat.cpu.all.idle%
1.43 -0.4 1.01 ? 5% mpstat.cpu.all.irq%
21.85 +9.1 30.94 mpstat.cpu.all.soft%
4.39 -1.3 3.10 mpstat.cpu.all.usr%
11920497 +37.7% 16417333 numa-numastat.node0.local_node
11960088 +37.7% 16473416 numa-numastat.node0.numa_hit
11819045 +38.4% 16362446 numa-numastat.node1.local_node
11894761 +38.0% 16420275 numa-numastat.node1.numa_hit
7316794 +27.4% 9320086 ? 2% numa-vmstat.node0.numa_hit
7277597 +27.3% 9262524 ? 2% numa-vmstat.node0.numa_local
6104502 +34.8% 8228122 ? 3% numa-vmstat.node1.numa_hit
5995904 +35.7% 8138907 ? 3% numa-vmstat.node1.numa_local
16.83 ? 2% -48.5% 8.67 ? 12% vmstat.cpu.id
123.00 +40.7% 173.00 vmstat.procs.r
5869578 +430.0% 31106336 vmstat.system.cs
759060 -58.5% 314852 vmstat.system.in
3271 -6.6% 3054 turbostat.Bzy_MHz
1.349e+08 ? 2% -85.2% 20007631 ? 20% turbostat.C1
7.21 ? 2% -6.4 0.86 ? 18% turbostat.C1%
23689617 -93.7% 1490049 ? 10% turbostat.C1E
5.95 ? 35% -4.3 1.64 ?103% turbostat.C1E%
49285246 -58.3% 20566340 turbostat.IRQ
40345 ? 2% +9.7% 44254 proc-vmstat.nr_shmem
86949 -7.8% 80178 proc-vmstat.nr_slab_unreclaimable
5499 ? 45% -74.7% 1390 ? 16% proc-vmstat.numa_hint_faults
1608 ? 14% -75.9% 388.17 ? 42% proc-vmstat.numa_hint_faults_local
23856870 +38.0% 32925200 proc-vmstat.numa_hit
23741562 +38.2% 32811289 proc-vmstat.numa_local
70447 ? 3% +9.0% 76757 ? 2% proc-vmstat.pgactivate
23854818 +38.0% 32923563 proc-vmstat.pgalloc_normal
443817 -2.1% 434710 proc-vmstat.pgfault
23637556 +38.3% 32697359 proc-vmstat.pgfree
21388 +8.0% 23108 ? 2% proc-vmstat.pgreuse
100466 ? 27% -82.6% 17526 ?116% sched_debug.cfs_rq:/.MIN_vruntime.avg
2432999 -53.8% 1124652 ? 99% sched_debug.cfs_rq:/.MIN_vruntime.max
476335 ? 14% -71.5% 135698 ?104% sched_debug.cfs_rq:/.MIN_vruntime.stddev
0.59 ? 5% +35.6% 0.79 ? 2% sched_debug.cfs_rq:/.h_nr_running.avg
5426 ? 3% +77.8% 9647 ? 39% sched_debug.cfs_rq:/.load.avg
100466 ? 27% -82.6% 17526 ?116% sched_debug.cfs_rq:/.max_vruntime.avg
2432999 -53.8% 1124652 ? 99% sched_debug.cfs_rq:/.max_vruntime.max
476335 ? 14% -71.5% 135698 ?104% sched_debug.cfs_rq:/.max_vruntime.stddev
2060801 -10.9% 1835538 ? 2% sched_debug.cfs_rq:/.min_vruntime.min
39916 ? 8% +20.6% 48123 ? 4% sched_debug.cfs_rq:/.min_vruntime.stddev
0.29 ? 7% -37.7% 0.18 ? 23% sched_debug.cfs_rq:/.nr_running.stddev
410.63 ? 4% -44.4% 228.36 ? 7% sched_debug.cfs_rq:/.runnable_avg.stddev
39957 ? 8% +20.6% 48195 ? 4% sched_debug.cfs_rq:/.spread0.stddev
2089 ? 12% -38.0% 1295 ? 7% sched_debug.cfs_rq:/.util_avg.max
399.61 ? 4% -51.4% 194.19 ? 5% sched_debug.cfs_rq:/.util_avg.stddev
327.45 ? 3% +41.1% 461.93 sched_debug.cfs_rq:/.util_est_enqueued.avg
243.96 ? 5% -38.9% 148.96 ? 6% sched_debug.cfs_rq:/.util_est_enqueued.stddev
694791 ? 21% +62.7% 1130760 ? 10% sched_debug.cpu.avg_idle.max
149474 ? 6% +43.6% 214589 ? 4% sched_debug.cpu.avg_idle.stddev
4.34 ? 12% +100.5% 8.70 ? 37% sched_debug.cpu.clock.stddev
3136 ? 2% +11.2% 3488 sched_debug.cpu.curr->pid.avg
1694 ? 7% -50.2% 843.45 ? 27% sched_debug.cpu.curr->pid.stddev
0.58 ? 3% +40.8% 0.81 ? 3% sched_debug.cpu.nr_running.avg
1422138 +431.9% 7563774 sched_debug.cpu.nr_switches.avg
1536440 ? 3% +412.3% 7871294 sched_debug.cpu.nr_switches.max
1211442 ? 2% +403.3% 6097679 ? 3% sched_debug.cpu.nr_switches.min
52546 ? 23% +280.0% 199667 ? 7% sched_debug.cpu.nr_switches.stddev
7.67 ? 2% -90.1% 0.76 ? 25% perf-stat.i.MPKI
5.135e+10 +32.3% 6.795e+10 perf-stat.i.branch-instructions
1.07 +0.4 1.44 perf-stat.i.branch-miss-rate%
5.208e+08 +79.7% 9.36e+08 perf-stat.i.branch-misses
2.43 ? 5% +20.1 22.48 ? 4% perf-stat.i.cache-miss-rate%
46712186 -46.5% 24986570 perf-stat.i.cache-misses
2.056e+09 -94.4% 1.144e+08 ? 5% perf-stat.i.cache-references
6076690 +431.7% 32310480 perf-stat.i.context-switches
1.41 -25.8% 1.05 perf-stat.i.cpi
3.795e+11 -2.7% 3.693e+11 perf-stat.i.cpu-cycles
412360 ? 2% -99.6% 1832 ? 9% perf-stat.i.cpu-migrations
8002 +81.2% 14502 perf-stat.i.cycles-between-cache-misses
0.01 ? 14% -0.0 0.01 ? 43% perf-stat.i.dTLB-load-miss-rate%
10157566 ? 6% -88.7% 1146598 ? 6% perf-stat.i.dTLB-load-misses
7.806e+10 +32.0% 1.03e+11 perf-stat.i.dTLB-loads
0.01 ? 14% -0.0 0.00 ? 30% perf-stat.i.dTLB-store-miss-rate%
1874231 ? 8% -87.9% 227410 perf-stat.i.dTLB-store-misses
4.738e+10 +30.5% 6.181e+10 perf-stat.i.dTLB-stores
2.666e+11 +33.5% 3.558e+11 perf-stat.i.instructions
0.71 +34.6% 0.96 perf-stat.i.ipc
2.96 -2.7% 2.89 perf-stat.i.metric.GHz
255.07 ? 2% +307.7% 1039 ? 3% perf-stat.i.metric.K/sec
1397 +30.2% 1819 perf-stat.i.metric.M/sec
66.73 +24.4 91.16 perf-stat.i.node-load-miss-rate%
4016095 -17.6% 3310212 ? 2% perf-stat.i.node-load-misses
2005931 -85.5% 291098 ? 8% perf-stat.i.node-loads
23.26 +48.7 71.99 perf-stat.i.node-store-miss-rate%
13385913 -88.9% 1488913 ? 3% perf-stat.i.node-stores
7.72 -95.8% 0.32 ? 5% perf-stat.overall.MPKI
1.02 +0.4 1.38 perf-stat.overall.branch-miss-rate%
2.29 +19.7 21.99 ? 4% perf-stat.overall.cache-miss-rate%
1.42 -27.1% 1.04 perf-stat.overall.cpi
8076 +82.2% 14714 perf-stat.overall.cycles-between-cache-misses
0.01 ? 6% -0.0 0.00 ? 6% perf-stat.overall.dTLB-load-miss-rate%
0.00 ? 8% -0.0 0.00 ? 2% perf-stat.overall.dTLB-store-miss-rate%
0.70 +37.2% 0.96 perf-stat.overall.ipc
66.24 +25.0 91.26 perf-stat.overall.node-load-miss-rate%
22.89 +49.7 72.60 perf-stat.overall.node-store-miss-rate%
5.055e+10 +32.5% 6.697e+10 perf-stat.ps.branch-instructions
5.13e+08 +79.8% 9.225e+08 perf-stat.ps.branch-misses
46290158 -46.5% 24744873 perf-stat.ps.cache-misses
2.026e+09 -94.4% 1.128e+08 ? 5% perf-stat.ps.cache-references
5981296 +432.5% 31852990 perf-stat.ps.context-switches
3.738e+11 -2.6% 3.64e+11 perf-stat.ps.cpu-cycles
406893 ? 2% -99.6% 1775 ? 9% perf-stat.ps.cpu-migrations
10048476 ? 6% -88.6% 1145857 ? 6% perf-stat.ps.dTLB-load-misses
7.685e+10 +32.1% 1.015e+11 perf-stat.ps.dTLB-loads
1844285 ? 7% -87.9% 223675 perf-stat.ps.dTLB-store-misses
4.664e+10 +30.6% 6.092e+10 perf-stat.ps.dTLB-stores
2.624e+11 +33.6% 3.507e+11 perf-stat.ps.instructions
3962058 -17.6% 3266404 ? 2% perf-stat.ps.node-load-misses
2019070 -84.5% 312407 ? 7% perf-stat.ps.node-loads
13289353 -88.8% 1493356 ? 3% perf-stat.ps.node-stores
1.645e+13 +35.2% 2.224e+13 perf-stat.total.instructions
3613065 +17.3% 4237109 ? 2% softirqs.CPU0.NET_RX
30760 ? 4% +38.7% 42678 ? 3% softirqs.CPU0.RCU
26366 ? 5% -74.2% 6809 ? 10% softirqs.CPU0.SCHED
13247 ? 3% -48.1% 6877 ? 2% softirqs.CPU0.TIMER
3614438 +19.0% 4302434 softirqs.CPU1.NET_RX
28024 +51.7% 42507 ? 2% softirqs.CPU1.RCU
24045 ? 3% -79.5% 4936 ? 18% softirqs.CPU1.SCHED
12244 -46.1% 6596 ? 3% softirqs.CPU1.TIMER
3612493 +17.6% 4248103 ? 2% softirqs.CPU10.NET_RX
27077 +48.3% 40149 ? 2% softirqs.CPU10.RCU
22734 ? 2% -84.9% 3438 ? 13% softirqs.CPU10.SCHED
12368 -47.0% 6552 ? 4% softirqs.CPU10.TIMER
3621450 +15.8% 4195321 ? 2% softirqs.CPU100.NET_RX
26447 +50.9% 39914 ? 2% softirqs.CPU100.RCU
22669 ? 4% -86.4% 3083 ? 21% softirqs.CPU100.SCHED
12504 ? 2% -48.7% 6409 softirqs.CPU100.TIMER
3615746 +17.1% 4233335 softirqs.CPU101.NET_RX
26385 +53.9% 40603 softirqs.CPU101.RCU
22338 ? 2% -86.4% 3038 ? 17% softirqs.CPU101.SCHED
12519 ? 2% -49.1% 6372 softirqs.CPU101.TIMER
3649966 +15.1% 4201301 softirqs.CPU102.NET_RX
26637 +49.6% 39852 ? 2% softirqs.CPU102.RCU
22663 ? 3% -87.2% 2911 ? 15% softirqs.CPU102.SCHED
12634 ? 3% -49.2% 6419 softirqs.CPU102.TIMER
3674424 +15.9% 4258664 softirqs.CPU103.NET_RX
26619 +52.5% 40597 softirqs.CPU103.RCU
22296 ? 2% -87.2% 2843 ? 18% softirqs.CPU103.SCHED
12457 -48.2% 6450 softirqs.CPU103.TIMER
3709657 +14.1% 4234036 ? 2% softirqs.CPU104.NET_RX
26782 +48.8% 39863 ? 3% softirqs.CPU104.RCU
22448 -87.9% 2724 ? 15% softirqs.CPU104.SCHED
12511 -48.6% 6436 softirqs.CPU104.TIMER
3631056 +16.2% 4220243 ? 2% softirqs.CPU105.NET_RX
26492 +51.4% 40112 ? 2% softirqs.CPU105.RCU
22383 -86.5% 3012 ? 24% softirqs.CPU105.SCHED
12320 -47.9% 6425 softirqs.CPU105.TIMER
3695635 +14.2% 4219356 ? 2% softirqs.CPU106.NET_RX
26780 +48.8% 39854 ? 2% softirqs.CPU106.RCU
22429 ? 2% -85.8% 3173 ? 27% softirqs.CPU106.SCHED
12445 ? 2% -48.4% 6416 softirqs.CPU106.TIMER
3688361 +14.2% 4213064 ? 2% softirqs.CPU107.NET_RX
26628 +50.6% 40108 ? 3% softirqs.CPU107.RCU
22334 -87.5% 2795 ? 19% softirqs.CPU107.SCHED
12485 ? 2% -48.6% 6422 softirqs.CPU107.TIMER
3601339 +16.4% 4192423 ? 2% softirqs.CPU108.NET_RX
26224 +52.5% 39986 ? 3% softirqs.CPU108.RCU
22217 ? 2% -85.5% 3211 ? 26% softirqs.CPU108.SCHED
12525 ? 2% -46.7% 6680 ? 9% softirqs.CPU108.TIMER
3629500 +16.0% 4209881 ? 2% softirqs.CPU109.NET_RX
26337 +51.3% 39860 ? 3% softirqs.CPU109.RCU
22154 -86.3% 3025 ? 23% softirqs.CPU109.SCHED
12417 ? 2% -48.2% 6438 softirqs.CPU109.TIMER
3672766 +17.2% 4303662 softirqs.CPU11.NET_RX
27062 +53.8% 41612 softirqs.CPU11.RCU
22539 ? 2% -84.0% 3602 ? 34% softirqs.CPU11.SCHED
12520 -49.0% 6386 softirqs.CPU11.TIMER
3642769 +17.8% 4290376 softirqs.CPU110.NET_RX
26083 +56.0% 40686 softirqs.CPU110.RCU
22109 -86.7% 2945 ? 18% softirqs.CPU110.SCHED
12606 ? 2% -48.9% 6441 softirqs.CPU110.TIMER
3657738 +15.2% 4215256 softirqs.CPU111.NET_RX
26506 +50.6% 39909 ? 3% softirqs.CPU111.RCU
22352 ? 2% -87.2% 2868 ? 18% softirqs.CPU111.SCHED
12239 ? 2% -47.3% 6451 softirqs.CPU111.TIMER
3649204 +16.5% 4251130 softirqs.CPU112.NET_RX
26355 +51.2% 39853 softirqs.CPU112.RCU
22318 ? 2% -87.0% 2908 ? 18% softirqs.CPU112.SCHED
12606 -49.0% 6433 softirqs.CPU112.TIMER
3611811 +15.6% 4175178 ? 2% softirqs.CPU113.NET_RX
26184 ? 2% +48.5% 38881 ? 3% softirqs.CPU113.RCU
22277 ? 2% -86.4% 3031 ? 14% softirqs.CPU113.SCHED
12579 ? 2% -48.3% 6507 softirqs.CPU113.TIMER
3588944 +18.5% 4253279 softirqs.CPU114.NET_RX
26177 +52.5% 39931 softirqs.CPU114.RCU
22289 ? 2% -86.6% 2978 ? 17% softirqs.CPU114.SCHED
12542 -48.4% 6469 softirqs.CPU114.TIMER
3689669 +13.8% 4199860 softirqs.CPU115.NET_RX
26547 +48.8% 39505 ? 3% softirqs.CPU115.RCU
22324 -85.3% 3283 ? 27% softirqs.CPU115.SCHED
12640 ? 2% -46.1% 6819 ? 13% softirqs.CPU115.TIMER
3670306 +15.3% 4230996 softirqs.CPU116.NET_RX
26503 +50.9% 39991 softirqs.CPU116.RCU
22375 ? 2% -86.5% 3023 ? 14% softirqs.CPU116.SCHED
12539 ? 2% -49.0% 6389 softirqs.CPU116.TIMER
3638583 +14.8% 4177232 ? 2% softirqs.CPU117.NET_RX
26329 +48.7% 39162 ? 3% softirqs.CPU117.RCU
22288 ? 2% -86.8% 2940 ? 13% softirqs.CPU117.SCHED
12527 -48.7% 6425 softirqs.CPU117.TIMER
3618613 +14.1% 4128766 ? 2% softirqs.CPU118.NET_RX
26412 +46.4% 38660 ? 2% softirqs.CPU118.RCU
22381 -86.7% 2975 ? 16% softirqs.CPU118.SCHED
12455 -47.9% 6487 softirqs.CPU118.TIMER
3681733 +15.7% 4261202 softirqs.CPU119.NET_RX
26542 +50.5% 39948 softirqs.CPU119.RCU
22275 -85.9% 3129 ? 20% softirqs.CPU119.SCHED
12435 ? 2% -46.4% 6661 ? 8% softirqs.CPU119.TIMER
3639537 +16.0% 4223340 ? 2% softirqs.CPU12.NET_RX
27137 +48.2% 40216 ? 2% softirqs.CPU12.RCU
22643 ? 2% -86.2% 3115 ? 20% softirqs.CPU12.SCHED
12512 ? 2% -48.8% 6407 ? 2% softirqs.CPU12.TIMER
3659696 +15.5% 4227664 softirqs.CPU120.NET_RX
26558 +49.0% 39566 ? 2% softirqs.CPU120.RCU
22438 ? 2% -87.0% 2909 ? 10% softirqs.CPU120.SCHED
12395 ? 2% -47.9% 6457 softirqs.CPU120.TIMER
3662175 +15.7% 4236037 softirqs.CPU121.NET_RX
26351 +52.1% 40078 softirqs.CPU121.RCU
22203 ? 2% -86.9% 2905 ? 16% softirqs.CPU121.SCHED
12410 -48.4% 6398 softirqs.CPU121.TIMER
3650595 +13.9% 4156471 ? 2% softirqs.CPU122.NET_RX
26327 +49.4% 39321 ? 2% softirqs.CPU122.RCU
22321 -86.7% 2977 ? 17% softirqs.CPU122.SCHED
12270 ? 3% -47.4% 6455 ? 2% softirqs.CPU122.TIMER
3608763 +16.1% 4190754 ? 2% softirqs.CPU123.NET_RX
26025 +50.3% 39127 ? 3% softirqs.CPU123.RCU
22048 ? 2% -86.4% 3007 ? 19% softirqs.CPU123.SCHED
12519 ? 2% -48.6% 6431 softirqs.CPU123.TIMER
3631782 +16.4% 4226236 softirqs.CPU124.NET_RX
26144 +51.6% 39625 ? 2% softirqs.CPU124.RCU
22153 -86.7% 2937 ? 18% softirqs.CPU124.SCHED
12573 -48.8% 6439 softirqs.CPU124.TIMER
3600753 +18.3% 4259040 softirqs.CPU125.NET_RX
26106 +54.1% 40219 softirqs.CPU125.RCU
22197 ? 2% -86.2% 3071 ? 32% softirqs.CPU125.SCHED
12560 ? 2% -48.4% 6480 softirqs.CPU125.TIMER
3643036 +15.9% 4221932 softirqs.CPU126.NET_RX
26416 +51.4% 39998 softirqs.CPU126.RCU
22361 ? 2% -85.7% 3194 ? 15% softirqs.CPU126.SCHED
12407 ? 2% -48.7% 6365 ? 2% softirqs.CPU126.TIMER
24630 ? 2% +43.5% 35356 ? 2% softirqs.CPU127.RCU
19919 ? 5% -84.5% 3097 ? 16% softirqs.CPU127.SCHED
10909 ? 5% -50.8% 5363 softirqs.CPU127.TIMER
3584110 +19.7% 4291793 softirqs.CPU13.NET_RX
26647 +56.5% 41702 ? 2% softirqs.CPU13.RCU
22555 ? 2% -83.5% 3729 ? 25% softirqs.CPU13.SCHED
12349 -44.9% 6810 ? 7% softirqs.CPU13.TIMER
3612703 +18.4% 4276695 softirqs.CPU14.NET_RX
26878 +52.6% 41016 softirqs.CPU14.RCU
22407 ? 2% -84.0% 3575 ? 22% softirqs.CPU14.SCHED
12332 -46.9% 6548 ? 6% softirqs.CPU14.TIMER
3632140 +17.8% 4280149 softirqs.CPU15.NET_RX
26970 +51.7% 40910 softirqs.CPU15.RCU
22697 -83.9% 3649 ? 19% softirqs.CPU15.SCHED
12380 -47.2% 6538 ? 4% softirqs.CPU15.TIMER
3615938 +17.3% 4241793 softirqs.CPU16.NET_RX
27866 ? 2% +45.8% 40631 softirqs.CPU16.RCU
23132 ? 3% -84.2% 3650 ? 20% softirqs.CPU16.SCHED
12466 ? 5% -47.9% 6494 ? 5% softirqs.CPU16.TIMER
3661467 +16.6% 4268092 softirqs.CPU17.NET_RX
27308 ? 2% +49.2% 40747 softirqs.CPU17.RCU
22603 ? 2% -84.6% 3476 ? 27% softirqs.CPU17.SCHED
12272 -47.2% 6477 ? 4% softirqs.CPU17.TIMER
3615459 +18.6% 4286163 softirqs.CPU18.NET_RX
26885 +53.9% 41388 ? 3% softirqs.CPU18.RCU
22564 ? 2% -83.8% 3648 ? 20% softirqs.CPU18.SCHED
12175 -46.7% 6491 ? 4% softirqs.CPU18.TIMER
3617218 +18.7% 4292429 softirqs.CPU19.NET_RX
27082 +51.6% 41044 softirqs.CPU19.RCU
22586 ? 2% -84.1% 3586 ? 15% softirqs.CPU19.SCHED
12244 ? 2% -48.1% 6356 softirqs.CPU19.TIMER
3656892 +18.3% 4327539 softirqs.CPU2.NET_RX
28667 ? 5% +45.9% 41827 ? 2% softirqs.CPU2.RCU
23403 ? 2% -80.8% 4500 ? 8% softirqs.CPU2.SCHED
12481 ? 4% -47.7% 6523 ? 2% softirqs.CPU2.TIMER
3680328 +16.8% 4297128 softirqs.CPU20.NET_RX
27910 ? 5% +47.7% 41237 softirqs.CPU20.RCU
22551 ? 2% -83.0% 3842 ? 11% softirqs.CPU20.SCHED
12410 ? 4% -47.1% 6560 ? 4% softirqs.CPU20.TIMER
3672767 +16.9% 4291707 softirqs.CPU21.NET_RX
27188 +50.7% 40985 softirqs.CPU21.RCU
22725 -83.9% 3652 ? 12% softirqs.CPU21.SCHED
12211 -47.9% 6356 softirqs.CPU21.TIMER
3645881 +17.9% 4299308 softirqs.CPU22.NET_RX
27031 ? 2% +52.0% 41083 softirqs.CPU22.RCU
22353 -83.0% 3790 ? 11% softirqs.CPU22.SCHED
12303 -48.3% 6356 softirqs.CPU22.TIMER
3648414 +15.6% 4218092 softirqs.CPU23.NET_RX
26615 +51.0% 40181 softirqs.CPU23.RCU
22251 ? 2% -83.7% 3630 ? 14% softirqs.CPU23.SCHED
12283 -48.7% 6299 ? 2% softirqs.CPU23.TIMER
3642421 +15.9% 4221316 ? 2% softirqs.CPU24.NET_RX
28828 ? 15% +40.5% 40491 ? 3% softirqs.CPU24.RCU
22624 ? 3% -84.3% 3550 ? 15% softirqs.CPU24.SCHED
12282 -48.5% 6326 softirqs.CPU24.TIMER
3613656 +17.7% 4254764 softirqs.CPU25.NET_RX
27143 ? 3% +48.9% 40430 softirqs.CPU25.RCU
22632 ? 3% -84.4% 3539 ? 22% softirqs.CPU25.SCHED
12295 -48.4% 6341 softirqs.CPU25.TIMER
3613067 +18.0% 4263149 softirqs.CPU26.NET_RX
26664 +52.7% 40706 softirqs.CPU26.RCU
22475 ? 3% -83.7% 3674 ? 19% softirqs.CPU26.SCHED
12279 -47.6% 6436 ? 4% softirqs.CPU26.TIMER
3613233 +18.7% 4288730 softirqs.CPU27.NET_RX
27290 ? 3% +47.8% 40332 softirqs.CPU27.RCU
22413 ? 2% -83.8% 3639 ? 18% softirqs.CPU27.SCHED
12416 ? 3% -49.1% 6315 softirqs.CPU27.TIMER
3600851 +19.0% 4285955 softirqs.CPU28.NET_RX
26779 ? 2% +51.5% 40580 softirqs.CPU28.RCU
22418 -84.5% 3483 ? 17% softirqs.CPU28.SCHED
12364 -48.5% 6364 softirqs.CPU28.TIMER
3567960 ? 4% +18.6% 4232238 ? 2% softirqs.CPU29.NET_RX
26704 ? 2% +49.8% 40008 ? 3% softirqs.CPU29.RCU
22290 ? 3% -83.2% 3735 ? 13% softirqs.CPU29.SCHED
12046 ? 6% -44.4% 6694 ? 9% softirqs.CPU29.TIMER
3680360 +16.0% 4270249 softirqs.CPU3.NET_RX
27123 +51.6% 41121 softirqs.CPU3.RCU
23119 ? 3% -83.9% 3723 ? 21% softirqs.CPU3.SCHED
12330 -47.9% 6427 softirqs.CPU3.TIMER
3643999 +17.4% 4278036 softirqs.CPU30.NET_RX
27134 +48.9% 40400 ? 2% softirqs.CPU30.RCU
22951 ? 4% -85.3% 3382 ? 15% softirqs.CPU30.SCHED
12366 ? 5% -48.6% 6352 softirqs.CPU30.TIMER
3606600 +19.4% 4304524 softirqs.CPU31.NET_RX
26870 +51.6% 40740 softirqs.CPU31.RCU
22750 ? 3% -85.2% 3360 ? 15% softirqs.CPU31.SCHED
12120 -47.3% 6386 softirqs.CPU31.TIMER
3584145 +17.4% 4208362 ? 2% softirqs.CPU32.NET_RX
26552 +50.8% 40049 ? 4% softirqs.CPU32.RCU
22296 ? 2% -86.5% 3003 ? 10% softirqs.CPU32.SCHED
12642 ? 2% -48.2% 6549 softirqs.CPU32.TIMER
3596209 +18.1% 4246254 softirqs.CPU33.NET_RX
26436 +53.4% 40549 softirqs.CPU33.RCU
22278 ? 2% -85.9% 3145 ? 15% softirqs.CPU33.SCHED
12667 ? 2% -48.3% 6546 ? 2% softirqs.CPU33.TIMER
3625597 +16.7% 4230469 softirqs.CPU34.NET_RX
26409 +51.3% 39967 ? 4% softirqs.CPU34.RCU
22113 ? 2% -85.7% 3153 ? 17% softirqs.CPU34.SCHED
12642 ? 2% -48.6% 6501 softirqs.CPU34.TIMER
3646457 +17.0% 4267332 softirqs.CPU35.NET_RX
26453 +53.7% 40670 ? 2% softirqs.CPU35.RCU
22411 ? 2% -86.9% 2935 ? 16% softirqs.CPU35.SCHED
13047 ? 6% -50.0% 6530 softirqs.CPU35.TIMER
3594222 +18.0% 4241189 softirqs.CPU36.NET_RX
26586 +52.9% 40650 ? 2% softirqs.CPU36.RCU
22185 ? 2% -86.1% 3083 ? 24% softirqs.CPU36.SCHED
12548 ? 2% -48.4% 6478 softirqs.CPU36.TIMER
3615955 +16.7% 4221514 ? 2% softirqs.CPU37.NET_RX
26595 +49.7% 39820 ? 3% softirqs.CPU37.RCU
22268 ? 2% -85.8% 3168 ? 16% softirqs.CPU37.SCHED
12545 ? 2% -47.9% 6533 softirqs.CPU37.TIMER
3650060 +17.1% 4272665 softirqs.CPU38.NET_RX
26817 +52.8% 40966 softirqs.CPU38.RCU
22608 ? 3% -86.6% 3040 ? 12% softirqs.CPU38.SCHED
12792 ? 3% -48.5% 6584 ? 4% softirqs.CPU38.TIMER
3676388 +13.9% 4185967 ? 2% softirqs.CPU39.NET_RX
26878 +48.0% 39767 ? 3% softirqs.CPU39.RCU
22221 -86.7% 2948 ? 20% softirqs.CPU39.SCHED
12490 -47.7% 6527 softirqs.CPU39.TIMER
3632448 +16.0% 4212263 ? 2% softirqs.CPU4.NET_RX
27052 +51.0% 40845 ? 4% softirqs.CPU4.RCU
22939 ? 2% -83.9% 3686 ? 15% softirqs.CPU4.SCHED
12352 -47.5% 6479 ? 5% softirqs.CPU4.TIMER
3687159 +15.2% 4249361 softirqs.CPU40.NET_RX
26954 +50.7% 40611 ? 2% softirqs.CPU40.RCU
22325 -87.6% 2775 ? 17% softirqs.CPU40.SCHED
12529 ? 2% -46.2% 6737 ? 7% softirqs.CPU40.TIMER
3656948 +15.6% 4226731 ? 2% softirqs.CPU41.NET_RX
27037 +48.9% 40257 ? 3% softirqs.CPU41.RCU
22420 ? 2% -86.9% 2941 ? 18% softirqs.CPU41.SCHED
12490 -48.0% 6499 softirqs.CPU41.TIMER
3686293 +15.0% 4239049 ? 2% softirqs.CPU42.NET_RX
26788 +50.0% 40177 ? 3% softirqs.CPU42.RCU
22308 ? 2% -87.4% 2816 ? 13% softirqs.CPU42.SCHED
12554 ? 2% -46.3% 6745 ? 7% softirqs.CPU42.TIMER
3690320 +15.3% 4255472 ? 2% softirqs.CPU43.NET_RX
26798 +51.2% 40512 ? 3% softirqs.CPU43.RCU
22349 -87.2% 2856 ? 15% softirqs.CPU43.SCHED
12593 ? 2% -48.2% 6528 softirqs.CPU43.TIMER
3597093 +18.2% 4250823 ? 2% softirqs.CPU44.NET_RX
26362 +53.2% 40394 ? 2% softirqs.CPU44.RCU
22245 -86.4% 3033 ? 19% softirqs.CPU44.SCHED
12619 ? 2% -48.4% 6506 softirqs.CPU44.TIMER
3670097 +16.2% 4263217 softirqs.CPU45.NET_RX
26561 +53.5% 40771 softirqs.CPU45.RCU
22620 ? 3% -86.5% 3059 ? 19% softirqs.CPU45.SCHED
13335 ? 12% -51.1% 6519 ? 2% softirqs.CPU45.TIMER
3631299 +14.0% 4140382 ? 2% softirqs.CPU46.NET_RX
26496 +48.4% 39308 ? 4% softirqs.CPU46.RCU
22106 ? 2% -84.9% 3337 ? 27% softirqs.CPU46.SCHED
12706 -49.0% 6476 softirqs.CPU46.TIMER
3629260 +17.5% 4265830 softirqs.CPU47.NET_RX
26725 +53.1% 40921 softirqs.CPU47.RCU
22471 -87.2% 2874 ? 18% softirqs.CPU47.SCHED
13048 ? 10% -50.3% 6485 softirqs.CPU47.TIMER
3647705 +15.9% 4227032 ? 2% softirqs.CPU48.NET_RX
26372 +50.5% 39677 ? 2% softirqs.CPU48.RCU
22175 ? 2% -85.9% 3124 ? 21% softirqs.CPU48.SCHED
12675 ? 2% -47.4% 6670 ? 6% softirqs.CPU48.TIMER
3624077 +17.9% 4271716 softirqs.CPU49.NET_RX
26469 +52.0% 40239 softirqs.CPU49.RCU
22306 -86.6% 2988 ? 16% softirqs.CPU49.SCHED
12699 ? 2% -48.8% 6501 softirqs.CPU49.TIMER
3669943 +16.6% 4277583 softirqs.CPU5.NET_RX
27410 +50.6% 41268 softirqs.CPU5.RCU
22796 ? 2% -83.9% 3668 ? 12% softirqs.CPU5.SCHED
12553 ? 2% -46.2% 6748 ? 12% softirqs.CPU5.TIMER
3600094 +16.6% 4198576 ? 2% softirqs.CPU50.NET_RX
26576 ? 2% +48.8% 39532 ? 3% softirqs.CPU50.RCU
22385 -86.4% 3034 ? 20% softirqs.CPU50.SCHED
12649 -48.7% 6488 softirqs.CPU50.TIMER
3665510 +16.1% 4256454 softirqs.CPU51.NET_RX
26651 +49.7% 39898 softirqs.CPU51.RCU
22159 ? 2% -86.3% 3039 ? 13% softirqs.CPU51.SCHED
12633 -48.7% 6486 softirqs.CPU51.TIMER
3661634 +16.3% 4258601 softirqs.CPU52.NET_RX
26462 +50.4% 39801 ? 2% softirqs.CPU52.RCU
22159 ? 2% -85.0% 3322 ? 24% softirqs.CPU52.SCHED
12607 ? 2% -46.6% 6727 ? 6% softirqs.CPU52.TIMER
3629833 +17.9% 4278596 softirqs.CPU53.NET_RX
26494 +51.8% 40211 softirqs.CPU53.RCU
22362 ? 2% -86.5% 3013 ? 12% softirqs.CPU53.SCHED
12648 -48.6% 6499 softirqs.CPU53.TIMER
3633047 +18.4% 4301719 softirqs.CPU54.NET_RX
26544 +51.7% 40264 softirqs.CPU54.RCU
22432 -86.5% 3023 ? 15% softirqs.CPU54.SCHED
12626 ? 2% -48.3% 6528 softirqs.CPU54.TIMER
3684799 +14.7% 4227457 softirqs.CPU55.NET_RX
26582 +49.2% 39670 softirqs.CPU55.RCU
22311 -86.7% 2964 ? 16% softirqs.CPU55.SCHED
12582 ? 2% -48.6% 6464 softirqs.CPU55.TIMER
3678360 +14.8% 4221340 softirqs.CPU56.NET_RX
26689 +48.3% 39586 ? 2% softirqs.CPU56.RCU
22502 ? 2% -86.7% 2982 ? 12% softirqs.CPU56.SCHED
12556 -48.3% 6489 softirqs.CPU56.TIMER
3696599 +14.8% 4244796 softirqs.CPU57.NET_RX
26612 +48.3% 39460 ? 3% softirqs.CPU57.RCU
22286 ? 2% -85.9% 3147 ? 16% softirqs.CPU57.SCHED
12594 ? 2% -48.3% 6510 softirqs.CPU57.TIMER
3650121 +18.0% 4307136 softirqs.CPU58.NET_RX
26702 +51.1% 40342 softirqs.CPU58.RCU
22431 ? 2% -86.2% 3084 ? 14% softirqs.CPU58.SCHED
13083 ? 11% -50.3% 6506 softirqs.CPU58.TIMER
3629890 +17.3% 4258589 softirqs.CPU59.NET_RX
26283 +51.9% 39933 ? 2% softirqs.CPU59.RCU
22057 ? 2% -85.6% 3174 ? 17% softirqs.CPU59.SCHED
12667 -48.9% 6473 softirqs.CPU59.TIMER
3681289 +16.4% 4285597 softirqs.CPU6.NET_RX
27708 +48.6% 41163 softirqs.CPU6.RCU
23158 ? 3% -83.9% 3737 ? 14% softirqs.CPU6.SCHED
12877 ? 5% -49.9% 6449 ? 2% softirqs.CPU6.TIMER
3618162 +17.5% 4251016 softirqs.CPU60.NET_RX
26384 +50.7% 39755 ? 2% softirqs.CPU60.RCU
22446 ? 3% -87.0% 2927 ? 16% softirqs.CPU60.SCHED
12641 ? 2% -48.6% 6497 softirqs.CPU60.TIMER
3590012 +17.9% 4234399 softirqs.CPU61.NET_RX
26267 +51.4% 39759 ? 2% softirqs.CPU61.RCU
22170 ? 2% -87.2% 2837 ? 21% softirqs.CPU61.SCHED
12595 ? 2% -48.0% 6555 ? 2% softirqs.CPU61.TIMER
3644555 +16.2% 4235606 softirqs.CPU62.NET_RX
26574 +48.4% 39449 ? 2% softirqs.CPU62.RCU
22368 ? 2% -84.8% 3407 ? 16% softirqs.CPU62.SCHED
12500 ? 2% -47.4% 6576 ? 3% softirqs.CPU62.TIMER
3631527 +12.7% 4093626 softirqs.CPU63.NET_RX
26773 ? 2% +46.1% 39121 ? 2% softirqs.CPU63.RCU
20929 ? 4% -85.3% 3072 ? 21% softirqs.CPU63.SCHED
12207 ? 2% -45.0% 6715 ? 6% softirqs.CPU63.TIMER
3617652 +16.3% 4207148 ? 2% softirqs.CPU64.NET_RX
26771 +52.9% 40932 softirqs.CPU64.RCU
22548 ? 2% -86.0% 3161 ? 19% softirqs.CPU64.SCHED
12411 ? 4% -48.9% 6344 ? 2% softirqs.CPU64.TIMER
3631482 +14.3% 4151252 softirqs.CPU65.NET_RX
26880 +51.8% 40815 ? 3% softirqs.CPU65.RCU
22564 -83.7% 3683 ? 26% softirqs.CPU65.SCHED
12223 -46.1% 6590 ? 11% softirqs.CPU65.TIMER
3655602 +12.3% 4105643 ? 2% softirqs.CPU66.NET_RX
27050 +48.9% 40289 ? 2% softirqs.CPU66.RCU
22563 ? 2% -84.6% 3471 ? 11% softirqs.CPU66.SCHED
12190 -48.7% 6258 softirqs.CPU66.TIMER
3681546 +14.5% 4215893 ? 2% softirqs.CPU67.NET_RX
27337 ? 2% +49.7% 40923 ? 3% softirqs.CPU67.RCU
22985 ? 2% -86.1% 3200 ? 13% softirqs.CPU67.SCHED
12297 -48.6% 6319 softirqs.CPU67.TIMER
3636012 +16.8% 4245963 softirqs.CPU68.NET_RX
27091 +54.0% 41729 softirqs.CPU68.RCU
22729 -84.6% 3501 ? 16% softirqs.CPU68.SCHED
12265 -48.7% 6287 softirqs.CPU68.TIMER
3669903 +14.8% 4211459 softirqs.CPU69.NET_RX
26760 +54.2% 41267 softirqs.CPU69.RCU
22711 ? 2% -85.0% 3407 ? 15% softirqs.CPU69.SCHED
12409 ? 2% -48.0% 6456 ? 7% softirqs.CPU69.TIMER
3641533 +17.0% 4258920 softirqs.CPU7.NET_RX
27174 +49.3% 40558 ? 2% softirqs.CPU7.RCU
22960 ? 3% -83.6% 3774 ? 13% softirqs.CPU7.SCHED
12194 -47.6% 6395 softirqs.CPU7.TIMER
3657044 +15.4% 4221328 softirqs.CPU70.NET_RX
27122 ? 2% +52.4% 41329 softirqs.CPU70.RCU
22663 ? 2% -84.1% 3603 ? 12% softirqs.CPU70.SCHED
12307 -46.3% 6612 ? 9% softirqs.CPU70.TIMER
3642201 +14.1% 4156875 ? 2% softirqs.CPU71.NET_RX
27465 ? 2% +46.9% 40348 ? 2% softirqs.CPU71.RCU
23158 ? 3% -84.2% 3661 ? 15% softirqs.CPU71.SCHED
12475 ? 4% -49.6% 6281 softirqs.CPU71.TIMER
3676096 +14.5% 4207672 softirqs.CPU72.NET_RX
27662 ? 2% +50.2% 41540 softirqs.CPU72.RCU
22927 ? 2% -84.7% 3509 ? 15% softirqs.CPU72.SCHED
12388 ? 3% -49.4% 6268 softirqs.CPU72.TIMER
3689025 +13.2% 4174568 ? 2% softirqs.CPU73.NET_RX
27008 +50.2% 40562 ? 3% softirqs.CPU73.RCU
22782 ? 3% -85.1% 3405 ? 21% softirqs.CPU73.SCHED
12506 ? 4% -49.2% 6355 ? 2% softirqs.CPU73.TIMER
3616064 +16.9% 4227496 softirqs.CPU74.NET_RX
26843 +53.6% 41238 softirqs.CPU74.RCU
22664 ? 2% -84.4% 3539 ? 16% softirqs.CPU74.SCHED
12315 -45.7% 6690 ? 13% softirqs.CPU74.TIMER
3671200 +13.5% 4168152 ? 2% softirqs.CPU75.NET_RX
27178 +49.2% 40536 ? 3% softirqs.CPU75.RCU
22627 ? 2% -84.8% 3430 ? 19% softirqs.CPU75.SCHED
12426 -49.1% 6323 softirqs.CPU75.TIMER
3681366 +16.0% 4270047 softirqs.CPU76.NET_RX
26915 ? 2% +54.5% 41570 softirqs.CPU76.RCU
22458 ? 3% -84.5% 3490 ? 10% softirqs.CPU76.SCHED
12388 -48.8% 6337 softirqs.CPU76.TIMER
3596148 +15.7% 4159328 ? 2% softirqs.CPU77.NET_RX
26726 +51.2% 40397 ? 3% softirqs.CPU77.RCU
22588 ? 3% -84.2% 3569 ? 12% softirqs.CPU77.SCHED
12341 ? 2% -48.2% 6393 ? 2% softirqs.CPU77.TIMER
3635295 +16.3% 4227381 softirqs.CPU78.NET_RX
26682 +55.7% 41535 softirqs.CPU78.RCU
22375 ? 2% -83.6% 3672 ? 11% softirqs.CPU78.SCHED
12306 -48.3% 6360 softirqs.CPU78.TIMER
3645083 +15.8% 4219789 softirqs.CPU79.NET_RX
26970 +53.5% 41411 softirqs.CPU79.RCU
22406 ? 2% -84.5% 3478 ? 17% softirqs.CPU79.SCHED
12373 -48.9% 6319 softirqs.CPU79.TIMER
3720755 +14.3% 4254642 softirqs.CPU8.NET_RX
27688 ? 2% +46.6% 40594 softirqs.CPU8.RCU
22942 ? 2% -83.9% 3697 ? 21% softirqs.CPU8.SCHED
12954 ? 11% -51.0% 6347 softirqs.CPU8.TIMER
3631957 +17.1% 4252782 softirqs.CPU80.NET_RX
27044 +50.5% 40690 softirqs.CPU80.RCU
22725 -84.0% 3637 ? 11% softirqs.CPU80.SCHED
12120 -48.1% 6294 softirqs.CPU80.TIMER
3660719 +16.0% 4247487 softirqs.CPU81.NET_RX
26763 +52.6% 40837 softirqs.CPU81.RCU
22568 ? 2% -84.3% 3554 ? 23% softirqs.CPU81.SCHED
12112 -46.9% 6435 ? 4% softirqs.CPU81.TIMER
3629949 +15.9% 4208198 softirqs.CPU82.NET_RX
27234 +49.5% 40712 softirqs.CPU82.RCU
22685 ? 3% -82.6% 3946 ? 19% softirqs.CPU82.SCHED
12229 ? 2% -48.2% 6334 ? 4% softirqs.CPU82.TIMER
3632157 +15.7% 4203914 softirqs.CPU83.NET_RX
27060 +51.5% 40995 ? 2% softirqs.CPU83.RCU
22736 ? 2% -81.7% 4162 ? 23% softirqs.CPU83.SCHED
12735 ? 12% -50.5% 6304 softirqs.CPU83.TIMER
3680232 +14.3% 4208276 softirqs.CPU84.NET_RX
26928 +52.4% 41052 softirqs.CPU84.RCU
22617 ? 2% -83.8% 3663 ? 19% softirqs.CPU84.SCHED
12124 -48.0% 6310 ? 2% softirqs.CPU84.TIMER
3655138 +15.6% 4226691 softirqs.CPU85.NET_RX
27031 +52.8% 41305 softirqs.CPU85.RCU
22699 -84.4% 3547 ? 19% softirqs.CPU85.SCHED
12033 -48.0% 6262 softirqs.CPU85.TIMER
3657642 +15.0% 4205027 ? 2% softirqs.CPU86.NET_RX
26494 +52.4% 40376 ? 3% softirqs.CPU86.RCU
22418 ? 2% -84.2% 3547 ? 21% softirqs.CPU86.SCHED
12284 -46.5% 6572 ? 9% softirqs.CPU86.TIMER
3656695 +16.6% 4263027 softirqs.CPU87.NET_RX
26728 +51.5% 40505 softirqs.CPU87.RCU
22431 ? 2% -83.5% 3697 ? 13% softirqs.CPU87.SCHED
12247 -48.7% 6286 softirqs.CPU87.TIMER
3622507 +16.2% 4207738 ? 2% softirqs.CPU88.NET_RX
26453 +50.8% 39898 ? 3% softirqs.CPU88.RCU
22402 ? 2% -84.7% 3422 ? 20% softirqs.CPU88.SCHED
12186 -48.1% 6328 softirqs.CPU88.TIMER
3616597 +17.2% 4239446 softirqs.CPU89.NET_RX
27224 ? 3% +50.0% 40825 softirqs.CPU89.RCU
22622 ? 2% -84.4% 3535 ? 16% softirqs.CPU89.SCHED
12355 -49.3% 6267 softirqs.CPU89.TIMER
3696490 +15.2% 4258106 ? 2% softirqs.CPU9.NET_RX
27811 +47.1% 40919 ? 2% softirqs.CPU9.RCU
23244 ? 3% -83.7% 3781 ? 13% softirqs.CPU9.SCHED
13128 ? 11% -51.4% 6380 softirqs.CPU9.TIMER
3637524 +16.2% 4225781 softirqs.CPU90.NET_RX
26646 +51.2% 40286 ? 2% softirqs.CPU90.RCU
22588 ? 3% -84.3% 3547 ? 13% softirqs.CPU90.SCHED
12322 -48.9% 6290 softirqs.CPU90.TIMER
3636814 +15.9% 4213776 softirqs.CPU91.NET_RX
26548 +53.8% 40842 softirqs.CPU91.RCU
22554 ? 3% -83.4% 3738 ? 19% softirqs.CPU91.SCHED
12242 -49.1% 6234 softirqs.CPU91.TIMER
3597522 +17.4% 4222507 softirqs.CPU92.NET_RX
26505 +52.9% 40529 softirqs.CPU92.RCU
22390 ? 3% -84.8% 3406 ? 18% softirqs.CPU92.SCHED
12221 -48.7% 6267 softirqs.CPU92.TIMER
3651412 +16.2% 4242870 ? 2% softirqs.CPU93.NET_RX
26954 +49.0% 40157 softirqs.CPU93.RCU
22826 ? 2% -84.8% 3465 ? 24% softirqs.CPU93.SCHED
12231 ? 4% -46.7% 6524 ? 8% softirqs.CPU93.TIMER
3630129 +15.3% 4183844 ? 2% softirqs.CPU94.NET_RX
26916 +48.0% 39838 ? 3% softirqs.CPU94.RCU
22702 ? 2% -85.6% 3270 ? 17% softirqs.CPU94.SCHED
12033 -47.9% 6275 softirqs.CPU94.TIMER
3631483 +15.3% 4185392 ? 2% softirqs.CPU95.NET_RX
26805 +49.2% 40002 ? 3% softirqs.CPU95.RCU
22824 -85.6% 3292 ? 17% softirqs.CPU95.SCHED
12047 -48.0% 6260 softirqs.CPU95.TIMER
3568253 +18.5% 4228069 softirqs.CPU96.NET_RX
26043 +55.1% 40402 ? 2% softirqs.CPU96.RCU
22147 ? 2% -87.2% 2838 ? 14% softirqs.CPU96.SCHED
12474 ? 2% -48.5% 6422 softirqs.CPU96.TIMER
3608937 +17.2% 4228799 softirqs.CPU97.NET_RX
26134 +54.5% 40368 softirqs.CPU97.RCU
22231 -86.5% 3009 ? 12% softirqs.CPU97.SCHED
12616 ? 2% -49.3% 6399 softirqs.CPU97.TIMER
3632811 +16.6% 4235794 softirqs.CPU98.NET_RX
26230 +53.6% 40295 softirqs.CPU98.RCU
22127 ? 2% -86.3% 3031 ? 18% softirqs.CPU98.SCHED
12578 ? 2% -47.2% 6641 ? 7% softirqs.CPU98.TIMER
3670765 +14.6% 4205482 softirqs.CPU99.NET_RX
26376 +51.7% 40025 ? 3% softirqs.CPU99.RCU
22274 ? 2% -86.9% 2911 ? 17% softirqs.CPU99.SCHED
12690 ? 2% -49.2% 6449 softirqs.CPU99.TIMER
4.657e+08 +16.2% 5.412e+08 softirqs.NET_RX
3431554 +50.6% 5169164 softirqs.RCU
2880521 ? 2% -85.2% 427430 ? 15% softirqs.SCHED
445625 ? 3% -99.2% 3534 ? 14% softirqs.TASKLET
1594175 -48.2% 825859 softirqs.TIMER
9.52 ? 4% -9.3 0.22 ?141% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
9.44 ? 4% -9.2 0.21 ?141% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
9.44 ? 4% -9.2 0.21 ?141% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
9.40 ? 4% -9.2 0.21 ?141% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
7.65 -7.6 0.00 perf-profile.calltrace.cycles-pp.release_sock.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
7.23 -7.2 0.00 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom
6.06 ? 5% -6.0 0.09 ?223% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
6.03 ? 5% -5.9 0.09 ?223% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
5.91 -5.9 0.00 perf-profile.calltrace.cycles-pp.dccp_v4_do_rcv.__release_sock.release_sock.dccp_recvmsg.sock_common_recvmsg
5.64 -5.6 0.00 perf-profile.calltrace.cycles-pp.dccp_rcv_established.dccp_v4_do_rcv.__release_sock.release_sock.dccp_recvmsg
5.59 ? 5% -5.6 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
5.21 -5.2 0.00 perf-profile.calltrace.cycles-pp.dccp_deliver_input_to_ccids.dccp_rcv_established.dccp_v4_do_rcv.__release_sock.release_sock
5.11 -5.1 0.00 perf-profile.calltrace.cycles-pp.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids.dccp_rcv_established.dccp_v4_do_rcv.__release_sock
5.74 ? 3% -4.7 1.04 ? 2% perf-profile.calltrace.cycles-pp.security_socket_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
22.38 -4.6 17.82 perf-profile.calltrace.cycles-pp.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
5.12 ? 3% -4.4 0.68 ? 2% perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
22.46 -4.4 18.04 perf-profile.calltrace.cycles-pp.sock_common_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
23.86 -3.9 19.98 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
23.92 -3.7 20.17 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
24.25 -2.7 21.60 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
3.03 ? 2% -2.4 0.58 perf-profile.calltrace.cycles-pp.lock_sock_nested.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
24.39 -2.3 22.10 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv
1.85 -1.2 0.70 perf-profile.calltrace.cycles-pp.__kfree_skb.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
1.24 -0.9 0.34 ? 70% perf-profile.calltrace.cycles-pp.accept
1.42 -0.9 0.52 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
1.45 -0.9 0.58 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.dccp_recvmsg
1.66 -0.8 0.88 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.dccp_recvmsg.sock_common_recvmsg
1.47 ? 2% -0.6 0.84 perf-profile.calltrace.cycles-pp.dccp_rcv_state_process.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish
1.22 ? 4% -0.6 0.66 perf-profile.calltrace.cycles-pp.sockfd_lookup_light.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.31 -0.5 0.79 perf-profile.calltrace.cycles-pp.__dccp_rcv_established.dccp_v4_do_rcv.__release_sock.release_sock.dccp_sendmsg
1.21 -0.5 0.71 perf-profile.calltrace.cycles-pp.__skb_checksum.skb_checksum.dccp_v4_send_check.dccp_transmit_skb.dccp_xmit_packet
1.23 -0.5 0.76 perf-profile.calltrace.cycles-pp.skb_checksum.dccp_v4_send_check.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit
2.31 -0.5 1.86 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom
25.11 -0.4 24.67 perf-profile.calltrace.cycles-pp.recv
1.31 -0.4 0.88 perf-profile.calltrace.cycles-pp.dccp_v4_send_check.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg
2.35 -0.4 1.92 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
2.47 -0.4 2.08 perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
0.91 -0.3 0.64 ? 2% perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
1.25 -0.2 1.02 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.dccp_transmit_skb.dccp_rcv_state_process
1.26 -0.2 1.04 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.dccp_transmit_skb.dccp_rcv_state_process.dccp_v4_do_rcv
1.27 -0.2 1.06 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.dccp_transmit_skb.dccp_rcv_state_process.dccp_v4_do_rcv.__release_sock
1.08 -0.2 0.88 perf-profile.calltrace.cycles-pp.__inet_lookup_established.dccp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
1.31 -0.2 1.12 perf-profile.calltrace.cycles-pp.dccp_transmit_skb.dccp_rcv_state_process.dccp_v4_do_rcv.__release_sock.release_sock
2.02 -0.1 1.90 perf-profile.calltrace.cycles-pp.dccp_rcv_state_process.dccp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect
0.93 -0.1 0.82 perf-profile.calltrace.cycles-pp.dccp_v4_conn_request.dccp_rcv_state_process.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu
0.65 -0.1 0.56 perf-profile.calltrace.cycles-pp.dccp_create_openreq_child.dccp_v4_request_recv_sock.dccp_check_req.dccp_v4_rcv.ip_protocol_deliver_rcu
2.06 -0.1 1.98 perf-profile.calltrace.cycles-pp.dccp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect
2.07 -0.1 2.00 perf-profile.calltrace.cycles-pp.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
2.06 -0.1 2.00 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect
0.72 -0.0 0.67 perf-profile.calltrace.cycles-pp.dccp_v4_request_recv_sock.dccp_check_req.dccp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
0.58 +0.0 0.62 perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.dccp_sendmsg
1.20 +0.1 1.25 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.dccp_transmit_skb.dccp_connect.dccp_v4_connect.__inet_stream_connect
1.61 +0.1 1.66 perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.dccp_sendmsg.sock_sendmsg
1.24 +0.1 1.31 perf-profile.calltrace.cycles-pp.dccp_transmit_skb.dccp_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect
1.29 +0.1 1.40 perf-profile.calltrace.cycles-pp.dccp_connect.dccp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
1.68 +0.1 1.79 perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.dccp_sendmsg.sock_sendmsg.__sys_sendto
0.87 +0.1 1.01 perf-profile.calltrace.cycles-pp.dccp_finish_passive_close.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
1.41 ? 2% +0.1 1.55 ? 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
0.68 +0.1 0.82 perf-profile.calltrace.cycles-pp.dccp_transmit_skb.dccp_finish_passive_close.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom
0.66 +0.2 0.82 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.dccp_transmit_skb.dccp_finish_passive_close.dccp_recvmsg
0.65 +0.2 0.81 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.dccp_transmit_skb.dccp_finish_passive_close
0.66 +0.2 0.82 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.dccp_transmit_skb.dccp_finish_passive_close.dccp_recvmsg.sock_common_recvmsg
0.65 +0.2 0.83 perf-profile.calltrace.cycles-pp.dccp_close.inet_release.__sock_release.sock_close.__fput
0.66 +0.2 0.84 perf-profile.calltrace.cycles-pp.inet_release.__sock_release.sock_close.__fput.task_work_run
0.68 +0.2 0.86 perf-profile.calltrace.cycles-pp.__sock_release.sock_close.__fput.task_work_run.exit_to_user_mode_prepare
0.68 +0.2 0.87 perf-profile.calltrace.cycles-pp.sock_close.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
3.60 +0.2 3.83 perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
3.65 +0.2 3.88 perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect
3.65 +0.2 3.87 perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect
3.66 +0.2 3.88 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect
0.56 +0.2 0.79 perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.dccp_recvmsg
3.61 +0.2 3.84 perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.52 +0.2 0.75 perf-profile.calltrace.cycles-pp.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe.__socket
0.51 +0.2 0.75 perf-profile.calltrace.cycles-pp.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe.__socket
3.66 +0.2 3.89 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.connect
0.52 +0.2 0.76 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__socket
0.52 +0.2 0.76 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__socket
0.53 +0.2 0.77 perf-profile.calltrace.cycles-pp.__socket
3.67 +0.2 3.91 perf-profile.calltrace.cycles-pp.connect
1.52 +0.3 1.79 perf-profile.calltrace.cycles-pp.dccp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
0.60 +0.3 0.87 perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.dccp_recvmsg.sock_common_recvmsg
3.34 +0.3 3.67 perf-profile.calltrace.cycles-pp.dccp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
1.01 +0.4 1.36 perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
1.04 +0.4 1.40 perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.05 +0.4 1.43 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
1.06 +0.4 1.43 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
1.11 +0.4 1.49 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
1.11 +0.4 1.50 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close
1.14 +0.4 1.56 perf-profile.calltrace.cycles-pp.__close
0.54 ? 2% +0.4 0.98 perf-profile.calltrace.cycles-pp._copy_from_iter.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
0.56 +0.4 1.00 perf-profile.calltrace.cycles-pp.dccp_send_ack.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids.dccp_rcv_established.dccp_v4_do_rcv
0.68 +0.5 1.16 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.dccp_transmit_skb.dccp_connect
0.68 +0.5 1.16 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.dccp_transmit_skb.dccp_connect.dccp_v4_connect
0.51 +0.5 1.00 perf-profile.calltrace.cycles-pp.skb_checksum.dccp_invalid_packet.dccp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
0.00 +0.5 0.50 perf-profile.calltrace.cycles-pp.ip_rcv_core.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll
0.00 +0.5 0.50 perf-profile.calltrace.cycles-pp.dccp_ackvec_clear_state.dccp_handle_ackvec_processing.dccp_rcv_established.dccp_v4_do_rcv.__release_sock
0.60 +0.5 1.11 perf-profile.calltrace.cycles-pp.dccp_insert_options.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg
0.00 +0.5 0.52 perf-profile.calltrace.cycles-pp.save_fpregs_to_fpstate.recv
0.00 +0.5 0.52 perf-profile.calltrace.cycles-pp.dccp_feat_entry_new.dccp_feat_push_change.dccp_feat_signal_nn_change.ccid2_hc_tx_packet_recv.dccp_rcv_established
0.00 +0.5 0.53 ? 2% perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
0.00 +0.5 0.53 ? 2% perf-profile.calltrace.cycles-pp.ret_from_fork
0.00 +0.5 0.53 ? 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
0.00 +0.5 0.53 perf-profile.calltrace.cycles-pp.__sock_create.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.5 0.54 perf-profile.calltrace.cycles-pp.dccp_feat_push_change.dccp_feat_signal_nn_change.ccid2_hc_tx_packet_recv.dccp_rcv_established.dccp_v4_do_rcv
0.00 +0.5 0.54 perf-profile.calltrace.cycles-pp.mod_timer.sk_reset_timer.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg
0.00 +0.5 0.55 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter.dccp_sendmsg.sock_sendmsg
0.00 +0.6 0.55 perf-profile.calltrace.cycles-pp.dccp_handle_ackvec_processing.dccp_rcv_established.dccp_v4_do_rcv.__release_sock.release_sock
0.00 +0.6 0.56 ? 4% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
0.00 +0.6 0.56 perf-profile.calltrace.cycles-pp.__cgroup_bpf_run_filter_skb.sk_filter_trim_cap.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish
0.00 +0.6 0.56 perf-profile.calltrace.cycles-pp.irqtime_account_irq.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
0.00 +0.6 0.56 perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.schedule_timeout
0.00 +0.6 0.57 perf-profile.calltrace.cycles-pp.__switch_to_asm.recv
0.61 +0.6 1.20 perf-profile.calltrace.cycles-pp.dccp_invalid_packet.dccp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
0.00 +0.6 0.58 perf-profile.calltrace.cycles-pp.__switch_to_asm.__send
0.00 +0.6 0.59 perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.__wake_up_common
0.00 +0.6 0.61 perf-profile.calltrace.cycles-pp.csum_partial.__skb_checksum.skb_checksum.dccp_v4_send_check.dccp_transmit_skb
0.00 +0.6 0.61 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.dccp_sendmsg.sock_sendmsg.__sys_sendto
0.00 +0.6 0.62 perf-profile.calltrace.cycles-pp.dccp_feat_signal_nn_change.ccid2_hc_tx_packet_recv.dccp_rcv_established.dccp_v4_do_rcv.__release_sock
0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp.load_new_mm_cr3.switch_mm_irqs_off.__schedule.preempt_schedule_common.__cond_resched
0.00 +0.7 0.65 perf-profile.calltrace.cycles-pp.enqueue_to_backlog.netif_rx_internal.netif_rx.loopback_xmit.dev_hard_start_xmit
0.00 +0.7 0.68 perf-profile.calltrace.cycles-pp.sk_reset_timer.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg.sock_sendmsg
0.08 ?223% +0.7 0.78 ? 3% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
0.00 +0.7 0.70 perf-profile.calltrace.cycles-pp.dccp_parse_options.dccp_rcv_established.dccp_v4_do_rcv.__release_sock.release_sock
0.00 +0.7 0.71 perf-profile.calltrace.cycles-pp.csum_partial.__skb_checksum.skb_checksum.dccp_invalid_packet.dccp_v4_rcv
0.00 +0.7 0.73 perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.__wake_up_common.__wake_up_common_lock
0.00 +0.8 0.78 perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.__wake_up_common.__wake_up_common_lock.dccp_write_space
0.00 +0.8 0.80 perf-profile.calltrace.cycles-pp.__netif_receive_skb_core.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
0.00 +0.8 0.82 perf-profile.calltrace.cycles-pp.dccp_insert_options.dccp_transmit_skb.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids.dccp_rcv_established
0.08 ?223% +0.8 0.93 perf-profile.calltrace.cycles-pp.__alloc_skb.dccp_send_ack.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids.dccp_rcv_established
0.64 ? 2% +0.9 1.50 ? 2% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common
0.00 +0.9 0.86 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
0.00 +0.9 0.89 perf-profile.calltrace.cycles-pp.__skb_checksum.skb_checksum.dccp_invalid_packet.dccp_v4_rcv.ip_protocol_deliver_rcu
0.00 +0.9 0.93 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
0.00 +1.0 0.96 perf-profile.calltrace.cycles-pp.dccp_parse_options.dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu
0.00 +1.0 0.97 perf-profile.calltrace.cycles-pp.load_new_mm_cr3.switch_mm_irqs_off.__schedule.schedule.schedule_timeout
0.00 +1.1 1.10 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
0.72 +1.1 1.84 perf-profile.calltrace.cycles-pp.ccid2_hc_tx_packet_recv.dccp_rcv_established.dccp_v4_do_rcv.__release_sock.release_sock
0.72 ? 3% +1.2 1.92 perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.__dccp_rcv_established
0.52 +1.2 1.73 perf-profile.calltrace.cycles-pp.netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
10.36 +1.2 11.57 perf-profile.calltrace.cycles-pp.dccp_transmit_skb.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids.dccp_rcv_established.dccp_v4_do_rcv
0.00 +1.2 1.24 perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
0.00 +1.2 1.24 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.preempt_schedule_common.__cond_resched.__release_sock
3.43 +1.2 4.68 perf-profile.calltrace.cycles-pp.dccp_v4_do_rcv.__release_sock.release_sock.dccp_sendmsg.sock_sendmsg
0.00 +1.3 1.27 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.wait_woken
0.00 +1.3 1.31 perf-profile.calltrace.cycles-pp.sk_filter_trim_cap.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
0.85 ? 3% +1.3 2.19 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__dccp_rcv_established.dccp_v4_do_rcv
0.00 +1.4 1.36 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.preempt_schedule_common.__cond_resched.__release_sock
0.00 +1.5 1.52 perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
7.01 +1.6 8.56 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.dccp_transmit_skb.ccid2_hc_rx_packet_recv
0.94 ? 2% +1.6 2.50 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb
1.42 +1.6 2.98 perf-profile.calltrace.cycles-pp.__dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish
0.08 ?223% +1.6 1.66 perf-profile.calltrace.cycles-pp.netif_rx_internal.netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit
0.00 +1.6 1.58 perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.06 ? 2% +1.6 2.69 perf-profile.calltrace.cycles-pp.sock_def_readable.__dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu
0.00 +1.7 1.66 perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
8.18 +1.7 9.84 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.dccp_transmit_skb.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids.dccp_rcv_established
1.59 +1.7 3.33 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woken
7.49 +1.7 9.24 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.dccp_transmit_skb.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids
2.07 +1.8 3.82 perf-profile.calltrace.cycles-pp.dccp_rcv_established.dccp_v4_do_rcv.__release_sock.release_sock.dccp_sendmsg
0.00 +1.9 1.89 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.schedule_timeout.wait_woken
0.92 ? 3% +2.0 2.90 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.dccp_write_space
0.70 ? 3% +2.0 2.70 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
0.00 +2.6 2.55 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
0.90 ? 3% +2.8 3.66 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
2.37 +3.1 5.52 perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.dccp_write_space.sock_wfree
2.60 +3.2 5.76 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.dccp_write_space.sock_wfree.loopback_xmit
2.96 +3.2 6.18 perf-profile.calltrace.cycles-pp.dccp_write_space.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit
2.69 +3.3 6.02 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.dccp_write_space.sock_wfree.loopback_xmit.dev_hard_start_xmit
3.06 +3.3 6.39 perf-profile.calltrace.cycles-pp.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
0.00 +4.0 4.04 perf-profile.calltrace.cycles-pp.__schedule.preempt_schedule_common.__cond_resched.__release_sock.release_sock
0.00 +4.2 4.21 perf-profile.calltrace.cycles-pp.preempt_schedule_common.__cond_resched.__release_sock.release_sock.dccp_sendmsg
0.00 +4.3 4.32 perf-profile.calltrace.cycles-pp.__cond_resched.__release_sock.release_sock.dccp_sendmsg.sock_sendmsg
5.31 +4.7 10.02 perf-profile.calltrace.cycles-pp.release_sock.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
4.48 ? 3% +5.0 9.44 perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
4.56 +5.2 9.78 perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit
6.17 +5.4 11.59 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.dccp_transmit_skb
3.78 +5.7 9.47 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.dccp_sendmsg.sock_sendmsg.__sys_sendto
3.10 ? 2% +6.1 9.21 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
3.14 ? 2% +6.4 9.56 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg
6.26 +6.5 12.78 perf-profile.calltrace.cycles-pp.ccid2_hc_rx_packet_recv.dccp_deliver_input_to_ccids.dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb
6.47 +6.6 13.07 perf-profile.calltrace.cycles-pp.dccp_deliver_input_to_ccids.dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu
3.18 ? 2% +6.7 9.90 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.dccp_recvmsg.sock_common_recvmsg
8.64 +6.8 15.49 perf-profile.calltrace.cycles-pp.dccp_rcv_established.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish
5.50 ? 2% +6.9 12.43 perf-profile.calltrace.cycles-pp.sk_wait_data.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
3.39 ? 2% +7.1 10.46 perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.dccp_recvmsg.sock_common_recvmsg.__sys_recvfrom
11.70 +7.9 19.64 perf-profile.calltrace.cycles-pp.dccp_v4_do_rcv.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
14.23 +8.3 22.49 perf-profile.calltrace.cycles-pp.__sk_receive_skb.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
18.53 +8.4 26.96 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
18.63 +8.6 27.21 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
18.74 +8.8 27.50 perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll
57.00 +8.9 65.93 perf-profile.calltrace.cycles-pp.__send
19.81 +8.9 28.75 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
20.39 +9.6 29.96 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
21.00 +10.0 30.98 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
17.92 +10.0 27.97 perf-profile.calltrace.cycles-pp.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg.sock_sendmsg
16.84 +10.1 26.90 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg
21.04 +10.1 31.13 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
49.64 +10.2 59.85 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
21.34 +10.3 31.66 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
49.24 +10.4 59.59 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
16.09 +10.5 26.54 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.dccp_transmit_skb.dccp_xmit_packet.dccp_write_xmit
15.64 +10.5 26.10 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.dccp_transmit_skb.dccp_xmit_packet
22.42 +10.5 32.91 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
22.61 +10.7 33.27 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
22.71 +10.8 33.46 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.dccp_transmit_skb
51.97 +11.4 63.42 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send
46.70 +11.5 58.19 perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
51.14 +11.8 62.91 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
24.86 +16.3 41.18 perf-profile.calltrace.cycles-pp.dccp_xmit_packet.dccp_write_xmit.dccp_sendmsg.sock_sendmsg.__sys_sendto
25.08 +16.4 41.44 perf-profile.calltrace.cycles-pp.dccp_write_xmit.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
39.95 +16.7 56.69 perf-profile.calltrace.cycles-pp.dccp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
9.52 ? 4% -9.0 0.50 ? 19% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
9.52 ? 4% -9.0 0.50 ? 19% perf-profile.children.cycles-pp.cpu_startup_entry
9.50 ? 4% -9.0 0.50 ? 19% perf-profile.children.cycles-pp.do_idle
9.44 ? 4% -8.9 0.50 ? 19% perf-profile.children.cycles-pp.start_secondary
6.11 ? 5% -5.7 0.41 ? 20% perf-profile.children.cycles-pp.cpuidle_enter
6.09 ? 5% -5.7 0.41 ? 20% perf-profile.children.cycles-pp.cpuidle_enter_state
5.68 ? 5% -5.3 0.38 ? 20% perf-profile.children.cycles-pp.intel_idle
6.57 -5.1 1.45 perf-profile.children.cycles-pp._raw_spin_lock_bh
6.24 ? 2% -4.8 1.41 ? 2% perf-profile.children.cycles-pp.aa_sk_perm
5.82 ? 3% -4.8 1.06 ? 2% perf-profile.children.cycles-pp.security_socket_sendmsg
6.13 -4.6 1.50 perf-profile.children.cycles-pp.lock_sock_nested
22.40 -4.5 17.87 perf-profile.children.cycles-pp.dccp_recvmsg
22.47 -4.4 18.06 perf-profile.children.cycles-pp.sock_common_recvmsg
23.88 -3.9 20.01 perf-profile.children.cycles-pp.__sys_recvfrom
23.93 -3.7 20.22 perf-profile.children.cycles-pp.__x64_sys_recvfrom
15.88 -3.2 12.65 perf-profile.children.cycles-pp.release_sock
13.72 -2.0 11.71 perf-profile.children.cycles-pp.__release_sock
1.90 -1.8 0.06 ? 7% perf-profile.children.cycles-pp.__slab_free
2.87 -1.6 1.26 perf-profile.children.cycles-pp.__kfree_skb
2.24 ? 3% -1.5 0.74 perf-profile.children.cycles-pp.__entry_text_start
2.21 ? 3% -1.3 0.90 perf-profile.children.cycles-pp.syscall_return_via_sysret
1.34 -1.1 0.21 ? 2% perf-profile.children.cycles-pp.skb_queue_tail
1.47 -1.1 0.35 perf-profile.children.cycles-pp.skb_set_owner_w
1.46 -0.9 0.60 perf-profile.children.cycles-pp.copyout
1.52 -0.8 0.70 perf-profile.children.cycles-pp.skb_release_data
1.17 -0.8 0.36 ? 2% perf-profile.children.cycles-pp.dst_release
1.35 -0.8 0.54 perf-profile.children.cycles-pp.skb_release_head_state
1.38 -0.8 0.60 perf-profile.children.cycles-pp.skb_release_all
0.89 -0.8 0.12 ? 4% perf-profile.children.cycles-pp.inet_accept
1.67 -0.8 0.91 perf-profile.children.cycles-pp._copy_to_iter
1.26 -0.8 0.51 perf-profile.children.cycles-pp.accept
0.83 -0.7 0.09 ? 5% perf-profile.children.cycles-pp.inet_csk_accept
1.14 -0.7 0.47 ? 2% perf-profile.children.cycles-pp.__sys_accept4_file
1.13 -0.7 0.45 perf-profile.children.cycles-pp.do_accept
1.15 -0.7 0.48 perf-profile.children.cycles-pp.__x64_sys_accept
1.15 -0.7 0.48 perf-profile.children.cycles-pp.__sys_accept4
0.79 ? 2% -0.7 0.13 ? 7% perf-profile.children.cycles-pp.ipv4_dst_check
1.50 ? 2% -0.7 0.84 ? 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
1.82 -0.6 1.18 perf-profile.children.cycles-pp.dccp_v4_send_check
0.90 -0.6 0.27 perf-profile.children.cycles-pp.__sk_dst_check
1.81 -0.6 1.19 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
1.42 ? 2% -0.6 0.79 ? 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
1.54 -0.6 0.96 perf-profile.children.cycles-pp.dccp_ackvec_clear_state
2.35 -0.6 1.76 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.61 -0.6 1.04 perf-profile.children.cycles-pp.dccp_handle_ackvec_processing
1.01 ? 4% -0.5 0.49 perf-profile.children.cycles-pp.update_cfs_group
1.52 -0.5 1.01 perf-profile.children.cycles-pp.__inet_lookup_established
0.81 -0.5 0.30 perf-profile.children.cycles-pp.__ksize
0.90 -0.5 0.41 perf-profile.children.cycles-pp.dccp_check_seqno
4.05 -0.5 3.59 perf-profile.children.cycles-pp.dccp_rcv_state_process
2.32 -0.4 1.88 perf-profile.children.cycles-pp.__skb_datagram_iter
2.35 -0.4 1.93 perf-profile.children.cycles-pp.skb_copy_datagram_iter
1.54 ? 4% -0.4 1.11 perf-profile.children.cycles-pp.sockfd_lookup_light
0.76 ? 2% -0.4 0.34 perf-profile.children.cycles-pp.irq_exit_rcu
1.17 ? 4% -0.4 0.77 perf-profile.children.cycles-pp.__fget_light
0.50 ? 5% -0.4 0.11 ? 3% perf-profile.children.cycles-pp.select_idle_cpu
2.48 -0.4 2.10 perf-profile.children.cycles-pp.sock_alloc_send_pskb
0.45 ? 4% -0.4 0.07 ? 5% perf-profile.children.cycles-pp.inet_sendmsg
0.65 ? 2% -0.4 0.30 perf-profile.children.cycles-pp.dccp_v4_send_response
2.19 -0.3 1.84 perf-profile.children.cycles-pp.__skb_checksum
0.58 -0.3 0.25 perf-profile.children.cycles-pp.ip_rcv_finish_core
0.48 ? 4% -0.3 0.16 ? 2% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
25.28 -0.3 24.96 perf-profile.children.cycles-pp.recv
0.56 -0.3 0.25 perf-profile.children.cycles-pp.dccp_enqueue_skb
0.60 ? 4% -0.3 0.29 ? 3% perf-profile.children.cycles-pp.select_idle_sibling
0.84 ? 2% -0.3 0.55 perf-profile.children.cycles-pp.ccid2_hc_tx_alloc_seq
0.85 -0.3 0.56 perf-profile.children.cycles-pp.ccid2_hc_tx_init
1.24 -0.3 0.96 perf-profile.children.cycles-pp.__might_resched
0.89 ? 2% -0.3 0.62 perf-profile.children.cycles-pp.dccp_hdlr_ccid
0.89 -0.3 0.61 perf-profile.children.cycles-pp.ccid_new
0.36 ? 2% -0.3 0.09 perf-profile.children.cycles-pp.dccp_child_process
0.34 ? 10% -0.3 0.07 ? 5% perf-profile.children.cycles-pp.task_tick_fair
0.83 ? 2% -0.3 0.57 perf-profile.children.cycles-pp.__might_sleep
2.29 -0.3 2.03 perf-profile.children.cycles-pp.skb_checksum
0.38 ? 10% -0.2 0.14 ? 3% perf-profile.children.cycles-pp.scheduler_tick
0.68 -0.2 0.44 perf-profile.children.cycles-pp.ip_rcv_finish
0.52 ? 7% -0.2 0.29 ? 4% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.48 ? 7% -0.2 0.25 ? 5% perf-profile.children.cycles-pp.tick_sched_timer
0.47 ? 7% -0.2 0.24 ? 6% perf-profile.children.cycles-pp.tick_sched_handle
0.46 ? 8% -0.2 0.24 ? 5% perf-profile.children.cycles-pp.update_process_times
1.06 -0.2 0.86 perf-profile.children.cycles-pp.dccp_feat_activate_values
0.64 ? 4% -0.2 0.44 ? 5% perf-profile.children.cycles-pp.hrtimer_interrupt
0.65 ? 4% -0.2 0.44 ? 4% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.55 ? 3% -0.2 0.37 ? 5% perf-profile.children.cycles-pp.stress_dccp_server
0.26 -0.2 0.09 ? 4% perf-profile.children.cycles-pp.dccp_feat_list_purge
0.83 ? 3% -0.2 0.67 perf-profile.children.cycles-pp.select_task_rq_fair
0.25 ? 4% -0.2 0.09 perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.41 ? 2% -0.2 0.26 perf-profile.children.cycles-pp.ccid2_hc_tx_packet_sent
0.24 ? 3% -0.2 0.09 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.36 ? 2% -0.2 0.21 perf-profile.children.cycles-pp.rcu_all_qs
0.32 ? 3% -0.1 0.20 perf-profile.children.cycles-pp._raw_spin_unlock_bh
4.06 -0.1 3.94 perf-profile.children.cycles-pp.dccp_v4_rcv
0.93 -0.1 0.82 perf-profile.children.cycles-pp.dccp_v4_conn_request
0.15 ? 3% -0.1 0.04 ? 44% perf-profile.children.cycles-pp.dccp_v4_reqsk_destructor
0.65 -0.1 0.56 perf-profile.children.cycles-pp.dccp_create_openreq_child
0.18 ? 2% -0.1 0.10 perf-profile.children.cycles-pp.dccp_feat_val_destructor
0.59 ? 2% -0.1 0.52 perf-profile.children.cycles-pp.finish_task_switch
0.17 ? 12% -0.1 0.11 ? 9% perf-profile.children.cycles-pp.ktime_get
0.13 -0.0 0.09 perf-profile.children.cycles-pp.inet_csk_destroy_sock
0.12 -0.0 0.08 perf-profile.children.cycles-pp.dccp_destroy_sock
0.14 -0.0 0.10 ? 3% perf-profile.children.cycles-pp.dccp_sk_destruct
0.18 ? 2% -0.0 0.14 perf-profile.children.cycles-pp.dccp_send_close
0.72 -0.0 0.68 perf-profile.children.cycles-pp.dccp_v4_request_recv_sock
0.20 ? 2% -0.0 0.16 ? 3% perf-profile.children.cycles-pp.dccp_set_state
0.09 -0.0 0.07 ? 5% perf-profile.children.cycles-pp.getsockname
0.07 ? 5% -0.0 0.05 perf-profile.children.cycles-pp.__x64_sys_getsockname
0.07 ? 7% -0.0 0.05 perf-profile.children.cycles-pp.__sys_getsockname
0.23 ? 3% -0.0 0.21 ? 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
0.09 -0.0 0.08 ? 4% perf-profile.children.cycles-pp.kmalloc_order_trace
0.07 +0.0 0.08 perf-profile.children.cycles-pp.enqueue_timer
0.05 +0.0 0.06 ? 6% perf-profile.children.cycles-pp.ccid2_hc_tx_parse_options
0.09 ? 5% +0.0 0.10 ? 3% perf-profile.children.cycles-pp.sk_prot_alloc
0.06 ? 7% +0.0 0.08 perf-profile.children.cycles-pp.rb_next
0.13 ? 2% +0.0 0.14 ? 3% perf-profile.children.cycles-pp.alloc_inode
0.17 ? 2% +0.0 0.19 ? 2% perf-profile.children.cycles-pp.sock_alloc
0.05 +0.0 0.07 perf-profile.children.cycles-pp.ioctl
0.06 ? 6% +0.0 0.08 perf-profile.children.cycles-pp.dccp_insert_fn_opt
0.08 ? 6% +0.0 0.10 ? 3% perf-profile.children.cycles-pp.sock_alloc_inode
1.04 +0.0 1.06 perf-profile.children.cycles-pp.sock_recvmsg
0.06 ? 6% +0.0 0.09 ? 4% perf-profile.children.cycles-pp.d_alloc_pseudo
0.06 +0.0 0.09 ? 6% perf-profile.children.cycles-pp.__d_alloc
0.10 +0.0 0.13 ? 2% perf-profile.children.cycles-pp.sk_alloc
0.34 +0.0 0.37 perf-profile.children.cycles-pp.dccp_rcv_close
0.11 +0.0 0.14 ? 3% perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
0.06 ? 6% +0.0 0.10 ? 5% perf-profile.children.cycles-pp.getpeername
0.11 ? 3% +0.0 0.14 ? 2% perf-profile.children.cycles-pp.dccp_rcv_closereq
0.07 +0.0 0.10 ? 4% perf-profile.children.cycles-pp.inet_csk_clear_xmit_timers
0.09 ? 4% +0.0 0.13 ? 2% perf-profile.children.cycles-pp.dccp_feat_push_confirm
0.05 +0.0 0.09 ? 4% perf-profile.children.cycles-pp.sched_clock
0.09 +0.0 0.13 perf-profile.children.cycles-pp.ip_route_output_flow
0.09 +0.0 0.13 ? 7% perf-profile.children.cycles-pp.eth_type_trans
0.02 ?141% +0.0 0.06 perf-profile.children.cycles-pp.apparmor_ip_postroute
0.07 +0.0 0.12 ? 4% perf-profile.children.cycles-pp.netdev_core_pick_tx
0.09 +0.0 0.14 ? 3% perf-profile.children.cycles-pp.sk_clone_lock
0.08 +0.0 0.12 ? 4% perf-profile.children.cycles-pp.__build_skb_around
0.12 ? 3% +0.0 0.17 perf-profile.children.cycles-pp.ip_route_output_key_hash
0.09 ? 5% +0.0 0.14 ? 2% perf-profile.children.cycles-pp.inet_csk_clone_lock
0.10 +0.0 0.15 perf-profile.children.cycles-pp.internal_add_timer
0.05 +0.1 0.10 perf-profile.children.cycles-pp.inet_ehash_insert
0.00 +0.1 0.05 perf-profile.children.cycles-pp.get_nohz_timer_target
0.00 +0.1 0.05 perf-profile.children.cycles-pp.apparmor_socket_sock_rcv_skb
0.00 +0.1 0.05 perf-profile.children.cycles-pp.skb_free_head
0.00 +0.1 0.05 perf-profile.children.cycles-pp.dccp_sample_rtt
0.00 +0.1 0.05 perf-profile.children.cycles-pp.move_addr_to_user
0.00 +0.1 0.05 perf-profile.children.cycles-pp.__usecs_to_jiffies
0.00 +0.1 0.05 perf-profile.children.cycles-pp.rcu_qs
0.00 +0.1 0.05 perf-profile.children.cycles-pp.evict
0.00 +0.1 0.05 perf-profile.children.cycles-pp.inet_csk_route_child_sock
0.00 +0.1 0.05 ? 7% perf-profile.children.cycles-pp.kfree_skb_reason
0.00 +0.1 0.05 ? 7% perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
0.00 +0.1 0.05 ? 8% perf-profile.children.cycles-pp.dccp_feat_propagate_ccid
1.50 +0.1 1.56 perf-profile.children.cycles-pp._raw_spin_lock
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.fib_table_lookup
0.02 ?141% +0.1 0.07 ? 9% perf-profile.children.cycles-pp.ksoftirqd_running
0.07 ? 5% +0.1 0.13 ? 2% perf-profile.children.cycles-pp.security_file_free
0.00 +0.1 0.06 ? 8% perf-profile.children.cycles-pp.dccp_feat_server_ccid_dependencies
0.15 ? 3% +0.1 0.21 ? 2% perf-profile.children.cycles-pp.skb_unlink
0.07 +0.1 0.13 ? 2% perf-profile.children.cycles-pp.apparmor_file_free_security
0.06 +0.1 0.12 ? 3% perf-profile.children.cycles-pp.skb_clone_tx_timestamp
0.05 +0.1 0.11 perf-profile.children.cycles-pp.inet_csk_reqsk_queue_hash_add
0.00 +0.1 0.06 perf-profile.children.cycles-pp.kfree_skbmem
0.00 +0.1 0.06 perf-profile.children.cycles-pp.perf_swevent_event
0.00 +0.1 0.06 perf-profile.children.cycles-pp.cr4_update_irqsoff
0.08 +0.1 0.14 perf-profile.children.cycles-pp.__inet_hash_connect
0.00 +0.1 0.06 ? 6% perf-profile.children.cycles-pp.dccp_timestamp
0.01 ?223% +0.1 0.07 perf-profile.children.cycles-pp.__x64_sys_getpeername
0.31 +0.1 0.37 perf-profile.children.cycles-pp.__ip_finish_output
0.00 +0.1 0.06 ? 7% perf-profile.children.cycles-pp.security_sk_clone
0.00 +0.1 0.06 ? 7% perf-profile.children.cycles-pp.apparmor_sk_clone_security
0.20 ? 5% +0.1 0.27 ? 5% perf-profile.children.cycles-pp.cpuacct_charge
0.15 ? 2% +0.1 0.22 ? 2% perf-profile.children.cycles-pp.dccp_done
0.07 +0.1 0.14 ? 3% perf-profile.children.cycles-pp.apparmor_file_alloc_security
0.97 +0.1 1.04 perf-profile.children.cycles-pp.dccp_send_ack
0.10 +0.1 0.17 ? 2% perf-profile.children.cycles-pp.dccp_make_response
0.07 ? 5% +0.1 0.14 ? 3% perf-profile.children.cycles-pp.kmalloc_slab
0.00 +0.1 0.07 ? 5% perf-profile.children.cycles-pp.dccp_time_wait
0.00 +0.1 0.07 ? 5% perf-profile.children.cycles-pp.inet_lhash2_lookup
0.00 +0.1 0.07 perf-profile.children.cycles-pp.perf_trace_run_bpf_submit
0.00 +0.1 0.07 perf-profile.children.cycles-pp.dccp_insert_options_rsk
0.00 +0.1 0.07 perf-profile.children.cycles-pp.__sys_getpeername
0.00 +0.1 0.07 perf-profile.children.cycles-pp.inet_unhash
0.07 ? 5% +0.1 0.14 ? 2% perf-profile.children.cycles-pp.dccp_qpolicy_top
0.09 ? 4% +0.1 0.16 ? 2% perf-profile.children.cycles-pp.__kmalloc_track_caller
0.01 ?223% +0.1 0.08 perf-profile.children.cycles-pp.__raise_softirq_irqoff
0.00 +0.1 0.07 ? 5% perf-profile.children.cycles-pp.__inet_lookup_listener
0.03 ? 70% +0.1 0.11 ? 4% perf-profile.children.cycles-pp.obj_cgroup_charge
0.00 +0.1 0.07 ? 6% perf-profile.children.cycles-pp.rb_insert_color
0.00 +0.1 0.07 ? 6% perf-profile.children.cycles-pp.apparmor_socket_post_create
0.09 ? 5% +0.1 0.17 ? 4% perf-profile.children.cycles-pp.should_failslab
0.00 +0.1 0.08 ? 6% perf-profile.children.cycles-pp.security_socket_post_create
0.14 ? 3% +0.1 0.21 ? 3% perf-profile.children.cycles-pp.memcpy_erms
0.12 ? 4% +0.1 0.20 perf-profile.children.cycles-pp.nf_hook_slow
0.00 +0.1 0.08 ? 6% perf-profile.children.cycles-pp.perf_exclude_event
0.00 +0.1 0.08 ? 11% perf-profile.children.cycles-pp.cgroup_rstat_updated
0.08 +0.1 0.16 perf-profile.children.cycles-pp.security_file_alloc
0.00 +0.1 0.08 ? 21% perf-profile.children.cycles-pp.propagate_protected_usage
0.09 +0.1 0.17 ? 2% perf-profile.children.cycles-pp.raw_local_deliver
0.00 +0.1 0.08 ? 5% perf-profile.children.cycles-pp.page_counter_charge
0.20 ? 2% +0.1 0.28 perf-profile.children.cycles-pp.ip_send_check
0.00 +0.1 0.09 ? 4% perf-profile.children.cycles-pp.dccp_ackvec_buflen
0.12 ? 4% +0.1 0.21 ? 3% perf-profile.children.cycles-pp.dccp_feat_init
0.00 +0.1 0.09 perf-profile.children.cycles-pp.detach_if_pending
0.21 +0.1 0.30 perf-profile.children.cycles-pp.dccp_qpolicy_pop
0.00 +0.1 0.09 ? 5% perf-profile.children.cycles-pp.obj_cgroup_charge_pages
0.16 +0.1 0.26 ? 3% perf-profile.children.cycles-pp.migrate_disable
0.12 ? 3% +0.1 0.22 ? 3% perf-profile.children.cycles-pp.dccp_v4_init_sock
0.11 +0.1 0.21 ? 3% perf-profile.children.cycles-pp.__feat_register_sp
0.31 +0.1 0.41 perf-profile.children.cycles-pp.read_tsc
0.56 ? 2% +0.1 0.66 perf-profile.children.cycles-pp.irqtime_account_irq
0.00 +0.1 0.10 perf-profile.children.cycles-pp.check_cfs_rq_runtime
0.00 +0.1 0.10 perf-profile.children.cycles-pp.rcu_note_context_switch
0.18 +0.1 0.28 perf-profile.children.cycles-pp.__alloc_file
0.00 +0.1 0.10 ? 3% perf-profile.children.cycles-pp.perf_trace_buf_alloc
0.18 ? 2% +0.1 0.29 perf-profile.children.cycles-pp.alloc_empty_file
0.19 +0.1 0.29 ? 2% perf-profile.children.cycles-pp.alloc_file
0.14 ? 5% +0.1 0.25 perf-profile.children.cycles-pp.__check_heap_object
0.01 ?223% +0.1 0.12 ? 3% perf-profile.children.cycles-pp.check_stack_object
0.00 +0.1 0.11 perf-profile.children.cycles-pp.rb_erase
1.29 +0.1 1.40 perf-profile.children.cycles-pp.dccp_connect
1.69 +0.1 1.80 perf-profile.children.cycles-pp.alloc_skb_with_frags
0.80 +0.1 0.92 perf-profile.children.cycles-pp.__kmalloc_node_track_caller
0.09 +0.1 0.21 ? 3% perf-profile.children.cycles-pp.dccp_insert_option_padding
0.00 +0.1 0.12 ? 4% perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
0.00 +0.1 0.12 ? 3% perf-profile.children.cycles-pp.___ratelimit
0.26 +0.1 0.39 perf-profile.children.cycles-pp.dccp_feat_insert_opts
0.07 +0.1 0.20 ? 2% perf-profile.children.cycles-pp.validate_xmit_xfrm
0.18 ? 2% +0.1 0.31 perf-profile.children.cycles-pp.inet_ehashfn
0.15 ? 2% +0.1 0.28 perf-profile.children.cycles-pp.migrate_enable
0.00 +0.1 0.13 ? 5% perf-profile.children.cycles-pp.switch_ldt
0.12 ? 3% +0.1 0.25 ? 2% perf-profile.children.cycles-pp.dccp_feat_clone_list
0.10 ? 3% +0.1 0.23 ? 3% perf-profile.children.cycles-pp.skb_network_protocol
0.23 +0.1 0.36 perf-profile.children.cycles-pp.inet_create
0.00 +0.1 0.13 perf-profile.children.cycles-pp.wakeup_preempt_entity
0.19 +0.1 0.32 ? 7% perf-profile.children.cycles-pp.stress_dccp_client
0.08 ? 4% +0.1 0.22 ? 2% perf-profile.children.cycles-pp.dccp_ackvec_add_new
0.87 +0.1 1.01 perf-profile.children.cycles-pp.dccp_finish_passive_close
0.16 ? 3% +0.1 0.30 ? 2% perf-profile.children.cycles-pp.skb_push
0.07 +0.1 0.21 perf-profile.children.cycles-pp.woken_wake_function
0.11 ? 4% +0.1 0.25 perf-profile.children.cycles-pp.__fdget
0.27 ? 2% +0.1 0.42 ? 2% perf-profile.children.cycles-pp.alloc_file_pseudo
0.00 +0.1 0.14 ? 2% perf-profile.children.cycles-pp.__enqueue_entity
0.28 +0.1 0.42 perf-profile.children.cycles-pp.sock_alloc_file
0.38 +0.1 0.52 perf-profile.children.cycles-pp.dccp_feat_parse_options
0.13 ? 5% +0.1 0.27 ? 8% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.00 +0.1 0.14 ? 3% perf-profile.children.cycles-pp.dput
0.17 +0.1 0.32 ? 2% perf-profile.children.cycles-pp.dccp_feat_clone_sp_val
0.16 ? 3% +0.1 0.31 perf-profile.children.cycles-pp.dccp_ackvec_input
0.13 ? 5% +0.1 0.28 perf-profile.children.cycles-pp.add_wait_queue
0.00 +0.1 0.15 ? 2% perf-profile.children.cycles-pp.clear_buddies
0.31 ? 2% +0.2 0.47 perf-profile.children.cycles-pp.__ip_local_out
0.00 +0.2 0.16 ? 2% perf-profile.children.cycles-pp.place_entity
0.21 ? 2% +0.2 0.38 ? 2% perf-profile.children.cycles-pp.kmemdup
0.16 ? 3% +0.2 0.33 perf-profile.children.cycles-pp.__might_fault
0.02 ? 99% +0.2 0.19 perf-profile.children.cycles-pp.update_irq_load_avg
0.87 +0.2 1.04 perf-profile.children.cycles-pp.kmalloc_reserve
0.37 +0.2 0.54 perf-profile.children.cycles-pp.__sock_create
0.59 +0.2 0.77 perf-profile.children.cycles-pp.dccp_ackvec_parsed_add
0.66 +0.2 0.84 perf-profile.children.cycles-pp.dccp_close
0.66 +0.2 0.84 perf-profile.children.cycles-pp.inet_release
0.68 +0.2 0.86 perf-profile.children.cycles-pp.__sock_release
0.36 ? 2% +0.2 0.55 perf-profile.children.cycles-pp.ip_local_out
0.68 +0.2 0.87 perf-profile.children.cycles-pp.sock_close
0.64 ? 3% +0.2 0.83 perf-profile.children.cycles-pp.rcu_do_batch
0.00 +0.2 0.19 perf-profile.children.cycles-pp.perf_trace_buf_update
0.65 ? 4% +0.2 0.86 perf-profile.children.cycles-pp.rcu_core
0.30 +0.2 0.50 perf-profile.children.cycles-pp.lock_timer_base
2.63 +0.2 2.84 perf-profile.children.cycles-pp.__alloc_skb
0.28 ? 2% +0.2 0.50 perf-profile.children.cycles-pp.remove_wait_queue
3.61 +0.2 3.83 perf-profile.children.cycles-pp.__inet_stream_connect
0.28 +0.2 0.51 perf-profile.children.cycles-pp.__list_del_entry_valid
0.24 +0.2 0.46 perf-profile.children.cycles-pp.del_timer
3.65 +0.2 3.88 perf-profile.children.cycles-pp.__x64_sys_connect
0.00 +0.2 0.23 perf-profile.children.cycles-pp.se_is_idle
0.41 +0.2 0.64 perf-profile.children.cycles-pp.mod_timer
3.65 +0.2 3.88 perf-profile.children.cycles-pp.__sys_connect
3.61 +0.2 3.84 perf-profile.children.cycles-pp.inet_stream_connect
0.36 +0.2 0.59 perf-profile.children.cycles-pp.sk_stop_timer
0.00 +0.2 0.23 ? 10% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.00 +0.2 0.23 ? 5% perf-profile.children.cycles-pp.perf_trace_sched_switch
0.52 +0.2 0.75 perf-profile.children.cycles-pp.__sys_socket
0.52 +0.2 0.75 perf-profile.children.cycles-pp.__x64_sys_socket
0.31 +0.2 0.55 perf-profile.children.cycles-pp.ip_rcv_core
0.54 +0.2 0.78 perf-profile.children.cycles-pp.__socket
0.22 ? 3% +0.2 0.46 ? 2% perf-profile.children.cycles-pp.netif_skb_features
3.67 +0.2 3.92 perf-profile.children.cycles-pp.connect
0.05 ? 7% +0.3 0.32 perf-profile.children.cycles-pp.__rdgsbase_inactive
0.48 +0.3 0.74 perf-profile.children.cycles-pp.sk_reset_timer
0.45 ? 3% +0.3 0.72 perf-profile.children.cycles-pp.native_sched_clock
0.37 ? 4% +0.3 0.64 perf-profile.children.cycles-pp.copyin
0.23 ? 3% +0.3 0.51 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
1.52 +0.3 1.80 perf-profile.children.cycles-pp.dccp_v4_connect
0.46 ? 2% +0.3 0.74 perf-profile.children.cycles-pp.kmem_cache_alloc_node
0.61 +0.3 0.89 perf-profile.children.cycles-pp.simple_copy_to_iter
0.10 ? 8% +0.3 0.39 perf-profile.children.cycles-pp.perf_tp_event
0.18 ? 7% +0.3 0.48 ? 2% perf-profile.children.cycles-pp.pick_next_entity
0.58 ? 4% +0.3 0.88 perf-profile.children.cycles-pp.update_rq_clock
0.56 +0.3 0.86 perf-profile.children.cycles-pp.ip_finish_output
0.00 +0.3 0.32 ? 3% perf-profile.children.cycles-pp.set_next_buddy
0.40 +0.3 0.72 perf-profile.children.cycles-pp.dccp_ackvec_parsed_cleanup
0.00 +0.3 0.32 perf-profile.children.cycles-pp.__calc_delta
0.12 ? 3% +0.3 0.45 perf-profile.children.cycles-pp.__virt_addr_valid
0.55 ? 2% +0.3 0.88 perf-profile.children.cycles-pp.sched_clock_cpu
0.15 ? 13% +0.3 0.49 ? 5% perf-profile.children.cycles-pp.page_counter_cancel
0.42 ? 2% +0.3 0.77 perf-profile.children.cycles-pp.validate_xmit_skb
1.02 +0.4 1.37 perf-profile.children.cycles-pp.__fput
0.13 ? 5% +0.4 0.49 ? 2% perf-profile.children.cycles-pp.run_ksoftirqd
0.52 +0.4 0.87 perf-profile.children.cycles-pp.dccp_ackvec_update_records
0.37 +0.4 0.73 ? 3% perf-profile.children.cycles-pp.ktime_get_with_offset
1.67 ? 2% +0.4 2.03 perf-profile.children.cycles-pp.kmem_cache_free
1.04 +0.4 1.40 perf-profile.children.cycles-pp.task_work_run
0.16 ? 4% +0.4 0.53 ? 2% perf-profile.children.cycles-pp.kthread
0.14 ? 5% +0.4 0.51 ? 2% perf-profile.children.cycles-pp.smpboot_thread_fn
0.16 ? 4% +0.4 0.53 ? 2% perf-profile.children.cycles-pp.ret_from_fork
0.08 ? 4% +0.4 0.48 perf-profile.children.cycles-pp.__wrgsbase_inactive
0.60 +0.4 1.01 perf-profile.children.cycles-pp.kmem_cache_alloc
0.16 ? 13% +0.4 0.56 ? 6% perf-profile.children.cycles-pp.page_counter_uncharge
0.14 ? 3% +0.4 0.54 perf-profile.children.cycles-pp.security_sock_rcv_skb
0.17 ? 14% +0.4 0.57 ? 6% perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
0.45 +0.4 0.86 perf-profile.children.cycles-pp.__netif_receive_skb_core
0.06 +0.4 0.48 ? 7% perf-profile.children.cycles-pp.update_min_vruntime
1.14 +0.4 1.57 perf-profile.children.cycles-pp.__close
0.99 +0.4 1.43 perf-profile.children.cycles-pp.csum_partial
3.65 +0.4 4.09 perf-profile.children.cycles-pp.__dccp_rcv_established
0.83 ? 3% +0.4 1.28 perf-profile.children.cycles-pp.set_next_entity
0.56 ? 2% +0.5 1.01 perf-profile.children.cycles-pp._copy_from_iter
0.84 +0.5 1.31 perf-profile.children.cycles-pp.__check_object_size
0.48 +0.5 0.96 perf-profile.children.cycles-pp.enqueue_to_backlog
0.22 ? 3% +0.5 0.72 perf-profile.children.cycles-pp.___perf_sw_event
0.86 +0.5 1.36 perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
0.19 ? 3% +0.5 0.69 perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
0.90 +0.5 1.41 perf-profile.children.cycles-pp.kfree
1.68 +0.5 2.19 perf-profile.children.cycles-pp.dccp_parse_options
0.21 +0.5 0.74 perf-profile.children.cycles-pp.__list_add_valid
0.08 ? 6% +0.5 0.62 perf-profile.children.cycles-pp.dccp_feat_push_change
0.40 ? 2% +0.5 0.94 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.08 +0.5 0.62 perf-profile.children.cycles-pp.dccp_feat_signal_nn_change
0.15 ? 3% +0.5 0.70 perf-profile.children.cycles-pp.dccp_feat_entry_new
0.73 +0.5 1.28 perf-profile.children.cycles-pp.dccp_invalid_packet
0.65 +0.8 1.41 perf-profile.children.cycles-pp.sk_filter_trim_cap
0.23 ? 4% +0.8 1.01 perf-profile.children.cycles-pp.check_preempt_curr
0.39 +0.8 1.17 perf-profile.children.cycles-pp.kmem_cache_alloc_trace
0.00 +0.8 0.79 perf-profile.children.cycles-pp.put_prev_entity
0.25 ? 3% +0.8 1.08 perf-profile.children.cycles-pp.ttwu_do_wakeup
1.19 +0.9 2.06 perf-profile.children.cycles-pp.enqueue_entity
0.90 +0.9 1.77 perf-profile.children.cycles-pp.netif_rx_internal
0.00 +0.9 0.89 perf-profile.children.cycles-pp.check_preempt_wakeup
1.09 +0.9 1.98 perf-profile.children.cycles-pp.ccid2_hc_tx_packet_recv
0.94 +0.9 1.84 perf-profile.children.cycles-pp.netif_rx
12.30 +0.9 13.22 perf-profile.children.cycles-pp.dccp_deliver_input_to_ccids
1.24 +0.9 2.17 perf-profile.children.cycles-pp.dccp_insert_options
0.18 ? 3% +0.9 1.11 ? 2% perf-profile.children.cycles-pp.__update_load_avg_se
0.09 ? 4% +0.9 1.04 perf-profile.children.cycles-pp.save_fpregs_to_fpstate
11.86 +1.0 12.84 perf-profile.children.cycles-pp.ccid2_hc_rx_packet_recv
0.32 ? 2% +1.0 1.35 perf-profile.children.cycles-pp.__switch_to
0.22 ? 2% +1.0 1.26 perf-profile.children.cycles-pp.__switch_to_asm
0.08 +1.1 1.14 ? 3% perf-profile.children.cycles-pp.reweight_entity
1.92 +1.1 2.98 perf-profile.children.cycles-pp.sock_def_readable
0.07 ? 5% +1.2 1.26 perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
0.17 ? 4% +1.5 1.70 perf-profile.children.cycles-pp.switch_fpu_return
1.81 +1.6 3.38 perf-profile.children.cycles-pp.dequeue_task_fair
1.40 +1.7 3.10 perf-profile.children.cycles-pp.update_load_avg
0.62 ? 2% +1.9 2.51 ? 2% perf-profile.children.cycles-pp.update_curr
1.52 ? 5% +2.0 3.51 perf-profile.children.cycles-pp.pick_next_task_fair
1.64 ? 2% +2.2 3.82 perf-profile.children.cycles-pp.enqueue_task_fair
1.66 ? 2% +2.3 3.92 perf-profile.children.cycles-pp.ttwu_do_activate
17.02 +2.4 19.44 perf-profile.children.cycles-pp.dccp_rcv_established
3.76 +2.7 6.43 perf-profile.children.cycles-pp.dccp_write_space
0.04 ? 44% +2.7 2.76 perf-profile.children.cycles-pp.load_new_mm_cr3
4.17 +2.8 6.96 perf-profile.children.cycles-pp.sock_wfree
24.30 +2.8 27.09 perf-profile.children.cycles-pp.dccp_v4_do_rcv
2.16 +3.1 5.31 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
1.64 +3.3 4.96 perf-profile.children.cycles-pp.exit_to_user_mode_prepare
4.30 +3.5 7.80 perf-profile.children.cycles-pp.try_to_wake_up
4.71 +3.5 8.25 perf-profile.children.cycles-pp.__wake_up_common
0.26 +3.6 3.82 perf-profile.children.cycles-pp.switch_mm_irqs_off
4.93 +4.0 8.89 perf-profile.children.cycles-pp.__wake_up_common_lock
5.43 +4.3 9.70 perf-profile.children.cycles-pp.loopback_xmit
5.60 +4.4 10.01 perf-profile.children.cycles-pp.dev_hard_start_xmit
0.75 ? 3% +4.5 5.28 perf-profile.children.cycles-pp.__cond_resched
0.00 +4.7 4.73 perf-profile.children.cycles-pp.preempt_schedule_common
6.98 +5.0 11.96 perf-profile.children.cycles-pp.__dev_queue_xmit
3.71 ? 2% +6.2 9.94 perf-profile.children.cycles-pp.schedule_timeout
5.51 ? 2% +7.0 12.48 perf-profile.children.cycles-pp.sk_wait_data
3.40 ? 2% +7.1 10.48 perf-profile.children.cycles-pp.wait_woken
3.68 ? 2% +7.5 11.21 perf-profile.children.cycles-pp.schedule
14.69 +8.0 22.74 perf-profile.children.cycles-pp.__sk_receive_skb
18.98 +8.3 27.24 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
19.07 +8.4 27.48 perf-profile.children.cycles-pp.ip_local_deliver_finish
19.18 +8.6 27.77 perf-profile.children.cycles-pp.ip_local_deliver
20.30 +8.8 29.06 perf-profile.children.cycles-pp.ip_rcv
57.14 +9.1 66.26 perf-profile.children.cycles-pp.__send
82.42 +9.1 91.57 perf-profile.children.cycles-pp.do_syscall_64
83.20 +9.3 92.54 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
20.88 +9.4 30.27 perf-profile.children.cycles-pp.__netif_receive_skb_one_core
5.91 ? 2% +9.6 15.52 perf-profile.children.cycles-pp.__schedule
21.52 +9.8 31.31 perf-profile.children.cycles-pp.process_backlog
21.58 +9.8 31.42 perf-profile.children.cycles-pp.__napi_poll
21.87 +10.1 31.98 perf-profile.children.cycles-pp.net_rx_action
49.74 +10.1 59.89 perf-profile.children.cycles-pp.__x64_sys_sendto
49.42 +10.2 59.63 perf-profile.children.cycles-pp.__sys_sendto
23.79 +10.3 34.07 perf-profile.children.cycles-pp.__softirqentry_text_start
23.92 +10.4 34.32 perf-profile.children.cycles-pp.__local_bh_enable_ip
23.12 +10.5 33.63 perf-profile.children.cycles-pp.do_softirq
32.41 +11.1 43.46 perf-profile.children.cycles-pp.dccp_transmit_skb
46.83 +11.4 58.23 perf-profile.children.cycles-pp.sock_sendmsg
28.91 +11.5 40.45 perf-profile.children.cycles-pp.__ip_queue_xmit
26.35 +11.8 38.19 perf-profile.children.cycles-pp.ip_finish_output2
27.37 +12.0 39.40 perf-profile.children.cycles-pp.ip_output
24.93 +16.3 41.22 perf-profile.children.cycles-pp.dccp_xmit_packet
25.16 +16.4 41.52 perf-profile.children.cycles-pp.dccp_write_xmit
40.19 +16.7 56.85 perf-profile.children.cycles-pp.dccp_sendmsg
5.50 ? 5% -5.1 0.38 ? 19% perf-profile.self.cycles-pp.intel_idle
5.24 ? 2% -4.1 1.14 ? 3% perf-profile.self.cycles-pp.aa_sk_perm
3.22 -1.9 1.33 perf-profile.self.cycles-pp._raw_spin_lock_bh
2.26 ? 3% -1.9 0.38 ? 4% perf-profile.self.cycles-pp.__send
1.86 -1.8 0.06 ? 7% perf-profile.self.cycles-pp.__slab_free
2.20 ? 3% -1.3 0.89 perf-profile.self.cycles-pp.syscall_return_via_sysret
1.44 -1.1 0.34 perf-profile.self.cycles-pp.skb_set_owner_w
1.55 -1.0 0.57 perf-profile.self.cycles-pp.dccp_recvmsg
1.14 -0.8 0.34 perf-profile.self.cycles-pp.dst_release
1.18 -0.8 0.41 perf-profile.self.cycles-pp.__skb_checksum
0.77 ? 2% -0.7 0.11 ? 7% perf-profile.self.cycles-pp.ipv4_dst_check
2.32 -0.6 1.69 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.34 -0.6 0.72 perf-profile.self.cycles-pp.__inet_lookup_established
1.78 -0.6 1.16 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
1.18 -0.6 0.63 perf-profile.self.cycles-pp.__sk_receive_skb
1.00 ? 4% -0.6 0.45 perf-profile.self.cycles-pp.update_cfs_group
0.97 ? 3% -0.5 0.44 perf-profile.self.cycles-pp.__sys_sendto
0.80 -0.5 0.28 perf-profile.self.cycles-pp.__ksize
0.88 -0.5 0.39 perf-profile.self.cycles-pp.dccp_ackvec_clear_state
0.88 -0.5 0.40 perf-profile.self.cycles-pp.dccp_check_seqno
0.60 ? 3% -0.4 0.22 ? 2% perf-profile.self.cycles-pp.__entry_text_start
1.10 ? 4% -0.4 0.75 perf-profile.self.cycles-pp.__fget_light
0.56 -0.3 0.23 perf-profile.self.cycles-pp.ip_rcv_finish_core
0.55 -0.3 0.24 perf-profile.self.cycles-pp.dccp_enqueue_skb
0.74 ? 2% -0.3 0.47 perf-profile.self.cycles-pp.ccid2_hc_tx_alloc_seq
0.30 ? 3% -0.2 0.05 perf-profile.self.cycles-pp.inet_sendmsg
1.09 -0.2 0.86 perf-profile.self.cycles-pp.__might_resched
0.82 ? 3% -0.2 0.59 perf-profile.self.cycles-pp.dccp_sendmsg
0.36 ? 2% -0.2 0.13 ? 2% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.53 -0.2 0.30 perf-profile.self.cycles-pp.release_sock
0.64 ? 2% -0.2 0.45 perf-profile.self.cycles-pp.__might_sleep
0.53 -0.2 0.34 perf-profile.self.cycles-pp.dccp_write_space
0.44 ? 2% -0.2 0.26 perf-profile.self.cycles-pp.sock_sendmsg
0.31 ? 4% -0.2 0.12 ? 4% perf-profile.self.cycles-pp.sockfd_lookup_light
0.45 -0.2 0.27 perf-profile.self.cycles-pp.__release_sock
0.47 ? 4% -0.2 0.29 perf-profile.self.cycles-pp.update_rq_clock
0.38 ? 3% -0.2 0.21 ? 3% perf-profile.self.cycles-pp.lock_sock_nested
0.29 ? 3% -0.2 0.13 ? 3% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
0.40 ? 2% -0.2 0.24 perf-profile.self.cycles-pp.ccid2_hc_tx_packet_sent
0.23 ? 3% -0.2 0.08 ? 4% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.48 ? 5% -0.1 0.34 perf-profile.self.cycles-pp.security_socket_sendmsg
0.42 ? 3% -0.1 0.28 ? 2% perf-profile.self.cycles-pp.stress_dccp_server
0.30 ? 2% -0.1 0.19 perf-profile.self.cycles-pp.dccp_ackvec_parsed_add
0.37 ? 3% -0.1 0.26 perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.60 -0.1 0.49 perf-profile.self.cycles-pp.try_to_wake_up
0.24 ? 3% -0.1 0.14 perf-profile.self.cycles-pp.rcu_all_qs
0.48 -0.1 0.38 perf-profile.self.cycles-pp.dccp_deliver_input_to_ccids
0.18 ? 4% -0.1 0.08 ? 4% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.33 ? 3% -0.1 0.24 perf-profile.self.cycles-pp.__wake_up_common
0.36 ? 5% -0.1 0.27 ? 2% perf-profile.self.cycles-pp.set_next_entity
0.22 ? 3% -0.1 0.13 perf-profile.self.cycles-pp._raw_spin_unlock_bh
0.19 ? 2% -0.1 0.14 ? 2% perf-profile.self.cycles-pp.skb_release_data
0.64 ? 2% -0.0 0.59 perf-profile.self.cycles-pp.do_syscall_64
0.70 -0.0 0.66 perf-profile.self.cycles-pp.__local_bh_enable_ip
0.08 ? 4% -0.0 0.05 perf-profile.self.cycles-pp.dccp_send_ack
0.36 ? 2% -0.0 0.33 perf-profile.self.cycles-pp.__cond_resched
0.28 ? 2% -0.0 0.25 perf-profile.self.cycles-pp.__x64_sys_sendto
0.19 ? 3% -0.0 0.16 ? 3% perf-profile.self.cycles-pp.sock_def_readable
0.06 -0.0 0.05 perf-profile.self.cycles-pp.qpolicy_simple_top
0.05 +0.0 0.06 perf-profile.self.cycles-pp.__dccp_feat_activate
0.08 ? 5% +0.0 0.10 ? 5% perf-profile.self.cycles-pp.dccp_ackvec_input
0.10 ? 4% +0.0 0.11 perf-profile.self.cycles-pp.__napi_poll
0.18 ? 2% +0.0 0.20 ? 2% perf-profile.self.cycles-pp.irqtime_account_irq
1.20 +0.0 1.22 perf-profile.self.cycles-pp.kmem_cache_free
0.06 +0.0 0.08 ? 5% perf-profile.self.cycles-pp.dccp_handle_ackvec_processing
0.10 ? 4% +0.0 0.14 ? 3% perf-profile.self.cycles-pp.dccp_v4_do_rcv
0.06 +0.0 0.09 perf-profile.self.cycles-pp.dccp_ackvec_parsed_cleanup
0.09 +0.0 0.12 ? 3% perf-profile.self.cycles-pp.sock_alloc_send_pskb
0.06 ? 9% +0.0 0.09 perf-profile.self.cycles-pp.netdev_core_pick_tx
0.11 ? 4% +0.0 0.15 ? 2% perf-profile.self.cycles-pp.__sk_dst_check
0.12 ? 3% +0.0 0.16 perf-profile.self.cycles-pp.dccp_write_xmit
0.08 ? 4% +0.0 0.12 ? 6% perf-profile.self.cycles-pp.eth_type_trans
0.07 +0.0 0.11 ? 3% perf-profile.self.cycles-pp.__sk_destruct
0.03 ? 70% +0.0 0.07 ? 5% perf-profile.self.cycles-pp.__inet_hash_connect
0.06 ? 6% +0.0 0.10 perf-profile.self.cycles-pp.__build_skb_around
0.08 +0.0 0.12 perf-profile.self.cycles-pp.sk_reset_timer
0.05 +0.0 0.09 perf-profile.self.cycles-pp.alloc_skb_with_frags
0.06 +0.0 0.10 perf-profile.self.cycles-pp.__skb_datagram_iter
0.06 +0.0 0.10 perf-profile.self.cycles-pp.skb_release_head_state
0.05 +0.0 0.09 ? 4% perf-profile.self.cycles-pp.should_failslab
0.05 ? 8% +0.0 0.10 ? 3% perf-profile.self.cycles-pp.check_preempt_curr
0.05 +0.0 0.10 ? 5% perf-profile.self.cycles-pp.ip_local_out
0.07 ? 5% +0.0 0.12 perf-profile.self.cycles-pp.sched_clock_cpu
0.07 +0.0 0.12 perf-profile.self.cycles-pp.__kmalloc_track_caller
0.06 ? 7% +0.0 0.11 ? 4% perf-profile.self.cycles-pp.__dccp_rcv_established
0.06 ? 6% +0.1 0.11 ? 3% perf-profile.self.cycles-pp.kmalloc_reserve
0.05 +0.1 0.10 perf-profile.self.cycles-pp.skb_clone_tx_timestamp
0.00 +0.1 0.05 perf-profile.self.cycles-pp.skb_copy_datagram_iter
0.00 +0.1 0.05 perf-profile.self.cycles-pp.skb_release_all
0.00 +0.1 0.05 perf-profile.self.cycles-pp.mod_objcg_state
0.00 +0.1 0.05 perf-profile.self.cycles-pp.skb_unlink
0.00 +0.1 0.05 perf-profile.self.cycles-pp.sk_alloc
0.00 +0.1 0.05 perf-profile.self.cycles-pp.rcu_note_context_switch
0.00 +0.1 0.05 perf-profile.self.cycles-pp.ccid2_hc_tx_parse_options
0.00 +0.1 0.05 perf-profile.self.cycles-pp.memset_erms
0.07 +0.1 0.12 ? 3% perf-profile.self.cycles-pp.apparmor_file_free_security
0.00 +0.1 0.05 ? 8% perf-profile.self.cycles-pp.netif_rx
0.00 +0.1 0.05 ? 8% perf-profile.self.cycles-pp.internal_add_timer
0.00 +0.1 0.05 ? 8% perf-profile.self.cycles-pp.check_cfs_rq_runtime
0.20 ? 4% +0.1 0.26 ? 5% perf-profile.self.cycles-pp.cpuacct_charge
0.00 +0.1 0.06 ? 9% perf-profile.self.cycles-pp.skb_queue_tail
0.00 +0.1 0.06 ? 9% perf-profile.self.cycles-pp.perf_trace_buf_alloc
0.02 ? 99% +0.1 0.08 ? 5% perf-profile.self.cycles-pp.dccp_qpolicy_top
0.00 +0.1 0.06 ? 6% perf-profile.self.cycles-pp.dccp_feat_activate_values
0.30 +0.1 0.36 perf-profile.self.cycles-pp.__ip_finish_output
0.00 +0.1 0.06 perf-profile.self.cycles-pp.sock_recvmsg
0.00 +0.1 0.06 perf-profile.self.cycles-pp.dccp_feat_signal_nn_change
0.00 +0.1 0.06 perf-profile.self.cycles-pp.rb_insert_color
0.14 ? 3% +0.1 0.20 perf-profile.self.cycles-pp._copy_to_iter
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.__raise_softirq_irqoff
0.10 ? 4% +0.1 0.17 ? 2% perf-profile.self.cycles-pp.__ip_local_out
0.08 ? 4% +0.1 0.14 ? 3% perf-profile.self.cycles-pp.ip_finish_output
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.ksoftirqd_running
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.dccp_insert_fn_opt
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.perf_exclude_event
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.apparmor_sk_clone_security
0.05 ? 7% +0.1 0.12 ? 3% perf-profile.self.cycles-pp.kmalloc_slab
0.00 +0.1 0.07 ? 7% perf-profile.self.cycles-pp.perf_trace_run_bpf_submit
0.00 +0.1 0.07 ? 11% perf-profile.self.cycles-pp.cgroup_rstat_updated
0.08 ? 5% +0.1 0.15 ? 3% perf-profile.self.cycles-pp.nf_hook_slow
0.00 +0.1 0.07 ? 5% perf-profile.self.cycles-pp.inet_lhash2_lookup
0.80 +0.1 0.87 perf-profile.self.cycles-pp.__dev_queue_xmit
0.09 +0.1 0.16 perf-profile.self.cycles-pp.dccp_rcv_established
0.06 ? 6% +0.1 0.13 ? 2% perf-profile.self.cycles-pp.apparmor_file_alloc_security
0.00 +0.1 0.07 perf-profile.self.cycles-pp.ttwu_do_wakeup
0.00 +0.1 0.07 perf-profile.self.cycles-pp.remove_wait_queue
0.00 +0.1 0.07 perf-profile.self.cycles-pp.apparmor_socket_post_create
0.09 +0.1 0.16 ? 2% perf-profile.self.cycles-pp.netif_rx_internal
0.06 ? 6% +0.1 0.13 ? 3% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.00 +0.1 0.07 ? 5% perf-profile.self.cycles-pp.perf_trace_buf_update
0.00 +0.1 0.07 ? 5% perf-profile.self.cycles-pp.detach_if_pending
0.00 +0.1 0.07 ? 5% perf-profile.self.cycles-pp.dccp_feat_entry_new
0.00 +0.1 0.07 ? 5% perf-profile.self.cycles-pp.dccp_ackvec_buflen
0.07 ? 5% +0.1 0.14 ? 3% perf-profile.self.cycles-pp.raw_local_deliver
0.11 +0.1 0.18 ? 2% perf-profile.self.cycles-pp.mod_timer
0.14 ? 2% +0.1 0.21 ? 2% perf-profile.self.cycles-pp.dccp_feat_parse_options
0.06 ? 8% +0.1 0.13 ? 2% perf-profile.self.cycles-pp.lock_timer_base
0.09 ? 4% +0.1 0.17 ? 2% perf-profile.self.cycles-pp.skb_checksum
0.00 +0.1 0.08 ? 6% perf-profile.self.cycles-pp.page_counter_charge
0.19 ? 3% +0.1 0.27 perf-profile.self.cycles-pp.ip_send_check
0.13 ? 3% +0.1 0.21 perf-profile.self.cycles-pp.memcpy_erms
0.09 +0.1 0.17 ? 3% perf-profile.self.cycles-pp.ip_rcv_finish
0.00 +0.1 0.08 perf-profile.self.cycles-pp.check_stack_object
0.00 +0.1 0.08 perf-profile.self.cycles-pp.rb_erase
0.00 +0.1 0.08 ? 21% perf-profile.self.cycles-pp.propagate_protected_usage
0.05 +0.1 0.13 ? 2% perf-profile.self.cycles-pp.dccp_insert_option_padding
0.00 +0.1 0.08 ? 4% perf-profile.self.cycles-pp.del_timer
0.26 +0.1 0.34 ? 2% perf-profile.self.cycles-pp.dccp_xmit_packet
0.12 ? 3% +0.1 0.20 perf-profile.self.cycles-pp.dccp_v4_send_check
0.14 ? 2% +0.1 0.23 ? 4% perf-profile.self.cycles-pp.migrate_disable
0.10 ? 4% +0.1 0.19 ? 3% perf-profile.self.cycles-pp.skb_push
0.00 +0.1 0.09 ? 4% perf-profile.self.cycles-pp.___ratelimit
0.00 +0.1 0.09 ? 4% perf-profile.self.cycles-pp.__might_fault
0.00 +0.1 0.09 perf-profile.self.cycles-pp.add_wait_queue
0.14 ? 2% +0.1 0.23 perf-profile.self.cycles-pp.dccp_feat_insert_opts
0.12 ? 3% +0.1 0.22 ? 2% perf-profile.self.cycles-pp.__check_heap_object
0.30 +0.1 0.39 perf-profile.self.cycles-pp.read_tsc
0.12 +0.1 0.22 ? 2% perf-profile.self.cycles-pp._copy_from_iter
0.20 +0.1 0.30 perf-profile.self.cycles-pp.validate_xmit_skb
0.00 +0.1 0.10 ? 3% perf-profile.self.cycles-pp.wakeup_preempt_entity
0.15 ? 2% +0.1 0.25 ? 2% perf-profile.self.cycles-pp.sk_filter_trim_cap
0.14 ? 3% +0.1 0.25 ? 2% perf-profile.self.cycles-pp.dccp_invalid_packet
0.00 +0.1 0.10 ? 4% perf-profile.self.cycles-pp.ttwu_do_activate
0.13 ? 2% +0.1 0.24 ? 2% perf-profile.self.cycles-pp.migrate_enable
0.17 ? 2% +0.1 0.28 perf-profile.self.cycles-pp.inet_ehashfn
0.00 +0.1 0.11 perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
0.09 ? 5% +0.1 0.20 ? 2% perf-profile.self.cycles-pp.ccid2_hc_rx_packet_recv
0.00 +0.1 0.11 ? 3% perf-profile.self.cycles-pp.clear_buddies
0.00 +0.1 0.11 ? 3% perf-profile.self.cycles-pp.__enqueue_entity
0.23 +0.1 0.34 ? 4% perf-profile.self.cycles-pp.recv
0.09 ? 5% +0.1 0.20 ? 2% perf-profile.self.cycles-pp.skb_network_protocol
0.00 +0.1 0.12 ? 4% perf-profile.self.cycles-pp.switch_ldt
0.00 +0.1 0.12 ? 4% perf-profile.self.cycles-pp.place_entity
0.41 +0.1 0.53 perf-profile.self.cycles-pp.sock_wfree
0.14 ? 2% +0.1 0.26 perf-profile.self.cycles-pp.dccp_ackvec_update_records
0.06 ? 6% +0.1 0.18 ? 3% perf-profile.self.cycles-pp.validate_xmit_xfrm
0.15 +0.1 0.27 ? 3% perf-profile.self.cycles-pp.stress_dccp_client
0.07 ? 7% +0.1 0.19 ? 3% perf-profile.self.cycles-pp.sock_common_recvmsg
0.64 +0.1 0.76 perf-profile.self.cycles-pp.dccp_v4_rcv
0.07 ? 6% +0.1 0.20 perf-profile.self.cycles-pp.dccp_ackvec_add_new
0.06 ? 7% +0.1 0.20 ? 2% perf-profile.self.cycles-pp.woken_wake_function
0.12 ? 4% +0.1 0.25 perf-profile.self.cycles-pp.netif_skb_features
0.07 +0.1 0.20 ? 2% perf-profile.self.cycles-pp.__wake_up_common_lock
0.12 ? 7% +0.1 0.26 ? 8% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.17 ? 2% +0.1 0.30 perf-profile.self.cycles-pp.dev_hard_start_xmit
0.00 +0.1 0.14 ? 2% perf-profile.self.cycles-pp.preempt_schedule_common
0.06 ? 7% +0.1 0.20 ? 2% perf-profile.self.cycles-pp.perf_tp_event
0.09 +0.2 0.24 perf-profile.self.cycles-pp.ip_local_deliver_finish
0.00 +0.2 0.16 ? 13% perf-profile.self.cycles-pp.__cgroup_account_cputime
0.04 ? 44% +0.2 0.20 perf-profile.self.cycles-pp.__x64_sys_recvfrom
0.84 ? 2% +0.2 0.99 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.16 ? 2% +0.2 0.32 ? 2% perf-profile.self.cycles-pp.do_softirq
0.00 +0.2 0.16 perf-profile.self.cycles-pp.se_is_idle
0.37 ? 6% +0.2 0.53 ? 3% perf-profile.self.cycles-pp.enqueue_task_fair
0.00 +0.2 0.16 ? 4% perf-profile.self.cycles-pp.select_idle_sibling
0.25 +0.2 0.41 perf-profile.self.cycles-pp.enqueue_to_backlog
0.05 ? 8% +0.2 0.22 ? 2% perf-profile.self.cycles-pp.__fdget
0.26 +0.2 0.44 perf-profile.self.cycles-pp.__list_del_entry_valid
0.10 ? 6% +0.2 0.27 perf-profile.self.cycles-pp.dequeue_entity
0.00 +0.2 0.18 ? 2% perf-profile.self.cycles-pp.update_irq_load_avg
0.11 ? 3% +0.2 0.29 perf-profile.self.cycles-pp.ip_local_deliver
1.07 +0.2 1.25 perf-profile.self.cycles-pp.dccp_transmit_skb
0.65 +0.2 0.83 perf-profile.self.cycles-pp.dccp_parse_options
0.67 +0.2 0.85 perf-profile.self.cycles-pp.ccid2_hc_tx_packet_recv
0.17 ? 6% +0.2 0.37 ? 3% perf-profile.self.cycles-pp.pick_next_entity
0.65 +0.2 0.85 ? 2% perf-profile.self.cycles-pp.__ip_queue_xmit
0.15 ? 3% +0.2 0.35 perf-profile.self.cycles-pp.ip_rcv
0.12 +0.2 0.33 ? 5% perf-profile.self.cycles-pp.ktime_get_with_offset
0.00 +0.2 0.21 ? 2% perf-profile.self.cycles-pp.put_prev_entity
0.39 +0.2 0.61 perf-profile.self.cycles-pp.__kmalloc_node_track_caller
0.00 +0.2 0.22 ? 4% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.11 ? 3% +0.2 0.33 perf-profile.self.cycles-pp.dequeue_task_fair
0.14 +0.2 0.36 perf-profile.self.cycles-pp.__netif_receive_skb_one_core
0.30 ? 2% +0.2 0.52 perf-profile.self.cycles-pp.ip_output
0.30 +0.2 0.53 perf-profile.self.cycles-pp.ip_rcv_core
0.12 +0.2 0.35 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.30 ? 2% +0.2 0.53 perf-profile.self.cycles-pp.kmem_cache_alloc_node
0.34 +0.2 0.57 perf-profile.self.cycles-pp.__softirqentry_text_start
0.82 ? 2% +0.2 1.06 perf-profile.self.cycles-pp.update_load_avg
0.05 ? 7% +0.2 0.30 perf-profile.self.cycles-pp.__rdgsbase_inactive
0.07 +0.3 0.32 perf-profile.self.cycles-pp.security_socket_recvmsg
0.43 ? 3% +0.3 0.69 perf-profile.self.cycles-pp.native_sched_clock
0.24 +0.3 0.49 perf-profile.self.cycles-pp.net_rx_action
0.42 +0.3 0.68 perf-profile.self.cycles-pp.__alloc_skb
0.16 ? 2% +0.3 0.42 perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
0.14 ? 2% +0.3 0.40 perf-profile.self.cycles-pp.__sys_recvfrom
0.00 +0.3 0.26 ? 4% perf-profile.self.cycles-pp.set_next_buddy
0.05 +0.3 0.32 perf-profile.self.cycles-pp.schedule_timeout
0.56 +0.3 0.84 perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
0.00 +0.3 0.29 perf-profile.self.cycles-pp.__calc_delta
0.37 +0.3 0.67 perf-profile.self.cycles-pp.kmem_cache_alloc
0.08 +0.3 0.38 ? 2% perf-profile.self.cycles-pp.select_task_rq_fair
0.35 +0.3 0.66 perf-profile.self.cycles-pp.dccp_insert_options
0.11 ? 3% +0.3 0.42 perf-profile.self.cycles-pp.__virt_addr_valid
0.00 +0.3 0.32 perf-profile.self.cycles-pp.schedule
0.31 +0.3 0.64 perf-profile.self.cycles-pp.process_backlog
0.09 ? 4% +0.3 0.43 ? 3% perf-profile.self.cycles-pp.sk_wait_data
0.15 ? 14% +0.3 0.49 ? 5% perf-profile.self.cycles-pp.page_counter_cancel
0.10 ? 5% +0.3 0.44 perf-profile.self.cycles-pp.switch_fpu_return
0.21 ? 4% +0.4 0.56 perf-profile.self.cycles-pp.wait_woken
0.31 +0.4 0.67 perf-profile.self.cycles-pp.kmem_cache_alloc_trace
1.11 +0.4 1.48 perf-profile.self.cycles-pp._raw_spin_lock
0.05 ? 7% +0.4 0.42 ? 8% perf-profile.self.cycles-pp.update_min_vruntime
0.08 ? 4% +0.4 0.46 perf-profile.self.cycles-pp.__wrgsbase_inactive
0.11 ? 4% +0.4 0.50 perf-profile.self.cycles-pp.security_sock_rcv_skb
0.20 ? 3% +0.4 0.59 perf-profile.self.cycles-pp.___perf_sw_event
0.44 +0.4 0.84 perf-profile.self.cycles-pp.__netif_receive_skb_core
0.00 +0.4 0.42 perf-profile.self.cycles-pp.check_preempt_wakeup
0.96 +0.4 1.39 perf-profile.self.cycles-pp.csum_partial
0.18 +0.5 0.65 perf-profile.self.cycles-pp.__list_add_valid
0.07 +0.5 0.54 ? 6% perf-profile.self.cycles-pp.reweight_entity
0.39 ? 2% +0.5 0.88 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.86 +0.5 1.36 perf-profile.self.cycles-pp.kfree
0.22 ? 2% +0.5 0.73 perf-profile.self.cycles-pp.loopback_xmit
0.49 +0.5 1.02 ? 2% perf-profile.self.cycles-pp.ip_finish_output2
0.06 +0.6 0.67 ? 2% perf-profile.self.cycles-pp.pick_next_task_fair
0.22 ? 3% +0.8 1.04 perf-profile.self.cycles-pp.switch_mm_irqs_off
0.19 ? 3% +0.8 1.03 perf-profile.self.cycles-pp.update_curr
0.17 ? 2% +0.9 1.06 ? 2% perf-profile.self.cycles-pp.__update_load_avg_se
0.09 ? 6% +0.9 1.02 perf-profile.self.cycles-pp.save_fpregs_to_fpstate
0.32 ? 2% +1.0 1.29 perf-profile.self.cycles-pp.__switch_to
0.21 ? 2% +1.0 1.24 perf-profile.self.cycles-pp.__switch_to_asm
0.07 ? 5% +1.2 1.25 perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
0.69 ? 2% +1.2 1.89 perf-profile.self.cycles-pp.__schedule
0.04 ? 44% +2.7 2.73 perf-profile.self.cycles-pp.load_new_mm_cr3
***************************************************************************************************
lkp-icl-2sp2: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode:
cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/50%/debian-10.4-x86_64-20200603.cgz/300s/5K/lkp-icl-2sp2/TCP_SENDFILE/netperf/0xd000280
commit:
6275d291a7 ("Merge irq/core into tip/master")
e9accc2386 ("sched: Stop searching for idle cpu if the LLC domain is overloaded")
6275d291a763090e e9accc2386f31d149ea339e8be4
---------------- ---------------------------
%stddev %change %stddev
\ | \
17338 +16.8% 20251 netperf.Throughput_Mbps
1109679 +16.8% 1296078 netperf.Throughput_total_Mbps
249210 ? 6% +14650.0% 36758554 ? 5% netperf.time.involuntary_context_switches
19077 +3.5% 19753 netperf.time.minor_page_faults
994.66 +24.4% 1237 netperf.time.user_time
8.128e+09 +16.8% 9.493e+09 netperf.workload
1575636 ? 5% +23.6% 1946780 meminfo.Mapped
13.78 +0.8 14.58 turbostat.C1%
15.43 -1.6 13.81 mpstat.cpu.all.soft%
3.41 +0.7 4.10 mpstat.cpu.all.usr%
440709 ? 10% +48.9% 656374 ? 21% numa-numastat.node0.local_node
505776 ? 6% +37.7% 696592 ? 24% numa-numastat.node0.numa_hit
4730818 +2.5% 4849620 vmstat.system.cs
2606454 -4.2% 2496267 vmstat.system.in
2644 ? 10% +43.0% 3782 ? 8% numa-meminfo.node0.Active
2644 ? 10% +43.0% 3782 ? 8% numa-meminfo.node0.Active(anon)
284023 ? 8% +98.7% 564442 ? 52% numa-meminfo.node0.Inactive
284023 ? 8% +98.7% 564442 ? 52% numa-meminfo.node0.Inactive(anon)
4735 ? 24% +58.5% 7506 ? 13% numa-meminfo.node0.PageTables
660.83 ? 10% +43.0% 944.83 ? 8% numa-vmstat.node0.nr_active_anon
71007 ? 8% +98.6% 141052 ? 52% numa-vmstat.node0.nr_inactive_anon
1184 ? 24% +58.2% 1874 ? 13% numa-vmstat.node0.nr_page_table_pages
660.83 ? 10% +43.0% 944.83 ? 8% numa-vmstat.node0.nr_zone_active_anon
71007 ? 8% +98.6% 141052 ? 52% numa-vmstat.node0.nr_zone_inactive_anon
1.14 ? 29% -90.2% 0.11 ?111% sched_debug.cfs_rq:/.load_avg.min
713960 -12.5% 624526 ? 3% sched_debug.cfs_rq:/.min_vruntime.stddev
101.58 ? 52% -89.2% 10.94 ?133% sched_debug.cfs_rq:/.runnable_avg.min
714113 -12.5% 624772 ? 3% sched_debug.cfs_rq:/.spread0.stddev
99.69 ? 50% -89.1% 10.86 ?135% sched_debug.cfs_rq:/.util_avg.min
154600 ? 4% +26.6% 195692 ? 3% sched_debug.cpu.avg_idle.avg
162529 ? 10% +46.9% 238804 ? 6% sched_debug.cpu.avg_idle.stddev
4.12 ? 2% +34.5% 5.54 ? 6% sched_debug.cpu.clock.stddev
2550588 ? 12% +37.9% 3517002 ? 3% sched_debug.cpu.nr_switches.min
1221544 ? 2% -8.5% 1117112 ? 3% sched_debug.cpu.nr_switches.stddev
2740194 -1.6% 2695985 proc-vmstat.nr_file_pages
2168032 -2.0% 2124294 proc-vmstat.nr_inactive_anon
396169 ? 5% +23.3% 488448 proc-vmstat.nr_mapped
2559 +7.8% 2759 proc-vmstat.nr_page_table_pages
2121254 -2.1% 2077045 proc-vmstat.nr_shmem
2168032 -2.0% 2124294 proc-vmstat.nr_zone_inactive_anon
714673 ? 2% +7.7% 769527 ? 2% proc-vmstat.numa_hint_faults
556124 ? 3% +8.9% 605412 ? 3% proc-vmstat.numa_hint_faults_local
6522277 -2.2% 6376540 proc-vmstat.numa_hit
6430747 -2.3% 6279842 proc-vmstat.numa_local
10312 ? 31% +117.3% 22405 ? 33% proc-vmstat.numa_pages_migrated
904795 ? 3% +10.0% 995168 ? 5% proc-vmstat.numa_pte_updates
6506590 -2.0% 6378588 proc-vmstat.pgalloc_normal
2035214 +4.3% 2123372 proc-vmstat.pgfault
10312 ? 31% +117.3% 22405 ? 33% proc-vmstat.pgmigrate_success
75710 ? 4% -8.9% 68938 ? 7% softirqs.CPU0.SCHED
47782 ? 7% -16.5% 39896 ? 10% softirqs.CPU1.SCHED
47511 ? 6% -14.3% 40738 ? 3% softirqs.CPU10.SCHED
57588 ? 3% -10.4% 51613 ? 6% softirqs.CPU114.SCHED
56773 ? 5% -8.5% 51962 ? 6% softirqs.CPU119.SCHED
60944 ? 8% -18.1% 49934 ? 7% softirqs.CPU123.SCHED
12115 ? 18% +25.6% 15211 ? 5% softirqs.CPU123.TIMER
49116 ? 8% -20.9% 38854 ? 6% softirqs.CPU18.SCHED
7100 ? 6% +24.3% 8824 ? 17% softirqs.CPU2.RCU
48215 ? 14% -25.4% 35957 ? 13% softirqs.CPU2.SCHED
47320 ? 9% -15.2% 40111 ? 11% softirqs.CPU20.SCHED
9098334 +11.3% 10127293 ? 3% softirqs.CPU24.NET_RX
48146 ? 4% -17.4% 39762 ? 7% softirqs.CPU30.SCHED
48439 ? 10% -20.5% 38522 ? 5% softirqs.CPU32.SCHED
45724 ? 3% -14.9% 38921 ? 4% softirqs.CPU34.SCHED
46626 ? 10% -17.6% 38405 ? 8% softirqs.CPU4.SCHED
21617 ? 6% -12.4% 18942 ? 8% softirqs.CPU49.TIMER
48749 ? 13% -22.1% 37979 ? 10% softirqs.CPU64.SCHED
9584889 ? 3% +9.1% 10452366 ? 3% softirqs.CPU66.NET_RX
17819 ? 14% -26.5% 13097 ? 13% softirqs.CPU66.TIMER
7481 ? 3% +19.4% 8936 ? 12% softirqs.CPU67.RCU
45843 ? 6% -16.9% 38115 ? 13% softirqs.CPU7.SCHED
56797 ? 5% -19.6% 45674 ? 8% softirqs.CPU88.SCHED
14629 ? 8% +20.6% 17647 ? 10% softirqs.CPU88.TIMER
6.79 -68.3% 2.15 ? 3% perf-stat.i.MPKI
5.079e+10 +12.9% 5.733e+10 perf-stat.i.branch-instructions
0.94 -0.1 0.87 perf-stat.i.branch-miss-rate%
4.755e+08 +4.0% 4.944e+08 perf-stat.i.branch-misses
24483752 ? 20% -53.1% 11479375 ? 16% perf-stat.i.cache-misses
1.778e+09 -64.6% 6.299e+08 ? 3% perf-stat.i.cache-references
4765951 +2.5% 4884855 perf-stat.i.context-switches
1.14 -12.8% 0.99 perf-stat.i.cpi
2.978e+11 -1.7% 2.928e+11 perf-stat.i.cpu-cycles
18360 ? 10% -81.0% 3488 ? 4% perf-stat.i.cpu-migrations
14283 ? 20% +126.2% 32316 ? 18% perf-stat.i.cycles-between-cache-misses
0.01 ? 3% -0.0 0.01 ? 5% perf-stat.i.dTLB-load-miss-rate%
8.033e+10 +12.9% 9.068e+10 perf-stat.i.dTLB-loads
0.00 ? 6% -0.0 0.00 ? 3% perf-stat.i.dTLB-store-miss-rate%
675078 ? 7% -60.8% 264803 ? 4% perf-stat.i.dTLB-store-misses
4.317e+10 +12.8% 4.871e+10 perf-stat.i.dTLB-stores
2.621e+11 +12.9% 2.959e+11 perf-stat.i.instructions
0.88 +14.8% 1.01 perf-stat.i.ipc
2.33 -1.7% 2.29 perf-stat.i.metric.GHz
112.86 ? 10% -18.6% 91.85 ? 9% perf-stat.i.metric.K/sec
1375 +12.1% 1541 perf-stat.i.metric.M/sec
6373 +4.7% 6671 perf-stat.i.minor-faults
1946110 ? 9% -44.8% 1075212 ? 5% perf-stat.i.node-loads
53.51 ? 3% +3.4 56.96 perf-stat.i.node-store-miss-rate%
1904568 ? 14% -28.7% 1358474 ? 11% perf-stat.i.node-stores
6374 +4.7% 6672 perf-stat.i.page-faults
6.79 -68.7% 2.13 ? 3% perf-stat.overall.MPKI
0.94 -0.1 0.86 perf-stat.overall.branch-miss-rate%
1.14 -12.9% 0.99 perf-stat.overall.cpi
12621 ? 17% +107.4% 26178 ? 15% perf-stat.overall.cycles-between-cache-misses
0.01 ? 3% -0.0 0.01 ? 5% perf-stat.overall.dTLB-load-miss-rate%
0.00 ? 7% -0.0 0.00 ? 4% perf-stat.overall.dTLB-store-miss-rate%
0.88 +14.9% 1.01 perf-stat.overall.ipc
9704 -3.3% 9381 perf-stat.overall.path-length
5.062e+10 +12.9% 5.714e+10 perf-stat.ps.branch-instructions
4.739e+08 +4.0% 4.928e+08 perf-stat.ps.branch-misses
24402201 ? 20% -53.1% 11437642 ? 16% perf-stat.ps.cache-misses
1.772e+09 -64.6% 6.274e+08 ? 3% perf-stat.ps.cache-references
4749921 +2.5% 4869190 perf-stat.ps.context-switches
2.968e+11 -1.7% 2.918e+11 perf-stat.ps.cpu-cycles
18325 ? 10% -81.0% 3475 ? 4% perf-stat.ps.cpu-migrations
8.006e+10 +12.9% 9.038e+10 perf-stat.ps.dTLB-loads
673376 ? 7% -60.8% 263708 ? 4% perf-stat.ps.dTLB-store-misses
4.302e+10 +12.8% 4.855e+10 perf-stat.ps.dTLB-stores
2.612e+11 +12.9% 2.949e+11 perf-stat.ps.instructions
6351 +4.7% 6648 perf-stat.ps.minor-faults
1937377 ? 9% -44.7% 1070713 ? 5% perf-stat.ps.node-loads
1898897 ? 14% -28.7% 1354269 ? 11% perf-stat.ps.node-stores
6352 +4.7% 6649 perf-stat.ps.page-faults
7.887e+13 +12.9% 8.906e+13 perf-stat.total.instructions
26.70 -4.0 22.72 perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage
27.16 -3.9 23.26 perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe
27.49 -3.8 23.67 perf-profile.calltrace.cycles-pp.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe
30.05 -3.8 26.25 perf-profile.calltrace.cycles-pp.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct
27.62 -3.8 23.83 perf-profile.calltrace.cycles-pp.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor
27.94 -3.7 24.22 perf-profile.calltrace.cycles-pp.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor
30.64 -3.7 26.97 perf-profile.calltrace.cycles-pp.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile
30.81 -3.6 27.19 perf-profile.calltrace.cycles-pp.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
43.22 -3.1 40.08 perf-profile.calltrace.cycles-pp.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
14.23 -2.9 11.34 perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
44.12 -2.8 41.27 perf-profile.calltrace.cycles-pp.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
14.42 -2.8 11.58 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
14.49 -2.8 11.70 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
14.56 -2.7 11.83 perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll
15.17 -2.6 12.52 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
16.21 -2.5 13.71 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
15.64 -2.5 13.15 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
6.84 ? 2% -2.5 4.38 ? 5% perf-profile.calltrace.cycles-pp.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
16.21 -2.4 13.79 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
16.51 -2.4 14.11 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
17.35 -2.3 15.02 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
5.57 ? 2% -2.3 3.28 ? 6% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
17.54 -2.3 15.25 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
17.64 -2.3 15.37 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
10.52 -2.0 8.51 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
10.60 -2.0 8.61 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
47.55 -2.0 45.59 perf-profile.calltrace.cycles-pp.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendfile
6.82 -1.9 4.95 ? 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage
5.98 -1.7 4.24 ? 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage
7.50 -1.7 5.81 ? 2% perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
5.98 ? 3% -1.6 4.41 ? 3% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv
3.93 ? 2% -1.5 2.40 ? 6% perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
3.91 ? 3% -1.5 2.38 ? 6% perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
49.83 -1.5 48.36 perf-profile.calltrace.cycles-pp.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendfile.sendfile_tcp_stream
3.60 ? 2% -1.3 2.27 ? 6% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage.inet_sendpage
3.57 ? 2% -1.3 2.25 ? 6% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage
14.18 -1.3 12.90 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
4.70 ? 2% -1.3 3.44 ? 3% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established
13.84 -1.2 12.66 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg
13.56 -1.2 12.39 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked
6.16 -1.2 5.00 ? 2% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
6.61 -1.1 5.47 ? 2% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
13.89 -1.0 12.86 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
3.99 -1.0 2.96 ? 2% perf-profile.calltrace.cycles-pp.filemap_get_read_batch.filemap_get_pages.filemap_read.generic_file_splice_read.splice_direct_to_actor
50.36 -1.0 49.39 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendfile.sendfile_tcp_stream.main
2.73 ? 3% -0.9 1.81 ? 2% perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
2.53 ? 2% -0.9 1.67 ? 4% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
2.90 ? 3% -0.9 2.05 ? 2% perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
4.31 -0.8 3.49 ? 2% perf-profile.calltrace.cycles-pp.filemap_get_pages.filemap_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
2.36 ? 3% -0.8 1.54 ? 4% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
2.42 ? 2% -0.8 1.62 ? 4% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
2.17 ? 3% -0.7 1.46 ? 4% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
2.99 ? 4% -0.4 2.55 perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
2.40 ? 4% -0.4 1.97 perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
1.38 ? 4% -0.3 1.04 ? 2% perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
1.40 ? 4% -0.3 1.08 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
8.53 -0.3 8.26 perf-profile.calltrace.cycles-pp.filemap_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile
1.22 -0.3 0.96 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
1.28 -0.2 1.04 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.91 -0.2 0.72 ? 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
1.53 ? 2% -0.2 1.38 ? 3% perf-profile.calltrace.cycles-pp.tcp_current_mss.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
0.82 ? 4% -0.1 0.70 perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
0.93 ? 4% -0.1 0.83 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woken
1.31 ? 2% -0.1 1.21 perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
1.86 ? 2% -0.1 1.76 ? 2% perf-profile.calltrace.cycles-pp.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
0.64 ? 5% +0.1 0.71 perf-profile.calltrace.cycles-pp.__tcp_send_ack.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
0.59 ? 5% +0.1 0.66 perf-profile.calltrace.cycles-pp.__alloc_skb.__tcp_send_ack.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
0.60 +0.1 0.69 perf-profile.calltrace.cycles-pp.ktime_get.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked
0.56 +0.1 0.66 ? 2% perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
2.36 +0.1 2.47 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages
0.53 ? 2% +0.1 0.65 perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.splice_direct_to_actor.do_splice_direct.do_sendfile
0.62 ? 2% +0.1 0.74 perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile
2.62 +0.1 2.74 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked
0.65 ? 2% +0.1 0.78 perf-profile.calltrace.cycles-pp.touch_atime.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
0.44 ? 44% +0.1 0.57 perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit
6.18 +0.1 6.31 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
2.32 ? 5% +0.1 2.46 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
0.53 ? 2% +0.1 0.67 perf-profile.calltrace.cycles-pp.touch_atime.filemap_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
6.21 +0.1 6.35 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.56 +0.1 0.70 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string._copy_from_user.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.42 ? 5% +0.1 2.57 perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
0.55 ? 2% +0.2 0.70 perf-profile.calltrace.cycles-pp.touch_atime.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile
2.15 ? 5% +0.2 2.30 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
0.56 ? 2% +0.2 0.72 perf-profile.calltrace.cycles-pp.__fget_light.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.68 ? 2% +0.2 0.85 ? 3% perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64
0.57 ? 2% +0.2 0.75 perf-profile.calltrace.cycles-pp.netperf_sendfile.sendfile_tcp_stream.main.__libc_start_main
2.19 ? 5% +0.2 2.37 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
0.60 ? 2% +0.2 0.80 ? 2% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.96 +0.2 1.18 perf-profile.calltrace.cycles-pp._copy_from_user.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendfile
1.07 ? 2% +0.2 1.31 perf-profile.calltrace.cycles-pp.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
1.06 ? 2% +0.3 1.31 perf-profile.calltrace.cycles-pp.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.44 +0.3 4.71 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
1.55 +0.3 1.83 perf-profile.calltrace.cycles-pp.copy_page_to_iter.filemap_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
0.26 ?100% +0.3 0.55 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg
4.55 +0.3 4.86 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
0.08 ?223% +0.5 0.58 ? 2% perf-profile.calltrace.cycles-pp.__fsnotify_parent.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.6 0.57 perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.filemap_read.generic_file_splice_read.splice_direct_to_actor
0.00 +0.6 0.58 perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
0.00 +0.6 0.62 perf-profile.calltrace.cycles-pp.__might_fault._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
0.17 ?141% +0.6 0.81 ? 6% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.wait_woken
0.64 +0.7 1.34 perf-profile.calltrace.cycles-pp.__entry_text_start.sendfile.sendfile_tcp_stream.main.__libc_start_main
10.96 ? 2% +0.8 11.72 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
0.00 +0.8 0.78 ? 6% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
11.09 ? 2% +0.8 11.88 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
5.21 ? 2% +1.0 6.17 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
5.42 ? 2% +1.0 6.44 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
6.64 ? 2% +1.4 8.06 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
27.10 -4.1 23.03 perf-profile.children.cycles-pp.tcp_sendpage
27.52 -4.0 23.53 perf-profile.children.cycles-pp.inet_sendpage
27.83 -3.9 23.91 perf-profile.children.cycles-pp.kernel_sendpage
27.96 -3.9 24.08 perf-profile.children.cycles-pp.sock_sendpage
28.28 -3.8 24.47 perf-profile.children.cycles-pp.pipe_to_sendpage
30.41 -3.8 26.61 perf-profile.children.cycles-pp.__splice_from_pipe
30.94 -3.7 27.25 perf-profile.children.cycles-pp.splice_from_pipe
31.08 -3.6 27.43 perf-profile.children.cycles-pp.direct_splice_actor
14.95 -3.4 11.59 perf-profile.children.cycles-pp.tcp_v4_do_rcv
14.76 -3.3 11.44 perf-profile.children.cycles-pp.tcp_rcv_established
43.27 -3.0 40.24 perf-profile.children.cycles-pp.splice_direct_to_actor
14.36 -2.9 11.44 perf-profile.children.cycles-pp.tcp_v4_rcv
14.53 -2.9 11.65 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
14.59 -2.8 11.75 perf-profile.children.cycles-pp.ip_local_deliver_finish
44.17 -2.8 41.34 perf-profile.children.cycles-pp.do_splice_direct
14.66 -2.8 11.89 perf-profile.children.cycles-pp.ip_local_deliver
15.28 -2.7 12.60 perf-profile.children.cycles-pp.ip_rcv
22.76 -2.6 20.18 perf-profile.children.cycles-pp.__tcp_transmit_skb
16.32 -2.5 13.78 perf-profile.children.cycles-pp.process_backlog
15.75 -2.5 13.22 perf-profile.children.cycles-pp.__netif_receive_skb_one_core
16.38 -2.5 13.86 perf-profile.children.cycles-pp.__napi_poll
21.43 -2.5 18.93 perf-profile.children.cycles-pp.__ip_queue_xmit
7.98 -2.4 5.53 ? 3% perf-profile.children.cycles-pp.release_sock
16.63 -2.4 14.19 perf-profile.children.cycles-pp.net_rx_action
20.62 -2.4 18.22 perf-profile.children.cycles-pp.ip_output
19.87 -2.4 17.47 perf-profile.children.cycles-pp.ip_finish_output2
17.52 -2.4 15.14 perf-profile.children.cycles-pp.__softirqentry_text_start
17.66 -2.3 15.33 perf-profile.children.cycles-pp.do_softirq
6.42 -2.3 4.12 ? 4% perf-profile.children.cycles-pp.__release_sock
18.20 -2.2 15.98 perf-profile.children.cycles-pp.__local_bh_enable_ip
8.26 -2.1 6.16 perf-profile.children.cycles-pp._raw_spin_lock_bh
47.68 -1.9 45.75 perf-profile.children.cycles-pp.do_sendfile
6.26 -1.9 4.36 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
8.13 -1.7 6.42 perf-profile.children.cycles-pp.lock_sock_nested
11.97 -1.5 10.44 perf-profile.children.cycles-pp.tcp_write_xmit
12.07 -1.5 10.56 perf-profile.children.cycles-pp.__tcp_push_pending_frames
49.97 -1.4 48.53 perf-profile.children.cycles-pp.__x64_sys_sendfile64
85.97 -1.3 84.62 perf-profile.children.cycles-pp.do_syscall_64
86.63 -1.2 85.46 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
4.06 -1.0 3.02 perf-profile.children.cycles-pp.filemap_get_read_batch
2.89 ? 2% -1.0 1.93 ? 2% perf-profile.children.cycles-pp.__check_object_size
3.98 -0.9 3.07 perf-profile.children.cycles-pp.tcp_ack
3.02 ? 2% -0.9 2.12 ? 2% perf-profile.children.cycles-pp.simple_copy_to_iter
3.24 -0.9 2.36 perf-profile.children.cycles-pp.tcp_clean_rtx_queue
2.62 ? 2% -0.9 1.75 ? 3% perf-profile.children.cycles-pp.sock_def_readable
4.38 -0.8 3.57 perf-profile.children.cycles-pp.filemap_get_pages
2.46 ? 2% -0.8 1.64 ? 2% perf-profile.children.cycles-pp.__wake_up_common_lock
2.38 ? 2% -0.8 1.57 ? 3% perf-profile.children.cycles-pp.__wake_up_common
2.58 -0.7 1.84 ? 3% perf-profile.children.cycles-pp.__kfree_skb
2.21 ? 2% -0.7 1.49 ? 3% perf-profile.children.cycles-pp.try_to_wake_up
2.14 -0.6 1.51 ? 2% perf-profile.children.cycles-pp.skb_release_data
0.84 ? 4% -0.3 0.53 ? 8% perf-profile.children.cycles-pp.ipv4_mtu
0.60 ? 3% -0.3 0.30 ? 12% perf-profile.children.cycles-pp.dst_release
0.92 -0.3 0.64 ? 2% perf-profile.children.cycles-pp.__inet_lookup_established
1.29 -0.2 1.05 perf-profile.children.cycles-pp.schedule_idle
0.49 -0.2 0.26 ? 2% perf-profile.children.cycles-pp.__ksize
0.64 -0.2 0.42 perf-profile.children.cycles-pp.__slab_free
0.40 -0.2 0.20 ? 6% perf-profile.children.cycles-pp.sock_put
8.75 -0.2 8.55 perf-profile.children.cycles-pp.filemap_read
0.42 ? 3% -0.2 0.24 ? 7% perf-profile.children.cycles-pp.tcp_add_backlog
0.49 -0.2 0.31 ? 2% perf-profile.children.cycles-pp.page_cache_pipe_buf_confirm
0.45 ? 2% -0.2 0.28 ? 3% perf-profile.children.cycles-pp.tcp_rate_check_app_limited
0.86 ? 3% -0.2 0.70 ? 2% perf-profile.children.cycles-pp.enqueue_task_fair
0.87 ? 4% -0.2 0.72 ? 2% perf-profile.children.cycles-pp.ttwu_do_activate
0.60 -0.2 0.44 ? 4% perf-profile.children.cycles-pp.enqueue_entity
0.23 ? 5% -0.2 0.08 ? 8% perf-profile.children.cycles-pp.select_idle_sibling
0.29 ? 3% -0.2 0.14 ? 5% perf-profile.children.cycles-pp.select_task_rq_fair
0.41 ? 2% -0.2 0.26 ? 3% perf-profile.children.cycles-pp.update_rq_clock
0.24 -0.1 0.09 ? 7% perf-profile.children.cycles-pp.skb_clone
0.40 ? 3% -0.1 0.26 ? 8% perf-profile.children.cycles-pp.skb_release_head_state
1.74 ? 2% -0.1 1.61 ? 2% perf-profile.children.cycles-pp.tcp_current_mss
0.44 ? 3% -0.1 0.31 ? 6% perf-profile.children.cycles-pp.skb_release_all
0.48 -0.1 0.34 ? 2% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.86 -0.1 0.73 perf-profile.children.cycles-pp.dequeue_entity
0.96 -0.1 0.84 perf-profile.children.cycles-pp.dequeue_task_fair
0.23 ? 2% -0.1 0.11 ? 4% perf-profile.children.cycles-pp.irqentry_enter
0.48 -0.1 0.36 ? 2% perf-profile.children.cycles-pp.sysvec_reschedule_ipi
2.52 -0.1 2.42 perf-profile.children.cycles-pp.__dev_queue_xmit
0.42 -0.1 0.32 ? 3% perf-profile.children.cycles-pp.set_next_entity
0.31 ? 3% -0.1 0.22 ? 7% perf-profile.children.cycles-pp.ipv4_dst_check
1.33 -0.1 1.24 perf-profile.children.cycles-pp.tcp_stream_alloc_skb
0.14 ? 3% -0.1 0.05 ? 8% perf-profile.children.cycles-pp.update_ts_time_stats
0.14 ? 3% -0.1 0.06 ? 9% perf-profile.children.cycles-pp.nr_iowait_cpu
0.24 ? 2% -0.1 0.16 ? 4% perf-profile.children.cycles-pp.tcp_mtu_probe
0.22 ? 2% -0.1 0.14 ? 3% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.17 ? 2% -0.1 0.09 ? 7% perf-profile.children.cycles-pp.remove_wait_queue
3.48 -0.1 3.40 perf-profile.children.cycles-pp.__schedule
1.96 ? 2% -0.1 1.88 perf-profile.children.cycles-pp.tcp_send_mss
0.58 ? 2% -0.1 0.51 perf-profile.children.cycles-pp.sk_reset_timer
0.54 ? 2% -0.1 0.47 perf-profile.children.cycles-pp.mod_timer
0.62 -0.1 0.55 perf-profile.children.cycles-pp.update_load_avg
0.92 -0.1 0.86 perf-profile.children.cycles-pp._raw_spin_lock
0.32 -0.1 0.25 ? 2% perf-profile.children.cycles-pp.__skb_clone
0.29 ? 2% -0.1 0.22 ? 4% perf-profile.children.cycles-pp.__sk_dst_check
0.43 -0.1 0.36 ? 2% perf-profile.children.cycles-pp.update_curr
0.23 ? 5% -0.1 0.16 ? 4% perf-profile.children.cycles-pp.tcp_eat_recv_skb
0.14 ? 6% -0.1 0.08 ? 6% perf-profile.children.cycles-pp.perf_tp_event
0.14 ? 3% -0.1 0.08 ? 6% perf-profile.children.cycles-pp.pick_next_entity
0.27 -0.1 0.21 ? 2% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.32 -0.1 0.26 perf-profile.children.cycles-pp.kmem_cache_free
0.19 ? 3% -0.1 0.14 ? 5% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
0.13 ? 3% -0.1 0.07 ? 6% perf-profile.children.cycles-pp.cpuacct_charge
0.22 ? 4% -0.1 0.17 ? 8% perf-profile.children.cycles-pp.ip_rcv_finish_core
0.20 ? 2% -0.0 0.15 ? 4% perf-profile.children.cycles-pp.migrate_disable
0.17 ? 2% -0.0 0.13 ? 2% perf-profile.children.cycles-pp.__list_del_entry_valid
0.18 ? 2% -0.0 0.14 ? 3% perf-profile.children.cycles-pp.tcp_ack_update_rtt
0.14 ? 2% -0.0 0.10 ? 8% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.14 ? 3% -0.0 0.10 ? 6% perf-profile.children.cycles-pp.tcp_update_pacing_rate
0.20 -0.0 0.17 ? 13% perf-profile.children.cycles-pp.tick_sched_timer
0.31 ? 3% -0.0 0.28 ? 5% perf-profile.children.cycles-pp.ip_rcv_finish
0.27 ? 3% -0.0 0.24 ? 2% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.19 ? 2% -0.0 0.16 ? 13% perf-profile.children.cycles-pp.tick_sched_handle
0.10 ? 3% -0.0 0.07 ? 5% perf-profile.children.cycles-pp.__list_add_valid
0.18 ? 2% -0.0 0.16 ? 13% perf-profile.children.cycles-pp.update_process_times
0.25 ? 2% -0.0 0.23 ? 2% perf-profile.children.cycles-pp.check_preempt_curr
0.06 -0.0 0.03 ? 70% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.14 ? 3% -0.0 0.12 ? 3% perf-profile.children.cycles-pp.tcp_chrono_stop
0.24 ? 2% -0.0 0.22 ? 5% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.09 -0.0 0.07 ? 7% perf-profile.children.cycles-pp.tcp_rate_skb_delivered
0.11 ? 7% -0.0 0.09 ? 5% perf-profile.children.cycles-pp.rcu_dynticks_inc
0.16 ? 2% -0.0 0.14 ? 3% perf-profile.children.cycles-pp.__tcp_select_window
0.14 ? 2% -0.0 0.12 perf-profile.children.cycles-pp.tcp_queue_rcv
0.15 -0.0 0.13 ? 2% perf-profile.children.cycles-pp.tcp_update_skb_after_send
0.07 ? 5% -0.0 0.05 perf-profile.children.cycles-pp.rcu_nmi_enter
0.55 -0.0 0.53 perf-profile.children.cycles-pp.tcp_event_new_data_sent
0.27 ? 2% -0.0 0.25 ? 2% perf-profile.children.cycles-pp.tcp_rcv_space_adjust
0.13 ? 2% -0.0 0.11 ? 3% perf-profile.children.cycles-pp.lock_timer_base
0.08 ? 5% -0.0 0.06 ? 7% perf-profile.children.cycles-pp.rb_first
0.15 ? 2% -0.0 0.13 perf-profile.children.cycles-pp.tcp_skb_entail
0.12 ? 3% -0.0 0.10 ? 5% perf-profile.children.cycles-pp.scheduler_tick
0.18 ? 2% -0.0 0.17 ? 2% perf-profile.children.cycles-pp.ip_send_check
0.30 -0.0 0.28 perf-profile.children.cycles-pp.kfree
0.16 -0.0 0.14 ? 3% perf-profile.children.cycles-pp.tcp_wfree
0.07 -0.0 0.06 perf-profile.children.cycles-pp.__sock_wfree
0.09 ? 5% +0.0 0.10 ? 4% perf-profile.children.cycles-pp.tcp_v4_fill_cb
0.08 ? 4% +0.0 0.09 perf-profile.children.cycles-pp.iov_iter_pipe
0.17 ? 2% +0.0 0.18 ? 2% perf-profile.children.cycles-pp.__ip_finish_output
0.05 ? 7% +0.0 0.06 ? 7% perf-profile.children.cycles-pp.__build_skb_around
0.05 ? 7% +0.0 0.06 ? 7% perf-profile.children.cycles-pp.tcp_options_write
0.06 +0.0 0.07 ? 6% perf-profile.children.cycles-pp.skb_push
0.05 +0.0 0.07 ? 7% perf-profile.children.cycles-pp.apparmor_socket_sock_rcv_skb
0.09 ? 5% +0.0 0.11 perf-profile.children.cycles-pp.tcp_v4_inbound_md5_hash
0.08 +0.0 0.10 ? 4% perf-profile.children.cycles-pp.__fdget
0.14 ? 3% +0.0 0.16 ? 2% perf-profile.children.cycles-pp.__update_load_avg_se
0.08 ? 4% +0.0 0.10 ? 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.08 ? 4% +0.0 0.10 perf-profile.children.cycles-pp.__wrgsbase_inactive
0.28 +0.0 0.30 ? 2% perf-profile.children.cycles-pp.native_irq_return_iret
0.10 ? 4% +0.0 0.12 ? 3% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.13 ? 3% +0.0 0.15 perf-profile.children.cycles-pp.inet_ehashfn
0.06 ? 7% +0.0 0.09 ? 4% perf-profile.children.cycles-pp.skb_network_protocol
0.12 +0.0 0.14 ? 3% perf-profile.children.cycles-pp.aa_sk_perm
0.11 ? 4% +0.0 0.14 ? 2% perf-profile.children.cycles-pp.migrate_enable
0.11 +0.0 0.14 ? 3% perf-profile.children.cycles-pp.generic_file_read_iter
0.13 ? 3% +0.0 0.15 ? 3% perf-profile.children.cycles-pp.xas_start
0.29 ? 2% +0.0 0.32 ? 3% perf-profile.children.cycles-pp.recv_tcp_stream
0.12 ? 3% +0.0 0.15 ? 4% perf-profile.children.cycles-pp.splice_from_pipe_next
0.17 +0.0 0.20 perf-profile.children.cycles-pp.tcp_tx_timestamp
0.10 ? 3% +0.0 0.13 ? 2% perf-profile.children.cycles-pp.security_sock_rcv_skb
0.10 ? 3% +0.0 0.13 ? 3% perf-profile.children.cycles-pp.demo_interval_tick
0.15 ? 3% +0.0 0.19 perf-profile.children.cycles-pp.tcp_release_cb
0.15 +0.0 0.19 ? 2% perf-profile.children.cycles-pp.tcp_push
0.16 ? 3% +0.0 0.19 ? 2% perf-profile.children.cycles-pp.apparmor_file_permission
0.13 ? 6% +0.0 0.17 ? 5% perf-profile.children.cycles-pp.update_sd_lb_stats
0.03 ? 70% +0.0 0.07 ? 5% perf-profile.children.cycles-pp.bpf_skops_write_hdr_opt
0.16 ? 3% +0.0 0.20 ? 2% perf-profile.children.cycles-pp.security_socket_recvmsg
0.20 ? 2% +0.0 0.24 ? 2% perf-profile.children.cycles-pp._raw_spin_unlock_bh
0.02 ?141% +0.0 0.06 ? 6% perf-profile.children.cycles-pp.update_irq_load_avg
0.20 ? 2% +0.0 0.25 ? 3% perf-profile.children.cycles-pp.sockfd_lookup_light
0.18 ? 2% +0.0 0.22 ? 3% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
1.08 ? 2% +0.0 1.13 perf-profile.children.cycles-pp.loopback_xmit
0.18 ? 2% +0.0 0.23 ? 2% perf-profile.children.cycles-pp.sock_recvmsg
0.10 ? 4% +0.0 0.14 ? 3% perf-profile.children.cycles-pp.save_fpregs_to_fpstate
0.16 ? 2% +0.0 0.21 ? 3% perf-profile.children.cycles-pp.__switch_to_asm
0.15 ? 2% +0.0 0.20 ? 2% perf-profile.children.cycles-pp.netif_skb_features
0.14 ? 7% +0.0 0.19 ? 7% perf-profile.children.cycles-pp.find_busiest_group
0.47 +0.0 0.52 ? 2% perf-profile.children.cycles-pp.tcp_mstamp_refresh
0.13 ? 2% +0.0 0.18 ? 3% perf-profile.children.cycles-pp.check_stack_object
0.00 +0.1 0.05 perf-profile.children.cycles-pp.rb_erase
0.00 +0.1 0.05 perf-profile.children.cycles-pp.tcp_newly_delivered
0.00 +0.1 0.05 perf-profile.children.cycles-pp.rb_insert_color
0.00 +0.1 0.05 perf-profile.children.cycles-pp.pipe_lock
0.22 +0.1 0.27 ? 2% perf-profile.children.cycles-pp.demo_stream_interval
0.67 +0.1 0.72 perf-profile.children.cycles-pp.__tcp_send_ack
0.25 +0.1 0.30 perf-profile.children.cycles-pp.ip_rcv_core
0.23 ? 3% +0.1 0.28 ? 2% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
0.07 ? 5% +0.1 0.13 ? 5% perf-profile.children.cycles-pp._find_next_bit
0.50 +0.1 0.56 perf-profile.children.cycles-pp.tcp_established_options
0.29 ? 2% +0.1 0.35 perf-profile.children.cycles-pp.rw_verify_area
0.18 ? 7% +0.1 0.24 ? 6% perf-profile.children.cycles-pp.load_balance
0.00 +0.1 0.06 ? 11% perf-profile.children.cycles-pp.sched_ttwu_pending
0.25 ? 3% +0.1 0.32 ? 2% perf-profile.children.cycles-pp.xas_load
0.09 ? 11% +0.1 0.16 ? 7% perf-profile.children.cycles-pp.update_blocked_averages
1.22 ? 2% +0.1 1.28 perf-profile.children.cycles-pp.dev_hard_start_xmit
0.25 ? 3% +0.1 0.32 ? 3% perf-profile.children.cycles-pp.__switch_to
0.00 +0.1 0.07 ? 5% perf-profile.children.cycles-pp.should_failslab
0.00 +0.1 0.07 perf-profile.children.cycles-pp.skb_clone_tx_timestamp
0.39 +0.1 0.46 ? 2% perf-profile.children.cycles-pp.native_sched_clock
0.22 ? 2% +0.1 0.30 ? 2% perf-profile.children.cycles-pp.sanity
0.00 +0.1 0.08 ? 10% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.33 ? 2% +0.1 0.41 perf-profile.children.cycles-pp.timestamp_truncate
0.31 +0.1 0.40 perf-profile.children.cycles-pp.rcu_all_qs
0.06 ? 45% +0.1 0.14 ? 12% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
0.16 ? 2% +0.1 0.25 perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
0.46 +0.1 0.54 perf-profile.children.cycles-pp.__kmalloc_node_track_caller
0.54 +0.1 0.62 perf-profile.children.cycles-pp.sk_filter_trim_cap
0.35 +0.1 0.44 perf-profile.children.cycles-pp.validate_xmit_skb
0.26 ? 2% +0.1 0.35 ? 4% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
0.47 +0.1 0.56 perf-profile.children.cycles-pp.sched_clock_cpu
2.38 +0.1 2.48 perf-profile.children.cycles-pp.schedule_timeout
0.51 ? 2% +0.1 0.60 ? 2% perf-profile.children.cycles-pp.kmalloc_reserve
0.00 +0.1 0.10 ? 11% perf-profile.children.cycles-pp.sysvec_call_function_single
0.38 ? 2% +0.1 0.47 ? 2% perf-profile.children.cycles-pp.__netif_receive_skb_core
2.48 +0.1 2.59 perf-profile.children.cycles-pp.wait_woken
0.46 +0.1 0.57 perf-profile.children.cycles-pp.__put_user_nocheck_8
0.09 ? 5% +0.1 0.20 ? 4% perf-profile.children.cycles-pp.__next_timer_interrupt
0.18 ? 2% +0.1 0.32 ? 2% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.84 +0.1 0.98 perf-profile.children.cycles-pp.read_tsc
0.38 ? 2% +0.1 0.51 ? 2% perf-profile.children.cycles-pp.kmem_cache_alloc_node
1.14 ? 3% +0.1 1.27 ? 2% perf-profile.children.cycles-pp.pick_next_task_fair
0.25 ? 3% +0.1 0.39 perf-profile.children.cycles-pp.tick_nohz_next_event
6.23 +0.1 6.38 perf-profile.children.cycles-pp.cpuidle_enter_state
0.57 +0.2 0.72 perf-profile.children.cycles-pp.aa_file_perm
6.26 +0.2 6.42 perf-profile.children.cycles-pp.cpuidle_enter
0.66 +0.2 0.82 perf-profile.children.cycles-pp.__cond_resched
0.40 ? 2% +0.2 0.55 perf-profile.children.cycles-pp.__virt_addr_valid
1.40 +0.2 1.56 ? 2% perf-profile.children.cycles-pp.ktime_get
0.36 ? 3% +0.2 0.52 ? 2% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.77 +0.2 0.94 perf-profile.children.cycles-pp.__fsnotify_parent
0.82 +0.2 1.00 perf-profile.children.cycles-pp.__entry_text_start
0.76 +0.2 0.96 perf-profile.children.cycles-pp.__fget_light
2.25 +0.2 2.45 perf-profile.children.cycles-pp.schedule
0.67 ? 2% +0.2 0.87 perf-profile.children.cycles-pp.netperf_sendfile
0.61 ? 2% +0.2 0.82 ? 2% perf-profile.children.cycles-pp.menu_select
0.84 ? 2% +0.2 1.05 perf-profile.children.cycles-pp.current_time
0.58 +0.2 0.79 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
10.87 +0.2 11.10 perf-profile.children.cycles-pp.do_tcp_sendpages
1.03 +0.2 1.26 perf-profile.children.cycles-pp._copy_from_user
1.12 +0.3 1.38 perf-profile.children.cycles-pp.do_splice_to
0.97 +0.3 1.22 perf-profile.children.cycles-pp.syscall_return_via_sysret
0.00 +0.3 0.26 ? 9% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.85 +0.3 1.13 perf-profile.children.cycles-pp.__might_sleep
1.62 +0.3 1.93 perf-profile.children.cycles-pp.copy_page_to_iter
0.47 ? 7% +0.3 0.78 ? 6% perf-profile.children.cycles-pp.newidle_balance
1.21 +0.3 1.56 perf-profile.children.cycles-pp.__might_fault
1.63 +0.4 2.00 ? 2% perf-profile.children.cycles-pp.common_file_perm
1.54 +0.4 1.92 perf-profile.children.cycles-pp.atime_needs_update
1.63 +0.4 2.01 perf-profile.children.cycles-pp.__might_resched
1.86 +0.5 2.32 perf-profile.children.cycles-pp.touch_atime
2.35 +0.5 2.88 perf-profile.children.cycles-pp.security_file_permission
11.05 ? 2% +0.8 11.84 perf-profile.children.cycles-pp.__skb_datagram_iter
11.09 ? 2% +0.8 11.89 perf-profile.children.cycles-pp.skb_copy_datagram_iter
5.48 ? 2% +1.0 6.53 perf-profile.children.cycles-pp.copyout
6.00 ? 2% +1.2 7.17 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
6.75 ? 2% +1.5 8.22 perf-profile.children.cycles-pp._copy_to_iter
6.21 -1.9 4.33 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
2.38 ? 2% -1.1 1.23 ? 4% perf-profile.self.cycles-pp.__check_object_size
3.79 -1.1 2.70 perf-profile.self.cycles-pp.filemap_get_read_batch
1.40 -0.4 0.98 ? 2% perf-profile.self.cycles-pp.skb_release_data
0.81 ? 4% -0.3 0.49 ? 8% perf-profile.self.cycles-pp.ipv4_mtu
0.73 -0.3 0.42 ? 4% perf-profile.self.cycles-pp.tcp_rcv_established
0.58 ? 3% -0.3 0.28 ? 13% perf-profile.self.cycles-pp.dst_release
0.79 -0.3 0.50 ? 2% perf-profile.self.cycles-pp.__inet_lookup_established
0.84 ? 2% -0.3 0.58 ? 4% perf-profile.self.cycles-pp.__dev_queue_xmit
2.00 -0.3 1.74 perf-profile.self.cycles-pp._raw_spin_lock_bh
0.48 -0.2 0.25 ? 2% perf-profile.self.cycles-pp.__ksize
0.63 -0.2 0.41 perf-profile.self.cycles-pp.__slab_free
0.40 -0.2 0.18 ? 8% perf-profile.self.cycles-pp.sock_put
0.94 -0.2 0.73 perf-profile.self.cycles-pp.tcp_v4_rcv
0.46 -0.2 0.27 ? 2% perf-profile.self.cycles-pp.page_cache_pipe_buf_confirm
0.61 -0.2 0.43 perf-profile.self.cycles-pp.tcp_clean_rtx_queue
0.30 -0.2 0.12 ? 4% perf-profile.self.cycles-pp.update_rq_clock
0.42 ? 3% -0.2 0.25 ? 4% perf-profile.self.cycles-pp.tcp_rate_check_app_limited
0.26 ? 2% -0.1 0.11 ? 6% perf-profile.self.cycles-pp.try_to_wake_up
0.26 ? 2% -0.1 0.12 ? 5% perf-profile.self.cycles-pp.set_next_entity
0.23 ? 3% -0.1 0.09 ? 7% perf-profile.self.cycles-pp.skb_clone
0.50 -0.1 0.37 ? 2% perf-profile.self.cycles-pp.tcp_sendpage_locked
0.46 -0.1 0.34 ? 2% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.18 -0.1 1.05 perf-profile.self.cycles-pp.__tcp_transmit_skb
0.27 ? 3% -0.1 0.15 ? 5% perf-profile.self.cycles-pp.enqueue_entity
0.14 -0.1 0.03 ?100% perf-profile.self.cycles-pp.nr_iowait_cpu
0.15 ? 2% -0.1 0.06 ? 9% perf-profile.self.cycles-pp.__wake_up_common
0.30 ? 3% -0.1 0.20 ? 8% perf-profile.self.cycles-pp.ipv4_dst_check
0.15 ? 3% -0.1 0.06 ? 9% perf-profile.self.cycles-pp.irqentry_enter
0.23 ? 3% -0.1 0.14 ? 4% perf-profile.self.cycles-pp.tcp_mtu_probe
0.11 ? 6% -0.1 0.03 ? 70% perf-profile.self.cycles-pp.perf_tp_event
0.16 ? 4% -0.1 0.08 ? 5% perf-profile.self.cycles-pp.tcp_stream_alloc_skb
0.13 ? 3% -0.1 0.07 ? 7% perf-profile.self.cycles-pp.pick_next_entity
0.25 -0.1 0.18 ? 2% perf-profile.self.cycles-pp.__skb_clone
0.26 -0.1 0.20 ? 2% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.15 ? 3% -0.1 0.08 ? 5% perf-profile.self.cycles-pp.__release_sock
0.14 ? 2% -0.1 0.08 ? 6% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.32 ? 2% -0.1 0.26 perf-profile.self.cycles-pp.kmem_cache_free
0.22 ? 5% -0.1 0.16 ? 9% perf-profile.self.cycles-pp.ip_rcv_finish_core
0.13 ? 3% -0.1 0.07 ? 5% perf-profile.self.cycles-pp.cpuacct_charge
0.16 ? 4% -0.1 0.10 ? 4% perf-profile.self.cycles-pp.sock_def_readable
0.13 ? 3% -0.1 0.08 ? 4% perf-profile.self.cycles-pp.tcp_eat_recv_skb
0.18 ? 4% -0.0 0.14 ? 5% perf-profile.self.cycles-pp.migrate_disable
0.14 -0.0 0.09 ? 4% perf-profile.self.cycles-pp.tcp_add_backlog
0.13 -0.0 0.08 ? 4% perf-profile.self.cycles-pp.schedule_timeout
0.16 ? 2% -0.0 0.11 ? 4% perf-profile.self.cycles-pp.tcp_ack_update_rtt
0.16 ? 3% -0.0 0.11 perf-profile.self.cycles-pp.__list_del_entry_valid
0.85 -0.0 0.80 ? 2% perf-profile.self.cycles-pp._raw_spin_lock
0.13 ? 3% -0.0 0.09 ? 5% perf-profile.self.cycles-pp.tcp_update_pacing_rate
0.25 -0.0 0.21 ? 3% perf-profile.self.cycles-pp.irqtime_account_irq
0.09 ? 5% -0.0 0.06 ? 9% perf-profile.self.cycles-pp.tcp_rcv_space_adjust
0.26 ? 3% -0.0 0.23 perf-profile.self.cycles-pp.mod_timer
0.11 ? 3% -0.0 0.08 perf-profile.self.cycles-pp.tcp_chrono_stop
0.09 ? 5% -0.0 0.06 ? 9% perf-profile.self.cycles-pp.__list_add_valid
1.27 -0.0 1.24 perf-profile.self.cycles-pp.tcp_write_xmit
0.06 -0.0 0.03 ? 70% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.40 -0.0 0.37 ? 3% perf-profile.self.cycles-pp.__schedule
0.09 ? 5% -0.0 0.06 ? 6% perf-profile.self.cycles-pp.tcp_rate_skb_delivered
0.22 ? 2% -0.0 0.20 ? 2% perf-profile.self.cycles-pp.update_load_avg
0.14 ? 3% -0.0 0.11 ? 4% perf-profile.self.cycles-pp.tcp_queue_rcv
0.10 ? 4% -0.0 0.08 ? 5% perf-profile.self.cycles-pp.rcu_dynticks_inc
0.08 ? 6% -0.0 0.05 ? 8% perf-profile.self.cycles-pp.rb_first
0.18 ? 2% -0.0 0.16 ? 3% perf-profile.self.cycles-pp.ip_send_check
0.15 ? 3% -0.0 0.13 ? 4% perf-profile.self.cycles-pp.__tcp_select_window
0.25 -0.0 0.23 perf-profile.self.cycles-pp.folio_mark_accessed
0.16 ? 2% -0.0 0.14 ? 2% perf-profile.self.cycles-pp.tcp_wfree
0.29 -0.0 0.27 perf-profile.self.cycles-pp.kfree
0.14 ? 2% -0.0 0.13 ? 3% perf-profile.self.cycles-pp.finish_task_switch
0.10 ? 5% -0.0 0.08 perf-profile.self.cycles-pp.dequeue_task_fair
0.11 ? 4% -0.0 0.10 ? 3% perf-profile.self.cycles-pp.__napi_poll
0.07 -0.0 0.06 ? 8% perf-profile.self.cycles-pp.__sock_wfree
0.12 ? 4% -0.0 0.11 ? 6% perf-profile.self.cycles-pp.kfree_skbmem
0.09 +0.0 0.10 ? 4% perf-profile.self.cycles-pp.__x64_sys_recvfrom
0.06 ? 6% +0.0 0.08 ? 6% perf-profile.self.cycles-pp.reweight_entity
0.05 ? 8% +0.0 0.07 ? 7% perf-profile.self.cycles-pp.aa_sk_perm
0.16 ? 2% +0.0 0.18 ? 2% perf-profile.self.cycles-pp.__ip_finish_output
0.06 ? 7% +0.0 0.08 ? 6% perf-profile.self.cycles-pp.skb_release_head_state
0.07 ? 5% +0.0 0.08 ? 5% perf-profile.self.cycles-pp.raw_local_deliver
0.07 ? 6% +0.0 0.09 ? 4% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.07 +0.0 0.08 ? 5% perf-profile.self.cycles-pp.asm_sysvec_reschedule_ipi
0.06 ? 7% +0.0 0.08 ? 4% perf-profile.self.cycles-pp.tcp_v4_inbound_md5_hash
0.21 ? 2% +0.0 0.22 perf-profile.self.cycles-pp.native_apic_msr_eoi_write
0.05 ? 8% +0.0 0.07 perf-profile.self.cycles-pp.security_sock_rcv_skb
0.11 ? 4% +0.0 0.13 ? 4% perf-profile.self.cycles-pp.cpuidle_enter_state
0.08 ? 4% +0.0 0.10 ? 3% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.06 +0.0 0.08 ? 6% perf-profile.self.cycles-pp.sched_clock_cpu
0.13 ? 3% +0.0 0.15 ? 2% perf-profile.self.cycles-pp.__update_load_avg_se
0.11 ? 4% +0.0 0.12 ? 4% perf-profile.self.cycles-pp.apparmor_file_permission
0.28 +0.0 0.30 ? 2% perf-profile.self.cycles-pp.native_irq_return_iret
0.13 ? 4% +0.0 0.15 perf-profile.self.cycles-pp.dev_hard_start_xmit
0.10 ? 3% +0.0 0.12 ? 3% perf-profile.self.cycles-pp.__tcp_push_pending_frames
0.11 ? 3% +0.0 0.13 ? 2% perf-profile.self.cycles-pp.inet_ehashfn
0.06 ? 6% +0.0 0.08 ? 4% perf-profile.self.cycles-pp.skb_network_protocol
0.08 ? 6% +0.0 0.10 ? 5% perf-profile.self.cycles-pp.ip_rcv_finish
0.23 ? 2% +0.0 0.25 perf-profile.self.cycles-pp.do_idle
0.22 ? 2% +0.0 0.24 ? 2% perf-profile.self.cycles-pp.recv
0.10 ? 4% +0.0 0.12 ? 3% perf-profile.self.cycles-pp.migrate_enable
0.38 +0.0 0.40 perf-profile.self.cycles-pp.__alloc_skb
0.10 ? 5% +0.0 0.12 perf-profile.self.cycles-pp.generic_file_read_iter
0.09 ? 4% +0.0 0.11 ? 4% perf-profile.self.cycles-pp.inet_recvmsg
0.20 +0.0 0.23 ? 2% perf-profile.self.cycles-pp.__sys_recvfrom
0.14 ? 2% +0.0 0.17 ? 2% perf-profile.self.cycles-pp.tcp_tx_timestamp
0.11 ? 4% +0.0 0.13 ? 3% perf-profile.self.cycles-pp.xas_start
0.12 ? 3% +0.0 0.15 perf-profile.self.cycles-pp.tcp_release_cb
0.10 ? 4% +0.0 0.13 ? 3% perf-profile.self.cycles-pp.splice_from_pipe_next
0.24 ? 3% +0.0 0.27 ? 3% perf-profile.self.cycles-pp.recv_tcp_stream
0.10 +0.0 0.13 perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
0.13 ? 4% +0.0 0.16 perf-profile.self.cycles-pp._raw_spin_unlock_bh
0.13 +0.0 0.16 ? 3% perf-profile.self.cycles-pp.tcp_push
0.09 ? 4% +0.0 0.12 ? 3% perf-profile.self.cycles-pp.netif_skb_features
0.08 +0.0 0.11 ? 5% perf-profile.self.cycles-pp.demo_interval_tick
0.12 ? 4% +0.0 0.16 ? 3% perf-profile.self.cycles-pp._copy_from_user
0.13 ? 2% +0.0 0.16 ? 2% perf-profile.self.cycles-pp.demo_stream_interval
0.14 ? 2% +0.0 0.17 ? 2% perf-profile.self.cycles-pp.sock_sendpage
0.13 ? 3% +0.0 0.16 ? 2% perf-profile.self.cycles-pp.xas_load
0.10 +0.0 0.14 ? 3% perf-profile.self.cycles-pp.check_stack_object
0.10 ? 4% +0.0 0.13 ? 2% perf-profile.self.cycles-pp.tcp_event_new_data_sent
0.10 ? 6% +0.0 0.14 ? 6% perf-profile.self.cycles-pp.update_sd_lb_stats
0.17 +0.0 0.21 ? 2% perf-profile.self.cycles-pp.validate_xmit_skb
0.16 ? 3% +0.0 0.19 perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.23 ? 2% +0.0 0.26 ? 2% perf-profile.self.cycles-pp.ip_output
0.45 ? 3% +0.0 0.48 ? 2% perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
0.18 ? 3% +0.0 0.22 ? 2% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.18 ? 2% +0.0 0.22 ? 2% perf-profile.self.cycles-pp.menu_select
0.06 ? 6% +0.0 0.10 ? 3% perf-profile.self.cycles-pp.ip_local_deliver_finish
0.12 ? 3% +0.0 0.16 ? 2% perf-profile.self.cycles-pp.direct_splice_actor
0.28 +0.0 0.33 ? 2% perf-profile.self.cycles-pp.process_backlog
0.13 ? 2% +0.0 0.17 perf-profile.self.cycles-pp.syscall_exit_to_user_mode
0.12 ? 4% +0.0 0.16 ? 2% perf-profile.self.cycles-pp.copyout
0.15 ? 3% +0.0 0.19 perf-profile.self.cycles-pp.do_softirq
0.10 ? 5% +0.0 0.14 perf-profile.self.cycles-pp.save_fpregs_to_fpstate
0.19 ? 3% +0.0 0.23 ? 2% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
0.19 ? 2% +0.0 0.23 ? 2% perf-profile.self.cycles-pp.do_splice_to
0.06 ? 8% +0.0 0.10 ? 4% perf-profile.self.cycles-pp._find_next_bit
0.21 ? 3% +0.0 0.26 perf-profile.self.cycles-pp.rw_verify_area
0.16 ? 3% +0.0 0.20 ? 3% perf-profile.self.cycles-pp.__switch_to_asm
0.00 +0.1 0.05 perf-profile.self.cycles-pp.schedule_idle
0.00 +0.1 0.05 perf-profile.self.cycles-pp.skb_push
0.00 +0.1 0.05 perf-profile.self.cycles-pp.tcp_options_write
0.00 +0.1 0.05 perf-profile.self.cycles-pp.update_irq_load_avg
0.24 +0.1 0.29 perf-profile.self.cycles-pp.ip_rcv_core
0.23 ? 2% +0.1 0.28 perf-profile.self.cycles-pp.lock_sock_nested
0.00 +0.1 0.05 ? 7% perf-profile.self.cycles-pp.schedule
0.01 ?223% +0.1 0.06 perf-profile.self.cycles-pp.pick_next_task_fair
0.00 +0.1 0.05 ? 7% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.00 +0.1 0.05 ? 8% perf-profile.self.cycles-pp.apparmor_socket_sock_rcv_skb
0.22 ? 2% +0.1 0.27 ? 2% perf-profile.self.cycles-pp.rcu_all_qs
0.25 +0.1 0.31 perf-profile.self.cycles-pp.__entry_text_start
0.00 +0.1 0.06 ? 9% perf-profile.self.cycles-pp.update_blocked_averages
0.14 ? 3% +0.1 0.19 perf-profile.self.cycles-pp.simple_copy_to_iter
0.31 ? 3% +0.1 0.37 ? 2% perf-profile.self.cycles-pp.__softirqentry_text_start
0.45 +0.1 0.51 perf-profile.self.cycles-pp.tcp_established_options
0.82 +0.1 0.88 perf-profile.self.cycles-pp.__skb_datagram_iter
0.10 ? 4% +0.1 0.16 ? 3% perf-profile.self.cycles-pp.__netif_receive_skb_one_core
0.00 +0.1 0.06 ? 6% perf-profile.self.cycles-pp.bpf_skops_write_hdr_opt
0.32 ? 2% +0.1 0.38 ? 2% perf-profile.self.cycles-pp.tcp_send_mss
0.27 +0.1 0.34 ? 2% perf-profile.self.cycles-pp.tcp_sendpage
0.09 ? 6% +0.1 0.15 ? 3% perf-profile.self.cycles-pp.ip_rcv
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.skb_clone_tx_timestamp
0.27 +0.1 0.33 ? 2% perf-profile.self.cycles-pp.__might_fault
0.07 +0.1 0.14 ? 3% perf-profile.self.cycles-pp.ip_local_deliver
0.48 +0.1 0.54 ? 2% perf-profile.self.cycles-pp.ip_finish_output2
0.24 ? 3% +0.1 0.30 ? 2% perf-profile.self.cycles-pp.__switch_to
0.38 +0.1 0.44 ? 2% perf-profile.self.cycles-pp.native_sched_clock
0.29 +0.1 0.36 ? 2% perf-profile.self.cycles-pp.__kmalloc_node_track_caller
0.29 ? 2% +0.1 0.36 perf-profile.self.cycles-pp.timestamp_truncate
0.31 ? 2% +0.1 0.38 perf-profile.self.cycles-pp.kernel_sendpage
0.19 +0.1 0.26 ? 2% perf-profile.self.cycles-pp.sanity
0.32 +0.1 0.39 perf-profile.self.cycles-pp.pipe_to_sendpage
0.21 ? 3% +0.1 0.28 perf-profile.self.cycles-pp.net_rx_action
0.14 +0.1 0.21 ? 2% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
0.04 ? 44% +0.1 0.12 ? 4% perf-profile.self.cycles-pp.__next_timer_interrupt
0.34 ? 2% +0.1 0.41 perf-profile.self.cycles-pp.inet_sendpage
0.43 ? 4% +0.1 0.51 ? 2% perf-profile.self.cycles-pp.__x64_sys_sendfile64
0.33 +0.1 0.41 perf-profile.self.cycles-pp.__cond_resched
0.27 ? 2% +0.1 0.35 ? 3% perf-profile.self.cycles-pp.do_splice_direct
0.33 ? 2% +0.1 0.41 ? 2% perf-profile.self.cycles-pp.current_time
0.34 ? 2% +0.1 0.43 ? 2% perf-profile.self.cycles-pp.touch_atime
0.37 +0.1 0.46 ? 2% perf-profile.self.cycles-pp.__netif_receive_skb_core
0.50 +0.1 0.59 perf-profile.self.cycles-pp.__local_bh_enable_ip
0.53 +0.1 0.63 perf-profile.self.cycles-pp.tcp_current_mss
0.16 ? 8% +0.1 0.26 ? 5% perf-profile.self.cycles-pp.newidle_balance
0.27 ? 2% +0.1 0.38 perf-profile.self.cycles-pp.kmem_cache_alloc_node
0.46 +0.1 0.57 perf-profile.self.cycles-pp.__put_user_nocheck_8
0.50 ? 2% +0.1 0.61 ? 2% perf-profile.self.cycles-pp.splice_from_pipe
0.42 +0.1 0.53 ? 2% perf-profile.self.cycles-pp.generic_file_splice_read
0.51 +0.1 0.62 perf-profile.self.cycles-pp.splice_direct_to_actor
0.58 +0.1 0.70 perf-profile.self.cycles-pp.do_syscall_64
0.62 +0.1 0.74 ? 2% perf-profile.self.cycles-pp.security_file_permission
0.81 +0.1 0.94 perf-profile.self.cycles-pp.read_tsc
0.48 +0.1 0.61 ? 2% perf-profile.self.cycles-pp.aa_file_perm
4.83 +0.1 4.96 perf-profile.self.cycles-pp.acpi_idle_do_entry
0.56 +0.1 0.71 perf-profile.self.cycles-pp.sendfile
0.37 ? 3% +0.1 0.51 ? 2% perf-profile.self.cycles-pp.__virt_addr_valid
0.73 +0.2 0.88 perf-profile.self.cycles-pp.__fsnotify_parent
1.78 +0.2 1.94 perf-profile.self.cycles-pp.filemap_read
0.67 +0.2 0.84 perf-profile.self.cycles-pp.sendfile_tcp_stream
0.73 +0.2 0.89 perf-profile.self.cycles-pp.__splice_from_pipe
0.67 +0.2 0.83 perf-profile.self.cycles-pp.atime_needs_update
0.60 ? 2% +0.2 0.78 perf-profile.self.cycles-pp.netperf_sendfile
0.73 +0.2 0.92 perf-profile.self.cycles-pp.__fget_light
0.68 +0.2 0.86 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.31 +0.2 0.53 ? 2% perf-profile.self.cycles-pp.filemap_get_pages
1.39 +0.2 1.62 perf-profile.self.cycles-pp.copy_page_to_iter
1.08 +0.2 1.32 ? 2% perf-profile.self.cycles-pp.common_file_perm
0.70 +0.2 0.94 perf-profile.self.cycles-pp.__might_sleep
0.82 ? 2% +0.2 1.06 perf-profile.self.cycles-pp._copy_to_iter
0.96 +0.2 1.21 perf-profile.self.cycles-pp.syscall_return_via_sysret
1.48 +0.3 1.81 perf-profile.self.cycles-pp.__might_resched
1.18 +0.3 1.51 ? 2% perf-profile.self.cycles-pp.do_sendfile
5.92 ? 2% +1.1 7.06 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation
Thanks,
Oliver Sang
Hello Chenyu,
On 2/7/2022 9:10 AM, Chen Yu wrote:
> [Problem Statement]
> Searching for an idle cpu/core in a LLC domain is time consuming
> when the LLC domain is relatively busy. Even worse, it is possible
> that after iterating the whole LLC domain, it might still
> fail to find an idle cpu/core.
We tested the patch running some standard benchmark on a dual
socket AMD Zen3 system (2 x 64C/128T). Following are the results
comparing the results seen on tip/sched/core (sched-tip) and
tip/sched/core + this patch (util-avg)
~~~~~~~~~
hackbench
~~~~~~~~~
NPS Mode - NPS1
Test: sched-tip util-avg
1-groups: 4.97 (0.00 pct) 5.06 (-1.81 pct)
2-groups: 5.71 (0.00 pct) 5.65 (1.05 pct)
4-groups: 6.33 (0.00 pct) 6.22 (1.73 pct)
8-groups: 7.93 (0.00 pct) 7.77 (2.01 pct)
16-groups: 12.35 (0.00 pct) 11.75 (4.85 pct)
NPS Mode - NPS2
Test: sched-tip util-avg
1-groups: 4.87 (0.00 pct) 4.97 (-2.05 pct)
2-groups: 5.43 (0.00 pct) 5.39 (0.73 pct)
4-groups: 6.07 (0.00 pct) 5.94 (2.14 pct)
8-groups: 7.46 (0.00 pct) 7.39 (0.93 pct)
16-groups: 10.27 (0.00 pct) 10.07 (1.94 pct)
NPS Mode - NPS4
Test: sched-tip util-avg
1-groups: 4.87 (0.00 pct) 4.89 (-0.41 pct)
2-groups: 5.40 (0.00 pct) 5.48 (-1.48 pct)
4-groups: 6.15 (0.00 pct) 6.17 (-0.32 pct)
8-groups: 7.63 (0.00 pct) 7.53 (1.31 pct)
16-groups: 10.24 (0.00 pct) 9.89 (3.41 pct)
~~~~~~~~
schbench
~~~~~~~~
NPS Mode - NPS1
#workers: sched-tip util-avg
1: 13.00 (0.00 pct) 14.50 (-11.53 pct)
2: 31.50 (0.00 pct) 34.00 (-7.93 pct)
4: 43.50 (0.00 pct) 37.50 (13.79 pct)
8: 56.00 (0.00 pct) 50.50 (9.82 pct)
16: 69.50 (0.00 pct) 68.50 (1.43 pct)
32: 107.00 (0.00 pct) 106.50 (0.46 pct)
64: 191.00 (0.00 pct) 191.50 (-0.26 pct)
128: 413.50 (0.00 pct) 408.50 (1.20 pct)
256: 950.00 (0.00 pct) 947.00 (0.31 pct)
512: 60352.00 (0.00 pct) 60288.00 (0.10 pct)
NPS Mode - NPS2
#workers: sched-tip util-avg
1: 11.50 (0.00 pct) 11.00 (4.34 pct)
2: 26.00 (0.00 pct) 30.00 (-15.38 pct)
4: 36.00 (0.00 pct) 39.00 (-8.33 pct)
8: 57.50 (0.00 pct) 49.50 (13.91 pct)
16: 68.50 (0.00 pct) 69.00 (-0.72 pct)
32: 105.50 (0.00 pct) 104.50 (0.94 pct)
64: 192.50 (0.00 pct) 189.50 (1.55 pct)
128: 403.50 (0.00 pct) 430.00 (-6.56 pct)
256: 945.00 (0.00 pct) 957.00 (-1.26 pct)
512: 60288.00 (0.00 pct) 60224.00 (0.10 pct)
NPS Mode - NPS4
#workers: sched-tip util-avg
1: 12.50 (0.00 pct) 11.50 (8.00 pct)
2: 29.00 (0.00 pct) 23.00 (20.68 pct)
4: 28.50 (0.00 pct) 29.50 (-3.50 pct)
8: 45.00 (0.00 pct) 46.50 (-3.33 pct)
16: 74.00 (0.00 pct) 84.00 (-13.51 pct)
32: 110.50 (0.00 pct) 108.50 (1.80 pct)
64: 198.00 (0.00 pct) 191.50 (3.28 pct)
128: 434.00 (0.00 pct) 418.00 (3.68 pct)
256: 964.00 (0.00 pct) 952.00 (1.24 pct)
512: 60352.00 (0.00 pct) 60352.00 (0.00 pct)
~~~~~~
tbench
~~~~~~
NPS Mode - NPS1
Clients: sched-tip util-avg
1 475.40 (0.00 pct) 480.46 (1.06 pct)
2 939.62 (0.00 pct) 929.13 (-1.11 pct)
4 1767.75 (0.00 pct) 1798.94 (1.76 pct)
8 3315.70 (0.00 pct) 3231.78 (-2.53 pct)
16 5887.79 (0.00 pct) 6037.36 (2.54 pct)
32 10202.20 (0.00 pct) 10223.65 (0.21 pct)
64 16816.42 (0.00 pct) 17122.40 (1.81 pct)
128 28423.90 (0.00 pct) 28680.46 (0.90 pct)
256 26726.29 (0.00 pct) 52502.83 (96.44 pct)
512 53082.43 (0.00 pct) 53635.54 (1.04 pct)
1024 52940.64 (0.00 pct) 53051.05 (0.20 pct)
NPS Mode - NPS2
Clients: sched-tip util-avg
1 481.84 (0.00 pct) 479.67 (-0.45 pct)
2 941.13 (0.00 pct) 927.93 (-1.40 pct)
4 1797.61 (0.00 pct) 1770.86 (-1.48 pct)
8 3353.37 (0.00 pct) 3360.29 (0.20 pct)
16 5779.49 (0.00 pct) 5925.27 (2.52 pct)
32 10810.27 (0.00 pct) 10855.17 (0.41 pct)
64 17046.83 (0.00 pct) 17457.64 (2.40 pct)
128 28799.32 (0.00 pct) 28855.49 (0.19 pct)
256 27654.49 (0.00 pct) 47126.18 (70.41 pct)
512 51831.52 (0.00 pct) 51219.55 (-1.18 pct)
1024 52579.87 (0.00 pct) 50213.68 (-4.50 pct)
NPS Mode - NPS4
Clients: sched-tip util-avg
1 474.08 (0.00 pct) 478.40 (0.91 pct)
2 928.23 (0.00 pct) 937.76 (1.02 pct)
4 1754.78 (0.00 pct) 1741.17 (-0.77 pct)
8 3219.31 (0.00 pct) 3318.00 (3.06 pct)
16 5791.53 (0.00 pct) 5687.78 (-1.79 pct)
32 9810.76 (0.00 pct) 10866.32 (10.75 pct)
64 17098.46 (0.00 pct) 17203.60 (0.61 pct)
128 28575.94 (0.00 pct) 28020.97 (-1.94 pct)
256 25997.38 (0.00 pct) 47735.83 (83.61 pct)
512 47698.94 (0.00 pct) 48220.54 (1.09 pct)
1024 51451.30 (0.00 pct) 50455.47 (-1.93 pct)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stream with 16 threads.
built with -DSTREAM_ARRAY_SIZE=128000000, -DNTIMES=10
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NPS Mode - NPS1
Test: sched-tip util-avg
Copy: 175278.66 (0.00 pct) 147847.67 (-15.64 pct)
Scale: 193235.46 (0.00 pct) 186582.76 (-3.44 pct)
Add: 220232.14 (0.00 pct) 208519.22 (-5.31 pct)
Triad: 215526.78 (0.00 pct) 204975.14 (-4.89 pct)
NPS Mode - NPS2
Test: sched-tip util-avg
Copy: 166600.09 (0.00 pct) 146456.45 (-12.09 pct)
Scale: 190820.84 (0.00 pct) 179603.16 (-5.87 pct)
Add: 214660.01 (0.00 pct) 198636.00 (-7.46 pct)
Triad: 210879.10 (0.00 pct) 196132.74 (-6.99 pct)
NPS Mode - NPS4
Test: sched-tip util-avg
Copy: 173888.83 (0.00 pct) 180948.16 (4.05 pct)
Scale: 193027.03 (0.00 pct) 199330.65 (3.26 pct)
Add: 218731.48 (0.00 pct) 221743.63 (1.37 pct)
Triad: 214651.06 (0.00 pct) 218036.35 (1.57 pct)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stream with 16 threads.
built with -DSTREAM_ARRAY_SIZE=128000000, -DNTIMES=100
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NPS Mode - NPS1
Test: sched-tip util-avg
Copy: 222954.79 (0.00 pct) 219066.70 (-1.74 pct)
Scale: 208255.97 (0.00 pct) 215060.20 (3.26 pct)
Add: 247694.16 (0.00 pct) 262893.33 (6.13 pct)
Triad: 236060.27 (0.00 pct) 250188.38 (5.98 pct)
NPS Mode - NPS2
Test: sched-tip util-avg
Copy: 226147.92 (0.00 pct) 228438.70 (1.01 pct)
Scale: 214059.09 (0.00 pct) 215959.36 (0.88 pct)
Add: 264978.05 (0.00 pct) 270058.63 (1.91 pct)
Triad: 253095.73 (0.00 pct) 257466.31 (1.72 pct)
NPS Mode - NPS4
Test: sched-tip util-avg
Copy: 223484.37 (0.00 pct) 266929.80 (19.44 pct)
Scale: 224198.90 (0.00 pct) 235316.80 (4.95 pct)
Add: 272008.98 (0.00 pct) 294452.15 (8.25 pct)
Triad: 258219.64 (0.00 pct) 285935.30 (10.73 pct)
--
We see great results in case of tbench with 256 workers when the
system is fully loaded which is exactly the kind of scenario this
patch optimizes for. Hackbench too shows improvements, especially
in the case of 16-groups.
--
Thanks and Regards,
Prateek
Hi Prateek,
On Wed, Feb 23, 2022 at 03:35:35PM +0530, K Prateek Nayak wrote:
> Hello Chenyu,
>
> On 2/7/2022 9:10 AM, Chen Yu wrote:
> > [Problem Statement]
> > Searching for an idle cpu/core in a LLC domain is time consuming
> > when the LLC domain is relatively busy. Even worse, it is possible
> > that after iterating the whole LLC domain, it might still
> > fail to find an idle cpu/core.
>
> We tested the patch running some standard benchmark on a dual
> socket AMD Zen3 system (2 x 64C/128T). Following are the results
> comparing the results seen on tip/sched/core (sched-tip) and
> tip/sched/core + this patch (util-avg)
>
> ~~~~~~~~~
> hackbench
> ~~~~~~~~~
>
> NPS Mode - NPS1
> Test: sched-tip util-avg
> 1-groups: 4.97 (0.00 pct) 5.06 (-1.81 pct)
> 2-groups: 5.71 (0.00 pct) 5.65 (1.05 pct)
> 4-groups: 6.33 (0.00 pct) 6.22 (1.73 pct)
> 8-groups: 7.93 (0.00 pct) 7.77 (2.01 pct)
> 16-groups: 12.35 (0.00 pct) 11.75 (4.85 pct)
>
> NPS Mode - NPS2
> Test: sched-tip util-avg
> 1-groups: 4.87 (0.00 pct) 4.97 (-2.05 pct)
> 2-groups: 5.43 (0.00 pct) 5.39 (0.73 pct)
> 4-groups: 6.07 (0.00 pct) 5.94 (2.14 pct)
> 8-groups: 7.46 (0.00 pct) 7.39 (0.93 pct)
> 16-groups: 10.27 (0.00 pct) 10.07 (1.94 pct)
>
> NPS Mode - NPS4
> Test: sched-tip util-avg
> 1-groups: 4.87 (0.00 pct) 4.89 (-0.41 pct)
> 2-groups: 5.40 (0.00 pct) 5.48 (-1.48 pct)
> 4-groups: 6.15 (0.00 pct) 6.17 (-0.32 pct)
> 8-groups: 7.63 (0.00 pct) 7.53 (1.31 pct)
> 16-groups: 10.24 (0.00 pct) 9.89 (3.41 pct)
>
> ~~~~~~~~
> schbench
> ~~~~~~~~
>
> NPS Mode - NPS1
> #workers: sched-tip util-avg
> 1: 13.00 (0.00 pct) 14.50 (-11.53 pct)
> 2: 31.50 (0.00 pct) 34.00 (-7.93 pct)
> 4: 43.50 (0.00 pct) 37.50 (13.79 pct)
> 8: 56.00 (0.00 pct) 50.50 (9.82 pct)
> 16: 69.50 (0.00 pct) 68.50 (1.43 pct)
> 32: 107.00 (0.00 pct) 106.50 (0.46 pct)
> 64: 191.00 (0.00 pct) 191.50 (-0.26 pct)
> 128: 413.50 (0.00 pct) 408.50 (1.20 pct)
> 256: 950.00 (0.00 pct) 947.00 (0.31 pct)
> 512: 60352.00 (0.00 pct) 60288.00 (0.10 pct)
>
> NPS Mode - NPS2
> #workers: sched-tip util-avg
> 1: 11.50 (0.00 pct) 11.00 (4.34 pct)
> 2: 26.00 (0.00 pct) 30.00 (-15.38 pct)
> 4: 36.00 (0.00 pct) 39.00 (-8.33 pct)
> 8: 57.50 (0.00 pct) 49.50 (13.91 pct)
> 16: 68.50 (0.00 pct) 69.00 (-0.72 pct)
> 32: 105.50 (0.00 pct) 104.50 (0.94 pct)
> 64: 192.50 (0.00 pct) 189.50 (1.55 pct)
> 128: 403.50 (0.00 pct) 430.00 (-6.56 pct)
> 256: 945.00 (0.00 pct) 957.00 (-1.26 pct)
> 512: 60288.00 (0.00 pct) 60224.00 (0.10 pct)
>
> NPS Mode - NPS4
> #workers: sched-tip util-avg
> 1: 12.50 (0.00 pct) 11.50 (8.00 pct)
> 2: 29.00 (0.00 pct) 23.00 (20.68 pct)
> 4: 28.50 (0.00 pct) 29.50 (-3.50 pct)
> 8: 45.00 (0.00 pct) 46.50 (-3.33 pct)
> 16: 74.00 (0.00 pct) 84.00 (-13.51 pct)
> 32: 110.50 (0.00 pct) 108.50 (1.80 pct)
> 64: 198.00 (0.00 pct) 191.50 (3.28 pct)
> 128: 434.00 (0.00 pct) 418.00 (3.68 pct)
> 256: 964.00 (0.00 pct) 952.00 (1.24 pct)
> 512: 60352.00 (0.00 pct) 60352.00 (0.00 pct)
>
> ~~~~~~
> tbench
> ~~~~~~
>
> NPS Mode - NPS1
> Clients: sched-tip util-avg
> 1 475.40 (0.00 pct) 480.46 (1.06 pct)
> 2 939.62 (0.00 pct) 929.13 (-1.11 pct)
> 4 1767.75 (0.00 pct) 1798.94 (1.76 pct)
> 8 3315.70 (0.00 pct) 3231.78 (-2.53 pct)
> 16 5887.79 (0.00 pct) 6037.36 (2.54 pct)
> 32 10202.20 (0.00 pct) 10223.65 (0.21 pct)
> 64 16816.42 (0.00 pct) 17122.40 (1.81 pct)
> 128 28423.90 (0.00 pct) 28680.46 (0.90 pct)
> 256 26726.29 (0.00 pct) 52502.83 (96.44 pct)
> 512 53082.43 (0.00 pct) 53635.54 (1.04 pct)
> 1024 52940.64 (0.00 pct) 53051.05 (0.20 pct)
>
> NPS Mode - NPS2
> Clients: sched-tip util-avg
> 1 481.84 (0.00 pct) 479.67 (-0.45 pct)
> 2 941.13 (0.00 pct) 927.93 (-1.40 pct)
> 4 1797.61 (0.00 pct) 1770.86 (-1.48 pct)
> 8 3353.37 (0.00 pct) 3360.29 (0.20 pct)
> 16 5779.49 (0.00 pct) 5925.27 (2.52 pct)
> 32 10810.27 (0.00 pct) 10855.17 (0.41 pct)
> 64 17046.83 (0.00 pct) 17457.64 (2.40 pct)
> 128 28799.32 (0.00 pct) 28855.49 (0.19 pct)
> 256 27654.49 (0.00 pct) 47126.18 (70.41 pct)
> 512 51831.52 (0.00 pct) 51219.55 (-1.18 pct)
> 1024 52579.87 (0.00 pct) 50213.68 (-4.50 pct)
>
> NPS Mode - NPS4
> Clients: sched-tip util-avg
> 1 474.08 (0.00 pct) 478.40 (0.91 pct)
> 2 928.23 (0.00 pct) 937.76 (1.02 pct)
> 4 1754.78 (0.00 pct) 1741.17 (-0.77 pct)
> 8 3219.31 (0.00 pct) 3318.00 (3.06 pct)
> 16 5791.53 (0.00 pct) 5687.78 (-1.79 pct)
> 32 9810.76 (0.00 pct) 10866.32 (10.75 pct)
> 64 17098.46 (0.00 pct) 17203.60 (0.61 pct)
> 128 28575.94 (0.00 pct) 28020.97 (-1.94 pct)
> 256 25997.38 (0.00 pct) 47735.83 (83.61 pct)
> 512 47698.94 (0.00 pct) 48220.54 (1.09 pct)
> 1024 51451.30 (0.00 pct) 50455.47 (-1.93 pct)
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Stream with 16 threads.
> built with -DSTREAM_ARRAY_SIZE=128000000, -DNTIMES=10
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> NPS Mode - NPS1
> Test: sched-tip util-avg
> Copy: 175278.66 (0.00 pct) 147847.67 (-15.64 pct)
> Scale: 193235.46 (0.00 pct) 186582.76 (-3.44 pct)
> Add: 220232.14 (0.00 pct) 208519.22 (-5.31 pct)
> Triad: 215526.78 (0.00 pct) 204975.14 (-4.89 pct)
>
> NPS Mode - NPS2
> Test: sched-tip util-avg
> Copy: 166600.09 (0.00 pct) 146456.45 (-12.09 pct)
> Scale: 190820.84 (0.00 pct) 179603.16 (-5.87 pct)
> Add: 214660.01 (0.00 pct) 198636.00 (-7.46 pct)
> Triad: 210879.10 (0.00 pct) 196132.74 (-6.99 pct)
>
> NPS Mode - NPS4
> Test: sched-tip util-avg
> Copy: 173888.83 (0.00 pct) 180948.16 (4.05 pct)
> Scale: 193027.03 (0.00 pct) 199330.65 (3.26 pct)
> Add: 218731.48 (0.00 pct) 221743.63 (1.37 pct)
> Triad: 214651.06 (0.00 pct) 218036.35 (1.57 pct)
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Stream with 16 threads.
> built with -DSTREAM_ARRAY_SIZE=128000000, -DNTIMES=100
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> NPS Mode - NPS1
> Test: sched-tip util-avg
> Copy: 222954.79 (0.00 pct) 219066.70 (-1.74 pct)
> Scale: 208255.97 (0.00 pct) 215060.20 (3.26 pct)
> Add: 247694.16 (0.00 pct) 262893.33 (6.13 pct)
> Triad: 236060.27 (0.00 pct) 250188.38 (5.98 pct)
>
> NPS Mode - NPS2
> Test: sched-tip util-avg
> Copy: 226147.92 (0.00 pct) 228438.70 (1.01 pct)
> Scale: 214059.09 (0.00 pct) 215959.36 (0.88 pct)
> Add: 264978.05 (0.00 pct) 270058.63 (1.91 pct)
> Triad: 253095.73 (0.00 pct) 257466.31 (1.72 pct)
>
> NPS Mode - NPS4
> Test: sched-tip util-avg
> Copy: 223484.37 (0.00 pct) 266929.80 (19.44 pct)
> Scale: 224198.90 (0.00 pct) 235316.80 (4.95 pct)
> Add: 272008.98 (0.00 pct) 294452.15 (8.25 pct)
> Triad: 258219.64 (0.00 pct) 285935.30 (10.73 pct)
> --
>
> We see great results in case of tbench with 256 workers when the
> system is fully loaded which is exactly the kind of scenario this
> patch optimizes for. Hackbench too shows improvements, especially
> in the case of 16-groups.
Thanks for the testing. Currently we have enhanced this patch
according to Peter's suggestion, to make the number of idle cpu
search more gradual. And we have launched some benchmarks to
evaluate the effect. Once it is done, we will send the v2 patch
out and it would be appreciated that you can also help give a glance
at it.
Thanks,
Chenyu
> --
> Thanks and Regards,
> Prateek