2021-03-16 08:21:10

by Aubrey Li

[permalink] [raw]
Subject: [PATCH v10] sched/fair: select idle cpu from idle cpumask for task wakeup

From: Aubrey Li <[email protected]>

Add idle cpumask to track idle cpus in sched domain. Every time
a CPU enters idle, the CPU is set in idle cpumask to be a wakeup
target. And if the CPU is not in idle, the CPU is cleared in idle
cpumask during scheduler tick to ratelimit idle cpumask update.

When a task wakes up to select an idle cpu, scanning idle cpumask
has lower cost than scanning all the cpus in last level cache domain,
especially when the system is heavily loaded.

v9->v10:
- Update scan cost only when the idle cpumask is scanned, i.e, the
idle cpumask is not empty

v8->v9:
- rebase on top of tip/sched/core, no code change

v7->v8:
- refine update_idle_cpumask, no functionality change
- fix a suspicious RCU usage warning with CONFIG_PROVE_RCU=y

v6->v7:
- place the whole idle cpumask mechanism under CONFIG_SMP

v5->v6:
- decouple idle cpumask update from stop_tick signal, set idle CPU
in idle cpumask every time the CPU enters idle

v4->v5:
- add update_idle_cpumask for s2idle case
- keep the same ordering of tick_nohz_idle_stop_tick() and update_
idle_cpumask() everywhere

v3->v4:
- change setting idle cpumask from every idle entry to tickless idle
if cpu driver is available
- move clearing idle cpumask to scheduler_tick to decouple nohz mode

v2->v3:
- change setting idle cpumask to every idle entry, otherwise schbench
has a regression of 99th percentile latency
- change clearing idle cpumask to nohz_balancer_kick(), so updating
idle cpumask is ratelimited in the idle exiting path
- set SCHED_IDLE cpu in idle cpumask to allow it as a wakeup target

v1->v2:
- idle cpumask is updated in the nohz routines, by initializing idle
cpumask with sched_domain_span(sd), nohz=off case remains the original
behavior

Cc: Peter Zijlstra <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Jiang Biao <[email protected]>
Cc: Tim Chen <[email protected]>
Signed-off-by: Aubrey Li <[email protected]>
---
include/linux/sched/topology.h | 13 ++++++++++++
kernel/sched/core.c | 2 ++
kernel/sched/fair.c | 47 ++++++++++++++++++++++++++++++++++++++++--
kernel/sched/idle.c | 5 +++++
kernel/sched/sched.h | 4 ++++
kernel/sched/topology.c | 3 ++-
6 files changed, 71 insertions(+), 3 deletions(-)

diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h
index 8f0f778..905e382 100644
--- a/include/linux/sched/topology.h
+++ b/include/linux/sched/topology.h
@@ -74,8 +74,21 @@ struct sched_domain_shared {
atomic_t ref;
atomic_t nr_busy_cpus;
int has_idle_cores;
+ /*
+ * Span of all idle CPUs in this domain.
+ *
+ * NOTE: this field is variable length. (Allocated dynamically
+ * by attaching extra space to the end of the structure,
+ * depending on how many CPUs the kernel has booted up with)
+ */
+ unsigned long idle_cpus_span[];
};

+static inline struct cpumask *sds_idle_cpus(struct sched_domain_shared *sds)
+{
+ return to_cpumask(sds->idle_cpus_span);
+}
+
struct sched_domain {
/* These fields must be setup */
struct sched_domain __rcu *parent; /* top domain must be null terminated */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ca2bb62..310bf9a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4552,6 +4552,7 @@ void scheduler_tick(void)

#ifdef CONFIG_SMP
rq->idle_balance = idle_cpu(cpu);
+ update_idle_cpumask(cpu, rq->idle_balance);
trigger_load_balance(rq);
#endif
}
@@ -8209,6 +8210,7 @@ void __init sched_init(void)
rq->idle_stamp = 0;
rq->avg_idle = 2*sysctl_sched_migration_cost;
rq->max_idle_balance_cost = sysctl_sched_migration_cost;
+ rq->last_idle_state = 1;

INIT_LIST_HEAD(&rq->cfs_tasks);

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 794c2cb..24384b4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6134,7 +6134,12 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
if (!this_sd)
return -1;

- cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr);
+ /*
+ * sched_domain_shared is set only at shared cache level,
+ * this works only because select_idle_cpu is called with
+ * sd_llc.
+ */
+ cpumask_and(cpus, sds_idle_cpus(sd->shared), p->cpus_ptr);

if (sched_feat(SIS_PROP) && !smt) {
u64 avg_cost, avg_idle, span_avg;
@@ -6173,7 +6178,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
if (smt)
set_idle_cores(this, false);

- if (sched_feat(SIS_PROP) && !smt) {
+ if (sched_feat(SIS_PROP) && !smt && (cpu < nr_cpumask_bits)) {
time = cpu_clock(this) - time;
update_avg(&this_sd->avg_scan_cost, time);
}
@@ -6838,6 +6843,44 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)

return newidle_balance(rq, rf) != 0;
}
+
+/*
+ * Update cpu idle state and record this information
+ * in sd_llc_shared->idle_cpus_span.
+ *
+ * This function is called with interrupts disabled.
+ */
+void update_idle_cpumask(int cpu, bool idle)
+{
+ struct sched_domain *sd;
+ struct rq *rq = cpu_rq(cpu);
+ int idle_state;
+
+ /*
+ * Also set SCHED_IDLE cpu in idle cpumask to
+ * allow SCHED_IDLE cpu as a wakeup target.
+ */
+ idle_state = idle || sched_idle_cpu(cpu);
+ /*
+ * No need to update idle cpumask if the state
+ * does not change.
+ */
+ if (rq->last_idle_state == idle_state)
+ return;
+ /*
+ * Called with irq disabled, rcu protection is not needed.
+ */
+ sd = per_cpu(sd_llc, cpu);
+ if (unlikely(!sd))
+ return;
+
+ if (idle_state)
+ cpumask_set_cpu(cpu, sds_idle_cpus(sd->shared));
+ else
+ cpumask_clear_cpu(cpu, sds_idle_cpus(sd->shared));
+
+ rq->last_idle_state = idle_state;
+}
#endif /* CONFIG_SMP */

static unsigned long wakeup_gran(struct sched_entity *se)
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index 7199e6f..9ff60f4 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -283,6 +283,11 @@ static void do_idle(void)
cpuhp_report_idle_dead();
arch_cpu_idle_dead();
}
+ /*
+ * The CPU is about to go idle, set it in idle cpumask
+ * to be a wake up target.
+ */
+ update_idle_cpumask(cpu, true);

arch_cpu_idle_enter();
rcu_nocb_flush_deferred_wakeup();
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 10a1522..8b85963 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -979,6 +979,7 @@ struct rq {

unsigned char nohz_idle_balance;
unsigned char idle_balance;
+ unsigned char last_idle_state;

unsigned long misfit_task_load;

@@ -1545,6 +1546,8 @@ static inline unsigned int group_first_cpu(struct sched_group *group)

extern int group_balance_cpu(struct sched_group *sg);

+void update_idle_cpumask(int cpu, bool idle);
+
#if defined(CONFIG_SCHED_DEBUG) && defined(CONFIG_SYSCTL)
void register_sched_domain_sysctl(void);
void dirty_sched_domain_sysctl(int cpu);
@@ -1565,6 +1568,7 @@ extern void flush_smp_call_function_from_idle(void);

#else /* !CONFIG_SMP: */
static inline void flush_smp_call_function_from_idle(void) { }
+static inline void update_idle_cpumask(int cpu, bool idle) { }
#endif

#include "stats.h"
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 09d3504..d480482 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -1466,6 +1466,7 @@ sd_init(struct sched_domain_topology_level *tl,
sd->shared = *per_cpu_ptr(sdd->sds, sd_id);
atomic_inc(&sd->shared->ref);
atomic_set(&sd->shared->nr_busy_cpus, sd_weight);
+ cpumask_copy(sds_idle_cpus(sd->shared), sched_domain_span(sd));
}

sd->private = sdd;
@@ -1825,7 +1826,7 @@ static int __sdt_alloc(const struct cpumask *cpu_map)

*per_cpu_ptr(sdd->sd, j) = sd;

- sds = kzalloc_node(sizeof(struct sched_domain_shared),
+ sds = kzalloc_node(sizeof(struct sched_domain_shared) + cpumask_size(),
GFP_KERNEL, cpu_to_node(j));
if (!sds)
return -ENOMEM;
--
2.7.4


2021-03-17 17:20:43

by Oliver Sang

[permalink] [raw]
Subject: [sched/fair] d619f7afd7: stress-ng.sock.ops_per_sec 69.4% improvement



Greeting,

FYI, we noticed a 69.4% improvement of stress-ng.sock.ops_per_sec due to commit:


commit: d619f7afd7731b1c3d56367e4583fd421ab1779e ("[PATCH v10] sched/fair: select idle cpu from idle cpumask for task wakeup")
url: https://github.com/0day-ci/linux/commits/Aubrey-Li/sched-fair-select-idle-cpu-from-idle-cpumask-for-task-wakeup/20210316-132415
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 13c2235b2b2870675195f0b551275d1abdd81068

in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
with following parameters:

nr_threads: 100%
disk: 1HDD
testtime: 60s
class: network
test: sock
cpufreq_governor: performance
ucode: 0x5003006






Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml
bin/lkp run compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
network/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/sock/stress-ng/60s/0x5003006

commit:
13c2235b2b ("sched: Remove unnecessary variable from schedule_tail()")
d619f7afd7 ("sched/fair: select idle cpu from idle cpumask for task wakeup")

13c2235b2b287067 d619f7afd7731b1c3d56367e458
---------------- ---------------------------
%stddev %change %stddev
\ | \
950326 +69.4% 1609719 ? 2% stress-ng.sock.ops
15838 +69.4% 26828 ? 2% stress-ng.sock.ops_per_sec
4.457e+08 -59.0% 1.828e+08 ? 9% stress-ng.time.involuntary_context_switches
18729 +16.1% 21750 stress-ng.time.minor_page_faults
6423 +18.3% 7597 stress-ng.time.percent_of_cpu_this_job_got
3812 +19.1% 4539 stress-ng.time.system_time
4.461e+08 -56.1% 1.96e+08 ? 8% stress-ng.time.voluntary_context_switches
7.44 +7.6% 8.00 iostat.cpu.idle
0.93 +0.2 1.10 ? 2% mpstat.cpu.all.irq%
22.86 -10.3 12.51 ? 4% mpstat.cpu.all.soft%
51.53 +10.7 62.24 mpstat.cpu.all.sys%
89.00 -1.1% 88.00 vmstat.cpu.sy
13737126 -56.8% 5929682 ? 8% vmstat.system.cs
210226 +7.7% 226386 vmstat.system.in
2382129 ? 25% +940.1% 24775963 ? 6% cpuidle.C1.time
183698 ? 73% +2123.6% 4084720 ? 7% cpuidle.C1.usage
421297 ? 81% +1721.9% 7675416 ? 6% cpuidle.POLL.time
120897 ? 82% +1582.7% 2034334 ? 5% cpuidle.POLL.usage
1141585 ? 37% +3269.7% 38467867 ? 5% numa-numastat.node0.local_node
1201302 ? 37% +3105.8% 38511127 ? 5% numa-numastat.node0.numa_hit
907692 ? 25% +4255.5% 39534228 ? 13% numa-numastat.node1.local_node
934562 ? 22% +4134.9% 39577574 ? 13% numa-numastat.node1.numa_hit
1387215 ? 12% +1308.4% 19537698 ? 4% numa-vmstat.node0.numa_hit
1288196 ? 12% +1413.2% 19493602 ? 4% numa-vmstat.node0.numa_local
1154758 ? 16% +1532.5% 18851309 ? 16% numa-vmstat.node1.numa_hit
1010907 ? 22% +1745.1% 18652251 ? 16% numa-vmstat.node1.numa_local
72843 +1.4% 73890 proc-vmstat.nr_inactive_anon
13858 ? 4% +8.7% 15058 ? 3% proc-vmstat.nr_mapped
23239 +8.2% 25139 proc-vmstat.nr_slab_reclaimable
64719 +4.9% 67881 proc-vmstat.nr_slab_unreclaimable
72843 +1.4% 73890 proc-vmstat.nr_zone_inactive_anon
432.33 ?106% +501.7% 2601 ? 75% proc-vmstat.numa_hint_faults_local
2127259 ? 26% +3531.9% 77260442 ? 6% proc-vmstat.numa_hit
2040648 ? 27% +3681.8% 77173834 ? 6% proc-vmstat.numa_local
14326984 ? 32% +4233.7% 6.209e+08 ? 6% proc-vmstat.pgalloc_normal
245168 +3.8% 254432 proc-vmstat.pgfault
14128459 ? 33% +4293.1% 6.207e+08 ? 6% proc-vmstat.pgfree
11746 ? 4% +11.4% 13087 ? 2% proc-vmstat.pgreuse
5137 ? 7% -50.9% 2522 ? 70% sched_debug.cfs_rq:/.load.min
3.92 ? 39% -76.6% 0.92 ? 48% sched_debug.cfs_rq:/.load_avg.min
1776013 +17.9% 2094409 sched_debug.cfs_rq:/.min_vruntime.avg
1883764 ? 3% +16.9% 2202539 sched_debug.cfs_rq:/.min_vruntime.max
1694658 +14.5% 1940737 sched_debug.cfs_rq:/.min_vruntime.min
1616 ? 9% +40.1% 2265 ? 17% sched_debug.cfs_rq:/.runnable_avg.max
199.99 ? 19% +66.2% 332.41 ? 16% sched_debug.cfs_rq:/.runnable_avg.stddev
1180 ? 14% +31.9% 1556 ? 10% sched_debug.cfs_rq:/.util_avg.max
455.00 ? 25% -62.5% 170.83 ? 21% sched_debug.cfs_rq:/.util_avg.min
169.21 ? 19% +47.8% 250.04 ? 9% sched_debug.cfs_rq:/.util_avg.stddev
448.20 ? 3% -16.5% 374.33 ? 5% sched_debug.cfs_rq:/.util_est_enqueued.avg
194.00 ? 22% -89.0% 21.42 ?219% sched_debug.cfs_rq:/.util_est_enqueued.min
162.52 ? 6% +20.6% 196.01 ? 6% sched_debug.cfs_rq:/.util_est_enqueued.stddev
14.15 ? 5% -50.6% 6.99 ? 38% sched_debug.cpu.clock.stddev
602.77 ? 4% +28.1% 772.40 ? 12% sched_debug.cpu.clock_task.stddev
21469 ? 15% +56.1% 33509 ? 31% sched_debug.cpu.max_idle_balance_cost.stddev
0.41 ? 3% +12.8% 0.46 ? 9% sched_debug.cpu.nr_running.stddev
4424282 -56.5% 1922647 ? 9% sched_debug.cpu.nr_switches.avg
4564285 -51.3% 2224187 ? 8% sched_debug.cpu.nr_switches.max
4082983 ? 2% -61.8% 1559208 ? 13% sched_debug.cpu.nr_switches.min
69357 ? 6% +245.7% 239751 ? 38% sched_debug.cpu.nr_switches.stddev
1304 ? 4% +17.2% 1527 ? 2% slabinfo.TCP.active_objs
1304 ? 4% +17.2% 1527 ? 2% slabinfo.TCP.num_objs
6641 ? 2% +48.0% 9833 ? 7% slabinfo.dmaengine-unmap-16.active_objs
6641 ? 2% +48.0% 9833 ? 7% slabinfo.dmaengine-unmap-16.num_objs
26279 ? 5% -12.7% 22947 slabinfo.filp.active_objs
855.50 ? 5% -14.6% 730.83 slabinfo.filp.active_slabs
27392 ? 5% -14.6% 23404 slabinfo.filp.num_objs
855.50 ? 5% -14.6% 730.83 slabinfo.filp.num_slabs
11247 ? 2% +30.8% 14710 slabinfo.kmalloc-1k.active_objs
354.67 ? 2% +30.5% 462.67 slabinfo.kmalloc-1k.active_slabs
11366 ? 2% +30.3% 14815 slabinfo.kmalloc-1k.num_objs
354.67 ? 2% +30.5% 462.67 slabinfo.kmalloc-1k.num_slabs
613.33 ? 8% -24.3% 464.00 ? 17% slabinfo.kmalloc-rcl-128.active_objs
613.33 ? 8% -24.3% 464.00 ? 17% slabinfo.kmalloc-rcl-128.num_objs
2107 ? 7% -11.6% 1862 ? 8% slabinfo.kmalloc-rcl-96.active_objs
2107 ? 7% -11.6% 1862 ? 8% slabinfo.kmalloc-rcl-96.num_objs
9878 ? 17% +89.8% 18752 slabinfo.skbuff_fclone_cache.active_objs
308.17 ? 17% +90.5% 587.00 slabinfo.skbuff_fclone_cache.active_slabs
9882 ? 17% +90.2% 18799 slabinfo.skbuff_fclone_cache.num_objs
308.17 ? 17% +90.5% 587.00 slabinfo.skbuff_fclone_cache.num_slabs
10637 ? 5% +41.6% 15066 ? 4% slabinfo.sock_inode_cache.active_objs
272.50 ? 5% +43.5% 391.17 ? 4% slabinfo.sock_inode_cache.active_slabs
10645 ? 5% +43.6% 15286 ? 4% slabinfo.sock_inode_cache.num_objs
272.50 ? 5% +43.5% 391.17 ? 4% slabinfo.sock_inode_cache.num_slabs
4.45 ? 3% +154.6% 11.33 ? 5% perf-stat.i.MPKI
3.343e+10 -24.5% 2.524e+10 ? 2% perf-stat.i.branch-instructions
1.57 -0.0 1.55 perf-stat.i.branch-miss-rate%
4.986e+08 -25.5% 3.716e+08 ? 2% perf-stat.i.branch-misses
2.63 ? 4% +15.2 17.87 ? 2% perf-stat.i.cache-miss-rate%
14369471 ? 4% +1660.1% 2.529e+08 ? 5% perf-stat.i.cache-misses
7.472e+08 ? 3% +88.0% 1.404e+09 ? 4% perf-stat.i.cache-references
14175402 -56.5% 6165131 ? 8% perf-stat.i.context-switches
1.42 +37.9% 1.96 ? 2% perf-stat.i.cpi
2.41e+11 +2.1% 2.46e+11 perf-stat.i.cpu-cycles
2786 ? 85% +7071.1% 199815 ? 6% perf-stat.i.cpu-migrations
16638 ? 4% -93.1% 1148 ? 20% perf-stat.i.cycles-between-cache-misses
0.00 ? 9% +0.0 0.02 ? 10% perf-stat.i.dTLB-load-miss-rate%
847687 ? 12% +570.6% 5684921 ? 9% perf-stat.i.dTLB-load-misses
4.994e+10 -27.0% 3.645e+10 ? 2% perf-stat.i.dTLB-loads
0.00 ? 11% +0.0 0.02 ? 18% perf-stat.i.dTLB-store-miss-rate%
131018 ? 17% +3378.5% 4557442 ? 17% perf-stat.i.dTLB-store-misses
2.874e+10 -29.4% 2.029e+10 ? 2% perf-stat.i.dTLB-stores
68.35 +6.4 74.79 perf-stat.i.iTLB-load-miss-rate%
3.051e+08 -31.3% 2.096e+08 ? 2% perf-stat.i.iTLB-load-misses
1.363e+08 -49.0% 69593955 ? 8% perf-stat.i.iTLB-loads
1.679e+11 -25.7% 1.248e+11 ? 2% perf-stat.i.instructions
702.32 +6.3% 746.70 perf-stat.i.instructions-per-iTLB-miss
0.71 -25.8% 0.52 ? 2% perf-stat.i.ipc
2.51 +2.1% 2.56 perf-stat.i.metric.GHz
0.72 ? 24% -82.7% 0.13 ? 23% perf-stat.i.metric.K/sec
1177 -26.0% 870.71 ? 2% perf-stat.i.metric.M/sec
3590 +3.6% 3719 perf-stat.i.minor-faults
84.06 -78.5 5.58 ? 20% perf-stat.i.node-load-miss-rate%
1437587 ? 3% +115.6% 3099118 ? 5% perf-stat.i.node-load-misses
367232 ? 14% +30953.2% 1.14e+08 ? 5% perf-stat.i.node-loads
90.98 -64.4 26.57 ? 5% perf-stat.i.node-store-miss-rate%
855437 ? 3% +269.7% 3162161 ? 7% perf-stat.i.node-store-misses
90133 ? 22% +9998.1% 9101716 ? 6% perf-stat.i.node-stores
3602 +3.6% 3731 perf-stat.i.page-faults
4.45 ? 3% +153.6% 11.29 ? 6% perf-stat.overall.MPKI
1.49 -0.0 1.47 perf-stat.overall.branch-miss-rate%
1.93 ? 6% +16.1 18.00 perf-stat.overall.cache-miss-rate%
1.44 +37.6% 1.97 ? 2% perf-stat.overall.cpi
16783 ? 4% -94.2% 974.85 ? 5% perf-stat.overall.cycles-between-cache-misses
0.00 ? 11% +0.0 0.02 ? 10% perf-stat.overall.dTLB-load-miss-rate%
0.00 ? 17% +0.0 0.02 ? 18% perf-stat.overall.dTLB-store-miss-rate%
69.11 +6.0 75.13 perf-stat.overall.iTLB-load-miss-rate%
550.55 +8.1% 595.17 perf-stat.overall.instructions-per-iTLB-miss
0.70 -27.3% 0.51 ? 2% perf-stat.overall.ipc
79.52 ? 2% -76.9 2.65 ? 3% perf-stat.overall.node-load-miss-rate%
90.36 -64.6 25.79 ? 2% perf-stat.overall.node-store-miss-rate%
3.29e+10 -24.5% 2.482e+10 ? 2% perf-stat.ps.branch-instructions
4.906e+08 -25.5% 3.654e+08 ? 2% perf-stat.ps.branch-misses
14156133 ? 4% +1659.4% 2.491e+08 ? 5% perf-stat.ps.cache-misses
7.355e+08 ? 3% +88.0% 1.383e+09 ? 4% perf-stat.ps.cache-references
13947799 -56.6% 6057134 ? 8% perf-stat.ps.context-switches
2.372e+11 +2.1% 2.421e+11 perf-stat.ps.cpu-cycles
2761 ? 85% +7024.8% 196786 ? 6% perf-stat.ps.cpu-migrations
863926 ? 11% +548.2% 5600307 ? 9% perf-stat.ps.dTLB-load-misses
4.914e+10 -27.0% 3.585e+10 ? 2% perf-stat.ps.dTLB-loads
130017 ? 17% +3348.4% 4483540 ? 17% perf-stat.ps.dTLB-store-misses
2.828e+10 -29.4% 1.995e+10 ? 2% perf-stat.ps.dTLB-stores
3.002e+08 -31.3% 2.062e+08 ? 2% perf-stat.ps.iTLB-load-misses
1.341e+08 -49.0% 68383313 ? 7% perf-stat.ps.iTLB-loads
1.652e+11 -25.7% 1.227e+11 ? 2% perf-stat.ps.instructions
3535 +3.6% 3662 perf-stat.ps.minor-faults
1414957 ? 3% +115.7% 3052181 ? 5% perf-stat.ps.node-load-misses
366486 ? 14% +30547.1% 1.123e+08 ? 5% perf-stat.ps.node-loads
842073 ? 3% +270.1% 3116228 ? 6% perf-stat.ps.node-store-misses
90486 ? 22% +9808.6% 8965949 ? 5% perf-stat.ps.node-stores
3547 +3.6% 3673 perf-stat.ps.page-faults
1.037e+13 -25.6% 7.715e+12 ? 2% perf-stat.total.instructions
4560843 -50.2% 2272786 ? 6% softirqs.CPU0.NET_RX
21241 ? 9% +23.3% 26193 ? 3% softirqs.CPU0.RCU
4498329 ? 3% -49.5% 2270538 ? 6% softirqs.CPU1.NET_RX
4543550 -49.7% 2285392 ? 6% softirqs.CPU10.NET_RX
18235 ? 2% +15.1% 20985 ? 3% softirqs.CPU10.RCU
4511949 -49.7% 2271032 ? 7% softirqs.CPU11.NET_RX
18035 +15.3% 20792 softirqs.CPU11.RCU
4481757 -49.5% 2261062 ? 7% softirqs.CPU12.NET_RX
18073 +15.3% 20840 ? 2% softirqs.CPU12.RCU
4499673 -49.9% 2252168 ? 6% softirqs.CPU13.NET_RX
18301 +14.3% 20926 ? 3% softirqs.CPU13.RCU
4508786 -49.3% 2284034 ? 7% softirqs.CPU14.NET_RX
18182 +16.4% 21162 softirqs.CPU14.RCU
4493226 -50.0% 2246240 ? 6% softirqs.CPU15.NET_RX
18292 +14.4% 20928 softirqs.CPU15.RCU
4513084 -49.7% 2272269 ? 7% softirqs.CPU16.NET_RX
18695 ? 3% +16.0% 21689 softirqs.CPU16.RCU
4429321 ? 2% -49.8% 2225171 ? 10% softirqs.CPU17.NET_RX
18950 ? 2% +13.7% 21548 ? 3% softirqs.CPU17.RCU
4493697 -49.4% 2274327 ? 7% softirqs.CPU18.NET_RX
19177 +14.7% 21992 softirqs.CPU18.RCU
4516955 -49.4% 2285842 ? 7% softirqs.CPU19.NET_RX
19178 +12.3% 21543 softirqs.CPU19.RCU
4551109 -50.5% 2251698 ? 6% softirqs.CPU2.NET_RX
18411 +14.4% 21064 softirqs.CPU2.RCU
4522378 -50.8% 2225906 ? 7% softirqs.CPU20.NET_RX
19110 +15.7% 22119 ? 2% softirqs.CPU20.RCU
4514623 -50.3% 2243136 ? 7% softirqs.CPU21.NET_RX
19218 ? 2% +12.8% 21687 softirqs.CPU21.RCU
4477606 ? 2% -50.1% 2234593 ? 7% softirqs.CPU22.NET_RX
18882 +14.5% 21618 softirqs.CPU22.RCU
4504922 -50.0% 2251753 ? 6% softirqs.CPU23.NET_RX
19164 +14.0% 21851 ? 2% softirqs.CPU23.RCU
4392559 -51.7% 2123133 ? 17% softirqs.CPU24.NET_RX
18593 +15.0% 21374 softirqs.CPU24.RCU
4491895 -52.4% 2139804 ? 17% softirqs.CPU25.NET_RX
4492562 -52.7% 2124465 ? 17% softirqs.CPU26.NET_RX
18774 +14.6% 21516 ? 3% softirqs.CPU26.RCU
4524012 -52.7% 2142018 ? 16% softirqs.CPU27.NET_RX
18705 +12.7% 21088 ? 3% softirqs.CPU27.RCU
4487532 -51.7% 2166923 ? 16% softirqs.CPU28.NET_RX
18890 ? 2% +13.0% 21337 softirqs.CPU28.RCU
4484561 -52.2% 2143458 ? 16% softirqs.CPU29.NET_RX
19019 ? 4% +12.3% 21355 ? 2% softirqs.CPU29.RCU
4551360 -49.9% 2281174 ? 7% softirqs.CPU3.NET_RX
18366 +15.4% 21197 softirqs.CPU3.RCU
4480179 -52.3% 2139257 ? 16% softirqs.CPU30.NET_RX
18520 ? 6% +14.7% 21245 softirqs.CPU30.RCU
4476574 -51.9% 2153958 ? 17% softirqs.CPU31.NET_RX
18594 +15.0% 21383 ? 2% softirqs.CPU31.RCU
4485217 -52.8% 2115908 ? 17% softirqs.CPU32.NET_RX
17909 +17.8% 21103 ? 2% softirqs.CPU32.RCU
4476301 -52.3% 2136742 ? 17% softirqs.CPU33.NET_RX
18182 ? 3% +16.6% 21207 ? 5% softirqs.CPU33.RCU
4470120 -52.4% 2126390 ? 17% softirqs.CPU34.NET_RX
17982 +16.8% 20996 ? 2% softirqs.CPU34.RCU
4443917 -52.1% 2128227 ? 16% softirqs.CPU35.NET_RX
18180 +15.9% 21075 ? 3% softirqs.CPU35.RCU
4473260 -52.6% 2122457 ? 16% softirqs.CPU36.NET_RX
18125 ? 2% +18.6% 21502 ? 4% softirqs.CPU36.RCU
4472379 -52.7% 2114795 ? 17% softirqs.CPU37.NET_RX
18007 +16.4% 20957 ? 2% softirqs.CPU37.RCU
4461059 -52.1% 2137066 ? 17% softirqs.CPU38.NET_RX
17935 +17.7% 21113 ? 2% softirqs.CPU38.RCU
4455317 -52.2% 2129969 ? 17% softirqs.CPU39.NET_RX
17911 +17.3% 21016 ? 3% softirqs.CPU39.RCU
4553331 -49.4% 2302532 ? 6% softirqs.CPU4.NET_RX
18113 +15.7% 20961 ? 2% softirqs.CPU4.RCU
4473435 -52.3% 2134944 ? 15% softirqs.CPU40.NET_RX
18248 ? 3% +14.5% 20893 ? 2% softirqs.CPU40.RCU
4465184 -51.6% 2163077 ? 16% softirqs.CPU41.NET_RX
18202 +14.4% 20817 ? 2% softirqs.CPU41.RCU
4489626 -52.6% 2126225 ? 17% softirqs.CPU42.NET_RX
18161 ? 2% +14.8% 20855 ? 2% softirqs.CPU42.RCU
4443047 -51.4% 2159443 ? 16% softirqs.CPU43.NET_RX
18129 ? 2% +15.3% 20908 ? 2% softirqs.CPU43.RCU
4476402 -52.0% 2146618 ? 16% softirqs.CPU44.NET_RX
17536 ? 4% +18.3% 20737 ? 2% softirqs.CPU44.RCU
4482489 -52.5% 2130397 ? 16% softirqs.CPU45.NET_RX
17834 +18.0% 21046 ? 3% softirqs.CPU45.RCU
4477349 -52.5% 2124980 ? 17% softirqs.CPU46.NET_RX
18077 +15.9% 20949 ? 2% softirqs.CPU46.RCU
4475117 -52.4% 2130668 ? 16% softirqs.CPU47.NET_RX
18436 ? 4% +13.5% 20933 ? 2% softirqs.CPU47.RCU
4373778 ? 3% -49.0% 2232155 ? 7% softirqs.CPU48.NET_RX
17745 ? 2% +15.6% 20514 ? 2% softirqs.CPU48.RCU
4496858 -50.7% 2217693 ? 6% softirqs.CPU49.NET_RX
17594 ? 3% +17.8% 20719 ? 4% softirqs.CPU49.RCU
4526397 -49.5% 2284088 ? 6% softirqs.CPU5.NET_RX
18140 +14.9% 20843 softirqs.CPU5.RCU
4517853 -49.6% 2274925 ? 6% softirqs.CPU50.NET_RX
4496903 -49.2% 2285523 ? 6% softirqs.CPU51.NET_RX
17902 +17.0% 20945 softirqs.CPU51.RCU
4520473 -50.0% 2261934 ? 6% softirqs.CPU52.NET_RX
4497228 -49.5% 2269608 ? 6% softirqs.CPU53.NET_RX
18098 +15.2% 20855 softirqs.CPU53.RCU
4511411 -49.9% 2261161 ? 6% softirqs.CPU54.NET_RX
18471 ? 2% +12.3% 20750 softirqs.CPU54.RCU
4520114 -49.6% 2280245 ? 6% softirqs.CPU55.NET_RX
18106 ? 2% +15.4% 20890 ? 3% softirqs.CPU55.RCU
4525942 -50.1% 2258939 ? 7% softirqs.CPU56.NET_RX
18002 +15.6% 20806 ? 2% softirqs.CPU56.RCU
4504366 -49.2% 2286767 ? 6% softirqs.CPU57.NET_RX
18075 +17.6% 21253 ? 3% softirqs.CPU57.RCU
4494686 -49.1% 2288598 ? 6% softirqs.CPU58.NET_RX
18444 ? 3% +13.3% 20888 softirqs.CPU58.RCU
4518913 -49.7% 2272563 ? 5% softirqs.CPU59.NET_RX
18077 +16.9% 21127 ? 4% softirqs.CPU59.RCU
4514808 -49.5% 2278657 ? 6% softirqs.CPU6.NET_RX
17961 +17.4% 21089 ? 2% softirqs.CPU6.RCU
4508378 -49.5% 2275322 ? 6% softirqs.CPU60.NET_RX
18163 +14.9% 20875 softirqs.CPU60.RCU
4507669 -49.8% 2262680 ? 6% softirqs.CPU61.NET_RX
17799 +16.1% 20659 softirqs.CPU61.RCU
4524653 -49.5% 2286102 ? 6% softirqs.CPU62.NET_RX
18109 +13.8% 20607 softirqs.CPU62.RCU
4519567 -50.0% 2257645 ? 7% softirqs.CPU63.NET_RX
17920 +15.1% 20624 ? 3% softirqs.CPU63.RCU
4515072 -49.9% 2260971 ? 6% softirqs.CPU64.NET_RX
19783 ? 4% +11.5% 22066 softirqs.CPU64.RCU
4507658 -49.6% 2271667 ? 5% softirqs.CPU65.NET_RX
19468 ? 3% +12.4% 21883 softirqs.CPU65.RCU
4519295 -49.4% 2286044 ? 7% softirqs.CPU66.NET_RX
19478 +11.6% 21739 softirqs.CPU66.RCU
4560346 -49.8% 2287499 ? 6% softirqs.CPU67.NET_RX
19841 ? 2% +10.3% 21880 softirqs.CPU67.RCU
4598839 -50.5% 2276110 ? 7% softirqs.CPU68.NET_RX
19499 +14.1% 22248 ? 3% softirqs.CPU68.RCU
4595176 -50.4% 2279210 ? 6% softirqs.CPU69.NET_RX
19782 ? 3% +11.4% 22033 softirqs.CPU69.RCU
4523914 -49.3% 2292970 ? 7% softirqs.CPU7.NET_RX
18066 +15.8% 20918 softirqs.CPU7.RCU
4596985 -50.5% 2273992 ? 8% softirqs.CPU70.NET_RX
19518 +12.3% 21912 softirqs.CPU70.RCU
4597696 -50.9% 2257307 ? 5% softirqs.CPU71.NET_RX
19724 +11.5% 21987 softirqs.CPU71.RCU
4398050 -52.0% 2110908 ? 16% softirqs.CPU72.NET_RX
19263 +11.0% 21389 ? 2% softirqs.CPU72.RCU
4482365 -52.1% 2145713 ? 16% softirqs.CPU73.NET_RX
19202 ? 2% +13.1% 21709 ? 2% softirqs.CPU73.RCU
4453475 -52.1% 2131900 ? 17% softirqs.CPU74.NET_RX
18981 +13.3% 21508 softirqs.CPU74.RCU
4407861 ? 3% -51.5% 2137316 ? 16% softirqs.CPU75.NET_RX
18845 ? 2% +13.6% 21400 ? 2% softirqs.CPU75.RCU
4455731 -52.1% 2133073 ? 16% softirqs.CPU76.NET_RX
18978 +13.0% 21450 ? 2% softirqs.CPU76.RCU
4473339 -52.4% 2127571 ? 16% softirqs.CPU77.NET_RX
19010 +12.6% 21407 ? 2% softirqs.CPU77.RCU
4451979 -52.0% 2136060 ? 16% softirqs.CPU78.NET_RX
19020 +12.6% 21420 ? 2% softirqs.CPU78.RCU
4450498 -51.7% 2148363 ? 17% softirqs.CPU79.NET_RX
18837 ? 5% +13.8% 21430 ? 2% softirqs.CPU79.RCU
4566338 -50.2% 2275865 ? 6% softirqs.CPU8.NET_RX
18206 +17.9% 21462 ? 3% softirqs.CPU8.RCU
4483014 -52.4% 2132319 ? 16% softirqs.CPU80.NET_RX
17798 +16.6% 20756 ? 2% softirqs.CPU80.RCU
4457292 -52.2% 2131632 ? 16% softirqs.CPU81.NET_RX
17826 +17.7% 20984 ? 3% softirqs.CPU81.RCU
4457882 -52.3% 2124800 ? 16% softirqs.CPU82.NET_RX
17995 ? 2% +15.5% 20790 ? 2% softirqs.CPU82.RCU
4468206 -52.6% 2117092 ? 17% softirqs.CPU83.NET_RX
17839 +16.4% 20766 ? 2% softirqs.CPU83.RCU
4466076 -52.3% 2131641 ? 17% softirqs.CPU84.NET_RX
17808 +19.9% 21347 ? 4% softirqs.CPU84.RCU
4460142 ? 2% -52.2% 2133593 ? 18% softirqs.CPU85.NET_RX
17759 +17.3% 20824 ? 2% softirqs.CPU85.RCU
4455148 -51.8% 2145521 ? 16% softirqs.CPU86.NET_RX
17811 +17.8% 20986 ? 3% softirqs.CPU86.RCU
4484463 -52.3% 2136959 ? 16% softirqs.CPU87.NET_RX
17794 +17.2% 20861 softirqs.CPU87.RCU
4493967 -52.5% 2135760 ? 16% softirqs.CPU88.NET_RX
17951 ? 2% +15.7% 20765 ? 2% softirqs.CPU88.RCU
4490187 -52.3% 2140828 ? 16% softirqs.CPU89.NET_RX
17910 +17.3% 21003 softirqs.CPU89.RCU
4530910 -49.6% 2283047 ? 7% softirqs.CPU9.NET_RX
18161 +15.4% 20964 ? 2% softirqs.CPU9.RCU
4488277 -52.4% 2136450 ? 16% softirqs.CPU90.NET_RX
17769 +17.0% 20797 softirqs.CPU90.RCU
4470274 -52.2% 2135528 ? 16% softirqs.CPU91.NET_RX
18367 ? 4% +12.7% 20706 ? 2% softirqs.CPU91.RCU
4481323 -52.2% 2140226 ? 17% softirqs.CPU92.NET_RX
4506115 -52.7% 2132735 ? 17% softirqs.CPU93.NET_RX
17943 ? 2% +16.5% 20906 ? 2% softirqs.CPU93.RCU
4490969 -52.5% 2133008 ? 17% softirqs.CPU94.NET_RX
17878 +17.2% 20960 ? 2% softirqs.CPU94.RCU
4289458 -52.9% 2018653 ? 17% softirqs.CPU95.NET_RX
18053 +14.9% 20735 softirqs.CPU95.RCU
4.313e+08 -51.0% 2.112e+08 ? 6% softirqs.NET_RX
1773319 +14.9% 2037601 softirqs.RCU
250242 ? 3% +118.6% 546975 ? 3% softirqs.SCHED
482254 +33.4% 643317 softirqs.TIMER
74585 ? 23% +888.7% 737428 ? 6% interrupts.CAL:Function_call_interrupts
949.50 ? 46% +759.3% 8159 ? 5% interrupts.CPU0.CAL:Function_call_interrupts
311.83 ? 79% +1239.8% 4177 ? 10% interrupts.CPU0.RES:Rescheduling_interrupts
1701 ? 32% +348.6% 7631 ? 14% interrupts.CPU1.CAL:Function_call_interrupts
430.00 ?152% +939.6% 4470 ? 21% interrupts.CPU1.RES:Rescheduling_interrupts
813.17 ? 36% +812.0% 7415 ? 3% interrupts.CPU10.CAL:Function_call_interrupts
167.50 ?103% +2386.9% 4165 ? 6% interrupts.CPU10.RES:Rescheduling_interrupts
915.83 ? 32% +765.5% 7926 ? 5% interrupts.CPU11.CAL:Function_call_interrupts
761.17 ?160% +549.6% 4944 ? 10% interrupts.CPU11.RES:Rescheduling_interrupts
849.33 ? 39% +757.2% 7280 ? 6% interrupts.CPU12.CAL:Function_call_interrupts
195.67 ? 92% +2177.6% 4456 ? 17% interrupts.CPU12.RES:Rescheduling_interrupts
841.00 ? 35% +748.5% 7135 ? 5% interrupts.CPU13.CAL:Function_call_interrupts
154.50 ? 69% +2810.5% 4496 ? 17% interrupts.CPU13.RES:Rescheduling_interrupts
988.83 ? 58% +687.6% 7787 ? 7% interrupts.CPU14.CAL:Function_call_interrupts
744.67 ?171% +483.4% 4344 ? 9% interrupts.CPU14.RES:Rescheduling_interrupts
875.83 ? 41% +735.4% 7316 ? 7% interrupts.CPU15.CAL:Function_call_interrupts
480.00 ?157% +769.2% 4172 ? 8% interrupts.CPU15.RES:Rescheduling_interrupts
831.67 ? 36% +773.6% 7265 ? 5% interrupts.CPU16.CAL:Function_call_interrupts
117.67 ? 18% +3542.1% 4285 ? 4% interrupts.CPU16.RES:Rescheduling_interrupts
791.00 ? 39% +871.6% 7685 ? 6% interrupts.CPU17.CAL:Function_call_interrupts
116.17 ? 35% +4163.7% 4953 ? 17% interrupts.CPU17.RES:Rescheduling_interrupts
858.50 ? 38% +795.7% 7689 ? 5% interrupts.CPU18.CAL:Function_call_interrupts
122.50 ? 49% +3339.9% 4213 ? 12% interrupts.CPU18.RES:Rescheduling_interrupts
845.67 ? 37% +813.6% 7725 ? 5% interrupts.CPU19.CAL:Function_call_interrupts
108.17 ? 35% +3729.7% 4142 ? 8% interrupts.CPU19.RES:Rescheduling_interrupts
1069 ? 42% +626.1% 7764 ? 9% interrupts.CPU2.CAL:Function_call_interrupts
245.50 ?138% +1761.0% 4568 ? 16% interrupts.CPU2.RES:Rescheduling_interrupts
831.67 ? 37% +775.0% 7276 ? 5% interrupts.CPU20.CAL:Function_call_interrupts
180.17 ? 97% +2586.4% 4840 ? 17% interrupts.CPU20.RES:Rescheduling_interrupts
868.67 ? 35% +756.8% 7443 ? 7% interrupts.CPU21.CAL:Function_call_interrupts
334.00 ?102% +1274.1% 4589 ? 11% interrupts.CPU21.RES:Rescheduling_interrupts
815.50 ? 38% +792.7% 7279 ? 7% interrupts.CPU22.CAL:Function_call_interrupts
207.00 ?112% +1974.5% 4294 ? 14% interrupts.CPU22.RES:Rescheduling_interrupts
877.00 ? 40% +737.8% 7347 ? 3% interrupts.CPU23.CAL:Function_call_interrupts
431.50 ? 91% +880.0% 4228 ? 6% interrupts.CPU23.RES:Rescheduling_interrupts
771.67 ? 24% +914.2% 7826 ? 14% interrupts.CPU24.CAL:Function_call_interrupts
669.17 ?123% +597.9% 4670 ? 12% interrupts.CPU24.RES:Rescheduling_interrupts
652.83 ? 21% +1166.0% 8265 ? 15% interrupts.CPU25.CAL:Function_call_interrupts
187.17 ? 68% +2520.1% 4904 ? 9% interrupts.CPU25.RES:Rescheduling_interrupts
710.67 ? 16% +998.2% 7804 ? 14% interrupts.CPU26.CAL:Function_call_interrupts
795.50 ? 94% +455.8% 4421 ? 9% interrupts.CPU26.RES:Rescheduling_interrupts
663.33 ? 19% +1117.2% 8074 ? 15% interrupts.CPU27.CAL:Function_call_interrupts
321.50 ?103% +1195.1% 4163 ? 10% interrupts.CPU27.RES:Rescheduling_interrupts
728.33 ? 24% +1008.5% 8073 ? 13% interrupts.CPU28.CAL:Function_call_interrupts
600.67 ?146% +628.2% 4374 ? 11% interrupts.CPU28.RES:Rescheduling_interrupts
646.67 ? 20% +1205.0% 8438 ? 14% interrupts.CPU29.CAL:Function_call_interrupts
135.00 ? 69% +3127.9% 4357 ? 10% interrupts.CPU29.RES:Rescheduling_interrupts
1045 ? 32% +638.8% 7723 ? 8% interrupts.CPU3.CAL:Function_call_interrupts
227.50 ? 70% +1781.5% 4280 ? 11% interrupts.CPU3.RES:Rescheduling_interrupts
668.83 ? 18% +1095.6% 7996 ? 15% interrupts.CPU30.CAL:Function_call_interrupts
286.67 ?159% +1636.2% 4977 ? 13% interrupts.CPU30.RES:Rescheduling_interrupts
657.50 ? 27% +1154.6% 8249 ? 15% interrupts.CPU31.CAL:Function_call_interrupts
404.33 ?149% +956.7% 4272 ? 12% interrupts.CPU31.RES:Rescheduling_interrupts
591.50 ? 17% +1204.1% 7713 ? 14% interrupts.CPU32.CAL:Function_call_interrupts
103.17 ? 77% +4475.8% 4720 ? 14% interrupts.CPU32.RES:Rescheduling_interrupts
716.17 ? 22% +1013.3% 7973 ? 15% interrupts.CPU33.CAL:Function_call_interrupts
502.67 ?122% +745.2% 4248 ? 13% interrupts.CPU33.RES:Rescheduling_interrupts
694.33 ? 27% +1097.4% 8313 ? 14% interrupts.CPU34.CAL:Function_call_interrupts
602.00 ?157% +642.4% 4469 ? 17% interrupts.CPU34.RES:Rescheduling_interrupts
668.17 ? 21% +1041.4% 7626 ? 15% interrupts.CPU35.CAL:Function_call_interrupts
392.33 ? 77% +1021.7% 4400 ? 14% interrupts.CPU35.RES:Rescheduling_interrupts
656.83 ? 24% +1073.3% 7706 ? 14% interrupts.CPU36.CAL:Function_call_interrupts
395.50 ?129% +1018.1% 4422 ? 8% interrupts.CPU36.RES:Rescheduling_interrupts
769.50 ? 36% +915.4% 7813 ? 13% interrupts.CPU37.CAL:Function_call_interrupts
644.67 ?161% +647.9% 4821 ? 12% interrupts.CPU37.RES:Rescheduling_interrupts
796.33 ? 28% +899.7% 7960 ? 16% interrupts.CPU38.CAL:Function_call_interrupts
1208 ?113% +283.0% 4628 ? 11% interrupts.CPU38.RES:Rescheduling_interrupts
676.00 ? 22% +1015.6% 7541 ? 14% interrupts.CPU39.CAL:Function_call_interrupts
650.83 ?131% +591.3% 4499 ? 15% interrupts.CPU39.RES:Rescheduling_interrupts
914.00 ? 29% +715.2% 7451 ? 6% interrupts.CPU4.CAL:Function_call_interrupts
173.17 ? 59% +2370.7% 4278 ? 11% interrupts.CPU4.RES:Rescheduling_interrupts
647.67 ? 29% +1118.6% 7892 ? 12% interrupts.CPU40.CAL:Function_call_interrupts
238.67 ?173% +2078.4% 5199 ? 15% interrupts.CPU40.RES:Rescheduling_interrupts
666.17 ? 17% +1137.1% 8241 ? 13% interrupts.CPU41.CAL:Function_call_interrupts
520.50 ?129% +778.7% 4573 ? 14% interrupts.CPU41.RES:Rescheduling_interrupts
608.83 ? 16% +1181.2% 7800 ? 15% interrupts.CPU42.CAL:Function_call_interrupts
180.83 ?147% +2343.0% 4417 ? 19% interrupts.CPU42.RES:Rescheduling_interrupts
656.17 ? 19% +1156.6% 8245 ? 13% interrupts.CPU43.CAL:Function_call_interrupts
256.00 ?165% +1744.3% 4721 ? 14% interrupts.CPU43.RES:Rescheduling_interrupts
718.67 ? 19% +1041.6% 8204 ? 13% interrupts.CPU44.CAL:Function_call_interrupts
750.00 ? 66% +586.9% 5151 ? 13% interrupts.CPU44.RES:Rescheduling_interrupts
614.17 ? 14% +1160.4% 7740 ? 12% interrupts.CPU45.CAL:Function_call_interrupts
93.33 ? 37% +4536.1% 4327 ? 9% interrupts.CPU45.RES:Rescheduling_interrupts
627.50 ? 23% +1113.4% 7614 ? 15% interrupts.CPU46.CAL:Function_call_interrupts
63.17 ? 41% +6762.3% 4334 ? 8% interrupts.CPU46.RES:Rescheduling_interrupts
673.50 ? 24% +1047.3% 7727 ? 14% interrupts.CPU47.CAL:Function_call_interrupts
447.67 ?171% +797.8% 4019 ? 10% interrupts.CPU47.RES:Rescheduling_interrupts
901.33 ? 40% +707.9% 7281 ? 3% interrupts.CPU48.CAL:Function_call_interrupts
273.00 ? 77% +1507.9% 4389 ? 22% interrupts.CPU48.RES:Rescheduling_interrupts
792.17 ? 34% +811.7% 7221 ? 7% interrupts.CPU49.CAL:Function_call_interrupts
220.33 ?119% +1938.2% 4490 ? 23% interrupts.CPU49.RES:Rescheduling_interrupts
885.17 ? 48% +781.6% 7803 ? 6% interrupts.CPU5.CAL:Function_call_interrupts
111.17 ? 37% +4122.3% 4693 ? 8% interrupts.CPU5.RES:Rescheduling_interrupts
854.50 ? 41% +796.8% 7663 ? 5% interrupts.CPU50.CAL:Function_call_interrupts
84.00 ? 43% +5481.5% 4688 ? 17% interrupts.CPU50.RES:Rescheduling_interrupts
812.17 ? 38% +839.6% 7631 ? 6% interrupts.CPU51.CAL:Function_call_interrupts
136.00 ? 62% +2912.5% 4097 ? 6% interrupts.CPU51.RES:Rescheduling_interrupts
793.00 ? 32% +886.8% 7825 ? 5% interrupts.CPU52.CAL:Function_call_interrupts
103.50 ? 35% +3944.0% 4185 ? 18% interrupts.CPU52.RES:Rescheduling_interrupts
814.50 ? 38% +843.9% 7688 ? 4% interrupts.CPU53.CAL:Function_call_interrupts
106.00 ? 54% +4367.1% 4735 ? 26% interrupts.CPU53.RES:Rescheduling_interrupts
816.67 ? 36% +810.2% 7433 ? 4% interrupts.CPU54.CAL:Function_call_interrupts
367.50 ?155% +1119.7% 4482 ? 14% interrupts.CPU54.RES:Rescheduling_interrupts
797.00 ? 38% +846.4% 7543 ? 5% interrupts.CPU55.CAL:Function_call_interrupts
188.50 ? 83% +2103.5% 4153 ? 17% interrupts.CPU55.RES:Rescheduling_interrupts
796.17 ? 33% +799.0% 7157 ? 4% interrupts.CPU56.CAL:Function_call_interrupts
115.50 ? 45% +3379.9% 4019 ? 8% interrupts.CPU56.RES:Rescheduling_interrupts
822.83 ? 38% +817.1% 7546 ? 5% interrupts.CPU57.CAL:Function_call_interrupts
118.00 ? 57% +3346.8% 4067 ? 7% interrupts.CPU57.RES:Rescheduling_interrupts
859.17 ? 39% +766.0% 7440 ? 3% interrupts.CPU58.CAL:Function_call_interrupts
194.33 ?109% +1974.0% 4030 ? 10% interrupts.CPU58.RES:Rescheduling_interrupts
937.33 ? 34% +713.6% 7626 ? 4% interrupts.CPU59.CAL:Function_call_interrupts
980.00 ?121% +380.7% 4711 ? 12% interrupts.CPU59.RES:Rescheduling_interrupts
803.33 ? 36% +843.5% 7579 ? 3% interrupts.CPU6.CAL:Function_call_interrupts
219.83 ? 85% +1885.5% 4364 ? 16% interrupts.CPU6.RES:Rescheduling_interrupts
821.67 ? 36% +778.9% 7221 ? 5% interrupts.CPU60.CAL:Function_call_interrupts
120.33 ? 54% +3372.9% 4179 ? 10% interrupts.CPU60.RES:Rescheduling_interrupts
822.67 ? 37% +751.5% 7005 ? 6% interrupts.CPU61.CAL:Function_call_interrupts
92.33 ? 46% +5290.6% 4977 ? 14% interrupts.CPU61.RES:Rescheduling_interrupts
818.67 ? 35% +830.7% 7619 ? 5% interrupts.CPU62.CAL:Function_call_interrupts
258.33 ?148% +1511.4% 4162 ? 7% interrupts.CPU62.RES:Rescheduling_interrupts
836.00 ? 39% +768.5% 7260 ? 6% interrupts.CPU63.CAL:Function_call_interrupts
405.17 ?157% +1005.6% 4479 ? 7% interrupts.CPU63.RES:Rescheduling_interrupts
847.17 ? 35% +746.2% 7168 ? 4% interrupts.CPU64.CAL:Function_call_interrupts
142.67 ? 44% +2921.0% 4310 ? 8% interrupts.CPU64.RES:Rescheduling_interrupts
813.67 ? 33% +783.4% 7187 ? 5% interrupts.CPU65.CAL:Function_call_interrupts
156.17 ? 73% +2880.9% 4655 ? 26% interrupts.CPU65.RES:Rescheduling_interrupts
871.00 ? 35% +781.8% 7680 ? 5% interrupts.CPU66.CAL:Function_call_interrupts
133.00 ? 29% +3291.4% 4510 ? 11% interrupts.CPU66.RES:Rescheduling_interrupts
807.17 ? 29% +815.9% 7392 ? 7% interrupts.CPU67.CAL:Function_call_interrupts
154.00 ? 42% +2559.7% 4096 ? 6% interrupts.CPU67.RES:Rescheduling_interrupts
799.00 ? 28% +774.9% 6990 ? 5% interrupts.CPU68.CAL:Function_call_interrupts
131.17 ? 25% +3205.7% 4336 ? 7% interrupts.CPU68.RES:Rescheduling_interrupts
775.83 ? 29% +809.8% 7058 ? 5% interrupts.CPU69.CAL:Function_call_interrupts
322.33 ?133% +1370.1% 4738 ? 22% interrupts.CPU69.RES:Rescheduling_interrupts
873.33 ? 44% +755.5% 7471 ? 5% interrupts.CPU7.CAL:Function_call_interrupts
869.17 ?151% +393.3% 4287 ? 10% interrupts.CPU7.RES:Rescheduling_interrupts
783.67 ? 30% +814.5% 7166 ? 6% interrupts.CPU70.CAL:Function_call_interrupts
119.67 ? 44% +3966.4% 4866 ? 22% interrupts.CPU70.RES:Rescheduling_interrupts
857.67 ? 39% +714.6% 6986 ? 5% interrupts.CPU71.CAL:Function_call_interrupts
1016 ?122% +310.8% 4177 ? 14% interrupts.CPU71.RES:Rescheduling_interrupts
771.67 ? 23% +906.2% 7764 ? 13% interrupts.CPU72.CAL:Function_call_interrupts
569.33 ? 94% +702.7% 4570 ? 14% interrupts.CPU72.RES:Rescheduling_interrupts
697.17 ? 23% +1079.9% 8225 ? 13% interrupts.CPU73.CAL:Function_call_interrupts
447.50 ?115% +918.2% 4556 ? 8% interrupts.CPU73.RES:Rescheduling_interrupts
712.33 ? 16% +962.1% 7565 ? 15% interrupts.CPU74.CAL:Function_call_interrupts
424.67 ? 79% +1011.3% 4719 ? 10% interrupts.CPU74.RES:Rescheduling_interrupts
656.00 ? 17% +1142.5% 8151 ? 14% interrupts.CPU75.CAL:Function_call_interrupts
161.67 ? 93% +2668.7% 4476 ? 11% interrupts.CPU75.RES:Rescheduling_interrupts
648.00 ? 23% +1167.7% 8214 ? 12% interrupts.CPU76.CAL:Function_call_interrupts
165.83 ?107% +2423.5% 4184 ? 11% interrupts.CPU76.RES:Rescheduling_interrupts
647.17 ? 17% +1163.1% 8174 ? 14% interrupts.CPU77.CAL:Function_call_interrupts
89.17 ? 38% +4762.1% 4335 ? 12% interrupts.CPU77.RES:Rescheduling_interrupts
702.00 ? 19% +1078.0% 8269 ? 15% interrupts.CPU78.CAL:Function_call_interrupts
342.83 ?115% +1267.9% 4689 ? 17% interrupts.CPU78.RES:Rescheduling_interrupts
725.00 ? 19% +1027.3% 8173 ? 14% interrupts.CPU79.CAL:Function_call_interrupts
487.83 ?103% +796.8% 4374 ? 14% interrupts.CPU79.RES:Rescheduling_interrupts
843.17 ? 42% +745.9% 7132 ? 4% interrupts.CPU8.CAL:Function_call_interrupts
479.50 ?153% +851.2% 4561 ? 16% interrupts.CPU8.RES:Rescheduling_interrupts
629.17 ? 17% +1119.4% 7671 ? 16% interrupts.CPU80.CAL:Function_call_interrupts
134.00 ? 94% +3176.1% 4390 ? 8% interrupts.CPU80.RES:Rescheduling_interrupts
739.17 ? 25% +1020.7% 8284 ? 14% interrupts.CPU81.CAL:Function_call_interrupts
588.67 ?131% +655.6% 4448 ? 9% interrupts.CPU81.RES:Rescheduling_interrupts
738.83 ? 28% +991.0% 8060 ? 15% interrupts.CPU82.CAL:Function_call_interrupts
936.00 ?141% +331.6% 4039 ? 12% interrupts.CPU82.RES:Rescheduling_interrupts
649.83 ? 26% +1076.6% 7645 ? 14% interrupts.CPU83.CAL:Function_call_interrupts
389.00 ?191% +925.0% 3987 ? 11% interrupts.CPU83.RES:Rescheduling_interrupts
647.67 ? 18% +1086.0% 7681 ? 16% interrupts.CPU84.CAL:Function_call_interrupts
157.33 ?115% +2630.4% 4295 ? 13% interrupts.CPU84.RES:Rescheduling_interrupts
674.50 ? 18% +1020.2% 7555 ? 15% interrupts.CPU85.CAL:Function_call_interrupts
439.83 ?122% +919.1% 4482 ? 11% interrupts.CPU85.RES:Rescheduling_interrupts
762.00 ? 26% +931.9% 7863 ? 13% interrupts.CPU86.CAL:Function_call_interrupts
1235 ? 55% +280.0% 4694 ? 10% interrupts.CPU86.RES:Rescheduling_interrupts
672.33 ? 17% +1062.5% 7816 ? 14% interrupts.CPU87.CAL:Function_call_interrupts
419.67 ? 95% +960.2% 4449 ? 4% interrupts.CPU87.RES:Rescheduling_interrupts
638.67 ? 24% +1100.3% 7665 ? 15% interrupts.CPU88.CAL:Function_call_interrupts
139.33 ?127% +3397.1% 4872 ? 13% interrupts.CPU88.RES:Rescheduling_interrupts
661.67 ? 17% +1151.1% 8277 ? 14% interrupts.CPU89.CAL:Function_call_interrupts
204.33 ? 93% +2116.4% 4528 ? 7% interrupts.CPU89.RES:Rescheduling_interrupts
800.33 ? 34% +839.1% 7515 ? 5% interrupts.CPU9.CAL:Function_call_interrupts
93.00 ? 59% +4509.1% 4286 ? 9% interrupts.CPU9.RES:Rescheduling_interrupts
605.17 ? 25% +1174.4% 7712 ? 13% interrupts.CPU90.CAL:Function_call_interrupts
188.33 ?110% +2269.0% 4461 ? 13% interrupts.CPU90.RES:Rescheduling_interrupts
677.83 ? 19% +1112.1% 8216 ? 14% interrupts.CPU91.CAL:Function_call_interrupts
310.00 ?152% +1462.7% 4844 ? 14% interrupts.CPU91.RES:Rescheduling_interrupts
909.33 ? 51% +790.0% 8092 ? 14% interrupts.CPU92.CAL:Function_call_interrupts
628.33 ? 20% +1085.4% 7448 ? 14% interrupts.CPU93.CAL:Function_call_interrupts
265.50 ?118% +1547.7% 4374 ? 10% interrupts.CPU93.RES:Rescheduling_interrupts
615.67 ? 16% +1133.3% 7593 ? 16% interrupts.CPU94.CAL:Function_call_interrupts
77.67 ? 43% +5157.7% 4083 ? 9% interrupts.CPU94.RES:Rescheduling_interrupts
755.67 ? 16% +807.7% 6859 ? 17% interrupts.CPU95.CAL:Function_call_interrupts
450.50 ?121% +816.7% 4129 ? 13% interrupts.CPU95.RES:Rescheduling_interrupts
33556 ? 32% +1173.3% 427283 ? 6% interrupts.RES:Rescheduling_interrupts
31.20 -17.8 13.41 ? 14% perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
31.04 -17.7 13.30 ? 14% perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
28.44 -17.1 11.34 ? 16% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
31.02 -15.8 15.19 ? 10% perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
26.09 -15.7 10.43 ? 16% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
30.64 -15.6 15.03 ? 10% perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
30.16 -15.4 14.79 ? 10% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
28.56 -14.5 14.09 ? 10% perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2
27.75 -14.0 13.74 ? 10% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq.__local_bh_enable_ip
27.59 -13.9 13.66 ? 10% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start.do_softirq
24.97 -13.9 11.10 ? 13% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
26.56 -13.6 12.98 ? 10% perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__softirqentry_text_start
23.82 -13.2 10.64 ? 13% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
25.11 -12.9 12.25 ? 10% perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
23.83 -12.2 11.59 ? 10% perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll
23.69 -12.2 11.51 ? 10% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
23.55 -12.1 11.44 ? 10% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
23.00 -11.8 11.20 ? 10% perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
18.52 -10.7 7.83 ? 12% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
18.87 -10.0 8.90 ? 10% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
16.78 -9.2 7.56 ? 15% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
15.34 -8.2 7.16 ? 14% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
13.14 -7.7 5.41 ? 16% perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
14.29 -7.5 6.79 ? 14% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg
11.53 -6.9 4.61 ? 17% perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
11.13 -6.9 4.23 ? 12% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
10.91 -6.9 4.04 ? 14% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
13.24 -6.8 6.42 ? 13% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked
11.10 -6.7 4.40 ? 17% perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
10.53 -6.6 3.89 ? 14% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
10.89 -6.6 4.30 ? 17% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
10.21 -6.5 3.68 ? 17% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
10.61 -6.4 4.16 ? 17% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
8.48 -4.6 3.93 ? 15% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
8.06 -4.5 3.57 ? 16% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
5.41 -3.5 1.91 ? 22% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
37.10 -3.5 33.65 perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
5.18 -3.4 1.82 ? 22% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
5.21 -3.1 2.15 ? 17% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
4.79 -2.9 1.88 ? 17% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woken
4.52 -2.8 1.74 ? 19% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
4.46 -2.8 1.71 ? 19% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
39.81 -2.7 37.16 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
39.67 -2.6 37.06 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.87 -2.4 1.47 ? 21% perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
38.48 -2.4 36.11 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
38.25 -2.3 35.96 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
2.72 -2.0 0.69 ? 15% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
3.15 -1.9 1.25 ? 27% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit
2.85 -1.8 1.04 ? 34% perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
2.38 -1.5 0.90 ? 19% perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
2.10 -1.5 0.64 ? 49% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode
2.21 -1.4 0.85 ? 20% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common
2.14 -1.3 0.87 ? 17% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
1.44 -1.0 0.40 ? 72% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.wait_woken
1.78 -1.0 0.77 ? 16% perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
1.34 -0.9 0.40 ? 72% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.schedule_timeout.wait_woken
1.37 -0.9 0.49 ? 46% perf-profile.calltrace.cycles-pp.__tcp_send_ack.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
1.24 -0.9 0.37 ? 72% perf-profile.calltrace.cycles-pp.__alloc_skb.__tcp_send_ack.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
1.96 -0.7 1.31 ? 5% perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
1.71 -0.6 1.08 ? 7% perf-profile.calltrace.cycles-pp.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
1.76 -0.3 1.49 ? 2% perf-profile.calltrace.cycles-pp.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.57 -0.2 1.33 ? 2% perf-profile.calltrace.cycles-pp.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.28 -0.2 1.07 ? 2% perf-profile.calltrace.cycles-pp.inet_ioctl.sock_do_ioctl.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl
1.35 -0.2 1.14 ? 2% perf-profile.calltrace.cycles-pp.sock_do_ioctl.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64
2.48 -0.2 2.30 perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.57 -0.1 0.43 ? 44% perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.99 -0.1 0.85 ? 2% perf-profile.calltrace.cycles-pp.tcp_ioctl.inet_ioctl.sock_do_ioctl.sock_ioctl.do_vfs_ioctl
0.72 -0.1 0.64 ? 2% perf-profile.calltrace.cycles-pp.tcp_rcv_space_adjust.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
0.52 +0.1 0.58 ? 2% perf-profile.calltrace.cycles-pp.tcp_send_mss.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
95.64 +0.6 96.22 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
0.00 +0.6 0.58 ? 4% perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
0.00 +0.6 0.59 ? 4% perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
0.00 +0.6 0.61 ? 6% perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
0.00 +0.6 0.61 ? 6% perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.6 0.62 ? 6% perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.6 0.62 ? 6% perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.6 0.62 ? 2% perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
0.00 +0.9 0.94 ? 11% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +0.9 0.94 ? 11% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +0.9 0.95 ? 11% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
0.00 +0.9 0.95 ? 10% perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
0.00 +1.0 0.96 ? 11% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
0.00 +1.0 0.97 ? 13% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg
0.00 +1.1 1.10 ? 11% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established
0.00 +1.2 1.18 ? 11% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv
0.00 +1.3 1.28 ? 13% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg
0.00 +1.3 1.29 ? 11% perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
0.00 +1.3 1.29 ? 11% perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
0.00 +1.4 1.36 ? 13% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
0.59 ? 2% +1.5 2.06 ? 10% perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
0.00 +1.6 1.64 ? 10% perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
0.00 +1.8 1.83 ? 10% perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue
0.00 +1.9 1.87 ? 18% perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
0.00 +1.9 1.88 ? 10% perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack
0.00 +1.9 1.89 ? 18% perf-profile.calltrace.cycles-pp.__sk_mem_schedule.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
0.00 +2.0 2.03 ? 10% perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
0.00 +2.3 2.32 ? 10% perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
0.00 +2.5 2.53 ? 9% perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
0.56 ? 10% +3.0 3.53 ? 9% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
0.45 ? 46% +3.0 3.50 ? 9% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
1.90 +3.1 5.01 ? 7% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
2.31 +3.1 5.45 ? 6% perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
1.93 +3.2 5.09 ? 7% perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked
3.11 +3.4 6.55 ? 6% perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
1.10 ? 7% +3.5 4.55 ? 8% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto
3.16 +3.5 6.61 ? 6% perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
38.75 +3.5 42.22 perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
1.45 ? 6% +3.5 4.98 ? 8% perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
1.92 ? 7% +4.2 6.16 ? 8% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg
2.02 ? 7% +4.3 6.29 ? 8% perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
2.35 ? 6% +4.7 7.04 ? 7% perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
86.42 +5.2 91.67 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
42.78 +7.1 49.89 ? 2% perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
42.58 +7.2 49.74 ? 2% perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
41.63 +7.2 48.88 ? 2% perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
40.90 +7.3 48.17 ? 2% perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
0.00 +10.3 10.27 ? 11% perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_recvmsg_locked
0.00 +10.5 10.54 ? 11% perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg
0.59 ? 2% +10.7 11.29 ? 10% perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
0.00 +11.0 10.95 ? 11% perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
0.00 +11.6 11.63 ? 11% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data
0.00 +11.8 11.81 ? 11% perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb
0.00 +11.8 11.83 ? 11% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.rmqueue.get_page_from_freelist.__alloc_pages_nodemask
0.00 +12.0 11.99 ? 11% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.rmqueue.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill
0.00 +12.7 12.72 ? 11% perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill
0.00 +13.0 13.01 ? 11% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked
0.00 +13.1 13.08 ? 11% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg
0.00 +13.4 13.37 ? 11% perf-profile.calltrace.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
0.00 +13.4 13.43 ? 11% perf-profile.calltrace.cycles-pp.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
46.23 -23.5 22.75 ? 11% perf-profile.children.cycles-pp.__tcp_transmit_skb
42.30 -21.3 20.97 ? 11% perf-profile.children.cycles-pp.__ip_queue_xmit
40.05 -20.0 20.02 ? 11% perf-profile.children.cycles-pp.ip_output
37.76 -18.8 19.00 ? 11% perf-profile.children.cycles-pp.ip_finish_output2
31.55 -15.7 15.88 ? 10% perf-profile.children.cycles-pp.__tcp_push_pending_frames
31.90 -15.4 16.46 ? 10% perf-profile.children.cycles-pp.__local_bh_enable_ip
31.52 -15.3 16.20 ? 10% perf-profile.children.cycles-pp.tcp_write_xmit
31.13 -15.1 16.00 ? 10% perf-profile.children.cycles-pp.do_softirq
30.87 -14.6 16.22 ? 9% perf-profile.children.cycles-pp.__softirqentry_text_start
29.02 -14.0 14.99 ? 10% perf-profile.children.cycles-pp.net_rx_action
28.19 -13.6 14.62 ? 10% perf-profile.children.cycles-pp.__napi_poll
28.04 -13.5 14.54 ? 10% perf-profile.children.cycles-pp.process_backlog
27.00 -13.0 14.01 ? 10% perf-profile.children.cycles-pp.__netif_receive_skb_one_core
25.54 -12.2 13.33 ? 9% perf-profile.children.cycles-pp.ip_rcv
24.23 -11.5 12.71 ? 9% perf-profile.children.cycles-pp.ip_local_deliver
24.09 -11.5 12.62 ? 9% perf-profile.children.cycles-pp.ip_local_deliver_finish
23.95 -11.4 12.55 ? 9% perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
23.42 -11.1 12.31 ? 9% perf-profile.children.cycles-pp.tcp_v4_rcv
16.32 -10.0 6.28 ? 18% perf-profile.children.cycles-pp.schedule
16.37 -9.7 6.65 ? 17% perf-profile.children.cycles-pp.__schedule
13.16 -7.7 5.41 ? 16% perf-profile.children.cycles-pp.sk_wait_data
11.53 -6.9 4.61 ? 17% perf-profile.children.cycles-pp.wait_woken
11.11 -6.6 4.46 ? 16% perf-profile.children.cycles-pp.schedule_timeout
11.24 -6.5 4.74 ? 13% perf-profile.children.cycles-pp.sock_def_readable
11.05 -6.4 4.62 ? 13% perf-profile.children.cycles-pp.__wake_up_common_lock
19.26 -6.3 12.99 ? 4% perf-profile.children.cycles-pp.tcp_rcv_established
10.65 -6.2 4.44 ? 13% perf-profile.children.cycles-pp.__wake_up_common
10.35 -6.0 4.30 ? 13% perf-profile.children.cycles-pp.try_to_wake_up
19.83 -5.2 14.65 ? 3% perf-profile.children.cycles-pp.tcp_v4_do_rcv
8.52 -4.5 3.97 ? 14% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
8.11 -4.5 3.60 ? 16% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
37.16 -3.5 33.68 perf-profile.children.cycles-pp.tcp_recvmsg_locked
4.80 -2.9 1.90 ? 17% perf-profile.children.cycles-pp.dequeue_task_fair
39.84 -2.7 37.19 perf-profile.children.cycles-pp.__x64_sys_recvfrom
5.42 -2.6 2.81 ? 11% perf-profile.children.cycles-pp.__dev_queue_xmit
39.68 -2.6 37.07 perf-profile.children.cycles-pp.__sys_recvfrom
4.58 -2.6 1.98 ? 16% perf-profile.children.cycles-pp.ttwu_do_activate
4.53 -2.6 1.95 ? 16% perf-profile.children.cycles-pp.enqueue_task_fair
38.49 -2.4 36.11 perf-profile.children.cycles-pp.inet_recvmsg
3.86 -2.4 1.49 ? 20% perf-profile.children.cycles-pp.pick_next_task_fair
38.30 -2.3 35.99 perf-profile.children.cycles-pp.tcp_recvmsg
3.60 -2.1 1.48 ? 16% perf-profile.children.cycles-pp.update_load_avg
3.22 -2.0 1.22 ? 20% perf-profile.children.cycles-pp.update_curr
2.75 -1.9 0.80 ? 12% perf-profile.children.cycles-pp.select_task_rq_fair
3.28 -1.6 1.67 ? 12% perf-profile.children.cycles-pp.dev_hard_start_xmit
2.08 -1.6 0.50 ? 12% perf-profile.children.cycles-pp.select_idle_cpu
2.67 -1.6 1.11 ? 17% perf-profile.children.cycles-pp.switch_mm_irqs_off
2.98 -1.5 1.50 ? 11% perf-profile.children.cycles-pp.loopback_xmit
3.09 -1.3 1.76 ? 8% perf-profile.children.cycles-pp.__alloc_skb
2.19 -1.3 0.91 ? 17% perf-profile.children.cycles-pp.dequeue_entity
2.28 -1.3 1.00 ? 16% perf-profile.children.cycles-pp.enqueue_entity
1.62 -1.1 0.53 ? 20% perf-profile.children.cycles-pp.reweight_entity
1.77 -1.0 0.73 ? 18% perf-profile.children.cycles-pp.load_new_mm_cr3
1.78 -1.0 0.77 ? 16% perf-profile.children.cycles-pp.switch_fpu_return
1.00 -0.9 0.13 ? 10% perf-profile.children.cycles-pp.available_idle_cpu
1.64 -0.8 0.81 ? 12% perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
1.64 ? 2% -0.8 0.81 ? 11% perf-profile.children.cycles-pp.netif_rx
1.60 -0.8 0.78 ? 11% perf-profile.children.cycles-pp.netif_rx_internal
1.43 -0.8 0.64 ? 12% perf-profile.children.cycles-pp.sk_filter_trim_cap
1.46 -0.8 0.70 ? 12% perf-profile.children.cycles-pp.__tcp_send_ack
1.24 -0.8 0.49 ? 15% perf-profile.children.cycles-pp.__switch_to_asm
1.07 -0.7 0.39 ? 19% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.94 -0.7 0.28 ? 22% perf-profile.children.cycles-pp.pick_next_entity
1.98 -0.6 1.33 ? 5% perf-profile.children.cycles-pp.sk_stream_alloc_skb
1.12 -0.6 0.48 ? 17% perf-profile.children.cycles-pp.set_next_entity
0.99 -0.6 0.35 ? 20% perf-profile.children.cycles-pp.check_preempt_curr
1.14 -0.6 0.51 ? 12% perf-profile.children.cycles-pp.__netif_receive_skb_core
1.04 -0.6 0.46 ? 14% perf-profile.children.cycles-pp.sched_clock_cpu
0.87 -0.6 0.29 ? 21% perf-profile.children.cycles-pp.check_preempt_wakeup
0.90 -0.6 0.33 ? 11% perf-profile.children.cycles-pp.update_rq_clock
0.86 -0.6 0.30 ? 19% perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
0.96 -0.6 0.41 ? 15% perf-profile.children.cycles-pp.__switch_to
0.88 -0.5 0.34 ? 18% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.86 -0.5 0.32 ? 22% perf-profile.children.cycles-pp.put_prev_entity
1.03 -0.5 0.52 ? 10% perf-profile.children.cycles-pp.kmalloc_reserve
0.92 -0.5 0.41 ? 13% perf-profile.children.cycles-pp.sched_clock
0.93 -0.5 0.42 ? 12% perf-profile.children.cycles-pp.irqtime_account_irq
0.91 ? 2% -0.5 0.41 ? 17% perf-profile.children.cycles-pp.validate_xmit_skb
0.79 -0.5 0.29 ? 15% perf-profile.children.cycles-pp.__update_load_avg_se
0.89 -0.5 0.40 ? 14% perf-profile.children.cycles-pp.native_sched_clock
0.96 -0.5 0.49 ? 12% perf-profile.children.cycles-pp.ip_finish_output
0.88 -0.4 0.46 ? 10% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
0.58 ? 3% -0.4 0.19 ? 26% perf-profile.children.cycles-pp.set_next_buddy
0.71 -0.4 0.33 ? 13% perf-profile.children.cycles-pp.tcp_schedule_loss_probe
0.74 -0.4 0.36 ? 11% perf-profile.children.cycles-pp.enqueue_to_backlog
0.63 ? 2% -0.4 0.26 ? 13% perf-profile.children.cycles-pp.finish_task_switch
0.80 -0.4 0.44 ? 9% perf-profile.children.cycles-pp.kmem_cache_alloc_node
0.92 -0.3 0.59 ? 7% perf-profile.children.cycles-pp.__inet_lookup_established
0.64 ? 2% -0.3 0.33 ? 12% perf-profile.children.cycles-pp.ktime_get_with_offset
0.59 -0.3 0.28 ? 15% perf-profile.children.cycles-pp.ip_rcv_finish
0.61 -0.3 0.30 ? 15% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
0.94 ? 5% -0.3 0.64 ? 7% perf-profile.children.cycles-pp.update_cfs_group
0.44 -0.3 0.14 ? 23% perf-profile.children.cycles-pp.clear_buddies
0.53 -0.3 0.24 ? 15% perf-profile.children.cycles-pp.__sk_dst_check
0.93 ? 2% -0.3 0.65 ? 5% perf-profile.children.cycles-pp.tcp_mstamp_refresh
0.63 ? 2% -0.3 0.34 ? 8% perf-profile.children.cycles-pp.__ksize
0.58 -0.3 0.29 ? 14% perf-profile.children.cycles-pp.___perf_sw_event
0.53 ? 2% -0.3 0.25 ? 12% perf-profile.children.cycles-pp.ip_rcv_core
0.45 ? 4% -0.3 0.18 ? 16% perf-profile.children.cycles-pp.ipv4_dst_check
1.77 -0.3 1.50 ? 2% perf-profile.children.cycles-pp.do_vfs_ioctl
0.65 -0.3 0.40 ? 9% perf-profile.children.cycles-pp.skb_release_all
0.55 ? 2% -0.3 0.30 ? 8% perf-profile.children.cycles-pp.kfree
0.38 ? 2% -0.2 0.14 ? 18% perf-profile.children.cycles-pp.cpumask_next_wrap
0.39 ? 3% -0.2 0.15 ? 23% perf-profile.children.cycles-pp.update_min_vruntime
0.42 -0.2 0.19 ? 14% perf-profile.children.cycles-pp.migrate_disable
0.43 ? 6% -0.2 0.19 ? 17% perf-profile.children.cycles-pp.netif_skb_features
0.60 -0.2 0.37 ? 9% perf-profile.children.cycles-pp.skb_release_head_state
0.38 -0.2 0.15 ? 14% perf-profile.children.cycles-pp.security_sock_rcv_skb
0.38 -0.2 0.16 ? 13% perf-profile.children.cycles-pp.tcp_ack_update_rtt
0.39 ? 2% -0.2 0.17 ? 15% perf-profile.children.cycles-pp.__calc_delta
1.39 ? 3% -0.2 1.17 perf-profile.children.cycles-pp.ktime_get
1.29 -0.2 1.08 ? 2% perf-profile.children.cycles-pp.inet_ioctl
0.38 -0.2 0.18 ? 19% perf-profile.children.cycles-pp.migrate_enable
0.52 -0.2 0.32 ? 12% perf-profile.children.cycles-pp.ip_local_out
0.35 ? 7% -0.2 0.16 ? 19% perf-profile.children.cycles-pp.cpuacct_charge
0.46 -0.2 0.27 ? 11% perf-profile.children.cycles-pp.__ip_local_out
0.37 ? 3% -0.2 0.18 ? 15% perf-profile.children.cycles-pp.nf_hook_slow
0.38 -0.2 0.19 ? 15% perf-profile.children.cycles-pp.__ip_finish_output
1.01 -0.2 0.82 ? 3% perf-profile.children.cycles-pp.read_tsc
0.33 -0.2 0.14 ? 16% perf-profile.children.cycles-pp.skb_entail
2.49 -0.2 2.31 perf-profile.children.cycles-pp.__x64_sys_ioctl
0.27 ? 3% -0.2 0.09 ? 18% perf-profile.children.cycles-pp.resched_curr
1.39 -0.2 1.21 perf-profile.children.cycles-pp.sock_do_ioctl
0.29 -0.2 0.11 ? 17% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.53 ? 3% -0.2 0.36 ? 6% perf-profile.children.cycles-pp.tcp_event_new_data_sent
0.79 -0.2 0.62 ? 6% perf-profile.children.cycles-pp.__cond_resched
0.32 ? 2% -0.2 0.16 ? 15% perf-profile.children.cycles-pp.perf_tp_event
1.04 -0.2 0.88 ? 2% perf-profile.children.cycles-pp.tcp_ioctl
0.33 ? 2% -0.2 0.17 ? 12% perf-profile.children.cycles-pp.ip_rcv_finish_core
0.22 ? 4% -0.2 0.07 ? 10% perf-profile.children.cycles-pp.tcp_v4_inbound_md5_hash
0.24 ? 3% -0.2 0.08 ? 17% perf-profile.children.cycles-pp.__usecs_to_jiffies
0.28 ? 3% -0.2 0.12 ? 15% perf-profile.children.cycles-pp.add_wait_queue
0.47 -0.1 0.32 ? 8% perf-profile.children.cycles-pp.preempt_schedule_common
0.25 ? 2% -0.1 0.10 ? 8% perf-profile.children.cycles-pp.tcp_event_data_recv
0.25 ? 9% -0.1 0.10 ? 22% perf-profile.children.cycles-pp.eth_type_trans
0.63 -0.1 0.49 ? 3% perf-profile.children.cycles-pp.__might_sleep
0.43 -0.1 0.29 ? 9% perf-profile.children.cycles-pp.__skb_clone
0.29 ? 3% -0.1 0.15 ? 12% perf-profile.children.cycles-pp.inet_ehashfn
0.25 ? 2% -0.1 0.12 ? 14% perf-profile.children.cycles-pp.__tcp_select_window
0.22 ? 5% -0.1 0.09 ? 16% perf-profile.children.cycles-pp.skb_network_protocol
0.22 ? 4% -0.1 0.10 ? 11% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.39 ? 4% -0.1 0.27 ? 7% perf-profile.children.cycles-pp.sk_reset_timer
0.22 ? 2% -0.1 0.10 ? 15% perf-profile.children.cycles-pp.tcp_options_write
0.29 -0.1 0.17 ? 9% perf-profile.children.cycles-pp.tcp_wfree
0.23 -0.1 0.11 ? 13% perf-profile.children.cycles-pp.tcp_v4_fill_cb
0.20 ? 2% -0.1 0.09 ? 14% perf-profile.children.cycles-pp.raw_local_deliver
0.19 ? 2% -0.1 0.08 ? 22% perf-profile.children.cycles-pp._find_next_bit
0.19 -0.1 0.08 ? 21% perf-profile.children.cycles-pp.bpf_skops_write_hdr_opt
0.15 ? 2% -0.1 0.05 ? 45% perf-profile.children.cycles-pp.tcp_grow_window
0.19 ? 14% -0.1 0.08 ? 25% perf-profile.children.cycles-pp.perf_trace_sched_switch
0.19 ? 4% -0.1 0.09 ? 13% perf-profile.children.cycles-pp.bictcp_acked
0.21 -0.1 0.11 ? 13% perf-profile.children.cycles-pp.tcp_update_pacing_rate
0.24 -0.1 0.14 ? 11% perf-profile.children.cycles-pp.remove_wait_queue
0.15 ? 23% -0.1 0.05 ? 46% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.19 ? 3% -0.1 0.09 ? 12% perf-profile.children.cycles-pp.tcp_update_skb_after_send
0.16 ? 4% -0.1 0.06 ? 14% perf-profile.children.cycles-pp.__rdgsbase_inactive
0.17 ? 3% -0.1 0.07 ? 12% perf-profile.children.cycles-pp.bictcp_cwnd_event
0.18 ? 2% -0.1 0.09 ? 14% perf-profile.children.cycles-pp.rb_next
0.30 ? 2% -0.1 0.21 ? 5% perf-profile.children.cycles-pp.tcp_established_options
0.25 -0.1 0.16 ? 9% perf-profile.children.cycles-pp.ip_send_check
0.16 ? 4% -0.1 0.07 ? 18% perf-profile.children.cycles-pp.tcp_rate_gen
0.17 ? 2% -0.1 0.08 ? 16% perf-profile.children.cycles-pp.woken_wake_function
0.15 ? 3% -0.1 0.07 ? 18% perf-profile.children.cycles-pp.__enqueue_entity
0.14 ? 3% -0.1 0.06 ? 46% perf-profile.children.cycles-pp.perf_trace_buf_alloc
0.17 ? 4% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.ksoftirqd_running
0.21 ? 8% -0.1 0.13 ? 7% perf-profile.children.cycles-pp.clockevents_program_event
0.14 ? 3% -0.1 0.07 ? 14% perf-profile.children.cycles-pp.apparmor_socket_sock_rcv_skb
0.13 ? 12% -0.1 0.05 ? 48% perf-profile.children.cycles-pp.switch_ldt
0.15 ? 3% -0.1 0.08 ? 16% perf-profile.children.cycles-pp.tcp_rearm_rto
0.72 -0.1 0.64 perf-profile.children.cycles-pp.tcp_rcv_space_adjust
0.27 ? 3% -0.1 0.20 ? 3% perf-profile.children.cycles-pp.import_single_range
0.17 ? 4% -0.1 0.09 ? 13% perf-profile.children.cycles-pp.__sock_wfree
0.31 ? 6% -0.1 0.24 ? 6% perf-profile.children.cycles-pp.mod_timer
0.34 ? 2% -0.1 0.27 ? 4% perf-profile.children.cycles-pp.dst_release
0.16 ? 4% -0.1 0.09 ? 11% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.12 ? 3% -0.1 0.05 ? 46% perf-profile.children.cycles-pp.perf_trace_buf_update
0.12 ? 3% -0.1 0.05 ? 49% perf-profile.children.cycles-pp.skb_csum_hwoffload_help
0.15 ? 7% -0.1 0.08 ? 16% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.17 ? 3% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.__copy_skb_header
0.22 ? 2% -0.1 0.15 ? 4% perf-profile.children.cycles-pp.sock_put
0.14 ? 4% -0.1 0.07 ? 12% perf-profile.children.cycles-pp._raw_spin_unlock_bh
0.58 -0.1 0.52 ? 2% perf-profile.children.cycles-pp.sock_recvmsg
0.15 ? 4% -0.1 0.09 ? 20% perf-profile.children.cycles-pp.kmalloc_slab
0.11 ? 5% -0.1 0.05 ? 48% perf-profile.children.cycles-pp.check_cfs_rq_runtime
0.12 ? 3% -0.1 0.06 ? 17% perf-profile.children.cycles-pp.tcp_send_delayed_ack
1.76 -0.1 1.70 perf-profile.children.cycles-pp.sock_ioctl
0.09 ? 4% -0.1 0.04 ? 71% perf-profile.children.cycles-pp.skb_clone_tx_timestamp
0.12 ? 4% -0.1 0.06 ? 14% perf-profile.children.cycles-pp.update_irq_load_avg
0.18 ? 2% -0.1 0.12 ? 11% perf-profile.children.cycles-pp.tcp_recv_timestamp
0.10 ? 4% -0.1 0.04 ? 45% perf-profile.children.cycles-pp.apparmor_ipv4_postroute
0.24 ? 2% -0.1 0.19 ? 5% perf-profile.children.cycles-pp.__build_skb_around
0.23 ? 2% -0.1 0.18 ? 7% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.54 -0.1 0.49 ? 2% perf-profile.children.cycles-pp.__might_fault
0.15 ? 3% -0.1 0.10 ? 15% perf-profile.children.cycles-pp.rb_insert_color
0.18 ? 2% -0.1 0.13 ? 3% perf-profile.children.cycles-pp.kfree_skbmem
0.11 ? 4% -0.1 0.06 ? 14% perf-profile.children.cycles-pp.validate_xmit_xfrm
0.10 -0.1 0.05 ? 46% perf-profile.children.cycles-pp.tcp_chrono_stop
0.12 ? 3% -0.1 0.07 ? 21% perf-profile.children.cycles-pp.bictcp_cong_avoid
0.11 ? 3% -0.1 0.06 ? 6% perf-profile.children.cycles-pp.smpboot_thread_fn
0.11 ? 4% -0.1 0.06 ? 8% perf-profile.children.cycles-pp.__fdget
0.12 ? 5% -0.0 0.07 ? 10% perf-profile.children.cycles-pp.ret_from_fork
0.12 ? 5% -0.0 0.07 ? 10% perf-profile.children.cycles-pp.kthread
1.24 -0.0 1.19 perf-profile.children.cycles-pp.lock_sock_nested
0.52 -0.0 0.47 ? 2% perf-profile.children.cycles-pp.security_socket_recvmsg
0.12 ? 3% -0.0 0.08 ? 6% perf-profile.children.cycles-pp.tcp_rate_check_app_limited
0.11 ? 4% -0.0 0.07 ? 10% perf-profile.children.cycles-pp.tcp_check_space
0.07 ? 8% -0.0 0.03 ? 99% perf-profile.children.cycles-pp.inet_send_prepare
0.26 -0.0 0.22 ? 4% perf-profile.children.cycles-pp.rcu_nocb_flush_deferred_wakeup
0.08 ? 4% -0.0 0.04 ? 71% perf-profile.children.cycles-pp.place_entity
0.20 ? 3% -0.0 0.16 ? 5% perf-profile.children.cycles-pp.tcp_release_cb
0.15 ? 5% -0.0 0.10 ? 9% perf-profile.children.cycles-pp.inet_sendmsg
0.13 ? 2% -0.0 0.09 ? 10% perf-profile.children.cycles-pp.rb_erase
0.07 ? 6% -0.0 0.03 ? 70% perf-profile.children.cycles-pp.sk_free
0.09 ? 4% -0.0 0.05 ? 7% perf-profile.children.cycles-pp.run_ksoftirqd
0.08 ? 5% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.ip_copy_addrs
0.12 ? 4% -0.0 0.09 ? 8% perf-profile.children.cycles-pp.sock_rfree
0.08 ? 5% -0.0 0.05 ? 8% perf-profile.children.cycles-pp.sk_forced_mem_schedule
0.78 -0.0 0.75 perf-profile.children.cycles-pp.__fget_light
0.11 ? 8% -0.0 0.09 ? 7% perf-profile.children.cycles-pp.__tcp_ack_snd_check
0.10 ? 4% -0.0 0.08 ? 5% perf-profile.children.cycles-pp.skb_clone
0.08 ? 6% -0.0 0.06 ? 7% perf-profile.children.cycles-pp.tcp_fin
0.12 ? 3% +0.0 0.14 ? 3% perf-profile.children.cycles-pp.tcp_close
0.16 ? 2% +0.0 0.18 ? 3% perf-profile.children.cycles-pp.__list_add_valid
0.11 ? 3% +0.0 0.14 ? 5% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.11 ? 3% +0.0 0.14 ? 4% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.14 ? 3% +0.0 0.18 ? 2% perf-profile.children.cycles-pp.sock_close
0.14 ? 4% +0.0 0.18 ? 2% perf-profile.children.cycles-pp.__sock_release
0.07 ? 7% +0.0 0.10 ? 9% perf-profile.children.cycles-pp.drain_obj_stock
0.14 ? 3% +0.0 0.18 ? 2% perf-profile.children.cycles-pp.inet_release
0.05 ? 8% +0.0 0.10 ? 5% perf-profile.children.cycles-pp.ksys_read
0.21 ? 3% +0.1 0.27 ? 4% perf-profile.children.cycles-pp.tcp_queue_rcv
0.08 ? 4% +0.1 0.13 ? 7% perf-profile.children.cycles-pp.__sys_getsockopt
0.01 ?223% +0.1 0.06 ? 13% perf-profile.children.cycles-pp.ktime_get_update_offsets_now
0.00 +0.1 0.05 ? 7% perf-profile.children.cycles-pp.ip_route_output_flow
0.04 ? 45% +0.1 0.10 ? 4% perf-profile.children.cycles-pp.vfs_read
0.34 ? 2% +0.1 0.39 ? 3% perf-profile.children.cycles-pp.__list_del_entry_valid
0.00 +0.1 0.05 ? 8% perf-profile.children.cycles-pp.tcp_validate_incoming
0.08 ? 5% +0.1 0.14 ? 6% perf-profile.children.cycles-pp.__x64_sys_getsockopt
0.00 +0.1 0.05 ? 9% perf-profile.children.cycles-pp.inet_create
0.03 ? 99% +0.1 0.08 ? 4% perf-profile.children.cycles-pp.tcp_conn_request
0.05 +0.1 0.11 ? 8% perf-profile.children.cycles-pp.__alloc_file
0.05 +0.1 0.11 ? 6% perf-profile.children.cycles-pp.alloc_empty_file
0.03 ? 70% +0.1 0.09 ? 10% perf-profile.children.cycles-pp.tcp_check_req
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.do_open
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.walk_component
0.09 ? 5% +0.1 0.15 ? 5% perf-profile.children.cycles-pp.__sys_setsockopt
0.00 +0.1 0.06 ? 6% perf-profile.children.cycles-pp.__sk_destruct
0.09 ? 5% +0.1 0.15 ? 7% perf-profile.children.cycles-pp.__x64_sys_setsockopt
0.02 ?141% +0.1 0.08 ? 7% perf-profile.children.cycles-pp.seq_read
0.00 +0.1 0.06 ? 7% perf-profile.children.cycles-pp.alloc_file_pseudo
0.00 +0.1 0.06 ? 7% perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
0.03 ? 70% +0.1 0.10 ? 3% perf-profile.children.cycles-pp.__dentry_kill
0.00 +0.1 0.07 ? 7% perf-profile.children.cycles-pp.sock_alloc_file
0.00 +0.1 0.07 ? 7% perf-profile.children.cycles-pp.ip_route_output_key_hash
0.00 +0.1 0.07 ? 13% perf-profile.children.cycles-pp.autoremove_wake_function
0.00 +0.1 0.07 ? 10% perf-profile.children.cycles-pp.lookup_fast
0.53 +0.1 0.60 ? 2% perf-profile.children.cycles-pp.tcp_send_mss
0.01 ?223% +0.1 0.08 ? 4% perf-profile.children.cycles-pp.seq_read_iter
0.00 +0.1 0.07 ? 12% perf-profile.children.cycles-pp.do_dentry_open
0.43 ? 2% +0.1 0.51 ? 2% perf-profile.children.cycles-pp.tcp_current_mss
0.06 ? 6% +0.1 0.13 ? 9% perf-profile.children.cycles-pp.__sys_socket
0.24 ? 6% +0.1 0.32 ? 7% perf-profile.children.cycles-pp.update_process_times
0.06 ? 6% +0.1 0.13 ? 7% perf-profile.children.cycles-pp.__x64_sys_socket
0.00 +0.1 0.08 ? 10% perf-profile.children.cycles-pp.do_tcp_setsockopt
0.72 ? 2% +0.1 0.80 ? 2% perf-profile.children.cycles-pp.aa_sk_perm
0.10 ? 6% +0.1 0.18 ? 9% perf-profile.children.cycles-pp.refill_obj_stock
0.00 +0.1 0.08 ? 8% perf-profile.children.cycles-pp.link_path_walk
0.00 +0.1 0.08 ? 13% perf-profile.children.cycles-pp.set_task_cpu
0.24 ? 6% +0.1 0.32 ? 6% perf-profile.children.cycles-pp.tick_sched_handle
0.14 ? 6% +0.1 0.22 ? 9% perf-profile.children.cycles-pp.scheduler_tick
0.10 ? 7% +0.1 0.18 ? 11% perf-profile.children.cycles-pp.task_tick_fair
0.00 +0.1 0.08 ? 7% perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
0.28 ? 3% +0.1 0.36 ? 5% perf-profile.children.cycles-pp.tick_sched_timer
0.00 +0.1 0.08 ? 8% perf-profile.children.cycles-pp.__x64_sys_accept
0.00 +0.1 0.08 ? 8% perf-profile.children.cycles-pp.dentry_open
0.00 +0.1 0.08 ? 11% perf-profile.children.cycles-pp.inet_csk_accept
0.00 +0.1 0.08 ? 11% perf-profile.children.cycles-pp.__x64_sys_accept4
0.74 +0.1 0.82 ? 2% perf-profile.children.cycles-pp.kmem_cache_free
0.00 +0.1 0.09 ? 12% perf-profile.children.cycles-pp.__zone_watermark_ok
0.12 ? 3% +0.1 0.21 ? 5% perf-profile.children.cycles-pp.tcp_connect
0.35 ? 3% +0.1 0.44 ? 4% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.00 +0.1 0.09 ? 7% perf-profile.children.cycles-pp.__sock_create
0.00 +0.1 0.09 ? 9% perf-profile.children.cycles-pp.inet_accept
0.00 +0.1 0.10 ? 9% perf-profile.children.cycles-pp.alloc_inode
0.00 +0.1 0.10 ? 9% perf-profile.children.cycles-pp.tcp_child_process
0.01 ?223% +0.1 0.11 ? 6% perf-profile.children.cycles-pp.__ns_get_path
0.37 ? 4% +0.1 0.47 ? 6% perf-profile.children.cycles-pp.security_socket_sendmsg
0.00 +0.1 0.10 ? 9% perf-profile.children.cycles-pp.new_inode_pseudo
0.08 +0.1 0.19 ? 7% perf-profile.children.cycles-pp.do_filp_open
0.08 +0.1 0.19 ? 7% perf-profile.children.cycles-pp.path_openat
0.07 ? 5% +0.1 0.19 ? 7% perf-profile.children.cycles-pp.kmem_cache_alloc
0.10 +0.1 0.21 ? 5% perf-profile.children.cycles-pp.open_related_ns
0.00 +0.1 0.12 ? 15% perf-profile.children.cycles-pp.skb_try_coalesce
0.09 ? 5% +0.1 0.21 ? 8% perf-profile.children.cycles-pp.do_sys_open
0.09 +0.1 0.21 ? 8% perf-profile.children.cycles-pp.do_sys_openat2
0.00 +0.1 0.12 ? 11% perf-profile.children.cycles-pp.prep_compound_page
0.17 ? 5% +0.1 0.29 ? 5% perf-profile.children.cycles-pp.tcp_v4_connect
0.00 +0.1 0.13 ? 12% perf-profile.children.cycles-pp.prep_new_page
0.00 +0.1 0.14 ? 10% perf-profile.children.cycles-pp.poll_idle
0.00 +0.1 0.14 ? 9% perf-profile.children.cycles-pp.tcp_add_backlog
0.00 +0.1 0.14 ? 13% perf-profile.children.cycles-pp.tcp_try_coalesce
0.26 ? 3% +0.1 0.40 ? 3% perf-profile.children.cycles-pp.__fput
0.00 +0.1 0.14 ? 14% perf-profile.children.cycles-pp.__slab_free
0.02 ?141% +0.1 0.16 ? 7% perf-profile.children.cycles-pp.__sys_accept4
0.27 ? 3% +0.2 0.42 ? 3% perf-profile.children.cycles-pp.task_work_run
0.01 ?223% +0.2 0.16 ? 8% perf-profile.children.cycles-pp.__sys_accept4_file
0.17 ? 2% +0.2 0.33 ? 6% perf-profile.children.cycles-pp.inet_shutdown
0.17 ? 2% +0.2 0.34 ? 5% perf-profile.children.cycles-pp.__x64_sys_shutdown
0.17 ? 2% +0.2 0.34 ? 5% perf-profile.children.cycles-pp.__sys_shutdown
0.12 ? 3% +0.2 0.30 ? 8% perf-profile.children.cycles-pp.tcp_rcv_synsent_state_process
0.45 ? 2% +0.2 0.63 ? 3% perf-profile.children.cycles-pp.simple_copy_to_iter
0.79 +0.2 0.99 perf-profile.children.cycles-pp.syscall_return_via_sysret
4.31 +0.2 4.53 perf-profile.children.cycles-pp.tcp_ack
0.06 ? 8% +0.2 0.28 ? 11% perf-profile.children.cycles-pp.tcp_tx_timestamp
0.27 ? 3% +0.2 0.50 ? 7% perf-profile.children.cycles-pp.rcu_core
0.24 ? 4% +0.2 0.48 ? 8% perf-profile.children.cycles-pp.rcu_do_batch
0.00 +0.3 0.26 ? 11% perf-profile.children.cycles-pp.__free_one_page
0.00 +0.3 0.27 ? 11% perf-profile.children.cycles-pp.schedule_idle
1.08 +0.3 1.35 ? 3% perf-profile.children.cycles-pp._raw_spin_lock_bh
0.16 ? 5% +0.3 0.45 ? 9% perf-profile.children.cycles-pp.irq_exit_rcu
0.00 +0.3 0.30 ? 19% perf-profile.children.cycles-pp.tcp_try_rmem_schedule
0.30 ? 2% +0.3 0.61 ? 6% perf-profile.children.cycles-pp.__inet_stream_connect
0.30 ? 3% +0.3 0.61 ? 6% perf-profile.children.cycles-pp.inet_stream_connect
0.31 ? 2% +0.3 0.62 ? 6% perf-profile.children.cycles-pp.__sys_connect
0.31 ? 2% +0.3 0.62 ? 6% perf-profile.children.cycles-pp.__x64_sys_connect
0.80 ? 3% +0.3 1.14 ? 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.76 ? 2% +0.3 1.10 ? 3% perf-profile.children.cycles-pp.__check_object_size
0.86 ? 3% +0.3 1.21 ? 5% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.10 ? 5% +0.4 0.45 ? 10% perf-profile.children.cycles-pp.iov_iter_advance
0.11 ? 6% +0.4 0.49 ? 8% perf-profile.children.cycles-pp.tcp_push
0.01 ?223% +0.4 0.44 ? 10% perf-profile.children.cycles-pp.intel_idle
0.20 ? 4% +0.6 0.78 ? 11% perf-profile.children.cycles-pp.tcp_data_queue
95.69 +0.6 96.27 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
0.01 ?223% +0.6 0.60 ? 11% perf-profile.children.cycles-pp.cpuidle_enter
0.01 ?223% +0.6 0.60 ? 11% perf-profile.children.cycles-pp.cpuidle_enter_state
0.02 ?223% +0.9 0.95 ? 11% perf-profile.children.cycles-pp.start_secondary
0.02 ?223% +0.9 0.96 ? 11% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
0.02 ?223% +0.9 0.96 ? 11% perf-profile.children.cycles-pp.cpu_startup_entry
0.02 ?223% +0.9 0.96 ? 11% perf-profile.children.cycles-pp.do_idle
2.68 +1.0 3.65 ? 2% perf-profile.children.cycles-pp.tcp_clean_rtx_queue
0.26 ? 3% +1.5 1.78 ? 9% perf-profile.children.cycles-pp.tcp_rcv_state_process
0.00 +2.2 2.25 ? 18% perf-profile.children.cycles-pp.__sk_mem_raise_allocated
0.00 +2.3 2.27 ? 18% perf-profile.children.cycles-pp.__sk_mem_schedule
2.33 +3.1 5.46 ? 6% perf-profile.children.cycles-pp._copy_to_iter
1.94 +3.2 5.10 ? 7% perf-profile.children.cycles-pp.copyout
3.13 +3.4 6.57 ? 6% perf-profile.children.cycles-pp.__skb_datagram_iter
3.16 +3.5 6.61 ? 6% perf-profile.children.cycles-pp.skb_copy_datagram_iter
38.80 +3.6 42.39 perf-profile.children.cycles-pp.tcp_sendmsg_locked
2.04 ? 7% +4.3 6.31 ? 8% perf-profile.children.cycles-pp.copyin
2.37 ? 6% +4.7 7.12 ? 7% perf-profile.children.cycles-pp._copy_from_iter_full
2.11 ? 4% +5.1 7.21 ? 8% perf-profile.children.cycles-pp.release_sock
1.25 ? 8% +5.1 6.37 ? 9% perf-profile.children.cycles-pp.__release_sock
86.46 +5.2 91.71 perf-profile.children.cycles-pp.do_syscall_64
42.80 +7.1 49.91 ? 2% perf-profile.children.cycles-pp.__x64_sys_sendto
42.60 +7.2 49.76 ? 2% perf-profile.children.cycles-pp.__sys_sendto
41.65 +7.2 48.89 ? 2% perf-profile.children.cycles-pp.sock_sendmsg
40.94 +7.3 48.19 ? 2% perf-profile.children.cycles-pp.tcp_sendmsg
3.87 ? 3% +7.4 11.29 ? 7% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
1.89 +11.2 13.07 ? 10% perf-profile.children.cycles-pp._raw_spin_lock
0.68 ? 2% +11.7 12.42 ? 10% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.01 ?223% +12.3 12.31 ? 11% perf-profile.children.cycles-pp.free_one_page
1.76 +12.4 14.17 ? 9% perf-profile.children.cycles-pp.__kfree_skb
0.01 ?223% +12.6 12.64 ? 11% perf-profile.children.cycles-pp.__free_pages_ok
0.01 ?223% +12.7 12.74 ? 11% perf-profile.children.cycles-pp.rmqueue
0.53 ? 3% +12.9 13.42 ? 10% perf-profile.children.cycles-pp.skb_release_data
0.01 ?223% +13.0 13.02 ? 11% perf-profile.children.cycles-pp.get_page_from_freelist
0.01 ?223% +13.1 13.09 ? 11% perf-profile.children.cycles-pp.__alloc_pages_nodemask
0.23 ? 10% +13.1 13.37 ? 11% perf-profile.children.cycles-pp.skb_page_frag_refill
0.30 ? 8% +13.2 13.48 ? 11% perf-profile.children.cycles-pp.sk_page_frag_refill
0.02 ?223% +24.1 24.14 ? 11% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.78 -1.2 0.61 ? 22% perf-profile.self.cycles-pp.update_curr
1.91 ? 2% -1.1 0.80 ? 16% perf-profile.self.cycles-pp.update_load_avg
1.76 -1.0 0.73 ? 18% perf-profile.self.cycles-pp.load_new_mm_cr3
1.76 -1.0 0.76 ? 17% perf-profile.self.cycles-pp.switch_fpu_return
2.06 -1.0 1.08 ? 10% perf-profile.self.cycles-pp.__tcp_transmit_skb
1.72 ? 2% -1.0 0.75 ? 16% perf-profile.self.cycles-pp.__schedule
1.84 -0.9 0.95 ? 4% perf-profile.self.cycles-pp._raw_spin_lock
0.98 -0.9 0.13 ? 12% perf-profile.self.cycles-pp.available_idle_cpu
1.22 -0.7 0.49 ? 14% perf-profile.self.cycles-pp.__switch_to_asm
1.11 -0.6 0.46 ? 12% perf-profile.self.cycles-pp.__ip_queue_xmit
1.13 -0.6 0.51 ? 13% perf-profile.self.cycles-pp.__netif_receive_skb_core
0.84 -0.5 0.29 ? 19% perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
0.84 -0.5 0.32 ? 19% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.87 -0.5 0.36 ? 15% perf-profile.self.cycles-pp.__switch_to
0.88 ? 2% -0.5 0.37 ? 16% perf-profile.self.cycles-pp.switch_mm_irqs_off
1.03 ? 2% -0.5 0.52 ? 11% perf-profile.self.cycles-pp.ip_finish_output2
0.69 -0.5 0.20 ? 17% perf-profile.self.cycles-pp.select_idle_cpu
0.78 -0.5 0.29 ? 14% perf-profile.self.cycles-pp.__update_load_avg_se
0.85 -0.5 0.38 ? 14% perf-profile.self.cycles-pp.native_sched_clock
0.97 -0.5 0.51 ? 9% perf-profile.self.cycles-pp.tcp_v4_rcv
0.70 ? 3% -0.4 0.28 ? 18% perf-profile.self.cycles-pp.enqueue_task_fair
0.85 -0.4 0.43 ? 9% perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
0.65 -0.4 0.28 ? 15% perf-profile.self.cycles-pp.ip_output
0.80 ? 2% -0.4 0.42 ? 12% perf-profile.self.cycles-pp.tcp_clean_rtx_queue
0.65 -0.4 0.28 ? 21% perf-profile.self.cycles-pp.pick_next_task_fair
0.62 ? 2% -0.4 0.25 ? 18% perf-profile.self.cycles-pp.enqueue_entity
0.84 -0.4 0.47 ? 11% perf-profile.self.cycles-pp.tcp_ack
0.65 -0.4 0.28 ? 12% perf-profile.self.cycles-pp.__softirqentry_text_start
0.54 ? 3% -0.4 0.18 ? 27% perf-profile.self.cycles-pp.set_next_buddy
0.68 -0.4 0.32 ? 15% perf-profile.self.cycles-pp.net_rx_action
0.50 -0.3 0.15 ? 5% perf-profile.self.cycles-pp.update_rq_clock
0.50 ? 2% -0.3 0.17 ? 14% perf-profile.self.cycles-pp.reweight_entity
0.47 ? 2% -0.3 0.14 ? 20% perf-profile.self.cycles-pp.pick_next_entity
0.93 -0.3 0.61 ? 5% perf-profile.self.cycles-pp.tcp_recvmsg_locked
0.89 ? 4% -0.3 0.58 ? 6% perf-profile.self.cycles-pp.__dev_queue_xmit
0.76 -0.3 0.46 ? 10% perf-profile.self.cycles-pp.__local_bh_enable_ip
0.54 ? 2% -0.3 0.23 ? 16% perf-profile.self.cycles-pp.select_task_rq_fair
0.62 -0.3 0.32 ? 11% perf-profile.self.cycles-pp.process_backlog
0.93 ? 4% -0.3 0.64 ? 6% perf-profile.self.cycles-pp.update_cfs_group
0.55 -0.3 0.28 ? 8% perf-profile.self.cycles-pp.__kmalloc_node_track_caller
0.61 ? 2% -0.3 0.34 ? 9% perf-profile.self.cycles-pp.__ksize
0.59 -0.3 0.32 ? 11% perf-profile.self.cycles-pp.kmem_cache_alloc_node
0.52 ? 2% -0.3 0.24 ? 13% perf-profile.self.cycles-pp.ip_rcv_core
0.46 -0.3 0.19 ? 15% perf-profile.self.cycles-pp.tcp_schedule_loss_probe
0.41 ? 2% -0.3 0.14 ? 16% perf-profile.self.cycles-pp.finish_task_switch
0.44 ? 5% -0.3 0.17 ? 18% perf-profile.self.cycles-pp.ipv4_dst_check
0.53 -0.3 0.28 ? 9% perf-profile.self.cycles-pp.kfree
0.38 -0.3 0.12 ? 24% perf-profile.self.cycles-pp.clear_buddies
0.52 -0.3 0.26 ? 16% perf-profile.self.cycles-pp.___perf_sw_event
0.38 ? 2% -0.2 0.14 ? 23% perf-profile.self.cycles-pp.update_min_vruntime
0.47 ? 2% -0.2 0.23 ? 11% perf-profile.self.cycles-pp.loopback_xmit
0.56 -0.2 0.33 ? 8% perf-profile.self.cycles-pp.tcp_rcv_established
0.38 -0.2 0.17 ? 16% perf-profile.self.cycles-pp.dequeue_task_fair
0.38 ? 3% -0.2 0.17 ? 16% perf-profile.self.cycles-pp.__calc_delta
0.41 -0.2 0.20 ? 12% perf-profile.self.cycles-pp.wait_woken
0.35 ? 3% -0.2 0.14 ? 24% perf-profile.self.cycles-pp.dequeue_entity
0.33 -0.2 0.12 ? 19% perf-profile.self.cycles-pp.check_preempt_wakeup
0.38 -0.2 0.18 ? 15% perf-profile.self.cycles-pp.validate_xmit_skb
0.40 -0.2 0.20 ? 18% perf-profile.self.cycles-pp.schedule
0.64 -0.2 0.44 ? 6% perf-profile.self.cycles-pp.__inet_lookup_established
0.33 ? 2% -0.2 0.13 ? 14% perf-profile.self.cycles-pp.tcp_ack_update_rtt
0.37 ? 2% -0.2 0.17 ? 9% perf-profile.self.cycles-pp.enqueue_to_backlog
0.35 ? 6% -0.2 0.16 ? 19% perf-profile.self.cycles-pp.cpuacct_charge
0.98 -0.2 0.79 ? 3% perf-profile.self.cycles-pp.read_tsc
0.36 ? 2% -0.2 0.17 ? 18% perf-profile.self.cycles-pp.migrate_enable
0.34 ? 11% -0.2 0.16 ? 16% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
0.33 ? 2% -0.2 0.15 ? 18% perf-profile.self.cycles-pp.sk_wait_data
0.26 ? 3% -0.2 0.08 ? 16% perf-profile.self.cycles-pp.resched_curr
0.35 -0.2 0.17 ? 14% perf-profile.self.cycles-pp.migrate_disable
0.29 -0.2 0.11 ? 17% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.69 ? 5% -0.2 0.52 ? 4% perf-profile.self.cycles-pp.ktime_get
0.34 -0.2 0.17 ? 15% perf-profile.self.cycles-pp.__netif_receive_skb_one_core
0.53 ? 2% -0.2 0.36 ? 7% perf-profile.self.cycles-pp.__alloc_skb
0.31 ? 3% -0.2 0.15 ? 13% perf-profile.self.cycles-pp.ktime_get_with_offset
0.27 -0.2 0.11 ? 16% perf-profile.self.cycles-pp.ip_rcv_finish
0.26 ? 2% -0.2 0.11 ? 11% perf-profile.self.cycles-pp.set_next_entity
0.33 ? 3% -0.2 0.17 ? 14% perf-profile.self.cycles-pp.do_softirq
0.37 ? 2% -0.1 0.22 ? 10% perf-profile.self.cycles-pp.try_to_wake_up
0.27 -0.1 0.12 ? 17% perf-profile.self.cycles-pp.skb_entail
0.28 ? 3% -0.1 0.13 ? 14% perf-profile.self.cycles-pp.irqtime_account_irq
0.24 -0.1 0.10 ? 10% perf-profile.self.cycles-pp.tcp_event_data_recv
0.29 ? 3% -0.1 0.15 ? 13% perf-profile.self.cycles-pp.ip_rcv_finish_core
0.21 ? 2% -0.1 0.07 ? 16% perf-profile.self.cycles-pp.__usecs_to_jiffies
0.42 ? 3% -0.1 0.28 ? 6% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.24 ? 10% -0.1 0.10 ? 22% perf-profile.self.cycles-pp.eth_type_trans
0.30 ? 2% -0.1 0.17 ? 14% perf-profile.self.cycles-pp.dev_hard_start_xmit
0.28 ? 2% -0.1 0.15 ? 15% perf-profile.self.cycles-pp.__ip_finish_output
0.25 -0.1 0.11 ? 16% perf-profile.self.cycles-pp.__tcp_select_window
0.24 ? 2% -0.1 0.11 ? 8% perf-profile.self.cycles-pp.sk_filter_trim_cap
0.19 ? 5% -0.1 0.05 ? 13% perf-profile.self.cycles-pp.tcp_v4_inbound_md5_hash
0.27 ? 3% -0.1 0.14 ? 11% perf-profile.self.cycles-pp.inet_ehashfn
0.20 ? 2% -0.1 0.08 ? 16% perf-profile.self.cycles-pp.security_sock_rcv_skb
0.22 ? 5% -0.1 0.10 ? 15% perf-profile.self.cycles-pp.netif_rx_internal
0.23 ? 7% -0.1 0.11 ? 15% perf-profile.self.cycles-pp.netif_skb_features
0.24 ? 7% -0.1 0.12 ? 8% perf-profile.self.cycles-pp.tcp_event_new_data_sent
0.17 ? 3% -0.1 0.05 ? 45% perf-profile.self.cycles-pp.cpumask_next_wrap
0.29 -0.1 0.17 ? 8% perf-profile.self.cycles-pp.tcp_wfree
0.72 -0.1 0.60 ? 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.21 ? 4% -0.1 0.10 ? 16% perf-profile.self.cycles-pp.nf_hook_slow
0.34 ? 2% -0.1 0.22 ? 6% perf-profile.self.cycles-pp.__sys_recvfrom
0.22 ? 2% -0.1 0.11 ? 16% perf-profile.self.cycles-pp.schedule_timeout
0.22 -0.1 0.11 ? 13% perf-profile.self.cycles-pp.tcp_v4_fill_cb
0.19 ? 4% -0.1 0.08 ? 19% perf-profile.self.cycles-pp.skb_network_protocol
0.18 ? 4% -0.1 0.07 ? 17% perf-profile.self.cycles-pp.put_prev_entity
0.19 ? 4% -0.1 0.08 ? 22% perf-profile.self.cycles-pp._find_next_bit
0.15 ? 2% -0.1 0.05 ? 45% perf-profile.self.cycles-pp.tcp_grow_window
0.25 ? 7% -0.1 0.14 ? 11% perf-profile.self.cycles-pp.mod_timer
0.18 ? 13% -0.1 0.08 ? 22% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.21 ? 2% -0.1 0.11 ? 18% perf-profile.self.cycles-pp.__sk_dst_check
0.19 -0.1 0.09 ? 16% perf-profile.self.cycles-pp.tcp_options_write
0.18 ? 2% -0.1 0.08 ? 14% perf-profile.self.cycles-pp.raw_local_deliver
0.19 ? 2% -0.1 0.09 ? 16% perf-profile.self.cycles-pp.bictcp_acked
0.21 ? 2% -0.1 0.11 ? 13% perf-profile.self.cycles-pp.__ip_local_out
0.20 ? 2% -0.1 0.10 ? 15% perf-profile.self.cycles-pp.perf_tp_event
0.17 ? 2% -0.1 0.07 ? 17% perf-profile.self.cycles-pp.bpf_skops_write_hdr_opt
0.54 -0.1 0.44 ? 3% perf-profile.self.cycles-pp.__might_sleep
0.21 ? 2% -0.1 0.11 ? 13% perf-profile.self.cycles-pp.tcp_update_pacing_rate
0.15 ? 3% -0.1 0.05 ? 48% perf-profile.self.cycles-pp.kmalloc_reserve
0.16 ? 2% -0.1 0.07 ? 18% perf-profile.self.cycles-pp.bictcp_cwnd_event
0.20 ? 3% -0.1 0.10 ? 15% perf-profile.self.cycles-pp.ip_rcv
0.16 ? 3% -0.1 0.06 ? 14% perf-profile.self.cycles-pp.__rdgsbase_inactive
0.12 ? 3% -0.1 0.03 ?102% perf-profile.self.cycles-pp.check_preempt_curr
0.16 ? 3% -0.1 0.07 ? 17% perf-profile.self.cycles-pp.tcp_rate_gen
0.25 -0.1 0.16 ? 6% perf-profile.self.cycles-pp.__might_fault
0.24 -0.1 0.15 ? 9% perf-profile.self.cycles-pp.ip_send_check
0.21 ? 3% -0.1 0.12 ? 7% perf-profile.self.cycles-pp.tcp_queue_rcv
0.29 -0.1 0.20 ? 5% perf-profile.self.cycles-pp.tcp_established_options
0.26 ? 2% -0.1 0.18 ? 5% perf-profile.self.cycles-pp.tcp_recvmsg
0.16 ? 2% -0.1 0.08 ? 16% perf-profile.self.cycles-pp.woken_wake_function
0.65 -0.1 0.57 ? 2% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.13 ? 4% -0.1 0.05 ? 45% perf-profile.self.cycles-pp.__enqueue_entity
0.17 ? 2% -0.1 0.09 ? 13% perf-profile.self.cycles-pp.__sock_wfree
0.16 ? 2% -0.1 0.09 ? 15% perf-profile.self.cycles-pp.skb_release_head_state
0.13 ? 12% -0.1 0.05 ? 48% perf-profile.self.cycles-pp.switch_ldt
0.21 ? 2% -0.1 0.13 ? 4% perf-profile.self.cycles-pp.lock_sock_nested
0.15 ? 3% -0.1 0.07 ? 12% perf-profile.self.cycles-pp.rb_next
0.14 ? 3% -0.1 0.06 ? 16% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.12 ? 4% -0.1 0.05 ? 45% perf-profile.self.cycles-pp.tcp_update_skb_after_send
0.16 ? 5% -0.1 0.09 ? 18% perf-profile.self.cycles-pp.ksoftirqd_running
0.11 ? 6% -0.1 0.03 ? 70% perf-profile.self.cycles-pp.sched_clock_cpu
0.26 ? 4% -0.1 0.19 ? 3% perf-profile.self.cycles-pp.import_single_range
0.21 ? 2% -0.1 0.14 ? 8% perf-profile.self.cycles-pp.sock_def_readable
0.13 ? 3% -0.1 0.06 ? 14% perf-profile.self.cycles-pp.apparmor_socket_sock_rcv_skb
0.19 ? 4% -0.1 0.12 ? 10% perf-profile.self.cycles-pp.inet_recvmsg
0.37 -0.1 0.30 ? 4% perf-profile.self.cycles-pp.tcp_rcv_space_adjust
0.16 ? 2% -0.1 0.09 ? 20% perf-profile.self.cycles-pp.__copy_skb_header
0.15 ? 9% -0.1 0.08 ? 16% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.14 ? 3% -0.1 0.07 ? 14% perf-profile.self.cycles-pp.__wake_up_common
0.14 ? 5% -0.1 0.07 ? 17% perf-profile.self.cycles-pp.ip_finish_output
0.12 ? 4% -0.1 0.06 ? 13% perf-profile.self.cycles-pp.__tcp_send_ack
0.26 ? 2% -0.1 0.20 ? 4% perf-profile.self.cycles-pp.__skb_clone
0.10 ? 3% -0.1 0.04 ? 71% perf-profile.self.cycles-pp.sk_reset_timer
0.21 ? 2% -0.1 0.15 ? 3% perf-profile.self.cycles-pp.sock_put
0.20 -0.1 0.13 ? 8% perf-profile.self.cycles-pp.sock_sendmsg
0.14 ? 3% -0.1 0.07 ? 16% perf-profile.self.cycles-pp.rcu_read_unlock_strict
0.13 ? 2% -0.1 0.07 ? 10% perf-profile.self.cycles-pp.__napi_poll
0.09 -0.1 0.03 ?100% perf-profile.self.cycles-pp.apparmor_ipv4_postroute
0.15 ? 3% -0.1 0.09 ? 17% perf-profile.self.cycles-pp.ip_local_deliver
0.14 ? 4% -0.1 0.08 ? 24% perf-profile.self.cycles-pp.kmalloc_slab
0.31 ? 2% -0.1 0.25 ? 4% perf-profile.self.cycles-pp.dst_release
0.12 -0.1 0.06 ? 13% perf-profile.self.cycles-pp.tcp_send_delayed_ack
0.12 ? 3% -0.1 0.07 ? 11% perf-profile.self.cycles-pp._raw_spin_unlock_bh
0.12 -0.1 0.07 ? 14% perf-profile.self.cycles-pp.ip_local_deliver_finish
0.12 ? 4% -0.1 0.06 ? 14% perf-profile.self.cycles-pp.update_irq_load_avg
0.16 ? 3% -0.1 0.10 ? 4% perf-profile.self.cycles-pp.tcp_v4_do_rcv
0.12 ? 3% -0.1 0.07 perf-profile.self.cycles-pp.tcp_rate_check_app_limited
0.08 ? 4% -0.1 0.03 ?100% perf-profile.self.cycles-pp.ip_copy_addrs
0.09 ? 5% -0.1 0.04 ? 71% perf-profile.self.cycles-pp.tcp_chrono_stop
0.11 ? 4% -0.1 0.06 ? 14% perf-profile.self.cycles-pp.validate_xmit_xfrm
0.11 ? 4% -0.1 0.06 ? 14% perf-profile.self.cycles-pp.tcp_rearm_rto
0.15 ? 4% -0.1 0.10 ? 6% perf-profile.self.cycles-pp.sk_stream_alloc_skb
0.24 ? 5% -0.1 0.19 ? 3% perf-profile.self.cycles-pp.inet_ioctl
0.12 ? 4% -0.0 0.07 ? 18% perf-profile.self.cycles-pp.bictcp_cong_avoid
0.16 ? 6% -0.0 0.11 ? 6% perf-profile.self.cycles-pp.__x64_sys_recvfrom
0.07 ? 6% -0.0 0.03 ? 99% perf-profile.self.cycles-pp.sk_free
0.14 ? 3% -0.0 0.10 ? 15% perf-profile.self.cycles-pp.rb_insert_color
0.20 ? 5% -0.0 0.15 ? 4% perf-profile.self.cycles-pp.__x64_sys_sendto
0.16 -0.0 0.11 ? 11% perf-profile.self.cycles-pp.tcp_recv_timestamp
0.07 ? 6% -0.0 0.03 ?100% perf-profile.self.cycles-pp.place_entity
0.26 ? 2% -0.0 0.21 ? 5% perf-profile.self.cycles-pp.do_syscall_64
0.17 ? 2% -0.0 0.12 ? 4% perf-profile.self.cycles-pp.kfree_skbmem
0.11 -0.0 0.06 ? 11% perf-profile.self.cycles-pp.tcp_check_space
0.08 ? 4% -0.0 0.04 ? 71% perf-profile.self.cycles-pp.lock_sock_fast
0.12 ? 4% -0.0 0.08 ? 6% perf-profile.self.cycles-pp.tcp_sendmsg
0.64 ? 2% -0.0 0.60 perf-profile.self.cycles-pp.kmem_cache_free
0.19 ? 3% -0.0 0.15 ? 5% perf-profile.self.cycles-pp.tcp_release_cb
0.25 -0.0 0.21 ? 3% perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
0.34 -0.0 0.30 ? 3% perf-profile.self.cycles-pp.__sys_sendto
0.15 ? 3% -0.0 0.12 ? 9% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.12 ? 4% -0.0 0.09 ? 9% perf-profile.self.cycles-pp.__tcp_push_pending_frames
0.77 -0.0 0.73 perf-profile.self.cycles-pp.__fget_light
0.12 ? 3% -0.0 0.09 ? 6% perf-profile.self.cycles-pp.sock_rfree
0.12 ? 4% -0.0 0.08 ? 14% perf-profile.self.cycles-pp.rb_erase
0.18 ? 2% -0.0 0.15 ? 7% perf-profile.self.cycles-pp.do_vfs_ioctl
0.18 ? 3% -0.0 0.15 ? 3% perf-profile.self.cycles-pp.tcp_ioctl
0.10 -0.0 0.07 ? 10% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.08 -0.0 0.05 perf-profile.self.cycles-pp.sk_forced_mem_schedule
0.20 ? 2% -0.0 0.18 ? 5% perf-profile.self.cycles-pp.release_sock
0.16 ? 4% -0.0 0.13 ? 7% perf-profile.self.cycles-pp.__x64_sys_ioctl
0.10 ? 4% -0.0 0.08 ? 7% perf-profile.self.cycles-pp.tcp_mstamp_refresh
0.10 ? 5% -0.0 0.07 ? 6% perf-profile.self.cycles-pp.tcp_update_recv_tstamps
0.09 ? 5% -0.0 0.07 ? 8% perf-profile.self.cycles-pp.__tcp_ack_snd_check
0.10 ? 4% -0.0 0.08 perf-profile.self.cycles-pp.skb_clone
0.24 ? 3% -0.0 0.21 ? 3% perf-profile.self.cycles-pp.sock_ioctl
0.07 ? 6% -0.0 0.06 ? 8% perf-profile.self.cycles-pp.inet_sendmsg
0.07 ? 5% -0.0 0.05 ? 8% perf-profile.self.cycles-pp.security_socket_recvmsg
0.10 ? 3% -0.0 0.09 ? 4% perf-profile.self.cycles-pp.copyin
0.07 ? 7% -0.0 0.05 ? 8% perf-profile.self.cycles-pp.apparmor_socket_recvmsg
0.15 ? 2% +0.0 0.17 ? 3% perf-profile.self.cycles-pp.__list_add_valid
0.09 ? 5% +0.0 0.12 ? 8% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.11 ? 3% +0.0 0.14 ? 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.21 ? 3% +0.0 0.23 ? 4% perf-profile.self.cycles-pp._copy_to_iter
0.20 ? 2% +0.0 0.24 perf-profile.self.cycles-pp.tcp_current_mss
0.18 ? 2% +0.0 0.22 ? 3% perf-profile.self.cycles-pp.skb_page_frag_refill
0.08 ? 5% +0.1 0.14 ? 7% perf-profile.self.cycles-pp.sk_page_frag_refill
0.00 +0.1 0.06 ? 13% perf-profile.self.cycles-pp.ktime_get_update_offsets_now
0.70 +0.1 0.76 perf-profile.self.cycles-pp.tcp_write_xmit
0.00 +0.1 0.07 ? 11% perf-profile.self.cycles-pp.refill_obj_stock
0.00 +0.1 0.07 ? 10% perf-profile.self.cycles-pp.__release_sock
0.30 ? 2% +0.1 0.37 ? 2% perf-profile.self.cycles-pp.__list_del_entry_valid
1.03 +0.1 1.10 ? 2% perf-profile.self.cycles-pp._raw_spin_lock_bh
0.00 +0.1 0.07 ? 12% perf-profile.self.cycles-pp.__alloc_pages_nodemask
0.00 +0.1 0.08 ? 12% perf-profile.self.cycles-pp.drain_obj_stock
0.00 +0.1 0.09 ? 10% perf-profile.self.cycles-pp.__zone_watermark_ok
0.00 +0.1 0.10 ? 9% perf-profile.self.cycles-pp.kmem_cache_alloc
0.11 ? 5% +0.1 0.23 ? 6% perf-profile.self.cycles-pp._copy_from_iter_full
0.00 +0.1 0.12 ? 11% perf-profile.self.cycles-pp.prep_compound_page
0.00 +0.1 0.12 ? 9% perf-profile.self.cycles-pp.__free_one_page
0.00 +0.1 0.12 ? 15% perf-profile.self.cycles-pp.skb_try_coalesce
0.24 ? 3% +0.1 0.36 ? 4% perf-profile.self.cycles-pp.__skb_datagram_iter
0.00 +0.1 0.12 ? 10% perf-profile.self.cycles-pp.poll_idle
0.00 +0.1 0.14 ? 13% perf-profile.self.cycles-pp.__slab_free
0.00 +0.2 0.17 ? 10% perf-profile.self.cycles-pp.__free_pages_ok
0.05 +0.2 0.24 ? 12% perf-profile.self.cycles-pp.tcp_tx_timestamp
0.78 ? 2% +0.2 0.98 perf-profile.self.cycles-pp.syscall_return_via_sysret
0.50 +0.2 0.72 ? 4% perf-profile.self.cycles-pp.skb_release_data
0.09 ? 8% +0.3 0.40 ? 10% perf-profile.self.cycles-pp.iov_iter_advance
0.10 ? 4% +0.3 0.41 ? 8% perf-profile.self.cycles-pp.tcp_push
0.29 ? 4% +0.3 0.60 ? 5% perf-profile.self.cycles-pp.__check_object_size
0.00 +0.3 0.33 ? 10% perf-profile.self.cycles-pp.rmqueue
0.01 ?223% +0.4 0.44 ? 10% perf-profile.self.cycles-pp.intel_idle
1.28 ? 2% +1.5 2.78 ? 6% perf-profile.self.cycles-pp.tcp_sendmsg_locked
0.00 +2.2 2.20 ? 18% perf-profile.self.cycles-pp.__sk_mem_raise_allocated
3.82 ? 3% +7.3 11.12 ? 7% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.02 ?223% +24.1 24.14 ? 11% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



stress-ng.time.system_time

4700 +--------------------------------------------------------------------+
4600 |-+ O O O O O |
|O O O OO O O O OOO O OOO O OO O O O O OO |
4500 |-+ O O O O O O O O |
4400 |-+ |
| |
4300 |-+ |
4200 |-+ |
4100 |-+ |
| |
4000 |-+ |
3900 |-+ |
| ++.+ .++.+++.+ .+|
3800 |+.++.+++.++.+++.++.++.+++.++.+++.++.+++.++.+ +.++.+++ + |
3700 +--------------------------------------------------------------------+


stress-ng.time.percent_of_cpu_this_job_got

7800 +--------------------------------------------------------------------+
| OO O O O OO O O OO |
7600 |O+ OO OO O OO O OO OOO OO O O O O OO |
7400 |-+ O O O O |
| |
7200 |-+ |
| |
7000 |-+ |
| |
6800 |-+ |
6600 |-+ + + |
| :: + : |
6400 |+.++.+++.++.+++.++.++.+++.++.+ +.++.+++.++ ++.++.++.+++.++.+++.++.+|
| |
6200 +--------------------------------------------------------------------+


stress-ng.time.voluntary_context_switches

5e+08 +-----------------------------------------------------------------+
| |
4.5e+08 |+.+++.+++.+++.++.+++.+++.+++ ++.+++.+++ ++.+++.++.+++.+++.+++.+|
| + : + : |
4e+08 |-+ + + |
| |
3.5e+08 |-+ |
| |
3e+08 |-+ |
| |
2.5e+08 |-+ |
| O O |
2e+08 |O+O O O O O O O OOO OOO O O O O O O O O O |
| OO O O O OO OO O O |
1.5e+08 +-----------------------------------------------------------------+


stress-ng.time.involuntary_context_switches

5e+08 +-----------------------------------------------------------------+
| |
4.5e+08 |+.+++.+++.+++.++.+++.+++.+++ ++.+++.+++ ++.+++.++.+++.+++.+++.+|
4e+08 |-+ + : + : |
| + + |
3.5e+08 |-+ |
| |
3e+08 |-+ |
| |
2.5e+08 |-+ |
2e+08 |-+ O O |
|O O OOO OO O O O OOO OOO O O O O O O O O O |
1.5e+08 |-+ OO O O OO OO O O |
| |
1e+08 +-----------------------------------------------------------------+


stress-ng.sock.ops

1.8e+06 +-----------------------------------------------------------------+
1.7e+06 |-+ O O |
| O O O O O OO OO OO O OO OO |
1.6e+06 |O+ O O O O O O O OO O O O O O O O |
1.5e+06 |-+ O O |
| |
1.4e+06 |-+ |
1.3e+06 |-+ |
1.2e+06 |-+ |
| |
1.1e+06 |-+ |
1e+06 |-+ +. |
|+.++ +++.+++.++.+++.+++.+++. ++.+++.+++. ++.+++.++.+++.+++.+++.+|
900000 |-+ + + |
800000 +-----------------------------------------------------------------+


stress-ng.sock.ops_per_sec

30000 +-------------------------------------------------------------------+
| O |
28000 |-+OO O O O O OO O O O O O |
26000 |O+ O O O O O OO O OOO O O O OO OO OO |
| O O |
24000 |-+ |
| |
22000 |-+ |
| |
20000 |-+ |
18000 |-+ |
| |
16000 |+.++.+++.+++.++.++ +. + .+ .+ +. +. .+ +.++.+++.+++.++.+|
| +.+ + + +. ++ + + + +.++ + |
14000 +-------------------------------------------------------------------+


[*] bisect-good sample
[O] bisect-bad sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation

Thanks,
Oliver Sang


Attachments:
(No filename) (130.26 kB)
config-5.12.0-rc2-00031-gd619f7afd773 (175.56 kB)
job-script (8.15 kB)
job.yaml (5.61 kB)
reproduce (347.00 B)
Download all attachments