LinuxLists.cc - [PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group

2020-03-25 12:58:20

Subject: [PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group

A huge number of load imbalance was observed when the local group
type is group_fully_busy, and the average load of local group is
greater than the selected busiest group, so the imbalance calculation
returns a negative value actually. Fix this problem by comparing the
average load before local group type check.

Signed-off-by: Aubrey Li <[email protected]>
---
kernel/sched/fair.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c1217bf..c524369 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8862,17 +8862,17 @@ static struct sched_group *find_busiest_group(struct lb_env *env)
goto out_balanced;

/*
+ * If the local group is more loaded than the selected
+ * busiest group don't try to pull any tasks.
+ */
+ if (local->avg_load >= busiest->avg_load)
+ goto out_balanced;
+
+ /*
* When groups are overloaded, use the avg_load to ensure fairness
* between tasks.
*/
if (local->group_type == group_overloaded) {
- /*
- * If the local group is more loaded than the selected
- * busiest group don't try to pull any tasks.
- */
- if (local->avg_load >= busiest->avg_load)
- goto out_balanced;
-
/* XXX broken for overlapping NUMA groups */
sds.avg_load = (sds.total_load * SCHED_CAPACITY_SCALE) /
sds.total_capacity;
--
2.7.4

2020-03-25 13:45:04

by Phil Auld

[permalink] [raw]

Subject: Re: [PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group

Hi Aubrey,

On Wed, Mar 25, 2020 at 08:46:28PM +0800 Aubrey Li wrote:
> A huge number of load imbalance was observed when the local group
> type is group_fully_busy, and the average load of local group is
> greater than the selected busiest group, so the imbalance calculation
> returns a negative value actually. Fix this problem by comparing the
> average load before local group type check.
>
> Signed-off-by: Aubrey Li <[email protected]>
> ---
> kernel/sched/fair.c | 14 +++++++-------
> 1 file changed, 7 insertions(+), 7 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index c1217bf..c524369 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8862,17 +8862,17 @@ static struct sched_group *find_busiest_group(struct lb_env *env)
> goto out_balanced;
>
> /*
> + * If the local group is more loaded than the selected
> + * busiest group don't try to pull any tasks.
> + */
> + if (local->avg_load >= busiest->avg_load)
> + goto out_balanced;
> +
> + /*
> * When groups are overloaded, use the avg_load to ensure fairness
> * between tasks.
> */
> if (local->group_type == group_overloaded) {
> - /*
> - * If the local group is more loaded than the selected
> - * busiest group don't try to pull any tasks.
> - */
> - if (local->avg_load >= busiest->avg_load)
> - goto out_balanced;
> -
> /* XXX broken for overlapping NUMA groups */
> sds.avg_load = (sds.total_load * SCHED_CAPACITY_SCALE) /
> sds.total_capacity;
> --
> 2.7.4
>

I'm not sure about this. I think this patch will undo a good bit of the
benefit of the load balancer rework. Avg_load is really most useful
when things are overloaded. If we go back to looking at it here we may
fail to balance when needed.

There are cases where, due to group scheduler load scaling, local average
may be higher but have spare CPUs still whereas busiest may have extra
processes which be balanced.

Cheers,
Phil

--

2020-03-25 14:00:11

by Vincent Guittot

[permalink] [raw]

Subject: Re: [PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group

Le mercredi 25 mars 2020 ? 20:46:28 (+0800), Aubrey Li a ?crit :
> A huge number of load imbalance was observed when the local group
> type is group_fully_busy, and the average load of local group is
> greater than the selected busiest group, so the imbalance calculation
> returns a negative value actually. Fix this problem by comparing the

Do you see any wrong task migration ? because if imbalance is < 0, detach_tasks should return early

> average load before local group type check.
>
> Signed-off-by: Aubrey Li <[email protected]>
> ---
> kernel/sched/fair.c | 14 +++++++-------
> 1 file changed, 7 insertions(+), 7 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index c1217bf..c524369 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8862,17 +8862,17 @@ static struct sched_group *find_busiest_group(struct lb_env *env)
> goto out_balanced;
>
> /*
> + * If the local group is more loaded than the selected
> + * busiest group don't try to pull any tasks.
> + */
> + if (local->avg_load >= busiest->avg_load)

If local is not overloaded, local->avg_load is null because it has not been already computed.

You should better add such test in calculate_imbalance after we computed local->avg_load and set imbalance to NULL

something like:

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 9e544e702f66..62f0861cefc0 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9152,6 +9152,15 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s

sds->avg_load = (sds->total_load * SCHED_CAPACITY_SCALE) /
sds->total_capacity;
+
+ /*
+ * If the local group is more loaded than the selected
+ * busiest group don't try to pull any tasks.
+ */
+ if (local->avg_load >= busiest->avg_load) {
+ env->imbalance =0;
+ return;
+ }
}

/*

> + goto out_balanced;
> +
> + /*
> * When groups are overloaded, use the avg_load to ensure fairness
> * between tasks.
> */
> if (local->group_type == group_overloaded) {
> - /*
> - * If the local group is more loaded than the selected
> - * busiest group don't try to pull any tasks.
> - */
> - if (local->avg_load >= busiest->avg_load)
> - goto out_balanced;
> -
> /* XXX broken for overlapping NUMA groups */
> sds.avg_load = (sds.total_load * SCHED_CAPACITY_SCALE) /
> sds.total_capacity;
> --
> 2.7.4
>

2020-03-26 01:52:31

by Li, Aubrey

[permalink] [raw]

Subject: Re: [PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group

On 2020/3/25 21:43, Phil Auld wrote:
> Hi Aubrey,
>
> On Wed, Mar 25, 2020 at 08:46:28PM +0800 Aubrey Li wrote:
>> A huge number of load imbalance was observed when the local group
>> type is group_fully_busy, and the average load of local group is
>> greater than the selected busiest group, so the imbalance calculation
>> returns a negative value actually. Fix this problem by comparing the
>> average load before local group type check.
>>
>> Signed-off-by: Aubrey Li <[email protected]>
>> ---
>> kernel/sched/fair.c | 14 +++++++-------
>> 1 file changed, 7 insertions(+), 7 deletions(-)
>>
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index c1217bf..c524369 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -8862,17 +8862,17 @@ static struct sched_group *find_busiest_group(struct lb_env *env)
>> goto out_balanced;
>>
>> /*
>> + * If the local group is more loaded than the selected
>> + * busiest group don't try to pull any tasks.
>> + */
>> + if (local->avg_load >= busiest->avg_load)
>> + goto out_balanced;
>> +
>> + /*
>> * When groups are overloaded, use the avg_load to ensure fairness
>> * between tasks.
>> */
>> if (local->group_type == group_overloaded) {
>> - /*
>> - * If the local group is more loaded than the selected
>> - * busiest group don't try to pull any tasks.
>> - */
>> - if (local->avg_load >= busiest->avg_load)
>> - goto out_balanced;
>> -
>> /* XXX broken for overlapping NUMA groups */
>> sds.avg_load = (sds.total_load * SCHED_CAPACITY_SCALE) /
>> sds.total_capacity;
>> --
>> 2.7.4
>>
>
> I'm not sure about this. I think this patch will undo a good bit of the
> benefit of the load balancer rework. Avg_load is really most useful
> when things are overloaded. If we go back to looking at it here we may
> fail to balance when needed.
>
> There are cases where, due to group scheduler load scaling, local average
> may be higher but have spare CPUs still whereas busiest may have extra
> processes which be balanced.
>
In my case, local group is fully busy, and the selected busiest group is overloaded,
should we compute avg_idle for group_fully_busy as well?

Thanks,
-Aubrey

2020-03-26 01:59:07

by Li, Aubrey

[permalink] [raw]

Subject: Re: [PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group

On 2020/3/25 21:58, Vincent Guittot wrote:
> Le mercredi 25 mars 2020 à 20:46:28 (+0800), Aubrey Li a écrit :
>> A huge number of load imbalance was observed when the local group
>> type is group_fully_busy, and the average load of local group is
>> greater than the selected busiest group, so the imbalance calculation
>> returns a negative value actually. Fix this problem by comparing the
>
> Do you see any wrong task migration ? because if imbalance is < 0, detach_tasks should return early

I didn't see wrong task migration, in this case the busiest group has only one CPU intensive
workload running.

>
>> average load before local group type check.
>>
>> Signed-off-by: Aubrey Li <[email protected]>
>> ---
>> kernel/sched/fair.c | 14 +++++++-------
>> 1 file changed, 7 insertions(+), 7 deletions(-)
>>
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index c1217bf..c524369 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -8862,17 +8862,17 @@ static struct sched_group *find_busiest_group(struct lb_env *env)
>> goto out_balanced;
>>
>> /*
>> + * If the local group is more loaded than the selected
>> + * busiest group don't try to pull any tasks.
>> + */
>> + if (local->avg_load >= busiest->avg_load)
>
> If local is not overloaded, local->avg_load is null because it has not been already computed.
>
> You should better add such test in calculate_imbalance after we computed local->avg_load and set imbalance to NULL

This makes more sense, will refine the patch for this.

Thanks,
-Aubrey

>
> something like:
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 9e544e702f66..62f0861cefc0 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9152,6 +9152,15 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s
>
> sds->avg_load = (sds->total_load * SCHED_CAPACITY_SCALE) /
> sds->total_capacity;
> +
> + /*
> + * If the local group is more loaded than the selected
> + * busiest group don't try to pull any tasks.
> + */
> + if (local->avg_load >= busiest->avg_load) {
> + env->imbalance =0;
> + return;
> + }
> }
>
> /*
>
>
>> + goto out_balanced;
>> +
>> + /*
>> * When groups are overloaded, use the avg_load to ensure fairness
>> * between tasks.
>> */
>> if (local->group_type == group_overloaded) {
>> - /*
>> - * If the local group is more loaded than the selected
>> - * busiest group don't try to pull any tasks.
>> - */
>> - if (local->avg_load >= busiest->avg_load)
>> - goto out_balanced;
>> -
>> /* XXX broken for overlapping NUMA groups */
>> sds.avg_load = (sds.total_load * SCHED_CAPACITY_SCALE) /
>> sds.total_capacity;
>> --
>> 2.7.4
>>

2020-03-28 13:24:43

by Chen, Rong A

[permalink] [raw]

Subject: [sched/fair] 59901cb452: netperf.Throughput_Mbps -27.3% regression

Greeting,

FYI, we

commit: url:
in testcase: netperf
on test with following parameters:

ip: ipv4
runtime: 300s
nr_threads: 25%
cluster: cs-localhost
send_size: 5K
test: TCP_SENDFILE
cpufreq_governor: performance
ucode: 0x2000065

test-description: test-url: In addition
+---------------- | testcase: | test machine | test parameters | | | | | | | +---------------- | testcase: | test machine | test parameters | | | | +---------------- | testcase: | test machine | test parameters | | | | +---------------- | testcase: | test machine | test parameters | | | | +----------------

If you fix Reported-by:

Details are as below:
-----------------

To reproduce:

git clone bin/lkp run job.yaml

================= cluster/compiler/ cs-localhost/gc
commit:
9c40365a65 59901cb452
9c40365a65d62d7c ---------------- fail:runs | 1:4 :4 %stddev \ 18708 336757 8293 ± 2% 2291 5103 4704 695.32 431.25 ± 51% 2.466e+09 526188 ± 5% 53993 ± 3% 1482325 ± 2% 150508 3999 ± 8% 3999 ± 8% 10897 10898 2.208e+08 5894730 ± 24% 19279 ± 3% 35085456 ± 9% 4822644 ± 12% 76807 13500 ± 3% 76807 15013 ± 91% 13589 ± 5% 833769 754892 152278 ± 42% 1169341 1169341 1.58 ± 6% 1.58 ± 6% 3098435 1573 ± 16% 56.75 ± 11% 6543 ± 8% 574.25 ± 30% 8042 ± 19% 34409 ± 22% 25847 ± 30% 6123 ± 8% 854.50 ± 19% 18421 ± 7% 34409 ± 22% 21046 ± 47% 32169 ± 19% 6543 ± 8% 1001821 ± 9% 2299 ± 30% 32169 ± 19% 102778 ± 7% 137587 ± 22% 137493 ± 22% 52091 ± 42% 103405 ± 30% 6116 ± 8% 1050635 ± 8% 3416 ± 19% 73684 ± 7% 108980 ± 7% 81937 ± 6% 21615 ± 70% 5010947 ± 28% 45054 ± 5% 4691793 ± 34% 48190 ± 9% 5045331 ± 36% 51391 ± 23% 52040 ± 11% 37936 ± 50% 46988 ± 16% 5967380 ± 30% 6170696 ± 42% 6718083 ± 40% 6397847 ± 27% 57352 ± 5% 4947229 ± 28% 4902059 ± 41% 3.636e+08 106.50 ± 96% 24013 14701 ± 60% 106.00 ± 97% 23189 ± 24% 23527 ± 34% 13122 ± 50% 21974 ± 29% 28365 ± 52% 27230 ± 58% 21075 ± 63% 6718 ± 19% 6718 ± 19% 7806 ± 72% 7343 ± 23% 7343 ± 23% 7679 ± 84% 35789 ± 57% 23882 ± 72% 21256 ± 72% 73.50 ±145% 32631 ± 47% 29653 ± 29% 25421 ± 21% 14735 ± 54% 11691 ± 85% 6240 ± 32% 6240 ± 32% 6781 ±120% 12981 ± 64% 14238 ± 44% 107.00 ± 5% 389189 ± 4% 389189 ± 4% 1296392 ± 4% 1588 ± 29% 10.18 1.552e+10 3.48e+08 1.52 ± 28% 11586741 ± 29% 8.042e+08 1492579 ± 2% 1.22 9.616e+10 162.88 ± 9% 10318 ± 43% 10338031 2.437e+10 0.00 ± 5% 12952 ± 3% 1.401e+10 49.00 11130067 ± 2% 11481481 7.868e+10 7290 ± 2% 0.82 2680 95.89 1617319 ± 41% 53805 ± 29% 96.01 213867 ± 31% 6788 ± 7% 2680 10.22 1.44 ± 29% 1.22 9226 ± 34% 0.00 ± 3% 49.22 7072 ± 2% 0.82 96.27 96.53 9587 1.547e+10 3.468e+08 11548381 ± 29% 8.015e+08 1487561 ± 2% 9.583e+10 162.41 ± 9% 10302904 2.429e+10 12914 ± 3% 1.396e+10 11092764 ± 2% 11443703 7.841e+10 2671 1611848 ± 41% 53714 ± 29% 213258 ± 31% 6777 ± 7% 2671 2.365e+13 37.98 ± 2% 38.08 ± 2% 38.03 ± 2% 40.82 ± 2% 40.23 ± 2% 40.24 ± 2% 40.24 ± 2% 0.89 ± 3% 0.62 ± 4% 0.59 ± 5% 0.54 ± 4% 0.42 ± 57% 0.58 ± 5% 1.74 0.57 ± 4% 0.68 ± 2% 0.63 ± 4% 0.68 ± 4% 1.09 ± 4% 2.04 ± 2% 0.61 ± 5% 0.63 ± 6% 1.91 ± 2% 0.83 ± 4% 0.69 ± 2% 2.07 ± 6% 0.72 ± 3% 2.17 ± 6% 2.04 0.84 ± 3% 0.87 ± 3% 1.15 ± 3% 1.03 ± 3% 1.06 ± 3% 0.13 ±173% 0.13 ±173% 1.70 ± 4% 1.25 ± 3% 1.38 ± 3% 0.00 0.00 3.01 ± 2% 0.00 1.50 ± 3% 1.25 0.00 0.13 ±173% 1.60 ± 3% 0.00 3.15 ± 2% 1.20 ± 21% 1.66 ± 3% 0.00 0.00 0.00 1.41 0.00 0.00 1.30 ± 8% 1.82 ± 2% 2.09 ± 4% 1.33 ± 9% 0.98 ± 28% 2.00 3.62 ± 3% 1.57 ± 9% 1.57 ± 9% 0.00 2.21 ± 9% 2.56 ± 4% 2.33 ± 3% 2.57 ± 4% 4.25 ± 2% 3.06 ± 2% 3.09 3.83 ± 4% 6.54 ± 2% 6.57 ± 2% 2.85 ± 7% 4.88 ± 2% 3.58 4.26 ± 3% 3.34 3.74 4.67 ± 4% 7.08 7.22 ± 2% 7.26 ± 2% 7.66 8.00 ± 2% 8.20 ± 2% 8.45 ± 2% 8.32 ± 2% 9.22 9.33 9.37 9.48 9.67 9.87 10.15 10.35 10.48 10.46 10.55 10.55 15.87 ± 2% 16.60 ± 2% 16.70 ± 2% 17.05 ± 2% 17.48 ± 2% 20.28 20.33 20.76 20.80 19.00 ± 2% 19.36 ± 2% 19.67 ± 2% 25.77 ± 2% 26.29 ± 2% 28.65 ± 2% 29.77 ± 2% 54.03 54.22 37.98 ± 2% 38.62 ± 2% 38.62 ± 2% 40.82 ± 2% 40.82 ± 2% 40.82 ± 2% 40.24 ± 2% 0.05 0.08 0.11 ± 3% 0.07 ± 6% 0.05 0.06 ± 7% 0.07 ± 10% 0.10 ± 10% 0.09 ± 5% 0.16 ± 6% 0.05 ± 9% 0.11 ± 8% 0.06 ± 6% 0.09 ± 7% 0.05 ± 8% 0.12 ± 7% 0.11 ± 11% 0.20 ± 2% 0.05 ± 9% 0.23 ± 2% 0.08 ± 5% 0.07 ± 6% 0.11 ± 4% 0.09 ± 8% 0.07 ± 5% 0.18 ± 4% 0.12 0.12 ± 4% 0.11 ± 8% 0.10 ± 8% 0.06 0.04 ± 57% 0.04 ± 58% 0.10 ± 7% 0.08 ± 8% 0.05 ± 8% 0.05 0.06 ± 9% 0.16 ± 2% 0.14 ± 3% 0.06 0.11 ± 7% 0.15 ± 3% 0.07 0.07 ± 11% 0.03 ±100% 0.13 ± 3% 0.10 ± 5% 0.18 ± 4% 0.06 ± 9% 0.22 ± 3% 0.12 ± 4% 0.09 ± 7% 0.21 ± 4% 0.07 ± 10% 0.01 ±173% 0.21 ± 6% 0.05 ± 8% 0.32 ± 3% 0.00 0.08 ± 10% 0.00 0.03 ±100% 0.21 ± 3% 0.15 ± 5% 0.15 ± 3% 0.00 0.00 0.00 0.07 ± 6% 0.01 ±173% 0.15 ± 9% 0.09 ± 7% 0.07 ± 12% 0.07 ± 7% 0.04 ± 57% 0.00 0.00 0.00 0.08 ± 10% 0.26 0.00 0.00 0.00 0.28 0.07 ± 10% 0.00 0.11 ± 4% 0.35 0.12 ± 7% 0.09 ± 4% 0.24 ± 2% 0.20 ± 5% 0.00 0.00 0.09 0.42 0.24 ± 4% 0.21 ± 2% 0.44 0.04 ± 57% 0.01 ±173% 0.23 ± 6% 0.00 0.17 ± 7% 0.11 ± 17% 0.17 ± 4% 0.00 0.55 ± 4% 0.12 ± 4% 0.00 0.30 ± 8% 0.16 ± 2% 0.17 ± 4% 0.38 0.17 ± 2% 0.12 ± 4% 0.00 0.11 ± 6% 0.00 0.37 0.25 ± 3% 0.17 ± 6% 0.11 ± 6% 0.36 ± 3% 0.20 ± 2% 0.12 ± 5% 0.00 0.04 ± 57% 0.26 ± 4% 0.49 ± 3% 0.12 ± 5% 0.27 ± 8% 0.24 ± 8% 0.21 ± 2% 0.48 ± 4% 0.21 ± 3% 0.23 ± 3% 0.32 ± 3% 0.32 ± 4% 0.10 ± 11% 0.18 ± 10% 0.12 ± 10% 0.53 0.31 ± 3% 0.14 ± 5% 0.10 ± 4% 0.35 ± 2% 0.30 ± 8% 0.34 ± 2% 0.41 ± 5% 0.29 ± 2% 0.26 ± 2% 0.34 ± 5% 0.08 ± 13% 0.18 ± 8% 0.33 ± 4% 0.35 ± 5% 0.55 ± 4% 0.31 ± 6% 0.78 ± 4% 1.16 0.44 0.45 ± 2% 0.48 ± 5% 0.48 ± 3% 0.59 ± 5% 0.76 0.53 ± 2% 1.77 0.48 ± 4% 1.24 ± 4% 0.69 ± 2% 0.65 ± 5% 0.49 ± 5% 0.62 ± 5% 0.63 ± 2% 2.04 ± 2% 0.42 ± 10% 0.44 ± 4% 0.89 ± 2% 0.57 ± 3% 2.17 ± 2% 0.58 ± 3% 0.58 ± 3% 0.53 ± 4% 0.72 ± 2% 2.08 ± 6% 0.77 ± 6% 2.06 2.19 ± 6% 0.88 ± 3% 0.96 ± 2% 1.16 ± 3% 1.01 ± 2% 0.98 0.66 ± 4% 1.06 ± 3% 1.32 ± 3% 0.98 ± 2% 1.84 0.77 ± 2% 0.56 ± 11% 0.65 ± 12% 1.25 ± 3% 1.38 ± 3% 1.49 ± 3% 1.77 ± 3% 3.19 ± 2% 1.66 ± 3% 3.17 ± 2% 1.67 ± 3% 1.36 1.41 2.01 ± 2% 1.54 ± 2% 3.65 ± 3% 1.91 ± 3% 2.45 ± 2% 4.28 ± 2% 2.78 ± 7% 6.55 ± 2% 6.57 ± 2% 3.92 ± 4% 4.88 ± 2% 3.58 ± 6% 4.98 ± 3% 4.96 ± 3% 7.11 7.67 7.20 7.25 9.24 9.33 9.37 9.49 8.91 9.07 9.67 9.88 10.16 10.36 10.48 10.56 10.52 10.80 11.79 12.06 12.61 13.35 15.91 ± 2% 16.62 ± 2% 16.71 ± 2% 17.06 ± 2% 17.48 ± 2% 20.29 20.33 20.76 20.81 19.04 ± 2% 19.37 ± 2% 19.68 ± 2% 25.77 ± 2% 26.31 ± 2% 28.69 ± 2% 29.79 ± 2% 54.10 54.27 37.98 ± 2% 0.10 ± 4% 0.09 0.06 0.08 ± 5% 0.06 ± 6% 0.06 ± 9% 0.05 ± 9% 0.11 ± 6% 0.09 ± 7% 0.08 ± 5% 0.17 ± 4% 0.10 ± 5% 0.11 ± 11% 0.10 ± 5% 0.05 0.11 ± 4% 0.10 ± 9% 0.08 ± 6% 0.07 ± 10% 0.07 ± 6% 0.09 ± 9% 0.22 ± 3% 0.17 ± 4% 0.09 ± 4% 0.10 ± 4% 0.06 0.11 ± 7% 0.15 ± 5% 0.05 0.15 ± 3% 0.15 ± 4% 0.05 ± 9% 0.05 ± 8% 0.04 ± 57% 0.04 ± 57% 0.04 ± 58% 0.07 ± 10% 0.05 ± 8% 0.16 ± 2% 0.07 ± 7% 0.06 0.04 ± 57% 0.04 ± 58% 0.05 0.05 ± 8% 0.05 0.06 ± 9% 0.06 ± 6% 0.07 ± 6% 0.11 ± 4% 0.01 ±173% 0.04 ± 58% 0.12 ± 4% 0.09 ± 7% 0.20 ± 4% 0.08 ± 8% 0.07 ± 6% 0.32 ± 3% 0.23 ± 4% 0.21 ± 6% 0.01 ±173% 0.24 ± 2% 0.10 ± 8% 0.04 ± 58% 0.06 ± 9% 0.21 ± 3% 0.07 ± 5% 0.09 ± 5% 0.01 ±173% 0.00 0.00 0.00 0.00 0.00 0.33 0.17 0.14 ± 7% 0.07 ± 6% 0.00 0.00 0.00 0.08 ± 5% 0.15 ± 4% 0.15 ± 4% 0.09 ± 7% 0.14 ± 14% 0.07 ± 7% 0.07 ± 7% 0.00 0.01 ±173% 0.16 ± 5% 0.15 ± 5% 0.00 0.00 0.00 0.00 0.00 0.10 ± 4% 0.13 ± 8% 0.00 0.09 ± 7% 0.35 0.11 ± 17% 0.20 ± 2% 0.00 0.20 ± 3% 0.17 ± 7% 0.09 0.00 0.14 ± 5% 0.15 ± 7% 0.00 0.00 0.00 0.01 ±173% 0.53 ± 5% 0.22 ± 5% 0.13 ± 6% 0.00 0.00 0.29 0.28 ± 2% 0.11 ± 4% 0.00 0.16 ± 10% 0.14 ± 3% 0.10 ± 8% 0.00 0.00 0.14 0.13 ± 3% 0.08 ± 5% 0.11 ± 6% 0.13 ± 6% 0.00 0.12 ± 3% 0.00 0.52 0.15 ± 8% 0.36 0.16 ± 5% 0.10 ± 8% 0.40 0.26 ± 4% 0.35 ± 5% 0.28 ± 2% 0.13 ± 5% 0.26 ± 7% 0.28 0.27 ± 7% 0.40 ± 2% 0.35 ± 2% 0.44 ± 4% 0.48 ± 3% 0.29 ± 4% 0.21 ± 6% 0.46 ± 4% 0.38 ± 2% 0.41 0.20 ± 4% 0.35 ± 3% 0.29 ± 4% 0.24 ± 3% 0.09 ± 4% 0.33 0.38 ± 4% 0.17 ± 8% 0.60 0.28 ± 3% 0.28 ± 7% 0.15 ± 12% 0.70 ± 4% 0.20 ± 6% 0.52 ± 3% 0.25 ± 11% 0.26 ± 3% 0.45 ± 9% 1.00 ± 5% 0.33 ± 3% 0.73 ± 2% 1.15 1.77 ± 5% 0.54 ± 5% 0.51 ± 4% 0.59 0.47 ± 3% 0.93 ± 3% 0.59 2.04 ± 2% 0.34 ± 6% 0.86 ± 2% 2.17 ± 2% 0.74 ± 6% 1.08 1.67 ± 2% 0.81 ± 2% 0.72 ± 3% 3.17 ± 2% 3.36 3.89 ± 4%

20000 18000 | 16000 |-+ 14000 |-O O | 12000 |-+ 10000 |-+ 8000 |-+ | 6000 |-+ 4000 |-+ | 2000 |-+ 0 350000 300000 |-+ | 250000 |-O O | 200000 |-+ | 150000 |-+ | 100000 |-+ | 50000 |-+ | 0 2.5e+09 | | 2e+09 |-+ | O O | 1.5e+09 |-+ | 1e+09 |-+ | | 5e+08 |-+ | | 700 +---------- | O O 600 |-+ | 500 |-+ | 400 |-+ | 300 |-+ | 200 |-+ | 100 |-+ | 0 +---------- 5000 +--------- 4500 |.+..+.+.+ | 4000 |-+ 3500 |-+ | 3000 |-+ 2500 |-+ 2000 |-+ | 1500 |-+ 1000 |-+ | 500 |-+ 0 +--------- [*] bisect-good sample
[O] bisect-bad sample

***************** lkp-bdw-ex2: ================= cluster/compiler/ cs-localhost/gc
commit:
9c40365a65 59901cb452
9c40365a65d62d7c ---------------- fail:runs | :4 %stddev \ 8310 332426 13748 ± 2% 587.05 1.217e+09 763112 ± 5% 10662925 ± 10% 3.24 2.02 72.00 2290192 6.88e+09 ± 6% 3.412e+08 ± 2% 45203190 ± 48% 25963 66104 9019 ± 34% 1.565e+09 1.565e+09 9.044e+09 9.044e+09 39376 ± 22% 3360482 ± 13% 333186 ± 18% 39376 ± 22% 3360482 ± 13% 333186 ± 18% 54158 ± 49% 1726678 ± 2% 1387 ± 7% 460.02 ± 2% 1726688 ± 2% 14277 ± 28% 1784858 6522465 ± 11% 2464 ± 11% 7868 ± 7% 677.50 ± 50% 5910 ± 22% 4052 ± 9% 6752 ± 2% 2939 ± 17% 434.25 ± 54% 4467 ± 8% 6931 ± 22% 17688 ± 8% 4052 ± 9% 1.873e+08 ± 2% 1.872e+08 ± 2% 25632 ± 87% 25596 ± 87% 25632 ± 87% 23643 ± 22% 7871 ± 7% 7269 ± 24% 737950 ± 7% 2710 ± 50% 23643 ± 22% 98061 ± 8% 16278 ± 10% 16210 ± 9% 27728 ± 22% 6748 ± 2% 11640 ± 18% 593633 ± 3% 1736 ± 55% 27728 ± 22% 70762 ± 8% 17873 ± 8% 98490 ± 5% 102626 ± 86% 102494 ± 87% 71781 ±102% 102350 ± 87% 9418 ± 4% 9418 ± 4% 10164 10164 7263 ± 6% 7263 ± 6% 52454 ± 5% 820.50 ± 5% 52535 ± 5% 820.50 ± 5% 5691 ± 2% 5691 ± 2% 18729 18729 11365 11366 3857 ± 4% 3859 ± 4% 6370 ± 5% 6372 ± 5% 9525 ± 4% 9525 ± 4% 2720 ± 3% 909.25 ± 3% 2728 ± 3% 909.25 ± 3% 62.27 1.506e+10 1.93 2.872e+08 0.75 ± 11% 34087325 ± 11% 4.561e+09 2306970 2.02 1.472e+11 778.21 ± 23% 4566 ± 11% 23558623 2.475e+10 0.06 1.639e+10 32.24 ± 6% 17483265 ± 6% 36909400 ± 4% 7.295e+10 0.50 7892760 ± 14% 82559 ± 5% 85.39 ± 2% 4935653 ± 12% 62.52 1.91 0.75 ± 11% 2.02 4380 ± 12% 0.06 32.16 ± 6% 0.50 85.86 ± 2% 18033 1.501e+10 2.862e+08 33952691 ± 11% 4.544e+09 2298503 1.466e+11 794.78 ± 24% 23475449 2.466e+10 1.633e+10 17419953 ± 6% 36786011 ± 4% 7.268e+10 7861951 ± 14% 82432 ± 5% 4916204 ± 12% 2.195e+13 3037 ± 66% 3037 ± 66% 2687 ± 97% 2687 ± 97% 2179 ± 35% 2179 ± 35% 3006 ± 56% 1996 ± 27% 1996 ± 27% 2101 ± 92% 1580 ± 57% 1362 ± 14% 3530 ± 35% 1343 ± 43% 2221 ± 39% 2221 ± 39% 3578 ± 57% 3578 ± 57% 2935 ± 38% 3052 ± 84% 2687 ± 13% 4192 ± 70% 4192 ± 70% 3234 ± 75% 3234 ± 75% 3496 ± 55% 3496 ± 55% 3794 ± 51% 3794 ± 51% 4717 ± 56% 4717 ± 56% 4786 ± 13% 3500 ± 64% 3500 ± 64% 2748 ± 4% 2748 ± 4% 3896 ± 50% 3896 ± 50% 6332 ± 22% 6332 ± 22% 2798 ± 62% 7445 ± 30% 2500 ± 31% 3157 ± 15% 4402 ± 61% 6686 ± 71% 5143 ± 37% 5143 ± 37% 5837 ± 32% 5837 ± 32% 3557 ± 47% 3370 ± 37% 4376 ± 37% 2807 ± 64% 7502 ± 52% 3688 ± 67% 2911 ± 16% 2911 ± 16% 2318 ± 21% 2318 ± 21% 2279 ± 27% 2279 ± 27% 5185 ± 35% 4118 ± 34% 5165 ± 22% 3514 ± 65% 3514 ± 65% 4784 ± 50% 4784 ± 50% 2389 ± 60% 3639 ± 19% 3639 ± 19% 2414 ± 45% 2414 ± 45% 3940 ± 28% 4768 ± 51% 4768 ± 51% 3071 ± 51% 2672 ± 40% 644992 ± 20% 141316 ± 2% 38660 ± 4% 40648 135335 ± 2% 39708 ± 24% 357756 ± 92% 140033 ± 4% 39682 ± 2% 1162992 ± 22% 36929 ± 7% 39419 ± 15% 37471 ± 14% 38725 ± 4% 380987 ±112% 684082 ±108% 35779 ± 14% 136179 ± 4% 136482 ± 3% 37701 ± 13% 138037 ± 5% 135813 ± 4% 136752 ± 3% 140989 ± 2% 140477 ± 4% 140183 ± 3% 41010 ± 3% 34115 ± 11% 1225282 ± 94% 38414 ± 9% 137423 ± 4% 30810 ± 31% 138560 ± 3% 42022 ± 5% 973700 ± 95% 977062 ± 77% 547399 ±162% 64145 ±114% 39496 26478 ± 28% 1601031 ± 85% 133259 ± 2% 137106 ± 2% 102176 ± 68% 1500188 ± 32% 1456825 ± 48% 30211 ± 21% 1249314 ± 45% 32688 ± 12% 614013 ± 33% 856129 ± 54% 826453 ± 88% 572160 ± 58% 1198932 ± 26% 764186 ± 42% 1957332 ± 39% 25563 ± 24% 1904800 ± 68% 136554 1937742 ± 65% 1965972 ± 41% 138112 ± 5% 938187 ± 79% 1859081 ± 63% 1418766 ± 55% 31415 ± 16% 1608741 ± 92% 1185884 ± 49% 2280687 ± 64% 989103 ± 66% 1357266 ± 62% 1353050 ± 14% 1061562 ± 62% 135927 ± 4% 1219305 ± 63% 138945 ± 3% 1342914 ± 22% 1370885 ± 17% 38700 ± 6% 37200 ± 11% 2042132 ± 26% 1162850 ± 36% 1454966 ± 30% 29191 ± 28% 1982705 ± 30% 26016 ± 21% 2230453 ± 9% 23850 ± 15% 1324547 ± 34% 30624 ± 11% 392004 ±131% 716600 ± 98% 1.739e+08 35.60 38.93 38.93 38.91 39.06 36.62 36.69 7.72 ± 2% 7.62 ± 2% 7.53 ± 2% 7.26 ± 2% 7.24 ± 2% 7.16 ± 2% 7.18 ± 2% 7.06 ± 2% 6.98 ± 2% 6.87 ± 2% 6.71 ± 2% 6.57 ± 2% 6.51 ± 2% 6.62 ± 2% 6.58 ± 2% 10.09 ± 2% 8.06 5.28 ± 2% 5.26 4.95 ± 2% 8.95 ± 2% 4.45 ± 2% 4.45 4.31 4.20 4.17 8.12 7.75 3.15 ± 2% 7.22 ± 2% 7.06 ± 2% 4.18 ± 3% 4.17 ± 3% 1.42 1.39 3.76 ± 3% 2.45 3.66 ± 3% 3.63 ± 3% 1.92 1.75 1.08 1.56 1.46 0.96 0.72 ± 2% 0.93 0.65 ± 2% 0.65 ± 2% 0.92 ± 2% 0.87 ± 2% 0.76 ± 4% 0.87 ± 3% 0.85 ± 3% 1.00 ± 3% 0.98 ± 3% 0.96 ± 3% 2.75 ± 5% 2.07 ± 5% 2.21 ± 6% 1.90 ± 6% 1.75 ± 6% 1.99 1.85 1.82 ± 2% 1.81 23.40 58.47 58.27 16.29 16.19 16.10 12.86 14.14 13.93 13.93 7.65 ± 2% 6.70 ± 2% 6.50 ± 2% 42.95 7.73 42.69 5.90 ± 2% 5.67 ± 2% 7.36 41.99 6.82 5.18 ± 2% 41.20 4.89 ± 2% 40.83 5.99 5.96 5.97 7.63 7.33 5.58 ± 2% 5.26 ± 2% 35.73 38.93 39.06 39.06 39.06 36.82 36.81 17.84 ± 2% 7.99 ± 2% 15.83 7.83 ± 2% 7.69 ± 2% 7.33 ± 2% 7.25 ± 2% 7.19 ± 2% 7.07 ± 2% 6.98 ± 2% 6.87 ± 2% 7.43 ± 2% 6.71 ± 2% 6.63 ± 2% 6.59 ± 2% 6.52 ± 2% 6.57 ± 2% 8.00 10.09 ± 2% 5.45 4.45 4.32 4.21 3.16 ± 2% 4.19 ± 3% 4.17 ± 3% 3.77 ± 3% 2.45 3.66 ± 3% 1.93 1.75 1.33 1.22 1.57 1.46 1.42 1.39 1.07 ± 6% 0.83 ± 2% 1.84 ± 3% 0.56 ± 3% 1.00 0.53 ± 3% 0.74 ± 8% 0.92 ± 2% 0.67 ± 9% 0.77 ± 4% 0.73 0.85 0.84 1.10 0.54 ± 8% 0.89 0.55 ± 2% 0.65 ± 2% 0.64 0.87 ± 3% 0.64 0.58 0.40 0.66 ± 2% 0.41 ± 2% 0.31 ± 14% 1.00 ± 3% 0.41 ± 7% 0.99 ± 3% 0.28 ± 15% 0.39 ± 2% 0.42 ± 2% 0.22 ± 12% 0.41 ± 8% 0.39 ± 7% 0.45 0.48 0.41 0.32 ± 2% 0.32 ± 4% 0.47 ± 2% 0.36 0.36 ± 2% 0.32 0.48 0.43 0.23 ± 4% 0.35 ± 3% 0.41 ± 3% 0.23 ± 3% 0.25 ± 2% 0.21 ± 3% 0.15 ± 9% 0.22 ± 5% 0.46 ± 5% 0.52 ± 3% 0.29 ± 2% 0.36 ± 5% 0.29 0.16 ± 2% 0.36 ± 3% 0.15 ± 5% 0.16 ± 2% 0.60 ± 2% 0.30 0.26 ± 3% 0.27 ± 4% 0.16 ± 9% 0.13 ± 6% 0.22 0.15 ± 3% 0.12 ± 5% 0.17 ± 4% 0.11 ± 3% 0.22 ± 3% 0.14 ± 5% 0.14 ± 8% 0.25 ± 3% 0.15 ± 3% 0.22 ± 3% 0.44 ± 5% 0.18 ± 6% 0.12 ± 8% 0.11 ± 15% 0.07 ± 7% 0.14 ± 11% 0.23 ± 2% 0.19 ± 5% 0.24 ± 2% 0.17 ± 2% 0.08 ± 6% 0.06 ± 6% 0.17 ± 5% 0.16 ± 2% 0.19 0.17 0.18 ± 2% 0.13 ± 9% 0.36 0.20 ± 4% 0.11 ± 4% 0.07 ± 5% 0.07 ± 6% 0.07 ± 7% 0.30 0.20 ± 4% 0.14 0.14 ± 3% 0.10 ± 7% 0.09 ± 9% 0.12 ± 8% 0.09 ± 4% 0.13 ± 6% 0.08 ± 10% 0.12 ± 5% 0.14 ± 6% 0.08 ± 6% 0.08 ± 5% 0.09 ± 4% 0.14 ± 6% 0.10 ± 4% 0.08 ± 5% 0.11 ± 3% 0.09 ± 4% 0.09 0.08 0.07 0.06 0.07 ± 10% 0.05 ± 60% 0.01 ±173% 0.00 2.75 ± 5% 25.85 23.41 25.96 19.22 58.51 58.31 16.37 16.19 16.11 12.87 7.65 ± 2% 7.37 ± 2% 6.81 ± 2% 42.95 7.74 42.70 7.52 42.00 41.21 40.84 7.75 ± 2% 8.12 ± 2% 7.47 ± 2% 8.98 7.92 7.87 7.77 7.64 7.35 6.01 ± 2% 12.88 ± 2% 35.73 7.95 5.40 2.65 0.78 0.81 0.81 ± 2% 1.00 0.92 ± 2% 0.51 ± 4% 0.57 0.70 ± 3% 0.75 ± 2% 0.38 0.41 0.30 ± 14% 0.28 ± 15% 0.40 ± 2% 0.33 ± 2% 0.39 ± 2% 0.34 ± 13% 0.21 ± 13% 0.39 ± 5% 0.46 ± 2% 0.30 ± 2% 0.33 0.40 0.27 ± 5% 0.26 ± 7% 0.18 ± 3% 0.18 ± 2% 0.23 ± 4% 0.23 ± 6% 0.33 ± 3% 0.26 ± 2% 0.22 ± 3% 0.22 ± 5% 0.20 ± 4% 0.15 ± 3% 0.24 0.28 0.31 ± 2% 0.18 ± 10% 0.27 ± 2% 0.29 ± 2% 0.12 ± 7% 0.29 ± 6% 0.27 ± 4% 0.26 ± 7% 0.15 ± 10% 0.11 ± 3% 0.22 ± 9% 0.14 ± 5% 0.15 ± 3% 0.13 ± 13% 0.15 ± 4% 0.08 ± 12% 0.07 ± 7% 0.07 ± 7% 0.33 ± 2% 0.12 ± 5% 0.08 ± 6% 0.14 ± 3% 0.18 ± 7% 0.17 ± 5% 0.21 ± 5% 0.20 ± 2% 0.15 ± 3% 0.14 ± 8% 0.14 ± 3% 0.16 ± 4% 0.16 ± 2% 0.13 ± 9% 0.18 ± 2% 0.20 ± 4% 0.09 ± 7% 0.15 ± 4% 0.12 ± 7% 0.11 ± 4% 0.10 ± 7% 0.14 ± 3% 0.16 ± 2% 0.12 ± 4% 0.12 ± 5% 0.08 ± 5% 0.06 ± 6% 0.10 ± 7% 0.09 ± 4% 0.11 ± 4% 0.11 ± 4% 0.09 ± 4% 0.09 ± 4% 0.08 ± 5% 0.14 ± 3% 0.07 ± 5% 0.08 ± 6% 0.09 ± 10% 0.11 ± 3% 0.11 ± 4% 0.08 ± 5% 0.11 ± 4% 0.09 ± 4% 0.08 0.08 ± 5% 0.06 0.06 ± 7% 12.88 ± 2%

***************** lkp-skl-fpga01: ================= compiler/cpufreq_ gcc-7/performan
commit:
9c40365a65 59901cb452
9c40365a65d62d7c ---------------- %stddev \ 23098 474.19 474.19 20407 11653 6674 60218701 51504 ± 2% 45407 ± 4% 0.00 ±117% 1505 9163 992.75 ± 12% 535099 ± 2% 9165 133774 ± 2% 133774 ± 2% 1.24 ± 16% 0.94 ± 18% 0.11 ± 36% 1.12 ± 17% 842912 ± 13% 851323 ± 14% 8420 ±172% 780061 ± 13% 805331 ± 14% 92.00 ± 6% 11349 ± 4% 19448 44561 13330 ± 44% 5558 ±107% 51552 ± 33% 5345 ± 12% 1739692 1018 ± 7% 1018 ± 7% 4826 4826 2826 ± 2% 2828 ± 2% 1151 ± 2% 1151 ± 2% 59534 ± 2% 1490 ± 2% 59640 ± 2% 1490 ± 2% 7.107e+09 12.39 ± 15% 1477 111.88 1.099e+10 3.823e+09 2132180 6.802e+10 0.88 4076 107204 ± 2% 14357 ± 5% 26949 ± 4% 4076 12.57 ± 16% 1.04 0.96 88.19 78.98 536500 7.106e+09 1474 111.66 1.098e+10 3.823e+09 2128273 6.801e+10 4061 107032 ± 2% 14320 ± 5% 26906 ± 4% 4061 51615 ± 9% 89952 ± 7% 6590 ± 39% 21438 ± 5% 1649313 ± 8% 2846100 ± 7% 225248 ± 36% 677866 ± 5% 130.54 ± 7% 19.52 ± 32% 130.54 ± 7% 19.52 ± 32% -1075371 121453 ± 95% -2499411 677861 ± 5% 6.00 ± 7% 25.29 ± 7% 241338 ± 5% 241342 ± 5% 241333 ± 5% 2.47 ± 2% 241338 ± 5% 241342 ± 5% 241333 ± 5% 2.47 ± 2% 4932 ± 3% 3525 ± 5% 2933 ± 5% 19139 ± 27% 2773 ± 10% 1280 ± 5% 9370 ± 27% 1374 ± 10% 1302 ± 5% 1455 ± 4% 594.81 ± 4% 2311 ± 24% 345.75 ± 7% 241334 ± 5% 240831 ± 5% 241752 ± 5% 74.25 ±159% 105.75 ± 40% 105.75 ± 40% 159.00 ± 10% 1558 ±163% 1558 ±163% 179.25 ± 6% 143.00 ± 82% 143.00 ± 82% 173.50 ± 19% 193.75 ± 6% 3261 ±140% 398.25 ±110% 146.25 ±160% 372.25 ± 79% 101.50 ±107% 98.75 ± 21% 94.50 ± 38% 100.25 ± 38% 81.50 ± 18% 88.75 ± 45% 71.25 ± 26% 65.25 ± 28% 45.75 ± 12% 75.00 ± 21% 618.25 ±144% 618.25 ±144% 80.25 ± 16% 861.75 ± 99% 861.75 ± 99% 69.25 ± 17% 96.25 ± 50% 1133 ± 91% 1133 ± 91% 617.00 ±146% 617.00 ±146% 617.00 ±146% 617.00 ±146% 81.00 ± 55% 3679 ± 63% 3679 ± 63% 3501 ± 37% 3501 ± 37% 4911 ± 6% 4911 ± 6% 4086 ± 36% 4086 ± 36% 245.25 ± 44% 63.00 ± 71% 2334 ± 51% 2334 ± 51% 132.75 ± 21% 128.25 ± 27% 3236 ±143% 2481 ± 43% 2481 ± 43% 132.50 ± 9% 131.50 ± 33% 1570 ± 63% 1570 ± 63% 131.75 ± 13% 769.50 ±130% 147.50 ± 19% 180.50 ± 9% 164.50 ± 20% 1573 ±162% 1573 ±162% 148.25 ± 20% 143.75 ± 8% 168.75 ± 6% 2281 ±106% 2281 ±106% 163.75 ± 17% 84.75 ± 27% 84.75 ± 27% 176.25 ± 8% 164.25 ± 8% 242630 ± 11% 242630 ± 11% 3842 ± 14% 39213 ± 2% 112479 ± 3% 47544 ± 5% 115525 ± 5% 42079 ± 4% 115645 ± 4% 42390 ± 5% 113237 ± 5% 43877 ± 3% 115048 ± 4% 42057 ± 2% 50287 ± 2% 51055 ± 3% 111557 ± 4% 51201 ± 8% 106509 ± 4% 106554 ± 5% 102888 ± 3% 102645 ± 4% 103074 ± 3% 40123 ± 2% 105947 ± 4% 56698 ± 3% 107845 ± 4% 104635 ± 2% 57459 ± 2% 107558 ± 4% 103404 ± 2% 57845 103795 ± 3% 116855 ± 3% 40401 ± 3% 117905 ± 5% 119161 155833 ± 4% 120298 ± 6% 165459 ± 5% 41776 ± 2% 111332 ± 3% 109782 ± 4% 163967 ± 3% 110345 ± 4% 110912 ± 4% 111899 ± 5% 164801 ± 4% 110069 ± 3% 166154 ± 2% 107927 ± 3% 167316 ± 2% 107149 ± 3% 52823 ± 5% 172148 ± 2% 106831 ± 4% 169088 ± 3% 109008 ± 4% 52842 ± 4% 168352 ± 2% 43344 ± 5% 109019 ± 4% 53829 ± 2% 107752 ± 9% 57665 104372 ± 4% 56657 176519 ± 2% 102140 ± 4% 57875 ± 2% 177283 ± 3% 102750 ± 4% 57039 ± 5% 180258 ± 3% 103951 ± 3% 57950 ± 2% 180059 ± 2% 106543 ± 4% 58936 ± 3% 175875 ± 2% 107599 ± 4% 61135 ± 11% 175037 ± 3% 104196 ± 4% 56551 ± 6% 178334 ± 5% 105321 ± 4% 176284 ± 2% 43643 ± 3% 104189 ± 4% 58720 ± 2% 180182 ± 3% 105511 ± 3% 106737 ± 4% 108068 ± 4% 50659 ± 3% 110776 ± 5% 49820 ± 3% 108892 ± 4% 48807 ± 3% 109303 ± 4% 48744 ± 4% 107665 ± 4% 46939 ± 4% 107899 ± 5% 47890 ± 6% 112252 ± 5% 45201 ± 5% 43271 ± 3% 42947 ± 4% 41118 ± 3% 43558 ± 9% 42404 ± 5% 44782 ± 6% 45270 ± 8% 46135 ± 6% 114120 ± 4% 43859 ± 4% 45007 ± 4% 46279 ± 4% 47194 ± 7% 45037 ± 4% 44473 ± 7% 116513 ± 5% 43065 ± 8% 114917 ± 4% 44462 114722 ± 3% 46507 ± 2% 109126 ± 4% 52943 ± 4% 113274 ± 2% 49330 ± 4% 114525 ± 4% 45993 ± 5% 107326 ± 4% 51636 ± 4% 103564 ± 4% 54621 ± 3% 106692 ± 5% 107429 ± 4% 48429 ± 3% 109037 ± 3% 111642 ± 5% 110949 ± 5% 110815 ± 5% 111916 ± 5% 42154 ± 5% 111872 ± 4% 47628 ± 3% 112803 ± 5% 45236 ± 5% 117379 ± 5% 116981 ± 3% 113561 ± 5% 45469 ± 3% 112230 ± 5% 113604 ± 4% 45422 ± 4% 112933 ± 4% 44320 ± 6% 113319 ± 5% 43834 ± 2% 116386 ± 6% 41001 ± 8% 116165 ± 5% 41299 ± 5% 11518008 ± 4% 17583486 ± 3%

***************** lkp-skl-fpga01: ================= compiler/cpufreq_ gcc-7/performan
commit:
9c40365a65 59901cb452
9c40365a65d62d7c ---------------- fail:runs | 1:4 %stddev \ 3115 15433 ± 2% 2114 7797 477.52 3.69e+08 4.873e+08 747856 ± 6% 1.40 181.50 ± 14% 45.00 ± 14% 76.00 3747208 237644 2932 ± 2% 2932 ± 2% 1160 ± 8% 1160 ± 8% 7.874e+09 8.948e+09 ± 82% 28927686 ± 31% 32968312 3321066 77470 44723 77470 11008 ± 26% 1806 ± 10% 10143 ± 5% 3687542 ± 3% 1247933 ± 4% 73039 46502 10133 ± 6% 12925 ± 8% 3687542 ± 3% 1247933 ± 4% 3425325 2191093 471746 ± 6% 0.24 ± 26% 2.64 ± 36% 0.51 ± 26% 1437 ± 3% 375.43 ± 2% -96957 471746 ± 6% 858.93 ± 5% 649.39 ± 6% 442164 ± 3% 514800 ± 4% 1444 ±148% 0.00 ± 3% 1.04 ± 5% 0.41 8064604 ± 2% 915034 ± 6% 8061279 ± 2% 914923 ± 6% 4029443 ± 2% 457354 ± 6% 3230031 4032123 457261 ± 6% 497.05 1512 354.64 ± 4% 130.66 ± 6% 97364 ± 2% 95732 98020 134854 ± 2% 100296 ± 4% 98665 ± 4% 101562 ± 2% 103241 102016 103072 ± 2% 103642 136281 104080 104161 103794 ± 2% 104881 106249 105901 106109 104208 ± 3% 105449 106650 110131 107950 ± 3% 110414 107954 ± 2% 107635 107434 ± 2% 106151 ± 2% 104935 ± 5% 103760 ± 2% 103250 ± 4% 104851 ± 2% 104007 101221 103382 ± 2% 99759 ± 3% 99723 99706 ± 2% 100802 99622 ± 2% 98714 ± 2% 99272 97699 ± 2% 30.70 ± 17% 6.229e+09 87521494 ± 2% 26161990 7.094e+08 3766575 3.51 ± 8% 8.561e+10 1199 ± 10% +12547.8% 3326 ± 8% 4368880 ± 5% 7.644e+09 83528 ± 41% 4.5e+09 4412580 29081328 2.835e+10 0.31 ± 4% 94.83 4078999 146125 95.95 3822185 11551 ± 3% 25.02 1.41 3.69 3.02 3272 0.06 ± 5% 0.00 ± 41% 13.18 6425 0.33 22735 6.21e+09 87263324 ± 2% 26081704 7.072e+08 3754979 8.535e+10 1197 ± 10% +12540.5% 4356788 ± 5% 7.621e+09 83463 ± 41% 4.487e+09 4399472 28993919 2.826e+10 4066345 145705 3810301 11533 ± 3% 1.108e+13 49518 75573 ± 2% 293.50 ± 46% 1651 ± 43% 107991 ± 2% 53.00 ± 95% 104860 ± 3% 101815 ± 3% 104224 ± 3% 3026 ± 8% 3026 ± 8% 97748 ± 3% 106274 ± 2% 109794 ± 3% 110641 ± 4% 112578 ± 4% 203.50 ±101% 115653 ± 3% 2336 ± 30% 2336 ± 30% 113166 ± 4% 117545 ± 6% 119816 ± 5% 208.00 ±115% 116298 ± 5% 118.50 ± 93% 471.50 ± 54% 119931 ± 4% 119193 ± 3% 119705 ± 4% 119224 239.50 ± 82% 114268 ± 5% 130.75 ± 54% 114570 ± 2% 401.00 ± 34% 2215 ± 26% 1056 ± 71% 94273 ± 4% 438.75 ± 54% 297.75 ± 52% 99323 ± 4% 101813 ± 4% 104108 ± 4% 108408 ± 6% 107851 ± 7% 3050 ± 5% 3050 ± 5% 110860 ± 5% 2994 ± 5% 2994 ± 5% 110701 ± 6% 2950 ± 3% 2950 ± 3% 114683 ± 2% 2959 ± 3% 2959 ± 3% 112794 ± 3% 96299 ± 6% 3084 ± 6% 3084 ± 6% 115740 ± 6% 336.75 ± 73% 3109 ± 5% 3109 ± 5% 117515 ± 3% 3074 ± 12% 3074 ± 12% 118729 ± 3% 3026 ± 7% 3026 ± 7% 116750 ± 3% 2999 ± 13% 2999 ± 13% 119184 119035 ± 2% 122760 ± 3% 122662 ± 2% 123349 ± 2% 123.00 ±107% 123038 ± 2% 174.50 ± 70% 94491 ± 3% 120948 ± 2% 121202 ± 3% 127597 ± 3% 122288 ± 4% 119757 ± 3% 232.50 ± 97% 114657 ± 4% 91.50 ±132% 112537 ± 6% 180.00 ± 94% 110747 ± 4% 309.25 ± 99% 108297 96801 ± 2% 104449 ± 5% 150.50 ± 85% 103863 ± 5% 99849 ± 5% 99399 ± 4% 134.50 ± 88% 94400 ± 3% 96.75 ±136% 93400 ± 6% 147.25 ±147% 101127 ± 6% 43.25 ±101% 93.00 ±134% 73.50 ±129% 78700 ± 4% 82332 ± 5% 80750 ± 3% 167.00 ± 86% 132925 ± 2% 130753 ± 3% 2129 ± 28% 2129 ± 28% 102071 ± 3% 130307 ± 5% 130176 ± 3% 125592 ± 2% 126379 ± 2% 122387 ± 4% 123520 ± 5% 122586 ± 6% 119406 ± 5% 119861 ± 4% 117115 ± 4% 104723 ± 2% 113197 ± 3% 117511 ± 3% 113502 ± 2% 111247 109067 ± 4% 109639 ± 2% 107661 ± 3% 2987 ± 7% 2987 ± 7% 107034 ± 2% 96.00 ±125% 107585 ± 2% 104668 ± 5% 93.25 ±108% 11221741 19833 ± 39% 30.31 31.04 34.59 36.44 36.85 36.88 36.89 39.76 40.65 40.72 38.95 39.08 39.31 39.59 40.70 40.77 46.74 46.77 46.78 47.29 6.01 ± 7% 6.03 ± 7% 6.17 ± 7% 6.16 ± 6% 6.33 ± 6% 0.53 ± 2% 0.55 ± 2% 0.70 ± 2% 0.70 ± 2% 0.54 1.16 0.64 ± 4% 0.78 ± 3% 1.53 1.02 ± 2% 1.27 ± 2% 0.26 ±100% 1.60 ± 3% 1.02 ± 6% 1.05 ± 6% 1.24 1.22 ± 7% 1.23 ± 7% 1.44 ± 7% 1.46 2.73 2.78 0.00 3.23 ± 6% 2.29 ± 2% 3.08 ± 7% 3.17 ± 6% 5.35 5.50 5.79 5.90 31.89 34.59 30.51 36.58 36.85 36.88 36.90 39.93 38.96 39.09 41.17 39.31 41.18 39.59 40.71 40.79 41.10 41.23 46.78 47.29 47.29 47.29 6.34 ± 6% 0.34 ± 5% 0.33 ± 5% 0.24 ± 3% 0.05 ± 8% 0.23 0.12 ± 4% 0.06 0.26 ± 4% 0.08 ± 5% 0.09 ± 7% 0.09 0.04 ± 57% 0.23 0.12 ± 7% 0.09 ± 4% 0.06 ± 6% 0.03 ±100% 0.12 ± 6% 0.21 ± 3% 0.11 0.09 ± 5% 0.01 ±173% 0.01 ±173% 0.04 ± 57% 0.06 ± 6% 0.09 ± 7% 0.06 ± 7% 0.01 ±173% 0.00 0.00 0.29 ± 4% 0.00 0.09 ± 5% 0.23 0.12 ± 5% 0.08 ± 5% 0.20 ± 2% 0.07 0.04 ± 57% 0.00 0.00 0.14 ± 3% 0.00 0.00 0.11 ± 6% 0.32 ± 6% 0.11 ± 4% 0.00 0.28 ± 3% 0.17 ± 4% 0.23 ± 3% 0.19 ± 4% 0.23 ± 2% 0.27 ± 4% 0.11 ± 11% 0.11 ± 3% 0.38 0.19 ± 3% 0.13 ± 9% 0.13 ± 6% 0.53 ± 3% 0.19 ± 5% 0.26 ± 2% 0.26 ± 3% 0.18 ± 2% 0.09 ± 8% 0.16 ± 4% 0.16 ± 2% 0.25 0.19 ± 4% 0.34 0.29 ± 2% 0.26 0.24 ± 3% 0.23 ± 7% 0.28 ± 3% 0.74 0.23 0.56 ± 5% 0.28 0.45 ± 3% 0.63 0.52 ± 4% 0.39 ± 2% 0.35 ± 3% 0.24 1.16 ± 5% 0.35 ± 4% 0.28 ± 5% 0.44 0.16 ± 2% 0.33 ± 3% 0.43 ± 4% 0.35 ± 4% 0.60 ± 2% 0.40 ± 2% 0.85 0.65 ± 3% 1.18 0.94 0.79 ± 3% 1.03 ± 2% 0.74 ± 2% 0.59 ± 2% 1.77 1.62 ± 3% 1.73 1.52 ± 2% 1.51 1.46 ± 6% 1.76 2.76 0.14 ± 5% 3.35 2.80 0.55 ± 2% 3.29 5.81 5.91 6.38 6.56 5.99 30.51 39.93 0.05 0.08 ± 5% 0.23 0.08 0.06 ± 6% 0.05 0.05 0.05 ± 8% 0.12 ± 4% 0.08 ± 5% 0.08 ± 5% 0.06 ± 11% 0.06 ± 6% 0.04 ± 58% 0.20 ± 4% 0.12 ± 6% 0.05 ± 8% 0.10 ± 4% 0.08 ± 8% 0.01 ±173% 0.05 ± 9% 0.18 ± 2% 0.04 ± 57% 0.09 ± 5% 0.05 ± 8% 0.01 ±173% 0.00 0.00 0.23 ± 3% 0.20 ± 2% 0.18 ± 3% 0.00 0.00 0.08 ± 8% 0.00 0.23 0.11 ± 7% 0.30 ± 2% 0.11 ± 4% 0.00 0.17 ± 2% 0.14 ± 3% 0.09 ± 7% 0.00 0.17 ± 2% 0.00 0.11 ± 6% 0.31 ± 6% 0.00 0.39 ± 4% 0.17 ± 2% 0.11 ± 4% 0.12 ± 10% 0.08 ± 8% 0.00 0.37 0.15 ± 3% 0.26 ± 4% 0.23 ± 2% 0.20 ± 2% 0.18 ± 4% 0.17 ± 8% 0.15 ± 4% 0.42 0.12 ± 4% 0.38 ± 2% 0.27 ± 3% 0.18 ± 4% 0.24 0.28 ± 2% 0.30 0.21 ± 2% 0.38 ± 3% 0.27 ± 3% 0.48 ± 4% 0.74 0.39 ± 2% 0.23 0.16 ± 14% 0.37 ± 3% 0.44 ± 6% 0.28 ± 4% 0.16 ± 2% 1.55 0.30 ± 3% 1.09 ± 2% 0.59 ± 3% 0.83 0.94 ± 2% 0.55 ± 2% 0.62 ± 2% 1.15 1.58 0.09 ± 5%

***************** lkp-hsw-d01: ================= compiler/cpufreq_ gcc-7/performan
commit:
9c40365a65 59901cb452
9c40365a65d62d7c ---------------- fail:runs | 1:4 1:4 1:4 :4 :4 :4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 1:4 0:4 0:4 0:4 0:4 %stddev \ 173.17 346.13 346.13 62328 119.99 1258 3976 ± 2% 8581 29.08 ± 6% 190.33 ± 5% 2.474e+08 ± 62% 361421 ± 60% 2265 ± 6% 2301 ± 5% 793.33 793.33 249422 138168 139666 138194 ± 2% 139323 ± 2% 1153221 267.67 ± 15% 3697 ± 14% 245.00 ± 19% 188.67 ± 19% 292.33 ± 9% 241.67 ± 23% 7195 ± 19% 7195 ± 19% 1361 ± 5% 0.00 0.00 77852 82443 72993 2882 ± 6% 119503 ± 2% 0.00 0.00 463036 489987 434924 17242 0.68 ± 6% 0.44 ± 3% 17242 180483 180483 180482 180483 180483 180482 9075 ± 6% 13910 6416 ± 2% 1.83 ± 19% 0.66 ± 18% 22.33 ± 28% 10.96 ± 9% 180482 179957 181437 13.61 3.762e+09 3.00 1.126e+08 8.73 20492578 2.365e+08 8639 0.92 1.582e+10 182.40 799.50 0.31 14810335 4.694e+09 2.55e+09 4617509 17720319 1.731e+10 3841 1.09 164253 13452948 1875909 ± 3% 164253 13.66 2.99 8.67 0.91 771.90 0.31 3749 1.09 3.751e+09 1.123e+08 20432412 2.358e+08 8613 1.577e+10 181.89 14766724 4.68e+09 2.543e+09 4603933 17668225 1.726e+10 163774 13413384 1870377 ± 3% 163774 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 3.46 ± 14% 3.27 ± 13% 3.20 ± 14% 3.20 ± 14% 2.16 ± 14% 1.02 ± 14% 1.24 ± 2% 1.24 ± 2% 0.56 ± 3% 0.84 ± 8% 0.84 ± 8% 0.84 ± 8% 0.80 ± 8% 0.80 ± 8% 0.79 ± 3% 0.73 ± 3% 0.73 ± 3% 0.63 ± 4% 0.98 ± 3% 0.65 ± 4% 0.58 ± 2% 0.66 ± 11% 0.65 ± 6% 0.34 ± 70% 2.11 ± 5% 1.14 ± 2% 8.11 ± 5% 8.13 ± 5% 4.19 ± 12% 4.21 ± 12% 4.18 ± 12% 4.17 ± 13% 4.07 ± 13% 4.07 ± 13% 4.07 ± 13% 4.65 ± 13% 4.73 ± 14% 4.09 ± 15% 3.99 ± 15% 3.89 ± 15% 3.80 ± 15% 2.71 ± 15% 1.27 ± 15% 0.63 ± 7% 0.62 ± 9% 0.62 ± 9% 0.62 ± 9% 0.61 ± 8% 0.59 ± 7% 0.59 ± 7% 0.50 ± 4% 0.49 ± 3% 0.86 ± 7% 0.84 ± 8% 0.84 ± 8% 0.84 ± 8% 0.85 ± 7% 0.62 ± 6% 0.61 ± 7% 0.52 ± 11% 0.84 ± 3% 0.92 ± 7% 0.70 ± 5% 0.90 ± 7% 0.69 ± 7% 1.25 0.49 ± 9% 0.45 0.22 ± 9% 0.24 ± 5% 0.24 ± 5% 0.20 ± 8% 0.10 ± 16% 0.20 ± 8% 0.32 ± 7% 0.29 ± 9% 0.27 ± 7% 0.12 ± 6% 0.20 ± 8% 0.12 ± 25% 0.09 ± 13% 0.17 ± 10% 0.18 ± 10% 0.14 ± 8% 0.24 ± 7% 0.14 ± 14% 0.09 ± 24% 0.25 ± 5% 0.12 ± 14% 0.16 ± 10% 0.14 ± 14% 0.11 ± 25% 0.11 ± 27% 0.09 ± 14% 0.09 ± 5% 0.19 ± 11% 0.12 ± 10% 0.15 ± 8% 0.07 ± 6% 0.18 ± 9% 0.12 ± 14% 0.10 ± 25% 0.16 ± 10% 0.17 ± 9% 0.11 ± 7% 0.09 ± 14% 0.08 ± 5% 0.10 ± 14% 0.12 ± 7% 0.09 ± 15% 0.13 ± 6% 0.09 ± 10% 0.10 ± 8% 0.10 ± 4% 0.09 ± 5% 0.09 ± 10% 0.07 ± 6% 0.05 ± 8% 0.09 ± 5% 0.04 ± 71% 0.13 ± 6% 0.10 ± 17% 0.09 ± 9% 0.08 ± 16% 0.04 ± 73% 0.09 ± 13% 0.11 ± 14% 0.03 ± 70% 0.03 ± 70% 0.04 ± 73% 0.04 ± 70% 0.16 ± 11% 0.04 ± 71% 0.02 ±141% 0.02 ±141% 0.02 ±141% 0.08 ± 11% 0.49 ± 4% 0.03 ± 70% 0.10 ± 4% 0.00 0.16 ± 10% 0.00 0.34 ± 13% 0.59 ± 3% 0.17 ± 14% 0.72 ± 10% 0.58 ± 5% 0.76 ± 4% 0.58 ± 11% 0.58 ± 11% 0.58 ± 11% 0.58 ± 11% 2.21 ± 5% 1.16 ± 3% 0.66 ± 26% 2.71 ± 15% 1.27 ± 15% 0.49 ± 3% 0.43 ± 3% 0.20 ± 8% 0.22 ± 5% 0.32 ± 7% 0.26 ± 8% 0.09 ± 18% 0.18 ± 10% 0.20 ± 8% 0.09 ± 13% 0.17 ± 5% 0.16 ± 10% 0.09 ± 24% 0.17 ± 9% 0.18 ± 12% 0.09 ± 14% 0.24 ± 5% 0.11 ± 18% 0.12 ± 10% 0.07 ± 6% 0.13 ± 12% 0.09 ± 14% 0.07 ± 7% 0.10 ± 8% 0.12 ± 7% 0.10 ± 9% 0.10 ± 4% 0.08 0.07 ± 6% 0.08 ± 5% 0.08 ± 5% 0.07 ± 11% 0.07 ± 7% 0.13 ± 6% 0.10 ± 12% 0.08 ± 10% 0.04 ± 71% 0.15 ± 11% 0.03 ± 70% 0.09 ± 10% 0.04 ± 71% 0.02 ±141% 0.08 ± 17% 0.02 ±141% 0.49 ± 3% 0.00 0.00 0.09 ± 5% 0.02 ±141% 0.16 ± 10% 0.29 ± 9% 0.13 ± 6% 0.57 ± 4% 0.70 ± 3% 2.14 ± 6% 1.12 ± 2%

Disclaimer:
Results for informational design or

Thanks,
Rong Chen

noticed a -27.3% regression of netperf.Throughput_Mbps due to commit:
59901cb4520c44bfce81f523bc61e7284a931ad1 ("[PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group")
href="https://github.com/0day-ci/linux/commits/Aubrey-Li/sched-fair-Don-t-pull-task-if-local-group-is-more-loaded-than-busiest-group/20200326-080334">https://github.com/0day-ci/linux/commits/Aubrey-Li/sched-fair-Don-t-pull-task-if-local-group-is-more-loaded-than-busiest-group/20200326-080334
machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
Netperf is a benchmark that can be use to measure various aspect of networking performance.
href="http://www.netperf.org/netperf/">http://www.netperf.org/netperf/
to that, the commit also has significant impact on the following tests:
--+------------------------------------------------------------------------+
change | netperf: netperf.Throughput_Mbps -31.1% regression |
| 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory |
| cluster=cs-localhost |
| cpufreq_governor=performance |
| ip=ipv4 |
| nr_threads=25% |
| runtime=300s |
| send_size=10K |
| test=SCTP_STREAM_MANY |
| ucode=0xb000038 |
--+------------------------------------------------------------------------+
change | unixbench: unixbench.score -4.0% regression |
| 104 threads Skylake with 192G memory |
| cpufreq_governor=performance |
| nr_task=30% |
| runtime=300s |
| test=whetstone-double |
| ucode=0x2000065 |
--+------------------------------------------------------------------------+
change | unixbench: unixbench.score 29.1% improvement |
| 104 threads Skylake with 192G memory |
| cpufreq_governor=performance |
| nr_task=30% |
| runtime=300s |
| test=context1 |
| ucode=0x2000065 |
--+------------------------------------------------------------------------+
change | kbuild: kbuild.buildtime_per_iteration 7.5% regression |
| 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory |
| cpufreq_governor=performance |
| nr_task=50% |
| runtime=300s |
| target=vmlinux_prereq |
| ucode=0x27 |
--+------------------------------------------------------------------------+
the issue, kindly add following tag
kernel test robot <[email protected]>
--------------------------------------------------------------------------------->
href="https://github.com/intel/lkp-tests.git">https://github.com/intel/lkp-tests.git
bin/lkp install job.yaml # job file is attached in this email
========================================================================
cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode:
c-7/performance/ipv4/x86_64-rhel-7.6/25%/debian-x86_64-20191114.cgz/300s/5K/lkp-skl-2sp7/TCP_SENDFILE/netperf/0x2000065
("threads: Update PID limit comment according to futex UAPI change")
("sched/fair: Don't pull task if local group is more loaded than busiest group")
59901cb4520c44bfce81f523bc6
---------------------------
%reproduction fail:runs
| |
-25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
50% 2:4 dmesg.WARNING:at_ip__fsnotify_parent/0x
%change %stddev
| \
-27.3% 13593 netperf.Throughput_Mbps
-27.3% 244677 netperf.Throughput_total_Mbps
+82.7% 15156 ± 57% netperf.time.involuntary_context_switches
+2.6% 2350 netperf.time.maximum_resident_set_size
-8.9% 4651 netperf.time.minor_page_faults
+1.4% 4771 netperf.time.system_time
-9.7% 627.59 netperf.time.user_time
+394.1% 2131 ± 59% netperf.time.voluntary_context_switches
-27.3% 1.792e+09 netperf.workload
-21.3% 414060 ± 7% meminfo.DirectMap4k
-13.9% 46507 ± 5% meminfo.Shmem
-24.4% 1120083 vmstat.system.cs
-1.5% 148218 vmstat.system.in
-10.8% 3566 ± 5% slabinfo.kmalloc-rcl-64.active_objs
-10.8% 3566 ± 5% slabinfo.kmalloc-rcl-64.num_objs
+12.6% 12274 ± 4% slabinfo.vmap_area.active_objs
+12.7% 12277 ± 4% slabinfo.vmap_area.num_objs
-20.9% 1.746e+08 cpuidle.C1.usage
+7005.1% 4.188e+08 ± 25% cpuidle.C6.time
+2594.5% 519470 ± 17% cpuidle.C6.usage
-39.3% 21307532 ± 7% cpuidle.POLL.time
-49.7% 2423791 ± 9% cpuidle.POLL.usage
-2.7% 74740 proc-vmstat.nr_active_anon
-13.8% 11639 ± 5% proc-vmstat.nr_shmem
-2.7% 74740 proc-vmstat.nr_zone_active_anon
-86.2% 2071 ±151% proc-vmstat.numa_hint_faults
-15.5% 11476 ± 11% proc-vmstat.pgactivate
-2.1% 815970 proc-vmstat.pgfault
+1.1% 762828 proc-vmstat.pgfree
-96.3% 5569 ± 23% sched_debug.cfs_rq:/.min_vruntime.min
+30.4% 1524450 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev
+30.4% 1524462 ± 7% sched_debug.cfs_rq:/.spread0.stddev
+22.0% 1.93 ± 7% sched_debug.cpu.clock.stddev
+22.1% 1.93 ± 7% sched_debug.cpu.clock_task.stddev
-28.6% 2211323 ± 9% sched_debug.cpu.nr_switches.avg
-21.6% 1233 ± 9% sched_debug.cpu.nr_switches.min
+18.5% 67.25 ± 4% sched_debug.cpu.nr_uninterruptible.max
+16.6% 7631 ± 5% numa-vmstat.node0.nr_kernel_stack
+86.8% 1072 ± 22% numa-vmstat.node0.nr_page_table_pages
+23.3% 9919 ± 13% numa-vmstat.node0.nr_slab_reclaimable
-56.6% 14946 ± 54% numa-vmstat.node1.nr_active_anon
-67.3% 8458 ± 93% numa-vmstat.node1.nr_anon_pages
-18.0% 5022 ± 8% numa-vmstat.node1.nr_kernel_stack
-56.5% 372.00 ± 65% numa-vmstat.node1.nr_page_table_pages
-17.9% 15125 ± 7% numa-vmstat.node1.nr_slab_unreclaimable
-56.6% 14946 ± 54% numa-vmstat.node1.nr_zone_active_anon
+55.4% 32703 ± 30% numa-vmstat.node1.numa_other
+23.3% 39680 ± 13% numa-meminfo.node0.KReclaimable
+16.6% 7630 ± 5% numa-meminfo.node0.KernelStack
+14.4% 1145780 ± 4% numa-meminfo.node0.MemUsed
+86.6% 4290 ± 22% numa-meminfo.node0.PageTables
+23.3% 39680 ± 13% numa-meminfo.node0.SReclaimable
+19.9% 123222 ± 6% numa-meminfo.node0.Slab
-56.6% 59688 ± 54% numa-meminfo.node1.Active
-56.6% 59648 ± 54% numa-meminfo.node1.Active(anon)
-68.4% 16453 ±161% numa-meminfo.node1.AnonHugePages
-67.3% 33836 ± 93% numa-meminfo.node1.AnonPages
-18.0% 5017 ± 8% numa-meminfo.node1.KernelStack
-13.4% 909860 ± 5% numa-meminfo.node1.MemUsed
-56.6% 1483 ± 65% numa-meminfo.node1.PageTables
-17.9% 60503 ± 7% numa-meminfo.node1.SUnreclaim
-19.1% 88152 ± 10% numa-meminfo.node1.Slab
+25.2% 102562 ± 9% softirqs.CPU16.RCU
+204.7% 65858 ± 22% softirqs.CPU16.SCHED
-67.3% 1637461 ±148% softirqs.CPU25.NET_RX
-21.6% 35310 ± 26% softirqs.CPU25.SCHED
-69.2% 1444457 ±170% softirqs.CPU26.NET_RX
-29.8% 33825 ± 34% softirqs.CPU26.SCHED
-77.2% 1148265 ±171% softirqs.CPU29.NET_RX
-69.3% 15752 ±133% softirqs.CPU36.SCHED
-73.0% 14042 ±111% softirqs.CPU46.SCHED
-78.3% 8234 ±104% softirqs.CPU47.SCHED
-67.0% 15513 ± 99% softirqs.CPU52.SCHED
-80.4% 1169737 ±173% softirqs.CPU54.NET_RX
-76.1% 1472055 ±173% softirqs.CPU57.NET_RX
-77.3% 1528221 ±173% softirqs.CPU58.NET_RX
-74.5% 1629273 ±173% softirqs.CPU59.NET_RX
-35.6% 36922 ± 10% softirqs.CPU63.SCHED
-76.6% 1159904 ±173% softirqs.CPU64.NET_RX
-72.8% 1331256 ±169% softirqs.CPU68.NET_RX
-23.0% 2.799e+08 softirqs.NET_RX
-96.7% 3.50 ±125% interrupts.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19
-1.8% 23590 interrupts.CAL:Function_call_interrupts
-99.2% 123.75 ± 89% interrupts.CPU11.RES:Rescheduling_interrupts
-97.2% 3.00 ±135% interrupts.CPU19.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19
-94.6% 1242 ±139% interrupts.CPU25.RES:Rescheduling_interrupts
-95.2% 1129 ±152% interrupts.CPU26.RES:Rescheduling_interrupts
-98.4% 207.50 ±143% interrupts.CPU29.RES:Rescheduling_interrupts
-77.8% 4872 ±172% interrupts.CPU3.RES:Rescheduling_interrupts
-79.1% 5941 ±171% interrupts.CPU36.RES:Rescheduling_interrupts
-98.1% 518.00 ± 98% interrupts.CPU46.RES:Rescheduling_interrupts
-99.0% 205.25 ± 58% interrupts.CPU47.RES:Rescheduling_interrupts
-43.9% 3772 ± 63% interrupts.CPU48.NMI:Non-maskable_interrupts
-43.9% 3772 ± 63% interrupts.CPU48.PMI:Performance_monitoring_interrupts
-73.1% 2100 ±147% interrupts.CPU52.RES:Rescheduling_interrupts
-57.7% 3106 ± 96% interrupts.CPU54.NMI:Non-maskable_interrupts
-57.7% 3106 ± 96% interrupts.CPU54.PMI:Performance_monitoring_interrupts
-98.4% 123.75 ±104% interrupts.CPU54.RES:Rescheduling_interrupts
-75.5% 8758 ±173% interrupts.CPU56.RES:Rescheduling_interrupts
-94.9% 1212 ±172% interrupts.CPU57.RES:Rescheduling_interrupts
-94.7% 1134 ±170% interrupts.CPU58.RES:Rescheduling_interrupts
-90.8% 6.75 ± 72% interrupts.CPU6.TLB:TLB_shootdowns
-68.2% 10387 ± 72% interrupts.CPU61.RES:Rescheduling_interrupts
-81.4% 5518 ±171% interrupts.CPU62.RES:Rescheduling_interrupts
-91.3% 2218 ±165% interrupts.CPU63.RES:Rescheduling_interrupts
-98.6% 200.50 ±155% interrupts.CPU64.RES:Rescheduling_interrupts
-64.2% 4184 ±172% interrupts.CPU66.RES:Rescheduling_interrupts
-68.2% 1982 ±140% interrupts.CPU67.NMI:Non-maskable_interrupts
-68.2% 1982 ±140% interrupts.CPU67.PMI:Performance_monitoring_interrupts
-93.8% 423.50 ±145% interrupts.CPU68.RES:Rescheduling_interrupts
-98.7% 174.50 ±105% interrupts.CPU70.RES:Rescheduling_interrupts
-65.1% 4975 ±168% interrupts.CPU71.RES:Rescheduling_interrupts
-17.1% 88.75 ± 11% interrupts.IWI:IRQ_work_interrupts
-31.5% 266460 ± 8% interrupts.NMI:Non-maskable_interrupts
-31.5% 266460 ± 8% interrupts.PMI:Performance_monitoring_interrupts
-34.6% 847860 ± 22% interrupts.RES:Rescheduling_interrupts
-61.8% 607.25 ± 35% interrupts.TLB:TLB_shootdowns
-4.0% 9.77 perf-stat.i.MPKI
-25.5% 1.157e+10 perf-stat.i.branch-instructions
-25.2% 2.605e+08 perf-stat.i.branch-misses
-1.1 0.43 ± 27% perf-stat.i.cache-miss-rate%
-83.7% 1894187 ± 32% perf-stat.i.cache-misses
-28.5% 5.749e+08 perf-stat.i.cache-references
-24.4% 1127729 perf-stat.i.context-switches
+23.5% 1.51 ± 2% perf-stat.i.cpi
-7.9% 8.853e+10 perf-stat.i.cpu-cycles
+39.8% 227.77 ± 4% perf-stat.i.cpu-migrations
+758.6% 88595 ± 30% perf-stat.i.cycles-between-cache-misses
-26.2% 7628495 perf-stat.i.dTLB-load-misses
-25.5% 1.815e+10 perf-stat.i.dTLB-loads
+0.0 0.00 ± 2% perf-stat.i.dTLB-store-miss-rate%
+12.2% 14538 ± 2% perf-stat.i.dTLB-store-misses
-25.6% 1.041e+10 perf-stat.i.dTLB-stores
-13.4 35.64 ± 6% perf-stat.i.iTLB-load-miss-rate%
-29.9% 7798895 ± 3% perf-stat.i.iTLB-load-misses
+22.6% 14080265 ± 7% perf-stat.i.iTLB-loads
-25.5% 5.862e+10 perf-stat.i.instructions
+6.5% 7762 perf-stat.i.instructions-per-iTLB-miss
-18.9% 0.66 ± 2% perf-stat.i.ipc
-2.2% 2621 perf-stat.i.minor-faults
-3.7 92.22 perf-stat.i.node-load-miss-rate%
-83.8% 262514 ± 32% perf-stat.i.node-load-misses
-57.3% 22960 ± 19% perf-stat.i.node-loads
-7.8 88.18 ± 2% perf-stat.i.node-store-miss-rate%
-86.0% 29841 ± 10% perf-stat.i.node-store-misses
-22.0% 5294 ± 12% perf-stat.i.node-stores
-2.2% 2621 perf-stat.i.page-faults
-4.1% 9.81 perf-stat.overall.MPKI
-1.1 0.33 ± 32% perf-stat.overall.cache-miss-rate%
+23.6% 1.51 ± 2% perf-stat.overall.cpi
+470.3% 52623 ± 35% perf-stat.overall.cycles-between-cache-misses
+0.0 0.00 ± 3% perf-stat.overall.dTLB-store-miss-rate%
-13.5 35.72 ± 6% perf-stat.overall.iTLB-load-miss-rate%
+6.3% 7520 perf-stat.overall.instructions-per-iTLB-miss
-19.0% 0.66 ± 2% perf-stat.overall.ipc
-4.6 91.64 perf-stat.overall.node-load-miss-rate%
-11.8 84.69 ± 3% perf-stat.overall.node-store-miss-rate%
+2.7% 9844 perf-stat.overall.path-length
-25.5% 1.153e+10 perf-stat.ps.branch-instructions
-25.1% 2.596e+08 perf-stat.ps.branch-misses
-83.6% 1888636 ± 32% perf-stat.ps.cache-misses
-28.5% 5.73e+08 perf-stat.ps.cache-references
-24.4% 1123929 perf-stat.ps.context-switches
-7.9% 8.823e+10 perf-stat.ps.cpu-cycles
+40.1% 227.57 ± 4% perf-stat.ps.cpu-migrations
-26.2% 7603035 perf-stat.ps.dTLB-load-misses
-25.5% 1.809e+10 perf-stat.ps.dTLB-loads
+12.4% 14511 ± 2% perf-stat.ps.dTLB-store-misses
-25.6% 1.038e+10 perf-stat.ps.dTLB-stores
-29.9% 7772733 ± 3% perf-stat.ps.iTLB-load-misses
+22.6% 14033574 ± 7% perf-stat.ps.iTLB-loads
-25.5% 5.843e+10 perf-stat.ps.instructions
-2.2% 2612 perf-stat.ps.minor-faults
-83.8% 261634 ± 32% perf-stat.ps.node-load-misses
-57.3% 22938 ± 19% perf-stat.ps.node-loads
-86.0% 29755 ± 10% perf-stat.ps.node-store-misses
-21.8% 5301 ± 12% perf-stat.ps.node-stores
-2.2% 2612 perf-stat.ps.page-faults
-25.4% 1.764e+13 perf-stat.total.instructions
-20.6 17.33 ± 5% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
-20.1 17.96 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
-20.1 17.91 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
-19.6 21.26 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64
-19.3 20.91 ± 4% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
-19.3 20.93 ± 4% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
-19.3 20.94 ± 4% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
+0.1 1.03 ± 2% perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
+0.1 0.77 ± 3% perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64
+0.2 0.75 ± 6% perf-profile.calltrace.cycles-pp.__inet_lookup_established.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
+0.2 0.73 ± 4% perf-profile.calltrace.cycles-pp.page_cache_pipe_buf_release.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor
+0.2 0.62 ± 2% perf-profile.calltrace.cycles-pp.page_cache_pipe_buf_confirm.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor
+0.2 0.80 ± 4% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.wait_woken
+0.2 1.96 perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
+0.2 0.79 ± 3% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
+0.3 0.94 perf-profile.calltrace.cycles-pp.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
+0.3 0.89 ± 2% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
+0.3 0.93 ± 3% perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
+0.3 1.37 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
+0.3 2.31 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
+0.3 0.89 perf-profile.calltrace.cycles-pp.copy_page_to_iter.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
+0.3 0.91 ± 3% perf-profile.calltrace.cycles-pp.ipv4_mtu.tcp_current_mss.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked
+0.3 2.19 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
+0.3 1.14 perf-profile.calltrace.cycles-pp.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
+0.3 1.02 perf-profile.calltrace.cycles-pp.__alloc_skb.sk_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
+0.3 2.40 ± 4% perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor
+0.3 1.06 perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
+0.4 2.53 ± 3% perf-profile.calltrace.cycles-pp.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
+0.4 2.41 perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
+0.4 1.21 ± 4% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
+0.4 1.24 ± 4% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
+0.4 1.59 ± 3% perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg
+0.4 1.48 ± 3% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_woken.sk_wait_data
+0.5 1.52 ± 3% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg
+0.5 0.60 ± 3% perf-profile.calltrace.cycles-pp._copy_from_user.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
+0.5 0.64 perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile
+0.5 2.20 perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
+0.5 1.76 ± 3% perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
+0.5 1.90 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
+0.5 0.53 perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
+0.5 0.53 ± 2% perf-profile.calltrace.cycles-pp.__x86_indirect_thunk_rax
+0.5 3.55 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
+0.5 0.54 ± 4% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg
+0.5 2.04 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
+0.6 1.81 ± 2% perf-profile.calltrace.cycles-pp.tcp_current_mss.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
+0.6 0.56 ± 4% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
+0.6 0.69 ± 5% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout
+0.6 2.16 ± 3% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
+0.6 0.57 ± 3% perf-profile.calltrace.cycles-pp.fsnotify.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
+0.6 3.74 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
+0.6 1.78 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
+0.6 2.26 ± 3% perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
+0.6 0.60 ± 2% perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
+0.6 0.61 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
+0.6 0.62 ± 13% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
+0.6 2.03 ± 2% perf-profile.calltrace.cycles-pp.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
+0.6 0.64 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
+0.7 0.68 ± 3% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg
+0.7 2.00 ± 9% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage
+0.7 2.52 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages
+0.7 2.79 perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
+0.7 2.04 ± 9% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage.inet_sendpage
+0.7 1.73 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
+0.8 2.77 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked
+0.8 4.41 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
+0.8 2.37 ± 9% perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
+0.8 2.38 ± 9% perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
+0.9 0.90 ± 29% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit
+1.1 3.30 ± 9% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
+1.1 3.65 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
+1.1 3.43 ± 5% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established
+1.1 3.68 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
+1.1 5.36 perf-profile.calltrace.cycles-pp.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile
+1.2 4.27 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
+1.2 4.32 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
+1.4 5.18 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage
+1.4 7.91 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
+1.4 7.96 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
+1.4 4.27 ± 7% perf-profile.calltrace.cycles-pp.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
+1.5 6.33 perf-profile.calltrace.cycles-pp.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
+1.5 5.04 ± 2% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
+1.5 5.76 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage
+1.5 4.88 ± 4% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv
+1.6 5.38 ± 2% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
+1.7 6.36 ± 2% perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
+2.7 9.80 perf-profile.calltrace.cycles-pp.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage
+2.7 9.96 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
+2.8 10.01 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
+2.8 10.47 perf-profile.calltrace.cycles-pp.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
+3.2 11.16 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
+3.3 11.50 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg
+3.3 11.79 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
+3.3 11.65 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg
+3.5 12.69 perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
+3.5 12.84 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
+3.5 12.90 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
+3.6 13.05 perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
+3.7 13.39 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
+3.9 13.76 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
+4.0 14.14 perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
+4.1 14.43 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
+4.2 14.65 perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
+4.2 14.64 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
+4.2 14.76 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
+4.2 14.78 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
+6.1 21.99 ± 2% perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage
+6.3 22.89 ± 2% perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe
+6.4 23.05 ± 2% perf-profile.calltrace.cycles-pp.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe
+6.4 23.46 ± 2% perf-profile.calltrace.cycles-pp.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor
+6.5 24.01 ± 2% perf-profile.calltrace.cycles-pp.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor
+6.8 27.05 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
+6.8 27.13 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
+6.9 27.67 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
+6.9 27.72 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
+7.0 26.05 ± 2% perf-profile.calltrace.cycles-pp.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct
+7.1 26.49 ± 2% perf-profile.calltrace.cycles-pp.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile
+7.3 26.97 ± 2% perf-profile.calltrace.cycles-pp.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
+9.3 35.08 perf-profile.calltrace.cycles-pp.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
+9.5 35.75 perf-profile.calltrace.cycles-pp.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
+10.4 39.06 perf-profile.calltrace.cycles-pp.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
+10.7 40.51 perf-profile.calltrace.cycles-pp.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
+18.5 72.54 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
+18.5 72.75 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
-20.6 17.34 ± 5% perf-profile.children.cycles-pp.intel_idle
-20.4 18.26 ± 5% perf-profile.children.cycles-pp.cpuidle_enter
-20.4 18.26 ± 5% perf-profile.children.cycles-pp.cpuidle_enter_state
-19.6 21.26 ± 4% perf-profile.children.cycles-pp.secondary_startup_64
-19.6 21.26 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry
-19.6 21.26 ± 4% perf-profile.children.cycles-pp.do_idle
-19.3 20.94 ± 4% perf-profile.children.cycles-pp.start_secondary
+0.0 0.06 ± 6% perf-profile.children.cycles-pp.hrtimer_get_next_event
+0.0 0.10 ± 5% perf-profile.children.cycles-pp.tcp_update_skb_after_send
+0.0 0.13 ± 5% perf-profile.children.cycles-pp.nr_iowait_cpu
+0.0 0.09 ± 5% perf-profile.children.cycles-pp.tick_nohz_idle_enter
+0.0 0.07 ± 12% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
+0.0 0.08 ± 6% perf-profile.children.cycles-pp.rb_next
+0.0 0.09 ± 7% perf-profile.children.cycles-pp.hrtimer_next_event_without
+0.0 0.12 ± 4% perf-profile.children.cycles-pp.switch_mm_irqs_off
+0.0 0.11 ± 4% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
+0.0 0.18 ± 4% perf-profile.children.cycles-pp.tcp_check_space
+0.0 0.08 ± 6% perf-profile.children.cycles-pp.__x86_indirect_thunk_r14
+0.0 0.13 ± 3% perf-profile.children.cycles-pp.rcu_idle_exit
+0.0 0.08 ± 5% perf-profile.children.cycles-pp.pick_next_entity
+0.0 0.11 ± 3% perf-profile.children.cycles-pp.sock_rfree
+0.0 0.08 ± 6% perf-profile.children.cycles-pp.tcp_tx_timestamp
+0.0 0.14 ± 5% perf-profile.children.cycles-pp.remove_wait_queue
+0.0 0.14 ± 3% perf-profile.children.cycles-pp.native_write_msr
+0.0 0.22 ± 3% perf-profile.children.cycles-pp.copy_user_generic_unrolled
+0.0 0.08 ± 8% perf-profile.children.cycles-pp.tcp_recv_timestamp
+0.0 0.25 ± 6% perf-profile.children.cycles-pp.ip_send_check
+0.0 0.11 ± 3% perf-profile.children.cycles-pp.__x86_indirect_thunk_r10
+0.0 0.10 ± 9% perf-profile.children.cycles-pp.cpus_share_cache
+0.0 0.14 ± 3% perf-profile.children.cycles-pp.check_stack_object
+0.0 0.12 ± 10% perf-profile.children.cycles-pp.update_cfs_group
+0.0 0.10 ± 4% perf-profile.children.cycles-pp.finish_task_switch
+0.0 0.21 ± 5% perf-profile.children.cycles-pp.generic_splice_sendpage
+0.0 0.15 ± 2% perf-profile.children.cycles-pp.aa_sk_perm
+0.0 0.16 ± 2% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
+0.0 0.14 ± 6% perf-profile.children.cycles-pp.update_ts_time_stats
+0.0 0.14 ± 3% perf-profile.children.cycles-pp.sched_clock
+0.0 0.09 ± 4% perf-profile.children.cycles-pp.tcp_event_data_recv
+0.0 0.07 perf-profile.children.cycles-pp.task_tick_fair
+0.0 0.08 ± 6% perf-profile.children.cycles-pp.bictcp_acked
+0.0 0.14 ± 3% perf-profile.children.cycles-pp.native_sched_clock
+0.0 0.12 ± 7% perf-profile.children.cycles-pp.check_preempt_curr
+0.0 0.09 ± 9% perf-profile.children.cycles-pp.fpregs_assert_state_consistent
+0.0 0.09 ± 4% perf-profile.children.cycles-pp.kfree_skbmem
+0.0 0.09 ± 4% perf-profile.children.cycles-pp.__copy_skb_header
+0.0 0.20 ± 3% perf-profile.children.cycles-pp.tcp_wfree
+0.0 0.17 ± 4% perf-profile.children.cycles-pp.sockfd_lookup_light
+0.0 0.10 ± 4% perf-profile.children.cycles-pp.newidle_balance
+0.0 0.15 ± 6% perf-profile.children.cycles-pp.sched_clock_cpu
+0.0 0.18 ± 2% perf-profile.children.cycles-pp.sock_recvmsg
+0.0 0.11 perf-profile.children.cycles-pp.inet_ehashfn
+0.0 0.11 ± 3% perf-profile.children.cycles-pp.tcp_update_pacing_rate
+0.0 0.07 ± 7% perf-profile.children.cycles-pp.tcp_chrono_stop
+0.0 0.17 ± 2% perf-profile.children.cycles-pp.security_socket_recvmsg
+0.0 0.14 ± 6% perf-profile.children.cycles-pp.ttwu_do_wakeup
+0.0 0.22 ± 3% perf-profile.children.cycles-pp.inet_send_prepare
+0.0 0.10 ± 8% perf-profile.children.cycles-pp.tcp_push
+0.0 0.26 ± 3% perf-profile.children.cycles-pp.select_task_rq_fair
+0.0 0.17 ± 2% perf-profile.children.cycles-pp.__switch_to_asm
+0.0 0.13 ± 6% perf-profile.children.cycles-pp.__tcp_select_window
+0.0 0.26 ± 4% perf-profile.children.cycles-pp.tcp_rcv_space_adjust
+0.0 0.12 ± 9% perf-profile.children.cycles-pp.netif_skb_features
+0.0 0.06 ± 14% perf-profile.children.cycles-pp.raw_local_deliver
+0.0 0.26 ± 6% perf-profile.children.cycles-pp.__put_user_8
+0.0 0.10 perf-profile.children.cycles-pp.___perf_sw_event
+0.0 0.36 perf-profile.children.cycles-pp.tcp_rate_check_app_limited
+0.1 0.05 perf-profile.children.cycles-pp.default_wake_function
+0.1 0.13 ± 9% perf-profile.children.cycles-pp.sanity
+0.1 0.05 ± 8% perf-profile.children.cycles-pp.tcp_v4_inbound_md5_hash
+0.1 0.08 ± 10% perf-profile.children.cycles-pp.ip_queue_xmit
+0.1 0.26 ± 3% perf-profile.children.cycles-pp.aa_file_perm
+0.1 0.20 ± 5% perf-profile.children.cycles-pp.__switch_to
+0.1 0.20 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_exit
+0.1 0.06 ± 9% perf-profile.children.cycles-pp.iov_iter_pipe
+0.1 0.06 ± 9% perf-profile.children.cycles-pp.__usecs_to_jiffies
+0.1 0.06 ± 9% perf-profile.children.cycles-pp.bictcp_cong_avoid
+0.1 0.12 ± 11% perf-profile.children.cycles-pp.__list_del_entry_valid
+0.1 0.07 ± 10% perf-profile.children.cycles-pp.resched_curr
+0.1 0.21 ± 3% perf-profile.children.cycles-pp.rcu_all_qs
+0.1 0.15 ± 12% perf-profile.children.cycles-pp._find_next_bit
+0.1 0.13 ± 6% perf-profile.children.cycles-pp.kfree
+0.1 0.12 ± 6% perf-profile.children.cycles-pp.tcp_ack_update_rtt
+0.1 0.10 ± 4% perf-profile.children.cycles-pp.apparmor_ipv4_postroute
+0.1 0.06 perf-profile.children.cycles-pp.tcp_v4_fill_cb
+0.1 0.06 perf-profile.children.cycles-pp._raw_spin_unlock_bh
+0.1 0.06 ± 11% perf-profile.children.cycles-pp.tcp_options_write
+0.1 0.14 ± 10% perf-profile.children.cycles-pp.scheduler_tick
+0.1 0.32 ± 4% perf-profile.children.cycles-pp.__ip_local_out
+0.1 0.06 ± 13% perf-profile.children.cycles-pp.eth_type_trans
+0.1 0.06 ± 6% perf-profile.children.cycles-pp.rcu_eqs_enter
+0.1 0.06 ± 6% perf-profile.children.cycles-pp.tcp_stream_memory_free
+0.1 0.34 ± 5% perf-profile.children.cycles-pp.ip_local_out
+0.1 0.14 ± 8% perf-profile.children.cycles-pp.ip_rcv_finish
+0.1 0.07 ± 7% perf-profile.children.cycles-pp.ksoftirqd_running
+0.1 0.17 ± 6% perf-profile.children.cycles-pp.tcp_v4_send_check
+0.1 0.42 ± 5% perf-profile.children.cycles-pp.__ksize
+0.1 0.18 ± 6% perf-profile.children.cycles-pp.validate_xmit_skb
+0.1 0.16 ± 5% perf-profile.children.cycles-pp.skb_entail
+0.1 0.32 ± 4% perf-profile.children.cycles-pp.__sk_dst_check
+0.1 0.27 ± 4% perf-profile.children.cycles-pp.sock_put
+0.1 0.07 perf-profile.children.cycles-pp.put_prev_task_fair
+0.1 0.07 ± 17% perf-profile.children.cycles-pp.account_entity_dequeue
+0.1 0.16 ± 7% perf-profile.children.cycles-pp.ip_rcv_core
+0.1 0.49 ± 3% perf-profile.children.cycles-pp.skb_release_head_state
+0.1 0.31 ± 2% perf-profile.children.cycles-pp.rw_verify_area
+0.1 0.28 ± 3% perf-profile.children.cycles-pp.ipv4_dst_check
+0.1 0.51 ± 3% perf-profile.children.cycles-pp.skb_release_all
+0.1 0.11 ± 7% perf-profile.children.cycles-pp.tcp_rearm_rto
+0.1 0.09 ± 4% perf-profile.children.cycles-pp.ip_rcv_finish_core
+0.1 0.30 ± 2% perf-profile.children.cycles-pp.update_curr
+0.1 0.08 ± 11% perf-profile.children.cycles-pp.__tcp_ack_snd_check
+0.1 0.25 ± 5% perf-profile.children.cycles-pp.skb_clone
+0.1 0.19 ± 3% perf-profile.children.cycles-pp.xas_start
+0.1 0.25 ± 5% perf-profile.children.cycles-pp.kmem_cache_free
+0.1 0.08 ± 10% perf-profile.children.cycles-pp.tcp_rack_advance
+0.1 0.64 ± 2% perf-profile.children.cycles-pp.page_cache_pipe_buf_confirm
+0.1 0.21 ± 3% perf-profile.children.cycles-pp.timestamp_truncate
+0.1 0.08 ± 5% perf-profile.children.cycles-pp.rb_first
+0.1 0.39 ± 2% perf-profile.children.cycles-pp._cond_resched
+0.1 0.25 ± 7% perf-profile.children.cycles-pp.tcp_add_backlog
+0.1 0.26 ± 9% perf-profile.children.cycles-pp.__next_timer_interrupt
+0.1 0.47 ± 5% perf-profile.children.cycles-pp.__skb_clone
+0.1 0.26 ± 4% perf-profile.children.cycles-pp.ktime_get_with_offset
+0.1 0.22 ± 11% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
+0.1 0.09 ± 4% perf-profile.children.cycles-pp.tcp_cleanup_rbuf
+0.1 0.20 ± 7% perf-profile.children.cycles-pp.__update_load_avg_se
+0.1 0.10 ± 11% perf-profile.children.cycles-pp.apparmor_socket_sock_rcv_skb
+0.1 0.47 ± 3% perf-profile.children.cycles-pp.dst_release
+0.1 0.35 ± 2% perf-profile.children.cycles-pp.enqueue_to_backlog
+0.1 0.28 ± 3% perf-profile.children.cycles-pp.__ip_finish_output
+0.1 0.22 ± 8% perf-profile.children.cycles-pp.nf_hook_slow
+0.1 0.47 ± 4% perf-profile.children.cycles-pp.set_next_entity
+0.1 0.31 ± 4% perf-profile.children.cycles-pp.kmem_cache_alloc_node
+0.1 0.23 ± 4% perf-profile.children.cycles-pp.update_process_times
+0.1 0.11 ± 15% perf-profile.children.cycles-pp.rcu_core
+0.1 0.15 ± 13% perf-profile.children.cycles-pp.security_sock_rcv_skb
+0.1 0.38 ± 6% perf-profile.children.cycles-pp.get_next_timer_interrupt
+0.1 0.61 ± 3% perf-profile.children.cycles-pp._copy_from_user
+0.1 0.24 ± 5% perf-profile.children.cycles-pp.tick_sched_handle
+0.1 0.39 perf-profile.children.cycles-pp.xas_load
+0.1 0.37 ± 4% perf-profile.children.cycles-pp.__hrtimer_run_queues
+0.1 0.34 perf-profile.children.cycles-pp.tcp_schedule_loss_probe
+0.1 0.60 perf-profile.children.cycles-pp.__fget_light
+0.1 0.34 ± 5% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
+0.1 0.36 ± 6% perf-profile.children.cycles-pp.__kmalloc_reserve
+0.1 0.46 ± 6% perf-profile.children.cycles-pp.__tcp_send_ack
+0.1 0.45 ± 5% perf-profile.children.cycles-pp.tick_nohz_next_event
+0.1 0.24 ± 8% perf-profile.children.cycles-pp.sk_filter_trim_cap
+0.1 0.32 ± 2% perf-profile.children.cycles-pp.__netif_receive_skb_core
+0.1 0.27 ± 14% perf-profile.children.cycles-pp.clockevents_program_event
+0.1 0.68 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
+0.1 0.45 ± 4% perf-profile.children.cycles-pp.tcp_established_options
+0.1 0.29 ± 5% perf-profile.children.cycles-pp.tick_sched_timer
+0.1 0.25 ± 5% perf-profile.children.cycles-pp.tcp_release_cb
+0.2 0.50 perf-profile.children.cycles-pp.__might_sleep
+0.2 0.45 ± 7% perf-profile.children.cycles-pp.__virt_addr_valid
+0.2 0.50 ± 6% perf-profile.children.cycles-pp.tcp_mstamp_refresh
+0.2 0.57 ± 4% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
+0.2 0.46 perf-profile.children.cycles-pp.__fsnotify_parent
+0.2 0.43 ± 4% perf-profile.children.cycles-pp.__slab_free
+0.2 0.52 ± 2% perf-profile.children.cycles-pp.mod_timer
+0.2 0.26 ± 15% perf-profile.children.cycles-pp.irq_exit
+0.2 0.36 ± 3% perf-profile.children.cycles-pp.apparmor_file_permission
+0.2 0.51 ± 4% perf-profile.children.cycles-pp.current_time
+0.2 0.54 ± 2% perf-profile.children.cycles-pp.sk_reset_timer
+0.2 0.73 ± 4% perf-profile.children.cycles-pp.page_cache_pipe_buf_release
+0.2 0.51 ± 2% perf-profile.children.cycles-pp.tcp_event_new_data_sent
+0.2 0.97 ± 3% perf-profile.children.cycles-pp.__inet_lookup_established
+0.2 1.37 ± 2% perf-profile.children.cycles-pp.skb_release_data
+0.2 0.65 perf-profile.children.cycles-pp.netif_rx_internal
+0.2 0.66 ± 2% perf-profile.children.cycles-pp.netif_rx
+0.2 0.70 ± 5% perf-profile.children.cycles-pp.dequeue_entity
+0.2 0.70 ± 3% perf-profile.children.cycles-pp.__might_fault
+0.2 0.81 ± 3% perf-profile.children.cycles-pp.dequeue_task_fair
+0.2 0.99 ± 4% perf-profile.children.cycles-pp.read_tsc
+0.2 0.76 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair
+0.2 2.00 perf-profile.children.cycles-pp.__check_object_size
+0.2 0.72 ± 3% perf-profile.children.cycles-pp.enqueue_entity
+0.2 1.48 perf-profile.children.cycles-pp.common_file_perm
+0.2 0.94 perf-profile.children.cycles-pp.do_splice_to
+0.3 0.90 ± 2% perf-profile.children.cycles-pp.menu_select
+0.3 0.76 ± 2% perf-profile.children.cycles-pp.fsnotify
+0.3 0.90 perf-profile.children.cycles-pp.copy_page_to_iter
+0.3 0.90 perf-profile.children.cycles-pp._raw_spin_lock
+0.3 2.32 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64
+0.3 0.71 ± 10% perf-profile.children.cycles-pp.hrtimer_interrupt
+0.3 0.74 ± 4% perf-profile.children.cycles-pp.update_load_avg
+0.3 1.20 perf-profile.children.cycles-pp.___might_sleep
+0.3 0.90 ± 4% perf-profile.children.cycles-pp.enqueue_task_fair
+0.3 2.49 perf-profile.children.cycles-pp.syscall_return_via_sysret
+0.3 0.93 ± 3% perf-profile.children.cycles-pp.ttwu_do_activate
+0.3 0.92 ± 3% perf-profile.children.cycles-pp.activate_task
+0.3 0.87 ± 2% perf-profile.children.cycles-pp.atime_needs_update
+0.3 1.07 perf-profile.children.cycles-pp.sk_stream_alloc_skb
+0.3 2.42 ± 4% perf-profile.children.cycles-pp.find_get_entry
+0.4 1.13 ± 3% perf-profile.children.cycles-pp.ipv4_mtu
+0.4 2.42 perf-profile.children.cycles-pp.simple_copy_to_iter
+0.4 2.55 ± 3% perf-profile.children.cycles-pp.pagecache_get_page
+0.4 1.25 ± 3% perf-profile.children.cycles-pp.schedule_idle
+0.4 1.37 perf-profile.children.cycles-pp.loopback_xmit
+0.4 1.59 ± 3% perf-profile.children.cycles-pp.schedule_timeout
+0.4 1.46 perf-profile.children.cycles-pp.dev_hard_start_xmit
+0.4 1.43 ± 2% perf-profile.children.cycles-pp.__alloc_skb
+0.5 1.12 ± 2% perf-profile.children.cycles-pp.touch_atime
+0.5 1.52 ± 3% perf-profile.children.cycles-pp.schedule
+0.5 1.78 ± 3% perf-profile.children.cycles-pp.try_to_wake_up
+0.5 1.45 perf-profile.children.cycles-pp.ktime_get
+0.5 2.31 ± 2% perf-profile.children.cycles-pp.__kfree_skb
+0.5 1.25 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
+0.5 1.04 ± 7% perf-profile.children.cycles-pp.smp_apic_timer_interrupt
+0.5 1.12 ± 6% perf-profile.children.cycles-pp.apic_timer_interrupt
+0.5 1.76 ± 3% perf-profile.children.cycles-pp.wait_woken
+0.5 1.91 ± 3% perf-profile.children.cycles-pp.__wake_up_common
+0.6 2.04 ± 2% perf-profile.children.cycles-pp.__wake_up_common_lock
+0.6 2.33 perf-profile.children.cycles-pp.security_file_permission
+0.6 3.78 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
+0.6 2.25 ± 2% perf-profile.children.cycles-pp.sock_def_readable
+0.6 3.75 perf-profile.children.cycles-pp.copyout
+0.6 2.27 ± 3% perf-profile.children.cycles-pp.sk_wait_data
+0.6 1.97 ± 2% perf-profile.children.cycles-pp.tcp_current_mss
+0.6 2.05 ± 2% perf-profile.children.cycles-pp.tcp_send_mss
+0.7 2.66 ± 2% perf-profile.children.cycles-pp.tcp_clean_rtx_queue
+0.7 2.21 perf-profile.children.cycles-pp.__dev_queue_xmit
+0.8 4.44 perf-profile.children.cycles-pp._copy_to_iter
+0.8 2.73 ± 3% perf-profile.children.cycles-pp.__sched_text_start
+0.9 3.35 ± 2% perf-profile.children.cycles-pp.tcp_ack
+1.1 5.41 perf-profile.children.cycles-pp.generic_file_read_iter
+1.3 4.10 ± 7% perf-profile.children.cycles-pp.__release_sock
+1.4 7.93 perf-profile.children.cycles-pp.__skb_datagram_iter
+1.4 7.96 perf-profile.children.cycles-pp.skb_copy_datagram_iter
+1.4 5.33 ± 3% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
+1.5 6.34 perf-profile.children.cycles-pp.generic_file_splice_read
+1.7 5.26 ± 6% perf-profile.children.cycles-pp.release_sock
+1.7 6.70 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_bh
+1.8 6.72 ± 2% perf-profile.children.cycles-pp.lock_sock_nested
+2.7 9.84 perf-profile.children.cycles-pp.do_tcp_sendpages
+2.8 10.50 perf-profile.children.cycles-pp.tcp_sendpage_locked
+3.1 10.33 ± 2% perf-profile.children.cycles-pp.tcp_write_xmit
+3.1 10.39 ± 2% perf-profile.children.cycles-pp.__tcp_push_pending_frames
+3.5 12.71 perf-profile.children.cycles-pp.tcp_v4_rcv
+3.5 12.84 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
+3.5 12.90 perf-profile.children.cycles-pp.ip_local_deliver_finish
+3.6 13.07 perf-profile.children.cycles-pp.ip_local_deliver
+3.6 12.53 ± 2% perf-profile.children.cycles-pp.tcp_rcv_established
+3.7 12.73 ± 2% perf-profile.children.cycles-pp.tcp_v4_do_rcv
+3.7 13.41 perf-profile.children.cycles-pp.ip_rcv
+3.9 13.77 perf-profile.children.cycles-pp.__netif_receive_skb_one_core
+4.0 14.16 perf-profile.children.cycles-pp.process_backlog
+4.1 14.44 perf-profile.children.cycles-pp.net_rx_action
+4.2 14.66 perf-profile.children.cycles-pp.do_softirq_own_stack
+4.2 14.79 perf-profile.children.cycles-pp.do_softirq
+4.3 14.84 perf-profile.children.cycles-pp.__softirqentry_text_start
+4.3 15.14 perf-profile.children.cycles-pp.__local_bh_enable_ip
+4.8 16.56 perf-profile.children.cycles-pp.ip_finish_output2
+5.0 17.04 perf-profile.children.cycles-pp.ip_output
+5.1 17.74 perf-profile.children.cycles-pp.__ip_queue_xmit
+5.5 18.82 perf-profile.children.cycles-pp.__tcp_transmit_skb
+6.1 22.05 ± 2% perf-profile.children.cycles-pp.tcp_sendpage
+6.3 22.92 ± 2% perf-profile.children.cycles-pp.inet_sendpage
+6.4 23.07 ± 2% perf-profile.children.cycles-pp.kernel_sendpage
+6.4 23.48 ± 2% perf-profile.children.cycles-pp.sock_sendpage
+6.5 24.02 ± 2% perf-profile.children.cycles-pp.pipe_to_sendpage
+6.8 27.07 perf-profile.children.cycles-pp.tcp_recvmsg
+6.8 27.13 perf-profile.children.cycles-pp.inet_recvmsg
+6.9 27.67 perf-profile.children.cycles-pp.__sys_recvfrom
+6.9 27.73 perf-profile.children.cycles-pp.__x64_sys_recvfrom
+7.1 26.10 ± 2% perf-profile.children.cycles-pp.__splice_from_pipe
+7.1 26.51 ± 2% perf-profile.children.cycles-pp.splice_from_pipe
+7.3 26.97 ± 2% perf-profile.children.cycles-pp.direct_splice_actor
+9.3 35.09 perf-profile.children.cycles-pp.splice_direct_to_actor
+9.5 35.77 perf-profile.children.cycles-pp.do_splice_direct
+10.4 39.12 perf-profile.children.cycles-pp.do_sendfile
+10.7 40.54 perf-profile.children.cycles-pp.__x64_sys_sendfile64
+18.5 72.61 perf-profile.children.cycles-pp.do_syscall_64
+18.5 72.80 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
-20.6 17.34 ± 5% perf-profile.self.cycles-pp.intel_idle
-0.0 0.08 ± 6% perf-profile.self.cycles-pp.schedule_timeout
+0.0 0.11 ± 4% perf-profile.self.cycles-pp.check_stack_object
+0.0 0.08 ± 6% perf-profile.self.cycles-pp.finish_task_switch
+0.0 0.10 ± 10% perf-profile.self.cycles-pp.tcp_rcv_space_adjust
+0.0 0.08 perf-profile.self.cycles-pp.pick_next_entity
+0.0 0.07 ± 5% perf-profile.self.cycles-pp.do_softirq
+0.0 0.07 ± 5% perf-profile.self.cycles-pp.tcp_recv_timestamp
+0.0 0.13 ± 5% perf-profile.self.cycles-pp.nr_iowait_cpu
+0.0 0.11 ± 6% perf-profile.self.cycles-pp.sock_rfree
+0.0 0.11 ± 4% perf-profile.self.cycles-pp.__x86_indirect_thunk_r10
+0.0 0.19 ± 4% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
+0.0 0.12 ± 3% perf-profile.self.cycles-pp.__sk_dst_check
+0.0 0.14 ± 3% perf-profile.self.cycles-pp.native_write_msr
+0.0 0.12 ± 5% perf-profile.self.cycles-pp.select_task_rq_fair
+0.0 0.08 ± 6% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
+0.0 0.14 ± 5% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
+0.0 0.12 ± 8% perf-profile.self.cycles-pp.__sys_recvfrom
+0.0 0.10 ± 4% perf-profile.self.cycles-pp.do_splice_direct
+0.0 0.10 ± 11% perf-profile.self.cycles-pp.kmem_cache_free
+0.0 0.10 ± 9% perf-profile.self.cycles-pp.cpus_share_cache
+0.0 0.12 ± 7% perf-profile.self.cycles-pp.update_cfs_group
+0.0 0.25 ± 4% perf-profile.self.cycles-pp.ip_send_check
+0.0 0.20 ± 3% perf-profile.self.cycles-pp.inet_send_prepare
+0.0 0.12 ± 3% perf-profile.self.cycles-pp.aa_sk_perm
+0.0 0.13 ± 6% perf-profile.self.cycles-pp.native_sched_clock
+0.0 0.09 ± 7% perf-profile.self.cycles-pp.do_splice_to
+0.0 0.14 ± 5% perf-profile.self.cycles-pp.pagecache_get_page
+0.0 0.18 perf-profile.self.cycles-pp._cond_resched
+0.0 0.08 ± 5% perf-profile.self.cycles-pp.kfree_skbmem
+0.0 0.19 ± 4% perf-profile.self.cycles-pp.tcp_wfree
+0.0 0.18 ± 5% perf-profile.self.cycles-pp.generic_splice_sendpage
+0.0 0.09 ± 4% perf-profile.self.cycles-pp.ktime_get_with_offset
+0.0 0.09 ± 5% perf-profile.self.cycles-pp.fpregs_assert_state_consistent
+0.0 0.07 ± 10% perf-profile.self.cycles-pp._copy_from_user
+0.0 0.07 perf-profile.self.cycles-pp.validate_xmit_skb
+0.0 0.07 ± 5% perf-profile.self.cycles-pp.rb_next
+0.0 0.10 ± 10% perf-profile.self.cycles-pp.__next_timer_interrupt
+0.0 0.09 ± 4% perf-profile.self.cycles-pp.dev_hard_start_xmit
+0.0 0.20 ± 5% perf-profile.self.cycles-pp.sock_def_readable
+0.0 0.10 ± 7% perf-profile.self.cycles-pp.copyout
+0.0 0.10 ± 5% perf-profile.self.cycles-pp.newidle_balance
+0.0 0.07 ± 5% perf-profile.self.cycles-pp.skb_release_head_state
+0.0 0.08 ± 6% perf-profile.self.cycles-pp.bictcp_acked
+0.0 0.09 ± 9% perf-profile.self.cycles-pp.tcp_push
+0.0 0.09 ± 7% perf-profile.self.cycles-pp.tcp_event_data_recv
+0.0 0.09 ± 14% perf-profile.self.cycles-pp.nf_hook_slow
+0.0 0.09 ± 4% perf-profile.self.cycles-pp.__copy_skb_header
+0.0 0.10 perf-profile.self.cycles-pp.tcp_schedule_loss_probe
+0.0 0.11 ± 4% perf-profile.self.cycles-pp.inet_ehashfn
+0.0 0.15 ± 3% perf-profile.self.cycles-pp.ip_local_deliver
+0.0 0.06 ± 9% perf-profile.self.cycles-pp.raw_local_deliver
+0.0 0.08 ± 10% perf-profile.self.cycles-pp.netif_skb_features
+0.0 0.17 ± 2% perf-profile.self.cycles-pp.__switch_to_asm
+0.0 0.13 ± 6% perf-profile.self.cycles-pp.__tcp_select_window
+0.0 0.24 ± 2% perf-profile.self.cycles-pp.set_next_entity
+0.0 0.12 ± 4% perf-profile.self.cycles-pp.skb_entail
+0.0 0.11 ± 3% perf-profile.self.cycles-pp.tcp_update_pacing_rate
+0.0 0.36 perf-profile.self.cycles-pp.tcp_rate_check_app_limited
+0.0 0.28 ± 6% perf-profile.self.cycles-pp.security_file_permission
+0.0 0.26 ± 4% perf-profile.self.cycles-pp.__put_user_8
+0.0 0.06 perf-profile.self.cycles-pp.__x86_indirect_thunk_r14
+0.0 0.29 ± 6% perf-profile.self.cycles-pp.tcp_sendpage_locked
+0.0 0.15 ± 2% perf-profile.self.cycles-pp.rcu_all_qs
+0.0 0.09 perf-profile.self.cycles-pp.sk_filter_trim_cap
+0.1 0.11 ± 8% perf-profile.self.cycles-pp.tcp_ack_update_rtt
+0.1 0.26 ± 2% perf-profile.self.cycles-pp.aa_file_perm
+0.1 0.12 ± 4% perf-profile.self.cycles-pp.tcp_event_new_data_sent
+0.1 0.14 ± 9% perf-profile.self.cycles-pp.kernel_sendpage
+0.1 0.07 ± 13% perf-profile.self.cycles-pp.inet_recvmsg
+0.1 0.05 ± 8% perf-profile.self.cycles-pp.tcp_v4_fill_cb
+0.1 0.05 ± 8% perf-profile.self.cycles-pp.ip_local_deliver_finish
+0.1 0.05 ± 8% perf-profile.self.cycles-pp._raw_spin_unlock_bh
+0.1 0.05 ± 8% perf-profile.self.cycles-pp.iov_iter_pipe
+0.1 0.05 ± 8% perf-profile.self.cycles-pp.bictcp_cong_avoid
+0.1 0.38 ± 6% perf-profile.self.cycles-pp.__skb_clone
+0.1 0.23 ± 3% perf-profile.self.cycles-pp.net_rx_action
+0.1 0.20 ± 4% perf-profile.self.cycles-pp.__switch_to
+0.1 0.12 ± 8% perf-profile.self.cycles-pp.sanity
+0.1 0.05 ± 9% perf-profile.self.cycles-pp.tcp_rack_advance
+0.1 0.06 ± 9% perf-profile.self.cycles-pp.tcp_chrono_stop
+0.1 0.06 ± 9% perf-profile.self.cycles-pp.__tcp_send_ack
+0.1 0.13 ± 6% perf-profile.self.cycles-pp.__ip_finish_output
+0.1 0.21 ± 2% perf-profile.self.cycles-pp.xas_load
+0.1 0.21 ± 2% perf-profile.self.cycles-pp.enqueue_to_backlog
+0.1 0.15 ± 15% perf-profile.self.cycles-pp._find_next_bit
+0.1 0.20 ± 2% perf-profile.self.cycles-pp.lock_sock_nested
+0.1 0.12 ± 8% perf-profile.self.cycles-pp.__list_del_entry_valid
+0.1 0.12 ± 12% perf-profile.self.cycles-pp.kfree
+0.1 0.06 ± 7% perf-profile.self.cycles-pp.tcp_options_write
+0.1 0.07 ± 10% perf-profile.self.cycles-pp.resched_curr
+0.1 0.22 perf-profile.self.cycles-pp.process_backlog
+0.1 0.21 ± 7% perf-profile.self.cycles-pp.release_sock
+0.1 0.06 perf-profile.self.cycles-pp.__ip_local_out
+0.1 0.06 perf-profile.self.cycles-pp.tcp_tx_timestamp
+0.1 0.06 ± 11% perf-profile.self.cycles-pp.rcu_eqs_enter
+0.1 0.06 ± 11% perf-profile.self.cycles-pp.netif_rx_internal
+0.1 0.06 ± 11% perf-profile.self.cycles-pp.tcp_stream_memory_free
+0.1 0.16 ± 6% perf-profile.self.cycles-pp.tcp_v4_send_check
+0.1 0.19 ± 3% perf-profile.self.cycles-pp.update_curr
+0.1 0.06 ± 13% perf-profile.self.cycles-pp.eth_type_trans
+0.1 0.15 ± 5% perf-profile.self.cycles-pp.ip_output
+0.1 0.42 ± 5% perf-profile.self.cycles-pp.__ksize
+0.1 0.17 ± 4% perf-profile.self.cycles-pp.xas_start
+0.1 0.26 ± 3% perf-profile.self.cycles-pp.ipv4_dst_check
+0.1 0.07 ± 7% perf-profile.self.cycles-pp.ksoftirqd_running
+0.1 0.27 ± 5% perf-profile.self.cycles-pp.sock_put
+0.1 0.24 ± 5% perf-profile.self.cycles-pp.menu_select
+0.1 0.16 ± 5% perf-profile.self.cycles-pp.ip_rcv_core
+0.1 0.07 ± 16% perf-profile.self.cycles-pp.account_entity_dequeue
+0.1 0.21 ± 6% perf-profile.self.cycles-pp.enqueue_entity
+0.1 0.22 ± 12% perf-profile.self.cycles-pp.__might_fault
+0.1 0.07 perf-profile.self.cycles-pp.__tcp_push_pending_frames
+0.1 0.07 ± 10% perf-profile.self.cycles-pp.__tcp_ack_snd_check
+0.1 0.07 ± 10% perf-profile.self.cycles-pp.ip_queue_xmit
+0.1 0.08 ± 5% perf-profile.self.cycles-pp.ip_rcv_finish_core
+0.1 0.60 ± 2% perf-profile.self.cycles-pp.page_cache_pipe_buf_confirm
+0.1 0.29 ± 2% perf-profile.self.cycles-pp.rw_verify_area
+0.1 0.21 ± 3% perf-profile.self.cycles-pp.mod_timer
+0.1 0.08 ± 6% perf-profile.self.cycles-pp.rb_first
+0.1 0.08 ± 6% perf-profile.self.cycles-pp.tcp_mstamp_refresh
+0.1 0.36 perf-profile.self.cycles-pp.sock_sendpage
+0.1 0.36 ± 4% perf-profile.self.cycles-pp.__x64_sys_sendfile64
+0.1 0.19 ± 4% perf-profile.self.cycles-pp.timestamp_truncate
+0.1 0.08 ± 5% perf-profile.self.cycles-pp.apparmor_ipv4_postroute
+0.1 0.24 ± 3% perf-profile.self.cycles-pp.skb_clone
+0.1 0.21 ± 5% perf-profile.self.cycles-pp.kmem_cache_alloc_node
+0.1 0.18 ± 8% perf-profile.self.cycles-pp.enqueue_task_fair
+0.1 0.08 perf-profile.self.cycles-pp.___perf_sw_event
+0.1 0.08 ± 8% perf-profile.self.cycles-pp.tcp_rearm_rto
+0.1 0.22 ± 3% perf-profile.self.cycles-pp.tcp_send_mss
+0.1 0.22 ± 6% perf-profile.self.cycles-pp.__kmalloc_node_track_caller
+0.1 0.17 ± 9% perf-profile.self.cycles-pp.wait_woken
+0.1 0.20 ± 7% perf-profile.self.cycles-pp.__update_load_avg_se
+0.1 0.21 ± 5% perf-profile.self.cycles-pp.current_time
+0.1 0.09 ± 14% perf-profile.self.cycles-pp.apparmor_socket_sock_rcv_skb
+0.1 0.21 ± 11% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
+0.1 0.09 ± 7% perf-profile.self.cycles-pp.tcp_cleanup_rbuf
+0.1 0.61 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
+0.1 0.24 ± 4% perf-profile.self.cycles-pp.direct_splice_actor
+0.1 0.45 ± 4% perf-profile.self.cycles-pp.dst_release
+0.1 0.25 perf-profile.self.cycles-pp.loopback_xmit
+0.1 0.19 ± 7% perf-profile.self.cycles-pp.__softirqentry_text_start
+0.1 0.50 ± 2% perf-profile.self.cycles-pp.pipe_to_sendpage
+0.1 0.36 ± 2% perf-profile.self.cycles-pp.splice_from_pipe
+0.1 0.46 ± 4% perf-profile.self.cycles-pp.__sched_text_start
+0.1 0.39 ± 2% perf-profile.self.cycles-pp.simple_copy_to_iter
+0.1 0.24 ± 5% perf-profile.self.cycles-pp.touch_atime
+0.1 0.37 perf-profile.self.cycles-pp.__local_bh_enable_ip
+0.1 0.39 ± 3% perf-profile.self.cycles-pp.__ip_queue_xmit
+0.1 0.38 ± 3% perf-profile.self.cycles-pp.ip_finish_output2
+0.1 0.52 ± 3% perf-profile.self.cycles-pp.tcp_rcv_established
+0.1 0.47 ± 4% perf-profile.self.cycles-pp.tcp_clean_rtx_queue
+0.1 0.56 perf-profile.self.cycles-pp.tcp_sendpage
+0.1 0.60 ± 2% perf-profile.self.cycles-pp.inet_sendpage
+0.1 0.41 ± 2% perf-profile.self.cycles-pp._copy_to_iter
+0.1 0.34 ± 2% perf-profile.self.cycles-pp.update_load_avg
+0.1 0.59 perf-profile.self.cycles-pp.__fget_light
+0.1 0.51 ± 3% perf-profile.self.cycles-pp.tcp_v4_rcv
+0.1 0.55 ± 3% perf-profile.self.cycles-pp.tcp_recvmsg
+0.1 0.33 ± 3% perf-profile.self.cycles-pp.__alloc_skb
+0.1 0.49 ± 2% perf-profile.self.cycles-pp.generic_file_splice_read
+0.1 0.43 ± 4% perf-profile.self.cycles-pp.tcp_established_options
+0.1 0.38 ± 4% perf-profile.self.cycles-pp.tcp_ack
+0.1 0.23 ± 6% perf-profile.self.cycles-pp.tcp_release_cb
+0.1 0.47 ± 2% perf-profile.self.cycles-pp.__might_sleep
+0.1 0.52 ± 4% perf-profile.self.cycles-pp.__dev_queue_xmit
+0.1 0.32 ± 2% perf-profile.self.cycles-pp.__netif_receive_skb_core
+0.1 0.75 perf-profile.self.cycles-pp.__skb_datagram_iter
+0.1 0.43 ± 2% perf-profile.self.cycles-pp.__fsnotify_parent
+0.2 0.44 ± 7% perf-profile.self.cycles-pp.__virt_addr_valid
+0.2 0.30 ± 5% perf-profile.self.cycles-pp.apparmor_file_permission
+0.2 0.86 ± 4% perf-profile.self.cycles-pp.__inet_lookup_established
+0.2 0.36 ± 4% perf-profile.self.cycles-pp.atime_needs_update
+0.2 0.69 ± 3% perf-profile.self.cycles-pp.page_cache_pipe_buf_release
+0.2 0.42 ± 3% perf-profile.self.cycles-pp.splice_direct_to_actor
+0.2 0.43 ± 4% perf-profile.self.cycles-pp.__slab_free
+0.2 0.63 ± 5% perf-profile.self.cycles-pp.tcp_current_mss
+0.2 1.20 perf-profile.self.cycles-pp.common_file_perm
+0.2 0.53 ± 3% perf-profile.self.cycles-pp.__splice_from_pipe
+0.2 0.93 ± 4% perf-profile.self.cycles-pp.read_tsc
+0.2 1.36 perf-profile.self.cycles-pp.skb_release_data
+0.2 1.98 ± 4% perf-profile.self.cycles-pp.find_get_entry
+0.2 0.77 perf-profile.self.cycles-pp.copy_page_to_iter
+0.2 0.75 perf-profile.self.cycles-pp.do_sendfile
+0.3 0.84 ± 3% perf-profile.self.cycles-pp.__tcp_transmit_skb
+0.3 0.73 perf-profile.self.cycles-pp.fsnotify
+0.3 1.19 perf-profile.self.cycles-pp.generic_file_read_iter
+0.3 0.85 perf-profile.self.cycles-pp._raw_spin_lock
+0.3 2.32 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64
+0.3 0.63 ± 5% perf-profile.self.cycles-pp.ktime_get
+0.3 1.15 perf-profile.self.cycles-pp.___might_sleep
+0.3 2.49 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret
+0.3 1.07 ± 4% perf-profile.self.cycles-pp.ipv4_mtu
+0.3 1.42 perf-profile.self.cycles-pp._raw_spin_lock_bh
+0.3 2.01 ± 2% perf-profile.self.cycles-pp.do_tcp_sendpages
+0.4 1.18 ± 2% perf-profile.self.cycles-pp.tcp_write_xmit
+0.4 1.16 perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
+0.6 3.75 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
+0.8 4.15 perf-profile.self.cycles-pp.do_syscall_64
+1.4 5.29 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath

netperf.Throughput_Mbps

+-------------------------------------------------------------------+
|.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.|
|
|
O O O O |
O O O O O O O O O O O O |
|
|
|
|
|
|
|
|
+-------------------------------------------------------------------+

netperf.Throughput_total_Mbps

+------------------------------------------------------------------+
|.+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+.+.+..+.+ +..+.+.+ +.+.|
|
|
O O O O O O O O O |
O O O O O O O |
|
|
|
|
|
|
|
|
+------------------------------------------------------------------+

netperf.workload

+-----------------------------------------------------------------+
+.+. |
|
|
O O O O O O O O O O O O |
O O O O |
|
|
|
|
|
|
|
|
0 +-----------------------------------------------------------------+

netperf.time.user_time

-----------------------------------------------------------+
+ O O O O O O |
O O O O O O O O O O |
|
|
|
|
|
|
|
|
|
|
|
-----------------------------------------------------------+

netperf.time.system_time

-----------------------------------------------------------+
..+.+.+.+..+.+.+..+.+.+..+.+.+.+..+.+.+..+.+.+..+.+.+.+..+.|
|
|
|
|
|
|
|
|
|
|
|
|
-----------------------------------------------------------+

**********************************************************************************
160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory
========================================================================
cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode:
c-7/performance/ipv4/x86_64-rhel-7.6/25%/debian-x86_64-20191114.cgz/300s/10K/lkp-bdw-ex2/SCTP_STREAM_MANY/netperf/0xb000038
("threads: Update PID limit comment according to futex UAPI change")
("sched/fair: Don't pull task if local group is more loaded than busiest group")
59901cb4520c44bfce81f523bc6
---------------------------
%reproduction fail:runs
| |
25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
%change %stddev
| \
-31.1% 5723 ± 8% netperf.Throughput_Mbps
-31.1% 228942 ± 8% netperf.Throughput_total_Mbps
-15.8% 11583 ± 7% netperf.time.minor_page_faults
-26.0% 434.31 ± 6% netperf.time.user_time
-31.1% 8.384e+08 ± 8% netperf.workload
+19.0% 908201 ± 4% meminfo.Committed_AS
+16.3% 12404278 meminfo.DirectMap2M
-0.8 2.48 ± 11% mpstat.cpu.all.soft%
-0.5 1.50 ± 6% mpstat.cpu.all.usr%
+2.1% 73.50 vmstat.cpu.id
-31.3% 1573919 ± 8% vmstat.system.cs
-54.1% 3.157e+09 ± 16% cpuidle.C1.time
-50.4% 1.691e+08 ± 17% cpuidle.C1.usage
+165.0% 1.198e+08 ± 22% cpuidle.C1E.usage
-4.1% 24904 proc-vmstat.nr_slab_reclaimable
-6.4% 61871 proc-vmstat.nr_slab_unreclaimable
-77.7% 2009 ± 72% proc-vmstat.numa_hint_faults_local
-31.1% 1.078e+09 ± 8% proc-vmstat.numa_hit
-31.1% 1.078e+09 ± 8% proc-vmstat.numa_local
-31.1% 6.228e+09 ± 8% proc-vmstat.pgalloc_normal
-31.1% 6.228e+09 ± 8% proc-vmstat.pgfree
-56.0% 17319 ± 38% sched_debug.cfs_rq:/.MIN_vruntime.avg
-57.8% 1416646 ± 30% sched_debug.cfs_rq:/.MIN_vruntime.max
-56.4% 145249 ± 31% sched_debug.cfs_rq:/.MIN_vruntime.stddev
-56.0% 17319 ± 38% sched_debug.cfs_rq:/.max_vruntime.avg
-57.8% 1416646 ± 30% sched_debug.cfs_rq:/.max_vruntime.max
-56.4% 145249 ± 31% sched_debug.cfs_rq:/.max_vruntime.stddev
-81.3% 10104 ± 32% sched_debug.cfs_rq:/.min_vruntime.min
+32.7% 2291681 ± 13% sched_debug.cfs_rq:/.min_vruntime.stddev
+21.8% 1689 ± 12% sched_debug.cfs_rq:/.runnable_avg.max
+11.2% 511.65 ± 4% sched_debug.cfs_rq:/.runnable_avg.stddev
+32.7% 2291718 ± 13% sched_debug.cfs_rq:/.spread0.stddev
-30.3% 9949 ± 4% sched_debug.cpu.avg_idle.min
-34.9% 1161189 ± 4% sched_debug.cpu.nr_switches.avg
+32.0% 8612184 ± 16% sched_debug.cpu.nr_switches.max
-21.9% 1924 ± 2% sched_debug.cpu.nr_switches.min
+10.8% 8717 ± 5% numa-vmstat.node0.nr_kernel_stack
+88.2% 1274 ± 29% numa-vmstat.node0.nr_page_table_pages
+56.5% 9251 ± 7% numa-vmstat.node0.nr_slab_reclaimable
-77.4% 917.50 ± 98% numa-vmstat.node1.nr_inactive_anon
-16.2% 5660 numa-vmstat.node1.nr_kernel_stack
-39.0% 1793 ± 20% numa-vmstat.node1.nr_mapped
-90.8% 39.75 ± 53% numa-vmstat.node1.nr_page_table_pages
-75.3% 1102 ± 82% numa-vmstat.node1.nr_shmem
-49.3% 3517 ± 13% numa-vmstat.node1.nr_slab_reclaimable
-37.7% 11019 ± 7% numa-vmstat.node1.nr_slab_unreclaimable
-77.4% 917.50 ± 98% numa-vmstat.node1.nr_zone_inactive_anon
-51.3% 91255968 ± 63% numa-vmstat.node1.numa_hit
-51.3% 91208577 ± 63% numa-vmstat.node1.numa_local
-75.5% 6288 ±130% numa-vmstat.node2.nr_active_anon
-75.5% 6267 ±131% numa-vmstat.node2.nr_anon_pages
-75.5% 6288 ±130% numa-vmstat.node2.nr_zone_active_anon
+56.5% 37005 ± 7% numa-meminfo.node0.KReclaimable
+10.7% 8713 ± 5% numa-meminfo.node0.KernelStack
+24.6% 9060 ± 20% numa-meminfo.node0.Mapped
+23.2% 909436 numa-meminfo.node0.MemUsed
+88.1% 5098 ± 29% numa-meminfo.node0.PageTables
+56.5% 37005 ± 7% numa-meminfo.node0.SReclaimable
+26.1% 123682 ± 6% numa-meminfo.node0.Slab
-77.1% 3725 ± 96% numa-meminfo.node1.Inactive
-77.3% 3672 ± 98% numa-meminfo.node1.Inactive(anon)
-49.3% 14069 ± 13% numa-meminfo.node1.KReclaimable
-16.0% 5670 numa-meminfo.node1.KernelStack
-39.3% 7065 ± 21% numa-meminfo.node1.Mapped
-10.8% 529651 ± 5% numa-meminfo.node1.MemUsed
-90.4% 167.00 ± 51% numa-meminfo.node1.PageTables
-49.3% 14069 ± 13% numa-meminfo.node1.SReclaimable
-37.7% 44079 ± 7% numa-meminfo.node1.SUnreclaim
-75.3% 4415 ± 82% numa-meminfo.node1.Shmem
-41.0% 58148 ± 8% numa-meminfo.node1.Slab
-75.4% 25195 ±130% numa-meminfo.node2.Active
-75.4% 25163 ±130% numa-meminfo.node2.Active(anon)
-84.3% 11236 ±162% numa-meminfo.node2.AnonHugePages
-75.5% 25082 ±131% numa-meminfo.node2.AnonPages
-25.7% 6997 ± 9% slabinfo.Acpi-State.active_objs
-25.7% 6997 ± 9% slabinfo.Acpi-State.num_objs
-9.7% 9182 ± 3% slabinfo.cred_jar.active_objs
-9.7% 9182 ± 3% slabinfo.cred_jar.num_objs
-30.8% 5027 ± 10% slabinfo.files_cache.active_objs
-30.8% 5027 ± 10% slabinfo.files_cache.num_objs
-16.9% 43590 ± 4% slabinfo.filp.active_objs
-16.6% 684.50 ± 4% slabinfo.filp.active_slabs
-16.6% 43839 ± 4% slabinfo.filp.num_objs
-16.6% 684.50 ± 4% slabinfo.filp.num_slabs
-25.7% 4226 ± 7% slabinfo.mm_struct.active_objs
-25.7% 4226 ± 7% slabinfo.mm_struct.num_objs
-13.4% 16227 ± 6% slabinfo.pde_opener.active_objs
-13.4% 16227 ± 6% slabinfo.pde_opener.num_objs
-23.0% 8755 ± 12% slabinfo.pool_workqueue.active_objs
-23.0% 8756 ± 12% slabinfo.pool_workqueue.num_objs
-18.9% 3126 ± 5% slabinfo.sighand_cache.active_objs
-18.8% 3135 ± 4% slabinfo.sighand_cache.num_objs
-22.4% 4942 ± 6% slabinfo.signal_cache.active_objs
-22.3% 4950 ± 6% slabinfo.signal_cache.num_objs
-24.2% 7218 ± 9% slabinfo.task_delay_info.active_objs
-24.2% 7218 ± 9% slabinfo.task_delay_info.num_objs
-12.5% 2379 slabinfo.task_struct.active_objs
-11.9% 801.00 slabinfo.task_struct.active_slabs
-11.9% 2403 slabinfo.task_struct.num_objs
-11.9% 801.00 slabinfo.task_struct.num_slabs
-6.7% 58.10 perf-stat.i.MPKI
-18.1% 1.233e+10 ± 4% perf-stat.i.branch-instructions
-0.2 1.68 ± 4% perf-stat.i.branch-miss-rate%
-28.3% 2.058e+08 ± 8% perf-stat.i.branch-misses
-0.4 0.38 ± 30% perf-stat.i.cache-miss-rate%
-61.2% 13229676 ± 41% perf-stat.i.cache-misses
-25.3% 3.405e+09 ± 6% perf-stat.i.cache-references
-31.2% 1587675 ± 8% perf-stat.i.context-switches
+21.0% 2.44 ± 2% perf-stat.i.cpi
-4.0% 1.413e+11 ± 2% perf-stat.i.cpu-cycles
+1447.9% 12045 ± 71% perf-stat.i.cpu-migrations
+244.8% 15744 ± 36% perf-stat.i.cycles-between-cache-misses
-23.0% 18146040 ± 8% perf-stat.i.dTLB-load-misses
-23.0% 1.905e+10 ± 5% perf-stat.i.dTLB-loads
+0.0 0.09 ± 9% perf-stat.i.dTLB-store-miss-rate%
-29.7% 1.153e+10 ± 8% perf-stat.i.dTLB-stores
-7.0 25.20 ± 6% perf-stat.i.iTLB-load-miss-rate%
-24.0% 13290898 ± 7% perf-stat.i.iTLB-load-misses
+7.2% 39555285 perf-stat.i.iTLB-loads
-20.2% 5.82e+10 ± 4% perf-stat.i.instructions
-17.0% 0.41 ± 3% perf-stat.i.ipc
-56.1% 3466546 ± 33% perf-stat.i.node-load-misses
-49.5% 41717 ± 30% perf-stat.i.node-loads
-21.2 64.20 ± 12% perf-stat.i.node-store-miss-rate%
-64.6% 1747970 ± 45% perf-stat.i.node-store-misses
-6.5% 58.45 ± 2% perf-stat.overall.MPKI
-0.2 1.67 ± 4% perf-stat.overall.branch-miss-rate%
-0.4 0.38 ± 34% perf-stat.overall.cache-miss-rate%
+20.6% 2.43 ± 3% perf-stat.overall.cpi
+188.2% 12623 ± 39% perf-stat.overall.cycles-between-cache-misses
+0.0 0.09 ± 10% perf-stat.overall.dTLB-store-miss-rate%
-7.0 25.13 ± 6% perf-stat.overall.iTLB-load-miss-rate%
-17.0% 0.41 ± 3% perf-stat.overall.ipc
-19.6 66.27 ± 14% perf-stat.overall.node-store-miss-rate%
+16.1% 20929 ± 3% perf-stat.overall.path-length
-18.2% 1.228e+10 ± 4% perf-stat.ps.branch-instructions
-28.4% 2.048e+08 ± 8% perf-stat.ps.branch-misses
-61.3% 13129750 ± 41% perf-stat.ps.cache-misses
-25.4% 3.39e+09 ± 6% perf-stat.ps.cache-references
-31.3% 1580015 ± 8% perf-stat.ps.context-switches
-4.0% 1.408e+11 ± 2% perf-stat.ps.cpu-cycles
+1419.0% 12072 ± 70% perf-stat.ps.cpu-migrations
-23.0% 18074421 ± 8% perf-stat.ps.dTLB-load-misses
-23.1% 1.896e+10 ± 5% perf-stat.ps.dTLB-loads
-29.8% 1.147e+10 ± 8% perf-stat.ps.dTLB-stores
-24.0% 13232122 ± 7% perf-stat.ps.iTLB-load-misses
+7.1% 39405794 ± 2% perf-stat.ps.iTLB-loads
-20.3% 5.795e+10 ± 4% perf-stat.ps.instructions
-56.2% 3443289 ± 33% perf-stat.ps.node-load-misses
-49.5% 41606 ± 30% perf-stat.ps.node-loads
-64.7% 1734707 ± 45% perf-stat.ps.node-store-misses
-20.3% 1.75e+13 ± 4% perf-stat.total.instructions
+68.9% 5130 ± 41% interrupts.CPU10.NMI:Non-maskable_interrupts
+68.9% 5130 ± 41% interrupts.CPU10.PMI:Performance_monitoring_interrupts
+145.3% 6592 ± 13% interrupts.CPU103.NMI:Non-maskable_interrupts
+145.3% 6592 ± 13% interrupts.CPU103.PMI:Performance_monitoring_interrupts
+216.1% 6887 ± 2% interrupts.CPU106.NMI:Non-maskable_interrupts
+216.1% 6887 ± 2% interrupts.CPU106.PMI:Performance_monitoring_interrupts
+126.1% 6798 ± 9% interrupts.CPU107.RES:Rescheduling_interrupts
+197.3% 5935 ± 31% interrupts.CPU108.NMI:Non-maskable_interrupts
+197.3% 5935 ± 31% interrupts.CPU108.PMI:Performance_monitoring_interrupts
+151.9% 5294 ± 49% interrupts.CPU109.RES:Rescheduling_interrupts
+330.4% 6801 ± 38% interrupts.CPU118.RES:Rescheduling_interrupts
+274.5% 5101 ± 39% interrupts.CPU119.RES:Rescheduling_interrupts
-83.7% 574.75 ±115% interrupts.CPU122.RES:Rescheduling_interrupts
-74.1% 348.00 ±147% interrupts.CPU137.RES:Rescheduling_interrupts
-60.7% 874.25 ±131% interrupts.CPU141.NMI:Non-maskable_interrupts
-60.7% 874.25 ±131% interrupts.CPU141.PMI:Performance_monitoring_interrupts
-80.8% 688.00 ±134% interrupts.CPU142.NMI:Non-maskable_interrupts
-80.8% 688.00 ±134% interrupts.CPU142.PMI:Performance_monitoring_interrupts
-73.7% 772.00 ±171% interrupts.CPU146.RES:Rescheduling_interrupts
+1035.4% 34656 ±135% interrupts.CPU15.RES:Rescheduling_interrupts
-59.0% 1102 ± 98% interrupts.CPU157.RES:Rescheduling_interrupts
-65.3% 1455 ±141% interrupts.CPU166.NMI:Non-maskable_interrupts
-65.3% 1455 ±141% interrupts.CPU166.PMI:Performance_monitoring_interrupts
-72.2% 898.25 ±139% interrupts.CPU167.NMI:Non-maskable_interrupts
-72.2% 898.25 ±139% interrupts.CPU167.PMI:Performance_monitoring_interrupts
-74.3% 897.00 ±133% interrupts.CPU172.NMI:Non-maskable_interrupts
-74.3% 897.00 ±133% interrupts.CPU172.PMI:Performance_monitoring_interrupts
-88.5% 435.00 ± 68% interrupts.CPU173.NMI:Non-maskable_interrupts
-88.5% 435.00 ± 68% interrupts.CPU173.PMI:Performance_monitoring_interrupts
-79.4% 972.75 ±136% interrupts.CPU174.NMI:Non-maskable_interrupts
-79.4% 972.75 ±136% interrupts.CPU174.PMI:Performance_monitoring_interrupts
-95.5% 213.25 ± 87% interrupts.CPU175.RES:Rescheduling_interrupts
-76.2% 832.50 ±107% interrupts.CPU184.NMI:Non-maskable_interrupts
-76.2% 832.50 ±107% interrupts.CPU184.PMI:Performance_monitoring_interrupts
-71.9% 772.25 ± 95% interrupts.CPU185.NMI:Non-maskable_interrupts
-71.9% 772.25 ± 95% interrupts.CPU185.PMI:Performance_monitoring_interrupts
-84.2% 616.50 ±115% interrupts.CPU186.NMI:Non-maskable_interrupts
-84.2% 616.50 ±115% interrupts.CPU186.PMI:Performance_monitoring_interrupts
-90.6% 597.50 ±110% interrupts.CPU187.NMI:Non-maskable_interrupts
-90.6% 597.50 ±110% interrupts.CPU187.PMI:Performance_monitoring_interrupts
-91.2% 246.25 ±163% interrupts.CPU188.RES:Rescheduling_interrupts
-89.6% 776.00 ±126% interrupts.CPU33.RES:Rescheduling_interrupts
-88.3% 293.75 ±158% interrupts.CPU35.RES:Rescheduling_interrupts
-81.9% 572.25 ±164% interrupts.CPU36.RES:Rescheduling_interrupts
-80.8% 844.50 ±168% interrupts.CPU39.RES:Rescheduling_interrupts
-91.7% 553.00 ±169% interrupts.CPU41.RES:Rescheduling_interrupts
-64.2% 1841 ±149% interrupts.CPU42.NMI:Non-maskable_interrupts
-64.2% 1841 ±149% interrupts.CPU42.PMI:Performance_monitoring_interrupts
-68.2% 1855 ±159% interrupts.CPU43.NMI:Non-maskable_interrupts
-68.2% 1855 ±159% interrupts.CPU43.PMI:Performance_monitoring_interrupts
-97.9% 75.25 ±126% interrupts.CPU44.RES:Rescheduling_interrupts
+130.1% 7754 ± 39% interrupts.CPU5.RES:Rescheduling_interrupts
-78.9% 924.75 ± 65% interrupts.CPU58.RES:Rescheduling_interrupts
+108.3% 5847 ± 36% interrupts.CPU6.RES:Rescheduling_interrupts
-83.6% 1226 ±112% interrupts.CPU61.RES:Rescheduling_interrupts
-88.3% 430.75 ±167% interrupts.CPU65.RES:Rescheduling_interrupts
-51.7% 1405 ± 62% interrupts.CPU75.NMI:Non-maskable_interrupts
-51.7% 1405 ± 62% interrupts.CPU75.PMI:Performance_monitoring_interrupts
-66.1% 786.25 ±117% interrupts.CPU76.NMI:Non-maskable_interrupts
-66.1% 786.25 ±117% interrupts.CPU76.PMI:Performance_monitoring_interrupts
-86.4% 310.75 ± 34% interrupts.CPU77.NMI:Non-maskable_interrupts
-86.4% 310.75 ± 34% interrupts.CPU77.PMI:Performance_monitoring_interrupts
-91.0% 465.00 ± 34% interrupts.CPU77.RES:Rescheduling_interrupts
-74.6% 1047 ± 87% interrupts.CPU79.RES:Rescheduling_interrupts
-63.7% 1875 ±107% interrupts.CPU86.RES:Rescheduling_interrupts
-70.4% 1041 ±106% interrupts.CPU88.NMI:Non-maskable_interrupts
-70.4% 1041 ±106% interrupts.CPU88.PMI:Performance_monitoring_interrupts
-78.3% 1037 ± 96% interrupts.CPU89.NMI:Non-maskable_interrupts
-78.3% 1037 ± 96% interrupts.CPU89.PMI:Performance_monitoring_interrupts
+193.3% 7009 ± 21% interrupts.CPU9.RES:Rescheduling_interrupts
-78.3% 789.75 ±128% interrupts.CPU90.NMI:Non-maskable_interrupts
-78.3% 789.75 ±128% interrupts.CPU90.PMI:Performance_monitoring_interrupts
-68.2% 768.50 ±126% interrupts.CPU91.NMI:Non-maskable_interrupts
-68.2% 768.50 ±126% interrupts.CPU91.PMI:Performance_monitoring_interrupts
-99.7% 11.00 ± 94% interrupts.CPU93.RES:Rescheduling_interrupts
-82.2% 849.75 ±136% interrupts.CPU94.NMI:Non-maskable_interrupts
-82.2% 849.75 ±136% interrupts.CPU94.PMI:Performance_monitoring_interrupts
-96.3% 114.25 ±129% interrupts.CPU94.RES:Rescheduling_interrupts
-73.5% 707.25 ±171% interrupts.CPU95.RES:Rescheduling_interrupts
+28.1% 826353 ± 9% interrupts.RES:Rescheduling_interrupts
-10.7% 126249 ± 8% softirqs.CPU10.RCU
-39.7% 23314 ± 40% softirqs.CPU10.SCHED
-24.3% 30768 ± 18% softirqs.CPU101.SCHED
-10.6% 120927 ± 7% softirqs.CPU102.RCU
-48.1% 20612 ± 50% softirqs.CPU102.SCHED
+233.0% 1191209 ± 45% softirqs.CPU108.NET_RX
-11.5% 123936 ± 10% softirqs.CPU108.RCU
-40.4% 23651 ± 53% softirqs.CPU108.SCHED
-62.5% 436694 ± 91% softirqs.CPU113.NET_RX
-37.5% 23090 ± 43% softirqs.CPU114.SCHED
-29.6% 27760 ± 23% softirqs.CPU116.SCHED
-20.4% 29821 ± 8% softirqs.CPU117.SCHED
-29.5% 27287 ± 24% softirqs.CPU122.SCHED
-100.0% 4.75 ± 60% softirqs.CPU124.NET_RX
-100.0% 6.50 ± 82% softirqs.CPU126.NET_RX
+17.7% 42122 ± 5% softirqs.CPU126.SCHED
-7.4% 126126 ± 5% softirqs.CPU136.RCU
-10.7% 121892 ± 3% softirqs.CPU137.RCU
-19.4% 30389 ± 19% softirqs.CPU14.SCHED
-7.9% 127161 ± 6% softirqs.CPU143.RCU
-11.1% 120675 ± 2% softirqs.CPU144.RCU
-9.5% 123693 ± 2% softirqs.CPU146.RCU
-7.0% 131118 ± 4% softirqs.CPU147.RCU
-7.0% 130636 ± 5% softirqs.CPU151.RCU
-9.2% 127229 ± 4% softirqs.CPU157.RCU
-41.3% 24083 ± 51% softirqs.CPU157.SCHED
+19.5% 40766 ± 12% softirqs.CPU159.SCHED
-94.6% 66471 ±173% softirqs.CPU160.NET_RX
-38.0% 23813 ± 45% softirqs.CPU161.SCHED
-7.0% 127861 ± 3% softirqs.CPU162.RCU
+27.3% 39228 ± 6% softirqs.CPU164.SCHED
-13.0% 120523 ± 5% softirqs.CPU165.RCU
-24.5% 31709 ± 38% softirqs.CPU165.SCHED
-96.8% 30927 ±173% softirqs.CPU172.NET_RX
-100.0% 2.75 ± 69% softirqs.CPU173.NET_RX
-100.0% 2.75 ± 86% softirqs.CPU179.NET_RX
-100.0% 7.00 ± 66% softirqs.CPU183.NET_RX
+10.4% 43614 ± 6% softirqs.CPU183.SCHED
+53.6% 40670 softirqs.CPU186.SCHED
-100.0% 477.00 ±173% softirqs.CPU187.NET_RX
-6.6% 124422 ± 4% softirqs.CPU189.RCU
-8.4% 125547 ± 2% softirqs.CPU190.RCU
-72.1% 28546 ±173% softirqs.CPU191.NET_RX
-71.1% 433553 ± 99% softirqs.CPU23.NET_RX
-78.7% 310211 ±145% softirqs.CPU27.NET_RX
+32.5% 40018 ± 16% softirqs.CPU27.SCHED
-83.4% 207506 ±173% softirqs.CPU28.NET_RX
+22.9% 40167 ± 10% softirqs.CPU28.SCHED
-91.8% 50330 ±172% softirqs.CPU31.NET_RX
-92.1% 67308 ±172% softirqs.CPU32.NET_RX
-92.6% 61272 ±153% softirqs.CPU33.NET_RX
-92.0% 45490 ±172% softirqs.CPU34.NET_RX
-80.2% 237504 ±173% softirqs.CPU35.NET_RX
-98.9% 8411 ±170% softirqs.CPU36.NET_RX
-97.6% 47130 ±172% softirqs.CPU38.NET_RX
+51.2% 38644 ± 12% softirqs.CPU38.SCHED
-69.5% 581766 ±173% softirqs.CPU39.NET_RX
-10.5% 122273 ± 3% softirqs.CPU41.RCU
-83.7% 316430 ±173% softirqs.CPU42.NET_RX
-84.5% 305614 ±173% softirqs.CPU43.NET_RX
-8.5% 126429 ± 4% softirqs.CPU44.RCU
-88.1% 111510 ±172% softirqs.CPU45.NET_RX
-100.0% 142.50 ± 6% softirqs.CPU47.NET_RX
-96.3% 51993 ±172% softirqs.CPU56.NET_RX
+33.4% 41922 ± 8% softirqs.CPU56.SCHED
-93.1% 110943 ±172% softirqs.CPU57.NET_RX
-52.3% 565791 ±113% softirqs.CPU58.NET_RX
-86.8% 301429 ±153% softirqs.CPU60.NET_RX
-84.0% 158376 ±173% softirqs.CPU63.NET_RX
-87.3% 172102 ±173% softirqs.CPU64.NET_RX
-77.5% 305024 ±170% softirqs.CPU67.NET_RX
-70.1% 317101 ±173% softirqs.CPU68.NET_RX
-9.8% 122670 ± 7% softirqs.CPU70.RCU
-66.8% 404442 ±163% softirqs.CPU71.NET_RX
-12.6% 121504 ± 7% softirqs.CPU71.RCU
-57.5% 570854 ±101% softirqs.CPU75.NET_RX
-60.3% 543996 ±107% softirqs.CPU76.NET_RX
-20.6% 30718 ± 21% softirqs.CPU77.SCHED
-32.2% 25215 ± 53% softirqs.CPU8.SCHED
-76.3% 484633 ±140% softirqs.CPU84.NET_RX
-69.4% 355800 ±116% softirqs.CPU85.NET_RX
-92.4% 109880 ±173% softirqs.CPU87.NET_RX
+48.1% 43243 ± 10% softirqs.CPU87.SCHED
-90.8% 181832 ± 87% softirqs.CPU88.NET_RX
+60.9% 41872 ± 10% softirqs.CPU88.SCHED
-92.5% 166989 ±173% softirqs.CPU89.NET_RX
+65.7% 39517 ± 6% softirqs.CPU89.SCHED
-100.0% 3.25 ± 70% softirqs.CPU90.NET_RX
+34.9% 41296 ± 10% softirqs.CPU90.SCHED
-100.0% 3.75 ± 87% softirqs.CPU91.NET_RX
-97.2% 20120 ±173% softirqs.CPU92.NET_RX
-31.1% 1.198e+08 ± 8% softirqs.NET_RX
-2.7 32.88 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
-2.7 36.26 ± 4% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
-2.7 36.26 ± 4% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
-2.7 36.25 ± 4% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
-2.6 36.42 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64
-2.5 34.09 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
-2.5 34.17 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
-1.7 5.99 ± 4% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
-1.7 5.91 ± 4% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
-1.7 5.83 ± 4% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush
-1.7 5.60 ± 4% perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.sctp_packet_transmit
-1.7 5.58 ± 4% perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
-1.6 5.52 ± 4% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
-1.6 5.54 ± 4% perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
-1.6 5.45 ± 4% perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
-1.6 5.36 ± 4% perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
-1.6 5.27 ± 4% perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
-1.6 5.14 ± 4% perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
-1.6 5.01 ± 4% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
-1.6 4.96 ± 4% perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
-1.6 5.07 ± 4% perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
-1.5 5.03 ± 4% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
-1.4 8.72 ± 3% perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
-1.3 6.75 ± 4% perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
-1.2 4.05 ± 4% perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
-1.2 4.09 ± 5% perf-profile.calltrace.cycles-pp.memcpy_erms.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
-1.2 3.78 ± 4% perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
-1.1 7.82 ± 3% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc
-1.1 3.36 ± 4% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu
-1.0 3.45 ± 5% perf-profile.calltrace.cycles-pp.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
-1.0 3.33 ± 5% perf-profile.calltrace.cycles-pp._copy_from_iter_full.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
-1.0 3.23 ± 5% perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
-0.9 3.22 ± 5% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.sctp_user_addto_chunk.sctp_datamsg_from_user
-0.9 7.23 ± 4% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
-0.9 6.87 ± 3% perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
-0.8 2.36 ± 5% perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv
-0.7 6.56 ± 3% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg
-0.6 6.44 ± 3% perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
-0.6 3.59 ± 7% perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
-0.6 3.57 ± 7% perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
-0.6 0.86 ± 57% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event
-0.6 0.84 ± 57% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready
-0.5 3.21 ± 7% perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg
-0.5 1.90 ± 3% perf-profile.calltrace.cycles-pp.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
-0.5 3.12 ± 7% perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg
-0.5 3.10 ± 7% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
-0.5 1.44 ± 5% perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
-0.4 1.33 ± 5% perf-profile.calltrace.cycles-pp.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm
-0.4 0.72 ± 5% perf-profile.calltrace.cycles-pp.lock_sock_nested.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
-0.4 1.21 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter
-0.3 1.14 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data
-0.3 0.64 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg
-0.3 0.40 ± 57% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv
-0.3 0.62 ± 6% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.sctp_skb_recv_datagram.sctp_recvmsg
-0.2 0.41 ± 57% perf-profile.calltrace.cycles-pp.copy_msghdr_from_user.sendmsg_copy_msghdr.___sys_sendmsg.__sys_sendmsg.do_syscall_64
-0.2 0.41 ± 57% perf-profile.calltrace.cycles-pp.sendmsg_copy_msghdr.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
-0.2 0.70 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
-0.2 0.67 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
-0.2 0.57 ± 4% perf-profile.calltrace.cycles-pp.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
-0.1 0.74 ± 6% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
-0.1 0.72 ± 7% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
-0.1 0.89 ± 3% perf-profile.calltrace.cycles-pp.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
-0.1 0.88 ± 3% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg
-0.1 0.86 ± 3% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg
+0.9 3.69 ± 13% perf-profile.calltrace.cycles-pp.kfree_skb.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg
+1.1 3.14 ± 14% perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.kfree_skb.sctp_recvmsg.inet_recvmsg
+1.1 3.27 ± 14% perf-profile.calltrace.cycles-pp.__free_pages_ok.kfree_skb.sctp_recvmsg.inet_recvmsg.____sys_recvmsg
+1.1 3.02 ± 14% perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.kfree_skb.sctp_recvmsg
+1.2 2.92 ± 15% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.kfree_skb
+1.3 3.29 ± 3% perf-profile.calltrace.cycles-pp.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
+1.3 3.19 ± 3% perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
+1.4 3.17 ± 3% perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush
+1.4 3.17 ± 3% perf-profile.calltrace.cycles-pp.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.sctp_packet_transmit
+2.0 25.38 ± 2% perf-profile.calltrace.cycles-pp.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
+3.2 61.64 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
+3.2 61.47 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
+3.2 19.53 ± 2% perf-profile.calltrace.cycles-pp.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg
+3.3 19.45 ± 2% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
+3.3 19.40 ± 2% perf-profile.calltrace.cycles-pp.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg
+3.4 16.27 ± 2% perf-profile.calltrace.cycles-pp.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg
+3.7 17.89 ± 2% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock
+3.8 17.70 ± 2% perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg
+3.8 17.70 ± 2% perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock
+4.2 11.90 perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv
+4.5 11.22 perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
+4.6 11.08 perf-profile.calltrace.cycles-pp.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm
+4.6 47.55 ± 2% perf-profile.calltrace.cycles-pp.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
+4.6 12.34 ± 2% perf-profile.calltrace.cycles-pp.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg
+4.7 47.35 ± 2% perf-profile.calltrace.cycles-pp.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
+4.7 10.56 ± 2% perf-profile.calltrace.cycles-pp.__free_pages_ok.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter
+4.7 10.33 ± 2% perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.consume_skb.sctp_chunk_put.sctp_outq_sack
+4.7 12.04 ± 2% perf-profile.calltrace.cycles-pp._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg
+4.8 46.77 ± 2% perf-profile.calltrace.cycles-pp.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
+4.8 11.62 ± 2% perf-profile.calltrace.cycles-pp.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc
+4.8 9.99 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.consume_skb.sctp_chunk_put
+4.9 46.11 ± 2% perf-profile.calltrace.cycles-pp.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64
+4.9 9.80 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.consume_skb
+5.0 45.80 ± 2% perf-profile.calltrace.cycles-pp.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg
+5.1 11.06 ± 2% perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user
+5.1 11.03 ± 2% perf-profile.calltrace.cycles-pp.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb._sctp_make_chunk
+5.1 11.05 ± 2% perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty
+6.5 14.09 ± 3% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb
+6.5 13.86 ± 3% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve
+6.9 12.47 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller
+7.0 12.24 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node
-2.7 33.02 ± 5% perf-profile.children.cycles-pp.intel_idle
-2.7 36.26 ± 4% perf-profile.children.cycles-pp.start_secondary
-2.6 36.42 ± 4% perf-profile.children.cycles-pp.do_idle
-2.6 36.42 ± 4% perf-profile.children.cycles-pp.secondary_startup_64
-2.6 36.42 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry
-2.5 34.32 ± 4% perf-profile.children.cycles-pp.cpuidle_enter
-2.5 34.32 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state
-2.2 15.63 ± 4% perf-profile.children.cycles-pp.sctp_outq_flush
-1.8 6.19 ± 4% perf-profile.children.cycles-pp.__ip_queue_xmit
-1.8 14.03 ± 4% perf-profile.children.cycles-pp.sctp_packet_transmit
-1.8 6.07 ± 4% perf-profile.children.cycles-pp.ip_output
-1.7 5.95 ± 4% perf-profile.children.cycles-pp.ip_finish_output2
-1.7 5.66 ± 4% perf-profile.children.cycles-pp.__local_bh_enable_ip
-1.7 5.59 ± 4% perf-profile.children.cycles-pp.do_softirq
-1.6 5.54 ± 4% perf-profile.children.cycles-pp.do_softirq_own_stack
-1.6 5.45 ± 4% perf-profile.children.cycles-pp.net_rx_action
-1.6 5.37 ± 4% perf-profile.children.cycles-pp.process_backlog
-1.6 5.28 ± 4% perf-profile.children.cycles-pp.__netif_receive_skb_one_core
-1.6 5.85 ± 5% perf-profile.children.cycles-pp.__softirqentry_text_start
-1.6 5.15 ± 4% perf-profile.children.cycles-pp.ip_rcv
-1.6 5.07 ± 4% perf-profile.children.cycles-pp.ip_local_deliver
-1.6 5.04 ± 4% perf-profile.children.cycles-pp.ip_local_deliver_finish
-1.6 4.97 ± 4% perf-profile.children.cycles-pp.sctp_rcv
-1.5 5.02 ± 4% perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
-1.5 6.48 ± 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
-1.4 8.72 ± 3% perf-profile.children.cycles-pp.sctp_primitive_SEND
-1.2 4.23 ± 5% perf-profile.children.cycles-pp.memcpy_erms
-1.0 3.46 ± 5% perf-profile.children.cycles-pp.sctp_user_addto_chunk
-1.0 3.33 ± 5% perf-profile.children.cycles-pp._copy_from_iter_full
-1.0 3.24 ± 5% perf-profile.children.cycles-pp.copyin
-0.8 2.37 ± 5% perf-profile.children.cycles-pp.sctp_ulpq_tail_data
-0.6 3.59 ± 7% perf-profile.children.cycles-pp.skb_copy_datagram_iter
-0.6 3.58 ± 7% perf-profile.children.cycles-pp.__skb_datagram_iter
-0.5 3.22 ± 7% perf-profile.children.cycles-pp._copy_to_iter
-0.5 1.91 ± 3% perf-profile.children.cycles-pp.sctp_skb_recv_datagram
-0.5 3.12 ± 7% perf-profile.children.cycles-pp.copyout
-0.5 1.46 ± 5% perf-profile.children.cycles-pp.sctp_ulpq_tail_event
-0.4 1.33 ± 5% perf-profile.children.cycles-pp.sctp_data_ready
-0.4 0.93 ± 4% perf-profile.children.cycles-pp.lock_sock_nested
-0.4 0.85 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_bh
-0.4 1.21 ± 6% perf-profile.children.cycles-pp.__wake_up_common_lock
-0.3 1.14 ± 6% perf-profile.children.cycles-pp.__wake_up_common
-0.3 1.10 ± 6% perf-profile.children.cycles-pp.autoremove_wake_function
-0.3 1.08 ± 6% perf-profile.children.cycles-pp.try_to_wake_up
-0.3 0.77 ± 5% perf-profile.children.cycles-pp.sctp_ulpevent_make_rcvmsg
-0.2 0.58 ± 5% perf-profile.children.cycles-pp.__list_del_entry_valid
-0.2 1.61 ± 4% perf-profile.children.cycles-pp.__sched_text_start
-0.2 0.33 ± 7% perf-profile.children.cycles-pp.sctp_datamsg_put
-0.2 0.77 perf-profile.children.cycles-pp.syscall_return_via_sysret
-0.2 0.31 ± 7% perf-profile.children.cycles-pp.sctp_chunk_free
-0.2 0.52 ± 4% perf-profile.children.cycles-pp.sctp_outq_select_transport
-0.2 0.70 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64
-0.2 0.46 ± 2% perf-profile.children.cycles-pp.sctp_packet_config
-0.2 0.57 ± 4% perf-profile.children.cycles-pp.sctp_ulpevent_free
-0.2 0.54 ± 3% perf-profile.children.cycles-pp.kmem_cache_free
-0.2 0.67 perf-profile.children.cycles-pp.skb_release_all
-0.2 0.66 perf-profile.children.cycles-pp.skb_release_head_state
-0.2 0.92 ± 4% perf-profile.children.cycles-pp.copy_msghdr_from_user
-0.2 0.37 ± 5% perf-profile.children.cycles-pp.__skb_clone
-0.1 0.74 ± 3% perf-profile.children.cycles-pp._copy_from_user
-0.1 0.41 ± 6% perf-profile.children.cycles-pp.enqueue_entity
-0.1 0.51 ± 7% perf-profile.children.cycles-pp.ttwu_do_activate
-0.1 0.50 ± 7% perf-profile.children.cycles-pp.enqueue_task_fair
-0.1 0.74 ± 6% perf-profile.children.cycles-pp.schedule_idle
-0.1 0.51 ± 8% perf-profile.children.cycles-pp.activate_task
-0.1 0.45 ± 5% perf-profile.children.cycles-pp.sctp_chunkify
-0.1 0.28 ± 5% perf-profile.children.cycles-pp.__list_add_valid
-0.1 0.54 ± 5% perf-profile.children.cycles-pp.sendmsg_copy_msghdr
-0.1 0.29 ± 5% perf-profile.children.cycles-pp.__slab_free
-0.1 0.20 ± 4% perf-profile.children.cycles-pp.ipv4_dst_check
-0.1 0.89 ± 3% perf-profile.children.cycles-pp.schedule_timeout
-0.1 0.30 ± 5% perf-profile.children.cycles-pp.sctp_hash_cmp
-0.1 0.88 ± 3% perf-profile.children.cycles-pp.schedule
-0.1 0.17 ± 4% perf-profile.children.cycles-pp.__copy_skb_header
-0.1 0.28 ± 4% perf-profile.children.cycles-pp.sctp_addrs_lookup_transport
-0.1 0.32 ± 3% perf-profile.children.cycles-pp.copy_user_generic_unrolled
-0.1 0.13 ± 5% perf-profile.children.cycles-pp.sctp_transport_hold
-0.1 0.33 ± 6% perf-profile.children.cycles-pp.sctp_endpoint_lookup_assoc
-0.1 0.30 ± 8% perf-profile.children.cycles-pp.sctp_epaddr_lookup_transport
-0.1 0.36 ± 4% perf-profile.children.cycles-pp.__dev_queue_xmit
-0.1 0.40 ± 7% perf-profile.children.cycles-pp.kmem_cache_alloc
-0.1 0.34 ± 5% perf-profile.children.cycles-pp.___might_sleep
-0.1 0.24 ± 6% perf-profile.children.cycles-pp.sctp_addto_chunk
-0.1 0.24 ± 4% perf-profile.children.cycles-pp.sctp_eat_data
-0.1 0.40 ± 4% perf-profile.children.cycles-pp.import_iovec
-0.1 0.29 ± 4% perf-profile.children.cycles-pp.sockfd_lookup_light
-0.1 0.29 ± 3% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
-0.1 0.25 ± 4% perf-profile.children.cycles-pp.security_socket_sendmsg
-0.1 0.41 ± 4% perf-profile.children.cycles-pp.update_load_avg
-0.1 0.36 ± 3% perf-profile.children.cycles-pp.sctp_sf_eat_data_6_2
-0.1 0.16 ± 6% perf-profile.children.cycles-pp.skb_release_data
-0.1 0.28 ± 4% perf-profile.children.cycles-pp.sctp_wfree
-0.1 0.35 ± 5% perf-profile.children.cycles-pp.rw_copy_check_uvector
-0.1 0.17 ± 5% perf-profile.children.cycles-pp.memset_erms
-0.1 0.19 ± 9% perf-profile.children.cycles-pp.sctp_sock_rfree
-0.1 0.15 ± 7% perf-profile.children.cycles-pp.__switch_to
-0.1 0.09 ± 8% perf-profile.children.cycles-pp.ip_local_out
-0.1 0.16 ± 6% perf-profile.children.cycles-pp.skb_set_owner_w
-0.1 0.40 ± 8% perf-profile.children.cycles-pp.recvmsg_copy_msghdr
-0.1 0.46 ± 3% perf-profile.children.cycles-pp.dequeue_task_fair
-0.1 0.23 ± 2% perf-profile.children.cycles-pp.__might_sleep
-0.1 0.31 ± 3% perf-profile.children.cycles-pp.aa_sk_perm
-0.1 0.24 ± 5% perf-profile.children.cycles-pp.__fget_light
-0.1 0.10 ± 4% perf-profile.children.cycles-pp.switch_mm_irqs_off
-0.1 0.30 ± 3% perf-profile.children.cycles-pp.sctp_sched_fcfs_dequeue
-0.1 0.10 ± 5% perf-profile.children.cycles-pp.rcu_idle_exit
-0.1 0.11 ± 6% perf-profile.children.cycles-pp.ttwu_do_wakeup
-0.1 0.55 ± 2% perf-profile.children.cycles-pp.__check_object_size
-0.1 0.24 ± 3% perf-profile.children.cycles-pp.dev_hard_start_xmit
-0.1 0.21 ± 5% perf-profile.children.cycles-pp.sctp_make_sack
-0.1 0.22 perf-profile.children.cycles-pp.loopback_xmit
-0.1 0.11 ± 7% perf-profile.children.cycles-pp.ipv4_mtu
-0.1 0.08 ± 5% perf-profile.children.cycles-pp.rcu_eqs_exit
-0.0 0.17 ± 6% perf-profile.children.cycles-pp.sctp_inq_pop
-0.0 0.11 ± 8% perf-profile.children.cycles-pp.check_preempt_curr
-0.0 0.07 ± 10% perf-profile.children.cycles-pp.ip_send_check
-0.0 0.12 ± 8% perf-profile.children.cycles-pp.sctp_outq_flush_ctrl
-0.0 0.07 ± 17% perf-profile.children.cycles-pp.resched_curr
-0.0 0.17 ± 4% perf-profile.children.cycles-pp.sock_kmalloc
-0.0 0.10 ± 4% perf-profile.children.cycles-pp.sock_wfree
-0.0 0.09 ± 7% perf-profile.children.cycles-pp.__ip_local_out
-0.0 0.20 ± 7% perf-profile.children.cycles-pp.sctp_check_transmitted
-0.0 0.11 ± 3% perf-profile.children.cycles-pp.sctp_transport_put
-0.0 0.18 ± 9% perf-profile.children.cycles-pp.kmem_cache_alloc_node
-0.0 0.40 ± 3% perf-profile.children.cycles-pp.dequeue_entity
-0.0 0.14 ± 5% perf-profile.children.cycles-pp.__update_load_avg_se
-0.0 0.08 ± 10% perf-profile.children.cycles-pp.sctp_chunk_hold
-0.0 0.07 ± 7% perf-profile.children.cycles-pp.__ip_finish_output
-0.0 0.03 ±100% perf-profile.children.cycles-pp.sctp_hash_key
-0.0 0.10 ± 11% perf-profile.children.cycles-pp.dst_release
-0.0 0.20 ± 4% perf-profile.children.cycles-pp.move_addr_to_kernel
-0.0 0.15 ± 4% perf-profile.children.cycles-pp.sctp_ulpevent_receive_data
-0.0 0.21 ± 7% perf-profile.children.cycles-pp.select_task_rq_fair
-0.0 0.14 ± 6% perf-profile.children.cycles-pp.sctp_v4_addr_valid
-0.0 0.04 ± 57% perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
-0.0 0.03 ±100% perf-profile.children.cycles-pp.sctp_bind_addr_match
-0.0 0.13 perf-profile.children.cycles-pp.sctp_association_put
-0.0 0.12 ± 6% perf-profile.children.cycles-pp.sock_kfree_s
-0.0 0.16 ± 7% perf-profile.children.cycles-pp.update_rq_clock
-0.0 0.14 ± 6% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
-0.0 0.15 ± 8% perf-profile.children.cycles-pp.__zone_watermark_ok
-0.0 0.10 ± 7% perf-profile.children.cycles-pp.__netif_receive_skb_core
-0.0 0.33 ± 5% perf-profile.children.cycles-pp.kfree
-0.0 0.17 ± 4% perf-profile.children.cycles-pp.sctp_sched_dequeue_common
-0.0 0.08 ± 14% perf-profile.children.cycles-pp.sctp_association_hold
-0.0 0.04 ± 57% perf-profile.children.cycles-pp.sctp_v4_from_skb
-0.0 0.04 ± 58% perf-profile.children.cycles-pp.pick_next_entity
-0.0 0.04 ± 57% perf-profile.children.cycles-pp.validate_xmit_skb
-0.0 0.28 ± 5% perf-profile.children.cycles-pp.sctp_packet_transmit_chunk
-0.0 0.18 ± 6% perf-profile.children.cycles-pp.update_curr
-0.0 0.12 ± 7% perf-profile.children.cycles-pp.sctp_sockaddr_af
-0.0 0.11 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
-0.0 0.08 ± 6% perf-profile.children.cycles-pp.del_timer
-0.0 0.06 ± 6% perf-profile.children.cycles-pp.sctp_transport_reset_t3_rtx
-0.0 0.10 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_exit
-0.0 0.07 ± 14% perf-profile.children.cycles-pp.sctp_assoc_rwnd_increase
-0.0 0.11 ± 10% perf-profile.children.cycles-pp.skb_clone
-0.0 0.06 ± 7% perf-profile.children.cycles-pp.sctp_v4_ecn_capable
-0.0 0.10 ± 7% perf-profile.children.cycles-pp.mod_timer
-0.0 0.12 ± 5% perf-profile.children.cycles-pp.__kmalloc
-0.0 0.06 ± 7% perf-profile.children.cycles-pp.account_entity_dequeue
-0.0 0.07 ± 6% perf-profile.children.cycles-pp.nr_iowait_cpu
-0.0 0.07 perf-profile.children.cycles-pp.lock_timer_base
-0.0 0.12 ± 3% perf-profile.children.cycles-pp.mod_node_page_state
-0.0 0.09 ± 4% perf-profile.children.cycles-pp.sctp_validate_data
-0.0 0.07 ± 6% perf-profile.children.cycles-pp.enqueue_to_backlog
-0.0 0.10 ± 4% perf-profile.children.cycles-pp.__switch_to_asm
-0.0 0.08 ± 5% perf-profile.children.cycles-pp.netif_rx
-0.0 0.08 ± 6% perf-profile.children.cycles-pp.netif_rx_internal
-0.0 0.07 ± 6% perf-profile.children.cycles-pp.tick_nohz_idle_enter
-0.0 0.06 perf-profile.children.cycles-pp.__x64_sys_sendmsg
-0.0 0.05 perf-profile.children.cycles-pp.__free_pages
+0.0 0.09 ± 7% perf-profile.children.cycles-pp.___perf_sw_event
+0.0 0.09 ± 15% perf-profile.children.cycles-pp.irq_enter
+0.1 0.07 ± 17% perf-profile.children.cycles-pp.tick_irq_enter
+0.1 0.05 ± 8% perf-profile.children.cycles-pp.load_balance
+0.9 3.69 ± 13% perf-profile.children.cycles-pp.kfree_skb
+2.0 27.81 ± 2% perf-profile.children.cycles-pp.sctp_cmd_interpreter
+2.0 25.38 ± 2% perf-profile.children.cycles-pp.sctp_sendmsg_to_asoc
+2.0 27.94 ± 2% perf-profile.children.cycles-pp.sctp_do_sm
+2.5 21.76 ± 2% perf-profile.children.cycles-pp.sctp_assoc_bh_rcv
+3.2 61.68 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
+3.2 61.52 ± 2% perf-profile.children.cycles-pp.do_syscall_64
+3.2 19.60 ± 2% perf-profile.children.cycles-pp.release_sock
+3.3 19.46 ± 2% perf-profile.children.cycles-pp.__release_sock
+3.3 19.40 ± 2% perf-profile.children.cycles-pp.sctp_backlog_rcv
+3.4 16.28 ± 2% perf-profile.children.cycles-pp.sctp_datamsg_from_user
+4.3 11.90 perf-profile.children.cycles-pp.sctp_outq_sack
+4.4 11.72 perf-profile.children.cycles-pp.sctp_chunk_put
+4.5 11.31 perf-profile.children.cycles-pp.consume_skb
+4.6 47.55 ± 2% perf-profile.children.cycles-pp.__sys_sendmsg
+4.6 12.34 ± 2% perf-profile.children.cycles-pp.sctp_make_datafrag_empty
+4.6 47.35 ± 2% perf-profile.children.cycles-pp.___sys_sendmsg
+4.7 12.17 ± 2% perf-profile.children.cycles-pp._sctp_make_chunk
+4.8 46.78 ± 2% perf-profile.children.cycles-pp.____sys_sendmsg
+4.9 46.12 ± 2% perf-profile.children.cycles-pp.sock_sendmsg
+5.0 45.82 ± 2% perf-profile.children.cycles-pp.sctp_sendmsg
+5.7 13.48 ± 4% perf-profile.children.cycles-pp.free_one_page
+5.7 13.86 ± 4% perf-profile.children.cycles-pp.__free_pages_ok
+5.8 13.30 ± 4% perf-profile.children.cycles-pp._raw_spin_lock
+6.1 15.05 ± 2% perf-profile.children.cycles-pp.__alloc_skb
+6.4 14.31 ± 3% perf-profile.children.cycles-pp.__kmalloc_reserve
+6.4 14.29 ± 2% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
+6.4 14.21 ± 2% perf-profile.children.cycles-pp.kmalloc_large_node
+6.5 14.10 ± 3% perf-profile.children.cycles-pp.__alloc_pages_nodemask
+6.5 13.88 ± 3% perf-profile.children.cycles-pp.get_page_from_freelist
+6.8 12.84 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
+12.8 25.66 ± 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
-2.7 33.02 ± 5% perf-profile.self.cycles-pp.intel_idle
-1.5 6.43 ± 2% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
-1.2 4.20 ± 5% perf-profile.self.cycles-pp.memcpy_erms
-0.6 2.05 perf-profile.self.cycles-pp.do_syscall_64
-0.3 0.50 ± 4% perf-profile.self.cycles-pp.__alloc_skb
-0.3 0.56 perf-profile.self.cycles-pp._raw_spin_lock
-0.2 0.56 ± 5% perf-profile.self.cycles-pp.__list_del_entry_valid
-0.2 0.77 perf-profile.self.cycles-pp.syscall_return_via_sysret
-0.2 0.70 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64
-0.2 0.29 ± 7% perf-profile.self.cycles-pp.sctp_datamsg_put
-0.2 0.38 ± 2% perf-profile.self.cycles-pp.sctp_datamsg_from_user
-0.2 0.53 ± 5% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
-0.1 0.61 ± 5% perf-profile.self.cycles-pp.get_page_from_freelist
-0.1 0.26 ± 7% perf-profile.self.cycles-pp.__list_add_valid
-0.1 0.30 ± 6% perf-profile.self.cycles-pp.sctp_recvmsg
-0.1 0.19 ± 6% perf-profile.self.cycles-pp.ipv4_dst_check
-0.1 0.17 ± 7% perf-profile.self.cycles-pp.__copy_skb_header
-0.1 0.29 ± 5% perf-profile.self.cycles-pp.__slab_free
-0.1 0.24 ± 7% perf-profile.self.cycles-pp.free_one_page
-0.1 0.30 ± 3% perf-profile.self.cycles-pp.copy_user_generic_unrolled
-0.1 0.25 perf-profile.self.cycles-pp.sctp_cmd_interpreter
-0.1 0.12 ± 4% perf-profile.self.cycles-pp.sctp_transport_hold
-0.1 0.31 ± 5% perf-profile.self.cycles-pp.sctp_packet_transmit
-0.1 0.38 ± 5% perf-profile.self.cycles-pp.sctp_outq_flush
-0.1 0.22 ± 3% perf-profile.self.cycles-pp.sctp_chunk_put
-0.1 0.25 ± 7% perf-profile.self.cycles-pp.kmem_cache_free
-0.1 0.33 ± 6% perf-profile.self.cycles-pp.___might_sleep
-0.1 0.20 ± 6% perf-profile.self.cycles-pp.sctp_rcv
-0.1 0.18 ± 8% perf-profile.self.cycles-pp.sctp_packet_config
-0.1 0.11 ± 11% perf-profile.self.cycles-pp.sctp_ulpq_tail_event
-0.1 0.11 perf-profile.self.cycles-pp.sctp_chunkify
-0.1 0.16 ± 6% perf-profile.self.cycles-pp.skb_release_data
-0.1 0.16 ± 5% perf-profile.self.cycles-pp.__alloc_pages_nodemask
-0.1 0.27 ± 7% perf-profile.self.cycles-pp.__sched_text_start
-0.1 0.20 ± 7% perf-profile.self.cycles-pp.__skb_clone
-0.1 0.16 ± 4% perf-profile.self.cycles-pp.memset_erms
-0.1 0.16 ± 5% perf-profile.self.cycles-pp.skb_set_owner_w
-0.1 0.14 ± 7% perf-profile.self.cycles-pp.__switch_to
-0.1 0.10 ± 5% perf-profile.self.cycles-pp.switch_mm_irqs_off
-0.1 0.18 ± 9% perf-profile.self.cycles-pp.sctp_sock_rfree
-0.1 0.22 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_bh
-0.1 0.26 ± 3% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
-0.1 0.12 ± 6% perf-profile.self.cycles-pp.sctp_data_ready
-0.1 0.21 ± 4% perf-profile.self.cycles-pp.__might_sleep
-0.1 0.24 ± 5% perf-profile.self.cycles-pp.__fget_light
-0.0 0.07 ± 6% perf-profile.self.cycles-pp.ip_send_check
-0.0 0.24 ± 4% perf-profile.self.cycles-pp.__check_object_size
-0.0 0.22 perf-profile.self.cycles-pp.sctp_wfree
-0.0 0.21 ± 5% perf-profile.self.cycles-pp.sctp_sendmsg
-0.0 0.10 ± 7% perf-profile.self.cycles-pp.ipv4_mtu
-0.0 0.07 ± 17% perf-profile.self.cycles-pp.resched_curr
-0.0 0.17 ± 4% perf-profile.self.cycles-pp.sctp_skb_recv_datagram
-0.0 0.10 ± 5% perf-profile.self.cycles-pp.sock_wfree
-0.0 0.10 ± 4% perf-profile.self.cycles-pp.sctp_eat_data
-0.0 0.09 ± 7% perf-profile.self.cycles-pp.dst_release
-0.0 0.11 ± 4% perf-profile.self.cycles-pp.sctp_transport_put
-0.0 0.04 ± 57% perf-profile.self.cycles-pp.sctp_ulpevent_free
-0.0 0.03 ±100% perf-profile.self.cycles-pp.pick_next_entity
-0.0 0.03 ±100% perf-profile.self.cycles-pp.sctp_hash_key
-0.0 0.29 ± 6% perf-profile.self.cycles-pp.__free_pages_ok
-0.0 0.08 ± 10% perf-profile.self.cycles-pp.sctp_chunk_hold
-0.0 0.04 ± 57% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
-0.0 0.10 ± 8% perf-profile.self.cycles-pp.enqueue_entity
-0.0 0.14 ± 5% perf-profile.self.cycles-pp.__update_load_avg_se
-0.0 0.13 perf-profile.self.cycles-pp.sctp_association_put
-0.0 0.18 ± 7% perf-profile.self.cycles-pp.____sys_recvmsg
-0.0 0.17 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
-0.0 0.12 ± 6% perf-profile.self.cycles-pp.sctp_v4_addr_valid
-0.0 0.10 ± 4% perf-profile.self.cycles-pp.sctp_epaddr_lookup_transport
-0.0 0.11 ± 3% perf-profile.self.cycles-pp.set_next_entity
-0.0 0.13 perf-profile.self.cycles-pp.do_idle
-0.0 0.13 ± 8% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
-0.0 0.10 ± 7% perf-profile.self.cycles-pp.__netif_receive_skb_core
-0.0 0.15 ± 8% perf-profile.self.cycles-pp.__zone_watermark_ok
-0.0 0.17 ± 7% perf-profile.self.cycles-pp.sctp_sendmsg_to_asoc
-0.0 0.06 ± 6% perf-profile.self.cycles-pp.sctp_assoc_rwnd_increase
-0.0 0.12 ± 12% perf-profile.self.cycles-pp.kmem_cache_alloc_node
-0.0 0.09 ± 7% perf-profile.self.cycles-pp.__softirqentry_text_start
-0.0 0.09 ± 4% perf-profile.self.cycles-pp.sctp_check_transmitted
-0.0 0.07 ± 5% perf-profile.self.cycles-pp.sctp_ulpevent_make_rcvmsg
-0.0 0.11 ± 6% perf-profile.self.cycles-pp.__skb_datagram_iter
-0.0 0.13 ± 3% perf-profile.self.cycles-pp.sctp_sched_fcfs_dequeue
-0.0 0.10 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
-0.0 0.10 ± 11% perf-profile.self.cycles-pp.ip_finish_output2
-0.0 0.05 ± 8% perf-profile.self.cycles-pp.sock_kmalloc
-0.0 0.04 ± 57% perf-profile.self.cycles-pp.sockfd_lookup_light
-0.0 0.08 ± 14% perf-profile.self.cycles-pp.sctp_association_hold
-0.0 0.07 ± 10% perf-profile.self.cycles-pp.__local_bh_enable_ip
-0.0 0.09 ± 5% perf-profile.self.cycles-pp.sctp_ulpevent_receive_data
-0.0 0.09 ± 5% perf-profile.self.cycles-pp.sctp_make_datafrag_empty
-0.0 0.07 ± 5% perf-profile.self.cycles-pp.consume_skb
-0.0 0.07 ± 5% perf-profile.self.cycles-pp.sctp_inq_pop
-0.0 0.06 ± 13% perf-profile.self.cycles-pp.__kmalloc_node_track_caller
-0.0 0.12 ± 3% perf-profile.self.cycles-pp.mod_node_page_state
-0.0 0.05 ± 9% perf-profile.self.cycles-pp.account_entity_dequeue
-0.0 0.06 ± 7% perf-profile.self.cycles-pp.sctp_v4_ecn_capable
-0.0 0.07 ± 6% perf-profile.self.cycles-pp.rw_copy_check_uvector
-0.0 0.10 ± 5% perf-profile.self.cycles-pp.__switch_to_asm
-0.0 0.10 ± 8% perf-profile.self.cycles-pp.sctp_sched_dequeue_common
-0.0 0.07 ± 6% perf-profile.self.cycles-pp.nr_iowait_cpu
-0.0 0.09 ± 4% perf-profile.self.cycles-pp.update_curr
-0.0 0.08 ± 5% perf-profile.self.cycles-pp.copy_msghdr_from_user
-0.0 0.07 ± 6% perf-profile.self.cycles-pp.__sctp_packet_append_chunk
-0.0 0.07 perf-profile.self.cycles-pp.sctp_addrs_lookup_transport
-0.0 0.05 perf-profile.self.cycles-pp.__free_pages
+0.0 0.08 ± 6% perf-profile.self.cycles-pp.___perf_sw_event
+12.8 25.66 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
**********************************************************************************
104 threads Skylake with 192G memory
========================================================================
governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
ce/x86_64-rhel-7.6/30%/debian-x86_64-20191114.cgz/300s/lkp-skl-fpga01/whetstone-double/unixbench/0x2000065
("threads: Update PID limit comment according to futex UAPI change")
("sched/fair: Don't pull task if local group is more loaded than busiest group")
59901cb4520c44bfce81f523bc6
---------------------------
%change %stddev
| \
-4.0% 22179 unixbench.score
+2.0% 483.52 unixbench.time.elapsed_time
+2.0% 483.52 unixbench.time.elapsed_time.max
-15.6% 17220 unixbench.time.involuntary_context_switches
+2.5% 11940 unixbench.time.user_time
-2.7% 6494 unixbench.time.voluntary_context_switches
-2.2% 58918844 unixbench.workload
+61.4% 83147 ± 16% cpuidle.C1.usage
+9.0% 49503 ± 6% meminfo.Shmem
+0.0 0.03 ± 78% mpstat.cpu.all.soft%
+5.3% 1584 vmstat.system.cs
-9.2% 8320 ± 6% numa-meminfo.node0.KernelStack
-49.9% 497.00 ± 65% numa-meminfo.node1.Mlocked
-7.3% 496178 ± 3% numa-meminfo.node1.Unevictable
-9.1% 8327 ± 6% numa-vmstat.node0.nr_kernel_stack
-7.3% 124044 ± 3% numa-vmstat.node1.nr_unevictable
-7.3% 124044 ± 3% numa-vmstat.node1.nr_zone_unevictable
-0.7 0.49 ± 32% perf-profile.children.cycles-pp.ktime_get
-0.5 0.43 ± 29% perf-profile.children.cycles-pp.clockevents_program_event
+0.1 0.20 ± 32% perf-profile.children.cycles-pp.tick_irq_enter
-0.7 0.38 ± 49% perf-profile.self.cycles-pp.ktime_get
-42.0% 489022 ± 36% numa-numastat.node0.local_node
-39.6% 514237 ± 31% numa-numastat.node0.numa_hit
+199.5% 25221 ± 57% numa-numastat.node0.other_node
+47.3% 1149237 ± 15% numa-numastat.node1.local_node
+43.8% 1157736 ± 14% numa-numastat.node1.numa_hit
-5.4% 87.00 proc-vmstat.nr_dirtied
+9.0% 12374 ± 6% proc-vmstat.nr_shmem
-1.5% 19147 proc-vmstat.nr_slab_reclaimable
-3.1% 43181 proc-vmstat.nr_slab_unreclaimable
-84.1% 2121 ± 82% proc-vmstat.numa_hint_faults
-99.3% 37.25 ±148% proc-vmstat.numa_hint_faults_local
-54.6% 23426 ± 53% proc-vmstat.numa_pte_updates
+24.6% 6662 ± 16% proc-vmstat.pgactivate
+1.0% 1756902 proc-vmstat.pgfree
-16.4% 850.75 ± 5% slabinfo.file_lock_cache.active_objs
-16.4% 850.75 ± 5% slabinfo.file_lock_cache.num_objs
-11.2% 4285 ± 3% slabinfo.files_cache.active_objs
-11.2% 4285 ± 3% slabinfo.files_cache.num_objs
-11.0% 2515 slabinfo.sighand_cache.active_objs
-11.0% 2517 slabinfo.sighand_cache.num_objs
-24.1% 873.25 ± 22% slabinfo.task_group.active_objs
-24.1% 873.25 ± 22% slabinfo.task_group.num_objs
-9.2% 54079 ± 3% slabinfo.vm_area_struct.active_objs
-9.1% 1354 ± 3% slabinfo.vm_area_struct.active_slabs
-9.1% 54195 ± 3% slabinfo.vm_area_struct.num_objs
-9.1% 1354 ± 3% slabinfo.vm_area_struct.num_slabs
-2.7% 6.915e+09 perf-stat.i.branch-instructions
-6.0 6.37 ± 23% perf-stat.i.cache-miss-rate%
+5.5% 1558 perf-stat.i.context-switches
-1.3% 110.46 perf-stat.i.cpu-migrations
-2.5% 1.071e+10 perf-stat.i.dTLB-loads
-2.6% 3.724e+09 perf-stat.i.dTLB-stores
-20.1% 1703908 ± 14% perf-stat.i.iTLB-loads
-2.7% 6.618e+10 perf-stat.i.instructions
-4.5% 0.84 ± 2% perf-stat.i.ipc
-1.5% 4014 perf-stat.i.minor-faults
-31.1% 73884 ± 11% perf-stat.i.node-load-misses
-9.6% 12974 ± 3% perf-stat.i.node-loads
-51.0% 13212 ± 18% perf-stat.i.node-store-misses
-1.5% 4014 perf-stat.i.page-faults
-6.1 6.42 ± 24% perf-stat.overall.cache-miss-rate%
+3.8% 1.08 perf-stat.overall.cpi
-3.6% 0.93 perf-stat.overall.ipc
-3.3 84.92 perf-stat.overall.node-load-miss-rate%
-11.9 67.04 ± 8% perf-stat.overall.node-store-miss-rate%
+1.3% 543249 perf-stat.overall.path-length
-2.8% 6.907e+09 perf-stat.ps.branch-instructions
+5.5% 1554 perf-stat.ps.context-switches
-1.3% 110.22 perf-stat.ps.cpu-migrations
-2.6% 1.07e+10 perf-stat.ps.dTLB-loads
-2.7% 3.719e+09 perf-stat.ps.dTLB-stores
-20.1% 1700626 ± 14% perf-stat.ps.iTLB-loads
-2.8% 6.61e+10 perf-stat.ps.instructions
-1.4% 4004 perf-stat.ps.minor-faults
-31.1% 73785 ± 11% perf-stat.ps.node-load-misses
-9.5% 12960 ± 3% perf-stat.ps.node-loads
-51.0% 13193 ± 18% perf-stat.ps.node-store-misses
-1.4% 4004 perf-stat.ps.page-faults
+38.7% 71599 ± 14% sched_debug.cfs_rq:/.exec_clock.avg
+79.0% 160981 ± 14% sched_debug.cfs_rq:/.exec_clock.max
-88.3% 770.93 ±168% sched_debug.cfs_rq:/.exec_clock.min
+140.7% 51593 ± 19% sched_debug.cfs_rq:/.exec_clock.stddev
+39.8% 2305637 ± 14% sched_debug.cfs_rq:/.min_vruntime.avg
+82.5% 5195365 ± 14% sched_debug.cfs_rq:/.min_vruntime.max
-87.1% 29096 ±143% sched_debug.cfs_rq:/.min_vruntime.min
+145.1% 1661129 ± 19% sched_debug.cfs_rq:/.min_vruntime.stddev
-54.9% 58.93 ±100% sched_debug.cfs_rq:/.removed.load_avg.max
-67.2% 6.41 ±100% sched_debug.cfs_rq:/.removed.load_avg.stddev
-54.9% 58.93 ±100% sched_debug.cfs_rq:/.removed.runnable_avg.max
-67.2% 6.41 ±100% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
-254.5% 1660977 ± 63% sched_debug.cfs_rq:/.spread0.avg
+3646.9% 4550704 ± 34% sched_debug.cfs_rq:/.spread0.max
-75.4% -615566 sched_debug.cfs_rq:/.spread0.min
+145.1% 1661130 ± 19% sched_debug.cfs_rq:/.spread0.stddev
-31.6% 4.10 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.avg
-22.8% 19.52 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.stddev
+12.8% 272122 ± 4% sched_debug.cpu.clock.avg
+12.8% 272127 ± 4% sched_debug.cpu.clock.max
+12.8% 272117 ± 4% sched_debug.cpu.clock.min
+12.0% 2.76 ± 4% sched_debug.cpu.clock.stddev
+12.8% 272122 ± 4% sched_debug.cpu.clock_task.avg
+12.8% 272127 ± 4% sched_debug.cpu.clock_task.max
+12.8% 272117 ± 4% sched_debug.cpu.clock_task.min
+12.0% 2.76 ± 4% sched_debug.cpu.clock_task.stddev
+12.7% 5557 ± 3% sched_debug.cpu.nr_switches.avg
+43.7% 5065 ± 17% sched_debug.cpu.nr_switches.stddev
+20.6% 3536 ± 5% sched_debug.cpu.sched_count.avg
+56.8% 30011 ± 48% sched_debug.cpu.sched_count.max
+55.8% 4320 ± 18% sched_debug.cpu.sched_count.stddev
+22.6% 1568 ± 6% sched_debug.cpu.sched_goidle.avg
+57.8% 14789 ± 48% sched_debug.cpu.sched_goidle.max
+56.0% 2144 ± 19% sched_debug.cpu.sched_goidle.stddev
+21.4% 1581 ± 6% sched_debug.cpu.ttwu_count.avg
+56.4% 2276 ± 19% sched_debug.cpu.ttwu_count.stddev
+28.4% 763.77 ± 7% sched_debug.cpu.ttwu_local.avg
+79.3% 4143 ± 10% sched_debug.cpu.ttwu_local.max
+105.0% 708.72 ± 4% sched_debug.cpu.ttwu_local.stddev
+12.8% 272118 ± 4% sched_debug.cpu_clk
+12.8% 271615 ± 4% sched_debug.ktime
+12.7% 272537 ± 4% sched_debug.sched_clk
-97.0% 2.25 ±101% interrupts.CPU1.TLB:TLB_shootdowns
+3409.2% 3711 ± 64% interrupts.CPU100.NMI:Non-maskable_interrupts
+3409.2% 3711 ± 64% interrupts.CPU100.PMI:Performance_monitoring_interrupts
+64.2% 261.00 ± 22% interrupts.CPU100.RES:Rescheduling_interrupts
+190.8% 4532 ± 53% interrupts.CPU101.NMI:Non-maskable_interrupts
+190.8% 4532 ± 53% interrupts.CPU101.PMI:Performance_monitoring_interrupts
+48.5% 266.25 ± 12% interrupts.CPU101.RES:Rescheduling_interrupts
+2066.4% 3098 ± 83% interrupts.CPU102.NMI:Non-maskable_interrupts
+2066.4% 3098 ± 83% interrupts.CPU102.PMI:Performance_monitoring_interrupts
+66.9% 289.50 ± 9% interrupts.CPU102.RES:Rescheduling_interrupts
+68.5% 326.50 ± 17% interrupts.CPU103.RES:Rescheduling_interrupts
-96.0% 129.00 ±128% interrupts.CPU12.RES:Rescheduling_interrupts
-99.4% 2.50 ±103% interrupts.CPU16.TLB:TLB_shootdowns
-97.9% 3.00 ± 62% interrupts.CPU26.TLB:TLB_shootdowns
-99.0% 3.75 ± 51% interrupts.CPU29.TLB:TLB_shootdowns
-97.0% 3.00 ± 84% interrupts.CPU3.TLB:TLB_shootdowns
+6523.3% 6540 ±168% interrupts.CPU31.RES:Rescheduling_interrupts
+110.3% 198.75 ± 22% interrupts.CPU33.RES:Rescheduling_interrupts
+114.2% 214.75 ± 31% interrupts.CPU34.RES:Rescheduling_interrupts
+140.8% 196.25 ± 49% interrupts.CPU35.RES:Rescheduling_interrupts
+5932.7% 5354 ±167% interrupts.CPU36.RES:Rescheduling_interrupts
+111.2% 150.50 ± 16% interrupts.CPU38.RES:Rescheduling_interrupts
+136.8% 154.50 ± 20% interrupts.CPU39.RES:Rescheduling_interrupts
+288.0% 177.50 ± 51% interrupts.CPU40.RES:Rescheduling_interrupts
+69.0% 126.75 ± 7% interrupts.CPU41.RES:Rescheduling_interrupts
+554.8% 4048 ± 51% interrupts.CPU42.NMI:Non-maskable_interrupts
+554.8% 4048 ± 51% interrupts.CPU42.PMI:Performance_monitoring_interrupts
+132.1% 186.25 ± 23% interrupts.CPU42.RES:Rescheduling_interrupts
+596.3% 6000 ± 30% interrupts.CPU43.NMI:Non-maskable_interrupts
+596.3% 6000 ± 30% interrupts.CPU43.PMI:Performance_monitoring_interrupts
+173.6% 189.50 ± 10% interrupts.CPU43.RES:Rescheduling_interrupts
+99.0% 191.50 ± 13% interrupts.CPU44.RES:Rescheduling_interrupts
+246.8% 3931 ± 27% interrupts.CPU45.NMI:Non-maskable_interrupts
+246.8% 3931 ± 27% interrupts.CPU45.PMI:Performance_monitoring_interrupts
+614.5% 4408 ± 38% interrupts.CPU47.NMI:Non-maskable_interrupts
+614.5% 4408 ± 38% interrupts.CPU47.PMI:Performance_monitoring_interrupts
+945.3% 6449 ± 22% interrupts.CPU49.NMI:Non-maskable_interrupts
+945.3% 6449 ± 22% interrupts.CPU49.PMI:Performance_monitoring_interrupts
+2323.5% 1963 ±159% interrupts.CPU49.RES:Rescheduling_interrupts
-96.8% 116.25 ± 40% interrupts.CPU5.NMI:Non-maskable_interrupts
-96.8% 116.25 ± 40% interrupts.CPU5.PMI:Performance_monitoring_interrupts
-96.4% 124.50 ± 43% interrupts.CPU57.NMI:Non-maskable_interrupts
-96.4% 124.50 ± 43% interrupts.CPU57.PMI:Performance_monitoring_interrupts
-72.3% 1362 ±157% interrupts.CPU62.NMI:Non-maskable_interrupts
-72.3% 1362 ±157% interrupts.CPU62.PMI:Performance_monitoring_interrupts
-67.1% 1342 ±157% interrupts.CPU64.NMI:Non-maskable_interrupts
-67.1% 1342 ±157% interrupts.CPU64.PMI:Performance_monitoring_interrupts
-85.4% 35.75 ± 97% interrupts.CPU7.RES:Rescheduling_interrupts
-94.4% 3.50 ± 95% interrupts.CPU77.RES:Rescheduling_interrupts
+217.0% 7399 ± 6% interrupts.CPU78.NMI:Non-maskable_interrupts
+217.0% 7399 ± 6% interrupts.CPU78.PMI:Performance_monitoring_interrupts
+157.6% 342.00 ± 22% interrupts.CPU78.RES:Rescheduling_interrupts
+103.5% 261.00 ± 14% interrupts.CPU79.RES:Rescheduling_interrupts
-99.0% 33.00 ±101% interrupts.CPU8.RES:Rescheduling_interrupts
+179.1% 6924 ± 17% interrupts.CPU80.NMI:Non-maskable_interrupts
+179.1% 6924 ± 17% interrupts.CPU80.PMI:Performance_monitoring_interrupts
+100.9% 266.25 ± 14% interrupts.CPU80.RES:Rescheduling_interrupts
+90.3% 250.25 ± 20% interrupts.CPU81.RES:Rescheduling_interrupts
+371.1% 7400 ± 6% interrupts.CPU82.NMI:Non-maskable_interrupts
+371.1% 7400 ± 6% interrupts.CPU82.PMI:Performance_monitoring_interrupts
+2020.3% 2793 ±159% interrupts.CPU82.RES:Rescheduling_interrupts
-87.3% 97.50 ±105% interrupts.CPU9.RES:Rescheduling_interrupts
+79.3% 264.50 ± 21% interrupts.CPU90.RES:Rescheduling_interrupts
+1677.8% 3209 ±158% interrupts.CPU91.RES:Rescheduling_interrupts
+119.1% 360.50 ± 26% interrupts.CPU93.RES:Rescheduling_interrupts
+307.6% 6411 ± 23% interrupts.CPU94.NMI:Non-maskable_interrupts
+307.6% 6411 ± 23% interrupts.CPU94.PMI:Performance_monitoring_interrupts
+76.9% 262.25 ± 17% interrupts.CPU94.RES:Rescheduling_interrupts
+68.5% 242.25 ± 16% interrupts.CPU95.RES:Rescheduling_interrupts
+127.0% 383.00 ± 48% interrupts.CPU96.RES:Rescheduling_interrupts
+181.2% 6415 ± 23% interrupts.CPU97.NMI:Non-maskable_interrupts
+181.2% 6415 ± 23% interrupts.CPU97.PMI:Performance_monitoring_interrupts
+71.3% 280.50 ± 16% interrupts.CPU97.RES:Rescheduling_interrupts
+5459.6% 4711 ± 64% interrupts.CPU98.NMI:Non-maskable_interrupts
+5459.6% 4711 ± 64% interrupts.CPU98.PMI:Performance_monitoring_interrupts
+91.2% 337.00 ± 23% interrupts.CPU98.RES:Rescheduling_interrupts
+61.5% 265.25 ± 20% interrupts.CPU99.RES:Rescheduling_interrupts
+27.8% 310009 ± 13% interrupts.NMI:Non-maskable_interrupts
+27.8% 310009 ± 13% interrupts.PMI:Performance_monitoring_interrupts
-81.4% 714.00 ± 62% interrupts.TLB:TLB_shootdowns
+59.4% 62496 ± 8% softirqs.CPU1.SCHED
+11.6% 125580 ± 6% softirqs.CPU10.RCU
+34.8% 64105 ± 6% softirqs.CPU10.SCHED
+34.5% 155325 ± 5% softirqs.CPU100.RCU
-34.4% 27593 ± 14% softirqs.CPU100.SCHED
+33.4% 154231 ± 4% softirqs.CPU101.RCU
-32.4% 28661 ± 9% softirqs.CPU101.SCHED
+37.9% 156145 ± 4% softirqs.CPU102.RCU
-42.3% 25316 ± 19% softirqs.CPU102.SCHED
+35.3% 155692 ± 6% softirqs.CPU103.RCU
-34.8% 27428 ± 19% softirqs.CPU103.SCHED
+26.8% 63739 ± 2% softirqs.CPU12.SCHED
+24.5% 63567 ± 2% softirqs.CPU13.SCHED
+15.1% 128441 ± 6% softirqs.CPU14.RCU
+21.9% 62437 ± 6% softirqs.CPU14.SCHED
+31.4% 139927 ± 7% softirqs.CPU15.RCU
+31.4% 140038 ± 10% softirqs.CPU16.RCU
+22.9% 126467 ± 7% softirqs.CPU17.RCU
+23.8% 127097 ± 8% softirqs.CPU18.RCU
+27.2% 131067 ± 8% softirqs.CPU19.RCU
+64.7% 66094 ± 16% softirqs.CPU2.SCHED
+31.3% 139080 ± 11% softirqs.CPU20.RCU
+10.3% 62534 ± 4% softirqs.CPU20.SCHED
+25.5% 135380 ± 7% softirqs.CPU21.RCU
+22.9% 128563 ± 7% softirqs.CPU22.RCU
+9.0% 62636 ± 4% softirqs.CPU22.SCHED
+26.9% 136472 ± 9% softirqs.CPU23.RCU
+21.9% 126077 ± 9% softirqs.CPU24.RCU
+8.9% 63018 ± 3% softirqs.CPU24.SCHED
+27.4% 132281 ± 10% softirqs.CPU25.RCU
+39.2% 162714 ± 3% softirqs.CPU26.RCU
+14.5% 46275 ± 10% softirqs.CPU26.SCHED
+38.2% 163001 ± 3% softirqs.CPU27.RCU
+36.5% 162677 ± 2% softirqs.CPU28.RCU
+31.8% 205398 ± 11% softirqs.CPU28.TIMER
+39.4% 167637 ± 3% softirqs.CPU29.RCU
+28.9% 213346 ± 9% softirqs.CPU29.TIMER
+47.3% 61528 ± 12% softirqs.CPU3.SCHED
+27.8% 142261 ± 3% softirqs.CPU30.RCU
+19.3% 130961 ± 4% softirqs.CPU31.RCU
+27.8% 209524 ± 10% softirqs.CPU31.TIMER
+25.3% 138304 ± 12% softirqs.CPU32.RCU
+27.2% 141121 ± 4% softirqs.CPU33.RCU
+27.5% 142697 ± 8% softirqs.CPU34.RCU
+25.8% 207277 ± 10% softirqs.CPU34.TIMER
+27.6% 140414 ± 11% softirqs.CPU35.RCU
+26.4% 210021 ± 9% softirqs.CPU35.TIMER
+26.8% 136882 ± 4% softirqs.CPU36.RCU
+23.4% 206530 ± 11% softirqs.CPU36.TIMER
+28.0% 137185 ± 7% softirqs.CPU37.RCU
-24.0% 40128 ± 16% softirqs.CPU37.SCHED
+20.5% 207427 ± 11% softirqs.CPU37.TIMER
+18.7% 126777 ± 6% softirqs.CPU38.RCU
+24.9% 211164 ± 9% softirqs.CPU38.TIMER
+29.2% 140872 ± 10% softirqs.CPU39.RCU
-12.9% 46051 ± 3% softirqs.CPU39.SCHED
+25.8% 211848 ± 8% softirqs.CPU39.TIMER
+40.9% 61065 ± 9% softirqs.CPU4.SCHED
+29.5% 141157 ± 8% softirqs.CPU40.RCU
-23.8% 41002 ± 25% softirqs.CPU40.SCHED
+19.7% 128985 ± 5% softirqs.CPU41.RCU
-16.3% 48293 ± 5% softirqs.CPU41.SCHED
+29.1% 134762 ± 6% softirqs.CPU42.RCU
-22.2% 44093 ± 8% softirqs.CPU42.SCHED
+18.9% 209864 ± 9% softirqs.CPU42.TIMER
+30.8% 133570 ± 3% softirqs.CPU43.RCU
-28.5% 41392 ± 6% softirqs.CPU43.SCHED
+17.1% 207557 ± 9% softirqs.CPU43.TIMER
+27.6% 131155 ± 6% softirqs.CPU44.RCU
-22.0% 44481 ± 7% softirqs.CPU44.SCHED
+17.0% 210978 ± 8% softirqs.CPU44.TIMER
+47.1% 152906 ± 2% softirqs.CPU45.RCU
-19.4% 46698 ± 7% softirqs.CPU45.SCHED
+17.5% 211546 ± 8% softirqs.CPU45.TIMER
+46.6% 156185 ± 3% softirqs.CPU46.RCU
-24.8% 44309 ± 6% softirqs.CPU46.SCHED
+16.8% 205464 ± 11% softirqs.CPU46.TIMER
+45.6% 156681 ± 2% softirqs.CPU47.RCU
-26.8% 44767 ± 9% softirqs.CPU47.SCHED
+18.6% 207548 ± 10% softirqs.CPU47.TIMER
+47.8% 154014 ± 3% softirqs.CPU48.RCU
-19.3% 45611 ± 16% softirqs.CPU48.SCHED
+18.4% 211119 ± 8% softirqs.CPU48.TIMER
+46.8% 154628 ± 3% softirqs.CPU49.RCU
+18.9% 209524 ± 9% softirqs.CPU49.TIMER
+40.7% 61423 ± 11% softirqs.CPU5.SCHED
+46.2% 152326 ± 3% softirqs.CPU50.RCU
-15.3% 49718 ± 11% softirqs.CPU50.SCHED
+16.7% 210254 ± 9% softirqs.CPU50.TIMER
+46.2% 154243 ± 2% softirqs.CPU51.RCU
+39.1% 148420 ± 10% softirqs.CPU52.RCU
+35.5% 146416 ± 11% softirqs.CPU53.RCU
+21.0% 61284 ± 9% softirqs.CPU53.SCHED
+29.5% 143423 ± 8% softirqs.CPU54.RCU
+24.6% 62062 ± 6% softirqs.CPU54.SCHED
+29.6% 141149 ± 9% softirqs.CPU55.RCU
+28.1% 62502 ± 6% softirqs.CPU55.SCHED
+30.2% 142335 ± 10% softirqs.CPU56.RCU
+29.7% 63244 ± 4% softirqs.CPU56.SCHED
+32.4% 142500 ± 8% softirqs.CPU57.RCU
+32.8% 62320 ± 6% softirqs.CPU57.SCHED
+32.1% 142541 ± 9% softirqs.CPU58.RCU
+29.7% 62112 ± 10% softirqs.CPU58.SCHED
+31.6% 147720 ± 9% softirqs.CPU59.RCU
+33.9% 60536 ± 11% softirqs.CPU59.SCHED
+44.9% 62698 ± 5% softirqs.CPU6.SCHED
+41.6% 60815 ± 10% softirqs.CPU60.SCHED
+48.2% 60919 ± 10% softirqs.CPU61.SCHED
+41.0% 61422 ± 8% softirqs.CPU62.SCHED
+45.0% 61473 ± 8% softirqs.CPU63.SCHED
+35.2% 60539 ± 11% softirqs.CPU65.SCHED
+34.9% 61049 ± 9% softirqs.CPU67.SCHED
+32.4% 61099 ± 12% softirqs.CPU69.SCHED
+13.0% 128950 ± 9% softirqs.CPU7.RCU
+42.6% 62558 ± 4% softirqs.CPU7.SCHED
+42.1% 63939 ± 2% softirqs.CPU70.SCHED
+31.2% 60706 ± 10% softirqs.CPU71.SCHED
+29.5% 61097 ± 9% softirqs.CPU72.SCHED
+37.5% 61916 ± 7% softirqs.CPU73.SCHED
+38.4% 61533 ± 8% softirqs.CPU74.SCHED
+32.7% 154642 ± 7% softirqs.CPU75.RCU
+41.5% 60952 ± 10% softirqs.CPU75.SCHED
+33.8% 153786 ± 7% softirqs.CPU76.RCU
+40.3% 62390 ± 14% softirqs.CPU76.SCHED
+34.1% 153870 ± 7% softirqs.CPU77.RCU
+31.3% 61066 ± 8% softirqs.CPU77.SCHED
+39.8% 152515 ± 3% softirqs.CPU78.RCU
-44.1% 29615 ± 22% softirqs.CPU78.SCHED
+34.9% 152829 ± 3% softirqs.CPU79.RCU
-39.3% 29950 ± 19% softirqs.CPU79.SCHED
+10.7% 126743 ± 6% softirqs.CPU8.RCU
+35.6% 62367 ± 5% softirqs.CPU8.SCHED
+40.3% 150622 ± 3% softirqs.CPU80.RCU
-40.0% 30997 ± 16% softirqs.CPU80.SCHED
+48.3% 153578 ± 3% softirqs.CPU81.RCU
-45.0% 30058 ± 18% softirqs.CPU81.SCHED
+38.1% 147383 ± 6% softirqs.CPU82.RCU
+38.3% 148549 ± 3% softirqs.CPU83.RCU
-29.7% 34026 ± 15% softirqs.CPU83.SCHED
+30.6% 142391 ± 11% softirqs.CPU84.RCU
+30.6% 145830 ± 7% softirqs.CPU85.RCU
+33.3% 147882 ± 3% softirqs.CPU86.RCU
+31.2% 145429 ± 7% softirqs.CPU87.RCU
+34.2% 150213 ± 3% softirqs.CPU88.RCU
-22.1% 32856 ± 15% softirqs.CPU88.SCHED
+33.9% 149766 ± 3% softirqs.CPU89.RCU
+32.5% 63122 ± 3% softirqs.CPU9.SCHED
+38.3% 155970 ± 5% softirqs.CPU90.RCU
-37.4% 28321 ± 23% softirqs.CPU90.SCHED
+32.4% 155399 ± 4% softirqs.CPU91.RCU
+26.6% 148101 ± 6% softirqs.CPU92.RCU
+39.4% 158341 ± 4% softirqs.CPU93.RCU
-43.2% 25823 ± 27% softirqs.CPU93.SCHED
+38.5% 155456 ± 3% softirqs.CPU94.RCU
+34.2% 152470 ± 5% softirqs.CPU95.RCU
-33.9% 30021 ± 17% softirqs.CPU95.SCHED
+38.3% 156199 ± 4% softirqs.CPU96.RCU
-40.9% 26191 ± 20% softirqs.CPU96.SCHED
+36.9% 155146 ± 6% softirqs.CPU97.RCU
-40.0% 26290 ± 19% softirqs.CPU97.SCHED
+32.5% 154262 ± 7% softirqs.CPU98.RCU
-31.1% 28239 ± 22% softirqs.CPU98.SCHED
+33.7% 155277 ± 5% softirqs.CPU99.RCU
-33.7% 27372 ± 20% softirqs.CPU99.SCHED
+25.8% 14489368 ± 4% softirqs.RCU
+16.4% 20469522 ± 4% softirqs.TIMER
**********************************************************************************
104 threads Skylake with 192G memory
========================================================================
governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
ce/x86_64-rhel-7.6/30%/debian-x86_64-20191114.cgz/300s/lkp-skl-fpga01/context1/unixbench/0x2000065
("threads: Update PID limit comment according to futex UAPI change")
("sched/fair: Don't pull task if local group is more loaded than busiest group")
59901cb4520c44bfce81f523bc6
---------------------------
%reproduction fail:runs
| |
-25% :4 kmsg.ipmi_si_dmi-ipmi-si.#:IRQ_index#not_found
%change %stddev
| \
+29.1% 4022 ± 3% unixbench.score
-36.2% 9850 ± 12% unixbench.time.involuntary_context_switches
-15.0% 1797 ± 3% unixbench.time.percent_of_cpu_this_job_got
-17.5% 6433 ± 3% unixbench.time.system_time
+24.2% 593.20 unixbench.time.user_time
+23.6% 4.561e+08 ± 2% unixbench.time.voluntary_context_switches
+29.0% 6.288e+08 ± 3% unixbench.workload
-13.1% 649552 ± 6% meminfo.DirectMap4k
+0.6 1.99 ± 3% mpstat.cpu.all.usr%
+205.5% 554.50 ± 61% numa-meminfo.node0.Mlocked
+207.2% 138.25 ± 61% numa-vmstat.node0.nr_mlock
+1.3% 77.00 vmstat.cpu.id
+20.0% 4498355 ± 2% vmstat.system.cs
-4.2% 227776 vmstat.system.in
-11.5% 2596 ± 5% slabinfo.khugepaged_mm_slot.active_objs
-11.5% 2596 ± 5% slabinfo.khugepaged_mm_slot.num_objs
-22.8% 895.75 ± 10% slabinfo.task_group.active_objs
-22.8% 895.75 ± 10% slabinfo.task_group.num_objs
-24.7% 5.932e+09 ± 5% cpuidle.C1.time
+99.3% 1.783e+10 ± 30% cpuidle.C1E.time
+37.6% 39802094 ± 20% cpuidle.C1E.usage
+1040.8% 3.761e+08 ± 16% cpuidle.POLL.time
+3571.2% 1.219e+08 ± 17% cpuidle.POLL.usage
-1.2% 76570 proc-vmstat.nr_active_anon
-1.6% 44026 proc-vmstat.nr_slab_unreclaimable
-1.2% 76570 proc-vmstat.nr_zone_active_anon
-56.0% 4844 ± 56% proc-vmstat.numa_hint_faults
-87.9% 218.75 ± 52% proc-vmstat.numa_hint_faults_local
-11.1% 9016 ± 3% proc-vmstat.pgactivate
-16.9% 3066128 ± 10% sched_debug.cfs_rq:/.MIN_vruntime.max
-15.6% 1053151 ± 11% sched_debug.cfs_rq:/.MIN_vruntime.stddev
-14.6% 62394 ± 5% sched_debug.cfs_rq:/.exec_clock.avg
-38.8% 28463 ± 28% sched_debug.cfs_rq:/.exec_clock.min
+94.8% 19737 ± 27% sched_debug.cfs_rq:/.exec_clock.stddev
+37.1% 17715 ± 27% sched_debug.cfs_rq:/.load.avg
-16.9% 3066129 ± 10% sched_debug.cfs_rq:/.max_vruntime.max
-15.6% 1053152 ± 11% sched_debug.cfs_rq:/.max_vruntime.stddev
-21.7% 2680459 ± 5% sched_debug.cfs_rq:/.min_vruntime.avg
-44.6% 1213462 ± 28% sched_debug.cfs_rq:/.min_vruntime.min
+77.8% 838686 ± 25% sched_debug.cfs_rq:/.min_vruntime.stddev
+90.8% 0.46 ± 19% sched_debug.cfs_rq:/.nr_spread_over.avg
+151.8% 6.65 ± 55% sched_debug.cfs_rq:/.nr_spread_over.max
+137.0% 1.20 ± 44% sched_debug.cfs_rq:/.nr_spread_over.stddev
+37.1% 1970 ± 6% sched_debug.cfs_rq:/.runnable_avg.max
+15.1% 432.07 ± 4% sched_debug.cfs_rq:/.runnable_avg.stddev
+1058.8% -1123518 sched_debug.cfs_rq:/.spread0.min
+77.8% 838694 ± 25% sched_debug.cfs_rq:/.spread0.stddev
+45.9% 1253 ± 13% sched_debug.cfs_rq:/.util_avg.max
+56.7% 1017 ± 15% sched_debug.cfs_rq:/.util_est_enqueued.max
+19.3% 527581 ± 3% sched_debug.cpu.avg_idle.avg
+15.5% 594740 ± 13% sched_debug.cpu.max_idle_balance_cost.max
+613.0% 10298 ± 72% sched_debug.cpu.max_idle_balance_cost.stddev
-16.2% 0.00 ± 8% sched_debug.cpu.next_balance.stddev
+81.0% 1.88 ± 22% sched_debug.cpu.nr_running.max
+12.1% 0.46 ± 6% sched_debug.cpu.nr_running.stddev
+33.5% 10769101 ± 11% sched_debug.cpu.nr_switches.max
+206.7% 2806624 ± 32% sched_debug.cpu.nr_switches.stddev
+33.5% 10761752 ± 11% sched_debug.cpu.sched_count.max
+206.7% 2806145 ± 32% sched_debug.cpu.sched_count.stddev
+29.7% 5227316 ± 11% sched_debug.cpu.sched_goidle.max
+196.6% 1356319 ± 31% sched_debug.cpu.sched_goidle.stddev
+17.6% 3799652 ± 8% sched_debug.cpu.ttwu_count.avg
+37.2% 5533499 ± 11% sched_debug.cpu.ttwu_count.max
+217.1% 1449903 ± 33% sched_debug.cpu.ttwu_count.stddev
+952.2% 5229 ± 17% sched_debug.cpu.ttwu_local.avg
+449.7% 8311 ± 24% sched_debug.cpu.ttwu_local.max
+688.8% 2797 ± 29% sched_debug.cpu.ttwu_local.min
+1584.8% 2201 ± 49% sched_debug.cpu.ttwu_local.stddev
-15.3% 82438 ± 10% softirqs.CPU100.SCHED
-13.2% 83091 ± 10% softirqs.CPU101.SCHED
-15.3% 83044 ± 11% softirqs.CPU102.SCHED
+9.5% 147696 ± 3% softirqs.CPU33.TIMER
-9.8% 90488 ± 8% softirqs.CPU34.SCHED
-10.0% 88752 ± 8% softirqs.CPU36.SCHED
-12.5% 88834 ± 8% softirqs.CPU37.SCHED
-14.5% 88256 ± 10% softirqs.CPU38.SCHED
-15.1% 86641 ± 9% softirqs.CPU39.SCHED
-15.9% 86642 ± 11% softirqs.CPU40.SCHED
-16.7% 86299 ± 10% softirqs.CPU41.SCHED
+8.1% 147364 ± 3% softirqs.CPU41.TIMER
-18.4% 84916 ± 11% softirqs.CPU42.SCHED
-25.6% 77445 ± 19% softirqs.CPU43.SCHED
-18.3% 84779 ± 11% softirqs.CPU44.SCHED
-19.0% 84947 ± 12% softirqs.CPU45.SCHED
-21.7% 83209 ± 11% softirqs.CPU46.SCHED
-21.9% 82708 ± 12% softirqs.CPU47.SCHED
-21.0% 83838 ± 10% softirqs.CPU48.SCHED
-20.5% 82800 ± 11% softirqs.CPU49.SCHED
-20.8% 83521 ± 10% softirqs.CPU50.SCHED
-12.8% 92988 ± 6% softirqs.CPU52.SCHED
-23.6% 84129 ± 11% softirqs.CPU78.SCHED
-22.3% 83897 ± 11% softirqs.CPU79.SCHED
-23.5% 84412 ± 10% softirqs.CPU80.SCHED
-21.3% 84966 ± 10% softirqs.CPU81.SCHED
-22.6% 83318 ± 13% softirqs.CPU82.SCHED
-22.4% 83344 ± 11% softirqs.CPU83.SCHED
-21.5% 83345 ± 12% softirqs.CPU84.SCHED
-20.0% 83933 ± 11% softirqs.CPU85.SCHED
-19.3% 83770 ± 11% softirqs.CPU86.SCHED
-18.6% 84093 ± 11% softirqs.CPU87.SCHED
-20.2% 83629 ± 12% softirqs.CPU88.SCHED
-19.5% 83676 ± 12% softirqs.CPU89.SCHED
-17.8% 83192 ± 11% softirqs.CPU90.SCHED
-19.4% 83322 ± 11% softirqs.CPU91.SCHED
-16.9% 82889 ± 12% softirqs.CPU92.SCHED
-15.8% 84012 ± 11% softirqs.CPU93.SCHED
-15.8% 83927 ± 11% softirqs.CPU94.SCHED
-17.3% 83351 ± 11% softirqs.CPU95.SCHED
-16.3% 83351 ± 10% softirqs.CPU96.SCHED
-16.4% 82483 ± 11% softirqs.CPU97.SCHED
-16.5% 82919 ± 11% softirqs.CPU98.SCHED
-15.7% 82320 ± 11% softirqs.CPU99.SCHED
-41.8% 17.86 ± 2% perf-stat.i.MPKI
+21.0% 7.537e+09 ± 2% perf-stat.i.branch-instructions
+26.1% 1.103e+08 ± 2% perf-stat.i.branch-misses
-61.0% 10190442 ± 20% perf-stat.i.cache-misses
-3.2% 6.866e+08 perf-stat.i.cache-references
+20.0% 4520959 ± 2% perf-stat.i.context-switches
-22.6% 2.72 ± 3% perf-stat.i.cpi
+2.8% 8.8e+10 perf-stat.i.cpu-cycles
151679 ± 18% perf-stat.i.cpu-migrations
+1896.7% 66426 ± 59% perf-stat.i.cycles-between-cache-misses
+260.9% 15765930 ± 11% perf-stat.i.dTLB-load-misses
+21.3% 9.275e+09 ± 2% perf-stat.i.dTLB-loads
+1345.4% 1207309 ± 15% perf-stat.i.dTLB-store-misses
+22.5% 5.512e+09 ± 2% perf-stat.i.dTLB-stores
+49.5% 6597934 ± 4% perf-stat.i.iTLB-load-misses
+25.9% 36603347 perf-stat.i.iTLB-loads
+21.5% 3.444e+10 ± 2% perf-stat.i.instructions
+21.5% 0.38 ± 3% perf-stat.i.ipc
-2.4 92.45 perf-stat.i.node-load-miss-rate%
-58.8% 1678711 ± 18% perf-stat.i.node-load-misses
-60.1% 58372 ± 13% perf-stat.i.node-loads
-10.6 85.36 perf-stat.i.node-store-miss-rate%
-59.7% 1539813 ± 20% perf-stat.i.node-store-misses
-8.5% 10571 ± 5% perf-stat.i.node-stores
-20.3% 19.94 ± 2% perf-stat.overall.MPKI
+0.1 1.46 perf-stat.overall.branch-miss-rate%
-2.2 1.49 ± 21% perf-stat.overall.cache-miss-rate%
-15.4% 2.56 ± 2% perf-stat.overall.cpi
+178.9% 9127 ± 25% perf-stat.overall.cycles-between-cache-misses
+0.1 0.17 ± 9% perf-stat.overall.dTLB-load-miss-rate%
+0.0 0.02 ± 13% perf-stat.overall.dTLB-store-miss-rate%
+2.1 15.27 ± 3% perf-stat.overall.iTLB-load-miss-rate%
-18.7% 5224 ± 2% perf-stat.overall.instructions-per-iTLB-miss
+18.2% 0.39 ± 2% perf-stat.overall.ipc
-5.8% 21419 perf-stat.overall.path-length
+21.1% 7.518e+09 ± 2% perf-stat.ps.branch-instructions
+26.1% 1.101e+08 ± 2% perf-stat.ps.branch-misses
-61.1% 10158408 ± 20% perf-stat.ps.cache-misses
-3.2% 6.848e+08 perf-stat.ps.cache-references
+20.1% 4509479 ± 2% perf-stat.ps.context-switches
+2.8% 8.777e+10 perf-stat.ps.cpu-cycles
151373 ± 18% perf-stat.ps.cpu-migrations
+261.1% 15731108 ± 11% perf-stat.ps.dTLB-load-misses
+21.4% 9.251e+09 ± 2% perf-stat.ps.dTLB-loads
+1343.5% 1204761 ± 15% perf-stat.ps.dTLB-store-misses
+22.5% 5.498e+09 ± 2% perf-stat.ps.dTLB-stores
+49.6% 6581866 ± 4% perf-stat.ps.iTLB-load-misses
+25.9% 36508403 perf-stat.ps.iTLB-loads
+21.6% 3.436e+10 ± 2% perf-stat.ps.instructions
-58.8% 1673367 ± 18% perf-stat.ps.node-load-misses
-60.0% 58229 ± 13% perf-stat.ps.node-loads
-59.7% 1534753 ± 20% perf-stat.ps.node-store-misses
-8.5% 10555 ± 5% perf-stat.ps.node-stores
+21.5% 1.346e+13 ± 2% perf-stat.total.instructions
-1.5% 48784 interrupts.CAL:Function_call_interrupts
-21.1% 59624 ± 19% interrupts.CPU0.RES:Rescheduling_interrupts
-83.7% 47.75 ± 39% interrupts.CPU0.TLB:TLB_shootdowns
-90.6% 155.75 ± 89% interrupts.CPU1.TLB:TLB_shootdowns
-25.4% 80516 ± 13% interrupts.CPU10.RES:Rescheduling_interrupts
-70.3% 15.75 ± 34% interrupts.CPU10.TLB:TLB_shootdowns
-45.3% 57407 ± 29% interrupts.CPU100.RES:Rescheduling_interrupts
-42.4% 58608 ± 28% interrupts.CPU101.RES:Rescheduling_interrupts
-45.2% 57146 ± 32% interrupts.CPU102.RES:Rescheduling_interrupts
-70.6% 891.00 ±149% interrupts.CPU103.NMI:Non-maskable_interrupts
-70.6% 891.00 ±149% interrupts.CPU103.PMI:Performance_monitoring_interrupts
-43.9% 54866 ± 25% interrupts.CPU103.RES:Rescheduling_interrupts
-25.2% 79442 ± 18% interrupts.CPU11.RES:Rescheduling_interrupts
-26.2% 81077 ± 13% interrupts.CPU12.RES:Rescheduling_interrupts
-28.4% 79170 ± 17% interrupts.CPU13.RES:Rescheduling_interrupts
-28.9% 80064 ± 13% interrupts.CPU14.RES:Rescheduling_interrupts
-92.1% 16.00 ± 76% interrupts.CPU14.TLB:TLB_shootdowns
-29.9% 81035 ± 16% interrupts.CPU15.RES:Rescheduling_interrupts
+56.1% 3646 ± 24% interrupts.CPU16.NMI:Non-maskable_interrupts
+56.1% 3646 ± 24% interrupts.CPU16.PMI:Performance_monitoring_interrupts
-31.0% 78040 ± 14% interrupts.CPU16.RES:Rescheduling_interrupts
-34.2% 77376 ± 15% interrupts.CPU17.RES:Rescheduling_interrupts
-35.9% 76821 ± 18% interrupts.CPU18.RES:Rescheduling_interrupts
-93.0% 14.50 ± 51% interrupts.CPU18.TLB:TLB_shootdowns
-33.1% 77798 ± 18% interrupts.CPU19.RES:Rescheduling_interrupts
-87.8% 14.50 ± 51% interrupts.CPU19.TLB:TLB_shootdowns
-97.1% 13.75 ± 37% interrupts.CPU2.TLB:TLB_shootdowns
-34.2% 78873 ± 16% interrupts.CPU20.RES:Rescheduling_interrupts
-33.7% 79035 ± 10% interrupts.CPU21.RES:Rescheduling_interrupts
-33.6% 79531 ± 21% interrupts.CPU22.RES:Rescheduling_interrupts
-33.6% 79145 ± 15% interrupts.CPU23.RES:Rescheduling_interrupts
-94.8% 12.50 ± 47% interrupts.CPU23.TLB:TLB_shootdowns
-32.0% 77747 ± 18% interrupts.CPU24.RES:Rescheduling_interrupts
-87.2% 16.75 ± 80% interrupts.CPU24.TLB:TLB_shootdowns
-27.4% 83205 ± 15% interrupts.CPU25.RES:Rescheduling_interrupts
-90.8% 36.75 ± 37% interrupts.CPU26.TLB:TLB_shootdowns
-81.2% 417.25 ± 96% interrupts.CPU27.TLB:TLB_shootdowns
-76.4% 249.75 ± 54% interrupts.CPU29.TLB:TLB_shootdowns
-14.7% 80458 ± 17% interrupts.CPU3.RES:Rescheduling_interrupts
-80.6% 85.25 ±138% interrupts.CPU3.TLB:TLB_shootdowns
-69.4% 91.00 ±123% interrupts.CPU30.TLB:TLB_shootdowns
-17.3% 82125 ± 18% interrupts.CPU31.RES:Rescheduling_interrupts
-22.1% 79310 ± 22% interrupts.CPU32.RES:Rescheduling_interrupts
-23.9% 79216 ± 21% interrupts.CPU33.RES:Rescheduling_interrupts
-30.2% 75649 ± 19% interrupts.CPU34.RES:Rescheduling_interrupts
-30.2% 75255 ± 22% interrupts.CPU35.RES:Rescheduling_interrupts
-38.5% 1875 ± 56% interrupts.CPU36.NMI:Non-maskable_interrupts
-38.5% 1875 ± 56% interrupts.CPU36.PMI:Performance_monitoring_interrupts
-36.0% 70961 ± 21% interrupts.CPU36.RES:Rescheduling_interrupts
-47.5% 1573 ± 56% interrupts.CPU37.NMI:Non-maskable_interrupts
-47.5% 1573 ± 56% interrupts.CPU37.PMI:Performance_monitoring_interrupts
-35.9% 70912 ± 21% interrupts.CPU37.RES:Rescheduling_interrupts
-60.7% 1158 ± 70% interrupts.CPU38.NMI:Non-maskable_interrupts
-60.7% 1158 ± 70% interrupts.CPU38.PMI:Performance_monitoring_interrupts
-39.3% 69605 ± 27% interrupts.CPU38.RES:Rescheduling_interrupts
-53.1% 1387 ± 64% interrupts.CPU39.NMI:Non-maskable_interrupts
-53.1% 1387 ± 64% interrupts.CPU39.PMI:Performance_monitoring_interrupts
-41.0% 66597 ± 21% interrupts.CPU39.RES:Rescheduling_interrupts
-19.1% 77895 ± 15% interrupts.CPU4.RES:Rescheduling_interrupts
-62.4% 1158 ± 67% interrupts.CPU40.NMI:Non-maskable_interrupts
-62.4% 1158 ± 67% interrupts.CPU40.PMI:Performance_monitoring_interrupts
-42.7% 66275 ± 28% interrupts.CPU40.RES:Rescheduling_interrupts
-94.3% 19.25 ± 40% interrupts.CPU40.TLB:TLB_shootdowns
-59.0% 1273 ± 57% interrupts.CPU41.NMI:Non-maskable_interrupts
-59.0% 1273 ± 57% interrupts.CPU41.PMI:Performance_monitoring_interrupts
-45.4% 64133 ± 26% interrupts.CPU41.RES:Rescheduling_interrupts
-64.3% 1097 ± 89% interrupts.CPU42.NMI:Non-maskable_interrupts
-64.3% 1097 ± 89% interrupts.CPU42.PMI:Performance_monitoring_interrupts
-47.7% 62131 ± 30% interrupts.CPU42.RES:Rescheduling_interrupts
-70.3% 898.50 ± 83% interrupts.CPU43.NMI:Non-maskable_interrupts
-70.3% 898.50 ± 83% interrupts.CPU43.PMI:Performance_monitoring_interrupts
-48.2% 60473 ± 29% interrupts.CPU43.RES:Rescheduling_interrupts
-71.5% 853.50 ±116% interrupts.CPU44.NMI:Non-maskable_interrupts
-71.5% 853.50 ±116% interrupts.CPU44.PMI:Performance_monitoring_interrupts
-48.6% 61233 ± 30% interrupts.CPU44.RES:Rescheduling_interrupts
-48.7% 61050 ± 32% interrupts.CPU45.RES:Rescheduling_interrupts
-52.1% 58842 ± 30% interrupts.CPU46.RES:Rescheduling_interrupts
-53.5% 57035 ± 33% interrupts.CPU47.RES:Rescheduling_interrupts
-51.7% 59540 ± 29% interrupts.CPU48.RES:Rescheduling_interrupts
-88.6% 14.00 ± 36% interrupts.CPU48.TLB:TLB_shootdowns
-53.4% 57374 ± 32% interrupts.CPU49.RES:Rescheduling_interrupts
-92.3% 13.50 ± 24% interrupts.CPU49.TLB:TLB_shootdowns
-16.2% 79161 ± 17% interrupts.CPU5.RES:Rescheduling_interrupts
-51.3% 58853 ± 28% interrupts.CPU50.RES:Rescheduling_interrupts
-53.5% 56355 ± 35% interrupts.CPU51.RES:Rescheduling_interrupts
-32.8% 85784 ± 14% interrupts.CPU52.RES:Rescheduling_interrupts
-33.4% 81461 ± 14% interrupts.CPU53.RES:Rescheduling_interrupts
-32.0% 81484 ± 19% interrupts.CPU54.RES:Rescheduling_interrupts
-94.3% 13.25 ± 86% interrupts.CPU54.TLB:TLB_shootdowns
-27.8% 82728 ± 16% interrupts.CPU55.RES:Rescheduling_interrupts
-84.7% 14.00 ± 59% interrupts.CPU55.TLB:TLB_shootdowns
-28.0% 80980 ± 19% interrupts.CPU56.RES:Rescheduling_interrupts
-95.1% 8.75 ± 74% interrupts.CPU56.TLB:TLB_shootdowns
-26.4% 81457 ± 17% interrupts.CPU57.RES:Rescheduling_interrupts
-97.1% 9.00 ± 73% interrupts.CPU57.TLB:TLB_shootdowns
-26.4% 79689 ± 20% interrupts.CPU58.RES:Rescheduling_interrupts
-18.7% 78664 ± 16% interrupts.CPU6.RES:Rescheduling_interrupts
-21.7% 81825 ± 18% interrupts.CPU60.RES:Rescheduling_interrupts
-93.7% 9.50 ± 28% interrupts.CPU60.TLB:TLB_shootdowns
-21.9% 81091 ± 19% interrupts.CPU61.RES:Rescheduling_interrupts
-18.8% 81035 ± 19% interrupts.CPU62.RES:Rescheduling_interrupts
-18.8% 80732 ± 16% interrupts.CPU63.RES:Rescheduling_interrupts
-92.8% 9.75 ± 38% interrupts.CPU64.TLB:TLB_shootdowns
-21.2% 74427 ± 22% interrupts.CPU66.RES:Rescheduling_interrupts
-89.1% 10.50 ± 86% interrupts.CPU66.TLB:TLB_shootdowns
-21.2% 73562 ± 23% interrupts.CPU68.RES:Rescheduling_interrupts
-94.2% 8.50 ± 55% interrupts.CPU69.TLB:TLB_shootdowns
-21.5% 79428 ± 17% interrupts.CPU7.RES:Rescheduling_interrupts
-76.9% 10.00 ± 86% interrupts.CPU71.TLB:TLB_shootdowns
-90.6% 8.75 ± 45% interrupts.CPU73.TLB:TLB_shootdowns
-86.4% 10.00 ± 57% interrupts.CPU74.TLB:TLB_shootdowns
-18.1% 64461 ± 26% interrupts.CPU75.RES:Rescheduling_interrupts
-23.2% 63213 ± 19% interrupts.CPU76.RES:Rescheduling_interrupts
-20.5% 64217 ± 21% interrupts.CPU77.RES:Rescheduling_interrupts
-53.9% 77.00 ±158% interrupts.CPU77.TLB:TLB_shootdowns
-55.5% 59127 ± 32% interrupts.CPU78.RES:Rescheduling_interrupts
-52.1% 62630 ± 28% interrupts.CPU79.RES:Rescheduling_interrupts
+106.7% 4402 ± 20% interrupts.CPU8.NMI:Non-maskable_interrupts
+106.7% 4402 ± 20% interrupts.CPU8.PMI:Performance_monitoring_interrupts
-20.1% 81537 ± 15% interrupts.CPU8.RES:Rescheduling_interrupts
-54.5% 59229 ± 29% interrupts.CPU80.RES:Rescheduling_interrupts
-52.9% 61349 ± 29% interrupts.CPU81.RES:Rescheduling_interrupts
-52.8% 59330 ± 35% interrupts.CPU82.RES:Rescheduling_interrupts
-53.4% 58871 ± 30% interrupts.CPU83.RES:Rescheduling_interrupts
-52.3% 58375 ± 32% interrupts.CPU84.RES:Rescheduling_interrupts
-51.6% 59756 ± 29% interrupts.CPU85.RES:Rescheduling_interrupts
-50.6% 60556 ± 30% interrupts.CPU86.RES:Rescheduling_interrupts
-49.8% 59963 ± 29% interrupts.CPU87.RES:Rescheduling_interrupts
-51.1% 58631 ± 33% interrupts.CPU88.RES:Rescheduling_interrupts
-50.0% 58574 ± 32% interrupts.CPU89.RES:Rescheduling_interrupts
-25.1% 78456 ± 20% interrupts.CPU9.RES:Rescheduling_interrupts
-49.1% 57667 ± 31% interrupts.CPU90.RES:Rescheduling_interrupts
-49.1% 59777 ± 31% interrupts.CPU91.RES:Rescheduling_interrupts
-48.6% 58311 ± 35% interrupts.CPU92.RES:Rescheduling_interrupts
-46.3% 59729 ± 29% interrupts.CPU93.RES:Rescheduling_interrupts
-45.0% 60027 ± 29% interrupts.CPU94.RES:Rescheduling_interrupts
-46.4% 58724 ± 29% interrupts.CPU95.RES:Rescheduling_interrupts
-45.2% 59031 ± 29% interrupts.CPU96.RES:Rescheduling_interrupts
-69.8% 900.75 ±117% interrupts.CPU97.NMI:Non-maskable_interrupts
-69.8% 900.75 ±117% interrupts.CPU97.PMI:Performance_monitoring_interrupts
-47.6% 56109 ± 29% interrupts.CPU97.RES:Rescheduling_interrupts
-86.5% 13.00 ± 12% interrupts.CPU97.TLB:TLB_shootdowns
-46.3% 57788 ± 31% interrupts.CPU98.RES:Rescheduling_interrupts
-46.5% 55991 ± 33% interrupts.CPU99.RES:Rescheduling_interrupts
-84.2% 14.75 ± 14% interrupts.CPU99.TLB:TLB_shootdowns
-34.2% 7378335 ± 9% interrupts.RES:Rescheduling_interrupts
-56.4% 8648 ± 38% interrupts.TLB:TLB_shootdowns
-8.5 21.77 ± 28% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
-8.5 22.53 ± 27% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
-6.9 27.65 ± 18% perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
-6.3 30.10 ± 15% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
-6.1 30.70 ± 14% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
-6.1 30.77 ± 14% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
-6.1 30.79 ± 14% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
-5.9 33.83 ± 14% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
-4.6 36.05 ± 8% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
-4.6 36.15 ± 8% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
-4.4 34.52 ± 8% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
-4.4 34.68 ± 8% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
-4.4 34.93 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
-4.3 35.27 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
-3.9 36.76 ± 7% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
-3.9 36.87 ± 7% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
-1.8 44.95 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
-1.8 45.00 ± 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
-1.8 45.02 ± 2% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
-1.5 45.81 ± 2% perf-profile.calltrace.cycles-pp.secondary_startup_64
-1.3 4.68 ± 9% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
-1.3 4.73 ± 9% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
-1.2 4.99 ± 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
-1.2 4.98 ± 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
-1.1 5.26 ± 9% perf-profile.calltrace.cycles-pp.write
+0.1 0.63 ± 14% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read
+0.2 0.74 ± 23% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
+0.2 0.91 ± 19% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
+0.2 0.93 ± 21% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
+0.2 0.79 ± 24% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule
+0.3 1.43 ± 11% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle
+0.3 0.95 ± 22% perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
+0.3 1.12 ± 21% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
+0.4 1.94 ± 10% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
+0.4 1.43 ± 21% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
+0.5 1.73 ± 22% perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
+0.6 0.85 ± 46% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
+0.7 2.28 ± 21% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
+0.8 1.83 ± 9% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
+0.8 1.87 ± 9% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
+0.8 2.08 ± 38% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.pipe_read
+0.9 2.16 ± 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
+0.9 2.18 ± 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
+1.1 2.56 ± 9% perf-profile.calltrace.cycles-pp.read
+1.4 2.90 ± 25% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.pipe_read.new_sync_read
+1.5 4.22 ± 28% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
+1.5 4.30 ± 27% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
+1.6 1.55 ± 85% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
+1.6 4.84 ± 21% perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
+1.8 4.12 ± 23% perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
+2.1 5.15 ± 26% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.pipe_read.new_sync_read.vfs_read
+2.1 5.27 ± 26% perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
+2.2 7.56 ± 25% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
+2.3 7.76 ± 25% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
+2.7 8.48 ± 21% perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
+2.8 8.67 ± 22% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
-8.3 23.59 ± 25% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
-6.9 27.66 ± 18% perf-profile.children.cycles-pp.__account_scheduler_latency
-6.9 23.60 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
-6.3 30.26 ± 15% perf-profile.children.cycles-pp.enqueue_entity
-6.1 30.71 ± 14% perf-profile.children.cycles-pp.enqueue_task_fair
-6.1 30.77 ± 14% perf-profile.children.cycles-pp.activate_task
-6.1 30.80 ± 14% perf-profile.children.cycles-pp.ttwu_do_activate
-6.0 33.95 ± 13% perf-profile.children.cycles-pp.intel_idle
-4.4 34.52 ± 8% perf-profile.children.cycles-pp.try_to_wake_up
-4.4 34.69 ± 8% perf-profile.children.cycles-pp.autoremove_wake_function
-4.4 36.79 ± 8% perf-profile.children.cycles-pp.cpuidle_enter_state
-4.4 34.93 ± 8% perf-profile.children.cycles-pp.__wake_up_common
-4.4 36.81 ± 8% perf-profile.children.cycles-pp.cpuidle_enter
-4.3 35.28 ± 8% perf-profile.children.cycles-pp.__wake_up_common_lock
-3.9 36.77 ± 7% perf-profile.children.cycles-pp.pipe_write
-3.9 36.89 ± 7% perf-profile.children.cycles-pp.new_sync_write
-3.8 37.30 ± 6% perf-profile.children.cycles-pp.vfs_write
-3.7 37.53 ± 6% perf-profile.children.cycles-pp.ksys_write
-1.8 45.02 ± 2% perf-profile.children.cycles-pp.start_secondary
-1.5 45.81 ± 2% perf-profile.children.cycles-pp.secondary_startup_64
-1.5 45.81 ± 2% perf-profile.children.cycles-pp.cpu_startup_entry
-1.5 45.82 ± 2% perf-profile.children.cycles-pp.do_idle
-1.1 5.29 ± 9% perf-profile.children.cycles-pp.write
-0.1 0.29 ± 12% perf-profile.children.cycles-pp.tick_sched_handle
-0.0 0.29 ± 11% perf-profile.children.cycles-pp.update_process_times
-0.0 0.20 ± 14% perf-profile.children.cycles-pp.scheduler_tick
+0.0 0.08 ± 16% perf-profile.children.cycles-pp.generic_pipe_buf_confirm
+0.0 0.26 ± 4% perf-profile.children.cycles-pp.nr_iowait_cpu
+0.0 0.16 ± 7% perf-profile.children.cycles-pp.cpus_share_cache
+0.0 0.09 ± 26% perf-profile.children.cycles-pp.atime_needs_update
+0.0 0.30 ± 9% perf-profile.children.cycles-pp.rcu_idle_exit
+0.0 0.12 ± 26% perf-profile.children.cycles-pp.copyin
+0.0 0.13 ± 25% perf-profile.children.cycles-pp.stack_access_ok
+0.0 0.13 ± 20% perf-profile.children.cycles-pp.__calc_delta
+0.0 0.08 ± 20% perf-profile.children.cycles-pp.is_bpf_text_address
+0.0 0.27 ± 6% perf-profile.children.cycles-pp.update_ts_time_stats
+0.0 0.16 ± 19% perf-profile.children.cycles-pp.hrtimer_get_next_event
+0.0 0.13 ± 25% perf-profile.children.cycles-pp.touch_atime
+0.0 0.10 ± 31% perf-profile.children.cycles-pp.__might_fault
+0.0 0.07 ± 6% perf-profile.children.cycles-pp.rcu_core
+0.0 0.17 ± 13% perf-profile.children.cycles-pp._raw_spin_lock_irq
+0.0 0.26 ± 10% perf-profile.children.cycles-pp.newidle_balance
+0.0 0.16 ± 23% perf-profile.children.cycles-pp.in_sched_functions
+0.0 0.13 ± 26% perf-profile.children.cycles-pp.__fsnotify_parent
+0.0 0.06 ± 11% perf-profile.children.cycles-pp.lapic_next_deadline
+0.1 0.06 ± 26% perf-profile.children.cycles-pp.__x64_sys_write
+0.1 0.09 ± 40% perf-profile.children.cycles-pp.set_next_task_idle
+0.1 0.12 ± 39% perf-profile.children.cycles-pp.pick_next_task_idle
+0.1 0.14 ± 26% perf-profile.children.cycles-pp.menu_reflect
+0.1 0.11 ± 25% perf-profile.children.cycles-pp.find_next_bit
+0.1 0.07 ± 19% perf-profile.children.cycles-pp.sched_ttwu_pending
+0.1 0.06 ± 15% perf-profile.children.cycles-pp.bpf_prog_kallsyms_find
+0.1 0.06 ± 15% perf-profile.children.cycles-pp.get_cpu_device
+0.1 0.35 ± 8% perf-profile.children.cycles-pp.copyout
+0.1 0.06 ± 28% perf-profile.children.cycles-pp.in_lock_functions
+0.1 0.15 ± 26% perf-profile.children.cycles-pp.current_time
+0.1 0.29 ± 10% perf-profile.children.cycles-pp.mutex_unlock
+0.1 0.18 ± 25% perf-profile.children.cycles-pp.fsnotify
+0.1 0.15 ± 30% perf-profile.children.cycles-pp.file_update_time
+0.1 0.26 ± 3% perf-profile.children.cycles-pp.__list_del_entry_valid
+0.1 0.13 ± 38% perf-profile.children.cycles-pp.__x86_indirect_thunk_rbx
+0.1 0.11 ± 24% perf-profile.children.cycles-pp.put_prev_entity
+0.1 0.06 ± 6% perf-profile.children.cycles-pp.tick_nohz_irq_exit
+0.1 0.06 ± 17% perf-profile.children.cycles-pp.tick_nohz_tick_stopped
+0.1 0.21 ± 21% perf-profile.children.cycles-pp.resched_curr
+0.1 0.07 ± 31% perf-profile.children.cycles-pp.timestamp_truncate
+0.1 0.07 ± 30% perf-profile.children.cycles-pp.apparmor_file_permission
+0.1 0.18 ± 21% perf-profile.children.cycles-pp.rcu_eqs_enter
+0.1 0.39 ± 11% perf-profile.children.cycles-pp.copy_user_generic_unrolled
+0.1 0.19 ± 23% perf-profile.children.cycles-pp.account_entity_dequeue
+0.1 0.07 ± 43% perf-profile.children.cycles-pp.__update_idle_core
+0.1 0.35 ± 6% perf-profile.children.cycles-pp.irq_exit
+0.1 0.25 ± 24% perf-profile.children.cycles-pp.__fget_light
+0.1 0.31 ± 20% perf-profile.children.cycles-pp.reweight_entity
+0.1 0.27 ± 13% perf-profile.children.cycles-pp.pick_next_entity
+0.1 0.30 ± 16% perf-profile.children.cycles-pp.check_preempt_curr
+0.1 0.35 ± 14% perf-profile.children.cycles-pp.native_write_msr
+0.1 0.19 ± 21% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
+0.1 0.20 ± 25% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
+0.1 0.46 ± 9% perf-profile.children.cycles-pp.___might_sleep
+0.1 0.28 ± 22% perf-profile.children.cycles-pp.__fdget_pos
+0.1 0.22 ± 15% perf-profile.children.cycles-pp.clockevents_program_event
+0.1 0.23 ± 25% perf-profile.children.cycles-pp.__might_sleep
+0.1 0.63 ± 11% perf-profile.children.cycles-pp.update_rq_clock
+0.1 0.29 ± 17% perf-profile.children.cycles-pp.read_tsc
+0.1 0.36 ± 17% perf-profile.children.cycles-pp.ttwu_do_wakeup
+0.1 0.36 ± 22% perf-profile.children.cycles-pp.common_file_perm
+0.1 0.28 ± 16% perf-profile.children.cycles-pp.tick_nohz_idle_enter
+0.1 0.19 ± 28% perf-profile.children.cycles-pp.put_prev_task_fair
+0.1 0.26 ± 32% perf-profile.children.cycles-pp.switch_fpu_return
+0.1 0.28 ± 30% perf-profile.children.cycles-pp.___perf_sw_event
+0.1 0.38 ± 20% perf-profile.children.cycles-pp.native_sched_clock
+0.1 0.32 ± 31% perf-profile.children.cycles-pp.orc_find
+0.1 0.47 ± 16% perf-profile.children.cycles-pp.tick_nohz_idle_exit
+0.1 0.43 ± 21% perf-profile.children.cycles-pp.__orc_find
+0.1 0.40 ± 20% perf-profile.children.cycles-pp.sched_clock
+0.1 0.38 ± 27% perf-profile.children.cycles-pp.finish_task_switch
+0.1 0.37 ± 22% perf-profile.children.cycles-pp.kernel_text_address
+0.1 0.42 ± 22% perf-profile.children.cycles-pp.__update_load_avg_se
+0.1 0.89 ± 6% perf-profile.children.cycles-pp.mutex_lock
+0.1 0.38 ± 26% perf-profile.children.cycles-pp.copy_page_from_iter
+0.2 0.71 ± 11% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
+0.2 0.43 ± 21% perf-profile.children.cycles-pp.sched_clock_cpu
+0.2 0.60 ± 13% perf-profile.children.cycles-pp.copy_page_to_iter
+0.2 0.79 ± 13% perf-profile.children.cycles-pp.prepare_to_wait_event
+0.2 0.69 ± 19% perf-profile.children.cycles-pp.__switch_to
+0.2 0.55 ± 18% perf-profile.children.cycles-pp.__switch_to_asm
+0.2 0.52 ± 23% perf-profile.children.cycles-pp.security_file_permission
+0.2 0.41 ± 32% perf-profile.children.cycles-pp._find_next_bit
+0.2 1.33 ± 5% perf-profile.children.cycles-pp.apic_timer_interrupt
+0.2 0.53 ± 28% perf-profile.children.cycles-pp.update_cfs_group
+0.2 0.45 ± 24% perf-profile.children.cycles-pp.__kernel_text_address
+0.2 0.65 ± 23% perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
+0.2 0.38 ± 51% perf-profile.children.cycles-pp.available_idle_cpu
+0.2 0.55 ± 26% perf-profile.children.cycles-pp.unwind_get_return_address
+0.2 0.67 ± 24% perf-profile.children.cycles-pp.__next_timer_interrupt
+0.3 0.61 ± 13% perf-profile.children.cycles-pp.ktime_get
+0.3 0.88 ± 16% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
+0.3 0.69 ± 28% perf-profile.children.cycles-pp.__unwind_start
+0.3 1.17 ± 19% perf-profile.children.cycles-pp.entry_SYSCALL_64
+0.3 0.97 ± 22% perf-profile.children.cycles-pp.get_next_timer_interrupt
+0.3 1.50 ± 12% perf-profile.children.cycles-pp.set_next_entity
+0.3 1.28 ± 17% perf-profile.children.cycles-pp.syscall_return_via_sysret
+0.4 1.15 ± 21% perf-profile.children.cycles-pp.tick_nohz_next_event
+0.4 1.45 ± 21% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
+0.4 1.17 ± 24% perf-profile.children.cycles-pp.update_curr
+0.5 1.13 ± 37% perf-profile.children.cycles-pp.select_task_rq_fair
+0.6 2.33 ± 13% perf-profile.children.cycles-pp.pick_next_task_fair
+0.7 2.33 ± 21% perf-profile.children.cycles-pp.menu_select
+0.8 2.57 ± 22% perf-profile.children.cycles-pp.unwind_next_frame
+0.9 2.44 ± 24% perf-profile.children.cycles-pp.update_load_avg
+0.9 2.46 ± 25% perf-profile.children.cycles-pp.dequeue_entity
+1.1 2.59 ± 9% perf-profile.children.cycles-pp.read
+1.1 2.91 ± 25% perf-profile.children.cycles-pp.dequeue_task_fair
+1.4 4.17 ± 23% perf-profile.children.cycles-pp.arch_stack_walk
+1.4 1.59 ± 85% perf-profile.children.cycles-pp.poll_idle
+1.5 4.84 ± 21% perf-profile.children.cycles-pp.stack_trace_save_tsk
+1.6 4.37 ± 28% perf-profile.children.cycles-pp.schedule_idle
+1.8 2.33 ± 71% perf-profile.children.cycles-pp._raw_spin_lock
+2.0 5.28 ± 26% perf-profile.children.cycles-pp.schedule
+2.7 8.51 ± 21% perf-profile.children.cycles-pp.pipe_read
+2.8 8.68 ± 22% perf-profile.children.cycles-pp.new_sync_read
+3.0 9.40 ± 21% perf-profile.children.cycles-pp.vfs_read
+3.1 9.63 ± 21% perf-profile.children.cycles-pp.ksys_read
+3.5 9.51 ± 27% perf-profile.children.cycles-pp.__sched_text_start
-6.9 23.59 ± 19% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
-6.0 33.94 ± 13% perf-profile.self.cycles-pp.intel_idle
+0.0 0.07 ± 13% perf-profile.self.cycles-pp.tick_nohz_next_event
+0.0 0.10 ± 8% perf-profile.self.cycles-pp.check_preempt_curr
+0.0 0.26 ± 4% perf-profile.self.cycles-pp.nr_iowait_cpu
+0.0 0.11 ± 19% perf-profile.self.cycles-pp.in_sched_functions
+0.0 0.09 ± 16% perf-profile.self.cycles-pp.schedule
+0.0 0.08 ± 20% perf-profile.self.cycles-pp.generic_pipe_buf_confirm
+0.0 0.08 ± 30% perf-profile.self.cycles-pp.__kernel_text_address
+0.0 0.08 ± 27% perf-profile.self.cycles-pp.put_task_stack
+0.0 0.16 ± 7% perf-profile.self.cycles-pp.cpus_share_cache
+0.0 0.11 ± 23% perf-profile.self.cycles-pp.stack_access_ok
+0.0 0.11 ± 17% perf-profile.self.cycles-pp.ksys_read
+0.0 0.09 ± 23% perf-profile.self.cycles-pp.kernel_text_address
+0.0 0.10 ± 21% perf-profile.self.cycles-pp.menu_reflect
+0.0 0.08 ± 19% perf-profile.self.cycles-pp.copy_page_from_iter
+0.0 0.24 ± 9% perf-profile.self.cycles-pp.newidle_balance
+0.0 0.16 ± 14% perf-profile.self.cycles-pp._raw_spin_lock_irq
+0.0 0.10 ± 39% perf-profile.self.cycles-pp.unwind_get_return_address
+0.0 0.15 ± 27% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
+0.0 0.13 ± 24% perf-profile.self.cycles-pp.__fsnotify_parent
+0.0 0.06 ± 20% perf-profile.self.cycles-pp.security_file_permission
+0.1 0.10 ± 35% perf-profile.self.cycles-pp.__x86_indirect_thunk_rbx
+0.1 0.24 ± 15% perf-profile.self.cycles-pp.prepare_to_wait_event
+0.1 0.09 ± 27% perf-profile.self.cycles-pp.put_prev_entity
+0.1 0.14 ± 19% perf-profile.self.cycles-pp.copy_page_to_iter
+0.1 0.11 ± 40% perf-profile.self.cycles-pp.pick_next_task_fair
+0.1 0.07 ± 19% perf-profile.self.cycles-pp.sched_ttwu_pending
+0.1 0.06 ± 15% perf-profile.self.cycles-pp.bpf_prog_kallsyms_find
+0.1 0.06 ± 15% perf-profile.self.cycles-pp.get_cpu_device
+0.1 0.29 ± 14% perf-profile.self.cycles-pp.reweight_entity
+0.1 0.25 ± 3% perf-profile.self.cycles-pp.__list_del_entry_valid
+0.1 0.24 ± 9% perf-profile.self.cycles-pp.pick_next_entity
+0.1 0.06 ± 22% perf-profile.self.cycles-pp.activate_task
+0.1 0.06 ± 14% perf-profile.self.cycles-pp.__is_insn_slot_addr
+0.1 0.14 ± 32% perf-profile.self.cycles-pp.arch_stack_walk
+0.1 0.06 ± 28% perf-profile.self.cycles-pp.__x64_sys_write
+0.1 0.29 ± 10% perf-profile.self.cycles-pp.mutex_unlock
+0.1 0.18 ± 25% perf-profile.self.cycles-pp.fsnotify
+0.1 0.36 ± 16% perf-profile.self.cycles-pp.try_to_wake_up
+0.1 0.17 ± 23% perf-profile.self.cycles-pp.rcu_eqs_enter
+0.1 0.06 ± 17% perf-profile.self.cycles-pp.cpuidle_governor_latency_req
+0.1 0.24 ± 24% perf-profile.self.cycles-pp.__unwind_start
+0.1 0.21 ± 21% perf-profile.self.cycles-pp.resched_curr
+0.1 0.15 ± 33% perf-profile.self.cycles-pp.new_sync_read
+0.1 0.07 ± 31% perf-profile.self.cycles-pp.timestamp_truncate
+0.1 0.24 ± 17% perf-profile.self.cycles-pp.__account_scheduler_latency
+0.1 0.07 ± 12% perf-profile.self.cycles-pp.ksys_write
+0.1 0.18 ± 24% perf-profile.self.cycles-pp.account_entity_dequeue
+0.1 0.38 ± 11% perf-profile.self.cycles-pp.copy_user_generic_unrolled
+0.1 0.07 ± 39% perf-profile.self.cycles-pp.__update_idle_core
+0.1 0.46 ± 8% perf-profile.self.cycles-pp.mutex_lock
+0.1 0.24 ± 23% perf-profile.self.cycles-pp.__fget_light
+0.1 0.18 ± 24% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
+0.1 0.20 ± 25% perf-profile.self.cycles-pp.__might_sleep
+0.1 0.16 ± 28% perf-profile.self.cycles-pp.dequeue_entity
+0.1 0.08 ± 34% perf-profile.self.cycles-pp.put_prev_task_fair
+0.1 0.46 ± 9% perf-profile.self.cycles-pp.___might_sleep
+0.1 0.24 ± 17% perf-profile.self.cycles-pp.vfs_read
+0.1 0.35 ± 14% perf-profile.self.cycles-pp.native_write_msr
+0.1 0.31 ± 21% perf-profile.self.cycles-pp.common_file_perm
+0.1 0.28 ± 24% perf-profile.self.cycles-pp.finish_task_switch
+0.1 0.27 ± 16% perf-profile.self.cycles-pp.read_tsc
+0.1 0.26 ± 22% perf-profile.self.cycles-pp.__next_timer_interrupt
+0.1 0.25 ± 31% perf-profile.self.cycles-pp.switch_fpu_return
+0.1 0.53 ± 17% perf-profile.self.cycles-pp.do_idle
+0.1 0.24 ± 31% perf-profile.self.cycles-pp.___perf_sw_event
+0.1 0.49 ± 16% perf-profile.self.cycles-pp.pipe_read
+0.1 0.39 ± 19% perf-profile.self.cycles-pp.enqueue_task_fair
+0.1 0.31 ± 29% perf-profile.self.cycles-pp.orc_find
+0.1 0.37 ± 19% perf-profile.self.cycles-pp.native_sched_clock
+0.1 0.41 ± 21% perf-profile.self.cycles-pp.__orc_find
+0.1 0.44 ± 24% perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched
+0.1 0.34 ± 22% perf-profile.self.cycles-pp.dequeue_task_fair
+0.1 0.52 ± 17% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
+0.1 0.41 ± 22% perf-profile.self.cycles-pp.__update_load_avg_se
+0.2 0.64 ± 18% perf-profile.self.cycles-pp.__switch_to
+0.2 0.91 ± 11% perf-profile.self.cycles-pp.enqueue_entity
+0.2 0.55 ± 19% perf-profile.self.cycles-pp.__switch_to_asm
+0.2 0.40 ± 32% perf-profile.self.cycles-pp._find_next_bit
+0.2 0.34 ± 13% perf-profile.self.cycles-pp.ktime_get
+0.2 0.55 ± 20% perf-profile.self.cycles-pp._raw_spin_lock
+0.2 0.62 ± 22% perf-profile.self.cycles-pp.menu_select
+0.2 0.46 ± 37% perf-profile.self.cycles-pp.update_cfs_group
+0.2 0.37 ± 51% perf-profile.self.cycles-pp.available_idle_cpu
+0.2 1.76 ± 9% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
+0.2 0.53 ± 28% perf-profile.self.cycles-pp.select_task_rq_fair
+0.2 1.32 ± 15% perf-profile.self.cycles-pp.__sched_text_start
+0.3 0.86 ± 17% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
+0.3 1.14 ± 19% perf-profile.self.cycles-pp.entry_SYSCALL_64
+0.3 1.27 ± 17% perf-profile.self.cycles-pp.syscall_return_via_sysret
+0.3 0.90 ± 24% perf-profile.self.cycles-pp.update_curr
+0.4 1.07 ± 29% perf-profile.self.cycles-pp.update_load_avg
+0.5 1.68 ± 21% perf-profile.self.cycles-pp.unwind_next_frame
+0.8 2.35 ± 22% perf-profile.self.cycles-pp.do_syscall_64
+1.4 1.50 ± 89% perf-profile.self.cycles-pp.poll_idle
**********************************************************************************
8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory
========================================================================
governor/kconfig/nr_task/rootfs/runtime/target/tbox_group/testcase/ucode:
ce/x86_64-rhel-7.6/50%/debian-x86_64-20191114.cgz/300s/vmlinux_prereq/lkp-hsw-d01/kbuild/0x27
("threads: Update PID limit comment according to futex UAPI change")
("sched/fair: Don't pull task if local group is more loaded than busiest group")
59901cb4520c44bfce81f523bc6
---------------------------
%reproduction fail:runs
| |
-25% :4 dmesg.BUG:Bad_page_map_in_process
-25% :4 dmesg.BUG:Bad_rss-counter_state_mm:#type:MM_ANONPAGES_val
-25% :4 dmesg.BUG:Bad_rss-counter_state_mm:#type:MM_SWAPENTS_val
25% 1:4 dmesg.Kernel_panic-not_syncing:Fatal_exception
25% 1:4 dmesg.RIP:__d_lookup_rcu
25% 1:4 dmesg.canonical_address#:#[##]
-25% :4 kmsg.file:libuuid.so.#fault:filemap_fault_mmap:generic_file_mmap_readpage:simple_readpage
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a07fffffa0ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a17fffffa1ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a27fffffa2ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a37fffffa3ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a47fffffa4ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a57fffffa5ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a67fffffa6ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a77fffffa7ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a7fffff8aff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a7fffff9aff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a87fffffa8ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a97fffffa9ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#aa7fffffaaff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ab7fffffabff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ac7fffffacff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ad7fffffadff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ae7fffffaeff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#af7fffffafff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b07fffffb0ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b17fffffb1ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b27fffffb2ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b37fffffb3ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b47fffffb4ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b57fffffb5ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b67fffffb6ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b77fffffb7ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b7fffff8bff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b7fffff9bff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b87fffffb8ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b97fffffb9ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ba7fffffbaff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#bb7fffffbbff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#bc7fffffbcff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#bd7fffffbdff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#be7fffffbeff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#bf7fffffbfff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c07fffffc0ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c17fffffc1ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c27fffffc2ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c37fffffc3ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c47fffffc4ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c57fffffc5ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c67fffffc6ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c77fffffc7ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c7fffff8cff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c7fffff9cff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c87fffffc8ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c97fffffc9ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ca7fffffcaff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#cb7fffffcbff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#cc7fffffccff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#cd7fffffcdff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ce7fffffceff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#cf7fffffcfff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d07fffffd0ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d17fffffd1ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d27fffffd2ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d37fffffd3ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d47fffffd4ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d57fffffd5ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d67fffffd6ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d77fffffd7ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d7fffff8dff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d7fffff9dff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d87fffffd8ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d97fffffd9ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#da7fffffdaff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#db7fffffdbff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#dc7fffffdcff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#dd7fffffddff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#de7fffffdeff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#df7fffffdfff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e07fffffe0ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e17fffffe1ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e27fffffe2ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e37fffffe3ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e47fffffe4ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e57fffffe5ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e67fffffe6ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e77fffffe7ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e7fffff8eff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e7fffff9eff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e87fffffe8ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e97fffffe9ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ea7fffffeaff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#eb7fffffebff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ec7fffffecff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ed7fffffedff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ee7fffffeeff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ef7fffffefff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f07ffffff0ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f17ffffff1ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f27ffffff2ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f37ffffff3ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f47ffffff4ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f57ffffff5ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f67ffffff6ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f77ffffff7ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f7fffff8fff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f7fffff9fff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f87ffffff8ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f97ffffff9ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fa7ffffffaff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fb7ffffffbff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fc7ffffffcff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fd7ffffffdff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fe7ffffffeff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ff7fffffffff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff80ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff81ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff82ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff83ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff84ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff85ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff86ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff87ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff88ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff89ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff90ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff91ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff92ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff93ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff94ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff95ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff96ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff97ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff98ff
-25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff99ff
0% 0:4 perf-profile.children.cycles-pp.warning_at
3% 0:4 perf-profile.children.cycles-pp.warning
4% 1:4 perf-profile.children.cycles-pp.error_entry
4% 0:4 perf-profile.self.cycles-pp.error_entry
%change %stddev
| \
+7.5% 186.17 kbuild.buildtime_per_iteration
+7.5% 372.13 kbuild.time.elapsed_time
+7.5% 372.13 kbuild.time.elapsed_time.max
-4.4% 59613 kbuild.time.involuntary_context_switches
+3.7% 124.42 kbuild.time.system_time
+7.8% 1357 kbuild.time.user_time
-5.2% 3768 ± 2% proc-vmstat.pgactivate
-7.6% 7925 vmstat.system.cs
+27.4% 37.06 ± 21% boot-time.boot
+31.2% 249.80 ± 20% boot-time.idle
+85.2% 4.58e+08 ± 21% cpuidle.C6.time
+70.5% 616268 ± 5% cpuidle.C6.usage
-16.4% 1895 ± 10% slabinfo.cred_jar.active_objs
-15.3% 1950 ± 9% slabinfo.cred_jar.num_objs
+14.3% 907.00 ± 4% slabinfo.task_delay_info.active_objs
+14.3% 907.00 ± 4% slabinfo.task_delay_info.num_objs
+10.7% 276098 ± 4% softirqs.CPU0.RCU
+30.3% 180101 ± 12% softirqs.CPU0.TIMER
+9.2% 152539 ± 2% softirqs.CPU2.TIMER
+11.1% 153534 ± 2% softirqs.CPU4.TIMER
+11.9% 155966 softirqs.CPU7.TIMER
+10.4% 1273097 softirqs.TIMER
-37.9% 166.33 ± 9% interrupts.CPU0.CAL:Function_call_interrupts
-17.8% 3040 ± 9% interrupts.CPU0.RES:Rescheduling_interrupts
-48.6% 126.00 ± 8% interrupts.CPU0.TLB:TLB_shootdowns
-39.2% 114.67 ± 22% interrupts.CPU1.TLB:TLB_shootdowns
-13.1% 254.00 ± 12% interrupts.CPU2.CAL:Function_call_interrupts
-31.6% 165.33 ± 7% interrupts.CPU2.TLB:TLB_shootdowns
-37.5% 4497 ± 29% interrupts.CPU4.NMI:Non-maskable_interrupts
-37.5% 4497 ± 29% interrupts.CPU4.PMI:Performance_monitoring_interrupts
-16.8% 1132 ± 10% interrupts.TLB:TLB_shootdowns
+1.4e+11% 1382 ±114% sched_debug.cfs_rq:/.MIN_vruntime.avg
+1.1e+12% 11058 ±114% sched_debug.cfs_rq:/.MIN_vruntime.max
+20.1% 93480 sched_debug.cfs_rq:/.exec_clock.avg
+20.3% 99210 sched_debug.cfs_rq:/.exec_clock.max
+18.4% 86426 sched_debug.cfs_rq:/.exec_clock.min
+33.9% 3858 ± 9% sched_debug.cfs_rq:/.exec_clock.stddev
+20.2% 143687 ± 8% sched_debug.cfs_rq:/.load.avg
+1.4e+11% 1382 ±114% sched_debug.cfs_rq:/.max_vruntime.avg
+1.1e+12% 11058 ±114% sched_debug.cfs_rq:/.max_vruntime.max
+18.8% 549986 sched_debug.cfs_rq:/.min_vruntime.avg
+18.7% 581626 sched_debug.cfs_rq:/.min_vruntime.max
+17.3% 510306 sched_debug.cfs_rq:/.min_vruntime.min
+24.4% 21446 ± 6% sched_debug.cfs_rq:/.min_vruntime.stddev
+18.1% 0.80 ± 3% sched_debug.cfs_rq:/.nr_running.avg
-18.2% 0.36 ± 5% sched_debug.cfs_rq:/.nr_running.stddev
+24.4% 21446 ± 6% sched_debug.cfs_rq:/.spread0.stddev
+21.0% 218362 ± 3% sched_debug.cpu.clock.avg
+21.0% 218363 ± 3% sched_debug.cpu.clock.max
+21.0% 218361 ± 3% sched_debug.cpu.clock.min
+21.0% 218362 ± 3% sched_debug.cpu.clock_task.avg
+21.0% 218363 ± 3% sched_debug.cpu.clock_task.max
+21.0% 218361 ± 3% sched_debug.cpu.clock_task.min
+37.6% 12486 sched_debug.cpu.curr->pid.avg
+37.2% 19088 sched_debug.cpu.curr->pid.max
+37.6% 8831 sched_debug.cpu.curr->pid.stddev
-24.7% 1.38 ± 12% sched_debug.cpu.nr_running.max
-30.6% 0.46 ± 4% sched_debug.cpu.nr_running.stddev
-53.9% 10.29 ± 11% sched_debug.cpu.nr_uninterruptible.max
-35.3% 7.10 ± 7% sched_debug.cpu.nr_uninterruptible.stddev
+21.0% 218362 ± 3% sched_debug.cpu_clk
+21.0% 217836 ± 3% sched_debug.ktime
+20.8% 219109 ± 3% sched_debug.sched_clk
+6.4% 14.47 perf-stat.i.MPKI
-7.5% 3.478e+09 perf-stat.i.branch-instructions
+0.1 3.08 perf-stat.i.branch-miss-rate%
-5.5% 1.064e+08 perf-stat.i.branch-misses
-0.6 8.17 ± 2% perf-stat.i.cache-miss-rate%
-10.3% 18373371 perf-stat.i.cache-misses
-2.6% 2.303e+08 perf-stat.i.cache-references
-7.7% 7977 perf-stat.i.context-switches
+9.1% 1.00 ± 2% perf-stat.i.cpi
-1.2% 1.562e+10 perf-stat.i.cpu-cycles
-11.9% 160.76 perf-stat.i.cpu-migrations
+9.7% 877.23 perf-stat.i.cycles-between-cache-misses
+0.0 0.33 perf-stat.i.dTLB-load-miss-rate%
-4.5% 14149852 perf-stat.i.dTLB-load-misses
-7.5% 4.341e+09 perf-stat.i.dTLB-loads
-7.0% 2.373e+09 perf-stat.i.dTLB-stores
+8.3% 4998953 ± 2% perf-stat.i.iTLB-load-misses
+7.6% 19060873 perf-stat.i.iTLB-loads
-7.5% 1.601e+10 perf-stat.i.instructions
-13.1% 3339 ± 2% perf-stat.i.instructions-per-iTLB-miss
-6.6% 1.02 perf-stat.i.ipc
-7.3% 152206 perf-stat.i.minor-faults
-11.3% 11934691 perf-stat.i.node-loads
-11.5% 1659475 ± 3% perf-stat.i.node-stores
-7.3% 152206 perf-stat.i.page-faults
+5.3% 14.38 perf-stat.overall.MPKI
+0.1 3.06 perf-stat.overall.branch-miss-rate%
-0.7 7.98 perf-stat.overall.cache-miss-rate%
+6.8% 0.98 perf-stat.overall.cpi
+10.2% 850.28 perf-stat.overall.cycles-between-cache-misses
+0.0 0.32 perf-stat.overall.dTLB-load-miss-rate%
-14.5% 3205 ± 3% perf-stat.overall.instructions-per-iTLB-miss
-6.4% 1.02 perf-stat.overall.ipc
-7.5% 3.469e+09 perf-stat.ps.branch-instructions
-5.5% 1.062e+08 perf-stat.ps.branch-misses
-10.3% 18323590 perf-stat.ps.cache-misses
-2.6% 2.297e+08 perf-stat.ps.cache-references
-7.6% 7955 perf-stat.ps.context-switches
-1.2% 1.558e+10 perf-stat.ps.cpu-cycles
-11.9% 160.32 perf-stat.ps.cpu-migrations
-4.4% 14111258 perf-stat.ps.dTLB-load-misses
-7.5% 4.33e+09 perf-stat.ps.dTLB-loads
-6.9% 2.366e+09 perf-stat.ps.dTLB-stores
+8.3% 4985424 ± 2% perf-stat.ps.iTLB-load-misses
+7.6% 19009302 perf-stat.ps.iTLB-loads
-7.5% 1.597e+10 perf-stat.ps.instructions
-7.3% 151790 perf-stat.ps.minor-faults
-11.3% 11902353 perf-stat.ps.node-loads
-11.5% 1654965 ± 3% perf-stat.ps.node-stores
-7.3% 151790 perf-stat.ps.page-faults
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.write._fini
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp._fini
-1.8 2.30 ± 21% perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write._fini
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.devkmsg_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
-1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write.ksys_write
-1.4 2.03 ± 21% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write
-1.4 1.90 ± 20% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit
-1.1 2.08 ± 29% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
-1.1 2.08 ± 29% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
-0.9 1.27 ± 18% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
-0.6 0.45 ± 72% perf-profile.calltrace.cycles-pp.delay_tsc.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
-0.3 0.95 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
-0.3 0.95 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
-0.2 0.36 ± 70% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
-0.2 0.66 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
-0.2 0.66 ± 3% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
-0.2 0.66 ± 3% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
-0.2 0.63 ± 3% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
-0.2 0.63 ± 3% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
-0.1 0.71 ± 6% perf-profile.calltrace.cycles-pp.read
-0.1 0.65 ± 7% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
-0.1 0.65 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
-0.1 0.56 ± 6% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
-0.1 0.91 ± 6% perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode
-0.1 0.58 ± 4% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
+0.1 0.67 ± 7% perf-profile.calltrace.cycles-pp.wide_int_to_tree
+0.1 0.78 ± 11% perf-profile.calltrace.cycles-pp.c_lex_with_flags
+0.2 0.81 ± 7% perf-profile.calltrace.cycles-pp.linemap_lookup
+0.2 0.54 ± 2% perf-profile.calltrace.cycles-pp.pop_scope
+0.2 2.36 ± 3% perf-profile.calltrace.cycles-pp._cpp_lex_direct
+0.3 1.44 ± 11% perf-profile.calltrace.cycles-pp.memchr
-2.4 5.75 ± 11% perf-profile.children.cycles-pp.do_syscall_64
-2.4 5.77 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
-1.8 2.36 ± 20% perf-profile.children.cycles-pp.ksys_write
-1.8 2.38 ± 20% perf-profile.children.cycles-pp.write
-1.8 2.36 ± 20% perf-profile.children.cycles-pp.vfs_write
-1.8 2.36 ± 20% perf-profile.children.cycles-pp.new_sync_write
-1.8 2.30 ± 20% perf-profile.children.cycles-pp._fini
-1.8 2.30 ± 20% perf-profile.children.cycles-pp.devkmsg_write
-1.8 2.30 ± 20% perf-profile.children.cycles-pp.devkmsg_emit
-1.5 3.13 ± 20% perf-profile.children.cycles-pp.vprintk_emit
-1.5 3.25 ± 16% perf-profile.children.cycles-pp.console_unlock
-1.2 2.94 ± 16% perf-profile.children.cycles-pp.serial8250_console_write
-1.1 2.88 ± 16% perf-profile.children.cycles-pp.wait_for_xmitr
-1.1 2.80 ± 16% perf-profile.children.cycles-pp.uart_console_write
-1.0 2.75 ± 15% perf-profile.children.cycles-pp.serial8250_console_putchar
-0.8 1.94 ± 15% perf-profile.children.cycles-pp.io_serial_in
-0.3 0.94 ± 18% perf-profile.children.cycles-pp.delay_tsc
-0.3 0.31 ± 18% perf-profile.children.cycles-pp.vt_console_print
-0.3 0.31 ± 18% perf-profile.children.cycles-pp.lf
-0.3 0.31 ± 18% perf-profile.children.cycles-pp.con_scroll
-0.3 0.31 ± 18% perf-profile.children.cycles-pp.fbcon_scroll
-0.3 0.30 ± 17% perf-profile.children.cycles-pp.fbcon_redraw
-0.3 0.29 ± 18% perf-profile.children.cycles-pp.fbcon_putcs
-0.3 0.29 ± 18% perf-profile.children.cycles-pp.bit_putcs
-0.3 0.24 ± 22% perf-profile.children.cycles-pp.drm_fb_helper_cfb_imageblit
-0.2 0.24 ± 22% perf-profile.children.cycles-pp.cfb_imageblit
-0.2 0.66 ± 3% perf-profile.children.cycles-pp.mmput
-0.2 0.66 ± 3% perf-profile.children.cycles-pp.__x64_sys_exit_group
-0.2 0.66 ± 3% perf-profile.children.cycles-pp.do_group_exit
-0.2 0.66 ± 3% perf-profile.children.cycles-pp.do_exit
-0.2 0.66 ± 3% perf-profile.children.cycles-pp.exit_mmap
-0.2 0.47 ± 3% perf-profile.children.cycles-pp.unmap_vmas
-0.1 0.47 ± 3% perf-profile.children.cycles-pp.unmap_page_range
-0.1 0.41 ± 7% perf-profile.children.cycles-pp.tlb_flush_mmu
-0.1 0.73 ± 6% perf-profile.children.cycles-pp.read
-0.1 0.82 ± 5% perf-profile.children.cycles-pp.do_sys_open
-0.1 0.61 ± 4% perf-profile.children.cycles-pp.ksys_read
-0.1 0.81 ± 5% perf-profile.children.cycles-pp.do_sys_openat2
-0.1 0.59 ± 5% perf-profile.children.cycles-pp.vfs_read
-0.1 1.16 ± 4% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
-0.1 0.41 ± 7% perf-profile.children.cycles-pp.release_pages
-0.1 0.37 ± 11% perf-profile.children.cycles-pp.bitmap_clear_bit
-0.1 0.14 ± 11% perf-profile.children.cycles-pp.execve
-0.1 0.17 ± 12% perf-profile.children.cycles-pp.__x64_sys_execve
-0.1 0.17 ± 12% perf-profile.children.cycles-pp.__do_execve_file
-0.1 0.13 ± 24% perf-profile.children.cycles-pp.statistics_fini_pass
-0.1 0.04 ± 70% perf-profile.children.cycles-pp.__libc_fork
-0.1 0.14 ± 17% perf-profile.children.cycles-pp.filemap_map_pages
-0.1 0.26 ± 9% perf-profile.children.cycles-pp.__list_del_entry_valid
-0.1 0.24 ± 5% perf-profile.children.cycles-pp.irq_exit
-0.1 0.21 ± 17% perf-profile.children.cycles-pp.bitmap_bit_p
-0.1 0.07 ± 25% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
-0.0 0.16 ± 13% perf-profile.children.cycles-pp.side_effects_p
-0.0 0.08 ± 30% perf-profile.children.cycles-pp.alloc_empty_file
-0.0 0.05 ± 72% perf-profile.children.cycles-pp.io_serial_out
-0.0 0.13 ± 9% perf-profile.children.cycles-pp.note_stores
-0.0 0.13 ± 16% perf-profile.children.cycles-pp.bitmap_copy
-0.0 0.10 ± 24% perf-profile.children.cycles-pp.kmem_cache_alloc
-0.0 0.19 ± 14% perf-profile.children.cycles-pp.rcu_core
-0.0 0.10 ± 8% perf-profile.children.cycles-pp.load_elf_binary
-0.0 0.05 ± 78% perf-profile.children.cycles-pp.copy_page
-0.0 0.20 ± 6% perf-profile.children.cycles-pp.init_alias_analysis
-0.0 0.07 ± 28% perf-profile.children.cycles-pp.find_unreachable_blocks
-0.0 0.12 ± 6% perf-profile.children.cycles-pp.syscall_return_via_sysret
-0.0 0.10 ± 4% perf-profile.children.cycles-pp.search_binary_handler
-0.0 0.07 ± 17% perf-profile.children.cycles-pp.ret_from_fork
-0.0 0.07 ± 35% perf-profile.children.cycles-pp.__alloc_file
-0.0 0.05 ± 72% perf-profile.children.cycles-pp.lra_inheritance
-0.0 0.05 ± 72% perf-profile.children.cycles-pp.page_remove_rmap
-0.0 0.15 ± 16% perf-profile.children.cycles-pp.get_attribute_name
-0.0 0.08 ± 32% perf-profile.children.cycles-pp.for_each_inc_dec
-0.0 0.11 ± 22% perf-profile.children.cycles-pp.update_stmt_operands
-0.0 0.04 ± 71% perf-profile.children.cycles-pp.wrapup_global_declaration_2
-0.0 0.15 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64
-0.0 0.08 ± 20% perf-profile.children.cycles-pp.gimple_simplify
-0.0 0.06 ± 19% perf-profile.children.cycles-pp.kthread
-0.0 0.13 ± 10% perf-profile.children.cycles-pp.free_pcppages_bulk
-0.0 0.14 ± 17% perf-profile.children.cycles-pp.walk_stmt_load_store_addr_ops
-0.0 0.08 ± 22% perf-profile.children.cycles-pp.execute_update_addresses_taken
-0.0 0.06 ± 8% perf-profile.children.cycles-pp.bitmap_and
-0.0 0.06 ± 8% perf-profile.children.cycles-pp.alloc_set_pte
-0.0 0.07 ± 6% perf-profile.children.cycles-pp.et_set_father
-0.0 0.10 ± 14% perf-profile.children.cycles-pp.hash_rtx_cb
-0.0 0.07 ± 7% perf-profile.children.cycles-pp.__do_munmap
-0.0 0.11 ± 11% perf-profile.children.cycles-pp.is_gimple_reg
-0.0 0.06 ± 7% perf-profile.children.cycles-pp.find_get_entry
-0.0 0.08 ± 6% perf-profile.children.cycles-pp.pagecache_get_page
-0.0 0.08 ± 6% perf-profile.children.cycles-pp.substitute_and_fold_dom_walker::before_dom_children
+0.0 0.11 ± 4% perf-profile.children.cycles-pp.realloc
+0.0 0.11 ± 12% perf-profile.children.cycles-pp.rtx_equal_p
+0.0 0.10 ± 12% perf-profile.children.cycles-pp.c_fully_fold
+0.0 0.08 ± 12% perf-profile.children.cycles-pp._cond_resched
+0.0 0.12 ± 8% perf-profile.children.cycles-pp.build_unary_op
+0.0 0.07 ± 7% perf-profile.children.cycles-pp.cpp_interpret_integer
+0.0 0.16 ± 5% perf-profile.children.cycles-pp.declspecs_add_type
+0.0 0.13 ± 6% perf-profile.children.cycles-pp.task_work_run
+0.0 0.12 ± 13% perf-profile.children.cycles-pp.pushdecl
+0.0 0.11 ± 12% perf-profile.children.cycles-pp.force_fit_type
+0.0 0.07 ± 12% perf-profile.children.cycles-pp.__fput
+0.0 0.13 ± 13% perf-profile.children.cycles-pp.is_typedef_decl
+0.0 0.15 ± 11% perf-profile.children.cycles-pp.convert
+0.0 0.07 ± 20% perf-profile.children.cycles-pp.__mod_node_page_state
+0.0 0.07 perf-profile.children.cycles-pp.do_dentry_open
+0.0 0.08 ± 16% perf-profile.children.cycles-pp.page_add_new_anon_rmap
+0.0 0.08 ± 10% perf-profile.children.cycles-pp.start_function
+0.0 0.21 ± 6% perf-profile.children.cycles-pp.decl_attributes
+0.0 0.08 perf-profile.children.cycles-pp.alloc_stmt_list
+0.0 0.06 ± 14% perf-profile.children.cycles-pp.tree_int_cst_sgn
+0.0 0.06 ± 13% perf-profile.children.cycles-pp.gimple_call_flags
+0.0 0.06 ± 13% perf-profile.children.cycles-pp.build_asm_expr
+0.0 0.13 ± 12% perf-profile.children.cycles-pp.wi::divmod_internal
+0.0 0.54 ± 2% perf-profile.children.cycles-pp.pop_scope
+0.1 0.09 ± 19% perf-profile.children.cycles-pp.pop_stmt_list
+0.1 0.16 ± 10% perf-profile.children.cycles-pp.get_range_from_loc
+0.1 0.06 ± 7% perf-profile.children.cycles-pp.build_component_ref
+0.1 0.23 ± 6% perf-profile.children.cycles-pp.build_null_declspecs
+0.1 0.07 ± 35% perf-profile.children.cycles-pp.vma_merge
+0.1 0.41 ± 6% perf-profile.children.cycles-pp.fold_binary_loc
+0.1 0.69 ± 7% perf-profile.children.cycles-pp.wide_int_to_tree
+0.1 0.28 ± 12% perf-profile.children.cycles-pp.linemap_add_macro_token
+0.1 0.85 ± 11% perf-profile.children.cycles-pp.c_lex_with_flags
+0.2 0.74 ± 16% perf-profile.children.cycles-pp.htab_find_slot_with_hash
+0.2 0.96 ± 10% perf-profile.children.cycles-pp.linemap_lookup
+0.2 0.83 ± 22% perf-profile.children.cycles-pp.irq_work_interrupt
+0.2 0.83 ± 22% perf-profile.children.cycles-pp.smp_irq_work_interrupt
+0.2 0.83 ± 22% perf-profile.children.cycles-pp.irq_work_run
+0.2 0.83 ± 22% perf-profile.children.cycles-pp.printk
+0.3 2.47 ± 3% perf-profile.children.cycles-pp._cpp_lex_direct
+0.3 1.46 ± 12% perf-profile.children.cycles-pp.memchr
+0.3 0.97 ± 9% perf-profile.children.cycles-pp.irq_work_run_list
-0.8 1.94 ± 15% perf-profile.self.cycles-pp.io_serial_in
-0.3 0.94 ± 18% perf-profile.self.cycles-pp.delay_tsc
-0.2 0.24 ± 22% perf-profile.self.cycles-pp.cfb_imageblit
-0.1 0.35 ± 12% perf-profile.self.cycles-pp.bitmap_clear_bit
-0.1 0.13 ± 24% perf-profile.self.cycles-pp.statistics_fini_pass
-0.1 0.16 ± 8% perf-profile.self.cycles-pp.unmap_page_range
-0.1 0.26 ± 11% perf-profile.self.cycles-pp.__list_del_entry_valid
-0.1 0.21 ± 19% perf-profile.self.cycles-pp.bitmap_bit_p
-0.1 0.04 ± 73% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
-0.1 0.12 ± 19% perf-profile.self.cycles-pp.bitmap_copy
-0.1 0.15 ± 13% perf-profile.self.cycles-pp.side_effects_p
-0.0 0.05 ± 72% perf-profile.self.cycles-pp.io_serial_out
-0.0 0.12 ± 6% perf-profile.self.cycles-pp.note_stores
-0.0 0.12 ± 10% perf-profile.self.cycles-pp.syscall_return_via_sysret
-0.0 0.05 ± 78% perf-profile.self.cycles-pp.copy_page
-0.0 0.13 ± 9% perf-profile.self.cycles-pp.entry_SYSCALL_64
-0.0 0.14 ± 9% perf-profile.self.cycles-pp.get_attribute_name
-0.0 0.04 ± 71% perf-profile.self.cycles-pp.lra_inheritance
-0.0 0.20 ± 8% perf-profile.self.cycles-pp.init_alias_analysis
-0.0 0.07 ± 28% perf-profile.self.cycles-pp.find_unreachable_blocks
-0.0 0.08 ± 32% perf-profile.self.cycles-pp.for_each_inc_dec
-0.0 0.04 ± 71% perf-profile.self.cycles-pp.wrapup_global_declaration_2
-0.0 0.10 ± 14% perf-profile.self.cycles-pp.free_pcppages_bulk
-0.0 0.06 ± 8% perf-profile.self.cycles-pp.bitmap_and
-0.0 0.04 ± 71% perf-profile.self.cycles-pp.page_remove_rmap
-0.0 0.07 ± 23% perf-profile.self.cycles-pp.update_stmt_operands
-0.0 0.10 ± 14% perf-profile.self.cycles-pp.hash_rtx_cb
-0.0 0.07 ± 6% perf-profile.self.cycles-pp.et_set_father
-0.0 0.08 ± 6% perf-profile.self.cycles-pp.substitute_and_fold_dom_walker::before_dom_children
+0.0 0.09 perf-profile.self.cycles-pp.realloc
+0.0 0.09 ± 5% perf-profile.self.cycles-pp.htab_traverse
+0.0 0.10 ± 4% perf-profile.self.cycles-pp.build_unary_op
+0.0 0.11 ± 11% perf-profile.self.cycles-pp.rtx_equal_p
+0.0 0.10 ± 9% perf-profile.self.cycles-pp.force_fit_type
+0.0 0.09 ± 10% perf-profile.self.cycles-pp.c_fully_fold
+0.0 0.16 ± 5% perf-profile.self.cycles-pp.declspecs_add_type
+0.0 0.13 ± 6% perf-profile.self.cycles-pp.convert
+0.0 0.11 ± 11% perf-profile.self.cycles-pp.pushdecl
+0.0 0.07 ± 11% perf-profile.self.cycles-pp.start_function
+0.0 0.18 ± 2% perf-profile.self.cycles-pp.decl_attributes
+0.0 0.07 ± 11% perf-profile.self.cycles-pp.malloc@plt
+0.0 0.13 ± 13% perf-profile.self.cycles-pp.is_typedef_decl
+0.0 0.08 perf-profile.self.cycles-pp.alloc_stmt_list
+0.0 0.06 ± 13% perf-profile.self.cycles-pp.mark_irreducible_loops
+0.1 0.13 ± 12% perf-profile.self.cycles-pp.wi::divmod_internal
+0.1 0.07 ± 25% perf-profile.self.cycles-pp.__mod_node_page_state
+0.1 0.54 ± 2% perf-profile.self.cycles-pp.pop_scope
+0.1 0.06 ± 8% perf-profile.self.cycles-pp.build_component_ref
+0.1 0.06 ± 16% perf-profile.self.cycles-pp.gimple_call_flags
+0.1 0.15 ± 12% perf-profile.self.cycles-pp.get_range_from_loc
+0.1 0.08 ± 12% perf-profile.self.cycles-pp.pop_stmt_list
+0.1 0.23 ± 4% perf-profile.self.cycles-pp.build_null_declspecs
+0.1 0.35 ± 9% perf-profile.self.cycles-pp.fold_binary_loc
+0.1 0.20 ± 2% perf-profile.self.cycles-pp.linemap_add_macro_token
+0.1 0.67 ± 8% perf-profile.self.cycles-pp.wide_int_to_tree
+0.2 0.89 ± 10% perf-profile.self.cycles-pp.linemap_lookup
+0.2 2.39 ± 3% perf-profile.self.cycles-pp._cpp_lex_direct
+0.3 1.40 ± 12% perf-profile.self.cycles-pp.memchr
have been estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
configuration may affect actual performance.

Attachments:

(No filename) (297.03 kB)
config-5.6.0-rc4-00106-g59901cb4520c4 (206.91 kB)
job-script (7.93 kB)
job.yaml (5.52 kB)
reproduce (1.44 kB)
Download all attachments

2020-03-29 05:02:23

by Aubrey Li

[permalink] [raw]

Subject: Re: [sched/fair] 59901cb452: netperf.Throughput_Mbps -27.3% regression

On Sat, Mar 28, 2020 at 9:24 PM kernel test robot <[email protected]> wrote:
>
> Greeting,
>
> FYI, we noticed a -27.3% regression of netperf.Throughput_Mbps due to commit:
>
>
> commit: 59901cb4520c44bfce81f523bc61e7284a931ad1 ("[PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group")

Thanks for the report, Robot, ;-)

This patch is abandoned, and replaced by another one:

- [PATCH] sched/fair: Fix negative imbalance in imbalance calculation

Thanks,
-Aubrey