LinuxLists.cc - [PATCH] sched/fair: update_pick_idlest() Select group with lowest group_util when idle

2020-06-16 16:50:29

Subject: [PATCH] sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal

From: Peter Puhov <[email protected]>

In slow path, when selecting idlest group, if both groups have type
group_has_spare, only idle_cpus count gets compared.
As a result, if multiple tasks are created in a tight loop,
and go back to sleep immediately
(while waiting for all tasks to be created),
they may be scheduled on the same core, because CPU is back to idle
when the new fork happen.

For example:
sudo perf record -e sched:sched_wakeup_new -- \
sysbench threads --threads=4 run
...
total number of events: 61582
...
sudo perf script
sysbench 129378 [006] 74586.633466: sched:sched_wakeup_new:
sysbench:129380 [120] success=1 CPU:007
sysbench 129378 [006] 74586.634718: sched:sched_wakeup_new:
sysbench:129381 [120] success=1 CPU:007
sysbench 129378 [006] 74586.635957: sched:sched_wakeup_new:
sysbench:129382 [120] success=1 CPU:007
sysbench 129378 [006] 74586.637183: sched:sched_wakeup_new:
sysbench:129383 [120] success=1 CPU:007

This may have negative impact on performance for workloads with frequent
creation of multiple threads.

In this patch we using group_util to select idlest group if both groups
have equal number of idle_cpus. In this case newly created tasks would be
better distributed. It is possible to use nr_running instead of group_util,
but result is less predictable.

With this patch:
sudo perf record -e sched:sched_wakeup_new -- \
sysbench threads --threads=4 run
...
total number of events: 74401
...
sudo perf script
sysbench 129455 [006] 75232.853257: sched:sched_wakeup_new:
sysbench:129457 [120] success=1 CPU:008
sysbench 129455 [006] 75232.854489: sched:sched_wakeup_new:
sysbench:129458 [120] success=1 CPU:009
sysbench 129455 [006] 75232.855732: sched:sched_wakeup_new:
sysbench:129459 [120] success=1 CPU:010
sysbench 129455 [006] 75232.856980: sched:sched_wakeup_new:
sysbench:129460 [120] success=1 CPU:011

We tested this patch with following benchmarks:
perf bench -f simple sched pipe -l 4000000
perf bench -f simple sched messaging -l 30000
perf bench -f simple mem memset -s 3GB -l 15 -f default
perf bench -f simple futex wake -s -t 640 -w 1
sysbench cpu --threads=8 --cpu-max-prime=10000 run
sysbench memory --memory-access-mode=rnd --threads=8 run
sysbench threads --threads=8 run
sysbench mutex --mutex-num=1 --threads=8 run
hackbench --loops 20000
hackbench --pipe --threads --loops 20000
hackbench --pipe --threads --loops 20000 --datasize 4096

and found some performance improvements in:
sysbench threads
sysbench mutex
perf bench futex wake
and no regressions in others.

master: 'commit b3a9e3b9622a ("Linux 5.8-rc1")'
$> sysbench threads --threads=16 run
sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 16
Initializing random number generator from current time
Initializing worker threads...
Threads started!
General statistics:
total time: 10.0079s
total number of events: 45526 << higher is better
Latency (ms):
min: 0.36
avg: 3.52
max: 54.22
95th percentile: 23.10
sum: 160044.33
Threads fairness:
events (avg/stddev): 2845.3750/94.18
execution time (avg/stddev): 10.0028/0.00

With patch:
$> sysbench threads --threads=16 run
sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 16
Initializing random number generator from current time
Initializing worker threads...
Threads started!
General statistics:
total time: 10.0053s
total number of events: 56567 << higher is better
Latency (ms):
min: 0.36
avg: 2.83
max: 27.65
95th percentile: 18.95
sum: 160003.83

Threads fairness:
events (avg/stddev): 3535.4375/147.38
execution time (avg/stddev): 10.0002/0.00

master: 'commit b3a9e3b9622a ("Linux 5.8-rc1")'
$> sysbench mutex --mutex-num=1 --threads=32 run
sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 32
Initializing random number generator from current time
Initializing worker threads...
Threads started!
General statistics:
total time: 1.0415s << lower is better
total number of events: 32
Latency (ms):
min: 940.57
avg: 959.24
max: 1041.05
95th percentile: 960.30
sum: 30695.84
Threads fairness:
events (avg/stddev): 1.0000/0.00
execution time (avg/stddev): 0.9592/0.02

With patch:
@> sysbench mutex --mutex-num=1 --threads=32 run
sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 32
Initializing random number generator from current time
Initializing worker threads...
Threads started!
General statistics:
total time: 0.9209s << lower is better
total number of events: 32
Latency (ms):
min: 867.37
avg: 892.09
max: 920.70
95th percentile: 909.80
sum: 28546.84
Threads fairness:
events (avg/stddev): 1.0000/0.00
execution time (avg/stddev): 0.8921/0.01

master: 'commit b3a9e3b9622a ("Linux 5.8-rc1")'
$> perf bench futex wake -s -t 128 -w 1
# Running 'futex/wake' benchmark:
Run summary [PID 2414]: blocking on 128 threads
(at [private] futex 0xaaaab663a154), waking up 1 at a time.
Wokeup 128 of 128 threads in 0.2852 ms (+-1.86%) << lower is better

With patch:
$> perf bench futex wake -s -t 128 -w 1
# Running 'futex/wake' benchmark:
Run summary [PID 5057]: blocking on 128 threads
(at [private] futex 0xaaaace461154), waking up 1 at a time.
Wokeup 128 of 128 threads in 0.2705 ms (+-1.84%) << lower is better

Signed-off-by: Peter Puhov <[email protected]>
---
kernel/sched/fair.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 02f323b85b6d..abcbdf80ee75 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8662,8 +8662,14 @@ static bool update_pick_idlest(struct sched_group *idlest,

case group_has_spare:
/* Select group with most idle CPUs */
- if (idlest_sgs->idle_cpus >= sgs->idle_cpus)
+ if (idlest_sgs->idle_cpus > sgs->idle_cpus)
return false;
+
+ /* Select group with lowest group_util */
+ if (idlest_sgs->idle_cpus == sgs->idle_cpus &&
+ idlest_sgs->group_util <= sgs->group_util)
+ return false;
+
break;
}

--
2.20.1

2020-06-17 10:52:56

by Valentin Schneider

[permalink] [raw]

Subject: Re: [PATCH] sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal

On 16/06/20 17:48, [email protected] wrote:
> From: Peter Puhov <[email protected]>
> We tested this patch with following benchmarks:
> perf bench -f simple sched pipe -l 4000000
> perf bench -f simple sched messaging -l 30000
> perf bench -f simple mem memset -s 3GB -l 15 -f default
> perf bench -f simple futex wake -s -t 640 -w 1
> sysbench cpu --threads=8 --cpu-max-prime=10000 run
> sysbench memory --memory-access-mode=rnd --threads=8 run
> sysbench threads --threads=8 run
> sysbench mutex --mutex-num=1 --threads=8 run
> hackbench --loops 20000
> hackbench --pipe --threads --loops 20000
> hackbench --pipe --threads --loops 20000 --datasize 4096
>
> and found some performance improvements in:
> sysbench threads
> sysbench mutex
> perf bench futex wake
> and no regressions in others.
>

One nitpick for the results of those: condensing them in a table form would
make them more reader-friendly. Perhaps something like:

| Benchmark | Metric | Lower is better? | BASELINE | SERIES | DELTA |
|------------------+----------+------------------+----------+--------+-------|
| Sysbench threads | # events | No | 45526 | 56567 | +24% |
| Sysbench mutex | ... | | | | |

If you want to include more stats for each benchmark, you could have one table
per (e.g. see [1]) - it'd still be a more readable form (or so I believe).

[1]: https://lore.kernel.org/lkml/[email protected]/

> ---
> kernel/sched/fair.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 02f323b85b6d..abcbdf80ee75 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8662,8 +8662,14 @@ static bool update_pick_idlest(struct sched_group *idlest,
>
> case group_has_spare:
> /* Select group with most idle CPUs */
> - if (idlest_sgs->idle_cpus >= sgs->idle_cpus)
> + if (idlest_sgs->idle_cpus > sgs->idle_cpus)
> return false;
> +
> + /* Select group with lowest group_util */
> + if (idlest_sgs->idle_cpus == sgs->idle_cpus &&
> + idlest_sgs->group_util <= sgs->group_util)
> + return false;
> +
> break;
> }

update_sd_pick_busiest() uses the group's nr_running instead. You mention
in the changelog that using nr_running is a possible alternative, did you
try benchmarking that and seeing how it compares to using group_util?

I think it would be nice to keep pick_busiest() and pick_idlest() aligned
wherever possible/sensible.

Also, there can be cases where one group has a few "big" tasks and another
has a handful more "small" tasks. Say something like

sgs_a->group_util = U
sgs_a->sum_nr_running = N

sgs_b->group_util = U*4/3
sgs_b->sum_nr_running = N*2/3

(sgs_b has more util per task, i.e. bigger tasks on average)

Given that we're in the 'group_has_spare' case, I would think picking the
group with the lesser amount of running tasks would make sense. Though I
guess you can find pathological cases where the util per task difference is
huge and we should look at util first...

2020-06-17 14:55:25

by Peter Puhov

[permalink] [raw]

Subject: Re: [PATCH] sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal

On Wed, 17 Jun 2020 at 06:50, Valentin Schneider
<[email protected]> wrote:
>
>
> On 16/06/20 17:48, [email protected] wrote:
> > From: Peter Puhov <[email protected]>
> > We tested this patch with following benchmarks:
> > perf bench -f simple sched pipe -l 4000000
> > perf bench -f simple sched messaging -l 30000
> > perf bench -f simple mem memset -s 3GB -l 15 -f default
> > perf bench -f simple futex wake -s -t 640 -w 1
> > sysbench cpu --threads=8 --cpu-max-prime=10000 run
> > sysbench memory --memory-access-mode=rnd --threads=8 run
> > sysbench threads --threads=8 run
> > sysbench mutex --mutex-num=1 --threads=8 run
> > hackbench --loops 20000
> > hackbench --pipe --threads --loops 20000
> > hackbench --pipe --threads --loops 20000 --datasize 4096
> >
> > and found some performance improvements in:
> > sysbench threads
> > sysbench mutex
> > perf bench futex wake
> > and no regressions in others.
> >
>
> One nitpick for the results of those: condensing them in a table form would
> make them more reader-friendly. Perhaps something like:
>
> | Benchmark | Metric | Lower is better? | BASELINE | SERIES | DELTA |
> |------------------+----------+------------------+----------+--------+-------|
> | Sysbench threads | # events | No | 45526 | 56567 | +24% |
> | Sysbench mutex | ... | | | | |
>
> If you want to include more stats for each benchmark, you could have one table
> per (e.g. see [1]) - it'd still be a more readable form (or so I believe).
>
> [1]: https://lore.kernel.org/lkml/[email protected]/
>
Good point.
I will reformat test results.

> > ---
> > kernel/sched/fair.c | 8 +++++++-
> > 1 file changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 02f323b85b6d..abcbdf80ee75 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -8662,8 +8662,14 @@ static bool update_pick_idlest(struct sched_group *idlest,
> >
> > case group_has_spare:
> > /* Select group with most idle CPUs */
> > - if (idlest_sgs->idle_cpus >= sgs->idle_cpus)
> > + if (idlest_sgs->idle_cpus > sgs->idle_cpus)
> > return false;
> > +
> > + /* Select group with lowest group_util */
> > + if (idlest_sgs->idle_cpus == sgs->idle_cpus &&
> > + idlest_sgs->group_util <= sgs->group_util)
> > + return false;
> > +
> > break;
> > }
>
> update_sd_pick_busiest() uses the group's nr_running instead. You mention
> in the changelog that using nr_running is a possible alternative, did you
> try benchmarking that and seeing how it compares to using group_util?
>
> I think it would be nice to keep pick_busiest() and pick_idlest() aligned
> wherever possible/sensible.
>
I agree with you.
> Also, there can be cases where one group has a few "big" tasks and another
> has a handful more "small" tasks. Say something like
>
> sgs_a->group_util = U
> sgs_a->sum_nr_running = N
>
> sgs_b->group_util = U*4/3
> sgs_b->sum_nr_running = N*2/3
>
> (sgs_b has more util per task, i.e. bigger tasks on average)
>
> Given that we're in the 'group_has_spare' case, I would think picking the
> group with the lesser amount of running tasks would make sense. Though I
> guess you can find pathological cases where the util per task difference is
> huge and we should look at util first...
I will re-run the tests with logic based on sum_nr_running and post results.
Thank you for suggestions.

2020-07-01 09:29:34

by Chen, Rong A

[permalink] [raw]

Subject: [sched/fair] 0b9730e694: vm-scalability.throughput 7.7% improvement

Greeting,

FYI, we

commit: url:
in testcase: vm-scalability
on test with following parameters:

runtime: 300s
size: 8T
test: anon-w-seq-mt
cpufreq_governor: performance
ucode: 0x2006906

test-description: test-url: In addition
+---------------- | testcase: | test machine | test parameters | | | | +----------------

Details are as below:
-----------------

To reproduce:

git clone bin/lkp run job.yaml

================= compiler/cpufreq_ gcc-9/performan
commit:
87e867b426 0b9730e694
87e867b4269f29da ---------------- %stddev \ 0.01 538577 4.26 ± 20% 3.11 ± 4% 56815424 242763 ± 7% 13544465 8173 10998 13712 629591 ± 2% 1.463e+10 1.219e+08 ±166% 7655935 ± 32% 47783 ± 37% 19.83 ± 3% 0.00 ± 11% 36.14 44.02 20.00 ± 3% 43.50 85.25 10286 ± 4% 217561 524.75 ± 5% 524.75 ± 5% 6790 ± 2% 6840 ± 2% 1068 ± 2% 1068 ± 2% 42637428 42637045 34573140 42547114 85538597 45201309 79011 20748538 ± 3% 20748299 ± 3% 16858671 ± 3% 20716928 ± 3% 21996436 ± 3% 38539 ± 2% 22305061 ± 5% 22304917 ± 5% 18167081 ± 5% 22247308 ± 5% 23623142 ± 4% 41615 ± 4% 5250950 ± 3% 5242969 ± 3% 8253 ± 4% 9616 ± 3% 5249215 ± 3% 5655438 ± 4% 5483453 ± 4% 5608225 ± 5% 5594460 ± 5% 8890 ± 5% 10446 ± 4% 5606844 ± 5% 10736731 10714236 17081 464.50 ± 2% 3751388 7511950 37803600 530.25 20048 44177 430.50 ± 2% 10736723 530.25 3830 ± 86% 20727730 17667 20693962 9108038 14037 ± 2% 3.26e+09 14355270 3.257e+09 6338210 109340 ± 10% 2092 ± 19% 18497 ± 3% 872088 ± 13% 5167 ± 23% 89568 ± 10% 3.71 ± 22% 11899740 ± 9% 12268244 ± 9% 11092219 ± 11% 250035 ± 9% 0.73 ± 5% 0.53 ± 23% 0.19 ± 17% 42.00 ± 22% 251.91 ± 24% 60.77 ± 21% 63.48 ±129% 63.52 ±129% 894.04 ± 10% 2625 ± 35% 459.19 ± 23% 104083 ±151% 472076 ± 35% 250006 ± 9% 718.36 ± 8% 1335 ± 10% 295.38 ± 26% 169.25 ± 11% 0.53 ± 23% 188416 ± 13% 18.94 ± 26% 18.94 ± 26% 10205 ± 5% 502.67 ± 36% 2734 ± 32% 0.75 ± 8% 2.27 ± 34% 0.53 ± 23% 157.77 ± 33% -102.27 31.78 ± 14% 4080 ± 12% 11397 ± 12% 1460 ± 18% 2074 ± 16% 18659 ± 25% 2942 ± 16% 5.038e+10 0.04 ± 2% 73.81 5.959e+08 7.992e+08 10301 ± 4% 1.39 2.211e+11 316.70 ± 2% 481094 ± 3% 3.953e+10 0.00 378091 1.152e+10 28.57 ± 3% 268142 804233 ± 2% 1.577e+11 615208 ± 2% 0.72 2.12 1.02 980.01 47481 75840108 ± 2% 47481 0.02 74.57 1.40 370.98 0.00 25.03 ± 2% 588828 0.71 5.007e+10 5.922e+08 7.942e+08 10230 ± 4% 2.197e+11 314.22 ± 2% 477788 ± 3% 3.928e+10 375508 1.144e+10 266199 ± 2% 797608 ± 2% 1.567e+11 47130 75363222 ± 2% 47130 4.747e+13 78.02 ± 2% 0.64 0.85 ± 12% 0.85 ± 12% 0.85 ± 12% 0.85 ± 12% 0.84 ± 12% 0.63 ± 14% 0.84 ± 12% 0.84 ± 12% 0.84 ± 12% 0.84 ± 12% 0.84 ± 12% 0.84 ± 12% 0.45 ± 59% 0.29 ±100% 0.29 ±100% 0.14 ±173% 1.61 ± 59% 1.72 ± 56% 1.73 ± 56% 1.78 ± 55% 1.78 ± 55% 1.78 ± 55% 1.79 ± 55% 66.80 ± 2% 1.98 ± 5% 0.14 ± 10% 0.12 ± 12% 0.07 ± 26% 0.11 ± 13% 0.14 ± 10% 0.13 ± 24% 0.13 ± 13% 0.13 ± 13% 0.15 ± 11% 0.15 ± 13% 0.15 ± 11% 0.15 ± 11% 0.27 ± 7% 0.25 ± 10% 0.01 ±173% 0.53 ± 9% 0.57 ± 10% 0.52 ± 12% 0.60 ± 11% 0.85 ± 12% 0.85 ± 12% 0.85 ± 12% 0.86 ± 11% 0.65 ± 12% 0.87 ± 11% 1.18 ± 10% 1.18 ± 10% 0.02 ±173% 0.02 ±173% 0.03 ±173% 1.63 ± 60% 1.78 ± 55% 1.75 ± 57% 1.75 ± 57% 1.79 ± 55% 1.79 ± 55% 1.79 ± 55% 1.69 ± 5% 0.54 ± 4% 0.10 ± 12% 0.07 ± 26% 0.10 ± 15% 0.25 ± 10% 0.51 ± 13% 0.02 ±173% 1.63 ± 60% 24783 ± 6% 14460 ± 6% 22296 ± 4% 13585 ± 3% 23996 ± 7% 12300 ± 2% 22392 ± 7% 12060 ± 3% 22300 ± 7% 12071 ± 2% 22418 ± 7% 12155 ± 3% 22475 ± 7% 12115 23395 ± 8% 11880 ± 4% 23634 ± 8% 11937 ± 5% 23617 ± 8% 11641 ± 6% 23654 ± 8% 12132 24073 ± 9% 11844 ± 4% 24318 ± 7% 11233 ± 7% 24480 ± 7% 11689 ± 5% 24552 ± 5% 11941 ± 5% 23989 ± 9% 12033 ± 4% 24304 ± 10% 12366 ± 4% 24320 ± 10% 11599 ± 8% 24150 ± 8% 11680 ± 5% 24166 ± 8% 11618 ± 4% 24129 ± 8% 11998 ± 3% 23975 ± 8% 11973 ± 4% 23862 ± 8% 11391 ± 6% 21729 ± 9% 13617 ± 9% 21141 ± 8% 11987 ± 3% 22180 ± 6% 11774 ± 4% 21072 ± 7% 11848 ± 6% 23495 ± 8% 12036 ± 5% 23130 ± 6% 12174 ± 3% 23136 ± 7% 12262 ± 2% 23472 ± 8% 12287 ± 2% 23564 ± 7% 12392 ± 3% 22893 ± 12% 12028 ± 2% 23270 ± 7% 11869 ± 5% 23159 ± 7% 12208 ± 2% 24507 ± 9% 12179 ± 4% 23516 ± 6% 12160 ± 3% 23688 ± 7% 12139 ± 2% 23739 ± 10% 11691 ± 3% 23418 ± 7% 11982 ± 2% 23222 ± 7% 12059 ± 3% 22787 ± 8% 12076 ± 3% 23099 ± 6% 12205 ± 3% 23538 ± 6% 12362 ± 4% 21975 ± 8% 12162 ± 2% 21884 ± 8% 12047 ± 3% 22283 ± 6% 12534 ± 7% 21918 ± 7% 12033 ± 2% 21793 ± 7% 12045 ± 3% 23337 ± 7% 12008 ± 3% 22065 ± 7% 12190 ± 2% 21952 ± 8% 10932 ± 6% 23038 ± 8% 11553 ± 7% 22728 ± 5% 11702 ± 3% 23652 ± 9% 11916 ± 3% 23314 ± 7% 11909 ± 4% 23410 ± 8% 11350 ± 9% 23567 ± 6% 11618 ± 4% 23485 ± 8% 11883 ± 2% 23186 ± 8% 11650 ± 3% 23423 ± 8% 11840 ± 4% 23408 ± 7% 11389 ± 6% 23640 ± 8% 12138 ± 3% 23653 ± 9% 11576 ± 7% 23693 ± 8% 11820 ± 4% 24003 ± 7% 11040 ± 7% 23497 ± 8% 11428 ± 6% 23861 ± 7% 12081 ± 4% 23392 ± 8% 11894 ± 2% 23361 ± 8% 11659 ± 4% 101304 ± 2% 23359 ± 8% 11613 ± 5% 23524 ± 6% 11877 ± 2% 23404 ± 8% 11517 ± 8% 23447 ± 8% 11737 ± 3% 23646 ± 7% 11853 ± 2% 23309 ± 7% 11029 ± 10% 23671 ± 9% 11840 ± 3% 20587 ± 10% 11410 ± 5% 20714 ± 11% 11749 ± 5% 20628 ± 11% 11581 ± 2% 23137 ± 7% 12028 ± 2% 23382 ± 8% 11926 ± 2% 23564 ± 8% 12048 ± 5% 23147 ± 8% 11618 ± 3% 23143 ± 7% 11834 ± 3% 25618 ± 15% 12173 ± 2% 23905 ± 8% 11861 ± 2% 23375 ± 8% 11894 ± 3% 23267 ± 7% 11945 ± 2% 22207 ± 15% 12102 ± 4% 23301 ± 7% 11843 ± 5% 23403 ± 7% 12155 ± 2% 23389 ± 6% 11801 ± 5% 23478 ± 8% 12089 ± 3% 22491 ± 7% 11867 ± 5% 22365 ± 7% 11943 22456 ± 6% 11755 ± 6% 22637 ± 8% 12258 21694 ± 10% 11727 ± 5% 22623 ± 8% 11591 ± 8% 22778 ± 7% 11996 ± 2% 22540 ± 8% 12054 ± 3% 22033 ± 7% 11664 ± 5% 21947 ± 6% 11548 ± 4% 2406782 ± 7% 1241423 2606651 24001 ± 10% 1294 ± 17% 22837 ± 12% 24970 ± 10% 5304 ± 8% 5304 ± 8% 23793 ± 9% 24945 ± 7% 24160 ± 9% 26539 ± 7% 26164 ± 8% 25753 ± 7% 25432 ± 8% 27130 ± 11% 2191 ± 67% 26628 ± 11% 26565 ± 9% 26254 ± 10% 24169 ± 12% 23839 ± 12% 23893 ± 12% 23697 ± 12% 23622 ± 11% 23316 ± 12% 23806 ± 10% 5317 ± 9% 5317 ± 9% 23778 ± 11% 24616 ± 8% 5349 ± 7% 5349 ± 7% 23748 ± 10% 23957 ± 9% 23627 ± 10% 23923 ± 12% 23564 ± 13% 24408 ± 10% 23948 ± 12% 24397 ± 8% 23493 ± 10% 24196 ± 11% 23837 ± 11% 24461 ± 9% 23888 ± 10% 24856 ± 8% 23805 ± 11% 24069 ± 9% 23714 ± 10% 24007 ± 10% 23529 ± 12% 24680 ± 9% 24159 ± 10% 24361 ± 9% 23727 ± 11% 26819 ± 11% 26314 ± 10% 26161 ± 11% 25778 ± 13% 26169 ± 9% 1105 ± 75% 26027 ± 11% 25973 ± 9% 25752 ± 9% 24234 ± 9% 23495 ± 10% 26185 ± 7% 25521 ± 10% 26134 ± 9% 25715 ± 10% 26017 ± 6% 25924 ± 8% 25759 ± 8% 25721 ± 9% 26350 ± 10% 26361 ± 10% 25688 ± 9% 5730 ± 9% 5730 ± 9% 25465 ± 9% 26476 ± 9% 25299 ± 9% 26120 ± 8% 26122 ± 9% 25800 ± 8% 25485 ± 11% 25674 ± 10% 25434 ± 10% 23756 ± 10% 5274 ± 8% 5274 ± 8% 23181 ± 12% 25540 ± 7% 25341 ± 8% 25598 ± 7% 25128 ± 9% 25322 ± 9% 5805 ± 5% 5805 ± 5% 25311 ± 10% 25461 ± 9% 25467 ± 9% 25772 ± 7% 25648 ± 9% 26004 ± 8% 25761 ± 10% 26517 ± 9% 26327 ± 10% 26088 ± 8% 25758 ± 10% 26074 ± 9% 25765 ± 11% 26773 ± 11% 25748 ± 10% 24208 ± 12% 23384 ± 13% 26594 ± 9% 26592 ± 9% 26067 ± 10% 26036 ± 11% 25324 ± 9% 23960 ± 10% 23969 ± 15% 23230 ± 13% 24615 ± 11% 23770 ± 12% 23405 ± 12% 5218 ± 8% 5218 ± 8% 22815 ± 12% 23722 ± 10% 22985 ± 12% 24226 ± 10% 5293 ± 8% 5293 ± 8% 23393 ± 12% 24006 ± 10% 5430 ± 8% 5430 ± 8% 23223 ± 11% 24840 ± 15% 23203 ± 13% 24857 ± 8% 24091 ± 8% 24617 ± 8% 5369 ± 5% 5369 ± 5% 23658 ± 10% 24148 ± 11% 23107 ± 13% 23325 ± 11% 22813 ± 12% 24771 ± 10% 23865 ± 11% 24478 ± 10% 1760 ±106% 23902 ± 11% 23545 ± 11% 5257 ± 8% 5257 ± 8% 22799 ± 11% 23472 ± 10% 23353 ± 11% 23826 ± 11% 5536 ± 7% 5536 ± 7% 23164 ± 12% 23689 ± 10% 23134 ± 12% 23593 ± 12% 22725 ± 14% 25294 ± 8% 23919 ± 10% 23343 ± 12% 23114 ± 13% 23701 ± 9% 23442 ± 10% 24572 ± 10% 23959 ± 11% 23571 ± 10% 23044 ± 12% 23446 ± 11% 22939 ± 12% 24200 ± 11% 5274 ± 8% 5274 ± 8% 23798 ± 11% 23724 ± 11% 23417 ± 12% 23835 ± 10% 23439 ± 12% 26028 ± 6% 25322 ± 10% 25722 ± 7% 25377 ± 9% 24086 ± 8% 23679 ± 10% 26378 ± 8% 25853 ± 9% 25298 ± 8% 25002 ± 9% 24700 ± 8% 24643 ± 8% 26133 ± 6% 6087 ± 6% 6087 ± 6% 25696 ± 10% 25825 ± 8% 25859 ± 8% 25611 ± 8% 25292 ± 8% 25606 ± 10% 25394 ± 10% 25580 ± 6% 25419 ± 8% 25493 ± 8% 24751 ± 8% 26074 ± 4% 25342 ± 8% 24062 ± 10% 23408 ± 11% 25583 ± 8% 25215 ± 9% 25536 ± 8% 25420 ± 8% 25510 ± 5% 24921 ± 8% 25076 ± 10% 24717 ± 10% 25460 ± 8% 25217 ± 9% 26120 ± 8% 25864 ± 9% 26369 ± 4% 25684 ± 9% 26552 ± 3% 25821 ± 9% 26734 ± 9% 26525 ± 9% 26309 ± 8% 26023 ± 9% 555164 ± 5% 555164 ± 5% 92265 ± 16% 2555541

6.2e+07 | 6.1e+07 |-+ | O | O 6e+07 |-+ | 5.9e+07 |-+ | O 5.8e+07 |-+ | |.. 5.7e+07 |-+ .+.+.. | +.+. 5.6e+07 14500 | 14000 |-.+.+..+ + 13500 |.+ + + | + 13000 |-+ | 12500 |-+ | 12000 |-+ 11500 |-+ | O O O 11000 |-+O O | 10500 11500 | 11000 |-.+.+..+. .. |. + | 10500 |-+ | 10000 |-+ | 9500 |-+ | O | O O O O 9000 |-+O | 8500 8600 +--------- 8400 |-+ | .+.+..+.. + 8200 |.+ + 8000 |-+ 7800 |-+ 7600 |-+ | 7400 |-+ 7200 |-+ 7000 |-+ 6800 |-+ O | O O O O 6600 |-+O 6400 +--------- 280000 260000 |-+ +..+ |+ +.. + : 240000 |-+ + + 220000 |-++ | 200000 |-+ 180000 |-+ 160000 |-+ | 140000 |-+ 120000 |-+ O O | 100000 |-+O O 80000 [*] bisect-good sample
[O] bisect-bad sample

***************** lkp-csl-2sp6: ================= compiler/cpufreq_ gcc-9/performan
commit:
87e867b426 0b9730e694
87e867b4269f29da ---------------- fail:runs | 6:4 5:4 6:4 0:4 %stddev \ 0.00 ± 9% 77440 7580390 173.87 ± 3% 173.87 ± 3% 8729 1231 ± 4% 13939 ± 2% 3286 790851 ± 3% 9.14 ± 14% 484919 ± 5% 506709 ± 4% 9.75 ± 13% 82.00 6.79 ±154% 6.79 ±154% 6.79 ±154% 116660 ± 13% 115715 ± 13% 64046 ± 40% 112193 ± 13% 166875 ± 9% 165953 ± 9% 86244 ± 32% 161863 ± 9% 28921 ± 13% 28043 ± 13% 28921 ± 13% 829325 ± 10% 85543 ± 88% 41512 ± 9% 40467 ± 9% 41512 ± 9% 772937 ± 11% 70404 3533411 7075463 35670935 1747 38184 70404 1073926 1042788 1046912 2621 ± 60% 3.28 ± 28% 573715 ± 41% 409.84 ± 9% 961.69 ± 4% 0.60 ± 26% 381.58 ± 2% 8.80 ± 36% 8.80 ± 36% 97.79 ± 35% 1752 ± 18% 35.00 ± 21% 287.52 ± 35% 11.29 1.672e+10 ± 3% 6038973 ± 4% 7.919e+08 ± 3% 8.13e+08 ± 3% 2.417e+11 171.33 ± 5% 1.987e+10 ± 3% 7.962e+09 ± 3% 88.28 ± 2% 238164 ± 16% 7.33e+10 ± 3% 101312 ± 3% 0.31 2.52 480.61 ± 3% 5778 ± 3% 31.34 ± 5% 402344 ± 5% 1389803 ± 8% 12.30 ± 25% 77615511 ± 19% 7.092e+08 ± 5% 5778 ± 3% 3.31 ± 2% 306.56 ± 2% 80.61 ± 3% 73133 ± 3% 0.30 ± 2% 22.19 ± 10% 10.04 ± 22% 1.651e+10 ± 3% 5954776 ± 3% 7.822e+08 ± 3% 8.032e+08 ± 3% 2.396e+11 169.26 ± 5% 1.963e+10 ± 3% 7.864e+09 ± 3% 239234 ± 16% 7.24e+10 ± 3% 5854 404234 ± 5% 1427369 ± 9% 77404942 ± 19% 6.997e+08 ± 6% 5854 62610 ± 4% 60394 ± 3% 61763 ± 5% 61376 ± 3% 61136 ± 4% 60791 ± 3% 60820 ± 3% 60770 ± 3% 3909 ± 10% 61374 ± 3% 61378 ± 3% 60787 ± 3% 3497 ± 11% 61469 ± 4% 61016 ± 3% 60803 ± 4% 62363 ± 3% 62147 ± 3% 62025 ± 3% 61692 ± 3% 63030 ± 4% 62007 ± 3% 61993 ± 4% 61879 ± 3% 62098 ± 3% 61928 ± 2% 3831 ± 6% 61915 ± 3% 4195 ± 18% 61875 ± 3% 61825 ± 3% 62079 ± 4% 63178 ± 6% 62506 ± 3% 61931 ± 3% 62059 ± 3% 62091 ± 4% 62208 ± 3% 61786 ± 4% 61397 ± 3% 61922 ± 3% 61990 ± 3% 61781 ± 4% 62311 ± 4% 63175 ± 3% 4376 ± 13% 61299 ± 3% 63297 ± 5% 63494 ± 5% 60467 ± 4% 62426 ± 3% 3730 ± 23% 60268 ± 4% 60384 ± 3% 59884 ± 4% 61465 ± 3% 60000 ± 4% 59859 ± 4% 60460 ± 4% 62593 ± 6% 60623 ± 3% 60870 ± 4% 60338 ± 4% 60023 ± 4% 60896 ± 4% 60505 ± 4% 60947 ± 4% 60324 ± 3% 61520 ± 7% 63228 ± 5% 60844 ± 4% 60471 ± 4% 61537 ± 3% 62250 ± 2% 61938 ± 4% 61631 ± 3% 61498 ± 4% 61189 ± 3% 62192 ± 4% 61744 ± 3% 62638 ± 3% 61442 ± 4% 61741 ± 3% 63379 ± 2% 62133 ± 3% 62184 ± 3% 61565 ± 4% 61814 ± 3% 61845 ± 3% 62581 ± 5% 61505 ± 3% 62105 ± 3% 61379 ± 5% 61306 ± 3% 61228 ± 3% 61437 ± 2% 61562 ± 3% 61909 ± 3% 397809 ± 7% 5953086 ± 3% 147.25 ± 6% 1.00 ±100% 349317 ± 3% 7610 ± 4% 7610 ± 4% 349158 ± 3% 349286 ± 3% 464.00 ± 2% 348627 ± 3% 7746 7746 195.50 ± 10% 349624 ± 3% 6575 ± 24% 6575 ± 24% 349667 ± 3% 458.25 ± 2% 349672 ± 3% 180.50 ± 6% 349277 ± 3% 349462 ± 3% 349570 ± 3% 349561 ± 3% 349579 ± 3% 348985 ± 3% 349387 ± 3% 349572 ± 3% 349446 ± 3% 349125 ± 3% 0.50 ±100% +12750.0% 349355 ± 3% 349310 ± 3% 1588 ± 40% 349129 ± 3% 349286 ± 3% 349099 ± 3% 349383 ± 3% 349417 ± 3% 349141 ± 3% 183.00 ± 7% 349301 ± 3% 348747 ± 3% 349092 ± 3% 349215 ± 3% 349257 ± 3% 349192 ± 3% 171.50 ± 2% 349125 ± 3% 349295 ± 3% 348979 ± 3% 171.00 ± 6% 349387 ± 3% 7633 ± 3% 7633 ± 3% 349285 ± 3% 179.25 ± 9% 349237 ± 3% 349280 ± 3% 349364 ± 3% 166.00 ± 9% 349207 ± 3% 349270 ± 3% 165.00 ± 6% 349398 ± 3% 170.75 ± 8% 349353 ± 3% 174.75 ± 15% 349297 ± 3% 349418 ± 3% 349291 ± 3% 349211 ± 3% 349086 ± 3% 7606 ± 5% 7606 ± 5% 349419 ± 3% 7778 7778 349468 ± 3% 349573 ± 3% 7807 7807 349365 ± 3% 7556 ± 6% 7556 ± 6% 349268 ± 3% 349325 ± 3% 349320 ± 3% 349440 ± 3% 349249 ± 3% 349499 ± 3% 349496 ± 3% 475.00 ± 5% 349547 ± 3% 7557 ± 6% 7557 ± 6% 348271 ± 3% 349332 ± 3% 349226 ± 3% 7595 ± 5% 7595 ± 5% 349538 ± 3% 7552 ± 6% 7552 ± 6% 349560 ± 3% 7556 ± 6% 7556 ± 6% 349516 ± 3% 7665 ± 3% 7665 ± 3% 349088 ± 3% 147.25 ± 6% 349480 ± 3% 348916 ± 3% 349575 ± 3% 7565 ± 6% 7565 ± 6% 349025 ± 3% 349214 ± 3% 349156 ± 3% 349051 ± 3% 349153 ± 3% 349043 ± 3% 349075 ± 3% 349151 ± 3% 186.25 ± 10% 349688 ± 3% 7543 ± 5% 7543 ± 5% 189.75 ± 6% 349068 ± 3% 349046 ± 3% 349015 ± 3% 349046 ± 3% 348881 ± 3% 348773 ± 3% 349137 ± 3% 348905 ± 3% 349051 ± 3% 348781 ± 3% 348918 ± 3% 349140 ± 3% 348953 ± 3% 349321 ± 3% 348937 ± 3% 349340 ± 3% 349401 ± 3% 33527497 ± 3% 666191 ± 4% 666191 ± 4%

Disclaimer:
Results for informational design or

Thanks,
Rong Chen

noticed a 7.7% improvement of vm-scalability.throughput due to commit:
0b9730e694e8c89df348e59e8d93b4985489ffee ("[PATCH] sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal")
href="https://github.com/0day-ci/linux/commits/peter-puhov-linaro-org/sched-fair-update_pick_idlest-Select-group-with-lowest-group_util-when-idle_cpus-are-equal/20200617-004854">https://github.com/0day-ci/linux/commits/peter-puhov-linaro-org/sched-fair-update_pick_idlest-Select-group-with-lowest-group_util-when-idle_cpus-are-equal/20200617-004854
machine: 104 threads Skylake with 192G memory
The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
href="https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/">https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/
to that, the commit also has significant impact on the following tests:
--+----------------------------------------------------------------------+
change | vm-scalability: vm-scalability.throughput -7.9% regression |
| 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory |
| cpufreq_governor=performance |
| runtime=300s |
| size=512G |
| test=anon-w-rand-hugetlb |
| ucode=0x5002f01 |
--+----------------------------------------------------------------------+
--------------------------------------------------------------------------------->
href="https://github.com/intel/lkp-tests.git">https://github.com/intel/lkp-tests.git
bin/lkp install job.yaml # job file is attached in this email
========================================================================
governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
ce/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/8T/lkp-skl-fpga01/anon-w-seq-mt/vm-scalability/0x2006906
("sched/pelt: Cleanup PELT divider")
("sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal")
0b9730e694e8c89df348e59e8d9
---------------------------
%change %stddev
| \
+11.9% 0.01 ± 5% vm-scalability.free_time
-1.8% 529143 vm-scalability.median
+4.3 8.56 ± 8% vm-scalability.median_stddev%
+6.8 9.88 ± 7% vm-scalability.stddev%
+7.7% 61205029 vm-scalability.throughput
-58.5% 100640 ± 5% vm-scalability.time.involuntary_context_switches
-5.8% 12752486 vm-scalability.time.minor_page_faults
-18.6% 6655 vm-scalability.time.percent_of_cpu_this_job_got
-18.4% 8977 vm-scalability.time.system_time
-18.6% 11155 vm-scalability.time.user_time
-19.3% 508253 vm-scalability.time.voluntary_context_switches
-5.9% 1.378e+10 vm-scalability.workload
-97.6% 2921999 ± 9% cpuidle.C1.time
+90.6% 14594155 ± 20% cpuidle.C1E.usage
+78.3% 85182 ± 11% cpuidle.POLL.usage
+14.7 34.56 mpstat.cpu.all.idle%
-0.0 0.00 ± 16% mpstat.cpu.all.soft%
-6.5 29.67 mpstat.cpu.all.sys%
-8.2 35.78 mpstat.cpu.all.usr%
+72.5% 34.50 vmstat.cpu.id
-19.5% 35.00 vmstat.cpu.us
-21.1% 67.25 vmstat.procs.r
-22.3% 7994 vmstat.system.cs
-1.2% 214961 vmstat.system.in
+28.6% 675.00 ± 9% slabinfo.UDP.active_objs
+28.6% 675.00 ± 9% slabinfo.UDP.num_objs
+8.5% 7367 ± 2% slabinfo.kmalloc-2k.active_objs
+8.4% 7414 ± 2% slabinfo.kmalloc-2k.num_objs
+26.3% 1349 ± 5% slabinfo.task_group.active_objs
+26.3% 1349 ± 5% slabinfo.task_group.num_objs
-23.8% 32497308 meminfo.Active
-23.8% 32496919 meminfo.Active(anon)
-22.5% 26785639 meminfo.AnonHugePages
-23.9% 32388157 meminfo.AnonPages
-18.4% 69791362 meminfo.Committed_AS
-22.5% 35025896 meminfo.Memused
-22.1% 61526 meminfo.PageTables
-30.0% 14525085 ± 9% numa-meminfo.node0.Active
-30.0% 14524731 ± 9% numa-meminfo.node0.Active(anon)
-29.3% 11925466 ± 10% numa-meminfo.node0.AnonHugePages
-30.1% 14486516 ± 9% numa-meminfo.node0.AnonPages
-28.4% 15759062 ± 9% numa-meminfo.node0.MemUsed
-26.5% 28330 ± 8% numa-meminfo.node0.PageTables
-17.2% 18479482 ± 10% numa-meminfo.node1.Active
-17.2% 18479448 ± 10% numa-meminfo.node1.Active(anon)
-16.4% 15196131 ± 9% numa-meminfo.node1.AnonHugePages
-17.3% 18409202 ± 10% numa-meminfo.node1.AnonPages
-16.3% 19775516 ± 9% numa-meminfo.node1.MemUsed
-17.0% 34527 ± 8% numa-meminfo.node1.PageTables
-32.0% 3568295 ± 8% numa-vmstat.node0.nr_active_anon
-32.2% 3556656 ± 8% numa-vmstat.node0.nr_anon_pages
-30.6% 5726 ± 9% numa-vmstat.node0.nr_anon_transparent_hugepages
-27.7% 6948 ± 8% numa-vmstat.node0.nr_page_table_pages
-32.0% 3567541 ± 8% numa-vmstat.node0.nr_zone_active_anon
-9.6% 5111268 ± 2% numa-vmstat.node0.numa_hit
-9.9% 4942508 ± 2% numa-vmstat.node0.numa_local
-20.0% 4486804 ± 10% numa-vmstat.node1.nr_active_anon
-20.1% 4470975 ± 10% numa-vmstat.node1.nr_anon_pages
-18.6% 7237 ± 10% numa-vmstat.node1.nr_anon_transparent_hugepages
-19.5% 8404 ± 9% numa-vmstat.node1.nr_page_table_pages
-20.0% 4486341 ± 10% numa-vmstat.node1.nr_zone_active_anon
-24.1% 8149870 proc-vmstat.nr_active_anon
-24.2% 8119391 proc-vmstat.nr_anon_pages
-23.3% 13103 proc-vmstat.nr_anon_transparent_hugepages
+7.1% 497.25 proc-vmstat.nr_dirtied
+6.9% 4011879 proc-vmstat.nr_dirty_background_threshold
+6.9% 8033567 proc-vmstat.nr_dirty_threshold
+6.9% 40399954 proc-vmstat.nr_free_pages
+2.5% 543.75 proc-vmstat.nr_inactive_file
-22.6% 15522 proc-vmstat.nr_page_table_pages
+1.0% 44603 proc-vmstat.nr_slab_unreclaimable
+6.1% 456.75 proc-vmstat.nr_written
-24.1% 8149865 proc-vmstat.nr_zone_active_anon
+2.5% 543.75 proc-vmstat.nr_zone_inactive_file
+135.6% 9022 ± 23% proc-vmstat.numa_hint_faults_local
-5.5% 19581418 proc-vmstat.numa_hit
-13.9% 15209 ± 7% proc-vmstat.numa_huge_pte_updates
-5.5% 19547594 proc-vmstat.numa_local
-13.9% 7845548 ± 7% proc-vmstat.numa_pte_updates
-12.0% 12353 ± 2% proc-vmstat.pgactivate
-5.9% 3.069e+09 proc-vmstat.pgalloc_normal
-5.4% 13580863 proc-vmstat.pgfault
-5.8% 3.068e+09 proc-vmstat.pgfree
-5.9% 5966713 proc-vmstat.thp_fault_alloc
-24.6% 82456 ± 2% sched_debug.cfs_rq:/.exec_clock.min
+411.2% 10698 ± 28% sched_debug.cfs_rq:/.exec_clock.stddev
+28.1% 23703 ± 4% sched_debug.cfs_rq:/.load.avg
+20.9% 1054326 ± 11% sched_debug.cfs_rq:/.load.max
-75.5% 1267 ±112% sched_debug.cfs_rq:/.load.min
+32.1% 118317 ± 11% sched_debug.cfs_rq:/.load.stddev
-75.3% 0.92 ± 77% sched_debug.cfs_rq:/.load_avg.min
-32.1% 8081747 sched_debug.cfs_rq:/.min_vruntime.avg
-26.1% 9072332 ± 2% sched_debug.cfs_rq:/.min_vruntime.max
-39.5% 6713734 ± 3% sched_debug.cfs_rq:/.min_vruntime.min
+216.5% 791257 ± 28% sched_debug.cfs_rq:/.min_vruntime.stddev
-37.6% 0.45 ± 23% sched_debug.cfs_rq:/.nr_running.avg
-76.2% 0.12 ±110% sched_debug.cfs_rq:/.nr_running.min
+77.6% 0.33 ± 6% sched_debug.cfs_rq:/.nr_running.stddev
-71.2% 12.10 ± 15% sched_debug.cfs_rq:/.nr_spread_over.avg
-50.0% 125.96 ± 31% sched_debug.cfs_rq:/.nr_spread_over.max
-67.4% 19.81 ± 26% sched_debug.cfs_rq:/.nr_spread_over.stddev
+301.6% 254.96 ± 33% sched_debug.cfs_rq:/.removed.load_avg.max
+301.4% 254.96 ± 33% sched_debug.cfs_rq:/.removed.runnable_avg.max
-37.2% 561.73 ± 19% sched_debug.cfs_rq:/.runnable_avg.avg
-41.9% 1524 ± 10% sched_debug.cfs_rq:/.runnable_avg.max
-87.4% 57.88 ±114% sched_debug.cfs_rq:/.runnable_avg.min
+620.3% 749680 ± 48% sched_debug.cfs_rq:/.spread0.avg
+268.7% 1740379 ± 29% sched_debug.cfs_rq:/.spread0.max
+216.4% 791111 ± 28% sched_debug.cfs_rq:/.spread0.stddev
-35.1% 466.32 ± 23% sched_debug.cfs_rq:/.util_avg.avg
-17.4% 1103 ± 3% sched_debug.cfs_rq:/.util_avg.max
-89.1% 32.25 ± 83% sched_debug.cfs_rq:/.util_avg.min
+60.7% 272.04 ± 8% sched_debug.cfs_rq:/.util_avg.stddev
-76.2% 0.12 ±110% sched_debug.cfs_rq:/.util_est_enqueued.min
-17.1% 156147 ± 7% sched_debug.cpu.avg_idle.stddev
-66.6% 6.33 ± 31% sched_debug.cpu.clock.stddev
-66.6% 6.33 ± 31% sched_debug.cpu.clock_task.stddev
-43.0% 5814 ± 23% sched_debug.cpu.curr->pid.avg
-70.5% 148.21 ±127% sched_debug.cpu.curr->pid.min
+47.9% 4043 ± 9% sched_debug.cpu.curr->pid.stddev
-42.2% 0.43 ± 24% sched_debug.cpu.nr_running.avg
-41.4% 1.33 ± 15% sched_debug.cpu.nr_running.max
-76.2% 0.12 ±110% sched_debug.cpu.nr_running.min
-67.4% 51.50 ± 28% sched_debug.cpu.nr_uninterruptible.max
-54.7% -46.33 sched_debug.cpu.nr_uninterruptible.min
-38.6% 19.50 ± 19% sched_debug.cpu.nr_uninterruptible.stddev
+19.6% 4882 ± 2% sched_debug.cpu.sched_goidle.avg
+52.1% 17336 ± 6% sched_debug.cpu.sched_goidle.max
+87.1% 2731 ± 7% sched_debug.cpu.sched_goidle.stddev
-53.7% 960.45 ± 11% sched_debug.cpu.ttwu_local.avg
-49.8% 9357 ± 53% sched_debug.cpu.ttwu_local.max
-64.8% 1036 ± 51% sched_debug.cpu.ttwu_local.stddev
-5.9% 4.74e+10 perf-stat.i.branch-instructions
+0.0 0.04 ± 9% perf-stat.i.branch-miss-rate%
-6.2 67.60 perf-stat.i.cache-miss-rate%
-13.0% 5.186e+08 perf-stat.i.cache-misses
-5.9% 7.521e+08 perf-stat.i.cache-references
-22.6% 7970 perf-stat.i.context-switches
-12.0% 1.23 perf-stat.i.cpi
-16.8% 1.839e+11 perf-stat.i.cpu-cycles
-13.5% 273.91 perf-stat.i.cpu-migrations
-9.2% 436689 ± 3% perf-stat.i.dTLB-load-misses
-5.8% 3.725e+10 perf-stat.i.dTLB-loads
-0.0 0.00 perf-stat.i.dTLB-store-miss-rate%
-11.8% 333435 perf-stat.i.dTLB-store-misses
-5.8% 1.085e+10 perf-stat.i.dTLB-stores
-12.0 16.59 ± 4% perf-stat.i.iTLB-load-miss-rate%
-23.3% 205627 ± 4% perf-stat.i.iTLB-load-misses
+47.1% 1183170 perf-stat.i.iTLB-loads
-5.7% 1.487e+11 perf-stat.i.instructions
+24.2% 764150 ± 3% perf-stat.i.instructions-per-iTLB-miss
+14.8% 0.83 perf-stat.i.ipc
-17.0% 1.76 perf-stat.i.metric.GHz
-4.9% 0.97 ± 2% perf-stat.i.metric.K/sec
-6.0% 921.69 perf-stat.i.metric.M/sec
-5.9% 44680 perf-stat.i.minor-faults
-4.8% 72212836 perf-stat.i.node-stores
-5.9% 44681 perf-stat.i.page-faults
+0.0 0.02 perf-stat.overall.branch-miss-rate%
-5.6 68.93 perf-stat.overall.cache-miss-rate%
-11.8% 1.24 perf-stat.overall.cpi
-4.4% 354.58 perf-stat.overall.cycles-between-cache-misses
-0.0 0.00 perf-stat.overall.dTLB-store-miss-rate%
-10.2 14.80 ± 4% perf-stat.overall.iTLB-load-miss-rate%
+22.8% 722834 ± 3% perf-stat.overall.instructions-per-iTLB-miss
+13.4% 0.81 perf-stat.overall.ipc
-5.9% 4.709e+10 perf-stat.ps.branch-instructions
-13.0% 5.151e+08 perf-stat.ps.cache-misses
-5.9% 7.473e+08 perf-stat.ps.cache-references
-22.4% 7939 perf-stat.ps.context-switches
-16.9% 1.826e+11 perf-stat.ps.cpu-cycles
-13.5% 271.91 perf-stat.ps.cpu-migrations
-9.0% 434647 ± 3% perf-stat.ps.dTLB-load-misses
-5.8% 3.7e+10 perf-stat.ps.dTLB-loads
-11.6% 331801 perf-stat.ps.dTLB-store-misses
-5.8% 1.078e+10 perf-stat.ps.dTLB-stores
-23.1% 204665 ± 4% perf-stat.ps.iTLB-load-misses
+47.7% 1178072 perf-stat.ps.iTLB-loads
-5.8% 1.477e+11 perf-stat.ps.instructions
-5.6% 44470 perf-stat.ps.minor-faults
-4.8% 71710440 perf-stat.ps.node-stores
-5.6% 44470 perf-stat.ps.page-faults
-5.4% 4.491e+13 perf-stat.total.instructions
-3.5 74.52 ± 2% perf-profile.calltrace.cycles-pp.do_access
-0.0 0.60 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.clear_page_erms.clear_subpage.clear_huge_page.do_huge_pmd_anonymous_page
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write._fini
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.write._fini
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp._fini
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.cold.new_sync_write
+0.1 0.78 ± 5% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.cold
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.vprintk_emit.devkmsg_emit.devkmsg_write.cold.new_sync_write.vfs_write
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.devkmsg_write.cold.new_sync_write.vfs_write.ksys_write.do_syscall_64
+0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.devkmsg_emit.devkmsg_write.cold.new_sync_write.vfs_write.ksys_write
+0.3 0.71 ± 5% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit
+0.3 0.62 ± 5% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
+0.3 0.62 ± 5% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
+0.4 0.56 ± 5% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
+4.6 6.25 ± 10% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
+4.7 6.43 ± 10% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
+4.7 6.45 ± 10% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
+4.8 6.53 ± 10% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
+4.8 6.53 ± 10% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
+4.8 6.53 ± 10% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
+4.8 6.64 ± 9% perf-profile.calltrace.cycles-pp.secondary_startup_64
-2.5 64.31 ± 2% perf-profile.children.cycles-pp.do_access
-0.3 1.70 ± 4% perf-profile.children.cycles-pp.___might_sleep
-0.0 0.11 ± 4% perf-profile.children.cycles-pp.prepare_exit_to_usermode
-0.0 0.10 ± 9% perf-profile.children.cycles-pp.__softirqentry_text_start
+0.0 0.10 ± 4% perf-profile.children.cycles-pp.io_serial_out
+0.0 0.16 ± 2% perf-profile.children.cycles-pp.__free_pages_ok
+0.1 0.20 ± 4% perf-profile.children.cycles-pp.release_pages
+0.1 0.19 ± 8% perf-profile.children.cycles-pp.munmap
+0.1 0.20 ± 2% perf-profile.children.cycles-pp.tlb_finish_mmu
+0.1 0.20 ± 2% perf-profile.children.cycles-pp.tlb_flush_mmu
+0.1 0.23 ± 2% perf-profile.children.cycles-pp.__do_munmap
+0.1 0.23 perf-profile.children.cycles-pp.unmap_region
+0.1 0.24 perf-profile.children.cycles-pp.__vm_munmap
+0.1 0.24 perf-profile.children.cycles-pp.__x64_sys_munmap
+0.1 0.36 ± 5% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
+0.1 0.34 ± 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
+0.1 0.11 ± 34% perf-profile.children.cycles-pp.start_kernel
+0.1 0.63 ± 6% perf-profile.children.cycles-pp.serial8250_console_putchar
+0.1 0.69 ± 5% perf-profile.children.cycles-pp.wait_for_xmitr
+0.1 0.64 ± 5% perf-profile.children.cycles-pp.io_serial_in
+0.1 0.72 ± 6% perf-profile.children.cycles-pp.uart_console_write
+0.1 0.99 ± 4% perf-profile.children.cycles-pp._fini
+0.1 0.99 ± 4% perf-profile.children.cycles-pp.devkmsg_write.cold
+0.1 0.99 ± 4% perf-profile.children.cycles-pp.devkmsg_emit
+0.1 1.01 ± 4% perf-profile.children.cycles-pp.console_unlock
+0.1 0.79 ± 5% perf-profile.children.cycles-pp.serial8250_console_write
+0.2 1.02 ± 4% perf-profile.children.cycles-pp.vprintk_emit
+0.2 1.37 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
+0.2 1.37 ± 3% perf-profile.children.cycles-pp.do_syscall_64
+0.7 0.72 ±112% perf-profile.children.cycles-pp.osq_lock
+0.7 0.76 ±112% perf-profile.children.cycles-pp.rwsem_optimistic_spin
+0.7 0.77 ±109% perf-profile.children.cycles-pp.rwsem_down_read_slowpath
+4.7 6.36 ± 10% perf-profile.children.cycles-pp.intel_idle
+4.8 6.53 ± 10% perf-profile.children.cycles-pp.start_secondary
+4.8 6.56 ± 9% perf-profile.children.cycles-pp.cpuidle_enter
+4.8 6.56 ± 9% perf-profile.children.cycles-pp.cpuidle_enter_state
+4.8 6.64 ± 9% perf-profile.children.cycles-pp.do_idle
+4.8 6.64 ± 9% perf-profile.children.cycles-pp.secondary_startup_64
+4.8 6.64 ± 9% perf-profile.children.cycles-pp.cpu_startup_entry
-0.2 1.46 ± 4% perf-profile.self.cycles-pp.___might_sleep
-0.1 0.46 ± 5% perf-profile.self.cycles-pp.rmqueue
-0.0 0.08 ± 5% perf-profile.self.cycles-pp.update_curr
+0.0 0.10 ± 4% perf-profile.self.cycles-pp.io_serial_out
+0.0 0.15 ± 2% perf-profile.self.cycles-pp.__free_pages_ok
+0.1 0.34 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
+0.1 0.64 ± 5% perf-profile.self.cycles-pp.io_serial_in
+0.7 0.72 ±112% perf-profile.self.cycles-pp.osq_lock
+4.7 6.36 ± 10% perf-profile.self.cycles-pp.intel_idle
-39.4% 15018 ± 4% softirqs.CPU0.RCU
+38.0% 19955 ± 4% softirqs.CPU0.SCHED
-35.2% 14448 ± 7% softirqs.CPU1.RCU
+44.3% 19605 ± 8% softirqs.CPU1.SCHED
-42.3% 13839 ± 5% softirqs.CPU10.RCU
+46.1% 17970 ± 9% softirqs.CPU10.SCHED
-40.5% 13332 ± 6% softirqs.CPU100.RCU
+22.1% 14731 ± 3% softirqs.CPU100.SCHED
-39.2% 13558 ± 5% softirqs.CPU101.RCU
+31.1% 15826 ± 9% softirqs.CPU101.SCHED
-40.3% 13382 ± 5% softirqs.CPU102.RCU
+31.8% 16015 ± 11% softirqs.CPU102.SCHED
-40.4% 13390 ± 5% softirqs.CPU103.RCU
+20.4% 14584 ± 12% softirqs.CPU103.SCHED
-40.5% 13917 ± 5% softirqs.CPU11.RCU
+59.6% 18961 ± 11% softirqs.CPU11.SCHED
-40.8% 13982 ± 6% softirqs.CPU12.RCU
+59.8% 19081 ± 5% softirqs.CPU12.SCHED
-41.4% 13838 ± 5% softirqs.CPU13.RCU
+59.3% 18543 ± 5% softirqs.CPU13.SCHED
-40.7% 14023 ± 4% softirqs.CPU14.RCU
+44.2% 17495 ± 7% softirqs.CPU14.SCHED
-41.5% 14088 ± 4% softirqs.CPU15.RCU
+58.9% 18817 ± 6% softirqs.CPU15.SCHED
-43.5% 13748 ± 3% softirqs.CPU16.RCU
+77.0% 19886 ± 7% softirqs.CPU16.SCHED
-42.5% 14069 ± 5% softirqs.CPU17.RCU
+61.9% 18920 ± 12% softirqs.CPU17.SCHED
-42.6% 14088 ± 6% softirqs.CPU18.RCU
+59.1% 18997 ± 7% softirqs.CPU18.SCHED
-41.5% 14026 ± 5% softirqs.CPU19.RCU
+53.2% 18433 ± 8% softirqs.CPU19.SCHED
-38.8% 14884 ± 9% softirqs.CPU2.RCU
+57.5% 19473 ± 11% softirqs.CPU2.SCHED
-42.0% 14097 ± 5% softirqs.CPU20.RCU
+57.6% 18277 ± 10% softirqs.CPU20.SCHED
-41.6% 14096 ± 4% softirqs.CPU21.RCU
+63.4% 19083 ± 7% softirqs.CPU21.SCHED
-42.3% 13939 ± 6% softirqs.CPU22.RCU
+59.9% 18577 ± 8% softirqs.CPU22.SCHED
-41.3% 14159 ± 5% softirqs.CPU23.RCU
+60.7% 19277 ± 6% softirqs.CPU23.SCHED
-41.4% 14058 ± 5% softirqs.CPU24.RCU
+55.6% 18629 ± 7% softirqs.CPU24.SCHED
-39.5% 14434 ± 5% softirqs.CPU25.RCU
+69.2% 19275 ± 11% softirqs.CPU25.SCHED
-44.2% 12128 ± 3% softirqs.CPU26.RCU
+20.6% 16422 ± 4% softirqs.CPU26.SCHED
-42.0% 12267 ± 5% softirqs.CPU27.RCU
+33.8% 16034 ± 12% softirqs.CPU27.SCHED
-44.7% 12257 ± 3% softirqs.CPU28.RCU
+35.8% 15993 ± 10% softirqs.CPU28.SCHED
-35.1% 13666 ± 18% softirqs.CPU29.RCU
+30.6% 15471 ± 10% softirqs.CPU29.SCHED
-39.7% 14157 ± 5% softirqs.CPU3.RCU
+63.9% 19727 ± 7% softirqs.CPU3.SCHED
-38.6% 14201 ± 7% softirqs.CPU30.RCU
+31.8% 16041 ± 8% softirqs.CPU30.SCHED
-37.7% 14423 ± 5% softirqs.CPU31.RCU
+31.5% 16120 ± 12% softirqs.CPU31.SCHED
-40.1% 14060 ± 5% softirqs.CPU32.RCU
+31.8% 16198 ± 11% softirqs.CPU32.SCHED
-40.8% 13960 ± 10% softirqs.CPU33.RCU
+29.9% 16092 ± 9% softirqs.CPU33.SCHED
-39.0% 13968 ± 6% softirqs.CPU34.RCU
+33.9% 16111 ± 9% softirqs.CPU34.SCHED
-39.2% 14139 ± 7% softirqs.CPU35.RCU
+31.6% 15625 ± 5% softirqs.CPU35.SCHED
-39.3% 14051 ± 6% softirqs.CPU36.RCU
+32.1% 16125 ± 10% softirqs.CPU36.SCHED
-41.6% 14315 ± 4% softirqs.CPU37.RCU
+24.5% 15169 ± 2% softirqs.CPU37.SCHED
-40.9% 13902 ± 5% softirqs.CPU38.RCU
+32.8% 16151 ± 13% softirqs.CPU38.SCHED
-40.6% 14064 ± 6% softirqs.CPU39.RCU
+32.2% 16044 ± 11% softirqs.CPU39.SCHED
-41.0% 14008 ± 6% softirqs.CPU4.RCU
+59.5% 18643 ± 7% softirqs.CPU4.SCHED
-40.5% 13930 ± 6% softirqs.CPU40.RCU
+35.9% 16279 ± 12% softirqs.CPU40.SCHED
-40.9% 13717 ± 9% softirqs.CPU41.RCU
+32.8% 16017 ± 8% softirqs.CPU41.SCHED
-38.2% 14076 ± 6% softirqs.CPU42.RCU
+33.1% 16073 ± 15% softirqs.CPU42.SCHED
-38.9% 14107 ± 5% softirqs.CPU43.RCU
+29.7% 15824 ± 9% softirqs.CPU43.SCHED
-40.6% 13981 ± 7% softirqs.CPU44.RCU
+27.7% 15788 ± 9% softirqs.CPU44.SCHED
-41.1% 12934 ± 5% softirqs.CPU45.RCU
+30.4% 15859 ± 7% softirqs.CPU45.SCHED
-39.7% 13193 ± 7% softirqs.CPU46.RCU
+32.6% 15970 ± 11% softirqs.CPU46.SCHED
-41.6% 13014 ± 7% softirqs.CPU47.RCU
+27.1% 15927 ± 13% softirqs.CPU47.SCHED
-40.4% 13066 ± 4% softirqs.CPU48.RCU
+31.5% 15824 ± 7% softirqs.CPU48.SCHED
-41.0% 12847 ± 4% softirqs.CPU49.RCU
+34.8% 16233 ± 10% softirqs.CPU49.SCHED
-39.8% 14038 ± 7% softirqs.CPU5.RCU
+62.9% 19559 ± 9% softirqs.CPU5.SCHED
-41.6% 12892 ± 4% softirqs.CPU50.RCU
+31.8% 16067 ± 13% softirqs.CPU50.SCHED
-41.3% 12894 ± 5% softirqs.CPU51.RCU
+37.5% 15029 ± 8% softirqs.CPU51.SCHED
-43.1% 13103 ± 5% softirqs.CPU52.RCU
+55.7% 17982 ± 7% softirqs.CPU52.SCHED
-42.1% 13151 ± 6% softirqs.CPU53.RCU
+64.1% 19208 ± 8% softirqs.CPU53.SCHED
-43.7% 13315 ± 5% softirqs.CPU54.RCU
+46.7% 17481 ± 10% softirqs.CPU54.SCHED
-42.1% 13496 ± 5% softirqs.CPU55.RCU
+65.3% 19687 ± 9% softirqs.CPU55.SCHED
-41.9% 13598 ± 6% softirqs.CPU56.RCU
+57.3% 17850 ± 9% softirqs.CPU56.SCHED
-42.8% 13485 ± 5% softirqs.CPU57.RCU
+59.6% 18544 ± 9% softirqs.CPU57.SCHED
-41.6% 13707 ± 5% softirqs.CPU58.RCU
+62.9% 19362 ± 7% softirqs.CPU58.SCHED
-42.8% 13264 ± 4% softirqs.CPU59.RCU
+57.5% 18347 ± 11% softirqs.CPU59.SCHED
-40.5% 13935 ± 5% softirqs.CPU6.RCU
+56.1% 18481 ± 6% softirqs.CPU6.SCHED
-41.8% 13618 ± 5% softirqs.CPU60.RCU
+63.9% 18671 ± 5% softirqs.CPU60.SCHED
-41.6% 13807 ± 4% softirqs.CPU61.RCU
+50.3% 18243 ± 5% softirqs.CPU61.SCHED
-42.4% 13618 ± 5% softirqs.CPU62.RCU
+63.7% 18953 ± 6% softirqs.CPU62.SCHED
-41.6% 13845 ± 5% softirqs.CPU63.RCU
+58.8% 18770 ± 12% softirqs.CPU63.SCHED
-42.8% 13723 ± 4% softirqs.CPU64.RCU
+73.6% 19168 ± 7% softirqs.CPU64.SCHED
-41.9% 13641 ± 4% softirqs.CPU65.RCU
+57.8% 18034 ± 16% softirqs.CPU65.SCHED
-41.5% 13954 ± 6% softirqs.CPU66.RCU
+49.6% 18074 ± 13% softirqs.CPU66.SCHED
-41.4% 13696 ± 5% softirqs.CPU67.RCU
+60.0% 19032 ± 7% softirqs.CPU67.SCHED
-44.2% 13034 ± 4% softirqs.CPU68.RCU
+58.8% 18519 ± 7% softirqs.CPU68.SCHED
+18.8% 120355 ± 18% softirqs.CPU68.TIMER
-41.0% 13776 ± 4% softirqs.CPU69.RCU
+68.0% 19509 ± 6% softirqs.CPU69.SCHED
-40.8% 13915 ± 5% softirqs.CPU7.RCU
+56.5% 18589 ± 10% softirqs.CPU7.SCHED
-40.8% 13857 ± 4% softirqs.CPU70.RCU
+66.3% 19153 ± 6% softirqs.CPU70.SCHED
-41.5% 13720 ± 4% softirqs.CPU71.RCU
+57.0% 18422 ± 5% softirqs.CPU71.SCHED
-41.8% 13752 ± 3% softirqs.CPU72.RCU
+58.4% 18773 ± 6% softirqs.CPU72.SCHED
-40.8% 13806 ± 6% softirqs.CPU73.RCU
+67.8% 18507 ± 4% softirqs.CPU73.SCHED
-42.1% 13710 ± 4% softirqs.CPU74.RCU
+62.3% 19222 ± 5% softirqs.CPU74.SCHED
-42.0% 11940 ± 4% softirqs.CPU75.RCU
+63.7% 18680 ± 9% softirqs.CPU75.SCHED
-42.0% 12014 ± 5% softirqs.CPU76.RCU
+53.2% 18005 ± 9% softirqs.CPU76.SCHED
-43.4% 11684 ± 3% softirqs.CPU77.RCU
+62.1% 18770 ± 7% softirqs.CPU77.SCHED
-42.9% 13211 ± 5% softirqs.CPU78.RCU
+28.6% 15465 ± 2% softirqs.CPU78.SCHED
-41.9% 13588 ± 6% softirqs.CPU79.RCU
+29.5% 15448 ± 9% softirqs.CPU79.SCHED
-38.5% 14491 ± 5% softirqs.CPU8.RCU
+41.0% 16989 ± 11% softirqs.CPU8.SCHED
-41.3% 13593 ± 4% softirqs.CPU80.RCU
+29.6% 15056 ± 3% softirqs.CPU80.SCHED
-41.2% 13619 ± 4% softirqs.CPU81.RCU
+30.2% 15410 ± 11% softirqs.CPU81.SCHED
-47.3% 13489 ± 5% softirqs.CPU82.RCU
+28.0% 15581 ± 7% softirqs.CPU82.SCHED
-33.7% 15859 ± 14% softirqs.CPU83.RCU
+35.0% 16013 ± 10% softirqs.CPU83.SCHED
-40.6% 13878 ± 4% softirqs.CPU84.RCU
+31.3% 15618 ± 12% softirqs.CPU84.SCHED
-37.6% 14529 ± 9% softirqs.CPU85.RCU
+30.3% 15559 ± 10% softirqs.CPU85.SCHED
-38.3% 13694 ± 5% softirqs.CPU86.RCU
+29.7% 15701 ± 10% softirqs.CPU86.SCHED
-41.2% 13693 ± 6% softirqs.CPU87.RCU
+28.7% 15238 ± 7% softirqs.CPU87.SCHED
-39.9% 14063 ± 6% softirqs.CPU88.RCU
+23.8% 15046 ± 3% softirqs.CPU88.SCHED
-40.1% 14019 ± 7% softirqs.CPU89.RCU
+31.2% 15484 ± 10% softirqs.CPU89.SCHED
-40.3% 14006 ± 6% softirqs.CPU9.RCU
+49.8% 18115 ± 9% softirqs.CPU9.SCHED
-40.0% 13504 ± 5% softirqs.CPU90.RCU
+36.2% 16166 ± 11% softirqs.CPU90.SCHED
-41.0% 13204 ± 6% softirqs.CPU91.RCU
+29.7% 15496 ± 11% softirqs.CPU91.SCHED
-40.8% 13293 ± 5% softirqs.CPU92.RCU
+33.7% 15715 ± 11% softirqs.CPU92.SCHED
-42.3% 13052 ± 10% softirqs.CPU93.RCU
+29.2% 15843 ± 9% softirqs.CPU93.SCHED
-38.0% 13446 ± 7% softirqs.CPU94.RCU
+35.6% 15901 ± 8% softirqs.CPU94.SCHED
-40.4% 13487 ± 6% softirqs.CPU95.RCU
+36.2% 15786 ± 10% softirqs.CPU95.SCHED
-39.7% 13737 ± 7% softirqs.CPU96.RCU
+33.9% 16059 ± 11% softirqs.CPU96.SCHED
-39.5% 13633 ± 7% softirqs.CPU97.RCU
+35.3% 16311 ± 8% softirqs.CPU97.SCHED
-39.4% 13357 ± 6% softirqs.CPU98.RCU
+33.6% 15587 ± 8% softirqs.CPU98.SCHED
-39.4% 13290 ± 7% softirqs.CPU99.RCU
+27.0% 14665 ± 3% softirqs.CPU99.SCHED
-40.9% 1422634 ± 5% softirqs.RCU
+44.4% 1793178 ± 2% softirqs.SCHED
-33.7% 1729139 ± 2% interrupts.CAL:Function_call_interrupts
-29.2% 17001 ± 14% interrupts.CPU0.CAL:Function_call_interrupts
-30.9% 895.00 ± 10% interrupts.CPU0.RES:Rescheduling_interrupts
-35.8% 14672 ± 5% interrupts.CPU0.TLB:TLB_shootdowns
-38.0% 15470 ± 5% interrupts.CPU1.CAL:Function_call_interrupts
-29.7% 3727 ± 16% interrupts.CPU1.NMI:Non-maskable_interrupts
-29.7% 3727 ± 16% interrupts.CPU1.PMI:Performance_monitoring_interrupts
-37.0% 14981 ± 5% interrupts.CPU1.TLB:TLB_shootdowns
-31.6% 17071 ± 5% interrupts.CPU10.CAL:Function_call_interrupts
-31.9% 16456 ± 5% interrupts.CPU10.TLB:TLB_shootdowns
-35.3% 17177 ± 6% interrupts.CPU100.CAL:Function_call_interrupts
-35.3% 16921 ± 7% interrupts.CPU100.TLB:TLB_shootdowns
-32.2% 17454 ± 5% interrupts.CPU101.CAL:Function_call_interrupts
-33.8% 16844 ± 5% interrupts.CPU101.TLB:TLB_shootdowns
-38.4% 16704 ± 6% interrupts.CPU102.CAL:Function_call_interrupts
-89.7% 226.25 ± 39% interrupts.CPU102.RES:Rescheduling_interrupts
-38.1% 16482 ± 6% interrupts.CPU102.TLB:TLB_shootdowns
-34.8% 17330 ± 7% interrupts.CPU103.CAL:Function_call_interrupts
-37.0% 16534 ± 5% interrupts.CPU103.TLB:TLB_shootdowns
-27.6% 17506 ± 12% interrupts.CPU11.CAL:Function_call_interrupts
-32.5% 16102 ± 7% interrupts.CPU11.TLB:TLB_shootdowns
-27.1% 17427 ± 5% interrupts.CPU12.CAL:Function_call_interrupts
-29.9% 16603 ± 2% interrupts.CPU12.TLB:TLB_shootdowns
-28.4% 16906 ± 7% interrupts.CPU13.CAL:Function_call_interrupts
-31.3% 16028 ± 6% interrupts.CPU13.TLB:TLB_shootdowns
-31.4% 16327 ± 2% interrupts.CPU14.CAL:Function_call_interrupts
-42.1% 3076 ± 25% interrupts.CPU14.NMI:Non-maskable_interrupts
-42.1% 3076 ± 25% interrupts.CPU14.PMI:Performance_monitoring_interrupts
-32.7% 16001 ± 3% interrupts.CPU14.TLB:TLB_shootdowns
-34.0% 16246 ± 3% interrupts.CPU15.CAL:Function_call_interrupts
-23.8% 4076 ± 17% interrupts.CPU15.NMI:Non-maskable_interrupts
-23.8% 4076 ± 17% interrupts.CPU15.PMI:Performance_monitoring_interrupts
-32.7% 15984 ± 3% interrupts.CPU15.TLB:TLB_shootdowns
-32.1% 16275 ± 4% interrupts.CPU16.CAL:Function_call_interrupts
-34.3% 15528 ± 4% interrupts.CPU16.TLB:TLB_shootdowns
-30.2% 16704 ± 3% interrupts.CPU17.CAL:Function_call_interrupts
-31.2% 16223 ± 3% interrupts.CPU17.TLB:TLB_shootdowns
-27.5% 17705 ± 4% interrupts.CPU18.CAL:Function_call_interrupts
-31.0% 16523 ± 5% interrupts.CPU18.TLB:TLB_shootdowns
-35.4% 15757 ± 6% interrupts.CPU19.CAL:Function_call_interrupts
-35.2% 15217 ± 7% interrupts.CPU19.TLB:TLB_shootdowns
-32.2% 16405 ± 5% interrupts.CPU2.CAL:Function_call_interrupts
-32.9% 15993 ± 6% interrupts.CPU2.TLB:TLB_shootdowns
-31.9% 16656 ± 4% interrupts.CPU20.CAL:Function_call_interrupts
-35.2% 15476 ± 6% interrupts.CPU20.TLB:TLB_shootdowns
-35.1% 16123 ± 6% interrupts.CPU21.CAL:Function_call_interrupts
-34.2% 15658 ± 5% interrupts.CPU21.TLB:TLB_shootdowns
-33.6% 15980 ± 4% interrupts.CPU22.CAL:Function_call_interrupts
-33.8% 15697 ± 5% interrupts.CPU22.TLB:TLB_shootdowns
-34.2% 15807 ± 9% interrupts.CPU23.CAL:Function_call_interrupts
-35.5% 15178 ± 8% interrupts.CPU23.TLB:TLB_shootdowns
-36.0% 15788 ± 5% interrupts.CPU24.CAL:Function_call_interrupts
-35.6% 15557 ± 5% interrupts.CPU24.TLB:TLB_shootdowns
-33.8% 16126 ± 4% interrupts.CPU25.CAL:Function_call_interrupts
-33.4% 15803 ± 4% interrupts.CPU25.TLB:TLB_shootdowns
-39.8% 16144 ± 13% interrupts.CPU26.CAL:Function_call_interrupts
-41.7% 15352 ± 12% interrupts.CPU26.TLB:TLB_shootdowns
-37.7% 16294 ± 5% interrupts.CPU27.CAL:Function_call_interrupts
-38.7% 15811 ± 6% interrupts.CPU27.TLB:TLB_shootdowns
-38.0% 16231 ± 5% interrupts.CPU28.CAL:Function_call_interrupts
-59.0% 452.75 ± 38% interrupts.CPU28.RES:Rescheduling_interrupts
-38.3% 16047 ± 6% interrupts.CPU28.TLB:TLB_shootdowns
-35.4% 16783 ± 4% interrupts.CPU29.CAL:Function_call_interrupts
-35.9% 16520 ± 4% interrupts.CPU29.TLB:TLB_shootdowns
-33.6% 16103 ± 4% interrupts.CPU3.CAL:Function_call_interrupts
-33.5% 15615 ± 2% interrupts.CPU3.TLB:TLB_shootdowns
-34.9% 17057 ± 7% interrupts.CPU30.CAL:Function_call_interrupts
-33.9% 16871 ± 7% interrupts.CPU30.TLB:TLB_shootdowns
-33.8% 17303 ± 2% interrupts.CPU31.CAL:Function_call_interrupts
-33.2% 17174 ± 2% interrupts.CPU31.TLB:TLB_shootdowns
-37.8% 16183 ± 2% interrupts.CPU32.CAL:Function_call_interrupts
-38.3% 15991 ± 2% interrupts.CPU32.TLB:TLB_shootdowns
-35.4% 16648 ± 5% interrupts.CPU33.CAL:Function_call_interrupts
-36.0% 16458 ± 5% interrupts.CPU33.TLB:TLB_shootdowns
-28.1% 18942 ± 26% interrupts.CPU34.CAL:Function_call_interrupts
-39.8% 15880 ± 3% interrupts.CPU34.TLB:TLB_shootdowns
-26.4% 18897 ± 19% interrupts.CPU35.CAL:Function_call_interrupts
-35.3% 3707 ± 34% interrupts.CPU35.NMI:Non-maskable_interrupts
-35.3% 3707 ± 34% interrupts.CPU35.PMI:Performance_monitoring_interrupts
-36.9% 16078 ± 7% interrupts.CPU35.TLB:TLB_shootdowns
-36.5% 16818 ± 8% interrupts.CPU36.CAL:Function_call_interrupts
-37.1% 15921 ± 7% interrupts.CPU36.TLB:TLB_shootdowns
-35.7% 16798 ± 6% interrupts.CPU37.CAL:Function_call_interrupts
-36.7% 16526 ± 7% interrupts.CPU37.TLB:TLB_shootdowns
-34.7% 16856 ± 5% interrupts.CPU38.CAL:Function_call_interrupts
-35.2% 16502 ± 5% interrupts.CPU38.TLB:TLB_shootdowns
-36.6% 16273 ± 4% interrupts.CPU39.CAL:Function_call_interrupts
-37.1% 15988 ± 5% interrupts.CPU39.TLB:TLB_shootdowns
-30.5% 16499 ± 6% interrupts.CPU4.CAL:Function_call_interrupts
-28.7% 3758 ± 21% interrupts.CPU4.NMI:Non-maskable_interrupts
-28.7% 3758 ± 21% interrupts.CPU4.PMI:Performance_monitoring_interrupts
-31.9% 15782 ± 2% interrupts.CPU4.TLB:TLB_shootdowns
-36.2% 16282 ± 4% interrupts.CPU40.CAL:Function_call_interrupts
-37.3% 15880 ± 5% interrupts.CPU40.TLB:TLB_shootdowns
-36.3% 16306 ± 5% interrupts.CPU41.CAL:Function_call_interrupts
-36.0% 16079 ± 7% interrupts.CPU41.TLB:TLB_shootdowns
-33.6% 16802 ± 3% interrupts.CPU42.CAL:Function_call_interrupts
-22.8% 4484 ± 25% interrupts.CPU42.NMI:Non-maskable_interrupts
-22.8% 4484 ± 25% interrupts.CPU42.PMI:Performance_monitoring_interrupts
-36.3% 16117 ± 6% interrupts.CPU42.TLB:TLB_shootdowns
-34.5% 16688 ± 5% interrupts.CPU43.CAL:Function_call_interrupts
-36.6% 16142 ± 3% interrupts.CPU43.TLB:TLB_shootdowns
-36.4% 16388 ± 5% interrupts.CPU44.CAL:Function_call_interrupts
-36.7% 16229 ± 6% interrupts.CPU44.TLB:TLB_shootdowns
-34.5% 17032 ± 7% interrupts.CPU45.CAL:Function_call_interrupts
-37.6% 16064 ± 9% interrupts.CPU45.TLB:TLB_shootdowns
-38.3% 16362 ± 5% interrupts.CPU46.CAL:Function_call_interrupts
-38.7% 16142 ± 6% interrupts.CPU46.TLB:TLB_shootdowns
-37.4% 16319 ± 5% interrupts.CPU47.CAL:Function_call_interrupts
-38.0% 15973 ± 7% interrupts.CPU47.TLB:TLB_shootdowns
-36.3% 16607 ± 4% interrupts.CPU48.CAL:Function_call_interrupts
-36.6% 16343 ± 4% interrupts.CPU48.TLB:TLB_shootdowns
-36.7% 16942 ± 6% interrupts.CPU49.CAL:Function_call_interrupts
-35.1% 16708 ± 6% interrupts.CPU49.TLB:TLB_shootdowns
-31.6% 16548 ± 6% interrupts.CPU5.CAL:Function_call_interrupts
-33.7% 15501 ± 7% interrupts.CPU5.TLB:TLB_shootdowns
-38.1% 16472 ± 4% interrupts.CPU50.CAL:Function_call_interrupts
-38.6% 16337 ± 5% interrupts.CPU50.TLB:TLB_shootdowns
-37.9% 16184 ± 4% interrupts.CPU51.CAL:Function_call_interrupts
-38.1% 16110 ± 4% interrupts.CPU51.TLB:TLB_shootdowns
-34.4% 16614 ± 4% interrupts.CPU52.CAL:Function_call_interrupts
-34.1% 15780 ± 3% interrupts.CPU52.TLB:TLB_shootdowns
-32.0% 16294 ± 2% interrupts.CPU53.CAL:Function_call_interrupts
-30.2% 16225 ± 2% interrupts.CPU53.TLB:TLB_shootdowns
-33.8% 16298 interrupts.CPU54.CAL:Function_call_interrupts
-32.8% 15983 interrupts.CPU54.TLB:TLB_shootdowns
-28.8% 16671 ± 2% interrupts.CPU55.CAL:Function_call_interrupts
-28.0% 3756 ± 17% interrupts.CPU55.NMI:Non-maskable_interrupts
-28.0% 3756 ± 17% interrupts.CPU55.PMI:Performance_monitoring_interrupts
-29.0% 16192 ± 3% interrupts.CPU55.TLB:TLB_shootdowns
-24.2% 17984 ± 13% interrupts.CPU56.CAL:Function_call_interrupts
-30.4% 16003 ± 5% interrupts.CPU56.TLB:TLB_shootdowns
-34.7% 15813 ± 6% interrupts.CPU57.CAL:Function_call_interrupts
-26.5% 3891 ± 24% interrupts.CPU57.NMI:Non-maskable_interrupts
-26.5% 3891 ± 24% interrupts.CPU57.PMI:Performance_monitoring_interrupts
-33.3% 15613 ± 5% interrupts.CPU57.TLB:TLB_shootdowns
-28.6% 17134 ± 9% interrupts.CPU58.CAL:Function_call_interrupts
-38.2% 3357 ± 21% interrupts.CPU58.NMI:Non-maskable_interrupts
-38.2% 3357 ± 21% interrupts.CPU58.PMI:Performance_monitoring_interrupts
-32.0% 15781 ± 8% interrupts.CPU58.TLB:TLB_shootdowns
-31.8% 16951 ± 6% interrupts.CPU59.CAL:Function_call_interrupts
-31.0% 16014 ± 7% interrupts.CPU59.TLB:TLB_shootdowns
-26.7% 18211 ± 13% interrupts.CPU6.CAL:Function_call_interrupts
-32.6% 16236 ± 6% interrupts.CPU6.TLB:TLB_shootdowns
-35.6% 15856 ± 5% interrupts.CPU60.CAL:Function_call_interrupts
-25.8% 3983 ± 19% interrupts.CPU60.NMI:Non-maskable_interrupts
-25.8% 3983 ± 19% interrupts.CPU60.PMI:Performance_monitoring_interrupts
-33.9% 15630 ± 5% interrupts.CPU60.TLB:TLB_shootdowns
-32.7% 16245 ± 6% interrupts.CPU61.CAL:Function_call_interrupts
-30.9% 15966 ± 6% interrupts.CPU61.TLB:TLB_shootdowns
-27.6% 16891 ± 10% interrupts.CPU62.CAL:Function_call_interrupts
-31.3% 15661 ± 6% interrupts.CPU62.TLB:TLB_shootdowns
-32.3% 16772 ± 5% interrupts.CPU63.CAL:Function_call_interrupts
-34.1% 15738 ± 4% interrupts.CPU63.TLB:TLB_shootdowns
-31.2% 16848 ± 4% interrupts.CPU64.CAL:Function_call_interrupts
-80.2% 348.75 ± 6% interrupts.CPU64.RES:Rescheduling_interrupts
-32.4% 16164 ± 4% interrupts.CPU64.TLB:TLB_shootdowns
-34.0% 15530 ± 6% interrupts.CPU65.CAL:Function_call_interrupts
-29.0% 3733 ± 23% interrupts.CPU65.NMI:Non-maskable_interrupts
-29.0% 3733 ± 23% interrupts.CPU65.PMI:Performance_monitoring_interrupts
-34.3% 14975 ± 6% interrupts.CPU65.TLB:TLB_shootdowns
-32.9% 15750 ± 6% interrupts.CPU66.CAL:Function_call_interrupts
-34.7% 15244 ± 4% interrupts.CPU66.TLB:TLB_shootdowns
-34.0% 15725 ± 6% interrupts.CPU67.CAL:Function_call_interrupts
-27.2% 4030 ± 29% interrupts.CPU67.NMI:Non-maskable_interrupts
-27.2% 4030 ± 29% interrupts.CPU67.PMI:Performance_monitoring_interrupts
-35.4% 14968 ± 8% interrupts.CPU67.TLB:TLB_shootdowns
-25.3% 17687 ± 19% interrupts.CPU68.CAL:Function_call_interrupts
-33.3% 15428 ± 7% interrupts.CPU68.TLB:TLB_shootdowns
-31.6% 16143 ± 5% interrupts.CPU69.CAL:Function_call_interrupts
-33.1% 15208 ± 4% interrupts.CPU69.TLB:TLB_shootdowns
-34.6% 16544 ± 2% interrupts.CPU7.CAL:Function_call_interrupts
-32.1% 16237 ± 3% interrupts.CPU7.TLB:TLB_shootdowns
-28.0% 16800 ± 5% interrupts.CPU70.CAL:Function_call_interrupts
-30.8% 15998 ± 6% interrupts.CPU70.TLB:TLB_shootdowns
-31.3% 16293 ± 6% interrupts.CPU71.CAL:Function_call_interrupts
-31.6% 16028 ± 6% interrupts.CPU71.TLB:TLB_shootdowns
-36.0% 15729 ± 6% interrupts.CPU72.CAL:Function_call_interrupts
-35.8% 15389 ± 6% interrupts.CPU72.TLB:TLB_shootdowns
-29.4% 16653 ± 5% interrupts.CPU73.CAL:Function_call_interrupts
-28.5% 16475 ± 5% interrupts.CPU73.TLB:TLB_shootdowns
-31.7% 16013 ± 8% interrupts.CPU74.CAL:Function_call_interrupts
-32.0% 15596 ± 8% interrupts.CPU74.TLB:TLB_shootdowns
-32.0% 16449 ± 3% interrupts.CPU75.CAL:Function_call_interrupts
-20.4% 4196 ± 18% interrupts.CPU75.NMI:Non-maskable_interrupts
-20.4% 4196 ± 18% interrupts.CPU75.PMI:Performance_monitoring_interrupts
-32.1% 16150 ± 3% interrupts.CPU75.TLB:TLB_shootdowns
-27.1% 17289 ± 4% interrupts.CPU76.CAL:Function_call_interrupts
-27.1% 17060 ± 5% interrupts.CPU76.TLB:TLB_shootdowns
-31.9% 16224 ± 6% interrupts.CPU77.CAL:Function_call_interrupts
-33.3% 15624 ± 5% interrupts.CPU77.TLB:TLB_shootdowns
-35.7% 16742 ± 5% interrupts.CPU78.CAL:Function_call_interrupts
-35.6% 16297 ± 3% interrupts.CPU78.TLB:TLB_shootdowns
-31.5% 17617 ± 5% interrupts.CPU79.CAL:Function_call_interrupts
-34.2% 16689 ± 2% interrupts.CPU79.TLB:TLB_shootdowns
-32.0% 16370 ± 5% interrupts.CPU8.CAL:Function_call_interrupts
-31.9% 16128 ± 5% interrupts.CPU8.TLB:TLB_shootdowns
-35.1% 17117 ± 3% interrupts.CPU80.CAL:Function_call_interrupts
-34.8% 16850 ± 3% interrupts.CPU80.TLB:TLB_shootdowns
-33.1% 16918 ± 6% interrupts.CPU81.CAL:Function_call_interrupts
-32.8% 16802 ± 8% interrupts.CPU81.TLB:TLB_shootdowns
-32.8% 16607 ± 4% interrupts.CPU82.CAL:Function_call_interrupts
-33.9% 16297 ± 5% interrupts.CPU82.TLB:TLB_shootdowns
-37.8% 16266 ± 5% interrupts.CPU83.CAL:Function_call_interrupts
-18.6% 4952 ± 13% interrupts.CPU83.NMI:Non-maskable_interrupts
-18.6% 4952 ± 13% interrupts.CPU83.PMI:Performance_monitoring_interrupts
-37.5% 16053 ± 5% interrupts.CPU83.TLB:TLB_shootdowns
-35.8% 16567 ± 6% interrupts.CPU84.CAL:Function_call_interrupts
-37.0% 16286 ± 7% interrupts.CPU84.TLB:TLB_shootdowns
-35.5% 16526 ± 6% interrupts.CPU85.CAL:Function_call_interrupts
-35.7% 16260 ± 7% interrupts.CPU85.TLB:TLB_shootdowns
-34.9% 16657 ± 6% interrupts.CPU86.CAL:Function_call_interrupts
-35.5% 16370 ± 7% interrupts.CPU86.TLB:TLB_shootdowns
-37.2% 16070 ± 5% interrupts.CPU87.CAL:Function_call_interrupts
-38.2% 15698 ± 7% interrupts.CPU87.TLB:TLB_shootdowns
-35.7% 16396 ± 7% interrupts.CPU88.CAL:Function_call_interrupts
-34.5% 16217 ± 8% interrupts.CPU88.TLB:TLB_shootdowns
-34.4% 17094 ± 4% interrupts.CPU89.CAL:Function_call_interrupts
-33.1% 16944 ± 5% interrupts.CPU89.TLB:TLB_shootdowns
-31.6% 16466 ± 6% interrupts.CPU9.CAL:Function_call_interrupts
-30.3% 16319 ± 6% interrupts.CPU9.TLB:TLB_shootdowns
-35.5% 16512 ± 8% interrupts.CPU90.CAL:Function_call_interrupts
-35.7% 16215 ± 9% interrupts.CPU90.TLB:TLB_shootdowns
-37.1% 16073 ± 7% interrupts.CPU91.CAL:Function_call_interrupts
-37.3% 15935 ± 8% interrupts.CPU91.TLB:TLB_shootdowns
-35.4% 16473 ± 4% interrupts.CPU92.CAL:Function_call_interrupts
-34.1% 16418 ± 5% interrupts.CPU92.TLB:TLB_shootdowns
-34.4% 16461 ± 7% interrupts.CPU93.CAL:Function_call_interrupts
-34.9% 16097 ± 8% interrupts.CPU93.TLB:TLB_shootdowns
-34.4% 16708 ± 4% interrupts.CPU94.CAL:Function_call_interrupts
-34.6% 16497 ± 5% interrupts.CPU94.TLB:TLB_shootdowns
-36.0% 16709 ± 7% interrupts.CPU95.CAL:Function_call_interrupts
-36.0% 16552 ± 8% interrupts.CPU95.TLB:TLB_shootdowns
-36.2% 16836 ± 7% interrupts.CPU96.CAL:Function_call_interrupts
-36.2% 16383 ± 8% interrupts.CPU96.TLB:TLB_shootdowns
-36.3% 16913 ± 5% interrupts.CPU97.CAL:Function_call_interrupts
-35.6% 16616 ± 5% interrupts.CPU97.TLB:TLB_shootdowns
-37.8% 16632 ± 4% interrupts.CPU98.CAL:Function_call_interrupts
-38.4% 16349 ± 4% interrupts.CPU98.TLB:TLB_shootdowns
-38.2% 16255 ± 6% interrupts.CPU99.CAL:Function_call_interrupts
-38.8% 15923 ± 7% interrupts.CPU99.TLB:TLB_shootdowns
-12.0% 488658 ± 3% interrupts.NMI:Non-maskable_interrupts
-12.0% 488658 ± 3% interrupts.PMI:Performance_monitoring_interrupts
-48.1% 47888 ± 7% interrupts.RES:Rescheduling_interrupts
-34.6% 1670205 ± 2% interrupts.TLB:TLB_shootdowns

vm-scalability.throughput

+-----------------------------------------------------------------+
O O O O O O |
O O O O O O O |
O O O O |
|
|
O O |
O |
|
|
|
.+.. |
.+.. +.. +. .+ +. .+.+.. .+.+..|
+.+..+ +.+.. + .. +. +. +.+. |
+-----------------------------------------------------------------+

vm-scalability.time.user_time

+-------------------------------------------------------------------+
+..+. .+. .+.. |
+.. .+.+. +. +. .+. .+..+. .+. .|
+. +..+.+..+. +. +. +. |
|
|
|
|
|
|
|
O O O O O O |
O O O O O O O O O O O O |
O |
+-------------------------------------------------------------------+

vm-scalability.time.system_time

+-------------------------------------------------------------------+
+..+.+.. + .+ |
+..+.+..+. .+.. .+.. + : .+.+. + |
+. + +.+.. + : .+. +..|
+ +. |
|
|
|
|
|
|
O O O O O O O O O |
O O O O O O |
O O O |
+-------------------------------------------------------------------+

vm-scalability.time.percent_of_cpu_this_job_got

-----------------------------------------------------------+
+..+..+ |
+ .+..+.+.. .+. +.. .+.+.. .+. |
+. +. +..+.+..+.. + +. +. +..|
+ |
|
|
|
|
|
|
O |
O O O O O O O O O O O |
O O O O O |
-----------------------------------------------------------+

vm-scalability.time.involuntary_context_switches

+------------------------------------------------------------------+
: : : : .+..+. +. + |
+. : : : : + +.. + + +.. : + |
: .. +..+ : : :.. + + : : +|
+..+ + + + + : + |
+ |
|
|
|
|
|
O O |
O O O O O |
O O O O O O O O O O O O |
+------------------------------------------------------------------+

**********************************************************************************
96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
========================================================================
governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
ce/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/512G/lkp-csl-2sp6/anon-w-rand-hugetlb/vm-scalability/0x5002f01
("sched/pelt: Cleanup PELT divider")
("sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal")
0b9730e694e8c89df348e59e8d9
---------------------------
%reproduction fail:runs
| |
-120% 1:4 perf-profile.calltrace.cycles-pp.error_entry
-109% 1:4 perf-profile.calltrace.cycles-pp.sync_regs.error_entry
-117% 1:4 perf-profile.children.cycles-pp.error_entry
-12% 0:4 perf-profile.self.cycles-pp.error_entry
%change %stddev
| \
+27.1% 0.00 ± 6% vm-scalability.free_time
-10.2% 69530 ± 2% vm-scalability.median
-7.9% 6980302 ± 2% vm-scalability.throughput
+20.9% 210.28 ± 5% vm-scalability.time.elapsed_time
+20.9% 210.28 ± 5% vm-scalability.time.elapsed_time.max
-7.2% 8099 ± 4% vm-scalability.time.percent_of_cpu_this_job_got
+22.5% 1508 ± 6% vm-scalability.time.system_time
+11.2% 15494 ± 4% vm-scalability.time.user_time
+7.8% 3542 ± 3% vm-scalability.time.voluntary_context_switches
-17.6% 651546 ± 6% meminfo.max_used_kB
+6.5 15.68 ± 23% mpstat.cpu.all.idle%
+19.5% 579550 ± 6% numa-numastat.node1.local_node
+16.8% 592066 ± 5% numa-numastat.node1.numa_hit
+66.7% 16.25 ± 23% vmstat.cpu.id
-7.9% 75.50 ± 4% vmstat.cpu.us
+28.3 35.12 ± 82% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
+28.4 35.16 ± 82% perf-profile.children.cycles-pp.intel_idle
+28.4 35.16 ± 82% perf-profile.self.cycles-pp.intel_idle
+103.1% 236933 ± 7% numa-meminfo.node0.Active
+103.6% 235539 ± 7% numa-meminfo.node0.Active(anon)
+131.7% 148384 ± 6% numa-meminfo.node0.AnonHugePages
+107.7% 233064 ± 6% numa-meminfo.node0.AnonPages
-68.3% 52918 ± 33% numa-meminfo.node1.Active
-68.4% 52464 ± 33% numa-meminfo.node1.Active(anon)
-83.0% 14632 ± 78% numa-meminfo.node1.AnonHugePages
-75.9% 39060 ± 39% numa-meminfo.node1.AnonPages
+103.6% 58883 ± 7% numa-vmstat.node0.nr_active_anon
+107.8% 58265 ± 6% numa-vmstat.node0.nr_anon_pages
+103.6% 58883 ± 7% numa-vmstat.node0.nr_zone_active_anon
-15.3% 702160 ± 7% numa-vmstat.node0.numa_local
+99.5% 170657 ± 4% numa-vmstat.node0.numa_other
-68.4% 13118 ± 33% numa-vmstat.node1.nr_active_anon
-75.9% 9765 ± 39% numa-vmstat.node1.nr_anon_pages
-68.4% 13118 ± 33% numa-vmstat.node1.nr_zone_active_anon
+21.1% 936063 ± 4% numa-vmstat.node1.numa_local
+2.3% 72002 proc-vmstat.nr_active_anon
+6.1% 3750236 ± 3% proc-vmstat.nr_dirty_background_threshold
+6.1% 7509642 ± 3% proc-vmstat.nr_dirty_threshold
+6.1% 37842330 ± 3% proc-vmstat.nr_free_pages
-2.6% 1702 proc-vmstat.nr_page_table_pages
+5.0% 40094 proc-vmstat.nr_shmem
+2.3% 72002 proc-vmstat.nr_zone_active_anon
+7.3% 1152673 ± 2% proc-vmstat.numa_hit
+7.5% 1121456 ± 2% proc-vmstat.numa_local
+9.6% 1147398 ± 3% proc-vmstat.pgfault
+84.0% 4823 ± 12% sched_debug.cfs_rq:/.exec_clock.stddev
+73.5% 5.69 ± 12% sched_debug.cfs_rq:/.nr_spread_over.avg
+63.1% 935910 ± 36% sched_debug.cfs_rq:/.spread0.max
+40.7% 576.84 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.avg
+18.6% 1140 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.max
+3648.3% 22.65 ± 40% sched_debug.cfs_rq:/.util_est_enqueued.min
-18.9% 309.54 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.stddev
+65.6% 14.56 ± 29% sched_debug.cpu.clock.stddev
+65.6% 14.56 ± 29% sched_debug.cpu.clock_task.stddev
-68.3% 31.00 ± 10% sched_debug.cpu.nr_uninterruptible.max
+49.7% 2623 ± 28% sched_debug.cpu.sched_goidle.max
+47.1% 51.50 ± 15% sched_debug.cpu.sched_goidle.min
+50.1% 431.68 ± 16% sched_debug.cpu.sched_goidle.stddev
-1.0% 11.18 perf-stat.i.MPKI
-17.1% 1.386e+10 ± 6% perf-stat.i.branch-instructions
-13.7% 5211681 ± 2% perf-stat.i.branch-misses
-17.2% 6.556e+08 ± 6% perf-stat.i.cache-misses
-17.2% 6.731e+08 ± 6% perf-stat.i.cache-references
-7.2% 2.244e+11 ± 4% perf-stat.i.cpu-cycles
-11.5% 151.59 ± 3% perf-stat.i.cpu-migrations
-17.1% 1.647e+10 ± 6% perf-stat.i.dTLB-loads
-17.0% 6.605e+09 ± 6% perf-stat.i.dTLB-stores
-8.8 79.50 ± 5% perf-stat.i.iTLB-load-miss-rate%
+86.6% 444313 ± 23% perf-stat.i.iTLB-loads
-17.1% 6.08e+10 ± 6% perf-stat.i.instructions
-20.1% 80899 ± 5% perf-stat.i.instructions-per-iTLB-miss
-8.4% 0.28 ± 3% perf-stat.i.ipc
-7.2% 2.34 ± 4% perf-stat.i.metric.GHz
-17.0% 398.76 ± 6% perf-stat.i.metric.M/sec
-9.4% 5234 perf-stat.i.minor-faults
+5.9 37.27 ± 5% perf-stat.i.node-load-miss-rate%
+8.6% 436835 ± 2% perf-stat.i.node-load-misses
-20.5% 1105497 ± 7% perf-stat.i.node-loads
+16.6 28.86 ± 12% perf-stat.i.node-store-miss-rate%
+115.7% 1.674e+08 ± 2% perf-stat.i.node-store-misses
-31.8% 4.839e+08 ± 9% perf-stat.i.node-stores
-9.4% 5235 perf-stat.i.page-faults
+12.2% 3.72 ± 4% perf-stat.overall.cpi
+12.4% 344.68 ± 4% perf-stat.overall.cycles-between-cache-misses
-10.6 70.04 ± 5% perf-stat.overall.iTLB-load-miss-rate%
-19.0% 59242 ± 6% perf-stat.overall.instructions-per-iTLB-miss
-10.8% 0.27 ± 4% perf-stat.overall.ipc
+6.2 28.43 ± 6% perf-stat.overall.node-load-miss-rate%
+15.9 25.90 ± 8% perf-stat.overall.node-store-miss-rate%
-16.9% 1.372e+10 ± 5% perf-stat.ps.branch-instructions
-13.2% 5169398 ± 2% perf-stat.ps.branch-misses
-17.1% 6.489e+08 ± 5% perf-stat.ps.cache-misses
-17.0% 6.662e+08 ± 5% perf-stat.ps.cache-references
-6.8% 2.233e+11 ± 4% perf-stat.ps.cpu-cycles
-11.2% 150.27 ± 3% perf-stat.ps.cpu-migrations
-16.9% 1.63e+10 ± 5% perf-stat.ps.dTLB-loads
-16.9% 6.536e+09 ± 5% perf-stat.ps.dTLB-stores
+84.6% 441573 ± 21% perf-stat.ps.iTLB-loads
-16.9% 6.017e+10 ± 5% perf-stat.ps.instructions
-9.3% 5309 ± 2% perf-stat.ps.minor-faults
+9.0% 440545 ± 3% perf-stat.ps.node-load-misses
-21.9% 1114896 ± 8% perf-stat.ps.node-loads
+114.7% 1.662e+08 ± 3% perf-stat.ps.node-store-misses
-31.6% 4.784e+08 ± 8% perf-stat.ps.node-stores
-9.3% 5309 ± 2% perf-stat.ps.page-faults
+50.6% 94283 ± 24% softirqs.CPU1.TIMER
+20.2% 72598 ± 5% softirqs.CPU10.TIMER
+34.0% 82736 ± 20% softirqs.CPU11.TIMER
+19.5% 73373 ± 4% softirqs.CPU12.TIMER
+19.9% 73315 ± 5% softirqs.CPU13.TIMER
+18.8% 72243 ± 5% softirqs.CPU14.TIMER
+18.4% 72036 ± 4% softirqs.CPU15.TIMER
+18.6% 72067 ± 5% softirqs.CPU16.TIMER
+93.7% 7574 ± 27% softirqs.CPU17.SCHED
+19.0% 73048 ± 5% softirqs.CPU17.TIMER
+18.0% 72401 ± 4% softirqs.CPU18.TIMER
+19.5% 72656 ± 5% softirqs.CPU19.TIMER
+116.3% 7564 ± 23% softirqs.CPU20.SCHED
+17.7% 72350 ± 5% softirqs.CPU21.TIMER
+17.5% 71676 ± 4% softirqs.CPU22.TIMER
+20.1% 73009 ± 5% softirqs.CPU23.TIMER
+18.7% 74038 ± 7% softirqs.CPU24.TIMER
+18.9% 73919 ± 7% softirqs.CPU25.TIMER
+18.6% 73534 ± 7% softirqs.CPU26.TIMER
+19.8% 73915 ± 8% softirqs.CPU27.TIMER
+18.3% 74562 ± 6% softirqs.CPU28.TIMER
+20.0% 74390 ± 8% softirqs.CPU29.TIMER
+16.7% 72335 ± 5% softirqs.CPU3.TIMER
+19.0% 73643 ± 7% softirqs.CPU30.TIMER
+19.3% 74059 ± 7% softirqs.CPU31.TIMER
+18.7% 73511 ± 7% softirqs.CPU32.TIMER
+77.5% 6802 ± 30% softirqs.CPU33.SCHED
+19.1% 73752 ± 7% softirqs.CPU33.TIMER
+59.8% 6705 ± 32% softirqs.CPU34.SCHED
+19.8% 74096 ± 8% softirqs.CPU34.TIMER
+19.1% 73620 ± 6% softirqs.CPU35.TIMER
+17.9% 73195 ± 7% softirqs.CPU36.TIMER
+16.4% 73528 ± 7% softirqs.CPU37.TIMER
+17.2% 73282 ± 7% softirqs.CPU38.TIMER
+18.1% 73113 ± 7% softirqs.CPU39.TIMER
+16.7% 72406 ± 5% softirqs.CPU4.TIMER
+17.2% 72780 ± 7% softirqs.CPU40.TIMER
+18.4% 73646 ± 7% softirqs.CPU41.TIMER
+18.7% 73322 ± 6% softirqs.CPU42.TIMER
+18.4% 72707 ± 6% softirqs.CPU43.TIMER
+18.8% 73569 ± 6% softirqs.CPU44.TIMER
+21.6% 75363 ± 9% softirqs.CPU45.TIMER
+17.9% 72813 ± 7% softirqs.CPU46.TIMER
+17.4% 73172 ± 7% softirqs.CPU47.TIMER
+15.3% 72836 ± 5% softirqs.CPU48.TIMER
+82.0% 7962 ± 22% softirqs.CPU49.SCHED
+22.7% 75220 ± 6% softirqs.CPU49.TIMER
+15.1% 72854 ± 5% softirqs.CPU5.TIMER
+13.6% 72161 ± 5% softirqs.CPU50.TIMER
+19.8% 72425 ± 5% softirqs.CPU51.TIMER
+15.8% 72299 ± 5% softirqs.CPU52.TIMER
+104.5% 7627 ± 27% softirqs.CPU53.SCHED
+20.6% 72704 ± 6% softirqs.CPU53.TIMER
+20.2% 72596 ± 5% softirqs.CPU54.TIMER
+21.3% 72641 ± 5% softirqs.CPU55.TIMER
+18.1% 72593 ± 5% softirqs.CPU56.TIMER
+36.9% 82133 ± 22% softirqs.CPU57.TIMER
+21.4% 72652 ± 3% softirqs.CPU58.TIMER
+20.3% 72740 ± 5% softirqs.CPU59.TIMER
+16.6% 72976 ± 5% softirqs.CPU6.TIMER
+18.9% 72092 ± 5% softirqs.CPU60.TIMER
+18.6% 72194 ± 5% softirqs.CPU61.TIMER
+20.8% 72913 ± 3% softirqs.CPU62.TIMER
+19.2% 71519 ± 4% softirqs.CPU63.TIMER
+19.2% 72590 ± 5% softirqs.CPU64.TIMER
+20.1% 72677 ± 5% softirqs.CPU65.TIMER
+18.4% 72137 ± 5% softirqs.CPU66.TIMER
+19.5% 72100 ± 5% softirqs.CPU67.TIMER
+16.6% 71744 ± 5% softirqs.CPU68.TIMER
+13.9% 72028 ± 5% softirqs.CPU69.TIMER
+19.1% 72437 ± 4% softirqs.CPU7.TIMER
+18.2% 71500 ± 4% softirqs.CPU70.TIMER
+17.8% 72482 ± 6% softirqs.CPU71.TIMER
+18.3% 73620 ± 7% softirqs.CPU72.TIMER
+17.3% 72639 ± 7% softirqs.CPU73.TIMER
+17.2% 72205 ± 7% softirqs.CPU74.TIMER
+17.6% 72297 ± 7% softirqs.CPU75.TIMER
+18.8% 72709 ± 7% softirqs.CPU76.TIMER
+18.0% 73396 ± 8% softirqs.CPU77.TIMER
+17.1% 72325 ± 7% softirqs.CPU78.TIMER
+15.6% 72388 ± 7% softirqs.CPU79.TIMER
+17.3% 72049 ± 5% softirqs.CPU8.TIMER
+17.2% 72379 ± 7% softirqs.CPU80.TIMER
+14.7% 72714 ± 7% softirqs.CPU81.TIMER
+17.1% 72734 ± 7% softirqs.CPU82.TIMER
+17.2% 72868 ± 7% softirqs.CPU83.TIMER
+18.5% 72962 ± 7% softirqs.CPU84.TIMER
+17.6% 72723 ± 7% softirqs.CPU85.TIMER
+17.1% 72417 ± 7% softirqs.CPU86.TIMER
+16.4% 72822 ± 7% softirqs.CPU87.TIMER
+18.2% 72684 ± 6% softirqs.CPU88.TIMER
+16.6% 72437 ± 7% softirqs.CPU89.TIMER
+19.1% 73092 ± 5% softirqs.CPU9.TIMER
+18.1% 72423 ± 7% softirqs.CPU90.TIMER
+18.1% 72309 ± 7% softirqs.CPU91.TIMER
+17.0% 71906 ± 7% softirqs.CPU92.TIMER
+17.1% 72076 ± 7% softirqs.CPU93.TIMER
+16.4% 72071 ± 7% softirqs.CPU94.TIMER
+51.8% 603839 ± 19% softirqs.SCHED
+18.2% 7036827 ± 6% softirqs.TIMER
+25.5% 184.75 ± 12% interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
+6325.0% 64.25 ± 92% interrupts.60:PCI-MSI.31981593-edge.i40e-eth0-TxRx-24
+21.1% 422991 ± 6% interrupts.CPU0.LOC:Local_timer_interrupts
-31.3% 5230 ± 33% interrupts.CPU0.NMI:Non-maskable_interrupts
-31.3% 5230 ± 33% interrupts.CPU0.PMI:Performance_monitoring_interrupts
+21.2% 423344 ± 6% interrupts.CPU1.LOC:Local_timer_interrupts
+21.2% 423243 ± 6% interrupts.CPU10.LOC:Local_timer_interrupts
+16.6% 541.00 ± 16% interrupts.CPU11.CAL:Function_call_interrupts
+21.4% 423349 ± 6% interrupts.CPU11.LOC:Local_timer_interrupts
-24.5% 5849 ± 27% interrupts.CPU11.NMI:Non-maskable_interrupts
-24.5% 5849 ± 27% interrupts.CPU11.PMI:Performance_monitoring_interrupts
+34.0% 262.00 ± 22% interrupts.CPU11.RES:Rescheduling_interrupts
+21.0% 423121 ± 6% interrupts.CPU12.LOC:Local_timer_interrupts
-36.7% 4162 ± 33% interrupts.CPU12.NMI:Non-maskable_interrupts
-36.7% 4162 ± 33% interrupts.CPU12.PMI:Performance_monitoring_interrupts
+21.0% 423134 ± 6% interrupts.CPU13.LOC:Local_timer_interrupts
+68.4% 771.75 ± 60% interrupts.CPU14.CAL:Function_call_interrupts
+21.1% 423341 ± 6% interrupts.CPU14.LOC:Local_timer_interrupts
+21.5% 219.25 ± 5% interrupts.CPU14.RES:Rescheduling_interrupts
+21.1% 422914 ± 6% interrupts.CPU15.LOC:Local_timer_interrupts
+21.1% 423128 ± 6% interrupts.CPU16.LOC:Local_timer_interrupts
+21.1% 423387 ± 6% interrupts.CPU17.LOC:Local_timer_interrupts
+21.1% 423252 ± 6% interrupts.CPU18.LOC:Local_timer_interrupts
+21.0% 423069 ± 6% interrupts.CPU19.LOC:Local_timer_interrupts
+21.3% 423252 ± 6% interrupts.CPU2.LOC:Local_timer_interrupts
+21.1% 423015 ± 6% interrupts.CPU20.LOC:Local_timer_interrupts
+21.1% 423268 ± 6% interrupts.CPU21.LOC:Local_timer_interrupts
+21.1% 423343 ± 6% interrupts.CPU22.LOC:Local_timer_interrupts
+21.2% 423279 ± 6% interrupts.CPU23.LOC:Local_timer_interrupts
64.25 ± 92% interrupts.CPU24.60:PCI-MSI.31981593-edge.i40e-eth0-TxRx-24
+21.0% 422848 ± 6% interrupts.CPU24.LOC:Local_timer_interrupts
+21.1% 423069 ± 6% interrupts.CPU25.LOC:Local_timer_interrupts
-65.5% 547.50 ± 3% interrupts.CPU26.CAL:Function_call_interrupts
+21.2% 423009 ± 6% interrupts.CPU26.LOC:Local_timer_interrupts
+21.1% 423073 ± 6% interrupts.CPU27.LOC:Local_timer_interrupts
+21.2% 422962 ± 6% interrupts.CPU28.LOC:Local_timer_interrupts
+21.1% 423047 ± 6% interrupts.CPU29.LOC:Local_timer_interrupts
+21.1% 423123 ± 6% interrupts.CPU3.LOC:Local_timer_interrupts
+21.1% 422972 ± 6% interrupts.CPU30.LOC:Local_timer_interrupts
+21.4% 222.25 ± 2% interrupts.CPU30.RES:Rescheduling_interrupts
+21.1% 423145 ± 6% interrupts.CPU31.LOC:Local_timer_interrupts
+21.3% 423017 ± 6% interrupts.CPU32.LOC:Local_timer_interrupts
+21.2% 423139 ± 6% interrupts.CPU33.LOC:Local_timer_interrupts
+21.2% 423088 ± 6% interrupts.CPU34.LOC:Local_timer_interrupts
+21.1% 422903 ± 6% interrupts.CPU35.LOC:Local_timer_interrupts
+21.1% 423034 ± 6% interrupts.CPU36.LOC:Local_timer_interrupts
+24.9% 214.25 ± 7% interrupts.CPU36.RES:Rescheduling_interrupts
+21.2% 423014 ± 6% interrupts.CPU37.LOC:Local_timer_interrupts
+21.1% 423030 ± 6% interrupts.CPU38.LOC:Local_timer_interrupts
+21.2% 422869 ± 6% interrupts.CPU39.LOC:Local_timer_interrupts
+29.8% 222.00 ± 15% interrupts.CPU39.RES:Rescheduling_interrupts
+21.1% 423056 ± 6% interrupts.CPU4.LOC:Local_timer_interrupts
-27.2% 5557 ± 20% interrupts.CPU4.NMI:Non-maskable_interrupts
-27.2% 5557 ± 20% interrupts.CPU4.PMI:Performance_monitoring_interrupts
+21.1% 423025 ± 6% interrupts.CPU40.LOC:Local_timer_interrupts
+13.0% 202.50 ± 7% interrupts.CPU40.RES:Rescheduling_interrupts
+21.1% 423008 ± 6% interrupts.CPU41.LOC:Local_timer_interrupts
+21.1% 422986 ± 6% interrupts.CPU42.LOC:Local_timer_interrupts
+21.1% 423033 ± 6% interrupts.CPU43.LOC:Local_timer_interrupts
+22.0% 202.50 ± 8% interrupts.CPU43.RES:Rescheduling_interrupts
+21.1% 423039 ± 6% interrupts.CPU44.LOC:Local_timer_interrupts
+21.1% 423126 ± 6% interrupts.CPU45.LOC:Local_timer_interrupts
+15.3% 190.25 ± 5% interrupts.CPU45.RES:Rescheduling_interrupts
+21.1% 423046 ± 6% interrupts.CPU46.LOC:Local_timer_interrupts
+21.8% 208.00 ± 6% interrupts.CPU46.RES:Rescheduling_interrupts
+21.1% 423026 ± 6% interrupts.CPU47.LOC:Local_timer_interrupts
+21.6% 212.50 ± 11% interrupts.CPU47.RES:Rescheduling_interrupts
+21.2% 423328 ± 6% interrupts.CPU48.LOC:Local_timer_interrupts
+21.1% 423220 ± 6% interrupts.CPU49.LOC:Local_timer_interrupts
+21.2% 423207 ± 6% interrupts.CPU5.LOC:Local_timer_interrupts
+21.2% 423232 ± 6% interrupts.CPU50.LOC:Local_timer_interrupts
+21.2% 423216 ± 6% interrupts.CPU51.LOC:Local_timer_interrupts
-47.4% 4004 ± 33% interrupts.CPU51.NMI:Non-maskable_interrupts
-47.4% 4004 ± 33% interrupts.CPU51.PMI:Performance_monitoring_interrupts
+21.0% 422768 ± 6% interrupts.CPU52.LOC:Local_timer_interrupts
-18.9% 6309 ± 5% interrupts.CPU52.NMI:Non-maskable_interrupts
-18.9% 6309 ± 5% interrupts.CPU52.PMI:Performance_monitoring_interrupts
+21.0% 422917 ± 6% interrupts.CPU53.LOC:Local_timer_interrupts
+21.1% 423194 ± 6% interrupts.CPU54.LOC:Local_timer_interrupts
-18.7% 6350 ± 7% interrupts.CPU54.NMI:Non-maskable_interrupts
-18.7% 6350 ± 7% interrupts.CPU54.PMI:Performance_monitoring_interrupts
+21.1% 423236 ± 6% interrupts.CPU55.LOC:Local_timer_interrupts
-37.1% 4751 ± 32% interrupts.CPU55.NMI:Non-maskable_interrupts
-37.1% 4751 ± 32% interrupts.CPU55.PMI:Performance_monitoring_interrupts
+21.2% 423209 ± 6% interrupts.CPU56.LOC:Local_timer_interrupts
+21.1% 423164 ± 6% interrupts.CPU57.LOC:Local_timer_interrupts
+21.1% 423127 ± 6% interrupts.CPU58.LOC:Local_timer_interrupts
+21.2% 423350 ± 6% interrupts.CPU59.LOC:Local_timer_interrupts
+21.2% 423191 ± 6% interrupts.CPU6.LOC:Local_timer_interrupts
+21.0% 423031 ± 6% interrupts.CPU60.LOC:Local_timer_interrupts
+21.1% 423202 ± 6% interrupts.CPU61.LOC:Local_timer_interrupts
+12.5% 534.25 ± 7% interrupts.CPU62.CAL:Function_call_interrupts
+21.0% 423065 ± 6% interrupts.CPU62.LOC:Local_timer_interrupts
-29.1% 5356 ± 24% interrupts.CPU62.NMI:Non-maskable_interrupts
-29.1% 5356 ± 24% interrupts.CPU62.PMI:Performance_monitoring_interrupts
+21.5% 423010 ± 6% interrupts.CPU63.LOC:Local_timer_interrupts
+21.2% 423400 ± 6% interrupts.CPU64.LOC:Local_timer_interrupts
+21.1% 422998 ± 6% interrupts.CPU65.LOC:Local_timer_interrupts
-27.6% 5501 ± 17% interrupts.CPU65.NMI:Non-maskable_interrupts
-27.6% 5501 ± 17% interrupts.CPU65.PMI:Performance_monitoring_interrupts
+21.1% 423214 ± 6% interrupts.CPU66.LOC:Local_timer_interrupts
-30.2% 5272 ± 23% interrupts.CPU66.NMI:Non-maskable_interrupts
-30.2% 5272 ± 23% interrupts.CPU66.PMI:Performance_monitoring_interrupts
+21.1% 423298 ± 6% interrupts.CPU67.LOC:Local_timer_interrupts
-20.7% 5989 ± 7% interrupts.CPU67.NMI:Non-maskable_interrupts
-20.7% 5989 ± 7% interrupts.CPU67.PMI:Performance_monitoring_interrupts
+21.0% 423040 ± 6% interrupts.CPU68.LOC:Local_timer_interrupts
-28.7% 5466 ± 29% interrupts.CPU68.NMI:Non-maskable_interrupts
-28.7% 5466 ± 29% interrupts.CPU68.PMI:Performance_monitoring_interrupts
+21.2% 423047 ± 6% interrupts.CPU69.LOC:Local_timer_interrupts
+25.5% 184.75 ± 12% interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
+21.1% 423165 ± 6% interrupts.CPU7.LOC:Local_timer_interrupts
+21.3% 423169 ± 6% interrupts.CPU70.LOC:Local_timer_interrupts
+21.1% 423199 ± 6% interrupts.CPU71.LOC:Local_timer_interrupts
-28.9% 5381 ± 23% interrupts.CPU71.NMI:Non-maskable_interrupts
-28.9% 5381 ± 23% interrupts.CPU71.PMI:Performance_monitoring_interrupts
+21.2% 422912 ± 6% interrupts.CPU72.LOC:Local_timer_interrupts
+21.1% 422921 ± 6% interrupts.CPU73.LOC:Local_timer_interrupts
+21.2% 423037 ± 6% interrupts.CPU74.LOC:Local_timer_interrupts
+21.1% 422808 ± 6% interrupts.CPU75.LOC:Local_timer_interrupts
+21.1% 422847 ± 6% interrupts.CPU76.LOC:Local_timer_interrupts
+21.2% 422957 ± 6% interrupts.CPU77.LOC:Local_timer_interrupts
+21.2% 422931 ± 6% interrupts.CPU78.LOC:Local_timer_interrupts
+21.1% 422951 ± 6% interrupts.CPU79.LOC:Local_timer_interrupts
-18.0% 152.75 ± 11% interrupts.CPU79.RES:Rescheduling_interrupts
+21.0% 423133 ± 6% interrupts.CPU8.LOC:Local_timer_interrupts
-13.9% 6492 ± 10% interrupts.CPU8.NMI:Non-maskable_interrupts
-13.9% 6492 ± 10% interrupts.CPU8.PMI:Performance_monitoring_interrupts
+25.3% 237.75 ± 11% interrupts.CPU8.RES:Rescheduling_interrupts
+21.2% 422917 ± 6% interrupts.CPU80.LOC:Local_timer_interrupts
+21.2% 422898 ± 6% interrupts.CPU81.LOC:Local_timer_interrupts
+21.2% 422981 ± 6% interrupts.CPU82.LOC:Local_timer_interrupts
+21.1% 422783 ± 6% interrupts.CPU83.LOC:Local_timer_interrupts
+21.2% 422860 ± 6% interrupts.CPU84.LOC:Local_timer_interrupts
+21.3% 422902 ± 6% interrupts.CPU85.LOC:Local_timer_interrupts
+21.1% 422949 ± 6% interrupts.CPU86.LOC:Local_timer_interrupts
+21.2% 422939 ± 6% interrupts.CPU87.LOC:Local_timer_interrupts
+21.2% 422898 ± 6% interrupts.CPU88.LOC:Local_timer_interrupts
+21.2% 422861 ± 6% interrupts.CPU89.LOC:Local_timer_interrupts
+21.3% 423363 ± 6% interrupts.CPU9.LOC:Local_timer_interrupts
+21.1% 422939 ± 6% interrupts.CPU90.LOC:Local_timer_interrupts
+21.2% 422872 ± 6% interrupts.CPU91.LOC:Local_timer_interrupts
+21.0% 422821 ± 6% interrupts.CPU92.LOC:Local_timer_interrupts
+21.2% 422932 ± 6% interrupts.CPU93.LOC:Local_timer_interrupts
+21.1% 422977 ± 6% interrupts.CPU94.LOC:Local_timer_interrupts
+21.1% 423128 ± 6% interrupts.CPU95.LOC:Local_timer_interrupts
+21.1% 40614868 ± 6% interrupts.LOC:Local_timer_interrupts
-7.3% 617233 ± 4% interrupts.NMI:Non-maskable_interrupts
-7.3% 617233 ± 4% interrupts.PMI:Performance_monitoring_interrupts
have been estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
configuration may affect actual performance.

Attachments:

(No filename) (99.09 kB)
config-5.8.0-rc1-00023-g0b9730e694e8c (209.49 kB)
job-script (7.52 kB)
job.yaml (5.17 kB)
reproduce (8.58 kB)
Download all attachments

2020-07-02 09:29:01

by Dietmar Eggemann

[permalink] [raw]

Subject: Re: [PATCH] sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal

On 17/06/2020 16:52, Peter Puhov wrote:
> On Wed, 17 Jun 2020 at 06:50, Valentin Schneider
> <[email protected]> wrote:
>>
>>
>> On 16/06/20 17:48, [email protected] wrote:
>>> From: Peter Puhov <[email protected]>
>>> We tested this patch with following benchmarks:
>>> perf bench -f simple sched pipe -l 4000000
>>> perf bench -f simple sched messaging -l 30000
>>> perf bench -f simple mem memset -s 3GB -l 15 -f default
>>> perf bench -f simple futex wake -s -t 640 -w 1
>>> sysbench cpu --threads=8 --cpu-max-prime=10000 run
>>> sysbench memory --memory-access-mode=rnd --threads=8 run
>>> sysbench threads --threads=8 run
>>> sysbench mutex --mutex-num=1 --threads=8 run
>>> hackbench --loops 20000
>>> hackbench --pipe --threads --loops 20000
>>> hackbench --pipe --threads --loops 20000 --datasize 4096
>>>
>>> and found some performance improvements in:
>>> sysbench threads
>>> sysbench mutex
>>> perf bench futex wake
>>> and no regressions in others.
>>>
>>
>> One nitpick for the results of those: condensing them in a table form would
>> make them more reader-friendly. Perhaps something like:
>>
>> | Benchmark | Metric | Lower is better? | BASELINE | SERIES | DELTA |
>> |------------------+----------+------------------+----------+--------+-------|
>> | Sysbench threads | # events | No | 45526 | 56567 | +24% |
>> | Sysbench mutex | ... | | | | |
>>
>> If you want to include more stats for each benchmark, you could have one table
>> per (e.g. see [1]) - it'd still be a more readable form (or so I believe).

Wouldn't Unix Bench's 'execl' and 'spawn' be the ultimate test cases
for those kind of changes?

I only see minor improvements with tip/sched/core as base on hikey620
(Arm64 octa-core).

base w/ patch
./Run spawn -c 8 -i 10 633.6 635.1

./Run execl -c 8 -i 10 1187.5 1190.7

At the end of find_idlest_group(), when comparing local and idlest, it
is explicitly mentioned that number of idle_cpus is used instead of
utilization.
The comparision between potential idle groups and local & idlest group
should probably follow the same rules.

I haven't tested it with

https://lkml.kernel.org/r/[email protected]

which might have an influence here too.

2020-07-02 13:30:15

by Mel Gorman

[permalink] [raw]

Subject: Re: [PATCH] sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal

On Thu, Jul 02, 2020 at 11:27:52AM +0200, Dietmar Eggemann wrote:
> On 17/06/2020 16:52, Peter Puhov wrote:
> > On Wed, 17 Jun 2020 at 06:50, Valentin Schneider
> > <[email protected]> wrote:
> >>
> >>
> >> On 16/06/20 17:48, [email protected] wrote:
> >>> From: Peter Puhov <[email protected]>
> >>> We tested this patch with following benchmarks:
> >>> perf bench -f simple sched pipe -l 4000000
> >>> perf bench -f simple sched messaging -l 30000
> >>> perf bench -f simple mem memset -s 3GB -l 15 -f default
> >>> perf bench -f simple futex wake -s -t 640 -w 1
> >>> sysbench cpu --threads=8 --cpu-max-prime=10000 run
> >>> sysbench memory --memory-access-mode=rnd --threads=8 run
> >>> sysbench threads --threads=8 run
> >>> sysbench mutex --mutex-num=1 --threads=8 run
> >>> hackbench --loops 20000
> >>> hackbench --pipe --threads --loops 20000
> >>> hackbench --pipe --threads --loops 20000 --datasize 4096
> >>>
> >>> and found some performance improvements in:
> >>> sysbench threads
> >>> sysbench mutex
> >>> perf bench futex wake
> >>> and no regressions in others.
> >>>
> >>
> >> One nitpick for the results of those: condensing them in a table form would
> >> make them more reader-friendly. Perhaps something like:
> >>
> >> | Benchmark | Metric | Lower is better? | BASELINE | SERIES | DELTA |
> >> |------------------+----------+------------------+----------+--------+-------|
> >> | Sysbench threads | # events | No | 45526 | 56567 | +24% |
> >> | Sysbench mutex | ... | | | | |
> >>
> >> If you want to include more stats for each benchmark, you could have one table
> >> per (e.g. see [1]) - it'd still be a more readable form (or so I believe).
>
> Wouldn't Unix Bench's 'execl' and 'spawn' be the ultimate test cases
> for those kind of changes?
>
> I only see minor improvements with tip/sched/core as base on hikey620
> (Arm64 octa-core).
>
> base w/ patch
> ./Run spawn -c 8 -i 10 633.6 635.1
>
> ./Run execl -c 8 -i 10 1187.5 1190.7
>
>
> At the end of find_idlest_group(), when comparing local and idlest, it
> is explicitly mentioned that number of idle_cpus is used instead of
> utilization.
> The comparision between potential idle groups and local & idlest group
> should probably follow the same rules.
>

There is the secondary hazard that what update_sd_pick_busiest returns
is checked later by find_busiest_group when considering the imbalance
between NUMA nodes. This particular patch splits basic communicating tasks
cross-node again at fork time so cross node communication is reintroduced
(same applies if sum_nr_running is used instead of group_util).

--
Mel Gorman
SUSE Labs

2020-07-02 13:48:25

by Vincent Guittot

[permalink] [raw]

Subject: Re: [PATCH] sched/fair: update_pick_idlest() Select group with lowest group_util when idle_cpus are equal

On Thu, 2 Jul 2020 at 15:29, Mel Gorman <[email protected]> wrote:
>
> On Thu, Jul 02, 2020 at 11:27:52AM +0200, Dietmar Eggemann wrote:
> > On 17/06/2020 16:52, Peter Puhov wrote:
> > > On Wed, 17 Jun 2020 at 06:50, Valentin Schneider
> > > <[email protected]> wrote:
> > >>
> > >>
> > >> On 16/06/20 17:48, [email protected] wrote:
> > >>> From: Peter Puhov <[email protected]>
> > >>> We tested this patch with following benchmarks:
> > >>> perf bench -f simple sched pipe -l 4000000
> > >>> perf bench -f simple sched messaging -l 30000
> > >>> perf bench -f simple mem memset -s 3GB -l 15 -f default
> > >>> perf bench -f simple futex wake -s -t 640 -w 1
> > >>> sysbench cpu --threads=8 --cpu-max-prime=10000 run
> > >>> sysbench memory --memory-access-mode=rnd --threads=8 run
> > >>> sysbench threads --threads=8 run
> > >>> sysbench mutex --mutex-num=1 --threads=8 run
> > >>> hackbench --loops 20000
> > >>> hackbench --pipe --threads --loops 20000
> > >>> hackbench --pipe --threads --loops 20000 --datasize 4096
> > >>>
> > >>> and found some performance improvements in:
> > >>> sysbench threads
> > >>> sysbench mutex
> > >>> perf bench futex wake
> > >>> and no regressions in others.
> > >>>
> > >>
> > >> One nitpick for the results of those: condensing them in a table form would
> > >> make them more reader-friendly. Perhaps something like:
> > >>
> > >> | Benchmark | Metric | Lower is better? | BASELINE | SERIES | DELTA |
> > >> |------------------+----------+------------------+----------+--------+-------|
> > >> | Sysbench threads | # events | No | 45526 | 56567 | +24% |
> > >> | Sysbench mutex | ... | | | | |
> > >>
> > >> If you want to include more stats for each benchmark, you could have one table
> > >> per (e.g. see [1]) - it'd still be a more readable form (or so I believe).
> >
> > Wouldn't Unix Bench's 'execl' and 'spawn' be the ultimate test cases
> > for those kind of changes?
> >
> > I only see minor improvements with tip/sched/core as base on hikey620
> > (Arm64 octa-core).
> >
> > base w/ patch
> > ./Run spawn -c 8 -i 10 633.6 635.1
> >
> > ./Run execl -c 8 -i 10 1187.5 1190.7
> >
> >
> > At the end of find_idlest_group(), when comparing local and idlest, it
> > is explicitly mentioned that number of idle_cpus is used instead of
> > utilization.
> > The comparision between potential idle groups and local & idlest group
> > should probably follow the same rules.

Comparing the number of idle cpu is not enough in the case described
by Peter because the newly forked thread sleeps immediately and before
we select cpu for the next one. This is shown in the trace where the
same CPU7 is selected for all wakeup_new events.
That's why, looking at utilization when there is the same number of
CPU is a good way to see where the previous task was placed. Using
nr_running doesn't solve the problem because newly forked task is not
running and the cpu would not have been idle in this case and an idle
CPU would have been selected instead

> >
>
> There is the secondary hazard that what update_sd_pick_busiest returns
> is checked later by find_busiest_group when considering the imbalance
> between NUMA nodes. This particular patch splits basic communicating tasks
> cross-node again at fork time so cross node communication is reintroduced
> (same applies if sum_nr_running is used instead of group_util).

As long as there is an idle cpu in the node, new thread doesn't cross
node like previously. The only difference happens inside the node

>
> --
> Mel Gorman
> SUSE Labs