LinuxLists.cc - [PATCH v2] sched/fair: Age the average idle time

2021-06-15 11:18:00

Subject: [PATCH v2] sched/fair: Age the average idle time

From: Peter Zijlstra (Intel) <[email protected]>

This is a partial forward-port of Peter Ziljstra's work first posted
at https://lore.kernel.org/lkml/[email protected]/.
His Signed-off has been removed because it is modified but will be restored
if he says it's still ok.

Currently select_idle_cpu()'s proportional scheme uses the average idle
time *for when we are idle*, that is temporally challenged. When a CPU
is not at all idle, we'll happily continue using whatever value we did
see when the CPU goes idle. To fix this, introduce a separate average
idle and age it (the existing value still makes sense for things like
new-idle balancing, which happens when we do go idle).

The overall goal is to not spend more time scanning for idle CPUs than
we're idle for. Otherwise we're inhibiting work. This means that we need to
consider the cost over all the wake-ups between consecutive idle periods.
To track this, the scan cost is subtracted from the estimated average
idle time.

The impact of this patch is related to workloads that have domains that
are fully busy or overloaded. Without the patch, the scan depth may be
too high because a CPU is not reaching idle.

Due to the nature of the patch, this is a regression magnet. It
potentially wins when domains are almost fully busy or overloaded --
at that point searches are likely to fail but idle is not being aged
as CPUs are active so search depth is too large and useless. It will
potentially show regressions when there are idle CPUs and a deep search is
beneficial. This tbench result on a 2-socket broadwell machine partially
illustates the problem

5.13.0-rc2 5.13.0-rc2
vanilla sched-avgidle-v1r5
Hmean 1 445.02 ( 0.00%) 451.36 * 1.42%*
Hmean 2 830.69 ( 0.00%) 846.03 * 1.85%*
Hmean 4 1350.80 ( 0.00%) 1505.56 * 11.46%*
Hmean 8 2888.88 ( 0.00%) 2586.40 * -10.47%*
Hmean 16 5248.18 ( 0.00%) 5305.26 * 1.09%*
Hmean 32 8914.03 ( 0.00%) 9191.35 * 3.11%*
Hmean 64 10663.10 ( 0.00%) 10192.65 * -4.41%*
Hmean 128 18043.89 ( 0.00%) 18478.92 * 2.41%*
Hmean 256 16530.89 ( 0.00%) 17637.16 * 6.69%*
Hmean 320 16451.13 ( 0.00%) 17270.97 * 4.98%*

Note that 8 was a regression point where a deeper search would have helped
but it gains for high thread counts when searches are useless. Hackbench
is a more extreme example although not perfect as the tasks idle rapidly

hackbench-process-pipes
5.13.0-rc2 5.13.0-rc2
vanilla sched-avgidle-v1r5
Amean 1 0.3950 ( 0.00%) 0.3887 ( 1.60%)
Amean 4 0.9450 ( 0.00%) 0.9677 ( -2.40%)
Amean 7 1.4737 ( 0.00%) 1.4890 ( -1.04%)
Amean 12 2.3507 ( 0.00%) 2.3360 * 0.62%*
Amean 21 4.0807 ( 0.00%) 4.0993 * -0.46%*
Amean 30 5.6820 ( 0.00%) 5.7510 * -1.21%*
Amean 48 8.7913 ( 0.00%) 8.7383 ( 0.60%)
Amean 79 14.3880 ( 0.00%) 13.9343 * 3.15%*
Amean 110 21.2233 ( 0.00%) 19.4263 * 8.47%*
Amean 141 28.2930 ( 0.00%) 25.1003 * 11.28%*
Amean 172 34.7570 ( 0.00%) 30.7527 * 11.52%*
Amean 203 41.0083 ( 0.00%) 36.4267 * 11.17%*
Amean 234 47.7133 ( 0.00%) 42.0623 * 11.84%*
Amean 265 53.0353 ( 0.00%) 47.7720 * 9.92%*
Amean 296 60.0170 ( 0.00%) 53.4273 * 10.98%*
Stddev 1 0.0052 ( 0.00%) 0.0025 ( 51.57%)
Stddev 4 0.0357 ( 0.00%) 0.0370 ( -3.75%)
Stddev 7 0.0190 ( 0.00%) 0.0298 ( -56.64%)
Stddev 12 0.0064 ( 0.00%) 0.0095 ( -48.38%)
Stddev 21 0.0065 ( 0.00%) 0.0097 ( -49.28%)
Stddev 30 0.0185 ( 0.00%) 0.0295 ( -59.54%)
Stddev 48 0.0559 ( 0.00%) 0.0168 ( 69.92%)
Stddev 79 0.1559 ( 0.00%) 0.0278 ( 82.17%)
Stddev 110 1.1728 ( 0.00%) 0.0532 ( 95.47%)
Stddev 141 0.7867 ( 0.00%) 0.0968 ( 87.69%)
Stddev 172 1.0255 ( 0.00%) 0.0420 ( 95.91%)
Stddev 203 0.8106 ( 0.00%) 0.1384 ( 82.92%)
Stddev 234 1.1949 ( 0.00%) 0.1328 ( 88.89%)
Stddev 265 0.9231 ( 0.00%) 0.0820 ( 91.11%)
Stddev 296 1.0456 ( 0.00%) 0.1327 ( 87.31%)

Again, higher thread counts benefit and the standard deviation
shows that results are also a lot more stable when the idle
time is aged.

The patch potentially matters when a socket was multiple LLCs as the
maximum search depth is lower. However, some of the test results were
suspiciously good (e.g. specjbb2005 gaining 50% on a Zen1 machine) and
other results were not dramatically different to other mcahines.

Given the nature of the patch, Peter's full series is not being forward
ported as each part should stand on its own. Preferably they would be
merged at different times to reduce the risk of false bisections.

Signed-off-by: Mel Gorman <[email protected]>
---
Changelog since v1
o No major change, rebase to 5.13-rc5 and retest, still passed

kernel/sched/core.c | 5 +++++
kernel/sched/fair.c | 25 +++++++++++++++++++++----
kernel/sched/sched.h | 3 +++
3 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5226cc26a095..6a3fdb9f4380 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2979,6 +2979,9 @@ static void ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags,
if (rq->avg_idle > max)
rq->avg_idle = max;

+ rq->wake_stamp = jiffies;
+ rq->wake_avg_idle = rq->avg_idle / 2;
+
rq->idle_stamp = 0;
}
#endif
@@ -8215,6 +8218,8 @@ void __init sched_init(void)
rq->online = 0;
rq->idle_stamp = 0;
rq->avg_idle = 2*sysctl_sched_migration_cost;
+ rq->wake_stamp = jiffies;
+ rq->wake_avg_idle = rq->avg_idle;
rq->max_idle_balance_cost = sysctl_sched_migration_cost;

INIT_LIST_HEAD(&rq->cfs_tasks);
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3248e24a90b0..cc7d1144a356 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6172,9 +6172,10 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
{
struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask);
int i, cpu, idle_cpu = -1, nr = INT_MAX;
+ struct rq *this_rq = this_rq();
int this = smp_processor_id();
struct sched_domain *this_sd;
- u64 time;
+ u64 time = 0;

this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc));
if (!this_sd)
@@ -6184,12 +6185,21 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool

if (sched_feat(SIS_PROP) && !has_idle_core) {
u64 avg_cost, avg_idle, span_avg;
+ unsigned long now = jiffies;

/*
- * Due to large variance we need a large fuzz factor;
- * hackbench in particularly is sensitive here.
+ * If we're busy, the assumption that the last idle period
+ * predicts the future is flawed; age away the remaining
+ * predicted idle time.
*/
- avg_idle = this_rq()->avg_idle / 512;
+ if (unlikely(this_rq->wake_stamp < now)) {
+ while (this_rq->wake_stamp < now && this_rq->wake_avg_idle) {
+ this_rq->wake_stamp++;
+ this_rq->wake_avg_idle >>= 1;
+ }
+ }
+
+ avg_idle = this_rq->wake_avg_idle;
avg_cost = this_sd->avg_scan_cost + 1;

span_avg = sd->span_weight * avg_idle;
@@ -6221,6 +6231,13 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool

if (sched_feat(SIS_PROP) && !has_idle_core) {
time = cpu_clock(this) - time;
+
+ /*
+ * Account for the scan cost of wakeups against the average
+ * idle time.
+ */
+ this_rq->wake_avg_idle -= min(this_rq->wake_avg_idle, time);
+
update_avg(&this_sd->avg_scan_cost, time);
}

diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a189bec13729..7bc20e5541cf 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1017,6 +1017,9 @@ struct rq {
u64 idle_stamp;
u64 avg_idle;

+ unsigned long wake_stamp;
+ u64 wake_avg_idle;
+
/* This is used to determine avg_idle's max value */
u64 max_idle_balance_cost;

2021-06-16 09:04:11

by Mel Gorman

[permalink] [raw]

Subject: Re: [PATCH v2] sched/fair: Age the average idle time

On Tue, Jun 15, 2021 at 10:42:28PM +0200, Peter Zijlstra wrote:
> On Tue, Jun 15, 2021 at 12:16:11PM +0100, Mel Gorman wrote:
> > From: Peter Zijlstra (Intel) <[email protected]>
> >
> > This is a partial forward-port of Peter Ziljstra's work first posted
> > at https://lore.kernel.org/lkml/[email protected]/.
>
> It's patches 2 and 3 together, right?
>

Patches 2, 3, 9 and 10. I saw limited value to preserving the feature
flag. Some of the series has since been obsoleted. The main patch of
interest that was dropped was patch 1 because the results were somewhat
inconclusive but leaning towards being an overall loss.

> > His Signed-off has been removed because it is modified but will be restored
> > if he says it's still ok.
>
> I suppose the SoB will auto-magically re-appear if I apply it :-)
>

Yep, it would and it would indicate that you didn't object to the copying
at least :P

> > The patch potentially matters when a socket was multiple LLCs as the
> > maximum search depth is lower. However, some of the test results were
> > suspiciously good (e.g. specjbb2005 gaining 50% on a Zen1 machine) and
> > other results were not dramatically different to other mcahines.
> >
> > Given the nature of the patch, Peter's full series is not being forward
> > ported as each part should stand on its own. Preferably they would be
> > merged at different times to reduce the risk of false bisections.
>
> I'm tempted to give it a go.. anyone object?

Thanks, so far no serious objection :)

The latest results as I see them have been copied to
https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/dashboard.html
They will move from here if the patch is accepted to 5-assembly replacing
3-perf-test. This naming is part of my workflow for evaluating topic
branches separetly and then putting them together for another round
of testing.

NAS shows small differences but NAS would see limited impact from the
patch. Specjbb shows small losses and some minor gains which is unfortunate
but the workload tends to see small gains and losses all the time.
redis is a mixed bag but has some wins. hackbench is the main benefit
because it's wakeup intensive and tends to overload machines where deep
searches hurt.

There are other results in there if you feel like digging around
such as sched-core tested with no processes getting tagged with prctl
https://beta.suse.com/private/mgorman/melt/v5.13-rc5/5-assembly/sched/sched-schedcore-v1r2/html/dashboard.html

--
Mel Gorman
SUSE Labs

2021-06-17 07:45:37

by Mel Gorman

[permalink] [raw]

Subject: Re: [PATCH v2] sched/fair: Age the average idle time

On Wed, Jun 16, 2021 at 05:52:25PM +0200, Vincent Guittot wrote:
> On Tue, 15 Jun 2021 at 22:43, Peter Zijlstra <[email protected]> wrote:
> >
> > On Tue, Jun 15, 2021 at 12:16:11PM +0100, Mel Gorman wrote:
> > > From: Peter Zijlstra (Intel) <[email protected]>
> > >
> > > This is a partial forward-port of Peter Ziljstra's work first posted
> > > at https://lore.kernel.org/lkml/[email protected]/.
> >
> > It's patches 2 and 3 together, right?
> >
> > > His Signed-off has been removed because it is modified but will be restored
> > > if he says it's still ok.
> >
> > I suppose the SoB will auto-magically re-appear if I apply it :-)
> >
> > > The patch potentially matters when a socket was multiple LLCs as the
> > > maximum search depth is lower. However, some of the test results were
> > > suspiciously good (e.g. specjbb2005 gaining 50% on a Zen1 machine) and
> > > other results were not dramatically different to other mcahines.
> > >
> > > Given the nature of the patch, Peter's full series is not being forward
> > > ported as each part should stand on its own. Preferably they would be
> > > merged at different times to reduce the risk of false bisections.
> >
> > I'm tempted to give it a go.. anyone object?
>
> Just finished running some tests on my large arm64 system.
> Tbench tests are a mixed between small gain and loss
>

Same for tbench on three x86 machines I reran tests for

https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/network-tbench/bing2/index.html#tbench4
Small gains and losses, gains at higher client counts where search depth
should be reduced

https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/network-tbench/hardy2/index.html#tbench4
Mostly gains, one counter-example at 4 clients

https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/network-tbench/marvin2/index.html#tbench4
Worst by far, 1 client took a major hit for unknown reasons, otherwise
mix of gains and losses. I'm not confident that the 1 client
results are meaningful because for this machine, there should
have been idle cores so the code the patch adjusts should not
even be executed.

> hackbench shows significant changes in both direction
> hackbench -g $group
>
> group tip/sched/core + this patch
> 1 13.358(+/- 1.82%) 12.850(+/- 2.21%) +4%
> 4 4.286(+/- 2.77%) 4.114(+/- 2.25%) +4%
> 16 3.175(+/- 0.55%) 3.559(+/- 0.43%) -12%
> 32 2.912(+/- 0.79%) 3.165(+/- 0.95%) -8%
> 64 2.859(+/- 1.12%) 2.937(+/- 0.91%) -3%
> 128 3.092(+/- 4.75%) 3.003(+/-5.18%) +3%
> 256 3.233(+/- 3.03%) 2.973(+/- 0.80%) +8%

Think this is processes and sockets. Of the hackbench results I had,
this one performed the worst

https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/scheduler-unbound/bing2/index.html#hackbench-process-sockets
Small gains and losses

https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/scheduler-unbound/hardy2/index.html#hackbench-process-sockets
Small gains and losses

https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/scheduler-unbound/marvin2/index.html#hackbench-process-sockets
Small gains and losses

One of the better results for hackbench was processes and pipes
https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/scheduler-unbound/bing2/index.html#hackbench-process-pipes
1-12% gains

For your arm machine, how many logical CPUs are online, what is the level
of SMT if any and is the machine NUMA?

Fundamentally though, as the changelog notes "due to the nature of the
patch, this is a regression magnet". There are going to be examples
where a deep search is better even if a machine is fully busy or
overloaded and examples where cutting off the search is better. I think
it's better to have an idle estimate that gets updated if CPUs are fully
busy even if it's not a universal win.

--
Mel Gorman
SUSE Labs

2021-06-17 11:00:33

by Mel Gorman

[permalink] [raw]

Subject: Re: [PATCH v2] sched/fair: Age the average idle time

On Thu, Jun 17, 2021 at 10:30:09AM +0200, Vincent Guittot wrote:
> > > > I'm tempted to give it a go.. anyone object?
> > >
> > > Just finished running some tests on my large arm64 system.
> > > Tbench tests are a mixed between small gain and loss
> > >
> >
> > Same for tbench on three x86 machines I reran tests for
> >
> > <SNIP>
> >
> > For your arm machine, how many logical CPUs are online, what is the level
> > of SMT if any and is the machine NUMA?
>
> It's a SMT4 x 28 cores x 2 NUMA nodes = 224 CPUs
>

Ok, SMT4 is what I was interested in. I suspected this was the case but
was not sure. I wondered about the possibility that SMT4 should be
accounted for in the scan depth.

> >
> > Fundamentally though, as the changelog notes "due to the nature of the
> > patch, this is a regression magnet". There are going to be examples
> > where a deep search is better even if a machine is fully busy or
> > overloaded and examples where cutting off the search is better. I think
> > it's better to have an idle estimate that gets updated if CPUs are fully
> > busy even if it's not a universal win.
>
> Although I agree that using a stall average idle time value of local
> is not good, I'm not sure this proposal is better. The main problem is
> that we use the avg_idle of the local CPU to estimate how many times
> we should loop and try to find another idle CPU. But there is no
> direct relation between both.

This is true. The idle time of the local CPU is used to estimate the
idle time of the domain which is inevitably going to be inaccurate but
tracking idle time for the domain will be cache write intensive and
potentially very expensive. I think this was discussed before but maybe
it is my imaginaction.

> Typically, a short average idle time on
> the local CPU doesn't mean that there are less idle CPUs and that's
> why we have a mix a gain and loss
>

Can you evaluate if scanning proportional to cores helps if applied on
top? The patch below is a bit of pick&mix and has only seen a basic build
test with a distro config. While I will queue this, I don't expect it to
have an impact on SMT2.

--8--
sched/fair: Make select_idle_cpu() proportional to cores

From: Peter Zijlstra (Intel) <[email protected]>

Instead of calculating how many (logical) CPUs to scan, compute how
many cores to scan.

This changes behaviour for anything !SMT2.

Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Signed-off-by: Mel Gorman <[email protected]>
---
kernel/sched/core.c | 17 ++++++++++++-----
kernel/sched/fair.c | 11 +++++++++--
kernel/sched/sched.h | 2 ++
3 files changed, 23 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 6a3fdb9f4380..1773e0707a5d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7846,11 +7846,18 @@ int sched_cpu_activate(unsigned int cpu)
balance_push_set(cpu, false);

#ifdef CONFIG_SCHED_SMT
- /*
- * When going up, increment the number of cores with SMT present.
- */
- if (cpumask_weight(cpu_smt_mask(cpu)) == 2)
- static_branch_inc_cpuslocked(&sched_smt_present);
+ do {
+ int weight = cpumask_weight(cpu_smt_mask(cpu));
+
+ if (weight > sched_smt_weight)
+ sched_smt_weight = weight;
+
+ /*
+ * When going up, increment the number of cores with SMT present.
+ */
+ if (cpumask_weight(cpu_smt_mask(cpu)) == 2)
+ static_branch_inc_cpuslocked(&sched_smt_present);
+ } while (0);
#endif
set_cpu_active(cpu, true);

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cc7d1144a356..4fc4e1f2eaae 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6037,6 +6037,8 @@ static inline int __select_idle_cpu(int cpu)
DEFINE_STATIC_KEY_FALSE(sched_smt_present);
EXPORT_SYMBOL_GPL(sched_smt_present);

+int __read_mostly sched_smt_weight = 1;
+
static inline void set_idle_cores(int cpu, int val)
{
struct sched_domain_shared *sds;
@@ -6151,6 +6153,8 @@ static inline bool test_idle_cores(int cpu, bool def)
return def;
}

+#define sched_smt_weight 1
+
static inline int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus, int *idle_cpu)
{
return __select_idle_cpu(core);
@@ -6163,6 +6167,8 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd

#endif /* CONFIG_SCHED_SMT */

+#define sis_min_cores 2
+
/*
* Scan the LLC domain for idle CPUs; this is dynamically regulated by
* comparing the average scan cost (tracked in sd->avg_scan_cost) against the
@@ -6203,11 +6209,12 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
avg_cost = this_sd->avg_scan_cost + 1;

span_avg = sd->span_weight * avg_idle;
- if (span_avg > 4*avg_cost)
+ if (span_avg > sis_min_cores * avg_cost)
nr = div_u64(span_avg, avg_cost);
else
- nr = 4;
+ nr = sis_min_cores;

+ nr *= sched_smt_weight;
time = cpu_clock(this);
}

diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 7bc20e5541cf..440a2bbc19d5 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1119,6 +1119,8 @@ static inline bool is_migration_disabled(struct task_struct *p)
#ifdef CONFIG_SCHED_SMT
extern void __update_idle_core(struct rq *rq);

+extern int sched_smt_weight;
+
static inline void update_idle_core(struct rq *rq)
{
if (static_branch_unlikely(&sched_smt_present))

2021-06-17 13:21:26

by Mel Gorman

[permalink] [raw]

Subject: Re: [PATCH v2] sched/fair: Age the average idle time

On Thu, Jun 17, 2021 at 12:02:56PM +0200, Vincent Guittot wrote:
> > > >
> > > > Fundamentally though, as the changelog notes "due to the nature of the
> > > > patch, this is a regression magnet". There are going to be examples
> > > > where a deep search is better even if a machine is fully busy or
> > > > overloaded and examples where cutting off the search is better. I think
> > > > it's better to have an idle estimate that gets updated if CPUs are fully
> > > > busy even if it's not a universal win.
> > >
> > > Although I agree that using a stall average idle time value of local
> > > is not good, I'm not sure this proposal is better. The main problem is
> > > that we use the avg_idle of the local CPU to estimate how many times
> > > we should loop and try to find another idle CPU. But there is no
> > > direct relation between both.
> >
> > This is true. The idle time of the local CPU is used to estimate the
> > idle time of the domain which is inevitably going to be inaccurate but
>
> I'm more and more convinced that using average idle time (of the
> local cpu or the full domain) is not the right metric. In
> select_idle_cpu(), we looks for an idle CPU but we don't care about
> how long it will be idle.

Can we predict that accurately? cpufreq for intel_pstate used to try
something like that but it was a bit fuzzy and I don't know if the
scheduler could do much better. There is some idle prediction stuff but
it's related to nohz which does not really help us if a machine is nearly
fully busy or overloaded.

I guess for tracking idle that revisiting
https://lore.kernel.org/lkml/[email protected]/
is an option now that the scan is somewhat unified. A two-pass scan
could be used to check potentially idle CPUs first and if there is
sufficient search depth left, scan other CPUs. There were some questions
on how accurate the idle mask was and how expensive it was to maintain.
Unfortunately, it would not help with scan depth calculations, it just
might reduce useless scanning.

Selecting based on avg idle time could be interesting but hazardous. If
for example, we prioritised selecting a CPU that is mostly idle, it'll
also pick CPUs that are potentially in a deep idle state incurring a
larger wakeup cost. Right now we are not much better because we just
select an idle CPU and hope for the best but always targetting the most
idle CPU could have problems. There would also be the cost of tracking
idle CPUs in priority order. It would eliminate the scan depth cost
calculations but the overall cost would be much worse.

Hence, I still think we can improve the scan depth costs in the short
term until a replacement is identified that works reasonably well.

> Even more, we can scan all CPUs whatever the
> avg idle time if there is a chance that there is an idle core.
>

That is an important, but separate topic. It's known that the idle core
detection can yield false positives. Putting core scanning under SIS_PROP
had mixed results when we last tried but things change. Again, it doesn't
help with scan depth calculations.

> > tracking idle time for the domain will be cache write intensive and
> > potentially very expensive. I think this was discussed before but maybe
> > it is my imaginaction.
> >
> > > Typically, a short average idle time on
> > > the local CPU doesn't mean that there are less idle CPUs and that's
> > > why we have a mix a gain and loss
> > >
> >
> > Can you evaluate if scanning proportional to cores helps if applied on
> > top? The patch below is a bit of pick&mix and has only seen a basic build
>
> I will queue it for some test later today
>

Thanks. The proposed patch since passed a build and boot test,
performance evaluation is under way but as it's x86 and SMT2, I'm mostly
just checking that it's neutral.

--
Mel Gorman
SUSE Labs

2021-06-17 16:21:58

by Mel Gorman

[permalink] [raw]

Subject: Re: [PATCH v2] sched/fair: Age the average idle time

On Thu, Jun 17, 2021 at 05:03:54PM +0200, Vincent Guittot wrote:
> On Thu, 17 Jun 2021 at 13:05, Mel Gorman <[email protected]> wrote:
> >
> > On Thu, Jun 17, 2021 at 12:02:56PM +0200, Vincent Guittot wrote:
> > > > > >
> > > > > > Fundamentally though, as the changelog notes "due to the nature of the
> > > > > > patch, this is a regression magnet". There are going to be examples
> > > > > > where a deep search is better even if a machine is fully busy or
> > > > > > overloaded and examples where cutting off the search is better. I think
> > > > > > it's better to have an idle estimate that gets updated if CPUs are fully
> > > > > > busy even if it's not a universal win.
> > > > >
> > > > > Although I agree that using a stall average idle time value of local
> > > > > is not good, I'm not sure this proposal is better. The main problem is
> > > > > that we use the avg_idle of the local CPU to estimate how many times
> > > > > we should loop and try to find another idle CPU. But there is no
> > > > > direct relation between both.
> > > >
> > > > This is true. The idle time of the local CPU is used to estimate the
> > > > idle time of the domain which is inevitably going to be inaccurate but
> > >
> > > I'm more and more convinced that using average idle time (of the
> > > local cpu or the full domain) is not the right metric. In
> > > select_idle_cpu(), we looks for an idle CPU but we don't care about
> > > how long it will be idle.
> >
> > Can we predict that accurately? cpufreq for intel_pstate used to try
> > something like that but it was a bit fuzzy and I don't know if the
> > scheduler could do much better. There is some idle prediction stuff but
> > it's related to nohz which does not really help us if a machine is nearly
> > fully busy or overloaded.
> >
> > I guess for tracking idle that revisiting
> > https://lore.kernel.org/lkml/[email protected]/
> > is an option now that the scan is somewhat unified. A two-pass scan
> > could be used to check potentially idle CPUs first and if there is
> > sufficient search depth left, scan other CPUs. There were some questions
>
> I think it's the other way around:
> a CPU is busy for sure if it is not set in the cpuidle_mask and we
> don't need to check it. But a cpu might not be idle even if it is set
> in the idle mask might because it's cleared during the tick
>

Tick is a long time so scan depth may still be a problem.

> > Selecting based on avg idle time could be interesting but hazardous. If
> > for example, we prioritised selecting a CPU that is mostly idle, it'll
> > also pick CPUs that are potentially in a deep idle state incurring a
> > larger wakeup cost. Right now we are not much better because we just
> > select an idle CPU and hope for the best but always targetting the most
> > idle CPU could have problems. There would also be the cost of tracking
> > idle CPUs in priority order. It would eliminate the scan depth cost
> > calculations but the overall cost would be much worse.
> >
> > Hence, I still think we can improve the scan depth costs in the short
> > term until a replacement is identified that works reasonably well.
> >
> > > Even more, we can scan all CPUs whatever the
> > > avg idle time if there is a chance that there is an idle core.
> > >
> >
> > That is an important, but separate topic. It's known that the idle core
> > detection can yield false positives. Putting core scanning under SIS_PROP
> > had mixed results when we last tried but things change. Again, it doesn't
> > help with scan depth calculations.
>
> my point was mainly to highlight that the path can take opposite
> decision for the same avg_idle value:
> - scan all cpus if has_idle_core is true whatever avg_idle
> - limit the depth if has_idle_core is false and avg_idle is short
>

I do understand the point but the idle core scan anomaly was not
intended to be addressed in the patch because putting the idle scan
under SIS_PROP potentially means using cpus with active idle siblings
prematurely.

> >
> > > > tracking idle time for the domain will be cache write intensive and
> > > > potentially very expensive. I think this was discussed before but maybe
> > > > it is my imaginaction.
> > > >
> > > > > Typically, a short average idle time on
> > > > > the local CPU doesn't mean that there are less idle CPUs and that's
> > > > > why we have a mix a gain and loss
> > > > >
> > > >
> > > > Can you evaluate if scanning proportional to cores helps if applied on
> > > > top? The patch below is a bit of pick&mix and has only seen a basic build
> > >
> > > I will queue it for some test later today
> > >
> >
> > Thanks. The proposed patch since passed a build and boot test,
> > performance evaluation is under way but as it's x86 and SMT2, I'm mostly
> > just checking that it's neutral.
>
> Results stay similar:
> group tip/sched/core + this patch + latest addon
> 1 13.358(+/- 1.82%) 12.850(+/- 2.21%) +4% 13.411(+/- 2.47%) -0%
> 4 4.286(+/- 2.77%) 4.114(+/- 2.25%) +4% 4.163(+/- 1.88%) +3%
> 16 3.175(+/- 0.55%) 3.559(+/- 0.43%) -12% 3.535(+/- 0.52%) -11%
> 32 2.912(+/- 0.79%) 3.165(+/- 0.95%) -8% 3.153(+/- 0.76%) -10%
> 64 2.859(+/- 1.12%) 2.937(+/- 0.91%) -3% 2.919(+/- 0.73%) -2%
> 128 3.092(+/- 4.75%) 3.003(+/-5.18%) +3% 2.973(+/- 0.90%) +4%
> 256 3.233(+/- 3.03%) 2.973(+/- 0.80%) +8% 3.036(+/- 1.05%) +6%
>

Ok, accounting for SMT4 didn't help.

--
Mel Gorman
SUSE Labs

2021-06-17 18:30:30

by Mel Gorman

[permalink] [raw]

Subject: Re: [PATCH v2] sched/fair: Age the average idle time

On Thu, Jun 17, 2021 at 11:01:16AM -0400, Phil Auld wrote:
> > Thanks, so far no serious objection :)
> >
> > The latest results as I see them have been copied to
> > https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/dashboard.html
> > They will move from here if the patch is accepted to 5-assembly replacing
> > 3-perf-test. This naming is part of my workflow for evaluating topic
> > branches separetly and then putting them together for another round
> > of testing.
> >
> > NAS shows small differences but NAS would see limited impact from the
> > patch. Specjbb shows small losses and some minor gains which is unfortunate
> > but the workload tends to see small gains and losses all the time.
> > redis is a mixed bag but has some wins. hackbench is the main benefit
> > because it's wakeup intensive and tends to overload machines where deep
> > searches hurt.
> >
> > There are other results in there if you feel like digging around
> > such as sched-core tested with no processes getting tagged with prctl
> > https://beta.suse.com/private/mgorman/melt/v5.13-rc5/5-assembly/sched/sched-schedcore-v1r2/html/dashboard.html
> >
>
> Thanks for the links. It's cool to see what your results dashboard looks like.
> It's really small, what are you plotting in those heat maps?
>
> It's hard for me to publish the results that come from our testing (web based
> on intranet) but we don't see any major differences with this patch. There
> are some gains here and there mostly balanced by some loses. Overall it comes
> out basically as a wash across our main performance test workload.
>

Ok, that's unfortunate. It's also somewhat surprising but then again, I
don't know what tests were executed.

> It'll be interesting to see if it effects a sensitive, proprietary perf test
> suite from a European company with a 3 letter name :)
>

I don't think it's worth the effort if it's failing microbenchmarks at
the moment.

--
Mel Gorman
SUSE Labs

2021-06-18 19:55:42

by Phil Auld

[permalink] [raw]

Subject: Re: [PATCH v2] sched/fair: Age the average idle time

On Thu, Jun 17, 2021 at 04:40:06PM +0100 Mel Gorman wrote:
> On Thu, Jun 17, 2021 at 11:01:16AM -0400, Phil Auld wrote:
> > > Thanks, so far no serious objection :)
> > >
> > > The latest results as I see them have been copied to
> > > https://beta.suse.com/private/mgorman/melt/v5.13-rc5/3-perf-test/sched/sched-avgidle-v1r6/html/dashboard.html
> > > They will move from here if the patch is accepted to 5-assembly replacing
> > > 3-perf-test. This naming is part of my workflow for evaluating topic
> > > branches separetly and then putting them together for another round
> > > of testing.
> > >
> > > NAS shows small differences but NAS would see limited impact from the
> > > patch. Specjbb shows small losses and some minor gains which is unfortunate
> > > but the workload tends to see small gains and losses all the time.
> > > redis is a mixed bag but has some wins. hackbench is the main benefit
> > > because it's wakeup intensive and tends to overload machines where deep
> > > searches hurt.
> > >
> > > There are other results in there if you feel like digging around
> > > such as sched-core tested with no processes getting tagged with prctl
> > > https://beta.suse.com/private/mgorman/melt/v5.13-rc5/5-assembly/sched/sched-schedcore-v1r2/html/dashboard.html
> > >
> >
> > Thanks for the links. It's cool to see what your results dashboard looks like.
> > It's really small, what are you plotting in those heat maps?
> >
> > It's hard for me to publish the results that come from our testing (web based
> > on intranet) but we don't see any major differences with this patch. There
> > are some gains here and there mostly balanced by some loses. Overall it comes
> > out basically as a wash across our main performance test workload.
> >
>
> Ok, that's unfortunate. It's also somewhat surprising but then again, I
> don't know what tests were executed.

Yes, I know, sorry. I get these really nice reports but they're hard to
summarize in text :)

The testing consists of NAS (mostly C, some D), stress-ng, libmicro, linpack
and stream, specjvm2008, specjbb2005 and a few others across a range of (x86_64)
hardware.

From libmicro there was these 2 which stand out

exit_100 52584 66975 [ -27.4%]
exit_100_nolibc 46831 57395 [ -22.6%]

Then some scattered gain and loss in the under +/-10% range.
That's on a 2-node Rome. It's a bit worse on a 8-node Rome
but with libmicro it's a few nsecs here or there.

Here are some pieces from the summary page I pulled out and trimmed as an example:
kernel testcase system top % change
5.13.0_rc5.idle_aging NASParallel-2_NUMA_nodes gold-2s-b NASParallel_ua_C_x: 4%
NASParallel_sp_C_x: -4%

5.13.0_rc5.idle_aging NASParallel-2_NUMA_nodes amd-epyc2-rome NASParallel_sp_C_x: 6%
NASParallel_sp_C_x: -10%

5.13.0_rc5.idle_aging LinpackAndStream-2_NUMA_nod… amd-epyc2-rome Linpack_stream_default: 4%
Linpack_linpackd_default: -11%

5.13.0_rc5.idle_aging LinpackAndStream-2_NUMA_nod… gold-2s-b Linpack_linpacks_default: 7%
Linpack_linpackd_default: -2%

5.13.0_rc5.idle_aging stress-ng-1_NUMA_nodes gold-1s STRESSngTest_getdent: 11%
STRESSngTest_madvise: -15%

5.13.0_rc5.idle_aging Intel_Linpack-8_NUMA_nodes gold-4s-b IntelLinpack_default: 12%
IntelLinpack_single: -12%

The "gold"s are Intel based.

As I said, the perf team reported no real difference one way or the other.
From their perspective this patch is basically neutral.

>
> > It'll be interesting to see if it effects a sensitive, proprietary perf test
> > suite from a European company with a 3 letter name :)
> >
>
> I don't think it's worth the effort if it's failing microbenchmarks at
> the moment.

Fair enough. I'm sensitive to this one because it can be a real pain to track down minor
trade-off differences. And I believe you have been there too...

Cheers,
Phil

>
> --
> Mel Gorman
> SUSE Labs
>

--

2021-06-23 14:32:38

by kernel test robot

[permalink] [raw]

Subject: [sched/fair] 5359f5ca0f: phoronix-test-suite.stress-ng.SystemVMessagePassing.bogo_ops_s 77.9% improvement

Greeting,

FYI, we noticed a

commit: 5359f5ca0f37ba59f5d url: in testcase: phoronix-test-suite
on test machine: 96 with following parameters:

test: stress-ng-1.2.2
option_a: System cpufreq_governor: performance
ucode: 0x5003006

test-description: test-url: Thanks,
Oliver Sang

77.9% improvement of phoronix-test-suite.stress-ng.SystemVMessagePassing.bogo_ops_s due to commit:
679beefbdb05184272846 ("[PATCH v2] sched/fair: Age the average idle time")
ub.com/0day-ci/linux/commits/Mel-Gorman/sched-fair-Age-the-average-idle-time/20210617-061733">https://github.com/0day-ci/linux/commits/Mel-Gorman/sched-fair-Age-the-average-idle-time/20210617-061733
.kernel.org/cgit/linux/kernel/git/tip/tip.git">https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 0159bb020ca9a43b17aa9149f1199643c1d49426
threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 128G memory
V Message Passing
The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added.
www.phoronix-test-suite.com/">http://www.phoronix-test-suite.com/
----------------------------------------------------------------------->
href="https://github.com/intel/lkp-tests.git">https://github.com/intel/lkp-tests.git
job.yaml # job file is attached in this email
--compatible job.yaml # generate the yaml file for lkp run
generated-yaml-file
==============================================================
config/option_a/rootfs/tbox_group/test/testcase/ucode:
rhel-8.3/System V Message Passing/debian-x86_64-phoronix/lkp-csl-2sp8/stress-ng-1.2.2/phoronix-test-suite/0x5003006
Add usecases, design and interface for core scheduling")
Age the average idle time")
37ba59f5d679beefb
-----------------
%stddev
\
+77.9% 10579766 phoronix-test-suite.stress-ng.SystemVMessagePassing.bogo_ops_s
469.13 ? 9% phoronix-test-suite.time.elapsed_time
469.13 ? 9% phoronix-test-suite.time.elapsed_time.max
+3040.3% 2.952e+09 ? 11% phoronix-test-suite.time.involuntary_context_switches
60060 ? 9% phoronix-test-suite.time.major_page_faults
338905 ? 4% phoronix-test-suite.time.minor_page_faults
8226 phoronix-test-suite.time.percent_of_cpu_this_job_got
36954 ? 9% phoronix-test-suite.time.system_time
1645 ? 10% phoronix-test-suite.time.user_time
+3022.2% 2.961e+09 ? 11% phoronix-test-suite.time.voluntary_context_switches
2721203 ? 33% cpuidle.C1.usage
463591 ? 9% cpuidle.POLL.usage
24.52 ? 9% mpstat.cpu.all.idle%
0.01 ? 21% mpstat.cpu.all.iowait%
502.56 ? 9% uptime.boot
8579 ? 6% uptime.idle
7884139 ? 10% numa-numastat.node0.local_node
7930255 ? 9% numa-numastat.node0.numa_hit
4698964 ? 15% numa-numastat.node1.local_node
4721144 ? 15% numa-numastat.node1.numa_hit
2712445 ? 33% turbostat.C1
11.04 ? 18% turbostat.CPU%c1
+352.1% 1.207e+08 ? 10% turbostat.IRQ
462022 ? 9% turbostat.POLL
259.29 turbostat.PkgWatt
13.22 ? 4% vmstat.cpu.id
216.00 ? 20% vmstat.io.bi
6233159 ? 12% vmstat.memory.cache
+637.3% 12517343 ? 3% vmstat.system.cs
+6.8% 255676 vmstat.system.in
417793 ?101% numa-meminfo.node0.Active(anon)
245946 ? 12% numa-meminfo.node0.AnonHugePages
367690 ? 3% numa-meminfo.node0.AnonPages
389727 ? 2% numa-meminfo.node0.AnonPages.max
2997760 ? 25% numa-meminfo.node1.Active
2962159 ? 26% numa-meminfo.node1.Active(anon)
159373 ? 7% numa-meminfo.node1.AnonPages.max
4093689 ? 19% numa-meminfo.node1.FilePages
+86.6% 1141181 ? 19% numa-meminfo.node1.Inactive
1093896 ? 18% numa-meminfo.node1.Inactive(anon)
346002 ? 32% numa-meminfo.node1.Mapped
4654338 ? 15% numa-meminfo.node1.MemUsed
4010040 ? 20% numa-meminfo.node1.Shmem
104584 ?101% numa-vmstat.node0.nr_active_anon
91925 ? 3% numa-vmstat.node0.nr_anon_pages
119.78 ? 12% numa-vmstat.node0.nr_anon_transparent_hugepages
104584 ?101% numa-vmstat.node0.nr_zone_active_anon
5112850 ? 9% numa-vmstat.node0.numa_hit
5062120 ? 9% numa-vmstat.node0.numa_local
741044 ? 26% numa-vmstat.node1.nr_active_anon
1023830 ? 19% numa-vmstat.node1.nr_file_pages
273372 ? 19% numa-vmstat.node1.nr_inactive_anon
86339 ? 33% numa-vmstat.node1.nr_mapped
1002918 ? 20% numa-vmstat.node1.nr_shmem
741044 ? 26% numa-vmstat.node1.nr_zone_active_anon
273372 ? 19% numa-vmstat.node1.nr_zone_inactive_anon
3370126 ? 5% numa-vmstat.node1.numa_hit
3340029 ? 5% numa-vmstat.node1.numa_local
3484665 ? 15% meminfo.Active
3382785 ? 15% meminfo.Active(anon)
250403 ? 12% meminfo.AnonHugePages
412449 meminfo.AnonPages
6112827 ? 12% meminfo.Cached
+51.3% 13133654 ? 5% meminfo.Committed_AS
+27.4% 3042942 ? 8% meminfo.Inactive
+45.0% 2081458 ? 12% meminfo.Inactive(anon)
115451 meminfo.KReclaimable
512718 ? 28% meminfo.Mapped
+95.0% 7600821 ? 9% meminfo.Memused
1303 ? 44% meminfo.Mlocked
115451 meminfo.SReclaimable
5049917 ? 14% meminfo.Shmem
1371 ? 42% meminfo.Unevictable
7954869 ? 9% meminfo.max_used_kB
+182.6% 1.445e+10 ? 2% perf-stat.i.branch-instructions
+216.3% 1.262e+08 ? 2% perf-stat.i.branch-misses
12.47 ? 8% perf-stat.i.cache-miss-rate%
+16.8% 68682862 ? 2% perf-stat.i.cache-misses
+100.8% 5.846e+08 ? 3% perf-stat.i.cache-references
+617.1% 12626462 ? 3% perf-stat.i.context-switches
3.75 ? 5% perf-stat.i.cpi
+3.3% 2.31e+11 perf-stat.i.cpu-cycles
1092 perf-stat.i.cpu-migrations
3381 ? 5% perf-stat.i.cycles-between-cache-misses
+179.8% 1.916e+10 ? 3% perf-stat.i.dTLB-loads
144605 ? 8% perf-stat.i.dTLB-store-misses
+215.9% 1.171e+10 ? 2% perf-stat.i.dTLB-stores
48.07 perf-stat.i.iTLB-load-miss-rate%
+209.2% 54234924 ? 2% perf-stat.i.iTLB-load-misses
+521.1% 60130535 ? 3% perf-stat.i.iTLB-loads
+180.5% 7.019e+10 ? 2% perf-stat.i.instructions
0.31 ? 3% perf-stat.i.ipc
-2.4% 128.83 perf-stat.i.major-faults
+3.2% 2406907 perf-stat.i.metric.GHz
+187.3% 4.791e+08 ? 2% perf-stat.i.metric.M/sec
4954 perf-stat.i.minor-faults
83.48 perf-stat.i.node-load-miss-rate%
+10.8% 9083633 ? 2% perf-stat.i.node-load-misses
+51.6% 1692321 ? 8% perf-stat.i.node-loads
85.56 perf-stat.i.node-store-miss-rate%
-13.8% 2655744 ? 2% perf-stat.i.node-store-misses
361888 ? 6% perf-stat.i.node-stores
5083 perf-stat.i.page-faults
40003 ? 3% slabinfo.anon_vma.active_objs
870.00 ? 3% slabinfo.anon_vma.active_slabs
40037 ? 3% slabinfo.anon_vma.num_objs
870.00 ? 3% slabinfo.anon_vma.num_slabs
85427 ? 3% slabinfo.anon_vma_chain.active_objs
1337 ? 3% slabinfo.anon_vma_chain.active_slabs
85640 ? 3% slabinfo.anon_vma_chain.num_objs
1337 ? 3% slabinfo.anon_vma_chain.num_slabs
1939 ? 5% slabinfo.dmaengine-unmap-16.active_objs
1939 ? 5% slabinfo.dmaengine-unmap-16.num_objs
5176 slabinfo.files_cache.active_objs
5176 slabinfo.files_cache.num_objs
3902 slabinfo.khugepaged_mm_slot.active_objs
3902 slabinfo.khugepaged_mm_slot.num_objs
-9.4% 143010 slabinfo.kmalloc-64.active_objs
2239 slabinfo.kmalloc-64.active_slabs
-9.4% 143338 slabinfo.kmalloc-64.num_objs
2239 slabinfo.kmalloc-64.num_slabs
3659 slabinfo.mm_struct.active_objs
3659 slabinfo.mm_struct.num_objs
15920 ? 2% slabinfo.proc_inode_cache.active_objs
15938 ? 2% slabinfo.proc_inode_cache.num_objs
34879 ? 8% slabinfo.radix_tree_node.active_objs
622.44 ? 8% slabinfo.radix_tree_node.active_slabs
34879 ? 8% slabinfo.radix_tree_node.num_objs
622.44 ? 8% slabinfo.radix_tree_node.num_slabs
3023 ? 3% slabinfo.sighand_cache.active_objs
3029 ? 3% slabinfo.sighand_cache.num_objs
14657 ? 4% slabinfo.vmap_area.active_objs
14662 ? 4% slabinfo.vmap_area.num_objs
845415 ? 15% proc-vmstat.nr_active_anon
25468 proc-vmstat.nr_active_file
103116 proc-vmstat.nr_anon_pages
121.78 ? 12% proc-vmstat.nr_anon_transparent_hugepages
-2.9% 3108484 proc-vmstat.nr_dirty_background_threshold
-2.9% 6224570 proc-vmstat.nr_dirty_threshold
1528310 ? 12% proc-vmstat.nr_file_pages
-2.9% 31019305 proc-vmstat.nr_free_pages
520348 ? 12% proc-vmstat.nr_inactive_anon
20712 proc-vmstat.nr_kernel_stack
128281 ? 27% proc-vmstat.nr_mapped
325.56 ? 44% proc-vmstat.nr_mlock
4130 proc-vmstat.nr_page_table_pages
1262178 ? 14% proc-vmstat.nr_shmem
28861 proc-vmstat.nr_slab_reclaimable
342.56 ? 42% proc-vmstat.nr_unevictable
845415 ? 15% proc-vmstat.nr_zone_active_anon
25468 proc-vmstat.nr_zone_active_file
520348 ? 12% proc-vmstat.nr_zone_inactive_anon
342.56 ? 42% proc-vmstat.nr_zone_unevictable
+165.6% 12670919 ? 8% proc-vmstat.numa_hit
1653 ? 24% proc-vmstat.numa_huge_pte_updates
+168.0% 12602615 ? 8% proc-vmstat.numa_local
8514 ? 30% proc-vmstat.numa_pages_migrated
1123301 ? 18% proc-vmstat.numa_pte_updates
1335860 ? 16% proc-vmstat.pgactivate
+163.3% 12862044 ? 8% proc-vmstat.pgalloc_normal
2520248 ? 8% proc-vmstat.pgfault
+187.7% 11452403 ? 8% proc-vmstat.pgfree
8514 ? 30% proc-vmstat.pgmigrate_success
102016 ? 19% proc-vmstat.pgpgin
290270 ? 9% proc-vmstat.pgreuse
20760 ? 12% softirqs.CPU0.RCU
18596 ? 10% softirqs.CPU0.SCHED
21904 ? 9% softirqs.CPU1.RCU
17903 ? 11% softirqs.CPU1.SCHED
21031 ? 9% softirqs.CPU10.RCU
16210 ? 7% softirqs.CPU10.SCHED
21063 ? 9% softirqs.CPU11.RCU
16212 ? 7% softirqs.CPU11.SCHED
20854 ? 14% softirqs.CPU12.RCU
15975 ? 7% softirqs.CPU12.SCHED
20710 ? 14% softirqs.CPU13.RCU
16137 ? 7% softirqs.CPU13.SCHED
21226 ? 10% softirqs.CPU14.RCU
16153 ? 8% softirqs.CPU14.SCHED
21202 ? 11% softirqs.CPU15.RCU
16040 ? 7% softirqs.CPU15.SCHED
21873 ? 11% softirqs.CPU16.RCU
16120 ? 7% softirqs.CPU16.SCHED
22392 ? 11% softirqs.CPU17.RCU
16351 ? 7% softirqs.CPU17.SCHED
22570 ? 11% softirqs.CPU18.RCU
16434 ? 8% softirqs.CPU18.SCHED
21608 ? 14% softirqs.CPU19.RCU
16232 ? 7% softirqs.CPU19.SCHED
21363 ? 10% softirqs.CPU2.RCU
16808 ? 9% softirqs.CPU2.SCHED
22262 ? 10% softirqs.CPU20.RCU
16531 ? 7% softirqs.CPU20.SCHED
22207 ? 10% softirqs.CPU21.RCU
16339 ? 7% softirqs.CPU21.SCHED
21674 ? 10% softirqs.CPU22.RCU
16101 ? 7% softirqs.CPU22.SCHED
22321 ? 11% softirqs.CPU23.RCU
16322 ? 8% softirqs.CPU23.SCHED
21063 ? 9% softirqs.CPU24.RCU
14567 ? 7% softirqs.CPU24.SCHED
20191 ? 10% softirqs.CPU25.RCU
14286 ? 9% softirqs.CPU25.SCHED
20844 ? 9% softirqs.CPU26.RCU
14447 ? 8% softirqs.CPU26.SCHED
21113 ? 10% softirqs.CPU27.RCU
14536 ? 9% softirqs.CPU27.SCHED
21301 ? 10% softirqs.CPU28.RCU
14329 ? 9% softirqs.CPU28.SCHED
21171 ? 9% softirqs.CPU29.RCU
14336 ? 9% softirqs.CPU29.SCHED
21329 ? 10% softirqs.CPU3.RCU
16472 ? 7% softirqs.CPU3.SCHED
20817 ? 9% softirqs.CPU30.RCU
14369 ? 9% softirqs.CPU30.SCHED
20369 ? 10% softirqs.CPU31.RCU
14280 ? 10% softirqs.CPU31.SCHED
21841 ? 10% softirqs.CPU32.RCU
14320 ? 9% softirqs.CPU32.SCHED
21871 ? 8% softirqs.CPU33.RCU
14308 ? 9% softirqs.CPU33.SCHED
21468 ? 9% softirqs.CPU34.RCU
14563 ? 9% softirqs.CPU34.SCHED
21893 ? 8% softirqs.CPU35.RCU
14222 ? 10% softirqs.CPU35.SCHED
21700 ? 10% softirqs.CPU36.RCU
14312 ? 9% softirqs.CPU36.SCHED
20864 ? 10% softirqs.CPU37.RCU
14274 ? 9% softirqs.CPU37.SCHED
21481 ? 10% softirqs.CPU38.RCU
14165 ? 9% softirqs.CPU38.SCHED
21376 ? 10% softirqs.CPU39.RCU
14271 ? 9% softirqs.CPU39.SCHED
21135 ? 11% softirqs.CPU4.RCU
16241 ? 8% softirqs.CPU4.SCHED
21619 ? 10% softirqs.CPU40.RCU
14232 ? 9% softirqs.CPU40.SCHED
21658 ? 8% softirqs.CPU41.RCU
14420 ? 9% softirqs.CPU41.SCHED
21968 ? 8% softirqs.CPU42.RCU
14246 ? 9% softirqs.CPU42.SCHED
22030 ? 9% softirqs.CPU43.RCU
14347 ? 8% softirqs.CPU43.SCHED
21883 ? 9% softirqs.CPU44.RCU
14489 ? 8% softirqs.CPU44.SCHED
21693 ? 8% softirqs.CPU45.RCU
14385 ? 8% softirqs.CPU45.SCHED
21665 ? 8% softirqs.CPU46.RCU
14424 ? 9% softirqs.CPU46.SCHED
21711 ? 9% softirqs.CPU47.RCU
14351 ? 9% softirqs.CPU47.SCHED
19951 ? 13% softirqs.CPU48.RCU
15680 ? 9% softirqs.CPU48.SCHED
21161 ? 10% softirqs.CPU49.RCU
15638 ? 5% softirqs.CPU49.SCHED
21111 ? 10% softirqs.CPU5.RCU
16374 ? 7% softirqs.CPU5.SCHED
21325 ? 10% softirqs.CPU50.RCU
15715 ? 8% softirqs.CPU50.SCHED
21631 ? 11% softirqs.CPU51.RCU
15974 ? 5% softirqs.CPU51.SCHED
21315 ? 12% softirqs.CPU52.RCU
16007 ? 7% softirqs.CPU52.SCHED
21866 ? 11% softirqs.CPU53.RCU
16098 ? 7% softirqs.CPU53.SCHED
21800 ? 12% softirqs.CPU54.RCU
16044 ? 8% softirqs.CPU54.SCHED
21260 ? 12% softirqs.CPU55.RCU
15807 ? 7% softirqs.CPU55.SCHED
21014 ? 11% softirqs.CPU56.RCU
15673 ? 8% softirqs.CPU56.SCHED
21450 ? 11% softirqs.CPU57.RCU
16138 ? 6% softirqs.CPU57.SCHED
21102 ? 9% softirqs.CPU58.RCU
15951 ? 8% softirqs.CPU58.SCHED
21550 ? 10% softirqs.CPU59.RCU
15993 ? 8% softirqs.CPU59.SCHED
21316 ? 10% softirqs.CPU6.RCU
16336 ? 7% softirqs.CPU6.SCHED
21673 ? 11% softirqs.CPU60.RCU
15894 ? 8% softirqs.CPU60.SCHED
21011 ? 11% softirqs.CPU61.RCU
15859 ? 7% softirqs.CPU61.SCHED
21298 ? 9% softirqs.CPU62.RCU
15838 ? 6% softirqs.CPU62.SCHED
20983 ? 10% softirqs.CPU63.RCU
15787 ? 7% softirqs.CPU63.SCHED
21826 ? 11% softirqs.CPU64.RCU
15982 ? 7% softirqs.CPU64.SCHED
22217 ? 9% softirqs.CPU65.RCU
16114 ? 7% softirqs.CPU65.SCHED
22138 ? 10% softirqs.CPU66.RCU
15847 ? 7% softirqs.CPU66.SCHED
21972 ? 12% softirqs.CPU67.RCU
15843 ? 9% softirqs.CPU67.SCHED
22522 ? 9% softirqs.CPU68.RCU
16434 ? 8% softirqs.CPU68.SCHED
21496 ? 14% softirqs.CPU69.RCU
16142 ? 8% softirqs.CPU69.SCHED
21404 ? 11% softirqs.CPU7.RCU
15974 ? 7% softirqs.CPU7.SCHED
22321 ? 11% softirqs.CPU70.RCU
16250 ? 8% softirqs.CPU70.SCHED
22152 ? 12% softirqs.CPU71.RCU
15870 ? 7% softirqs.CPU71.SCHED
21072 ? 8% softirqs.CPU72.RCU
14533 ? 8% softirqs.CPU72.SCHED
20380 ? 11% softirqs.CPU73.RCU
14243 ? 8% softirqs.CPU73.SCHED
20952 ? 9% softirqs.CPU74.RCU
14233 ? 8% softirqs.CPU74.SCHED
21123 ? 11% softirqs.CPU75.RCU
14497 ? 9% softirqs.CPU75.SCHED
21398 ? 10% softirqs.CPU76.RCU
14306 ? 9% softirqs.CPU76.SCHED
21183 ? 10% softirqs.CPU77.RCU
14335 ? 9% softirqs.CPU77.SCHED
20929 ? 9% softirqs.CPU78.RCU
14249 ? 9% softirqs.CPU78.SCHED
20420 ? 9% softirqs.CPU79.RCU
14360 ? 8% softirqs.CPU79.SCHED
21290 ? 9% softirqs.CPU8.RCU
16171 ? 9% softirqs.CPU8.SCHED
21098 ? 9% softirqs.CPU80.RCU
14261 ? 9% softirqs.CPU80.SCHED
22054 ? 8% softirqs.CPU81.RCU
14644 ? 8% softirqs.CPU81.SCHED
21037 ? 8% softirqs.CPU82.RCU
14460 ? 8% softirqs.CPU82.SCHED
21778 ? 9% softirqs.CPU83.RCU
14263 ? 9% softirqs.CPU83.SCHED
21486 ? 9% softirqs.CPU84.RCU
14317 ? 8% softirqs.CPU84.SCHED
20979 ? 9% softirqs.CPU85.RCU
14342 ? 8% softirqs.CPU85.SCHED
21194 ? 8% softirqs.CPU86.RCU
14279 ? 8% softirqs.CPU86.SCHED
21010 ? 9% softirqs.CPU87.RCU
14426 ? 8% softirqs.CPU87.SCHED
21253 ? 9% softirqs.CPU88.RCU
14211 ? 8% softirqs.CPU88.SCHED
21691 ? 9% softirqs.CPU89.RCU
14550 ? 8% softirqs.CPU89.SCHED
21094 ? 10% softirqs.CPU9.RCU
16514 ? 8% softirqs.CPU9.SCHED
22064 ? 9% softirqs.CPU90.RCU
14640 ? 8% softirqs.CPU90.SCHED
21793 ? 10% softirqs.CPU91.RCU
14440 ? 8% softirqs.CPU91.SCHED
21890 ? 11% softirqs.CPU92.RCU
14528 ? 8% softirqs.CPU92.SCHED
21318 ? 8% softirqs.CPU93.RCU
14336 ? 7% softirqs.CPU93.SCHED
21497 ? 8% softirqs.CPU94.RCU
14476 ? 8% softirqs.CPU94.SCHED
21773 ? 6% softirqs.CPU95.RCU
14570 ? 7% softirqs.CPU95.SCHED
7767 ? 64% softirqs.NET_RX
2057758 ? 9% softirqs.RCU
1467162 ? 7% softirqs.SCHED
76310 ? 8% softirqs.TIMER
941.78 ? 9% interrupts.9:IO-APIC.9-fasteoi.acpi
8489677 ? 10% interrupts.CAL:Function_call_interrupts
124471 ? 16% interrupts.CPU0.CAL:Function_call_interrupts
935623 ? 10% interrupts.CPU0.LOC:Local_timer_interrupts
283667 ? 10% interrupts.CPU0.RES:Rescheduling_interrupts
941.78 ? 9% interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
117666 ? 8% interrupts.CPU1.CAL:Function_call_interrupts
935343 ? 10% interrupts.CPU1.LOC:Local_timer_interrupts
284822 ? 9% interrupts.CPU1.RES:Rescheduling_interrupts
106515 ? 8% interrupts.CPU10.CAL:Function_call_interrupts
935134 ? 10% interrupts.CPU10.LOC:Local_timer_interrupts
290995 ? 9% interrupts.CPU10.RES:Rescheduling_interrupts
110171 ? 19% interrupts.CPU11.CAL:Function_call_interrupts
934976 ? 10% interrupts.CPU11.LOC:Local_timer_interrupts
292318 ? 11% interrupts.CPU11.RES:Rescheduling_interrupts
102743 ? 9% interrupts.CPU12.CAL:Function_call_interrupts
934784 ? 10% interrupts.CPU12.LOC:Local_timer_interrupts
273144 ? 8% interrupts.CPU12.RES:Rescheduling_interrupts
103538 ? 13% interrupts.CPU13.CAL:Function_call_interrupts
935052 ? 10% interrupts.CPU13.LOC:Local_timer_interrupts
281862 ? 14% interrupts.CPU13.RES:Rescheduling_interrupts
108669 ? 12% interrupts.CPU14.CAL:Function_call_interrupts
934869 ? 10% interrupts.CPU14.LOC:Local_timer_interrupts
290709 ? 16% interrupts.CPU14.RES:Rescheduling_interrupts
103709 ? 14% interrupts.CPU15.CAL:Function_call_interrupts
934691 ? 10% interrupts.CPU15.LOC:Local_timer_interrupts
273590 ? 11% interrupts.CPU15.RES:Rescheduling_interrupts
102852 ? 11% interrupts.CPU16.CAL:Function_call_interrupts
934943 ? 10% interrupts.CPU16.LOC:Local_timer_interrupts
279796 ? 8% interrupts.CPU16.RES:Rescheduling_interrupts
109406 ? 10% interrupts.CPU17.CAL:Function_call_interrupts
934620 ? 10% interrupts.CPU17.LOC:Local_timer_interrupts
307405 ? 12% interrupts.CPU17.RES:Rescheduling_interrupts
102309 ? 10% interrupts.CPU18.CAL:Function_call_interrupts
934692 ? 10% interrupts.CPU18.LOC:Local_timer_interrupts
280608 ? 13% interrupts.CPU18.RES:Rescheduling_interrupts
100908 ? 12% interrupts.CPU19.CAL:Function_call_interrupts
935220 ? 10% interrupts.CPU19.LOC:Local_timer_interrupts
274601 ? 12% interrupts.CPU19.RES:Rescheduling_interrupts
104675 ? 13% interrupts.CPU2.CAL:Function_call_interrupts
934820 ? 10% interrupts.CPU2.LOC:Local_timer_interrupts
272230 ? 10% interrupts.CPU2.RES:Rescheduling_interrupts
126131 ? 11% interrupts.CPU20.CAL:Function_call_interrupts
934872 ? 10% interrupts.CPU20.LOC:Local_timer_interrupts
352318 ? 9% interrupts.CPU20.RES:Rescheduling_interrupts
103814 ? 11% interrupts.CPU21.CAL:Function_call_interrupts
934640 ? 10% interrupts.CPU21.LOC:Local_timer_interrupts
282960 ? 11% interrupts.CPU21.RES:Rescheduling_interrupts
103258 ? 11% interrupts.CPU22.CAL:Function_call_interrupts
935122 ? 10% interrupts.CPU22.LOC:Local_timer_interrupts
280753 ? 10% interrupts.CPU22.RES:Rescheduling_interrupts
102893 ? 10% interrupts.CPU23.CAL:Function_call_interrupts
934607 ? 10% interrupts.CPU23.LOC:Local_timer_interrupts
278513 ? 11% interrupts.CPU23.RES:Rescheduling_interrupts
107849 ? 17% interrupts.CPU24.CAL:Function_call_interrupts
932326 ? 10% interrupts.CPU24.LOC:Local_timer_interrupts
187658 ? 18% interrupts.CPU24.RES:Rescheduling_interrupts
91796 ? 19% interrupts.CPU25.CAL:Function_call_interrupts
932502 ? 10% interrupts.CPU25.LOC:Local_timer_interrupts
178992 ? 11% interrupts.CPU25.RES:Rescheduling_interrupts
90653 ? 24% interrupts.CPU26.CAL:Function_call_interrupts
932180 ? 10% interrupts.CPU26.LOC:Local_timer_interrupts
173689 ? 16% interrupts.CPU26.RES:Rescheduling_interrupts
86935 ? 17% interrupts.CPU27.CAL:Function_call_interrupts
932156 ? 10% interrupts.CPU27.LOC:Local_timer_interrupts
193694 ? 18% interrupts.CPU27.RES:Rescheduling_interrupts
69785 ? 15% interrupts.CPU28.CAL:Function_call_interrupts
932126 ? 10% interrupts.CPU28.LOC:Local_timer_interrupts
172295 ? 19% interrupts.CPU28.RES:Rescheduling_interrupts
69004 ? 11% interrupts.CPU29.CAL:Function_call_interrupts
932218 ? 10% interrupts.CPU29.LOC:Local_timer_interrupts
169218 ? 11% interrupts.CPU29.RES:Rescheduling_interrupts
125693 ? 9% interrupts.CPU3.CAL:Function_call_interrupts
935540 ? 10% interrupts.CPU3.LOC:Local_timer_interrupts
334631 ? 8% interrupts.CPU3.RES:Rescheduling_interrupts
63313 ? 18% interrupts.CPU30.CAL:Function_call_interrupts
932297 ? 10% interrupts.CPU30.LOC:Local_timer_interrupts
157907 ? 22% interrupts.CPU30.RES:Rescheduling_interrupts
72249 ? 15% interrupts.CPU31.CAL:Function_call_interrupts
932201 ? 10% interrupts.CPU31.LOC:Local_timer_interrupts
177711 ? 22% interrupts.CPU31.RES:Rescheduling_interrupts
73506 ? 21% interrupts.CPU32.CAL:Function_call_interrupts
932032 ? 10% interrupts.CPU32.LOC:Local_timer_interrupts
166548 ? 20% interrupts.CPU32.RES:Rescheduling_interrupts
86654 ? 16% interrupts.CPU33.CAL:Function_call_interrupts
932395 ? 10% interrupts.CPU33.LOC:Local_timer_interrupts
212836 ? 16% interrupts.CPU33.RES:Rescheduling_interrupts
76711 ? 18% interrupts.CPU34.CAL:Function_call_interrupts
932057 ? 10% interrupts.CPU34.LOC:Local_timer_interrupts
181222 ? 19% interrupts.CPU34.RES:Rescheduling_interrupts
72528 ? 15% interrupts.CPU35.CAL:Function_call_interrupts
932098 ? 10% interrupts.CPU35.LOC:Local_timer_interrupts
177651 ? 18% interrupts.CPU35.RES:Rescheduling_interrupts
68370 ? 15% interrupts.CPU36.CAL:Function_call_interrupts
932241 ? 10% interrupts.CPU36.LOC:Local_timer_interrupts
173184 ? 14% interrupts.CPU36.RES:Rescheduling_interrupts
65373 ? 16% interrupts.CPU37.CAL:Function_call_interrupts
932306 ? 10% interrupts.CPU37.LOC:Local_timer_interrupts
166879 ? 17% interrupts.CPU37.RES:Rescheduling_interrupts
72019 ? 14% interrupts.CPU38.CAL:Function_call_interrupts
932259 ? 10% interrupts.CPU38.LOC:Local_timer_interrupts
172118 ? 15% interrupts.CPU38.RES:Rescheduling_interrupts
67013 ? 22% interrupts.CPU39.CAL:Function_call_interrupts
932067 ? 10% interrupts.CPU39.LOC:Local_timer_interrupts
165667 ? 19% interrupts.CPU39.RES:Rescheduling_interrupts
100612 ? 11% interrupts.CPU4.CAL:Function_call_interrupts
934594 ? 10% interrupts.CPU4.LOC:Local_timer_interrupts
270954 ? 9% interrupts.CPU4.RES:Rescheduling_interrupts
67782 ? 18% interrupts.CPU40.CAL:Function_call_interrupts
932137 ? 10% interrupts.CPU40.LOC:Local_timer_interrupts
165778 ? 19% interrupts.CPU40.RES:Rescheduling_interrupts
75787 ? 19% interrupts.CPU41.CAL:Function_call_interrupts
931979 ? 10% interrupts.CPU41.LOC:Local_timer_interrupts
180915 ? 11% interrupts.CPU41.RES:Rescheduling_interrupts
71343 ? 18% interrupts.CPU42.CAL:Function_call_interrupts
932018 ? 10% interrupts.CPU42.LOC:Local_timer_interrupts
175085 ? 21% interrupts.CPU42.RES:Rescheduling_interrupts
71779 ? 14% interrupts.CPU43.CAL:Function_call_interrupts
932145 ? 10% interrupts.CPU43.LOC:Local_timer_interrupts
161615 ? 14% interrupts.CPU43.RES:Rescheduling_interrupts
81548 ? 19% interrupts.CPU44.CAL:Function_call_interrupts
932041 ? 10% interrupts.CPU44.LOC:Local_timer_interrupts
209248 ? 15% interrupts.CPU44.RES:Rescheduling_interrupts
72776 ? 14% interrupts.CPU45.CAL:Function_call_interrupts
932031 ? 10% interrupts.CPU45.LOC:Local_timer_interrupts
177960 ? 12% interrupts.CPU45.RES:Rescheduling_interrupts
68039 ? 15% interrupts.CPU46.CAL:Function_call_interrupts
932079 ? 10% interrupts.CPU46.LOC:Local_timer_interrupts
165252 ? 15% interrupts.CPU46.RES:Rescheduling_interrupts
77068 ? 23% interrupts.CPU47.CAL:Function_call_interrupts
932018 ? 10% interrupts.CPU47.LOC:Local_timer_interrupts
178419 ? 20% interrupts.CPU47.RES:Rescheduling_interrupts
108365 ? 8% interrupts.CPU48.CAL:Function_call_interrupts
935137 ? 10% interrupts.CPU48.LOC:Local_timer_interrupts
278221 ? 6% interrupts.CPU48.RES:Rescheduling_interrupts
100127 ? 12% interrupts.CPU49.CAL:Function_call_interrupts
935364 ? 10% interrupts.CPU49.LOC:Local_timer_interrupts
266724 ? 10% interrupts.CPU49.RES:Rescheduling_interrupts
108740 ? 12% interrupts.CPU5.CAL:Function_call_interrupts
934784 ? 10% interrupts.CPU5.LOC:Local_timer_interrupts
284200 ? 9% interrupts.CPU5.RES:Rescheduling_interrupts
99608 ? 15% interrupts.CPU50.CAL:Function_call_interrupts
934744 ? 10% interrupts.CPU50.LOC:Local_timer_interrupts
248580 ? 9% interrupts.CPU50.RES:Rescheduling_interrupts
123091 ? 9% interrupts.CPU51.CAL:Function_call_interrupts
935794 ? 10% interrupts.CPU51.LOC:Local_timer_interrupts
323015 ? 9% interrupts.CPU51.RES:Rescheduling_interrupts
98035 ? 15% interrupts.CPU52.CAL:Function_call_interrupts
935096 ? 10% interrupts.CPU52.LOC:Local_timer_interrupts
254126 ? 10% interrupts.CPU52.RES:Rescheduling_interrupts
100220 ? 10% interrupts.CPU53.CAL:Function_call_interrupts
935127 ? 10% interrupts.CPU53.LOC:Local_timer_interrupts
269042 ? 13% interrupts.CPU53.RES:Rescheduling_interrupts
99372 ? 13% interrupts.CPU54.CAL:Function_call_interrupts
935306 ? 10% interrupts.CPU54.LOC:Local_timer_interrupts
267661 ? 12% interrupts.CPU54.RES:Rescheduling_interrupts
98036 ? 8% interrupts.CPU55.CAL:Function_call_interrupts
934796 ? 10% interrupts.CPU55.LOC:Local_timer_interrupts
267954 ? 6% interrupts.CPU55.RES:Rescheduling_interrupts
89423 ? 17% interrupts.CPU56.CAL:Function_call_interrupts
934536 ? 10% interrupts.CPU56.LOC:Local_timer_interrupts
244608 ? 14% interrupts.CPU56.RES:Rescheduling_interrupts
112067 ? 10% interrupts.CPU57.CAL:Function_call_interrupts
935390 ? 10% interrupts.CPU57.LOC:Local_timer_interrupts
312436 ? 11% interrupts.CPU57.RES:Rescheduling_interrupts
103800 ? 16% interrupts.CPU58.CAL:Function_call_interrupts
934852 ? 10% interrupts.CPU58.LOC:Local_timer_interrupts
274612 ? 14% interrupts.CPU58.RES:Rescheduling_interrupts
102334 ? 14% interrupts.CPU59.CAL:Function_call_interrupts
935578 ? 10% interrupts.CPU59.LOC:Local_timer_interrupts
275509 ? 12% interrupts.CPU59.RES:Rescheduling_interrupts
106459 ? 7% interrupts.CPU6.CAL:Function_call_interrupts
934671 ? 10% interrupts.CPU6.LOC:Local_timer_interrupts
281718 ? 9% interrupts.CPU6.RES:Rescheduling_interrupts
92676 ? 8% interrupts.CPU60.CAL:Function_call_interrupts
935138 ? 10% interrupts.CPU60.LOC:Local_timer_interrupts
255621 ? 8% interrupts.CPU60.RES:Rescheduling_interrupts
95134 ? 12% interrupts.CPU61.CAL:Function_call_interrupts
934826 ? 10% interrupts.CPU61.LOC:Local_timer_interrupts
260844 ? 11% interrupts.CPU61.RES:Rescheduling_interrupts
99149 ? 9% interrupts.CPU62.CAL:Function_call_interrupts
935664 ? 10% interrupts.CPU62.LOC:Local_timer_interrupts
268981 ? 8% interrupts.CPU62.RES:Rescheduling_interrupts
101790 ? 11% interrupts.CPU63.CAL:Function_call_interrupts
935095 ? 10% interrupts.CPU63.LOC:Local_timer_interrupts
271638 ? 11% interrupts.CPU63.RES:Rescheduling_interrupts
101075 ? 12% interrupts.CPU64.CAL:Function_call_interrupts
934597 ? 10% interrupts.CPU64.LOC:Local_timer_interrupts
272998 ? 12% interrupts.CPU64.RES:Rescheduling_interrupts
101448 ? 12% interrupts.CPU65.CAL:Function_call_interrupts
934858 ? 10% interrupts.CPU65.LOC:Local_timer_interrupts
274931 ? 9% interrupts.CPU65.RES:Rescheduling_interrupts
93350 ? 6% interrupts.CPU66.CAL:Function_call_interrupts
934966 ? 10% interrupts.CPU66.LOC:Local_timer_interrupts
252400 ? 8% interrupts.CPU66.RES:Rescheduling_interrupts
92368 ? 8% interrupts.CPU67.CAL:Function_call_interrupts
934778 ? 10% interrupts.CPU67.LOC:Local_timer_interrupts
257039 ? 9% interrupts.CPU67.RES:Rescheduling_interrupts
120547 ? 13% interrupts.CPU68.CAL:Function_call_interrupts
935344 ? 10% interrupts.CPU68.LOC:Local_timer_interrupts
332222 ? 15% interrupts.CPU68.RES:Rescheduling_interrupts
100719 ? 10% interrupts.CPU69.CAL:Function_call_interrupts
934545 ? 10% interrupts.CPU69.LOC:Local_timer_interrupts
270904 ? 9% interrupts.CPU69.RES:Rescheduling_interrupts
104924 ? 11% interrupts.CPU7.CAL:Function_call_interrupts
934463 ? 10% interrupts.CPU7.LOC:Local_timer_interrupts
274393 ? 9% interrupts.CPU7.RES:Rescheduling_interrupts
104226 ? 9% interrupts.CPU70.CAL:Function_call_interrupts
934745 ? 10% interrupts.CPU70.LOC:Local_timer_interrupts
276066 ? 10% interrupts.CPU70.RES:Rescheduling_interrupts
93447 ? 9% interrupts.CPU71.CAL:Function_call_interrupts
934356 ? 10% interrupts.CPU71.LOC:Local_timer_interrupts
255055 ? 9% interrupts.CPU71.RES:Rescheduling_interrupts
78176 ? 19% interrupts.CPU72.CAL:Function_call_interrupts
932320 ? 10% interrupts.CPU72.LOC:Local_timer_interrupts
183605 ? 19% interrupts.CPU72.RES:Rescheduling_interrupts
72380 ? 17% interrupts.CPU73.CAL:Function_call_interrupts
932446 ? 10% interrupts.CPU73.LOC:Local_timer_interrupts
159549 ? 14% interrupts.CPU73.RES:Rescheduling_interrupts
66730 ? 19% interrupts.CPU74.CAL:Function_call_interrupts
932219 ? 10% interrupts.CPU74.LOC:Local_timer_interrupts
154807 ? 21% interrupts.CPU74.RES:Rescheduling_interrupts
88347 ? 22% interrupts.CPU75.CAL:Function_call_interrupts
932240 ? 10% interrupts.CPU75.LOC:Local_timer_interrupts
184144 ? 21% interrupts.CPU75.RES:Rescheduling_interrupts
66130 ? 21% interrupts.CPU76.CAL:Function_call_interrupts
932554 ? 10% interrupts.CPU76.LOC:Local_timer_interrupts
165752 ? 19% interrupts.CPU76.RES:Rescheduling_interrupts
65419 ? 12% interrupts.CPU77.CAL:Function_call_interrupts
932296 ? 10% interrupts.CPU77.LOC:Local_timer_interrupts
167619 ? 15% interrupts.CPU77.RES:Rescheduling_interrupts
67450 ? 25% interrupts.CPU78.CAL:Function_call_interrupts
932566 ? 10% interrupts.CPU78.LOC:Local_timer_interrupts
159393 ? 17% interrupts.CPU78.RES:Rescheduling_interrupts
65653 ? 16% interrupts.CPU79.CAL:Function_call_interrupts
932535 ? 10% interrupts.CPU79.LOC:Local_timer_interrupts
159820 ? 17% interrupts.CPU79.RES:Rescheduling_interrupts
102822 ? 16% interrupts.CPU8.CAL:Function_call_interrupts
935038 ? 10% interrupts.CPU8.LOC:Local_timer_interrupts
264927 ? 14% interrupts.CPU8.RES:Rescheduling_interrupts
65021 ? 18% interrupts.CPU80.CAL:Function_call_interrupts
932142 ? 10% interrupts.CPU80.LOC:Local_timer_interrupts
160105 ? 21% interrupts.CPU80.RES:Rescheduling_interrupts
75910 ? 19% interrupts.CPU81.CAL:Function_call_interrupts
932262 ? 10% interrupts.CPU81.LOC:Local_timer_interrupts
182315 ? 15% interrupts.CPU81.RES:Rescheduling_interrupts
68118 ? 19% interrupts.CPU82.CAL:Function_call_interrupts
932128 ? 10% interrupts.CPU82.LOC:Local_timer_interrupts
161457 ? 18% interrupts.CPU82.RES:Rescheduling_interrupts
61193 ? 17% interrupts.CPU83.CAL:Function_call_interrupts
932154 ? 10% interrupts.CPU83.LOC:Local_timer_interrupts
161329 ? 19% interrupts.CPU83.RES:Rescheduling_interrupts
65082 ? 13% interrupts.CPU84.CAL:Function_call_interrupts
932195 ? 10% interrupts.CPU84.LOC:Local_timer_interrupts
157575 ? 13% interrupts.CPU84.RES:Rescheduling_interrupts
60385 ? 20% interrupts.CPU85.CAL:Function_call_interrupts
932602 ? 10% interrupts.CPU85.LOC:Local_timer_interrupts
153742 ? 16% interrupts.CPU85.RES:Rescheduling_interrupts
60518 ? 17% interrupts.CPU86.CAL:Function_call_interrupts
932168 ? 10% interrupts.CPU86.LOC:Local_timer_interrupts
154349 ? 14% interrupts.CPU86.RES:Rescheduling_interrupts
61039 ? 19% interrupts.CPU87.CAL:Function_call_interrupts
932145 ? 10% interrupts.CPU87.LOC:Local_timer_interrupts
149913 ? 19% interrupts.CPU87.RES:Rescheduling_interrupts
62300 ? 10% interrupts.CPU88.CAL:Function_call_interrupts
932276 ? 10% interrupts.CPU88.LOC:Local_timer_interrupts
157364 ? 13% interrupts.CPU88.RES:Rescheduling_interrupts
67200 ? 9% interrupts.CPU89.CAL:Function_call_interrupts
931996 ? 10% interrupts.CPU89.LOC:Local_timer_interrupts
161361 ? 15% interrupts.CPU89.RES:Rescheduling_interrupts
128730 ? 11% interrupts.CPU9.CAL:Function_call_interrupts
935056 ? 10% interrupts.CPU9.LOC:Local_timer_interrupts
348530 ? 10% interrupts.CPU9.RES:Rescheduling_interrupts
68227 ? 19% interrupts.CPU90.CAL:Function_call_interrupts
932106 ? 10% interrupts.CPU90.LOC:Local_timer_interrupts
165869 ? 17% interrupts.CPU90.RES:Rescheduling_interrupts
65189 ? 21% interrupts.CPU91.CAL:Function_call_interrupts
932316 ? 10% interrupts.CPU91.LOC:Local_timer_interrupts
157890 ? 16% interrupts.CPU91.RES:Rescheduling_interrupts
75411 ? 26% interrupts.CPU92.CAL:Function_call_interrupts
932121 ? 10% interrupts.CPU92.LOC:Local_timer_interrupts
186532 ? 22% interrupts.CPU92.RES:Rescheduling_interrupts
62752 ? 15% interrupts.CPU93.CAL:Function_call_interrupts
932090 ? 10% interrupts.CPU93.LOC:Local_timer_interrupts
158788 ? 14% interrupts.CPU93.RES:Rescheduling_interrupts
66731 ? 19% interrupts.CPU94.CAL:Function_call_interrupts
932096 ? 10% interrupts.CPU94.LOC:Local_timer_interrupts
164911 ? 15% interrupts.CPU94.RES:Rescheduling_interrupts
72276 ? 24% interrupts.CPU95.CAL:Function_call_interrupts
932068 ? 10% interrupts.CPU95.LOC:Local_timer_interrupts
173807 ? 25% interrupts.CPU95.RES:Rescheduling_interrupts
2602 ? 17% interrupts.IWI:IRQ_work_interrupts
+338.0% 89624783 ? 10% interrupts.LOC:Local_timer_interrupts
+452.0% 21654414 ? 9% interrupts.RES:Rescheduling_interrupts
108.78 ? 15% interrupts.TLB:TLB_shootdowns
3.47 ? 17% perf-profile.calltrace.cycles-pp._raw_spin_lock.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.49 ? 20% perf-profile.calltrace.cycles-pp._raw_spin_lock.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
46.32 perf-profile.calltrace.cycles-pp.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.64 ? 18% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.95 ? 21% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
96.02 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
96.34 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
1.48 ? 6% perf-profile.calltrace.cycles-pp.ksys_msgctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.30 ? 5% perf-profile.calltrace.cycles-pp.msgctl_info.ksys_msgctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.87 ? 5% perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.__kmalloc.load_msg.do_msgsnd.do_syscall_64
0.71 ? 2% perf-profile.calltrace.cycles-pp.store_msg.do_msg_fill.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.05 perf-profile.calltrace.cycles-pp.do_msg_fill.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.43 ? 13% perf-profile.calltrace.cycles-pp.__mod_memcg_state.__mod_memcg_lruvec_state.kfree.free_msg.do_msgrcv
1.43 ? 12% perf-profile.calltrace.cycles-pp.__mod_memcg_state.__mod_memcg_lruvec_state.__kmalloc.load_msg.do_msgsnd
1.50 ? 12% perf-profile.calltrace.cycles-pp.__mod_memcg_lruvec_state.kfree.free_msg.do_msgrcv.do_syscall_64
1.51 ? 11% perf-profile.calltrace.cycles-pp.__mod_memcg_lruvec_state.__kmalloc.load_msg.do_msgsnd.do_syscall_64
4.04 ? 7% perf-profile.calltrace.cycles-pp.__kmalloc.load_msg.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.60 ? 3% perf-profile.calltrace.cycles-pp.wake_q_add.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.62 ? 5% perf-profile.calltrace.cycles-pp.pick_next_task_fair.pick_next_task.__schedule.schedule.do_msgrcv
3.20 ? 9% perf-profile.calltrace.cycles-pp.kfree.free_msg.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.70 ? 7% perf-profile.calltrace.cycles-pp.load_msg.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.66 ? 8% perf-profile.calltrace.cycles-pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.wake_up_q
0.93 ? 7% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.wake_up_q.do_msgsnd
0.74 ? 5% perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.08 ? 6% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.wake_up_q.do_msgsnd.do_syscall_64
0.76 ? 4% perf-profile.calltrace.cycles-pp.pick_next_task.__schedule.schedule.do_msgrcv.do_syscall_64
0.84 ? 3% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.do_msgrcv.do_syscall_64
0.85 ? 3% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode
0.98 ? 6% perf-profile.calltrace.cycles-pp.pick_next_task_fair.pick_next_task.__schedule.schedule.exit_to_user_mode_prepare
1.07 ? 5% perf-profile.calltrace.cycles-pp.pick_next_task.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode
4.66 ? 6% perf-profile.calltrace.cycles-pp.__get_user_nocheck_8.perf_callchain_user.get_perf_callchain.perf_callchain.perf_prepare_sample
4.89 ? 7% perf-profile.calltrace.cycles-pp.unwind_next_frame.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
6.03 ? 7% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template
6.05 ? 7% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.ttwu_do_wakeup
6.06 ? 7% perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.ttwu_do_wakeup.try_to_wake_up
6.14 ? 7% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.do_msgrcv
6.37 ? 7% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.do_msgrcv.do_syscall_64
6.29 ? 7% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.exit_to_user_mode_prepare
6.32 ? 7% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.ttwu_do_wakeup.try_to_wake_up.wake_up_q
6.45 ? 7% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.ttwu_do_wakeup.try_to_wake_up.wake_up_q.do_msgsnd
6.53 ? 7% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode
6.74 ? 6% perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
6.90 ? 6% perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.wake_up_q.do_msgsnd.do_syscall_64
6.92 ? 6% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
6.91 ? 7% perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.enqueue_entity
7.07 ? 6% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair.__schedule
7.08 ? 7% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.enqueue_entity.enqueue_task_fair
7.24 ? 7% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate
7.44 ? 6% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
7.55 ? 7% perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
7.78 ? 6% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.do_msgrcv
8.00 ? 7% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.wake_up_q
8.57 ? 6% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.do_msgrcv.do_syscall_64
8.77 ? 7% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.wake_up_q.do_msgsnd
8.82 ? 7% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.wake_up_q.do_msgsnd.do_syscall_64
9.76 ? 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
9.84 ? 6% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
10.73 ? 6% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
8.68 ? 7% perf-profile.calltrace.cycles-pp.unwind_next_frame.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain
9.52 ? 7% perf-profile.calltrace.cycles-pp.perf_callchain_user.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
9.92 ? 7% perf-profile.calltrace.cycles-pp.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
12.03 ? 7% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch
12.09 ? 7% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
12.11 ? 7% perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
10.90 ? 7% perf-profile.calltrace.cycles-pp.asm_exc_page_fault.__get_user_nocheck_8.perf_callchain_user.get_perf_callchain.perf_callchain
13.56 ? 7% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime
13.63 ? 7% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
17.81 ? 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
18.05 ? 6% perf-profile.calltrace.cycles-pp.wake_up_q.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
17.97 ? 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
18.06 ? 6% perf-profile.calltrace.cycles-pp.schedule.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
17.94 ? 7% perf-profile.calltrace.cycles-pp.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
28.23 ? 7% perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
28.34 ? 7% perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow
30.04 ? 7% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event
6.80 ? 16% perf-profile.children.cycles-pp._raw_spin_lock
4.74 ? 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
46.35 perf-profile.children.cycles-pp.do_msgsnd
96.87 perf-profile.children.cycles-pp.do_syscall_64
97.18 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
1.49 ? 6% perf-profile.children.cycles-pp.ksys_msgctl
0.26 ? 13% perf-profile.children.cycles-pp.__slab_free
1.30 ? 5% perf-profile.children.cycles-pp.msgctl_info
0.93 ? 3% perf-profile.children.cycles-pp.ipc_obtain_object_check
0.15 ? 15% perf-profile.children.cycles-pp.msgctl_stat
0.07 ? 7% perf-profile.children.cycles-pp.___slab_alloc
0.09 ? 5% perf-profile.children.cycles-pp.obj_cgroup_charge_pages
0.07 ? 9% perf-profile.children.cycles-pp.__slab_alloc
0.06 ? 10% perf-profile.children.cycles-pp.page_counter_try_charge
0.14 ? 4% perf-profile.children.cycles-pp.obj_cgroup_charge
0.12 ? 10% perf-profile.children.cycles-pp.down_read
0.11 ? 4% perf-profile.children.cycles-pp.drain_obj_stock
0.18 ? 3% perf-profile.children.cycles-pp.refill_obj_stock
0.08 ? 12% perf-profile.children.cycles-pp.up_read
0.08 ? 7% perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
0.04 ? 35% perf-profile.children.cycles-pp.page_counter_uncharge
0.11 ? 5% perf-profile.children.cycles-pp.__list_del_entry_valid
0.07 ? 22% perf-profile.children.cycles-pp.shmem_write_end
0.42 ? 4% perf-profile.children.cycles-pp.task_tick_fair
0.11 ? 7% perf-profile.children.cycles-pp.ipcperms
0.09 ? 10% perf-profile.children.cycles-pp.__cond_resched
0.08 ? 17% perf-profile.children.cycles-pp.__mod_lruvec_page_state
0.16 ? 3% perf-profile.children.cycles-pp.__might_sleep
0.07 ? 4% perf-profile.children.cycles-pp.security_msg_msg_free
0.13 ? 3% perf-profile.children.cycles-pp.__get_user_8
0.21 ? 2% perf-profile.children.cycles-pp.___might_sleep
0.16 ? 4% perf-profile.children.cycles-pp.copy_user_generic_unrolled
0.12 ? 15% perf-profile.children.cycles-pp.mem_cgroup_charge
0.22 ? 3% perf-profile.children.cycles-pp.__entry_text_start
0.06 ? 7% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.05 ? 7% perf-profile.children.cycles-pp.__list_add_valid
0.23 ? 13% perf-profile.children.cycles-pp.shmem_add_to_page_cache
0.06 ? 11% perf-profile.children.cycles-pp.local_clock
0.06 ? 11% perf-profile.children.cycles-pp.__rdgsbase_inactive
0.11 ? 5% perf-profile.children.cycles-pp.resched_curr
0.06 ? 5% perf-profile.children.cycles-pp.__x64_sys_msgrcv
0.28 ? 3% perf-profile.children.cycles-pp.__check_heap_object
0.38 ? 13% perf-profile.children.cycles-pp.shmem_getpage_gfp
0.07 ? 10% perf-profile.children.cycles-pp.rb_erase
0.07 ? 7% perf-profile.children.cycles-pp.__bad_area_nosemaphore
0.18 ? 9% perf-profile.children.cycles-pp.do_user_addr_fault
0.38 ? 13% perf-profile.children.cycles-pp.shmem_write_begin
0.07 ? 10% perf-profile.children.cycles-pp.ex_handler_uaccess
0.07 ? 10% perf-profile.children.cycles-pp.cpumask_next
0.07 ? 7% perf-profile.children.cycles-pp.__x64_sys_msgsnd
0.07 ? 8% perf-profile.children.cycles-pp.rb_insert_color
0.07 ? 9% perf-profile.children.cycles-pp.in_task_stack
0.19 ? 6% perf-profile.children.cycles-pp.__virt_addr_valid
0.07 ? 6% perf-profile.children.cycles-pp.__enqueue_entity
0.27 ? 2% perf-profile.children.cycles-pp._copy_to_user
0.08 ? 8% perf-profile.children.cycles-pp.is_module_text_address
0.25 ? 3% perf-profile.children.cycles-pp._copy_from_user
0.12 ? 9% perf-profile.children.cycles-pp.cpumask_next_wrap
0.09 ? 11% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.25 ? 4% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.15 ? 5% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.09 ? 6% perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
0.88 ? 5% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.10 ? 14% perf-profile.children.cycles-pp.rcu_is_watching
0.10 ? 8% perf-profile.children.cycles-pp.clear_buddies
0.11 ? 8% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.28 ? 8% perf-profile.children.cycles-pp.available_idle_cpu
0.11 ? 8% perf-profile.children.cycles-pp.perf_trace_run_bpf_submit
0.32 ? 4% perf-profile.children.cycles-pp.ksys_msgsnd
0.11 ? 7% perf-profile.children.cycles-pp._find_next_bit
0.30 ? 8% perf-profile.children.cycles-pp.__radix_tree_lookup
0.12 ? 6% perf-profile.children.cycles-pp.update_min_vruntime
0.12 ? 6% perf-profile.children.cycles-pp.wakeup_preempt_entity
0.12 ? 7% perf-profile.children.cycles-pp.ftrace_ops_trampoline
0.47 perf-profile.children.cycles-pp.syscall_return_via_sysret
0.13 ? 10% perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
0.13 ? 7% perf-profile.children.cycles-pp.is_ftrace_trampoline
0.13 ? 11% perf-profile.children.cycles-pp.perf_instruction_pointer
0.13 ? 12% perf-profile.children.cycles-pp.set_next_buddy
0.14 ? 9% perf-profile.children.cycles-pp.get_stack_info_noinstr
0.18 ? 6% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.72 ? 2% perf-profile.children.cycles-pp.store_msg
0.45 ? 2% perf-profile.children.cycles-pp.__might_fault
0.23 ? 3% perf-profile.children.cycles-pp.update_rq_clock
0.17 ? 8% perf-profile.children.cycles-pp.__calc_delta
0.24 ? 6% perf-profile.children.cycles-pp.___perf_sw_event
0.18 ? 6% perf-profile.children.cycles-pp.perf_misc_flags
0.20 ? 10% perf-profile.children.cycles-pp.cpuacct_charge
0.71 ? 3% perf-profile.children.cycles-pp.__check_object_size
0.68 ? 3% perf-profile.children.cycles-pp.wake_q_add
0.28 ? 6% perf-profile.children.cycles-pp.update_cfs_group
0.22 ? 6% perf-profile.children.cycles-pp.__is_insn_slot_addr
0.19 ? 12% perf-profile.children.cycles-pp.get_callchain_entry
0.26 ? 6% perf-profile.children.cycles-pp.get_stack_info
0.22 ? 9% perf-profile.children.cycles-pp.perf_trace_buf_update
0.22 ? 18% perf-profile.children.cycles-pp.bpf_ksym_find
0.25 ? 8% perf-profile.children.cycles-pp.perf_trace_buf_alloc
0.29 ? 7% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.21 ? 8% perf-profile.children.cycles-pp.kvm_is_in_guest
0.31 ? 4% perf-profile.children.cycles-pp.pick_next_entity
0.32 ? 5% perf-profile.children.cycles-pp.check_preempt_wakeup
0.30 ? 6% perf-profile.children.cycles-pp.__update_load_avg_se
0.49 ? 16% perf-profile.children.cycles-pp.memcpy_erms
0.36 ? 6% perf-profile.children.cycles-pp.check_preempt_curr
0.67 ? 8% perf-profile.children.cycles-pp.select_idle_cpu
0.32 ? 12% perf-profile.children.cycles-pp.is_bpf_text_address
0.58 ? 15% perf-profile.children.cycles-pp.perf_output_copy
0.29 ? 7% perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
1.06 perf-profile.children.cycles-pp.do_msg_fill
0.34 ? 7% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.47 ? 4% perf-profile.children.cycles-pp.finish_task_switch
0.36 ? 6% perf-profile.children.cycles-pp.put_prev_entity
0.54 ? 5% perf-profile.children.cycles-pp.__switch_to
0.29 ? 13% perf-profile.children.cycles-pp.perf_tp_event_match
0.37 ? 8% perf-profile.children.cycles-pp.bad_get_user
0.73 ? 2% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
0.43 ? 6% perf-profile.children.cycles-pp.native_sched_clock
0.83 ? 15% perf-profile.children.cycles-pp.perf_output_sample
0.43 ? 8% perf-profile.children.cycles-pp.__switch_to_asm
0.42 ? 8% perf-profile.children.cycles-pp.ftrace_graph_ret_addr
0.48 ? 7% perf-profile.children.cycles-pp.perf_event_pid_type
0.50 ? 6% perf-profile.children.cycles-pp.sched_clock_cpu
0.54 ? 7% perf-profile.children.cycles-pp.reweight_entity
0.55 ? 5% perf-profile.children.cycles-pp.set_next_entity
0.95 ? 7% perf-profile.children.cycles-pp.select_idle_sibling
0.59 ? 8% perf-profile.children.cycles-pp.perf_output_begin_forward
4.07 ? 7% perf-profile.children.cycles-pp.__kmalloc
0.65 ? 7% perf-profile.children.cycles-pp.load_new_mm_cr3
0.75 ? 5% perf-profile.children.cycles-pp.switch_fpu_return
1.08 ? 6% perf-profile.children.cycles-pp.select_task_rq_fair
3.26 ? 9% perf-profile.children.cycles-pp.kfree
4.71 ? 7% perf-profile.children.cycles-pp.load_msg
2.93 ? 12% perf-profile.children.cycles-pp.__mod_memcg_state
3.09 ? 11% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.97 ? 7% perf-profile.children.cycles-pp.__perf_event_header__init_id
0.99 ? 7% perf-profile.children.cycles-pp.cmp_ex_search
1.12 ? 7% perf-profile.children.cycles-pp.update_load_avg
1.71 ? 3% perf-profile.children.cycles-pp.switch_mm_irqs_off
1.37 ? 7% perf-profile.children.cycles-pp.kernel_text_address
1.67 ? 5% perf-profile.children.cycles-pp.pick_next_task_fair
1.59 ? 7% perf-profile.children.cycles-pp.stack_access_ok
1.65 ? 7% perf-profile.children.cycles-pp.__kernel_text_address
1.68 ? 6% perf-profile.children.cycles-pp.native_irq_return_iret
1.61 ? 7% perf-profile.children.cycles-pp.bsearch
1.69 ? 7% perf-profile.children.cycles-pp.search_extable
1.85 ? 5% perf-profile.children.cycles-pp.pick_next_task
1.74 ? 7% perf-profile.children.cycles-pp.search_exception_tables
1.88 ? 7% perf-profile.children.cycles-pp.fixup_exception
2.02 ? 7% perf-profile.children.cycles-pp.unwind_get_return_address
2.03 ? 7% perf-profile.children.cycles-pp.kernelmode_fixup_or_oops
2.18 ? 7% perf-profile.children.cycles-pp.orc_find
2.47 ? 6% perf-profile.children.cycles-pp.exc_page_fault
3.56 ? 7% perf-profile.children.cycles-pp.__orc_find
6.51 ? 6% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
6.97 ? 6% perf-profile.children.cycles-pp.ttwu_do_wakeup
7.01 ? 7% perf-profile.children.cycles-pp.asm_exc_page_fault
7.82 ? 6% perf-profile.children.cycles-pp.dequeue_entity
8.08 ? 7% perf-profile.children.cycles-pp.enqueue_entity
8.62 ? 6% perf-profile.children.cycles-pp.dequeue_task_fair
8.84 ? 7% perf-profile.children.cycles-pp.enqueue_task_fair
8.89 ? 7% perf-profile.children.cycles-pp.ttwu_do_activate
9.18 ? 7% perf-profile.children.cycles-pp.__get_user_nocheck_8
9.66 ? 7% perf-profile.children.cycles-pp.perf_callchain_user
10.75 ? 6% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
10.17 ? 7% perf-profile.children.cycles-pp.__unwind_start
12.97 ? 7% perf-profile.children.cycles-pp.perf_trace_sched_switch
14.73 ? 7% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
14.32 ? 7% perf-profile.children.cycles-pp.unwind_next_frame
15.84 ? 7% perf-profile.children.cycles-pp.update_curr
17.91 ? 6% perf-profile.children.cycles-pp.try_to_wake_up
18.16 ? 6% perf-profile.children.cycles-pp.wake_up_q
18.38 ? 7% perf-profile.children.cycles-pp.perf_callchain_kernel
27.96 ? 6% perf-profile.children.cycles-pp.__schedule
27.99 ? 6% perf-profile.children.cycles-pp.schedule
28.68 ? 7% perf-profile.children.cycles-pp.get_perf_callchain
28.79 ? 7% perf-profile.children.cycles-pp.perf_callchain
30.54 ? 7% perf-profile.children.cycles-pp.perf_prepare_sample
32.13 ? 7% perf-profile.children.cycles-pp.perf_event_output_forward
32.28 ? 7% perf-profile.children.cycles-pp.__perf_event_overflow
32.33 ? 7% perf-profile.children.cycles-pp.perf_swevent_overflow
33.29 ? 7% perf-profile.children.cycles-pp.perf_tp_event
19.69 ? 6% perf-profile.self.cycles-pp.do_msgsnd
9.61 ? 10% perf-profile.self.cycles-pp.do_msgrcv
4.69 ? 19% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
2.06 ? 9% perf-profile.self.cycles-pp._raw_spin_lock
0.26 ? 13% perf-profile.self.cycles-pp.__slab_free
1.08 ? 5% perf-profile.self.cycles-pp.msgctl_info
0.23 ? 9% perf-profile.self.cycles-pp.ipc_obtain_object_check
0.06 ? 10% perf-profile.self.cycles-pp.page_counter_try_charge
0.12 ? 12% perf-profile.self.cycles-pp.down_read
0.08 ? 12% perf-profile.self.cycles-pp.up_read
0.10 ? 6% perf-profile.self.cycles-pp.__list_del_entry_valid
0.10 ? 6% perf-profile.self.cycles-pp.ipcperms
0.20 ? 4% perf-profile.self.cycles-pp.__check_object_size
0.13 ? 3% perf-profile.self.cycles-pp.__get_user_8
0.21 ? 3% perf-profile.self.cycles-pp.___might_sleep
0.15 ? 3% perf-profile.self.cycles-pp.copy_user_generic_unrolled
0.15 ? 13% perf-profile.self.cycles-pp.perf_output_sample
0.22 ? 3% perf-profile.self.cycles-pp.__entry_text_start
0.06 ? 8% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.09 ? 11% perf-profile.self.cycles-pp.ttwu_do_wakeup
0.05 ? 6% perf-profile.self.cycles-pp.__list_add_valid
0.05 ? 7% perf-profile.self.cycles-pp.perf_swevent_overflow
0.11 ? 4% perf-profile.self.cycles-pp.resched_curr
0.05 ? 12% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.06 ? 8% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.06 ? 11% perf-profile.self.cycles-pp.search_exception_tables
0.06 ? 12% perf-profile.self.cycles-pp.ex_handler_uaccess
0.06 ? 9% perf-profile.self.cycles-pp.__rdgsbase_inactive
0.06 ? 7% perf-profile.self.cycles-pp.__x64_sys_msgrcv
0.06 ? 10% perf-profile.self.cycles-pp.in_task_stack
0.06 ? 10% perf-profile.self.cycles-pp.rb_erase
0.15 ? 6% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.06 ? 10% perf-profile.self.cycles-pp.sched_clock_cpu
0.07 ? 4% perf-profile.self.cycles-pp.do_msg_fill
0.06 ? 10% perf-profile.self.cycles-pp.cpumask_next_wrap
0.28 ? 3% perf-profile.self.cycles-pp.__check_heap_object
0.07 ? 14% perf-profile.self.cycles-pp.__cgroup_account_cputime
0.07 ? 10% perf-profile.self.cycles-pp.get_stack_info_noinstr
0.07 ? 7% perf-profile.self.cycles-pp.__bad_area_nosemaphore
0.07 ? 7% perf-profile.self.cycles-pp.__x64_sys_msgsnd
0.18 ? 6% perf-profile.self.cycles-pp.__virt_addr_valid
0.07 ? 7% perf-profile.self.cycles-pp.__enqueue_entity
0.07 ? 8% perf-profile.self.cycles-pp.rb_insert_color
0.18 ? 16% perf-profile.self.cycles-pp.perf_output_copy
0.08 ? 12% perf-profile.self.cycles-pp.search_extable
0.81 ? 4% perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.08 ? 12% perf-profile.self.cycles-pp.is_bpf_text_address
0.08 ? 8% perf-profile.self.cycles-pp.fixup_exception
0.15 ? 5% perf-profile.self.cycles-pp.__might_fault
0.09 ? 8% perf-profile.self.cycles-pp.put_prev_entity
0.09 ? 7% perf-profile.self.cycles-pp.perf_instruction_pointer
0.25 ? 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.09 ? 9% perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
0.09 ? 14% perf-profile.self.cycles-pp.perf_trace_buf_update
0.10 ? 7% perf-profile.self.cycles-pp.clear_buddies
0.13 ? 9% perf-profile.self.cycles-pp.dequeue_entity
0.16 ? 5% perf-profile.self.cycles-pp.do_syscall_64
0.24 ? 5% perf-profile.self.cycles-pp.wake_up_q
0.10 ? 16% perf-profile.self.cycles-pp.rcu_is_watching
0.27 ? 8% perf-profile.self.cycles-pp.available_idle_cpu
0.10 ? 5% perf-profile.self.cycles-pp.update_rq_clock
0.10 ? 7% perf-profile.self.cycles-pp.ftrace_ops_trampoline
0.10 ? 9% perf-profile.self.cycles-pp.perf_trace_run_bpf_submit
0.10 ? 7% perf-profile.self.cycles-pp._find_next_bit
0.11 ? 6% perf-profile.self.cycles-pp.rcu_read_unlock_strict
0.11 ? 11% perf-profile.self.cycles-pp.__perf_event_overflow
0.11 ? 8% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.11 ? 12% perf-profile.self.cycles-pp.perf_callchain
0.29 ? 8% perf-profile.self.cycles-pp.__radix_tree_lookup
0.11 ? 8% perf-profile.self.cycles-pp.wakeup_preempt_entity
0.11 ? 8% perf-profile.self.cycles-pp.perf_misc_flags
0.13 ? 9% perf-profile.self.cycles-pp.do_user_addr_fault
0.12 ? 9% perf-profile.self.cycles-pp.update_min_vruntime
0.12 ? 8% perf-profile.self.cycles-pp.schedule
0.46 perf-profile.self.cycles-pp.syscall_return_via_sysret
0.12 ? 12% perf-profile.self.cycles-pp.set_next_buddy
0.12 ? 7% perf-profile.self.cycles-pp.get_stack_info
0.12 ? 6% perf-profile.self.cycles-pp.perf_event_output_forward
0.12 ? 9% perf-profile.self.cycles-pp.perf_event_pid_type
0.12 ? 10% perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
0.13 ? 7% perf-profile.self.cycles-pp.check_preempt_wakeup
0.13 ? 8% perf-profile.self.cycles-pp.select_task_rq_fair
0.14 ? 10% perf-profile.self.cycles-pp.finish_task_switch
0.20 ? 4% perf-profile.self.cycles-pp.pick_next_entity
0.21 ? 6% perf-profile.self.cycles-pp.___perf_sw_event
0.15 ? 9% perf-profile.self.cycles-pp.perf_trace_buf_alloc
0.15 ? 7% perf-profile.self.cycles-pp.select_idle_sibling
0.15 ? 7% perf-profile.self.cycles-pp.exc_page_fault
0.15 ? 6% perf-profile.self.cycles-pp.set_next_entity
0.18 ? 6% perf-profile.self.cycles-pp.pick_next_task
0.15 ? 8% perf-profile.self.cycles-pp.dequeue_task_fair
0.15 ? 9% perf-profile.self.cycles-pp.__perf_event_header__init_id
0.16 ? 4% perf-profile.self.cycles-pp.kernelmode_fixup_or_oops
0.16 ? 6% perf-profile.self.cycles-pp.__calc_delta
0.31 ? 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.23 ? 6% perf-profile.self.cycles-pp.reweight_entity
0.19 ? 10% perf-profile.self.cycles-pp.enqueue_entity
0.35 ? 3% perf-profile.self.cycles-pp.try_to_wake_up
0.19 ? 11% perf-profile.self.cycles-pp.cpuacct_charge
0.17 ? 7% perf-profile.self.cycles-pp.__is_insn_slot_addr
0.67 ? 3% perf-profile.self.cycles-pp.wake_q_add
0.19 ? 11% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.18 ? 11% perf-profile.self.cycles-pp.get_callchain_entry
0.18 ? 9% perf-profile.self.cycles-pp.kvm_is_in_guest
0.28 ? 5% perf-profile.self.cycles-pp.update_cfs_group
0.27 ? 6% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.22 ? 17% perf-profile.self.cycles-pp.bpf_ksym_find
0.24 ? 7% perf-profile.self.cycles-pp.enqueue_task_fair
0.27 ? 7% perf-profile.self.cycles-pp.asm_exc_page_fault
0.26 ? 9% perf-profile.self.cycles-pp.__kernel_text_address
0.30 ? 5% perf-profile.self.cycles-pp.__update_load_avg_se
0.48 ? 16% perf-profile.self.cycles-pp.memcpy_erms
0.35 ? 8% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.32 ? 7% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.33 ? 5% perf-profile.self.cycles-pp.pick_next_task_fair
0.32 ? 8% perf-profile.self.cycles-pp.perf_callchain_user
0.50 ? 5% perf-profile.self.cycles-pp.__switch_to
0.28 ? 6% perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
0.29 ? 13% perf-profile.self.cycles-pp.perf_tp_event_match
0.33 ? 9% perf-profile.self.cycles-pp.ftrace_graph_ret_addr
1.30 ? 9% perf-profile.self.cycles-pp.__kmalloc
0.37 ? 8% perf-profile.self.cycles-pp.unwind_get_return_address
0.42 ? 7% perf-profile.self.cycles-pp.perf_tp_event
0.42 ? 6% perf-profile.self.cycles-pp.native_sched_clock
0.42 ? 7% perf-profile.self.cycles-pp.__switch_to_asm
1.54 ? 8% perf-profile.self.cycles-pp.kfree
0.49 ? 6% perf-profile.self.cycles-pp.perf_prepare_sample
0.44 ? 8% perf-profile.self.cycles-pp.get_perf_callchain
1.04 ? 2% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.53 ? 9% perf-profile.self.cycles-pp.update_load_avg
0.52 ? 6% perf-profile.self.cycles-pp.kernel_text_address
0.54 ? 8% perf-profile.self.cycles-pp.update_curr
0.56 ? 7% perf-profile.self.cycles-pp.perf_output_begin_forward
0.65 ? 7% perf-profile.self.cycles-pp.load_new_mm_cr3
0.65 ? 8% perf-profile.self.cycles-pp.bsearch
0.74 ? 5% perf-profile.self.cycles-pp.switch_fpu_return
0.95 ? 5% perf-profile.self.cycles-pp.__schedule
2.93 ? 12% perf-profile.self.cycles-pp.__mod_memcg_state
0.89 ? 7% perf-profile.self.cycles-pp.__unwind_start
0.95 ? 7% perf-profile.self.cycles-pp.cmp_ex_search
1.12 ? 6% perf-profile.self.cycles-pp.perf_callchain_kernel
1.37 ? 7% perf-profile.self.cycles-pp.stack_access_ok
1.67 ? 6% perf-profile.self.cycles-pp.native_irq_return_iret
2.06 ? 7% perf-profile.self.cycles-pp.orc_find
3.55 ? 7% perf-profile.self.cycles-pp.__orc_find
4.58 ? 7% perf-profile.self.cycles-pp.__get_user_nocheck_8
6.84 ? 7% perf-profile.self.cycles-pp.unwind_next_frame

uite.stress-ng.SystemVMessagePassing.bogo_ops_s

-------------------------------------------------+
|
|
|
O O OO O OO OO O OO O OO O OO OO |
O O O O O O O |
|
|
|
|
|
|
+.++.++.++.++.++.++.++.++.++.++.++.++.+ .++.++.++.++. +.+|
+ + |
-------------------------------------------------+

phoronix-test-suite.time.voluntary_context_switches

-------------------------------------------------+
OO OO O O OO O O O |
O O |
O O |
O O O OO O |
|
|
|
|
|
|
|
|
.+ +. |
-------------------------------------------------+

est-suite.time.involuntary_context_switches

-------------------------------------------------+
OO OO O O OO O O O |
O O |
O O |
O O O OO O |
|
|
|
|
|
|
|
|
.+ +. |
-------------------------------------------------+

estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
may affect actual performance.
Open Source Technology Center
org/hyperkitty/list/lkp@lists.01.org">https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation

Attachments:

(No filename) (101.07 kB)
config-5.13.0-rc4-00057-g5359f5ca0f37 (176.86 kB)
job-script (7.69 kB)
job.yaml (5.00 kB)
reproduce (311.00 B)
Download all attachments