LinuxLists.cc - [PATCH v7 0/4] lockinig/rwsem: Fix rwsem bugs & enable true lock handoff

2023-02-13 12:32:29

by Peter Zijlstra

[permalink] [raw]

Subject: Re: [PATCH v7 4/4] locking/rwsem: Enable direct rwsem lock handoff

On Wed, Jan 25, 2023 at 07:36:28PM -0500, Waiman Long wrote:

> @@ -609,6 +618,12 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem,
>
> lockdep_assert_held(&sem->wait_lock);
>
> + if (!waiter->task) {
> + /* Write lock handed off */
> + smp_acquire__after_ctrl_dep();
> + return true;
> + }
> +
> count = atomic_long_read(&sem->count);
> do {
> bool has_handoff = !!(count & RWSEM_FLAG_HANDOFF);
> @@ -754,6 +769,10 @@ rwsem_spin_on_owner(struct rw_semaphore *sem)
>
> owner = rwsem_owner_flags(sem, &flags);
> state = rwsem_owner_state(owner, flags);
> +
> + if (owner == current)
> + return OWNER_NONSPINNABLE; /* Handoff granted */
> +
> if (state != OWNER_WRITER)
> return state;
>
> @@ -1168,21 +1186,23 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)
> * without sleeping.
> */
> if (waiter.handoff_set) {
> - enum owner_state owner_state;
> -
> - owner_state = rwsem_spin_on_owner(sem);
> - if (owner_state == OWNER_NULL)
> - goto trylock_again;
> + rwsem_spin_on_owner(sem);
> + if (!READ_ONCE(waiter.task)) {
> + /* Write lock handed off */
> + smp_acquire__after_ctrl_dep();
> + set_current_state(TASK_RUNNING);
> + goto out;
> + }
> }
>
> schedule_preempt_disabled();
> lockevent_inc(rwsem_sleep_writer);
> set_current_state(state);
> -trylock_again:
> raw_spin_lock_irq(&sem->wait_lock);
> }
> __set_current_state(TASK_RUNNING);
> raw_spin_unlock_irq(&sem->wait_lock);
> +out:
> lockevent_inc(rwsem_wlock);
> trace_contention_end(sem, 0);
> return sem;
> @@ -1190,6 +1210,11 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)
> out_nolock:
> __set_current_state(TASK_RUNNING);
> raw_spin_lock_irq(&sem->wait_lock);
> + if (!waiter.task) {
> + smp_acquire__after_ctrl_dep();
> + raw_spin_unlock_irq(&sem->wait_lock);
> + goto out;
> + }
> rwsem_del_wake_waiter(sem, &waiter, &wake_q);
> lockevent_inc(rwsem_wlock_fail);
> trace_contention_end(sem, -EINTR);
> @@ -1202,14 +1227,41 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)
> */
> static struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
> {
> - unsigned long flags;
> DEFINE_WAKE_Q(wake_q);
> + unsigned long flags;
> + unsigned long count;
>
> raw_spin_lock_irqsave(&sem->wait_lock, flags);
>
> - if (!list_empty(&sem->wait_list))
> - rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
> + if (list_empty(&sem->wait_list))
> + goto unlock_out;
> +
> + /*
> + * If the rwsem is free and handoff flag is set with wait_lock held,
> + * no other CPUs can take an active lock.
> + */
> + count = atomic_long_read(&sem->count);
> + if (!(count & RWSEM_LOCK_MASK) && (count & RWSEM_FLAG_HANDOFF)) {
> + /*
> + * Since rwsem_mark_wake() will handle the handoff to reader
> + * properly, we don't need to do anything extra for reader.
> + * Special handoff processing will only be needed for writer.
> + */
> + struct rwsem_waiter *waiter = rwsem_first_waiter(sem);
> + long adj = RWSEM_WRITER_LOCKED - RWSEM_FLAG_HANDOFF;
> +
> + if (waiter->type == RWSEM_WAITING_FOR_WRITE) {
> + atomic_long_set(&sem->owner, (long)waiter->task);
> + atomic_long_add(adj, &sem->count);
> + wake_q_add(&wake_q, waiter->task);
> + rwsem_del_waiter(sem, waiter);
> + waiter->task = NULL; /* Signal the handoff */
> + goto unlock_out;
> + }
> + }
> + rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
>
> +unlock_out:
> raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
> wake_up_q(&wake_q);
>

I am once again confused...

*WHY* are you changing the writer wake-up path? The comments added here
don't clarify anything.

If we set handoff, we terminate/disallow the spinning/stealing. The
direct consequence is that the slowpath/wait-list becomes the only way
forward.

Since we don't take wait_lock on up, we fundamentally have a race
condition. But *WHY* do you insist on handling that in rwsem_wake()?
Delaying all that until rwsem_try_write_lock()? Doing so would render
pretty much all of the above pointless, no?

After all, rwsem_mark_wake() already wakes the writer if it is first,
no? Why invent yet another special way to wake up the writer.

Also; and I asked this last time around; why do we care about the
handoff to writer *at*all* ? It is the readers that set HANDOFF.

2023-02-13 17:15:48

by Waiman Long

[permalink] [raw]

Subject: Re: [PATCH v7 4/4] locking/rwsem: Enable direct rwsem lock handoff

On 2/13/23 07:31, Peter Zijlstra wrote:
> On Wed, Jan 25, 2023 at 07:36:28PM -0500, Waiman Long wrote:
>
>> @@ -609,6 +618,12 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem,
>>
>> lockdep_assert_held(&sem->wait_lock);
>>
>> + if (!waiter->task) {
>> + /* Write lock handed off */
>> + smp_acquire__after_ctrl_dep();
>> + return true;
>> + }
>> +
>> count = atomic_long_read(&sem->count);
>> do {
>> bool has_handoff = !!(count & RWSEM_FLAG_HANDOFF);
>> @@ -754,6 +769,10 @@ rwsem_spin_on_owner(struct rw_semaphore *sem)
>>
>> owner = rwsem_owner_flags(sem, &flags);
>> state = rwsem_owner_state(owner, flags);
>> +
>> + if (owner == current)
>> + return OWNER_NONSPINNABLE; /* Handoff granted */
>> +
>> if (state != OWNER_WRITER)
>> return state;
>>
>> @@ -1168,21 +1186,23 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)
>> * without sleeping.
>> */
>> if (waiter.handoff_set) {
>> - enum owner_state owner_state;
>> -
>> - owner_state = rwsem_spin_on_owner(sem);
>> - if (owner_state == OWNER_NULL)
>> - goto trylock_again;
>> + rwsem_spin_on_owner(sem);
>> + if (!READ_ONCE(waiter.task)) {
>> + /* Write lock handed off */
>> + smp_acquire__after_ctrl_dep();
>> + set_current_state(TASK_RUNNING);
>> + goto out;
>> + }
>> }
>>
>> schedule_preempt_disabled();
>> lockevent_inc(rwsem_sleep_writer);
>> set_current_state(state);
>> -trylock_again:
>> raw_spin_lock_irq(&sem->wait_lock);
>> }
>> __set_current_state(TASK_RUNNING);
>> raw_spin_unlock_irq(&sem->wait_lock);
>> +out:
>> lockevent_inc(rwsem_wlock);
>> trace_contention_end(sem, 0);
>> return sem;
>> @@ -1190,6 +1210,11 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)
>> out_nolock:
>> __set_current_state(TASK_RUNNING);
>> raw_spin_lock_irq(&sem->wait_lock);
>> + if (!waiter.task) {
>> + smp_acquire__after_ctrl_dep();
>> + raw_spin_unlock_irq(&sem->wait_lock);
>> + goto out;
>> + }
>> rwsem_del_wake_waiter(sem, &waiter, &wake_q);
>> lockevent_inc(rwsem_wlock_fail);
>> trace_contention_end(sem, -EINTR);
>> @@ -1202,14 +1227,41 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)
>> */
>> static struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
>> {
>> - unsigned long flags;
>> DEFINE_WAKE_Q(wake_q);
>> + unsigned long flags;
>> + unsigned long count;
>>
>> raw_spin_lock_irqsave(&sem->wait_lock, flags);
>>
>> - if (!list_empty(&sem->wait_list))
>> - rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
>> + if (list_empty(&sem->wait_list))
>> + goto unlock_out;
>> +
>> + /*
>> + * If the rwsem is free and handoff flag is set with wait_lock held,
>> + * no other CPUs can take an active lock.
>> + */
>> + count = atomic_long_read(&sem->count);
>> + if (!(count & RWSEM_LOCK_MASK) && (count & RWSEM_FLAG_HANDOFF)) {
>> + /*
>> + * Since rwsem_mark_wake() will handle the handoff to reader
>> + * properly, we don't need to do anything extra for reader.
>> + * Special handoff processing will only be needed for writer.
>> + */
>> + struct rwsem_waiter *waiter = rwsem_first_waiter(sem);
>> + long adj = RWSEM_WRITER_LOCKED - RWSEM_FLAG_HANDOFF;
>> +
>> + if (waiter->type == RWSEM_WAITING_FOR_WRITE) {
>> + atomic_long_set(&sem->owner, (long)waiter->task);
>> + atomic_long_add(adj, &sem->count);
>> + wake_q_add(&wake_q, waiter->task);
>> + rwsem_del_waiter(sem, waiter);
>> + waiter->task = NULL; /* Signal the handoff */
>> + goto unlock_out;
>> + }
>> + }
>> + rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
>>
>> +unlock_out:
>> raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
>> wake_up_q(&wake_q);
>>
> I am once again confused...
>
> *WHY* are you changing the writer wake-up path? The comments added here
> don't clarify anything.
>
> If we set handoff, we terminate/disallow the spinning/stealing. The
> direct consequence is that the slowpath/wait-list becomes the only way
> forward.
Yes, that is true.
>
> Since we don't take wait_lock on up, we fundamentally have a race
> condition. But *WHY* do you insist on handling that in rwsem_wake()?
> Delaying all that until rwsem_try_write_lock()? Doing so would render
> pretty much all of the above pointless, no?

There is an advantage in doing the handover earlier, if possible. A
reader that comes in between can spoils the takeover of the rwsem in
rwsem_try_write_lock() and cause it to sleep again. Since we will have
to take the wait lock anyway in rwsem_wake(), there isn't much
additional cost to do some additional check.

Note that the kernel test robot had detected a 19.3% improvement of
will-it-scale.per_thread_ops [1] due to this commit. That indicates this
commit is good to have. I am planning to update the commit log to
include that information as well as additional reasoning as discussed here.

[1]
https://lore.kernel.org/lkml/[email protected]/

>
> After all, rwsem_mark_wake() already wakes the writer if it is first,
> no? Why invent yet another special way to wake up the writer.
As I said before, waking up the writer does not mean it can always get
the rwsem on the first rwsem_try_write_lock(). Doing early handoff in
rwsem_wake() can remove that ambiguity.
>
> Also; and I asked this last time around; why do we care about the
> handoff to writer *at*all* ? It is the readers that set HANDOFF.

HANDOFF can happen for both readers and writers. Handoff to writer is
actually more important than to readers.

Cheers,
Longman

2023-02-13 21:52:56

by Peter Zijlstra

[permalink] [raw]

Subject: Re: [PATCH v7 4/4] locking/rwsem: Enable direct rwsem lock handoff

On Mon, Feb 13, 2023 at 12:14:59PM -0500, Waiman Long wrote:

> > I am once again confused...
> >
> > *WHY* are you changing the writer wake-up path? The comments added here
> > don't clarify anything.
> >
> > If we set handoff, we terminate/disallow the spinning/stealing. The
> > direct consequence is that the slowpath/wait-list becomes the only way
> > forward.
> Yes, that is true.
> >
> > Since we don't take wait_lock on up, we fundamentally have a race
> > condition. But *WHY* do you insist on handling that in rwsem_wake()?
> > Delaying all that until rwsem_try_write_lock()? Doing so would render
> > pretty much all of the above pointless, no?
>
> There is an advantage in doing the handover earlier, if possible. A reader
> that comes in between can spoils the takeover of the rwsem in

How ?!? since all the stealing and spinning is out, the wait-list reigns
supreme. A new reader will queue.

Are you worried about the spurious elevation of ->count due to that
uncondition increment on down_read() ? This is always going to be the
case.

> rwsem_try_write_lock() and cause it to sleep again. Since we will have to
> take the wait lock anyway in rwsem_wake(), there isn't much additional cost
> to do some additional check.

There is a complexity cost -- and so far I've not seen a single line of
justification for the added complexity.

> Note that the kernel test robot had detected a 19.3% improvement of
> will-it-scale.per_thread_ops [1] due to this commit. That indicates this
> commit is good to have. I am planning to update the commit log to include
> that information as well as additional reasoning as discussed here.

Seen that; but who's saying a simpler implementation will not also have
those gains. And if not, then we have a clear separation into
functionality and optimization and justifications for it.

But now, we have neither. I'm not saying the patch is wrong -- I'm
saying it is needlessly complicated without justification.

All this stuff is hard enough -- we should really try to keep is as
simple as possible, having two distinct ways to wake up a writer is not
'as simple as possible'.

> > After all, rwsem_mark_wake() already wakes the writer if it is first,
> > no? Why invent yet another special way to wake up the writer.
> As I said before, waking up the writer does not mean it can always get the
> rwsem on the first rwsem_try_write_lock(). Doing early handoff in
> rwsem_wake() can remove that ambiguity.

But what if it is rwsem_down_read_slowpath() that drops ->count to 0 and
does rwsem_mark_wake() and misses your fancy new path?

That is, I'm thinking there's an argument to be had that rwsem_wake() is
fundamentally the wrong place to do anything.

OTOH, you are right in that rwsem_mark_wake() is in a special position;
it *KNOWS* the reader count has hit 0 and can ignore future spurious
increments because by knowing it was 0 it knows they *WILL* fail and
queue (provided WAITERS||HANDOFF) -- but I've not seen you
articulate this point.

(note how rwsem_cond_wake_waiter() relies on exactly this to select
WAKE_ANY)

You are also right in that having these different means of waking
readers and writers is 'confusing'.

But you failed to take things to their natural conclusion -- change the
way we do writer wakeups, all writer wakeups.

So change rwsem_mark_wake()'s writer wakeup and
rwsem_down_write_slowpath() to always so that waiter->task thing that
the readers already do.

It means putting all the logic for waking writers into
rwsem_mark_wake(), but then we can make full use of this 'we hit zero
future readers are temporary noise'. Althought I suspect you might need
to pass count into it, so we can observe the flags at the time 0 was
observed.

Is all that making sense? -- it has been a long day :-)

> > Also; and I asked this last time around; why do we care about the
> > handoff to writer *at*all* ? It is the readers that set HANDOFF.
>
> HANDOFF can happen for both readers and writers. Handoff to writer is
> actually more important than to readers.

Hmm, clearly I missed something. This is rwsem_try_write_lock() setting
it, right?

2023-02-14 02:04:29

by Waiman Long

[permalink] [raw]

Subject: Re: [PATCH v7 4/4] locking/rwsem: Enable direct rwsem lock handoff

On 2/13/23 16:52, Peter Zijlstra wrote:
> On Mon, Feb 13, 2023 at 12:14:59PM -0500, Waiman Long wrote:
>
>>> I am once again confused...
>>>
>>> *WHY* are you changing the writer wake-up path? The comments added here
>>> don't clarify anything.
>>>
>>> If we set handoff, we terminate/disallow the spinning/stealing. The
>>> direct consequence is that the slowpath/wait-list becomes the only way
>>> forward.
>> Yes, that is true.
>>> Since we don't take wait_lock on up, we fundamentally have a race
>>> condition. But *WHY* do you insist on handling that in rwsem_wake()?
>>> Delaying all that until rwsem_try_write_lock()? Doing so would render
>>> pretty much all of the above pointless, no?
>> There is an advantage in doing the handover earlier, if possible. A reader
>> that comes in between can spoils the takeover of the rwsem in
> How ?!? since all the stealing and spinning is out, the wait-list reigns
> supreme. A new reader will queue.
>
> Are you worried about the spurious elevation of ->count due to that
> uncondition increment on down_read() ? This is always going to be the
> case.

Yes, that is what I am talking about.

>
>> rwsem_try_write_lock() and cause it to sleep again. Since we will have to
>> take the wait lock anyway in rwsem_wake(), there isn't much additional cost
>> to do some additional check.
> There is a complexity cost -- and so far I've not seen a single line of
> justification for the added complexity.
>
>> Note that the kernel test robot had detected a 19.3% improvement of
>> will-it-scale.per_thread_ops [1] due to this commit. That indicates this
>> commit is good to have. I am planning to update the commit log to include
>> that information as well as additional reasoning as discussed here.
> Seen that; but who's saying a simpler implementation will not also have
> those gains. And if not, then we have a clear separation into
> functionality and optimization and justifications for it.
>
> But now, we have neither. I'm not saying the patch is wrong -- I'm
> saying it is needlessly complicated without justification.
>
> All this stuff is hard enough -- we should really try to keep is as
> simple as possible, having two distinct ways to wake up a writer is not
> 'as simple as possible'.

OK, a simpler implementation will be to also check for handoff readiness
in rwsem_wake() and set a flag in waiter structure. When the writer
wakes up and calls rwsem_try_write_lock(), it can ignore any spurious
READER_BIAS and take over the lock directly if the flag is set. Will you
be OK with that?

I believe the one of the reasons for the performance improvement
reported by the test robot is that the check for handoff readiness in
both rwsem_wake() and rwsem_try_write_lock() will increase the chance
that a handoff will be successful in the first attempt. Another possible
reason is that writer may not need to take the wait_lock anymore if all
the handoff work is done by the unlocker at rwsem_wake(). That will be
lost if we do a simpler solution as outlined above.

>
>>> After all, rwsem_mark_wake() already wakes the writer if it is first,
>>> no? Why invent yet another special way to wake up the writer.
>> As I said before, waking up the writer does not mean it can always get the
>> rwsem on the first rwsem_try_write_lock(). Doing early handoff in
>> rwsem_wake() can remove that ambiguity.
> But what if it is rwsem_down_read_slowpath() that drops ->count to 0 and
> does rwsem_mark_wake() and misses your fancy new path?

As said above, the current patch allows handoff to be done at both
rwsem_wake() and rwsem_try_write_lock(). I haven't removed the code to
do handoff in rwsem_try_write_lock().

>
> That is, I'm thinking there's an argument to be had that rwsem_wake() is
> fundamentally the wrong place to do anything.
>
> OTOH, you are right in that rwsem_mark_wake() is in a special position;
> it *KNOWS* the reader count has hit 0 and can ignore future spurious
> increments because by knowing it was 0 it knows they *WILL* fail and
> queue (provided WAITERS||HANDOFF) -- but I've not seen you
> articulate this point.
My bad, I sometimes miss stating assumptions and rationale that can
properly justify the patch. I usually need more back and fro like what
we are doing now to fully write out all the missing pieces. So I really
appreciate your time in reviewing the patch and your critiques :-)
>
> (note how rwsem_cond_wake_waiter() relies on exactly this to select
> WAKE_ANY)
>
> You are also right in that having these different means of waking
> readers and writers is 'confusing'.
>
> But you failed to take things to their natural conclusion -- change the
> way we do writer wakeups, all writer wakeups.
>
> So change rwsem_mark_wake()'s writer wakeup and
> rwsem_down_write_slowpath() to always so that waiter->task thing that
> the readers already do.
>
> It means putting all the logic for waking writers into
> rwsem_mark_wake(), but then we can make full use of this 'we hit zero
> future readers are temporary noise'. Althought I suspect you might need
> to pass count into it, so we can observe the flags at the time 0 was
> observed.
>
> Is all that making sense? -- it has been a long day :-)

I guess it is possible to do that, but it may reduce the performance
improvement due to optimistic spinning and it will become more like the
old way of strictly following the wait queue order in granting lock. I
need to run some tests to verify the performance impact.

>
>>> Also; and I asked this last time around; why do we care about the
>>> handoff to writer *at*all* ? It is the readers that set HANDOFF.
>> HANDOFF can happen for both readers and writers. Handoff to writer is
>> actually more important than to readers.
> Hmm, clearly I missed something. This is rwsem_try_write_lock() setting
> it, right?

Yes, handoff can be set and granted in rwsem_try_write_lock().

Cheers,
Longman

2023-02-23 01:48:15

by kernel test robot

[permalink] [raw]

Subject: Re: [PATCH v7 4/4] locking/rwsem: Enable direct rwsem lock handoff

Greeting,

FYI, we noticed a -19.5% regression of aim7.jobs-per-min due to commit:

commit: 12ea6b720a0e614611f626df1067f6a2281da65b ("[PATCH v7 4/4] locking/rwsem: Enable direct rwsem lock handoff")
url: https://github.com/intel-lab-lkp/linux/commits/Waiman-Long/locking-rwsem-Prevent-non-first-waiter-from-spinning-in-down_write-slowpath/20230128-123758
base: https://git.kernel.org/cgit/linux/kernel/git/will/linux.git for-next/perf
patch link: https://lore.kernel.org/all/[email protected]/
patch subject: [PATCH v7 4/4] locking/rwsem: Enable direct rwsem lock handoff

in testcase: aim7
on test machine: 128 threads 4 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory
with following parameters:

disk: 1BRD_48G
fs: xfs
test: creat-clo
load: 1500
cpufreq_governor: performance

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops 18.2% improvement |
| test machine | 104 threads 2 sockets (Skylake) with 192G memory |
| test parameters | cpufreq_governor=performance |
| | mode=thread |
| | nr_task=100% |
| | test=mmap2 |
+------------------+------------------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops 19.3% improvement |
| test machine | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz (Cascade Lake) with 128G memory |
| test parameters | cpufreq_governor=performance |
| | mode=thread |
| | nr_task=100% |
| | test=mmap2 |
+------------------+------------------------------------------------------------------------------------------------+

If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-lkp/[email protected]

Details are as below:

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase:
gcc-11/performance/1BRD_48G/xfs/x86_64-rhel-8.3/1500/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp2/creat-clo/aim7

commit:
6eb0bdc24a ("locking/rwsem: Disable preemption at all down_write*() and up_write() code paths")
12ea6b720a ("locking/rwsem: Enable direct rwsem lock handoff")

6eb0bdc24acb1b4e 12ea6b720a0e614611f626df106
---------------- ---------------------------
%stddev %change %stddev
\ | \
78510 -19.5% 63223 ? 8% aim7.jobs-per-min
114.78 +25.1% 143.60 ? 8% aim7.time.elapsed_time
114.78 +25.1% 143.60 ? 8% aim7.time.elapsed_time.max
181629 -35.5% 117183 ? 32% aim7.time.involuntary_context_switches
216106 ? 2% -19.8% 173358 ? 12% aim7.time.minor_page_faults
10838 ? 2% -29.3% 7665 ? 26% aim7.time.system_time
528056 +1813.0% 10101571 ? 51% aim7.time.voluntary_context_switches
3.986e+09 +172.6% 1.087e+10 ? 33% cpuidle..time
8246637 +277.0% 31093448 ? 39% cpuidle..usage
27.13 ? 2% +109.1% 56.74 ? 25% iostat.cpu.idle
72.60 -40.8% 43.01 ? 33% iostat.cpu.system
185.47 +16.7% 216.45 ? 5% uptime.boot
12021 +59.0% 19112 ? 18% uptime.idle
478874 +11.4% 533387 ? 4% meminfo.Inactive
477678 +11.4% 532183 ? 4% meminfo.Inactive(anon)
102085 ? 6% +52.8% 156021 ? 13% meminfo.Mapped
25.92 ? 2% +30.2 56.14 ? 26% mpstat.cpu.all.idle%
0.08 ? 2% +0.1 0.13 ? 9% mpstat.cpu.all.soft%
72.73 -30.2 42.51 ? 34% mpstat.cpu.all.sys%
416025 ? 9% -29.4% 293807 ? 23% numa-numastat.node1.local_node
527710 ? 7% +31.8% 695357 ? 7% numa-numastat.node3.local_node
583042 ? 6% +26.4% 736891 ? 9% numa-numastat.node3.numa_hit
26.83 ? 2% +109.9% 56.33 ? 26% vmstat.cpu.id
72.00 -41.0% 42.50 ? 34% vmstat.cpu.sy
93.83 -43.2% 53.33 ? 34% vmstat.procs.r
16919 ? 2% +718.8% 138536 ? 43% vmstat.system.cs
279240 +18.0% 329399 ? 7% vmstat.system.in
18933 ? 57% +100.0% 37867 ? 39% numa-meminfo.node0.Mapped
6669 ? 78% -75.3% 1647 ? 70% numa-meminfo.node1.Shmem
13638 ? 13% +33.9% 18258 ? 15% numa-meminfo.node2.Mapped
5341 ? 76% -57.8% 2255 ? 46% numa-meminfo.node2.Shmem
1403 ? 99% +1046.6% 16090 ? 47% numa-meminfo.node3.AnonHugePages
95567 ? 34% +140.5% 229823 ? 16% numa-meminfo.node3.Inactive
95398 ? 34% +140.5% 229480 ? 16% numa-meminfo.node3.Inactive(anon)
29044 ? 9% +138.2% 69196 ? 20% numa-meminfo.node3.Mapped
1917 -40.2% 1147 ? 33% turbostat.Avg_MHz
73.89 -29.7 44.22 ? 33% turbostat.Busy%
8206994 +277.9% 31016940 ? 39% turbostat.C1
26.48 +30.3 56.75 ? 26% turbostat.C1%
26.11 +113.7% 55.78 ? 26% turbostat.CPU%c1
33137178 +47.8% 48969692 ? 15% turbostat.IRQ
1603 ?102% +2380.2% 39766 ? 51% turbostat.POLL
324.04 -7.9% 298.44 ? 4% turbostat.PkgWatt
105792 -2.2% 103451 proc-vmstat.nr_anon_pages
119289 +11.7% 133280 ? 4% proc-vmstat.nr_inactive_anon
25429 ? 6% +54.9% 39384 ? 14% proc-vmstat.nr_mapped
88988 -8.0% 81837 ? 2% proc-vmstat.nr_slab_unreclaimable
119289 +11.7% 133280 ? 4% proc-vmstat.nr_zone_inactive_anon
45599 ? 7% -19.1% 36876 ? 13% proc-vmstat.numa_hint_faults_local
80031 ? 67% +238.4% 270822 ? 3% proc-vmstat.pgactivate
901595 +6.3% 958492 proc-vmstat.pgfault
984192 +21.6% 1196800 ? 7% proc-vmstat.unevictable_pgs_scanned
4578 ? 59% +107.0% 9476 ? 39% numa-vmstat.node0.nr_mapped
1612 ? 78% -74.5% 411.50 ? 71% numa-vmstat.node1.nr_shmem
415920 ? 9% -29.3% 293852 ? 23% numa-vmstat.node1.numa_local
3231 ? 10% +41.6% 4574 ? 14% numa-vmstat.node2.nr_mapped
1270 ? 70% -55.6% 563.50 ? 46% numa-vmstat.node2.nr_shmem
23523 ? 35% +145.6% 57776 ? 15% numa-vmstat.node3.nr_inactive_anon
6918 ? 10% +155.9% 17702 ? 18% numa-vmstat.node3.nr_mapped
23524 ? 35% +145.6% 57776 ? 15% numa-vmstat.node3.nr_zone_inactive_anon
583266 ? 6% +26.3% 736780 ? 9% numa-vmstat.node3.numa_hit
527934 ? 7% +31.7% 695246 ? 7% numa-vmstat.node3.numa_local
0.35 ? 8% +16.8% 0.41 ? 4% sched_debug.cfs_rq:/.h_nr_running.stddev
0.34 ? 9% +19.0% 0.40 ? 5% sched_debug.cfs_rq:/.nr_running.stddev
512.00 -33.6% 339.89 sched_debug.cfs_rq:/.removed.load_avg.max
245.97 ? 4% -28.8% 175.25 ? 12% sched_debug.cfs_rq:/.runnable_avg.stddev
240.01 ? 4% -27.7% 173.41 ? 11% sched_debug.cfs_rq:/.util_avg.stddev
183.56 ? 11% -73.1% 49.31 ? 81% sched_debug.cfs_rq:/.util_est_enqueued.avg
886.17 ? 6% -40.0% 532.00 ? 23% sched_debug.cfs_rq:/.util_est_enqueued.max
158.68 ? 5% -46.7% 84.59 ? 36% sched_debug.cfs_rq:/.util_est_enqueued.stddev
880432 -10.7% 786314 ? 2% sched_debug.cpu.avg_idle.avg
99262 +32.3% 131372 sched_debug.cpu.clock.avg
99276 +32.3% 131387 sched_debug.cpu.clock.max
99248 +32.4% 131356 sched_debug.cpu.clock.min
98647 +32.2% 130421 sched_debug.cpu.clock_task.avg
98939 +32.1% 130728 sched_debug.cpu.clock_task.max
81511 +39.0% 113274 sched_debug.cpu.clock_task.min
6565 +19.8% 7863 sched_debug.cpu.curr->pid.max
1753 ? 7% +29.3% 2266 ? 5% sched_debug.cpu.curr->pid.stddev
0.35 ? 9% +17.8% 0.41 ? 4% sched_debug.cpu.nr_running.stddev
6386 ? 2% +923.2% 65345 ? 42% sched_debug.cpu.nr_switches.avg
24662 ? 21% +280.4% 93813 ? 23% sched_debug.cpu.nr_switches.max
3912 ? 2% +1402.9% 58803 ? 44% sched_debug.cpu.nr_switches.min
3083 ? 13% +39.9% 4312 ? 5% sched_debug.cpu.nr_switches.stddev
99248 +32.4% 131357 sched_debug.cpu_clk
97982 +32.8% 130092 sched_debug.ktime
100450 +32.0% 132558 sched_debug.sched_clk
2.63 +50.2% 3.94 ? 18% perf-stat.i.MPKI
1.343e+10 -35.6% 8.649e+09 ? 27% perf-stat.i.branch-instructions
0.22 ? 2% +0.1 0.31 ? 13% perf-stat.i.branch-miss-rate%
18270941 ? 3% +8.2% 19775640 ? 3% perf-stat.i.branch-misses
59405514 ? 2% -10.2% 53324512 ? 4% perf-stat.i.cache-misses
1.684e+08 -8.8% 1.535e+08 ? 4% perf-stat.i.cache-references
17093 ? 2% +729.7% 141824 ? 42% perf-stat.i.context-switches
3.65 -11.1% 3.24 ? 5% perf-stat.i.cpi
2.461e+11 -40.7% 1.46e+11 ? 32% perf-stat.i.cpu-cycles
3065 ? 2% +169.7% 8267 ? 14% perf-stat.i.cpu-migrations
4781 -33.5% 3178 ? 23% perf-stat.i.cycles-between-cache-misses
0.01 ? 4% +0.0 0.03 ? 16% perf-stat.i.dTLB-load-miss-rate%
1008257 ? 13% +107.2% 2089007 ? 10% perf-stat.i.dTLB-load-misses
1.839e+10 -35.1% 1.193e+10 ? 26% perf-stat.i.dTLB-loads
0.00 ? 3% +0.0 0.01 ? 6% perf-stat.i.dTLB-store-miss-rate%
92257 ? 4% +78.4% 164589 ? 5% perf-stat.i.dTLB-store-misses
3.009e+09 -13.9% 2.59e+09 ? 6% perf-stat.i.dTLB-stores
6.76e+10 -35.4% 4.365e+10 ? 26% perf-stat.i.instructions
0.29 +14.7% 0.33 ? 4% perf-stat.i.ipc
1.92 -40.7% 1.14 ? 32% perf-stat.i.metric.GHz
230.51 +15.7% 266.74 ? 5% perf-stat.i.metric.K/sec
273.22 -33.4% 182.02 ? 24% perf-stat.i.metric.M/sec
14493557 ? 2% -8.6% 13246799 ? 2% perf-stat.i.node-load-misses
242903 ? 8% -16.2% 203542 ? 5% perf-stat.i.node-loads
75.33 -2.3 72.98 perf-stat.i.node-store-miss-rate%
10389085 ? 3% -13.7% 8967198 ? 6% perf-stat.i.node-store-misses
2.49 +49.7% 3.73 ? 22% perf-stat.overall.MPKI
0.14 ? 3% +0.1 0.24 ? 23% perf-stat.overall.branch-miss-rate%
3.64 -9.6% 3.29 ? 6% perf-stat.overall.cpi
4144 ? 2% -34.4% 2718 ? 29% perf-stat.overall.cycles-between-cache-misses
0.01 ? 14% +0.0 0.02 ? 31% perf-stat.overall.dTLB-load-miss-rate%
0.00 ? 4% +0.0 0.01 ? 9% perf-stat.overall.dTLB-store-miss-rate%
0.27 +11.1% 0.31 ? 6% perf-stat.overall.ipc
76.57 -2.4 74.17 perf-stat.overall.node-store-miss-rate%
1.342e+10 -35.3% 8.679e+09 ? 27% perf-stat.ps.branch-instructions
18164364 ? 3% +8.4% 19689502 ? 3% perf-stat.ps.branch-misses
59393969 ? 2% -10.1% 53379111 ? 4% perf-stat.ps.cache-misses
1.684e+08 -8.7% 1.537e+08 ? 4% perf-stat.ps.cache-references
16982 ? 2% +729.5% 140871 ? 43% perf-stat.ps.context-switches
2.46e+11 -40.3% 1.467e+11 ? 33% perf-stat.ps.cpu-cycles
3059 ? 2% +171.3% 8299 ? 14% perf-stat.ps.cpu-migrations
1042183 ? 14% +101.0% 2095105 ? 10% perf-stat.ps.dTLB-load-misses
1.838e+10 -34.8% 1.197e+10 ? 26% perf-stat.ps.dTLB-loads
92536 ? 4% +77.6% 164309 ? 5% perf-stat.ps.dTLB-store-misses
3.004e+09 -13.9% 2.587e+09 ? 6% perf-stat.ps.dTLB-stores
6.754e+10 -35.1% 4.38e+10 ? 27% perf-stat.ps.instructions
14490881 ? 2% -8.4% 13266549 ? 2% perf-stat.ps.node-load-misses
243051 ? 8% -16.1% 203822 ? 5% perf-stat.ps.node-loads
10386482 ? 3% -13.6% 8976461 ? 6% perf-stat.ps.node-store-misses
7.842e+12 -20.8% 6.212e+12 ? 18% perf-stat.total.instructions
97.91 -2.8 95.09 ? 2% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_creat.do_syscall_64
97.93 -2.8 95.11 ? 2% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_creat.do_syscall_64.entry_SYSCALL_64_after_hwframe
98.24 -2.7 95.54 ? 2% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_creat.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
98.25 -2.7 95.56 ? 2% perf-profile.calltrace.cycles-pp.__x64_sys_creat.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
98.30 -2.7 95.65 ? 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
98.34 -2.7 95.68 ? 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat64
98.43 -2.6 95.83 ? 2% perf-profile.calltrace.cycles-pp.creat64
2.14 ? 21% -1.1 1.03 ? 36% perf-profile.calltrace.cycles-pp._raw_spin_lock.lockref_get_not_dead.__legitimize_path.try_to_unlazy.open_last_lookups
2.14 ? 21% -1.1 1.02 ? 36% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.lockref_get_not_dead.__legitimize_path.try_to_unlazy
2.14 ? 21% -1.1 1.03 ? 37% perf-profile.calltrace.cycles-pp._raw_spin_lock.dput.terminate_walk.path_openat.do_filp_open
2.14 ? 21% -1.1 1.02 ? 36% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.dput.terminate_walk.path_openat
2.46 ? 18% -1.0 1.45 ? 20% perf-profile.calltrace.cycles-pp.lockref_get_not_dead.__legitimize_path.try_to_unlazy.open_last_lookups.path_openat
2.40 ? 19% -1.0 1.39 ? 22% perf-profile.calltrace.cycles-pp.dput.terminate_walk.path_openat.do_filp_open.do_sys_openat2
2.43 ? 18% -1.0 1.42 ? 21% perf-profile.calltrace.cycles-pp.terminate_walk.path_openat.do_filp_open.do_sys_openat2.__x64_sys_creat
2.50 ? 17% -1.0 1.50 ? 18% perf-profile.calltrace.cycles-pp.__legitimize_path.try_to_unlazy.open_last_lookups.path_openat.do_filp_open
2.51 ? 17% -1.0 1.53 ? 18% perf-profile.calltrace.cycles-pp.try_to_unlazy.open_last_lookups.path_openat.do_filp_open.do_sys_openat2
0.08 ?223% +1.2 1.24 ? 41% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.rwsem_wake.up_write.open_last_lookups.path_openat
0.98 ? 2% +1.6 2.56 ? 41% perf-profile.calltrace.cycles-pp.rwsem_wake.up_write.open_last_lookups.path_openat.do_filp_open
1.13 ? 2% +1.7 2.80 ? 40% perf-profile.calltrace.cycles-pp.up_write.open_last_lookups.path_openat.do_filp_open.do_sys_openat2
0.00 +2.5 2.52 ? 67% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +2.5 2.52 ? 67% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +2.5 2.52 ? 67% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
0.00 +2.5 2.54 ? 67% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
97.96 -2.8 95.13 ? 2% perf-profile.children.cycles-pp.path_openat
97.98 -2.8 95.14 ? 2% perf-profile.children.cycles-pp.do_filp_open
98.29 -2.7 95.58 ? 2% perf-profile.children.cycles-pp.do_sys_openat2
98.26 -2.7 95.56 ? 2% perf-profile.children.cycles-pp.__x64_sys_creat
98.47 -2.6 95.88 ? 2% perf-profile.children.cycles-pp.creat64
99.28 -2.4 96.90 perf-profile.children.cycles-pp.do_syscall_64
99.32 -2.4 96.94 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
4.63 ? 19% -2.1 2.56 ? 25% perf-profile.children.cycles-pp._raw_spin_lock
2.48 ? 18% -1.0 1.46 ? 20% perf-profile.children.cycles-pp.lockref_get_not_dead
2.44 ? 18% -1.0 1.44 ? 21% perf-profile.children.cycles-pp.terminate_walk
2.52 ? 17% -1.0 1.52 ? 18% perf-profile.children.cycles-pp.__legitimize_path
2.53 ? 17% -1.0 1.54 ? 18% perf-profile.children.cycles-pp.try_to_unlazy
2.68 ? 16% -0.9 1.74 ? 14% perf-profile.children.cycles-pp.dput
0.18 ? 3% -0.1 0.11 ? 13% perf-profile.children.cycles-pp.ret_from_fork
0.18 ? 3% -0.1 0.11 ? 13% perf-profile.children.cycles-pp.kthread
0.11 ? 4% +0.0 0.15 ? 19% perf-profile.children.cycles-pp.strncpy_from_user
0.05 ? 7% +0.0 0.09 ? 20% perf-profile.children.cycles-pp.newidle_balance
0.04 ? 44% +0.0 0.09 ? 21% perf-profile.children.cycles-pp.update_sd_lb_stats
0.04 ? 44% +0.1 0.09 ? 23% perf-profile.children.cycles-pp.find_busiest_group
0.39 ? 6% +0.1 0.44 ? 2% perf-profile.children.cycles-pp.update_process_times
0.06 ? 8% +0.1 0.11 ? 23% perf-profile.children.cycles-pp.load_balance
0.39 ? 6% +0.1 0.44 ? 3% perf-profile.children.cycles-pp.tick_sched_handle
0.01 ?223% +0.1 0.08 ? 23% perf-profile.children.cycles-pp.update_sg_lb_stats
0.41 ? 6% +0.1 0.48 ? 4% perf-profile.children.cycles-pp.tick_sched_timer
0.08 ? 18% +0.1 0.15 ? 30% perf-profile.children.cycles-pp.clockevents_program_event
0.05 ? 7% +0.1 0.13 ? 33% perf-profile.children.cycles-pp.pick_next_task_fair
0.16 ? 3% +0.1 0.24 ? 28% perf-profile.children.cycles-pp.osq_unlock
0.20 ? 2% +0.1 0.30 ? 23% perf-profile.children.cycles-pp.xlog_ticket_alloc
0.10 ? 10% +0.1 0.19 ? 26% perf-profile.children.cycles-pp.update_load_avg
0.17 ? 3% +0.1 0.26 ? 31% perf-profile.children.cycles-pp.rcu_core
0.06 ? 17% +0.1 0.17 ? 40% perf-profile.children.cycles-pp.ktime_get
0.16 ? 4% +0.1 0.28 ? 31% perf-profile.children.cycles-pp.wake_q_add
0.45 ? 5% +0.1 0.56 ? 8% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.20 ? 3% +0.2 0.36 ? 35% perf-profile.children.cycles-pp.__do_softirq
0.55 ? 3% +0.2 0.77 ? 15% perf-profile.children.cycles-pp.hrtimer_interrupt
0.57 ? 3% +0.2 0.81 ? 15% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.06 ? 6% +0.2 0.30 ? 52% perf-profile.children.cycles-pp.schedule
0.10 ? 4% +0.3 0.36 ? 43% perf-profile.children.cycles-pp.__irq_exit_rcu
0.00 +0.3 0.27 ? 60% perf-profile.children.cycles-pp.schedule_preempt_disabled
0.07 ? 8% +0.3 0.39 ? 55% perf-profile.children.cycles-pp.__schedule
0.27 ? 3% +0.4 0.72 ? 42% perf-profile.children.cycles-pp.try_to_wake_up
0.30 +0.5 0.80 ? 43% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.08 ? 8% +0.5 0.60 ? 61% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.30 ? 2% +0.5 0.83 ? 44% perf-profile.children.cycles-pp.wake_up_q
0.70 ? 2% +0.6 1.26 ? 27% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.00 +0.6 0.57 ? 71% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.69 ? 4% +1.0 1.71 ? 39% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.84 ? 2% +1.3 2.11 ? 42% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.98 ? 2% +1.6 2.57 ? 42% perf-profile.children.cycles-pp.rwsem_wake
1.19 ? 2% +1.7 2.88 ? 40% perf-profile.children.cycles-pp.up_write
0.23 ? 4% +2.0 2.18 ? 67% perf-profile.children.cycles-pp.acpi_idle_do_entry
0.23 ? 2% +2.0 2.18 ? 67% perf-profile.children.cycles-pp.acpi_idle_enter
0.24 ? 4% +2.0 2.26 ? 67% perf-profile.children.cycles-pp.cpuidle_enter_state
0.24 ? 4% +2.0 2.26 ? 67% perf-profile.children.cycles-pp.cpuidle_enter
0.25 ? 4% +2.1 2.40 ? 67% perf-profile.children.cycles-pp.cpuidle_idle_call
0.26 ? 3% +2.3 2.52 ? 67% perf-profile.children.cycles-pp.start_secondary
0.26 ? 3% +2.3 2.54 ? 67% perf-profile.children.cycles-pp.do_idle
0.26 ? 3% +2.3 2.54 ? 67% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
0.26 ? 3% +2.3 2.54 ? 67% perf-profile.children.cycles-pp.cpu_startup_entry
0.06 ? 6% +0.0 0.09 ? 24% perf-profile.self.cycles-pp.do_dentry_open
0.05 +0.0 0.08 ? 27% perf-profile.self.cycles-pp.__alloc_file
0.15 ? 3% +0.0 0.19 ? 20% perf-profile.self.cycles-pp.kmem_cache_alloc
0.00 +0.1 0.06 ? 17% perf-profile.self.cycles-pp.update_sg_lb_stats
0.04 ? 45% +0.1 0.11 ? 25% perf-profile.self.cycles-pp.update_load_avg
0.16 ? 3% +0.1 0.24 ? 28% perf-profile.self.cycles-pp.osq_unlock
0.04 ? 72% +0.1 0.14 ? 43% perf-profile.self.cycles-pp.ktime_get
0.20 ? 2% +0.1 0.31 ? 26% perf-profile.self.cycles-pp.up_write
0.00 +0.1 0.11 ? 45% perf-profile.self.cycles-pp.rwsem_wake
0.16 ? 4% +0.1 0.27 ? 31% perf-profile.self.cycles-pp.wake_q_add
0.00 +0.1 0.12 ? 50% perf-profile.self.cycles-pp.wake_up_q
0.10 ? 3% +0.2 0.26 ? 44% perf-profile.self.cycles-pp.rwsem_down_write_slowpath
0.30 ? 2% +0.2 0.47 ? 29% perf-profile.self.cycles-pp._raw_spin_lock
0.30 +0.2 0.51 ? 31% perf-profile.self.cycles-pp.down_write
0.34 ? 2% +0.2 0.56 ? 28% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.29 +0.5 0.79 ? 43% perf-profile.self.cycles-pp.rwsem_spin_on_owner
0.11 ? 6% +1.1 1.23 ? 71% perf-profile.self.cycles-pp.acpi_idle_do_entry

***************************************************************************************************
lkp-skl-fpga01: 104 threads 2 sockets (Skylake) with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-11/performance/x86_64-rhel-8.3/thread/100%/debian-11.1-x86_64-20220510.cgz/lkp-skl-fpga01/mmap2/will-it-scale

commit:
6eb0bdc24a ("locking/rwsem: Disable preemption at all down_write*() and up_write() code paths")
12ea6b720a ("locking/rwsem: Enable direct rwsem lock handoff")

6eb0bdc24acb1b4e 12ea6b720a0e614611f626df106
---------------- ---------------------------
%stddev %change %stddev
\ | \
122193 +18.2% 144442 will-it-scale.104.threads
2.05 ? 2% +4184.9% 87.91 will-it-scale.104.threads_idle
1174 +18.2% 1388 will-it-scale.per_thread_ops
122193 +18.2% 144442 will-it-scale.workload
5901 +464.6% 33319 uptime.idle
8.974e+08 ? 3% +3041.2% 2.819e+10 cpuidle..time
10588909 ? 5% +953.7% 1.116e+08 cpuidle..usage
2.83 +85.1 87.97 mpstat.cpu.all.idle%
0.01 +0.1 0.06 ? 2% mpstat.cpu.all.soft%
95.86 -85.1 10.72 ? 12% mpstat.cpu.all.sys%
69923 ? 2% -76.6% 16357 ? 2% meminfo.Active
69819 ? 2% -76.7% 16243 ? 2% meminfo.Active(anon)
8145299 +12.0% 9122891 meminfo.Committed_AS
63874 ? 2% -12.9% 55604 ? 3% meminfo.Mapped
99558 -59.5% 40282 ? 2% meminfo.Shmem
47377 ? 3% -12.2% 41591 ? 3% numa-meminfo.node0.Mapped
19233 ? 19% -53.5% 8952 ? 50% numa-meminfo.node0.Shmem
65775 ? 2% -80.5% 12848 ? 6% numa-meminfo.node1.Active
65691 ? 2% -80.4% 12848 ? 6% numa-meminfo.node1.Active(anon)
80391 ? 4% -61.2% 31192 ? 12% numa-meminfo.node1.Shmem
11819 ? 3% -11.9% 10412 ? 3% numa-vmstat.node0.nr_mapped
4810 ? 19% -53.4% 2241 ? 51% numa-vmstat.node0.nr_shmem
16450 ? 2% -80.5% 3215 ? 6% numa-vmstat.node1.nr_active_anon
20104 ? 4% -61.1% 7828 ? 12% numa-vmstat.node1.nr_shmem
16450 ? 2% -80.5% 3215 ? 6% numa-vmstat.node1.nr_zone_active_anon
3.00 +2822.2% 87.67 vmstat.cpu.id
96.00 -88.2% 11.33 ? 12% vmstat.cpu.sy
94.17 -90.1% 9.33 ? 14% vmstat.procs.r
67738 ? 6% +497.5% 404708 ? 2% vmstat.system.cs
216928 +12.1% 243249 vmstat.system.in
17464 ? 2% -77.2% 3984 ? 2% proc-vmstat.nr_active_anon
751355 -2.0% 736520 proc-vmstat.nr_file_pages
96091 -1.8% 94358 proc-vmstat.nr_inactive_anon
16034 ? 2% -13.0% 13952 ? 3% proc-vmstat.nr_mapped
24902 -59.7% 10026 ? 3% proc-vmstat.nr_shmem
17464 ? 2% -77.2% 3984 ? 2% proc-vmstat.nr_zone_active_anon
96091 -1.8% 94358 proc-vmstat.nr_zone_inactive_anon
935656 +1.4% 948960 proc-vmstat.numa_hit
841902 +1.5% 854785 proc-vmstat.numa_local
278938 ? 8% -25.1% 208918 ? 10% proc-vmstat.numa_pte_updates
1015659 -2.8% 987018 proc-vmstat.pgfault
982847 +2.4% 1006082 proc-vmstat.pgfree
48959 ? 2% -9.2% 44468 ? 3% proc-vmstat.pgreuse
2716 -88.1% 322.33 ? 10% turbostat.Avg_MHz
97.25 -85.4 11.86 ? 10% turbostat.Busy%
2799 -2.8% 2721 turbostat.Bzy_MHz
713088 ? 9% +316.0% 2966569 ? 51% turbostat.C1
0.10 ? 25% +1.2 1.27 ?117% turbostat.C1%
1874841 ? 16% +3121.6% 60400704 ? 21% turbostat.C1E
1.74 ? 40% +66.2 67.94 ? 42% turbostat.C1E%
2.60 ? 2% +3286.7% 88.00 turbostat.CPU%c1
44.67 -10.8% 39.83 turbostat.CoreTmp
0.03 +88.9% 0.06 ? 8% turbostat.IPC
66288538 +11.9% 74185443 turbostat.IRQ
7619276 ? 6% +421.6% 39741832 ? 2% turbostat.POLL
0.08 ? 4% +0.6 0.66 ? 16% turbostat.POLL%
45.17 -11.1% 40.17 turbostat.PkgTmp
289.90 -23.7% 221.06 turbostat.PkgWatt
26.59 +1.5% 27.00 turbostat.RAMWatt
0.72 ? 8% -81.8% 0.13 ? 14% sched_debug.cfs_rq:/.h_nr_running.avg
1.72 ? 7% -38.7% 1.06 ? 11% sched_debug.cfs_rq:/.h_nr_running.max
7422 ? 81% +315.9% 30869 ? 20% sched_debug.cfs_rq:/.load.stddev
218.86 ? 21% +123.1% 488.21 ? 35% sched_debug.cfs_rq:/.load_avg.max
2.97 ? 19% -100.0% 0.00 sched_debug.cfs_rq:/.load_avg.min
38.66 ? 26% +78.4% 68.97 ? 18% sched_debug.cfs_rq:/.load_avg.stddev
14312599 -99.0% 142063 ? 23% sched_debug.cfs_rq:/.min_vruntime.avg
14367713 -98.8% 172523 ? 18% sched_debug.cfs_rq:/.min_vruntime.max
14126661 -99.2% 119038 ? 28% sched_debug.cfs_rq:/.min_vruntime.min
32193 ? 6% -71.0% 9337 ? 12% sched_debug.cfs_rq:/.min_vruntime.stddev
0.71 ? 8% -81.6% 0.13 ? 13% sched_debug.cfs_rq:/.nr_running.avg
30.30 ? 39% +71.5% 51.95 ? 23% sched_debug.cfs_rq:/.removed.load_avg.stddev
78.33 ? 15% +141.6% 189.27 ? 41% sched_debug.cfs_rq:/.removed.runnable_avg.max
12.15 ? 42% +101.3% 24.45 ? 24% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
78.33 ? 15% +141.6% 189.27 ? 41% sched_debug.cfs_rq:/.removed.util_avg.max
12.15 ? 42% +101.3% 24.45 ? 24% sched_debug.cfs_rq:/.removed.util_avg.stddev
767.16 ? 5% -83.8% 123.91 ? 6% sched_debug.cfs_rq:/.runnable_avg.avg
1358 ? 9% -49.7% 683.63 ? 5% sched_debug.cfs_rq:/.runnable_avg.max
304.25 ? 18% -99.8% 0.51 ? 65% sched_debug.cfs_rq:/.runnable_avg.min
33275 ? 53% -107.3% -2417 sched_debug.cfs_rq:/.spread0.avg
88428 ? 23% -68.3% 28043 ? 54% sched_debug.cfs_rq:/.spread0.max
-152998 -83.4% -25443 sched_debug.cfs_rq:/.spread0.min
32195 ? 6% -71.0% 9338 ? 12% sched_debug.cfs_rq:/.spread0.stddev
765.09 ? 5% -83.8% 123.88 ? 6% sched_debug.cfs_rq:/.util_avg.avg
1276 ? 9% -46.5% 683.51 ? 5% sched_debug.cfs_rq:/.util_avg.max
289.25 ? 24% -99.8% 0.51 ? 65% sched_debug.cfs_rq:/.util_avg.min
624.78 ? 9% -96.7% 20.45 ? 20% sched_debug.cfs_rq:/.util_est_enqueued.avg
1169 ? 4% -68.0% 374.37 ? 5% sched_debug.cfs_rq:/.util_est_enqueued.max
274.94 ? 16% -75.1% 68.44 ? 10% sched_debug.cfs_rq:/.util_est_enqueued.stddev
439610 ? 12% +59.3% 700395 ? 2% sched_debug.cpu.avg_idle.avg
5.25 ? 10% -29.4% 3.71 sched_debug.cpu.clock.stddev
3812 ? 8% -82.5% 668.76 ? 13% sched_debug.cpu.curr->pid.avg
0.00 ? 35% -81.2% 0.00 ? 22% sched_debug.cpu.next_balance.stddev
0.72 ? 8% -82.4% 0.13 ? 13% sched_debug.cpu.nr_running.avg
1.72 ? 7% -37.1% 1.08 ? 17% sched_debug.cpu.nr_running.max
102653 ? 5% +407.5% 520923 ? 13% sched_debug.cpu.nr_switches.avg
314676 ? 34% +97.2% 620406 ? 15% sched_debug.cpu.nr_switches.max
42954 ? 10% +936.0% 445007 ? 11% sched_debug.cpu.nr_switches.min
1.14 ? 5% +641.1% 8.43 ? 5% perf-stat.i.MPKI
5.856e+09 -77.1% 1.34e+09 ? 5% perf-stat.i.branch-instructions
0.25 +1.1 1.38 ? 4% perf-stat.i.branch-miss-rate%
14856545 +27.7% 18977248 perf-stat.i.branch-misses
42.87 -5.3 37.57 ? 2% perf-stat.i.cache-miss-rate%
14113155 ? 6% +46.8% 20719763 ? 2% perf-stat.i.cache-misses
33283236 ? 5% +65.9% 55220432 perf-stat.i.cache-references
68565 ? 6% +496.9% 409268 ? 2% perf-stat.i.context-switches
9.66 -49.2% 4.90 ? 5% perf-stat.i.cpi
2.826e+11 -88.5% 3.248e+10 ? 11% perf-stat.i.cpu-cycles
168.63 -17.1% 139.86 perf-stat.i.cpu-migrations
20166 ? 6% -92.3% 1559 ? 12% perf-stat.i.cycles-between-cache-misses
0.01 ? 5% +0.0 0.04 ? 3% perf-stat.i.dTLB-load-miss-rate%
489364 ? 5% +41.0% 690198 ? 5% perf-stat.i.dTLB-load-misses
7.767e+09 -77.6% 1.74e+09 ? 5% perf-stat.i.dTLB-loads
0.01 ? 34% -0.0 0.00 ? 12% perf-stat.i.dTLB-store-miss-rate%
4.227e+08 +67.0% 7.058e+08 perf-stat.i.dTLB-stores
56.60 ? 3% -45.4 11.21 ? 4% perf-stat.i.iTLB-load-miss-rate%
329472 ? 6% +22.9% 404958 ? 2% perf-stat.i.iTLB-load-misses
277356 +1057.3% 3209822 ? 3% perf-stat.i.iTLB-loads
2.927e+10 -77.4% 6.628e+09 ? 5% perf-stat.i.instructions
89841 ? 6% -81.8% 16390 ? 7% perf-stat.i.instructions-per-iTLB-miss
0.10 +105.5% 0.21 ? 5% perf-stat.i.ipc
0.11 ? 20% -91.4% 0.01 ? 77% perf-stat.i.major-faults
2.72 -88.5% 0.31 ? 11% perf-stat.i.metric.GHz
383.43 ? 4% +65.9% 636.18 perf-stat.i.metric.K/sec
135.05 -73.0% 36.40 ? 4% perf-stat.i.metric.M/sec
2759 -3.4% 2664 perf-stat.i.minor-faults
89.22 +1.4 90.58 perf-stat.i.node-load-miss-rate%
4761231 +12.1% 5338568 ? 2% perf-stat.i.node-load-misses
957855 +67.2% 1601103 ? 2% perf-stat.i.node-store-misses
26974 ? 5% -36.0% 17268 ? 12% perf-stat.i.node-stores
2759 -3.4% 2664 perf-stat.i.page-faults
1.14 ? 5% +632.7% 8.36 ? 4% perf-stat.overall.MPKI
0.25 +1.2 1.42 ? 4% perf-stat.overall.branch-miss-rate%
42.29 -4.8 37.53 ? 2% perf-stat.overall.cache-miss-rate%
9.65 -49.4% 4.89 ? 5% perf-stat.overall.cpi
20088 ? 6% -92.2% 1570 ? 12% perf-stat.overall.cycles-between-cache-misses
0.01 ? 5% +0.0 0.04 ? 2% perf-stat.overall.dTLB-load-miss-rate%
0.01 ? 31% -0.0 0.00 ? 12% perf-stat.overall.dTLB-store-miss-rate%
54.22 ? 3% -43.0 11.22 ? 4% perf-stat.overall.iTLB-load-miss-rate%
89173 ? 6% -81.6% 16389 ? 7% perf-stat.overall.instructions-per-iTLB-miss
0.10 +98.2% 0.21 ? 5% perf-stat.overall.ipc
89.18 +1.4 90.59 perf-stat.overall.node-load-miss-rate%
97.25 +1.7 98.93 perf-stat.overall.node-store-miss-rate%
72308298 -80.9% 13846989 ? 5% perf-stat.overall.path-length
5.836e+09 -77.1% 1.336e+09 ? 5% perf-stat.ps.branch-instructions
14803236 +27.8% 18915506 perf-stat.ps.branch-misses
14070120 ? 6% +46.8% 20652380 ? 2% perf-stat.ps.cache-misses
33255853 ? 5% +65.5% 55047209 perf-stat.ps.cache-references
68252 ? 6% +497.6% 407866 ? 2% perf-stat.ps.context-switches
2.816e+11 -88.5% 3.238e+10 ? 11% perf-stat.ps.cpu-cycles
168.00 -17.0% 139.48 perf-stat.ps.cpu-migrations
494045 ? 5% +39.4% 688570 ? 5% perf-stat.ps.dTLB-load-misses
7.741e+09 -77.6% 1.734e+09 ? 5% perf-stat.ps.dTLB-loads
4.215e+08 +66.9% 7.035e+08 perf-stat.ps.dTLB-stores
328360 ? 6% +22.9% 403621 ? 2% perf-stat.ps.iTLB-load-misses
276724 +1056.1% 3199136 ? 3% perf-stat.ps.iTLB-loads
2.917e+10 -77.3% 6.606e+09 ? 5% perf-stat.ps.instructions
0.11 ? 20% -91.4% 0.01 ? 77% perf-stat.ps.major-faults
2752 -3.5% 2657 perf-stat.ps.minor-faults
4744971 +12.1% 5320794 ? 2% perf-stat.ps.node-load-misses
954867 +67.1% 1595736 ? 2% perf-stat.ps.node-store-misses
26949 ? 6% -36.0% 17240 ? 12% perf-stat.ps.node-stores
2753 -3.5% 2657 perf-stat.ps.page-faults
8.835e+12 -77.4% 2e+12 ? 5% perf-stat.total.instructions
47.61 -37.6 10.02 ? 11% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
47.56 -37.4 10.11 ? 10% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
48.04 -35.5 12.50 ? 8% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
48.09 -35.5 12.56 ? 9% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
48.15 -33.2 14.95 ? 6% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
48.18 -33.0 15.15 ? 6% perf-profile.calltrace.cycles-pp.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
48.20 -32.8 15.43 ? 6% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
48.22 -32.6 15.63 ? 6% perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
48.71 -28.6 20.10 ? 5% perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
48.71 -28.6 20.12 ? 4% perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
48.75 -28.3 20.43 ? 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
48.77 -28.2 20.60 ? 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
48.82 -27.7 21.15 ? 4% perf-profile.calltrace.cycles-pp.__munmap
48.77 -27.4 21.37 ? 4% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
48.78 -27.3 21.46 ? 4% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
48.82 -27.0 21.82 ? 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
48.84 -26.8 21.99 ? 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
48.89 -26.3 22.58 ? 4% perf-profile.calltrace.cycles-pp.__mmap
0.00 +0.6 0.56 ? 4% perf-profile.calltrace.cycles-pp.mas_store_prealloc.do_mas_align_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
0.00 +0.6 0.56 ? 8% perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff
0.00 +0.6 0.58 ? 11% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen
0.00 +0.6 0.62 ? 7% perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.00 +0.6 0.62 ? 8% perf-profile.calltrace.cycles-pp.unmap_region.do_mas_align_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
0.00 +0.6 0.63 ? 12% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn
0.00 +0.6 0.65 ? 8% perf-profile.calltrace.cycles-pp.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
0.00 +0.6 0.65 ? 11% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn.perf_mmap__push
0.00 +0.7 0.68 ? 7% perf-profile.calltrace.cycles-pp.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
0.00 +0.7 0.71 ? 13% perf-profile.calltrace.cycles-pp.__libc_write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
0.00 +0.7 0.71 ? 6% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.rwsem_wake.up_read
0.00 +0.7 0.71 ? 13% perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record
0.00 +0.7 0.72 ? 13% perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
0.00 +0.8 0.78 ? 14% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
0.00 +0.8 0.79 ? 10% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
0.00 +0.8 0.80 ? 9% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.00 +0.8 0.82 ? 10% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
0.00 +0.9 0.90 ? 22% perf-profile.calltrace.cycles-pp.queue_event.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events
0.00 +0.9 0.91 ? 35% perf-profile.calltrace.cycles-pp.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
0.00 +0.9 0.92 ? 22% perf-profile.calltrace.cycles-pp.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events.record__finish_output
0.00 +0.9 0.94 ? 9% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle
0.00 +1.0 0.97 ? 8% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle.do_idle
0.00 +1.0 0.97 ? 17% perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.cmd_sched
0.00 +1.0 0.97 ? 7% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.rwsem_wake.up_write
0.00 +1.0 0.97 ? 22% perf-profile.calltrace.cycles-pp.process_simple.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record
0.00 +1.0 1.01 ? 9% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
0.00 +1.0 1.05 ? 7% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +1.0 1.05 ? 7% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.00 +1.1 1.12 ? 17% perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.__cmd_record.cmd_record.cmd_sched.run_builtin
0.00 +1.3 1.25 ? 21% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template
0.00 +1.3 1.33 ? 19% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.00 +1.5 1.47 ? 8% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +1.5 1.48 ? 9% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime
0.00 +1.5 1.51 ? 7% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up
0.00 +1.5 1.52 ? 7% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q
0.00 +1.5 1.55 ? 5% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
0.00 +1.6 1.55 ? 9% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
0.00 +1.6 1.58 ? 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.up_read.__vm_munmap
0.00 +1.6 1.58 ? 6% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.schedule_preempt_disabled
0.00 +1.6 1.60 ? 7% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.rwsem_wake
0.00 +1.6 1.62 ? 5% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath
0.00 +1.6 1.62 ? 7% perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.up_read.__vm_munmap.__x64_sys_munmap
0.00 +1.7 1.66 ? 8% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
0.00 +1.7 1.67 ? 8% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
0.00 +1.7 1.71 ? 8% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
0.00 +1.7 1.74 ? 8% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair.__schedule
0.00 +1.8 1.83 ? 5% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
0.00 +1.9 1.88 ? 8% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
0.00 +1.9 1.93 ? 6% perf-profile.calltrace.cycles-pp.rwsem_wake.up_read.__vm_munmap.__x64_sys_munmap.do_syscall_64
0.00 +1.9 1.95 ? 5% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
0.00 +2.0 1.95 ? 7% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.00 +2.0 1.96 ? 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.up_write.vm_mmap_pgoff
0.00 +2.0 1.98 ? 7% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +2.0 2.03 ? 7% perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.up_write.vm_mmap_pgoff.ksys_mmap_pgoff
0.00 +2.1 2.05 ? 5% perf-profile.calltrace.cycles-pp.up_read.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +2.1 2.09 ? 32% perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.cmd_sched.run_builtin.main
0.00 +2.1 2.09 ? 32% perf-profile.calltrace.cycles-pp.cmd_record.cmd_sched.run_builtin.main.__libc_start_main
0.00 +2.1 2.09 ? 32% perf-profile.calltrace.cycles-pp.cmd_sched.run_builtin.main.__libc_start_main
0.00 +2.1 2.09 ? 8% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_preempt_disabled
0.00 +2.1 2.13 ? 6% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch
0.00 +2.1 2.15 ? 31% perf-profile.calltrace.cycles-pp.__libc_start_main
0.00 +2.1 2.15 ? 31% perf-profile.calltrace.cycles-pp.main.__libc_start_main
0.00 +2.1 2.15 ? 31% perf-profile.calltrace.cycles-pp.run_builtin.main.__libc_start_main
0.00 +2.2 2.15 ? 8% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath
0.00 +2.2 2.17 ? 9% perf-profile.calltrace.cycles-pp.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
0.00 +2.2 2.18 ? 9% perf-profile.calltrace.cycles-pp.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
0.00 +2.3 2.29 ? 5% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
0.00 +2.4 2.39 ? 7% perf-profile.calltrace.cycles-pp.rwsem_wake.up_write.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
0.00 +2.5 2.46 ? 5% perf-profile.calltrace.cycles-pp.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
0.00 +2.5 2.46 ? 5% perf-profile.calltrace.cycles-pp.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
0.00 +2.5 2.47 ? 6% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
0.00 +2.6 2.55 ? 7% perf-profile.calltrace.cycles-pp.up_write.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +2.7 2.68 ? 6% perf-profile.calltrace.cycles-pp.do_mas_align_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +2.9 2.86 ? 8% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.00 +2.9 2.92 ? 8% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.00 +3.1 3.10 ? 5% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +4.2 4.17 ? 7% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
0.00 +4.6 4.56 ? 10% perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
0.00 +4.6 4.58 ? 9% perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event
0.00 +4.6 4.61 ? 7% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable
0.00 +4.6 4.61 ? 7% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
1.08 ? 71% +29.8 30.88 ? 47% perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
1.08 ? 71% +29.9 30.98 ? 47% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
1.84 ? 15% +44.9 46.76 ? 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
1.84 ? 15% +44.9 46.78 ? 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
1.84 ? 15% +46.8 48.60 ? 3% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.92 ? 14% +49.6 51.47 ? 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.92 ? 14% +49.6 51.50 ? 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.92 ? 14% +49.6 51.50 ? 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
1.94 ? 15% +50.1 52.08 ? 3% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
95.21 -75.1 20.14 ? 11% perf-profile.children.cycles-pp.osq_lock
96.14 -71.0 25.11 ? 8% perf-profile.children.cycles-pp.rwsem_optimistic_spin
96.35 -66.0 30.38 ? 6% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
96.40 -65.6 30.78 ? 6% perf-profile.children.cycles-pp.down_write_killable
97.67 -54.2 43.52 ? 4% perf-profile.children.cycles-pp.do_syscall_64
97.71 -53.8 43.90 ? 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
48.71 -28.6 20.11 ? 5% perf-profile.children.cycles-pp.__vm_munmap
48.71 -28.6 20.12 ? 5% perf-profile.children.cycles-pp.__x64_sys_munmap
48.82 -27.6 21.17 ? 4% perf-profile.children.cycles-pp.__munmap
48.77 -27.4 21.39 ? 4% perf-profile.children.cycles-pp.vm_mmap_pgoff
48.78 -27.3 21.47 ? 4% perf-profile.children.cycles-pp.ksys_mmap_pgoff
48.89 -26.3 22.60 ? 4% perf-profile.children.cycles-pp.__mmap
0.54 ? 8% -0.3 0.21 ? 7% perf-profile.children.cycles-pp.task_tick_fair
0.04 ? 45% +0.0 0.08 ? 14% perf-profile.children.cycles-pp.update_cfs_group
0.04 ? 44% +0.0 0.08 ? 14% perf-profile.children.cycles-pp.mas_wr_spanning_store
0.02 ?141% +0.1 0.07 ? 19% perf-profile.children.cycles-pp.mas_spanning_rebalance
0.00 +0.1 0.06 ? 13% perf-profile.children.cycles-pp.select_idle_sibling
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.trigger_load_balance
0.00 +0.1 0.06 ? 13% perf-profile.children.cycles-pp.enqueue_hrtimer
0.00 +0.1 0.06 ? 13% perf-profile.children.cycles-pp.get_stack_info
0.00 +0.1 0.06 ? 30% perf-profile.children.cycles-pp.do_read_fault
0.00 +0.1 0.06 ? 19% perf-profile.children.cycles-pp.shmem_mmap
0.00 +0.1 0.06 ? 14% perf-profile.children.cycles-pp.seq_printf
0.00 +0.1 0.07 ? 30% perf-profile.children.cycles-pp.do_fault
0.00 +0.1 0.07 ? 19% perf-profile.children.cycles-pp.record__write
0.00 +0.1 0.07 ? 19% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.00 +0.1 0.07 ? 15% perf-profile.children.cycles-pp.ct_idle_exit
0.00 +0.1 0.07 ? 17% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.00 +0.1 0.07 ? 17% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
0.00 +0.1 0.07 ? 20% perf-profile.children.cycles-pp.mas_next_nentry
0.00 +0.1 0.07 ? 11% perf-profile.children.cycles-pp.rcu_pending
0.00 +0.1 0.07 ? 10% perf-profile.children.cycles-pp.newidle_balance
0.00 +0.1 0.08 ? 17% perf-profile.children.cycles-pp.down_write
0.00 +0.1 0.08 ? 16% perf-profile.children.cycles-pp.select_task_rq_fair
0.00 +0.1 0.08 ? 22% perf-profile.children.cycles-pp.security_mmap_file
0.00 +0.1 0.08 ? 27% perf-profile.children.cycles-pp.__intel_pmu_enable_all
0.00 +0.1 0.08 ? 13% perf-profile.children.cycles-pp.machines__deliver_event
0.00 +0.1 0.08 ? 14% perf-profile.children.cycles-pp.error_entry
0.00 +0.1 0.08 ? 19% perf-profile.children.cycles-pp.llist_reverse_order
0.00 +0.1 0.08 ? 13% perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.00 +0.1 0.09 ? 8% perf-profile.children.cycles-pp.zap_pmd_range
0.00 +0.1 0.09 ? 33% perf-profile.children.cycles-pp.__perf_mmap__read_init
0.00 +0.1 0.09 ? 10% perf-profile.children.cycles-pp.__update_load_avg_se
0.00 +0.1 0.09 ? 7% perf-profile.children.cycles-pp.rwsem_mark_wake
0.00 +0.1 0.09 ? 7% perf-profile.children.cycles-pp.free_pgd_range
0.00 +0.1 0.09 ? 23% perf-profile.children.cycles-pp.perf_output_begin_forward
0.00 +0.1 0.09 ? 19% perf-profile.children.cycles-pp.shmem_write_end
0.00 +0.1 0.09 ? 22% perf-profile.children.cycles-pp.read
0.00 +0.1 0.09 ? 10% perf-profile.children.cycles-pp.prepend_path
0.00 +0.1 0.09 ? 17% perf-profile.children.cycles-pp.__cond_resched
0.00 +0.1 0.09 ? 14% perf-profile.children.cycles-pp.__might_sleep
0.00 +0.1 0.10 ? 15% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.00 +0.1 0.10 ? 7% perf-profile.children.cycles-pp.proc_reg_read
0.00 +0.1 0.10 ? 12% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
0.00 +0.1 0.10 ? 18% perf-profile.children.cycles-pp.core_kernel_text
0.00 +0.1 0.10 ? 12% perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.00 +0.1 0.10 ? 15% perf-profile.children.cycles-pp.update_sg_lb_stats
0.00 +0.1 0.10 ? 18% perf-profile.children.cycles-pp.tlb_gather_mmu
0.00 +0.1 0.10 ? 23% perf-profile.children.cycles-pp.fput
0.00 +0.1 0.10 ? 23% perf-profile.children.cycles-pp.remove_vma
0.00 +0.1 0.10 ? 15% perf-profile.children.cycles-pp._raw_spin_trylock
0.00 +0.1 0.10 ? 35% perf-profile.children.cycles-pp.ret_from_fork
0.00 +0.1 0.10 ? 35% perf-profile.children.cycles-pp.kthread
0.00 +0.1 0.10 ? 9% perf-profile.children.cycles-pp.perf_iterate_sb
0.00 +0.1 0.10 ? 16% perf-profile.children.cycles-pp.__update_blocked_fair
0.00 +0.1 0.10 ? 13% perf-profile.children.cycles-pp.__hrtimer_next_event_base
0.00 +0.1 0.10 ? 9% perf-profile.children.cycles-pp.seq_read
0.00 +0.1 0.11 ? 13% perf-profile.children.cycles-pp.__switch_to_asm
0.00 +0.1 0.11 ? 11% perf-profile.children.cycles-pp.mas_leaf_max_gap
0.00 +0.1 0.11 ? 28% perf-profile.children.cycles-pp.perf_poll
0.00 +0.1 0.11 ? 14% perf-profile.children.cycles-pp.select_task_rq
0.00 +0.1 0.11 ? 12% perf-profile.children.cycles-pp.io_serial_in
0.00 +0.1 0.11 ? 12% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.00 +0.1 0.12 ? 13% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.00 +0.1 0.12 ? 12% perf-profile.children.cycles-pp.seq_read_iter
0.00 +0.1 0.12 ? 30% perf-profile.children.cycles-pp.perf_mmap__read_init
0.00 +0.1 0.12 ? 10% perf-profile.children.cycles-pp.__might_resched
0.00 +0.1 0.12 ? 10% perf-profile.children.cycles-pp.__handle_mm_fault
0.00 +0.1 0.13 ? 10% perf-profile.children.cycles-pp.__libc_read
0.00 +0.1 0.13 ? 8% perf-profile.children.cycles-pp.prepare_task_switch
0.00 +0.1 0.13 ? 26% perf-profile.children.cycles-pp.do_poll
0.00 +0.1 0.13 ? 30% perf-profile.children.cycles-pp.ring_buffer_read_head
0.00 +0.1 0.13 ? 25% perf-profile.children.cycles-pp.evsel__parse_sample
0.00 +0.1 0.13 ? 6% perf-profile.children.cycles-pp.set_next_entity
0.00 +0.1 0.13 ? 13% perf-profile.children.cycles-pp.update_sd_lb_stats
0.00 +0.1 0.13 ? 10% perf-profile.children.cycles-pp.handle_mm_fault
0.00 +0.1 0.13 ? 15% perf-profile.children.cycles-pp.d_path
0.00 +0.1 0.13 ? 19% perf-profile.children.cycles-pp.fault_in_iov_iter_readable
0.00 +0.1 0.13 ? 19% perf-profile.children.cycles-pp.fault_in_readable
0.00 +0.1 0.13 ? 12% perf-profile.children.cycles-pp.hrtimer_next_event_without
0.00 +0.1 0.14 ? 10% perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
0.00 +0.1 0.14 ? 11% perf-profile.children.cycles-pp.mas_store_gfp
0.00 +0.1 0.14 ? 14% perf-profile.children.cycles-pp.find_busiest_group
0.00 +0.1 0.14 ? 7% perf-profile.children.cycles-pp.io__get_hex
0.01 ?223% +0.1 0.15 ? 17% perf-profile.children.cycles-pp.ktime_get_update_offsets_now
0.00 +0.1 0.14 ? 28% perf-profile.children.cycles-pp.calc_global_load_tick
0.00 +0.1 0.14 ? 13% perf-profile.children.cycles-pp.__switch_to
0.00 +0.1 0.14 ? 17% perf-profile.children.cycles-pp.perf_rotate_context
0.00 +0.1 0.14 ? 9% perf-profile.children.cycles-pp.wait_for_lsr
0.00 +0.1 0.15 ? 33% perf-profile.children.cycles-pp.perf_mmap__read_head
0.00 +0.1 0.15 ? 19% perf-profile.children.cycles-pp.shmem_write_begin
0.00 +0.1 0.15 ? 25% perf-profile.children.cycles-pp.__x64_sys_poll
0.00 +0.1 0.15 ? 25% perf-profile.children.cycles-pp.do_sys_poll
0.00 +0.2 0.15 ? 5% perf-profile.children.cycles-pp.mas_wr_node_store
0.00 +0.2 0.15 ? 11% perf-profile.children.cycles-pp.serial8250_console_write
0.00 +0.2 0.15 ? 25% perf-profile.children.cycles-pp.__poll
0.00 +0.2 0.15 ? 7% perf-profile.children.cycles-pp.unmap_page_range
0.00 +0.2 0.15 ? 10% perf-profile.children.cycles-pp.mas_wr_walk
0.00 +0.2 0.15 ? 36% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.00 +0.2 0.15 ? 17% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.sysvec_irq_work
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.__sysvec_irq_work
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.irq_work_run
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.irq_work_single
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp._printk
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.vprintk_emit
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.console_unlock
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.console_flush_all
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.console_emit_next_record
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.asm_sysvec_irq_work
0.00 +0.2 0.16 ? 15% perf-profile.children.cycles-pp.shmem_get_folio_gfp
0.00 +0.2 0.16 ? 12% perf-profile.children.cycles-pp.vm_area_alloc
0.00 +0.2 0.16 ? 32% perf-profile.children.cycles-pp.tick_sched_do_timer
0.00 +0.2 0.16 ? 17% perf-profile.children.cycles-pp.copyin
0.00 +0.2 0.16 ? 9% perf-profile.children.cycles-pp.mas_update_gap
0.00 +0.2 0.17 ? 10% perf-profile.children.cycles-pp.update_blocked_averages
0.00 +0.2 0.17 ? 10% perf-profile.children.cycles-pp.irq_work_run_list
0.00 +0.2 0.17 ? 15% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.00 +0.2 0.17 ? 9% perf-profile.children.cycles-pp.run_rebalance_domains
0.00 +0.2 0.17 ? 16% perf-profile.children.cycles-pp.copy_page_from_iter_atomic
0.00 +0.2 0.17 ? 12% perf-profile.children.cycles-pp.load_balance
0.00 +0.2 0.17 ? 14% perf-profile.children.cycles-pp.__perf_event_header__init_id
0.00 +0.2 0.18 ? 7% perf-profile.children.cycles-pp.do_user_addr_fault
0.00 +0.2 0.18 ? 15% perf-profile.children.cycles-pp.mas_next_entry
0.00 +0.2 0.18 ? 4% perf-profile.children.cycles-pp.mas_wr_modify
0.00 +0.2 0.18 ? 13% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.00 +0.2 0.18 ? 8% perf-profile.children.cycles-pp.irqtime_account_irq
0.00 +0.2 0.18 ? 8% perf-profile.children.cycles-pp.free_pgtables
0.00 +0.2 0.19 ? 11% perf-profile.children.cycles-pp.ksys_read
0.00 +0.2 0.19 ? 10% perf-profile.children.cycles-pp.vfs_read
0.00 +0.2 0.20 ? 5% perf-profile.children.cycles-pp.llist_add_batch
0.00 +0.2 0.20 ? 19% perf-profile.children.cycles-pp.evlist__parse_sample
0.00 +0.2 0.20 ? 5% perf-profile.children.cycles-pp.__smp_call_single_queue
0.00 +0.2 0.20 ? 6% perf-profile.children.cycles-pp.kmem_cache_free
0.00 +0.2 0.20 ? 15% perf-profile.children.cycles-pp.tick_irq_enter
0.04 ? 44% +0.2 0.25 ? 11% perf-profile.children.cycles-pp.mas_walk
0.00 +0.2 0.21 ? 11% perf-profile.children.cycles-pp.cmp_ex_search
0.00 +0.2 0.22 ? 5% perf-profile.children.cycles-pp.unmap_vmas
0.00 +0.2 0.22 ? 9% perf-profile.children.cycles-pp.update_rq_clock
0.00 +0.2 0.22 ? 16% perf-profile.children.cycles-pp.irq_enter_rcu
0.58 ? 7% +0.2 0.81 ? 10% perf-profile.children.cycles-pp.scheduler_tick
0.05 ? 7% +0.2 0.28 ? 11% perf-profile.children.cycles-pp.mas_find
0.00 +0.2 0.24 ? 11% perf-profile.children.cycles-pp.kallsyms__parse
0.04 ? 71% +0.2 0.28 ? 75% perf-profile.children.cycles-pp.intel_idle_irq
0.00 +0.2 0.25 ? 6% perf-profile.children.cycles-pp.lapic_next_deadline
0.06 ? 8% +0.2 0.30 ? 10% perf-profile.children.cycles-pp.do_mas_munmap
0.00 +0.2 0.25 ? 9% perf-profile.children.cycles-pp.read_tsc
0.00 +0.3 0.25 ? 10% perf-profile.children.cycles-pp._raw_spin_lock
0.00 +0.3 0.25 ? 13% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
0.00 +0.3 0.26 ? 13% perf-profile.children.cycles-pp.osq_unlock
0.00 +0.3 0.26 ? 11% perf-profile.children.cycles-pp.rebalance_domains
0.00 +0.3 0.26 ? 10% perf-profile.children.cycles-pp.kmem_cache_free_bulk
0.00 +0.3 0.27 ? 7% perf-profile.children.cycles-pp.stack_access_ok
0.00 +0.3 0.27 ? 12% perf-profile.children.cycles-pp.enqueue_entity
0.00 +0.3 0.28 ? 6% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.01 ?223% +0.3 0.29 ? 7% perf-profile.children.cycles-pp.update_load_avg
0.00 +0.3 0.28 ? 8% perf-profile.children.cycles-pp.pick_next_task_fair
0.00 +0.3 0.28 ? 14% perf-profile.children.cycles-pp.map__load
0.00 +0.3 0.29 ? 11% perf-profile.children.cycles-pp.dso__load
0.00 +0.3 0.29 ? 11% perf-profile.children.cycles-pp.__dso__load_kallsyms
0.00 +0.3 0.29 ? 15% perf-profile.children.cycles-pp.tick_nohz_irq_exit
0.00 +0.3 0.29 ? 11% perf-profile.children.cycles-pp.arch_scale_freq_tick
0.00 +0.3 0.29 ? 10% perf-profile.children.cycles-pp.downgrade_write
0.01 ?223% +0.3 0.30 ? 10% perf-profile.children.cycles-pp.kernel_text_address
0.00 +0.3 0.30 ? 5% perf-profile.children.cycles-pp.native_sched_clock
0.00 +0.3 0.30 ? 10% perf-profile.children.cycles-pp.__entry_text_start
0.00 +0.3 0.30 ? 11% perf-profile.children.cycles-pp.orc_find
0.00 +0.3 0.32 ? 7% perf-profile.children.cycles-pp.sysvec_call_function_single
0.02 ?141% +0.3 0.34 ? 6% perf-profile.children.cycles-pp.memset_erms
0.01 ?223% +0.3 0.33 ? 10% perf-profile.children.cycles-pp.mas_wr_store_entry
0.00 +0.3 0.32 ? 14% perf-profile.children.cycles-pp.thread__find_map
0.00 +0.3 0.33 ? 10% perf-profile.children.cycles-pp.bsearch
0.06 ? 9% +0.3 0.39 ? 7% perf-profile.children.cycles-pp.mas_destroy
0.06 ? 7% +0.3 0.40 ? 9% perf-profile.children.cycles-pp.mas_rev_awalk
0.00 +0.3 0.34 ? 11% perf-profile.children.cycles-pp.enqueue_task_fair
0.00 +0.3 0.34 ? 9% perf-profile.children.cycles-pp.search_extable
0.00 +0.3 0.34 ? 7% perf-profile.children.cycles-pp.sched_clock_cpu
0.00 +0.3 0.34 ? 10% perf-profile.children.cycles-pp.wake_q_add
0.00 +0.3 0.34 ? 7% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.00 +0.3 0.34 ? 9% perf-profile.children.cycles-pp.search_exception_tables
0.01 ?223% +0.3 0.35 ? 7% perf-profile.children.cycles-pp.memcpy_erms
0.00 +0.3 0.34 ? 8% perf-profile.children.cycles-pp.kmem_cache_alloc
0.05 +0.3 0.40 ? 10% perf-profile.children.cycles-pp.perf_event_mmap_event
0.00 +0.3 0.35 ? 9% perf-profile.children.cycles-pp.ttwu_do_activate
0.00 +0.4 0.36 ? 18% perf-profile.children.cycles-pp.timekeeping_max_deferment
0.05 +0.4 0.42 ? 10% perf-profile.children.cycles-pp.perf_event_mmap
0.00 +0.4 0.37 ? 4% perf-profile.children.cycles-pp.perf_output_copy
0.00 +0.4 0.38 ? 9% perf-profile.children.cycles-pp.fixup_exception
0.08 ? 6% +0.4 0.46 ? 8% perf-profile.children.cycles-pp.mas_empty_area_rev
0.69 ? 7% +0.4 1.08 ? 9% perf-profile.children.cycles-pp.update_process_times
0.01 ?223% +0.4 0.40 ? 11% perf-profile.children.cycles-pp.build_id__mark_dso_hit
0.00 +0.4 0.41 ? 11% perf-profile.children.cycles-pp.kernelmode_fixup_or_oops
0.04 ? 45% +0.4 0.45 ? 10% perf-profile.children.cycles-pp.__kernel_text_address
0.03 ? 70% +0.4 0.44 ? 6% perf-profile.children.cycles-pp.__output_copy
0.70 ? 6% +0.4 1.11 ? 9% perf-profile.children.cycles-pp.tick_sched_handle
0.07 ? 6% +0.4 0.50 ? 3% perf-profile.children.cycles-pp.native_irq_return_iret
0.00 +0.4 0.43 ? 8% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.08 ? 7% +0.4 0.53 ? 7% perf-profile.children.cycles-pp.vm_unmapped_area
0.09 ? 5% +0.5 0.57 ? 8% perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
0.08 ? 8% +0.5 0.58 ? 7% perf-profile.children.cycles-pp.kmem_cache_alloc_bulk
0.05 +0.5 0.56 ? 6% perf-profile.children.cycles-pp.perf_output_sample
0.07 ? 7% +0.5 0.59 ? 9% perf-profile.children.cycles-pp.unwind_get_return_address
0.00 +0.5 0.53 ? 8% perf-profile.children.cycles-pp.__do_softirq
0.11 ? 6% +0.5 0.65 ? 8% perf-profile.children.cycles-pp.shmem_get_unmapped_area
0.09 ? 7% +0.5 0.63 ? 8% perf-profile.children.cycles-pp.unmap_region
0.02 ?141% +0.5 0.56 ? 9% perf-profile.children.cycles-pp.generic_perform_write
0.02 ? 99% +0.5 0.56 ? 9% perf-profile.children.cycles-pp.__generic_file_write_iter
0.00 +0.5 0.54 ? 8% perf-profile.children.cycles-pp.flush_smp_call_function_queue
0.02 ? 99% +0.6 0.58 ? 10% perf-profile.children.cycles-pp.generic_file_write_iter
0.12 ? 6% +0.6 0.68 ? 8% perf-profile.children.cycles-pp.get_unmapped_area
0.00 +0.6 0.56 ? 5% perf-profile.children.cycles-pp.sched_ttwu_pending
0.07 ? 10% +0.6 0.64 ? 6% perf-profile.children.cycles-pp.exc_page_fault
0.06 ? 9% +0.6 0.63 ? 10% perf-profile.children.cycles-pp.__orc_find
0.00 +0.6 0.59 ? 56% perf-profile.children.cycles-pp.start_kernel
0.00 +0.6 0.59 ? 56% perf-profile.children.cycles-pp.arch_call_rest_init
0.00 +0.6 0.59 ? 56% perf-profile.children.cycles-pp.rest_init
0.04 ? 71% +0.6 0.62 ? 10% perf-profile.children.cycles-pp.vfs_write
0.07 +0.6 0.67 ? 5% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.04 ? 71% +0.6 0.64 ? 11% perf-profile.children.cycles-pp.ksys_write
0.06 ? 6% +0.6 0.68 ? 8% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.00 +0.6 0.62 ? 16% perf-profile.children.cycles-pp.tick_nohz_next_event
0.72 ? 6% +0.6 1.36 ? 7% perf-profile.children.cycles-pp.tick_sched_timer
0.04 ? 45% +0.7 0.72 ? 13% perf-profile.children.cycles-pp.record__pushfn
0.00 +0.7 0.68 ? 7% perf-profile.children.cycles-pp.__irq_exit_rcu
0.06 ? 47% +0.7 0.76 ? 11% perf-profile.children.cycles-pp.perf_session__deliver_event
0.00 +0.7 0.72 ? 6% perf-profile.children.cycles-pp.finish_task_switch
0.06 ? 47% +0.7 0.79 ? 10% perf-profile.children.cycles-pp.__ordered_events__flush
0.10 ? 4% +0.7 0.83 ? 6% perf-profile.children.cycles-pp.mas_preallocate
0.06 ? 47% +0.7 0.79 ? 10% perf-profile.children.cycles-pp.perf_session__process_user_event
0.04 ? 45% +0.7 0.78 ? 13% perf-profile.children.cycles-pp.writen
0.11 ? 6% +0.7 0.85 ? 6% perf-profile.children.cycles-pp.mas_alloc_nodes
0.04 ? 45% +0.7 0.79 ? 13% perf-profile.children.cycles-pp.__libc_write
0.00 +0.8 0.76 ? 10% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.00 +0.8 0.79 ? 14% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.18 ? 4% +0.9 1.06 ? 4% perf-profile.children.cycles-pp.mas_store_prealloc
0.06 ? 11% +0.9 0.97 ? 17% perf-profile.children.cycles-pp.perf_mmap__push
0.12 ? 18% +1.0 1.07 ? 16% perf-profile.children.cycles-pp.queue_event
0.12 ? 18% +1.0 1.10 ? 15% perf-profile.children.cycles-pp.ordered_events__queue
0.12 ? 20% +1.1 1.18 ? 15% perf-profile.children.cycles-pp.process_simple
0.76 ? 6% +1.1 1.83 ? 8% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.06 ? 13% +1.1 1.12 ? 17% perf-profile.children.cycles-pp.record__mmap_read_evlist
0.12 ? 8% +1.1 1.20 ? 8% perf-profile.children.cycles-pp.asm_exc_page_fault
0.16 ? 8% +1.1 1.28 ? 7% perf-profile.children.cycles-pp.__get_user_nocheck_8
0.00 +1.1 1.12 ? 7% perf-profile.children.cycles-pp.clockevents_program_event
0.16 ? 8% +1.2 1.32 ? 7% perf-profile.children.cycles-pp.perf_callchain_user
0.13 ? 3% +1.3 1.39 ? 8% perf-profile.children.cycles-pp.__unwind_start
0.04 ? 45% +1.3 1.32 ? 10% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.06 ? 11% +1.3 1.34 ? 19% perf-profile.children.cycles-pp.poll_idle
0.00 +1.4 1.42 ? 8% perf-profile.children.cycles-pp.ktime_get
0.48 ? 8% +1.4 1.92 ? 8% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.00 +1.6 1.58 ? 9% perf-profile.children.cycles-pp.menu_select
0.50 ? 8% +1.6 2.09 ? 8% perf-profile.children.cycles-pp.update_curr
0.08 ? 10% +1.6 1.72 ? 7% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
0.20 ? 19% +1.9 2.08 ? 12% perf-profile.children.cycles-pp.reader__read_event
0.20 ? 18% +1.9 2.09 ? 12% perf-profile.children.cycles-pp.perf_session__process_events
0.18 ? 42% +1.9 2.09 ? 32% perf-profile.children.cycles-pp.cmd_record
0.18 ? 47% +1.9 2.09 ? 12% perf-profile.children.cycles-pp.record__finish_output
0.17 ? 43% +1.9 2.09 ? 32% perf-profile.children.cycles-pp.cmd_sched
0.06 ? 11% +1.9 1.99 ? 7% perf-profile.children.cycles-pp.schedule_idle
0.10 ? 11% +2.0 2.05 ? 5% perf-profile.children.cycles-pp.up_read
0.19 ? 40% +2.0 2.15 ? 31% perf-profile.children.cycles-pp.__libc_start_main
0.19 ? 40% +2.0 2.15 ? 31% perf-profile.children.cycles-pp.main
0.19 ? 40% +2.0 2.15 ? 31% perf-profile.children.cycles-pp.run_builtin
0.31 +2.0 2.31 ? 5% perf-profile.children.cycles-pp.mmap_region
0.26 ? 5% +2.0 2.30 ? 6% perf-profile.children.cycles-pp.unwind_next_frame
0.08 ? 6% +2.1 2.14 ? 8% perf-profile.children.cycles-pp.dequeue_entity
0.08 ? 6% +2.1 2.19 ? 8% perf-profile.children.cycles-pp.dequeue_task_fair
0.39 +2.3 2.70 ? 6% perf-profile.children.cycles-pp.do_mas_align_munmap
0.86 ? 5% +2.4 3.31 ? 7% perf-profile.children.cycles-pp.hrtimer_interrupt
0.87 ? 5% +2.5 3.38 ? 8% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.10 ? 8% +2.6 2.66 ? 7% perf-profile.children.cycles-pp.perf_trace_sched_switch
0.12 ? 5% +2.6 2.67 ? 7% perf-profile.children.cycles-pp.up_write
0.44 +2.7 3.12 ? 6% perf-profile.children.cycles-pp.do_mmap
0.37 ? 5% +3.0 3.37 ? 7% perf-profile.children.cycles-pp.perf_callchain_kernel
0.26 ? 33% +3.1 3.37 ? 11% perf-profile.children.cycles-pp.__cmd_record
0.74 ? 2% +3.2 3.94 ? 5% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.13 ? 8% +3.5 3.60 ? 7% perf-profile.children.cycles-pp.try_to_wake_up
0.13 ? 8% +3.6 3.70 ? 7% perf-profile.children.cycles-pp.wake_up_q
0.90 ? 5% +3.8 4.70 ? 7% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.16 ? 10% +4.2 4.32 ? 7% perf-profile.children.cycles-pp.rwsem_wake
0.54 ? 5% +4.3 4.86 ? 7% perf-profile.children.cycles-pp.get_perf_callchain
0.55 ? 5% +4.3 4.88 ? 7% perf-profile.children.cycles-pp.perf_callchain
1.00 ? 5% +4.4 5.38 ? 8% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.18 ? 7% +4.5 4.64 ? 7% perf-profile.children.cycles-pp.schedule_preempt_disabled
0.19 ? 7% +4.5 4.71 ? 7% perf-profile.children.cycles-pp.schedule
0.57 ? 5% +4.6 5.19 ? 7% perf-profile.children.cycles-pp.perf_prepare_sample
0.64 ? 4% +5.3 5.90 ? 7% perf-profile.children.cycles-pp.perf_event_output_forward
0.64 ? 4% +5.3 5.94 ? 7% perf-profile.children.cycles-pp.__perf_event_overflow
0.65 ? 4% +5.5 6.12 ? 7% perf-profile.children.cycles-pp.perf_tp_event
0.25 ? 8% +6.4 6.66 ? 6% perf-profile.children.cycles-pp.__schedule
1.21 ? 51% +29.8 31.05 ? 47% perf-profile.children.cycles-pp.intel_idle
1.78 ? 15% +38.4 40.22 ? 5% perf-profile.children.cycles-pp.mwait_idle_with_hints
1.86 ? 15% +45.4 47.29 ? 4% perf-profile.children.cycles-pp.cpuidle_enter_state
1.86 ? 15% +45.5 47.31 ? 4% perf-profile.children.cycles-pp.cpuidle_enter
1.86 ? 15% +47.3 49.19 ? 3% perf-profile.children.cycles-pp.cpuidle_idle_call
1.92 ? 14% +49.6 51.50 ? 3% perf-profile.children.cycles-pp.start_secondary
1.94 ? 15% +50.1 52.08 ? 3% perf-profile.children.cycles-pp.do_idle
1.94 ? 15% +50.1 52.08 ? 3% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
1.94 ? 15% +50.1 52.08 ? 3% perf-profile.children.cycles-pp.cpu_startup_entry
94.32 -74.4 19.96 ? 11% perf-profile.self.cycles-pp.osq_lock
0.04 ? 45% +0.0 0.08 ? 14% perf-profile.self.cycles-pp.update_cfs_group
0.00 +0.1 0.05 ? 8% perf-profile.self.cycles-pp.mas_empty_area_rev
0.00 +0.1 0.06 ? 14% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.00 +0.1 0.06 ? 17% perf-profile.self.cycles-pp.prepend_path
0.00 +0.1 0.06 ? 14% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.00 +0.1 0.06 ? 17% perf-profile.self.cycles-pp.fault_in_readable
0.00 +0.1 0.06 ? 17% perf-profile.self.cycles-pp.unmap_page_range
0.00 +0.1 0.06 ? 21% perf-profile.self.cycles-pp.mas_alloc_nodes
0.00 +0.1 0.07 ? 16% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.00 +0.1 0.07 ? 14% perf-profile.self.cycles-pp.evlist__parse_sample
0.00 +0.1 0.07 ? 17% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.00 +0.1 0.07 ? 11% perf-profile.self.cycles-pp.enqueue_task_fair
0.00 +0.1 0.07 ? 20% perf-profile.self.cycles-pp.mas_next_nentry
0.00 +0.1 0.07 ? 18% perf-profile.self.cycles-pp.core_kernel_text
0.00 +0.1 0.08 ? 12% perf-profile.self.cycles-pp.update_sg_lb_stats
0.00 +0.1 0.08 ? 14% perf-profile.self.cycles-pp.shmem_get_unmapped_area
0.00 +0.1 0.08 ? 12% perf-profile.self.cycles-pp.__hrtimer_run_queues
0.00 +0.1 0.08 ? 19% perf-profile.self.cycles-pp.update_curr
0.00 +0.1 0.08 ? 17% perf-profile.self.cycles-pp.error_entry
0.00 +0.1 0.08 ? 14% perf-profile.self.cycles-pp.dequeue_entity
0.00 +0.1 0.08 ? 20% perf-profile.self.cycles-pp.irqtime_account_irq
0.00 +0.1 0.08 ? 27% perf-profile.self.cycles-pp.__intel_pmu_enable_all
0.00 +0.1 0.08 ? 10% perf-profile.self.cycles-pp.perf_iterate_sb
0.00 +0.1 0.08 ? 19% perf-profile.self.cycles-pp.reader__read_event
0.00 +0.1 0.08 ? 16% perf-profile.self.cycles-pp.prepare_task_switch
0.00 +0.1 0.08 ? 16% perf-profile.self.cycles-pp.__hrtimer_next_event_base
0.00 +0.1 0.08 ? 17% perf-profile.self.cycles-pp.hrtimer_interrupt
0.00 +0.1 0.09 ? 12% perf-profile.self.cycles-pp.__update_load_avg_se
0.00 +0.1 0.09 ? 8% perf-profile.self.cycles-pp.rwsem_mark_wake
0.00 +0.1 0.09 ? 19% perf-profile.self.cycles-pp.perf_output_begin_forward
0.00 +0.1 0.09 ? 15% perf-profile.self.cycles-pp.__might_sleep
0.00 +0.1 0.09 ? 18% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.00 +0.1 0.09 ? 23% perf-profile.self.cycles-pp.do_idle
0.00 +0.1 0.09 ? 12% perf-profile.self.cycles-pp.unmap_region
0.00 +0.1 0.09 ? 11% perf-profile.self.cycles-pp.perf_prepare_sample
0.00 +0.1 0.09 ? 19% perf-profile.self.cycles-pp.mas_next_entry
0.00 +0.1 0.09 ? 17% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
0.00 +0.1 0.10 ? 8% perf-profile.self.cycles-pp.mas_wr_node_store
0.00 +0.1 0.10 ? 8% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.00 +0.1 0.10 ? 11% perf-profile.self.cycles-pp.perf_output_sample
0.00 +0.1 0.10 ? 22% perf-profile.self.cycles-pp.fput
0.00 +0.1 0.10 ? 18% perf-profile.self.cycles-pp.tlb_gather_mmu
0.01 ?223% +0.1 0.11 ? 24% perf-profile.self.cycles-pp.ktime_get_update_offsets_now
0.00 +0.1 0.10 ? 15% perf-profile.self.cycles-pp._raw_spin_trylock
0.00 +0.1 0.10 ? 9% perf-profile.self.cycles-pp.update_rq_clock
0.00 +0.1 0.11 ? 14% perf-profile.self.cycles-pp.mas_leaf_max_gap
0.00 +0.1 0.11 ? 13% perf-profile.self.cycles-pp.__switch_to_asm
0.00 +0.1 0.11 ? 18% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.00 +0.1 0.11 ? 25% perf-profile.self.cycles-pp.mas_wr_store_entry
0.00 +0.1 0.11 ? 17% perf-profile.self.cycles-pp.downgrade_write
0.00 +0.1 0.11 ? 14% perf-profile.self.cycles-pp.enqueue_entity
0.00 +0.1 0.11 ? 12% perf-profile.self.cycles-pp.io_serial_in
0.00 +0.1 0.11 ? 11% perf-profile.self.cycles-pp.rwsem_wake
0.00 +0.1 0.11 ? 4% perf-profile.self.cycles-pp.up_read
0.00 +0.1 0.12 ? 10% perf-profile.self.cycles-pp.bsearch
0.00 +0.1 0.12 ? 6% perf-profile.self.cycles-pp.update_load_avg
0.00 +0.1 0.12 ? 13% perf-profile.self.cycles-pp.__kernel_text_address
0.00 +0.1 0.12 ? 8% perf-profile.self.cycles-pp.perf_tp_event
0.00 +0.1 0.12 ? 22% perf-profile.self.cycles-pp.tick_nohz_next_event
0.00 +0.1 0.12 ? 9% perf-profile.self.cycles-pp.kmem_cache_alloc_bulk
0.00 +0.1 0.12 ? 10% perf-profile.self.cycles-pp.__might_resched
0.00 +0.1 0.12 ? 8% perf-profile.self.cycles-pp.kmem_cache_alloc
0.00 +0.1 0.12 ? 13% perf-profile.self.cycles-pp.kernel_text_address
0.00 +0.1 0.12 ? 14% perf-profile.self.cycles-pp.wake_up_q
0.00 +0.1 0.12 ? 19% perf-profile.self.cycles-pp.do_mmap
0.00 +0.1 0.12 ? 22% perf-profile.self.cycles-pp.evsel__parse_sample
0.00 +0.1 0.13 ? 30% perf-profile.self.cycles-pp.ring_buffer_read_head
0.00 +0.1 0.13 ? 10% perf-profile.self.cycles-pp.__kmem_cache_alloc_bulk
0.00 +0.1 0.13 ? 11% perf-profile.self.cycles-pp.get_perf_callchain
0.00 +0.1 0.14 ? 13% perf-profile.self.cycles-pp.mas_wr_walk
0.00 +0.1 0.14 ? 27% perf-profile.self.cycles-pp.calc_global_load_tick
0.00 +0.1 0.14 ? 12% perf-profile.self.cycles-pp.__switch_to
0.00 +0.1 0.14 ? 10% perf-profile.self.cycles-pp.__output_copy
0.00 +0.1 0.14 ? 40% perf-profile.self.cycles-pp.tick_sched_do_timer
0.00 +0.1 0.15 ? 37% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.00 +0.2 0.16 ? 13% perf-profile.self.cycles-pp.__unwind_start
0.00 +0.2 0.16 ? 8% perf-profile.self.cycles-pp.unwind_get_return_address
0.00 +0.2 0.16 ? 15% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.00 +0.2 0.17 ? 7% perf-profile.self.cycles-pp.kmem_cache_free
0.00 +0.2 0.18 ? 10% perf-profile.self.cycles-pp.cpuidle_idle_call
0.00 +0.2 0.20 ? 5% perf-profile.self.cycles-pp.llist_add_batch
0.04 ? 44% +0.2 0.25 ? 11% perf-profile.self.cycles-pp.mas_walk
0.00 +0.2 0.21 ? 11% perf-profile.self.cycles-pp.cmp_ex_search
0.00 +0.2 0.23 ? 9% perf-profile.self.cycles-pp._raw_spin_lock
0.00 +0.2 0.24 ? 6% perf-profile.self.cycles-pp.stack_access_ok
0.00 +0.2 0.24 ? 6% perf-profile.self.cycles-pp.lapic_next_deadline
0.00 +0.2 0.25 ? 10% perf-profile.self.cycles-pp.read_tsc
0.00 +0.2 0.25 ? 9% perf-profile.self.cycles-pp.kmem_cache_free_bulk
0.00 +0.3 0.25 ? 13% perf-profile.self.cycles-pp.osq_unlock
0.00 +0.3 0.26 ? 17% perf-profile.self.cycles-pp.do_mas_align_munmap
0.00 +0.3 0.26 ? 12% perf-profile.self.cycles-pp.__entry_text_start
0.05 ? 7% +0.3 0.32 ? 9% perf-profile.self.cycles-pp.mmap_region
0.00 +0.3 0.27 ? 6% perf-profile.self.cycles-pp.rwsem_down_write_slowpath
0.00 +0.3 0.28 ? 10% perf-profile.self.cycles-pp.up_write
0.00 +0.3 0.29 ? 9% perf-profile.self.cycles-pp.__schedule
0.00 +0.3 0.29 ? 11% perf-profile.self.cycles-pp.arch_scale_freq_tick
0.00 +0.3 0.29 ? 9% perf-profile.self.cycles-pp.perf_callchain_kernel
0.00 +0.3 0.30 ? 4% perf-profile.self.cycles-pp.native_sched_clock
0.00 +0.3 0.30 ? 10% perf-profile.self.cycles-pp.orc_find
0.01 ?223% +0.3 0.32 ? 7% perf-profile.self.cycles-pp.memset_erms
0.06 ? 7% +0.3 0.39 ? 8% perf-profile.self.cycles-pp.mas_rev_awalk
0.00 +0.3 0.34 ? 5% perf-profile.self.cycles-pp.finish_task_switch
0.00 +0.3 0.34 ? 10% perf-profile.self.cycles-pp.wake_q_add
0.00 +0.3 0.34 ? 8% perf-profile.self.cycles-pp.memcpy_erms
0.01 ?223% +0.4 0.36 ? 6% perf-profile.self.cycles-pp.down_write_killable
0.00 +0.4 0.36 ? 18% perf-profile.self.cycles-pp.timekeeping_max_deferment
0.00 +0.4 0.39 ? 9% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.07 ? 6% +0.4 0.50 ? 3% perf-profile.self.cycles-pp.native_irq_return_iret
0.00 +0.5 0.50 ? 8% perf-profile.self.cycles-pp.try_to_wake_up
0.02 ?141% +0.5 0.52 ? 9% perf-profile.self.cycles-pp.__get_user_nocheck_8
0.00 +0.6 0.56 ? 5% perf-profile.self.cycles-pp.cpuidle_enter_state
0.06 ? 7% +0.6 0.63 ? 5% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
0.06 ? 9% +0.6 0.63 ? 10% perf-profile.self.cycles-pp.__orc_find
0.02 ?141% +0.6 0.60 ? 16% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.06 ? 6% +0.6 0.68 ? 8% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.00 +0.7 0.69 ? 8% perf-profile.self.cycles-pp.menu_select
0.00 +0.8 0.76 ? 11% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.17 ? 2% +0.8 0.95 ? 6% perf-profile.self.cycles-pp.rwsem_optimistic_spin
0.12 ? 20% +0.9 1.04 ? 16% perf-profile.self.cycles-pp.queue_event
0.13 ? 5% +1.0 1.08 ? 6% perf-profile.self.cycles-pp.unwind_next_frame
0.00 +1.2 1.21 ? 8% perf-profile.self.cycles-pp.ktime_get
0.06 ? 13% +1.3 1.32 ? 19% perf-profile.self.cycles-pp.poll_idle
0.73 ? 2% +3.1 3.86 ? 5% perf-profile.self.cycles-pp.rwsem_spin_on_owner
1.78 ? 15% +38.4 40.19 ? 5% perf-profile.self.cycles-pp.mwait_idle_with_hints

***************************************************************************************************
lkp-csl-2sp9: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz (Cascade Lake) with 128G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-11/performance/x86_64-rhel-8.3/thread/100%/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp9/mmap2/will-it-scale

commit:
6eb0bdc24a ("locking/rwsem: Disable preemption at all down_write*() and up_write() code paths")
12ea6b720a ("locking/rwsem: Enable direct rwsem lock handoff")

6eb0bdc24acb1b4e 12ea6b720a0e614611f626df106
---------------- ---------------------------
%stddev %change %stddev
\ | \
134457 +19.3% 160439 will-it-scale.88.threads
1.73 +4582.2% 81.16 will-it-scale.88.threads_idle
1527 +19.3% 1822 will-it-scale.per_thread_ops
134457 +19.3% 160439 will-it-scale.workload
4425 +487.1% 25984 uptime.idle
6.861e+08 ? 2% +3132.4% 2.218e+10 cpuidle..time
10162729 ? 3% +960.5% 1.078e+08 ? 2% cpuidle..usage
404235 ? 10% +30.4% 527073 ? 8% numa-numastat.node1.local_node
448886 ? 11% +25.0% 561227 ? 4% numa-numastat.node1.numa_hit
60605 ? 4% -72.7% 16527 ? 3% meminfo.Active
60482 ? 4% -72.9% 16401 ? 3% meminfo.Active(anon)
86284 ? 3% -42.7% 49465 ? 2% meminfo.Shmem
57004 ? 5% -76.8% 13201 ? 8% numa-meminfo.node1.Active
56946 ? 5% -77.0% 13097 ? 8% numa-meminfo.node1.Active(anon)
69677 ? 7% -46.8% 37034 ? 18% numa-meminfo.node1.Shmem
2.46 ? 5% +78.8 81.27 mpstat.cpu.all.idle%
0.86 -0.1 0.75 ? 3% mpstat.cpu.all.irq%
0.01 ? 3% +0.0 0.06 mpstat.cpu.all.soft%
96.57 -78.7 17.83 ? 7% mpstat.cpu.all.sys%
14240 ? 5% -77.0% 3278 ? 8% numa-vmstat.node1.nr_active_anon
17414 ? 7% -46.7% 9278 ? 18% numa-vmstat.node1.nr_shmem
14240 ? 5% -77.0% 3278 ? 8% numa-vmstat.node1.nr_zone_active_anon
448824 ? 11% +25.0% 560977 ? 4% numa-vmstat.node1.numa_hit
404174 ? 10% +30.3% 526823 ? 8% numa-vmstat.node1.numa_local
2.67 ? 17% +2925.0% 80.67 vmstat.cpu.id
96.17 -81.3% 18.00 ? 6% vmstat.cpu.sy
80.83 -83.7% 13.17 ? 9% vmstat.procs.r
65143 ? 4% +578.4% 441903 ? 2% vmstat.system.cs
182913 +16.1% 212312 vmstat.system.in
15238 ? 5% -73.1% 4098 ? 3% proc-vmstat.nr_active_anon
747965 -1.2% 738910 proc-vmstat.nr_file_pages
91814 +1.6% 93316 proc-vmstat.nr_inactive_anon
21587 ? 3% -42.7% 12377 ? 2% proc-vmstat.nr_shmem
15238 ? 5% -73.1% 4098 ? 3% proc-vmstat.nr_zone_active_anon
91814 +1.6% 93316 proc-vmstat.nr_zone_inactive_anon
864519 +10.9% 959067 proc-vmstat.numa_hit
784745 +12.1% 879399 proc-vmstat.numa_local
139344 +47.1% 204977 ? 3% proc-vmstat.pgactivate
956317 +8.9% 1041881 proc-vmstat.pgalloc_normal
911190 +11.4% 1015162 proc-vmstat.pgfree
2724 -81.2% 511.17 ? 6% turbostat.Avg_MHz
97.56 -79.1 18.41 ? 6% turbostat.Busy%
1051738 ? 29% +196.9% 3122271 ? 56% turbostat.C1
0.18 ? 57% +1.4 1.58 ? 79% turbostat.C1%
1307315 ? 31% +3885.8% 52107440 ? 14% turbostat.C1E
1.60 ? 43% +68.6 70.20 ? 30% turbostat.C1E%
2.28 ? 4% +3472.2% 81.45 turbostat.CPU%c1
0.08 +12.5% 0.09 turbostat.IPC
55869605 +15.9% 64754284 turbostat.IRQ
7602308 ? 4% +545.0% 49038375 ? 4% turbostat.POLL
0.09 ? 5% +0.9 0.99 ? 11% turbostat.POLL%
57.33 -10.2% 51.50 turbostat.PkgTmp
224.85 -28.7% 160.26 turbostat.PkgWatt
12.60 +2.6% 12.92 turbostat.RAMWatt
0.81 ? 5% -76.1% 0.19 ? 9% sched_debug.cfs_rq:/.h_nr_running.avg
1.64 ? 13% -37.3% 1.03 ? 6% sched_debug.cfs_rq:/.h_nr_running.max
0.26 ? 16% +48.3% 0.39 ? 3% sched_debug.cfs_rq:/.h_nr_running.stddev
11354 ? 5% +12.1% 12725 ? 6% sched_debug.cfs_rq:/.load.avg
7400 ? 81% +249.7% 25877 ? 7% sched_debug.cfs_rq:/.load.stddev
218.31 ? 24% +117.5% 474.92 ? 25% sched_debug.cfs_rq:/.load_avg.max
4.42 ? 19% -100.0% 0.00 sched_debug.cfs_rq:/.load_avg.min
44.35 ? 17% +77.3% 78.64 ? 15% sched_debug.cfs_rq:/.load_avg.stddev
12264992 -97.5% 307686 ? 13% sched_debug.cfs_rq:/.min_vruntime.avg
12311125 -97.0% 371535 ? 14% sched_debug.cfs_rq:/.min_vruntime.max
12151231 -97.9% 254886 ? 13% sched_debug.cfs_rq:/.min_vruntime.min
0.80 ? 4% -75.9% 0.19 ? 8% sched_debug.cfs_rq:/.nr_running.avg
0.24 ? 22% +65.3% 0.39 ? 3% sched_debug.cfs_rq:/.nr_running.stddev
170.67 +150.0% 426.67 ? 30% sched_debug.cfs_rq:/.removed.load_avg.max
36.65 ? 21% +83.2% 67.14 ? 19% sched_debug.cfs_rq:/.removed.load_avg.stddev
86.89 +140.7% 209.11 ? 28% sched_debug.cfs_rq:/.removed.runnable_avg.max
16.38 ? 21% +86.2% 30.50 ? 20% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
86.89 +140.7% 209.11 ? 28% sched_debug.cfs_rq:/.removed.util_avg.max
16.38 ? 21% +86.2% 30.50 ? 20% sched_debug.cfs_rq:/.removed.util_avg.stddev
819.44 ? 5% -77.8% 182.12 ? 6% sched_debug.cfs_rq:/.runnable_avg.avg
1256 ? 8% -42.5% 722.28 ? 3% sched_debug.cfs_rq:/.runnable_avg.max
365.75 ? 19% -99.7% 0.94 ? 59% sched_debug.cfs_rq:/.runnable_avg.min
817.44 ? 5% -77.7% 182.08 ? 6% sched_debug.cfs_rq:/.util_avg.avg
1190 ? 7% -39.3% 722.25 ? 3% sched_debug.cfs_rq:/.util_avg.max
353.69 ? 18% -99.7% 0.94 ? 59% sched_debug.cfs_rq:/.util_avg.min
685.44 ? 6% -94.0% 41.09 ? 18% sched_debug.cfs_rq:/.util_est_enqueued.avg
1083 ? 10% -56.5% 471.50 ? 6% sched_debug.cfs_rq:/.util_est_enqueued.max
219.37 ? 15% -52.4% 104.46 ? 10% sched_debug.cfs_rq:/.util_est_enqueued.stddev
358410 ? 6% +76.1% 631092 ? 2% sched_debug.cpu.avg_idle.avg
286334 ? 10% +30.9% 374891 sched_debug.cpu.avg_idle.stddev
4.80 ? 15% -45.2% 2.63 ? 2% sched_debug.cpu.clock.stddev
3953 ? 5% -76.7% 921.70 ? 8% sched_debug.cpu.curr->pid.avg
1250 ? 18% +56.7% 1959 ? 3% sched_debug.cpu.curr->pid.stddev
0.00 ? 35% -54.3% 0.00 ? 7% sched_debug.cpu.next_balance.stddev
0.81 ? 5% -77.2% 0.18 ? 9% sched_debug.cpu.nr_running.avg
1.64 ? 14% -37.3% 1.03 ? 6% sched_debug.cpu.nr_running.max
0.27 ? 15% +42.9% 0.38 ? 4% sched_debug.cpu.nr_running.stddev
117814 ? 4% +537.5% 751075 ? 2% sched_debug.cpu.nr_switches.avg
326346 ? 11% +195.8% 965323 ? 7% sched_debug.cpu.nr_switches.max
48447 ? 9% +1083.0% 573116 ? 7% sched_debug.cpu.nr_switches.min
53541 ? 21% +138.2% 127556 ? 36% sched_debug.cpu.nr_switches.stddev
0.53 ? 3% +839.3% 4.94 ? 6% perf-stat.i.MPKI
1.278e+10 -77.3% 2.898e+09 ? 5% perf-stat.i.branch-instructions
0.11 ? 2% +0.4 0.56 ? 4% perf-stat.i.branch-miss-rate%
13846247 ? 2% +16.7% 16157972 perf-stat.i.branch-misses
45.75 ? 2% -6.0 39.78 ? 3% perf-stat.i.cache-miss-rate%
15121997 ? 4% +80.9% 27354104 ? 3% perf-stat.i.cache-misses
33343137 ? 3% +106.6% 68874470 ? 2% perf-stat.i.cache-references
65852 ? 4% +578.0% 446459 ? 2% perf-stat.i.context-switches
3.77 -17.0% 3.13 perf-stat.i.cpi
2.403e+11 -81.5% 4.458e+10 ? 6% perf-stat.i.cpu-cycles
147.90 -16.0% 124.29 perf-stat.i.cpu-migrations
15980 ? 4% -89.9% 1621 ? 5% perf-stat.i.cycles-between-cache-misses
1.689e+10 -78.2% 3.683e+09 ? 5% perf-stat.i.dTLB-loads
0.00 ? 5% -0.0 0.00 ? 3% perf-stat.i.dTLB-store-miss-rate%
4.344e+08 +73.6% 7.539e+08 perf-stat.i.dTLB-stores
93.88 -40.9 52.98 perf-stat.i.iTLB-load-miss-rate%
1566558 ? 8% +53.1% 2398677 perf-stat.i.iTLB-load-misses
109261 ? 2% +1829.3% 2107955 ? 2% perf-stat.i.iTLB-loads
6.373e+10 -77.8% 1.414e+10 ? 5% perf-stat.i.instructions
42122 ? 8% -85.7% 6031 ? 6% perf-stat.i.instructions-per-iTLB-miss
0.27 +22.5% 0.32 perf-stat.i.ipc
0.18 ? 21% -85.9% 0.02 ? 52% perf-stat.i.major-faults
2.73 -81.4% 0.51 ? 6% perf-stat.i.metric.GHz
460.27 ? 3% +100.0% 920.38 ? 2% perf-stat.i.metric.K/sec
342.15 -75.6% 83.36 ? 4% perf-stat.i.metric.M/sec
88.02 +2.5 90.51 perf-stat.i.node-load-miss-rate%
4569607 +48.0% 6763657 ? 3% perf-stat.i.node-load-misses
624630 ? 4% +13.1% 706201 ? 4% perf-stat.i.node-loads
2159769 ? 3% +45.6% 3144636 ? 4% perf-stat.i.node-store-misses
0.52 ? 3% +832.4% 4.89 ? 6% perf-stat.overall.MPKI
0.11 ? 2% +0.5 0.56 ? 4% perf-stat.overall.branch-miss-rate%
45.31 ? 2% -5.6 39.72 ? 3% perf-stat.overall.cache-miss-rate%
3.77 -16.5% 3.15 perf-stat.overall.cpi
15919 ? 4% -89.8% 1629 ? 5% perf-stat.overall.cycles-between-cache-misses
0.00 ? 7% -0.0 0.00 ? 4% perf-stat.overall.dTLB-store-miss-rate%
93.43 -40.2 53.23 perf-stat.overall.iTLB-load-miss-rate%
40956 ? 8% -85.6% 5898 ? 6% perf-stat.overall.instructions-per-iTLB-miss
0.27 +19.8% 0.32 perf-stat.overall.ipc
87.96 +2.6 90.55 perf-stat.overall.node-load-miss-rate%
1.432e+08 -81.4% 26640660 ? 6% perf-stat.overall.path-length
1.274e+10 -77.3% 2.888e+09 ? 5% perf-stat.ps.branch-instructions
13804828 ? 2% +16.7% 16104208 perf-stat.ps.branch-misses
15077890 ? 4% +80.8% 27263085 ? 3% perf-stat.ps.cache-misses
33278822 ? 3% +106.3% 68645541 ? 2% perf-stat.ps.cache-references
65557 ? 4% +578.7% 444967 ? 2% perf-stat.ps.context-switches
2.395e+11 -81.4% 4.443e+10 ? 6% perf-stat.ps.cpu-cycles
147.63 -16.1% 123.87 perf-stat.ps.cpu-migrations
1.684e+10 -78.2% 3.671e+09 ? 5% perf-stat.ps.dTLB-loads
4.332e+08 +73.5% 7.514e+08 perf-stat.ps.dTLB-stores
1561170 ? 8% +53.1% 2390817 perf-stat.ps.iTLB-load-misses
109136 ? 2% +1825.1% 2100961 ? 2% perf-stat.ps.iTLB-loads
6.351e+10 -77.8% 1.409e+10 ? 5% perf-stat.ps.instructions
0.18 ? 21% -85.9% 0.02 ? 52% perf-stat.ps.major-faults
4554869 +48.0% 6741102 ? 3% perf-stat.ps.node-load-misses
623691 ? 4% +12.9% 703864 ? 4% perf-stat.ps.node-loads
2152844 ? 3% +45.6% 3134169 ? 4% perf-stat.ps.node-store-misses
1.925e+13 -77.8% 4.27e+12 ? 5% perf-stat.total.instructions
49.10 -35.2 13.89 ? 7% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
49.68 -32.9 16.80 ? 6% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
46.15 -32.2 13.95 ? 7% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
49.79 -30.4 19.40 ? 6% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
49.81 -30.2 19.61 ? 6% perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
46.70 -29.9 16.78 ? 7% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
46.81 -27.6 19.24 ? 6% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
46.83 -27.4 19.46 ? 6% perf-profile.calltrace.cycles-pp.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
50.42 -25.2 25.17 ? 5% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
50.43 -25.2 25.25 ? 5% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
50.44 -25.1 25.30 ? 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
50.44 -25.1 25.32 ? 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
50.46 -25.0 25.48 ? 5% perf-profile.calltrace.cycles-pp.__mmap
47.45 -22.7 24.78 ? 5% perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
47.45 -22.7 24.79 ? 5% perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
47.46 -22.6 24.84 ? 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
47.46 -22.6 24.85 ? 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
47.48 -22.5 25.01 ? 5% perf-profile.calltrace.cycles-pp.__munmap
0.00 +0.6 0.59 ? 6% perf-profile.calltrace.cycles-pp.vm_unmapped_area.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.get_unmapped_area.do_mmap
0.00 +0.6 0.62 ? 6% perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff
0.00 +0.6 0.62 ? 8% perf-profile.calltrace.cycles-pp.mas_store_prealloc.do_mas_align_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
0.00 +0.6 0.64 ? 10% perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.cmd_sched.run_builtin.main
0.00 +0.6 0.64 ? 10% perf-profile.calltrace.cycles-pp.cmd_record.cmd_sched.run_builtin.main.__libc_start_main
0.00 +0.6 0.64 ? 10% perf-profile.calltrace.cycles-pp.cmd_sched.run_builtin.main.__libc_start_main
0.00 +0.6 0.64 ? 4% perf-profile.calltrace.cycles-pp.unmap_region.do_mas_align_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
0.00 +0.7 0.66 ? 9% perf-profile.calltrace.cycles-pp.__libc_start_main
0.00 +0.7 0.66 ? 9% perf-profile.calltrace.cycles-pp.main.__libc_start_main
0.00 +0.7 0.66 ? 9% perf-profile.calltrace.cycles-pp.run_builtin.main.__libc_start_main
0.00 +0.7 0.70 ? 5% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.00 +0.7 0.70 ? 5% perf-profile.calltrace.cycles-pp.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
0.00 +0.7 0.71 ? 10% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +0.7 0.73 ? 4% perf-profile.calltrace.cycles-pp.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
0.00 +0.8 0.78 ? 5% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.rwsem_wake.up_read
0.00 +0.8 0.82 ? 7% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle
0.00 +0.8 0.84 ? 4% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.rwsem_wake.up_write
0.00 +0.8 0.84 ? 6% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle.do_idle
0.00 +0.9 0.87 ? 15% perf-profile.calltrace.cycles-pp.queue_event.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events
0.00 +0.9 0.88 ? 7% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
0.00 +0.9 0.89 ? 15% perf-profile.calltrace.cycles-pp.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events.record__finish_output
0.00 +0.9 0.95 ? 14% perf-profile.calltrace.cycles-pp.process_simple.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record
0.00 +1.0 0.96 ? 3% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +1.2 1.18 ? 12% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
0.00 +1.2 1.21 ? 2% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template
0.00 +1.3 1.32 ? 4% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime
0.00 +1.4 1.38 ? 4% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
0.00 +1.4 1.42 ? 2% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up
0.00 +1.4 1.42 ? 4% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.schedule_preempt_disabled
0.00 +1.4 1.43 ? 2% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q
0.00 +1.5 1.46 ? 4% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath
0.00 +1.5 1.50 ? 4% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
0.00 +1.5 1.51 ? 4% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
0.00 +1.5 1.52 ? 2% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.rwsem_wake
0.00 +1.5 1.54 ? 4% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
0.00 +1.6 1.57 ? 4% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair.__schedule
0.00 +1.6 1.58 ? 4% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.up_read.__vm_munmap
0.00 +1.6 1.61 ? 15% perf-profile.calltrace.cycles-pp.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record
0.00 +1.6 1.61 ? 15% perf-profile.calltrace.cycles-pp.record__finish_output.__cmd_record
0.00 +1.6 1.61 ? 15% perf-profile.calltrace.cycles-pp.perf_session__process_events.record__finish_output.__cmd_record
0.00 +1.7 1.66 ? 5% perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.up_read.__vm_munmap.__x64_sys_munmap
0.00 +1.7 1.68 ? 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.up_write.vm_mmap_pgoff
0.00 +1.7 1.72 ? 12% perf-profile.calltrace.cycles-pp.__cmd_record
0.00 +1.7 1.73 ? 4% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
0.00 +1.8 1.76 ? 3% perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.up_write.vm_mmap_pgoff.ksys_mmap_pgoff
0.00 +1.8 1.80 ? 5% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch
0.00 +1.8 1.82 ? 5% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.00 +1.8 1.84 ? 5% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +1.9 1.86 ? 5% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.00 +1.9 1.90 ? 5% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.00 +1.9 1.92 ? 4% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_preempt_disabled
0.00 +2.0 1.98 ? 4% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath
0.00 +2.1 2.07 ? 13% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.00 +2.1 2.11 ? 4% perf-profile.calltrace.cycles-pp.rwsem_wake.up_read.__vm_munmap.__x64_sys_munmap.do_syscall_64
0.00 +2.1 2.14 ? 4% perf-profile.calltrace.cycles-pp.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
0.00 +2.1 2.14 ? 4% perf-profile.calltrace.cycles-pp.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
0.00 +2.2 2.18 ? 5% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
0.00 +2.2 2.19 ? 6% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
0.00 +2.2 2.19 ? 4% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
0.00 +2.2 2.21 ? 2% perf-profile.calltrace.cycles-pp.rwsem_wake.up_write.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
0.00 +2.2 2.22 ? 3% perf-profile.calltrace.cycles-pp.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
0.00 +2.2 2.22 ? 3% perf-profile.calltrace.cycles-pp.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
0.00 +2.2 2.24 ? 3% perf-profile.calltrace.cycles-pp.up_read.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +2.3 2.27 ? 5% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
0.00 +2.4 2.41 ? 2% perf-profile.calltrace.cycles-pp.up_write.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +2.7 2.72 ? 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
0.00 +2.9 2.86 ? 5% perf-profile.calltrace.cycles-pp.do_mas_align_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +3.0 3.02 ? 5% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.00 +3.1 3.06 ? 5% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +4.0 4.02 ? 8% perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
0.00 +4.0 4.03 ? 7% perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_tp_event
0.00 +4.3 4.32 ? 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write_killable
1.48 ? 31% +34.8 36.25 ? 9% perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
1.48 ? 31% +34.8 36.26 ? 9% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
1.64 ? 27% +40.4 42.09 ? 7% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
1.64 ? 27% +40.5 42.13 ? 7% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
1.65 ? 27% +41.8 43.49 ? 6% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.73 ? 26% +44.3 46.07 ? 6% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.73 ? 26% +44.4 46.09 ? 6% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.73 ? 26% +44.4 46.09 ? 6% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
1.75 ? 26% +44.9 46.65 ? 6% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
95.29 -67.4 27.86 ? 7% perf-profile.children.cycles-pp.osq_lock
96.40 -62.7 33.69 ? 6% perf-profile.children.cycles-pp.rwsem_optimistic_spin
96.60 -58.0 38.65 ? 6% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
96.65 -57.6 39.07 ? 6% perf-profile.children.cycles-pp.down_write_killable
97.99 -46.9 51.04 ? 5% perf-profile.children.cycles-pp.do_syscall_64
98.00 -46.9 51.08 ? 5% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
50.42 -25.2 25.18 ? 5% perf-profile.children.cycles-pp.vm_mmap_pgoff
50.43 -25.2 25.26 ? 5% perf-profile.children.cycles-pp.ksys_mmap_pgoff
50.46 -25.0 25.50 ? 5% perf-profile.children.cycles-pp.__mmap
47.45 -22.7 24.79 ? 5% perf-profile.children.cycles-pp.__x64_sys_munmap
47.45 -22.7 24.79 ? 5% perf-profile.children.cycles-pp.__vm_munmap
47.48 -22.4 25.04 ? 5% perf-profile.children.cycles-pp.__munmap
0.45 ? 7% -0.2 0.21 ? 6% perf-profile.children.cycles-pp.task_tick_fair
0.05 ? 7% +0.0 0.09 ? 17% perf-profile.children.cycles-pp.mas_wr_spanning_store
0.03 ? 70% +0.0 0.08 ? 17% perf-profile.children.cycles-pp.mas_spanning_rebalance
0.02 ?141% +0.1 0.07 ? 11% perf-profile.children.cycles-pp.update_cfs_group
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.newidle_balance
0.00 +0.1 0.06 ? 13% perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.00 +0.1 0.06 ? 8% perf-profile.children.cycles-pp.seq_printf
0.00 +0.1 0.06 ? 13% perf-profile.children.cycles-pp.shmem_write_end
0.00 +0.1 0.06 ? 13% perf-profile.children.cycles-pp.prepend_path
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.__hrtimer_next_event_base
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.vsnprintf
0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.core_kernel_text
0.00 +0.1 0.06 ? 14% perf-profile.children.cycles-pp.error_entry
0.00 +0.1 0.06 ? 11% perf-profile.children.cycles-pp.cpuacct_charge
0.00 +0.1 0.06 ? 17% perf-profile.children.cycles-pp.irqentry_enter
0.00 +0.1 0.06 ? 7% perf-profile.children.cycles-pp.__intel_pmu_enable_all
0.00 +0.1 0.06 ? 19% perf-profile.children.cycles-pp.io_serial_in
0.00 +0.1 0.07 ? 20% perf-profile.children.cycles-pp.select_task_rq_fair
0.00 +0.1 0.07 ? 17% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.00 +0.1 0.07 ? 19% perf-profile.children.cycles-pp.update_sg_lb_stats
0.00 +0.1 0.07 ? 14% perf-profile.children.cycles-pp.__handle_mm_fault
0.00 +0.1 0.07 ? 16% perf-profile.children.cycles-pp.set_next_entity
0.00 +0.1 0.07 ? 14% perf-profile.children.cycles-pp.__cond_resched
0.00 +0.1 0.07 ? 12% perf-profile.children.cycles-pp.zap_pmd_range
0.00 +0.1 0.08 ? 12% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.00 +0.1 0.08 ? 15% perf-profile.children.cycles-pp.perf_output_begin_forward
0.00 +0.1 0.08 ? 13% perf-profile.children.cycles-pp.proc_reg_read
0.00 +0.1 0.08 ? 19% perf-profile.children.cycles-pp.hrtimer_next_event_without
0.00 +0.1 0.08 ? 10% perf-profile.children.cycles-pp.handle_mm_fault
0.00 +0.1 0.08 ? 8% perf-profile.children.cycles-pp.__update_blocked_fair
0.00 +0.1 0.08 ? 13% perf-profile.children.cycles-pp.down_write
0.00 +0.1 0.08 ? 22% perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.00 +0.1 0.08 ? 22% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
0.00 +0.1 0.08 ? 27% perf-profile.children.cycles-pp.fput
0.00 +0.1 0.08 ? 8% perf-profile.children.cycles-pp.seq_read
0.00 +0.1 0.09 ? 8% perf-profile.children.cycles-pp.perf_iterate_sb
0.00 +0.1 0.09 ? 10% perf-profile.children.cycles-pp.llist_reverse_order
0.00 +0.1 0.09 ? 8% perf-profile.children.cycles-pp.serial8250_console_write
0.00 +0.1 0.09 ? 8% perf-profile.children.cycles-pp.wait_for_lsr
0.01 ?223% +0.1 0.10 ? 10% perf-profile.children.cycles-pp.ktime_get_update_offsets_now
0.00 +0.1 0.09 ? 12% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.00 +0.1 0.09 ? 12% perf-profile.children.cycles-pp.__libc_read
0.00 +0.1 0.09 ? 20% perf-profile.children.cycles-pp.perf_poll
0.00 +0.1 0.09 ? 6% perf-profile.children.cycles-pp.free_pgd_range
0.00 +0.1 0.09 ? 12% perf-profile.children.cycles-pp.__update_load_avg_se
0.00 +0.1 0.09 ? 23% perf-profile.children.cycles-pp.update_sd_lb_stats
0.00 +0.1 0.09 ? 14% perf-profile.children.cycles-pp._raw_spin_trylock
0.00 +0.1 0.09 ? 9% perf-profile.children.cycles-pp._printk
0.00 +0.1 0.09 ? 9% perf-profile.children.cycles-pp.vprintk_emit
0.00 +0.1 0.09 ? 9% perf-profile.children.cycles-pp.console_unlock
0.00 +0.1 0.09 ? 9% perf-profile.children.cycles-pp.console_flush_all
0.00 +0.1 0.09 ? 9% perf-profile.children.cycles-pp.console_emit_next_record
0.00 +0.1 0.09 ? 11% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.00 +0.1 0.09 ? 11% perf-profile.children.cycles-pp.cmp_ex_search
0.00 +0.1 0.10 ? 8% perf-profile.children.cycles-pp.sysvec_irq_work
0.00 +0.1 0.10 ? 8% perf-profile.children.cycles-pp.__sysvec_irq_work
0.00 +0.1 0.10 ? 8% perf-profile.children.cycles-pp.irq_work_run
0.00 +0.1 0.10 ? 8% perf-profile.children.cycles-pp.irq_work_single
0.00 +0.1 0.10 ? 16% perf-profile.children.cycles-pp.select_task_rq
0.00 +0.1 0.10 ? 7% perf-profile.children.cycles-pp.tlb_gather_mmu
0.00 +0.1 0.10 ? 7% perf-profile.children.cycles-pp.d_path
0.00 +0.1 0.10 ? 7% perf-profile.children.cycles-pp.asm_sysvec_irq_work
0.00 +0.1 0.10 ? 7% perf-profile.children.cycles-pp.irq_work_run_list
0.00 +0.1 0.10 ? 7% perf-profile.children.cycles-pp.perf_rotate_context
0.00 +0.1 0.10 ? 22% perf-profile.children.cycles-pp.find_busiest_group
0.00 +0.1 0.10 ? 10% perf-profile.children.cycles-pp.seq_read_iter
0.00 +0.1 0.10 ? 15% perf-profile.children.cycles-pp.__might_sleep
0.00 +0.1 0.10 ? 20% perf-profile.children.cycles-pp.__switch_to_asm
0.00 +0.1 0.10 ? 24% perf-profile.children.cycles-pp.evsel__parse_sample
0.00 +0.1 0.11 ? 16% perf-profile.children.cycles-pp.do_poll
0.00 +0.1 0.11 ? 17% perf-profile.children.cycles-pp.calc_global_load_tick
0.00 +0.1 0.11 ? 29% perf-profile.children.cycles-pp.fault_in_iov_iter_readable
0.00 +0.1 0.11 ? 29% perf-profile.children.cycles-pp.fault_in_readable
0.00 +0.1 0.11 ? 13% perf-profile.children.cycles-pp.irqtime_account_irq
0.00 +0.1 0.11 ? 17% perf-profile.children.cycles-pp.shmem_get_folio_gfp
0.00 +0.1 0.11 ? 17% perf-profile.children.cycles-pp.tick_irq_enter
0.00 +0.1 0.11 ? 11% perf-profile.children.cycles-pp.io__get_hex
0.00 +0.1 0.12 ? 10% perf-profile.children.cycles-pp.mas_leaf_max_gap
0.00 +0.1 0.12 ? 19% perf-profile.children.cycles-pp.shmem_write_begin
0.00 +0.1 0.12 ? 13% perf-profile.children.cycles-pp.irq_enter_rcu
0.00 +0.1 0.12 ? 23% perf-profile.children.cycles-pp.load_balance
0.49 ? 6% +0.1 0.61 ? 6% perf-profile.children.cycles-pp.scheduler_tick
0.00 +0.1 0.12 ? 18% perf-profile.children.cycles-pp.rwsem_mark_wake
0.00 +0.1 0.13 ? 11% perf-profile.children.cycles-pp.__poll
0.00 +0.1 0.13 ? 11% perf-profile.children.cycles-pp.__x64_sys_poll
0.00 +0.1 0.13 ? 11% perf-profile.children.cycles-pp.do_sys_poll
0.00 +0.1 0.13 ? 23% perf-profile.children.cycles-pp.do_user_addr_fault
0.00 +0.1 0.13 ? 8% perf-profile.children.cycles-pp.update_blocked_averages
0.00 +0.1 0.13 ? 8% perf-profile.children.cycles-pp.run_rebalance_domains
0.00 +0.1 0.13 ? 10% perf-profile.children.cycles-pp.__might_resched
0.00 +0.1 0.13 ? 14% perf-profile.children.cycles-pp.vfs_read
0.00 +0.1 0.13 ? 13% perf-profile.children.cycles-pp.mas_store_gfp
0.00 +0.1 0.14 ? 38% perf-profile.children.cycles-pp.tick_sched_do_timer
0.00 +0.1 0.14 ? 12% perf-profile.children.cycles-pp.ksys_read
0.00 +0.1 0.14 ? 8% perf-profile.children.cycles-pp.prepare_task_switch
0.00 +0.1 0.14 ? 9% perf-profile.children.cycles-pp.mas_wr_walk
0.00 +0.1 0.14 ? 9% perf-profile.children.cycles-pp.vm_area_alloc
0.00 +0.1 0.14 ? 12% perf-profile.children.cycles-pp.bsearch
0.00 +0.1 0.14 ? 13% perf-profile.children.cycles-pp.__perf_event_header__init_id
0.00 +0.1 0.14 ? 17% perf-profile.children.cycles-pp.read_tsc
0.00 +0.1 0.14 ? 24% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.00 +0.1 0.15 ? 12% perf-profile.children.cycles-pp.search_extable
0.00 +0.2 0.15 ? 14% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.00 +0.2 0.15 ? 8% perf-profile.children.cycles-pp.__switch_to
0.00 +0.2 0.15 ? 12% perf-profile.children.cycles-pp.search_exception_tables
0.00 +0.2 0.15 ? 17% perf-profile.children.cycles-pp.mas_next_entry
0.00 +0.2 0.15 ? 18% perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
0.00 +0.2 0.16 ? 12% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.00 +0.2 0.16 ? 12% perf-profile.children.cycles-pp.copyin
0.00 +0.2 0.16 ? 11% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
0.00 +0.2 0.16 ? 14% perf-profile.children.cycles-pp.unmap_page_range
0.00 +0.2 0.16 ? 12% perf-profile.children.cycles-pp.copy_page_from_iter_atomic
0.00 +0.2 0.16 ? 21% perf-profile.children.cycles-pp.evlist__parse_sample
0.00 +0.2 0.17 ? 7% perf-profile.children.cycles-pp.pick_next_task_fair
0.00 +0.2 0.17 ? 10% perf-profile.children.cycles-pp.mas_wr_node_store
0.00 +0.2 0.18 ? 7% perf-profile.children.cycles-pp.mas_update_gap
0.00 +0.2 0.18 ? 12% perf-profile.children.cycles-pp.fixup_exception
0.00 +0.2 0.18 ? 9% perf-profile.children.cycles-pp.arch_scale_freq_tick
0.00 +0.2 0.18 ? 6% perf-profile.children.cycles-pp.kernel_text_address
0.00 +0.2 0.18 ? 4% perf-profile.children.cycles-pp.free_pgtables
0.00 +0.2 0.19 ? 9% perf-profile.children.cycles-pp.mas_wr_modify
0.00 +0.2 0.19 ? 12% perf-profile.children.cycles-pp.kallsyms__parse
0.00 +0.2 0.19 ? 7% perf-profile.children.cycles-pp.llist_add_batch
0.00 +0.2 0.20 ? 10% perf-profile.children.cycles-pp.update_rq_clock
0.00 +0.2 0.20 ? 7% perf-profile.children.cycles-pp.__smp_call_single_queue
0.00 +0.2 0.20 ? 12% perf-profile.children.cycles-pp.kernelmode_fixup_or_oops
0.00 +0.2 0.20 ? 13% perf-profile.children.cycles-pp.lapic_next_deadline
0.00 +0.2 0.20 ? 6% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.00 +0.2 0.20 ? 15% perf-profile.children.cycles-pp.rebalance_domains
0.01 ?223% +0.2 0.22 ? 7% perf-profile.children.cycles-pp.unmap_vmas
0.58 ? 5% +0.2 0.79 ? 7% perf-profile.children.cycles-pp.update_process_times
0.05 +0.2 0.26 ? 8% perf-profile.children.cycles-pp.kmem_cache_free_bulk
0.00 +0.2 0.22 ? 17% perf-profile.children.cycles-pp.tick_nohz_irq_exit
0.00 +0.2 0.22 ? 10% perf-profile.children.cycles-pp.kmem_cache_free
0.05 ? 7% +0.2 0.27 ? 15% perf-profile.children.cycles-pp.mas_walk
0.00 +0.2 0.22 ? 12% perf-profile.children.cycles-pp.__dso__load_kallsyms
0.58 ? 6% +0.2 0.80 ? 7% perf-profile.children.cycles-pp.tick_sched_handle
0.00 +0.2 0.22 ? 11% perf-profile.children.cycles-pp.dso__load
0.00 +0.2 0.23 ? 12% perf-profile.children.cycles-pp.map__load
0.00 +0.2 0.23 ? 9% perf-profile.children.cycles-pp._raw_spin_lock
0.00 +0.2 0.24 ? 6% perf-profile.children.cycles-pp.__kernel_text_address
0.01 ?223% +0.2 0.25 ? 10% perf-profile.children.cycles-pp.update_load_avg
0.06 ? 8% +0.2 0.30 ? 12% perf-profile.children.cycles-pp.mas_find
0.00 +0.2 0.25 ? 14% perf-profile.children.cycles-pp.native_sched_clock
0.00 +0.3 0.25 ? 7% perf-profile.children.cycles-pp.enqueue_entity
0.07 ? 7% +0.3 0.32 ? 13% perf-profile.children.cycles-pp.do_mas_munmap
0.00 +0.3 0.26 ? 9% perf-profile.children.cycles-pp.stack_access_ok
0.00 +0.3 0.26 ? 15% perf-profile.children.cycles-pp.thread__find_map
0.00 +0.3 0.26 ? 11% perf-profile.children.cycles-pp.sched_clock_cpu
0.02 ? 99% +0.3 0.29 ? 6% perf-profile.children.cycles-pp.unwind_get_return_address
0.04 ? 44% +0.3 0.31 ? 6% perf-profile.children.cycles-pp.kmem_cache_alloc
0.05 +0.3 0.32 ? 3% perf-profile.children.cycles-pp.perf_event_mmap_event
0.05 +0.3 0.33 ? 16% perf-profile.children.cycles-pp.mas_wr_store_entry
0.05 +0.3 0.34 ? 3% perf-profile.children.cycles-pp.perf_event_mmap
0.00 +0.3 0.29 ? 10% perf-profile.children.cycles-pp.orc_find
0.05 +0.3 0.34 ? 7% perf-profile.children.cycles-pp.memset_erms
0.00 +0.3 0.31 ? 17% perf-profile.children.cycles-pp.build_id__mark_dso_hit
0.00 +0.3 0.32 ? 5% perf-profile.children.cycles-pp.enqueue_task_fair
0.05 +0.3 0.38 ? 9% perf-profile.children.cycles-pp.osq_unlock
0.00 +0.3 0.33 ? 6% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.00 +0.3 0.33 ? 5% perf-profile.children.cycles-pp.ttwu_do_activate
0.07 ? 7% +0.3 0.40 ? 10% perf-profile.children.cycles-pp.mas_destroy
0.00 +0.3 0.34 ? 27% perf-profile.children.cycles-pp.timekeeping_max_deferment
0.00 +0.3 0.34 ? 6% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.06 ? 8% +0.4 0.41 ? 9% perf-profile.children.cycles-pp.exc_page_fault
0.08 +0.4 0.44 ? 3% perf-profile.children.cycles-pp.mas_rev_awalk
0.00 +0.4 0.36 ? 5% perf-profile.children.cycles-pp.sysvec_call_function_single
0.00 +0.4 0.37 ? 6% perf-profile.children.cycles-pp.wake_q_add
0.09 ? 30% +0.4 0.47 ? 88% perf-profile.children.cycles-pp.intel_idle_irq
0.60 ? 5% +0.4 1.00 ? 5% perf-profile.children.cycles-pp.tick_sched_timer
0.00 +0.4 0.41 ? 4% perf-profile.children.cycles-pp.memcpy_erms
0.00 +0.4 0.42 ? 8% perf-profile.children.cycles-pp.downgrade_write
0.00 +0.4 0.42 ? 11% perf-profile.children.cycles-pp.__do_softirq
0.09 +0.4 0.52 ? 6% perf-profile.children.cycles-pp.mas_empty_area_rev
0.00 +0.5 0.46 ? 2% perf-profile.children.cycles-pp.perf_output_copy
0.00 +0.5 0.46 ? 9% perf-profile.children.cycles-pp.generic_perform_write
0.00 +0.5 0.46 ? 8% perf-profile.children.cycles-pp.__generic_file_write_iter
0.00 +0.5 0.46 ? 8% perf-profile.children.cycles-pp.generic_file_write_iter
0.00 +0.5 0.48 ? 7% perf-profile.children.cycles-pp.vfs_write
0.00 +0.5 0.48 ? 8% perf-profile.children.cycles-pp.flush_smp_call_function_queue
0.00 +0.5 0.49 ? 10% perf-profile.children.cycles-pp.record__pushfn
0.07 ? 20% +0.5 0.56 ? 3% perf-profile.children.cycles-pp.native_irq_return_iret
0.00 +0.5 0.49 ? 7% perf-profile.children.cycles-pp.ksys_write
0.00 +0.5 0.50 ? 11% perf-profile.children.cycles-pp.__libc_write
0.10 ? 4% +0.5 0.59 ? 6% perf-profile.children.cycles-pp.vm_unmapped_area
0.00 +0.5 0.50 ? 11% perf-profile.children.cycles-pp.writen
0.00 +0.5 0.50 ? 5% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.00 +0.5 0.51 ? 10% perf-profile.children.cycles-pp.__irq_exit_rcu
0.09 ? 4% +0.5 0.61 ? 9% perf-profile.children.cycles-pp.kmem_cache_alloc_bulk
0.10 +0.5 0.62 ? 6% perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
0.05 +0.5 0.57 ? 5% perf-profile.children.cycles-pp.__output_copy
0.12 ? 4% +0.5 0.64 ? 3% perf-profile.children.cycles-pp.unmap_region
0.04 ? 45% +0.5 0.58 ? 17% perf-profile.children.cycles-pp.perf_session__deliver_event
0.04 ? 45% +0.5 0.59 ? 17% perf-profile.children.cycles-pp.__ordered_events__flush
0.04 ? 45% +0.5 0.59 ? 17% perf-profile.children.cycles-pp.perf_session__process_user_event
0.02 ? 99% +0.5 0.57 ? 9% perf-profile.children.cycles-pp.perf_mmap__push
0.00 +0.6 0.55 ? 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.00 +0.6 0.56 ? 22% perf-profile.children.cycles-pp.start_kernel
0.00 +0.6 0.56 ? 22% perf-profile.children.cycles-pp.arch_call_rest_init
0.00 +0.6 0.56 ? 22% perf-profile.children.cycles-pp.rest_init
0.00 +0.6 0.56 ? 21% perf-profile.children.cycles-pp.tick_nohz_next_event
0.00 +0.6 0.56 ? 5% perf-profile.children.cycles-pp.sched_ttwu_pending
0.07 ? 6% +0.6 0.64 ? 10% perf-profile.children.cycles-pp.cmd_record
0.07 ? 7% +0.6 0.64 ? 10% perf-profile.children.cycles-pp.cmd_sched
0.12 ? 5% +0.6 0.70 ? 5% perf-profile.children.cycles-pp.shmem_get_unmapped_area
0.02 ? 99% +0.6 0.61 ? 9% perf-profile.children.cycles-pp.record__mmap_read_evlist
0.08 ? 6% +0.6 0.66 ? 9% perf-profile.children.cycles-pp.__libc_start_main
0.08 ? 6% +0.6 0.66 ? 9% perf-profile.children.cycles-pp.main
0.08 ? 6% +0.6 0.66 ? 9% perf-profile.children.cycles-pp.run_builtin
0.13 ? 4% +0.6 0.73 ? 4% perf-profile.children.cycles-pp.get_unmapped_area
0.06 ? 9% +0.6 0.68 ? 4% perf-profile.children.cycles-pp.perf_output_sample
0.06 ? 11% +0.6 0.70 ? 5% perf-profile.children.cycles-pp.__orc_find
0.64 ? 5% +0.7 1.30 ? 3% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.00 +0.7 0.67 ? 19% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
0.12 ? 3% +0.7 0.87 ? 10% perf-profile.children.cycles-pp.mas_preallocate
0.12 ? 3% +0.7 0.87 ? 9% perf-profile.children.cycles-pp.mas_alloc_nodes
0.12 ? 13% +0.8 0.88 ? 15% perf-profile.children.cycles-pp.queue_event
0.12 ? 13% +0.8 0.89 ? 15% perf-profile.children.cycles-pp.ordered_events__queue
0.00 +0.8 0.79 ? 10% perf-profile.children.cycles-pp.clockevents_program_event
0.12 ? 13% +0.8 0.95 ? 14% perf-profile.children.cycles-pp.process_simple
0.00 +0.8 0.84 ? 6% perf-profile.children.cycles-pp.finish_task_switch
0.20 ? 2% +0.9 1.10 ? 9% perf-profile.children.cycles-pp.mas_store_prealloc
0.00 +1.0 0.96 ? 8% perf-profile.children.cycles-pp.ktime_get
0.10 ? 9% +1.0 1.10 ? 3% perf-profile.children.cycles-pp.asm_exc_page_fault
0.12 ? 10% +1.1 1.21 ? 6% perf-profile.children.cycles-pp.__unwind_start
0.00 +1.2 1.20 ? 12% perf-profile.children.cycles-pp.menu_select
0.14 ? 14% +1.2 1.36 perf-profile.children.cycles-pp.__get_user_nocheck_8
0.04 ? 70% +1.2 1.27 ? 11% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.15 ? 12% +1.3 1.41 perf-profile.children.cycles-pp.perf_callchain_user
0.42 ? 6% +1.3 1.74 ? 4% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.18 ? 14% +1.4 1.61 ? 15% perf-profile.children.cycles-pp.reader__read_event
0.18 ? 14% +1.4 1.61 ? 15% perf-profile.children.cycles-pp.record__finish_output
0.18 ? 14% +1.4 1.61 ? 15% perf-profile.children.cycles-pp.perf_session__process_events
0.43 ? 7% +1.5 1.92 ? 4% perf-profile.children.cycles-pp.update_curr
0.08 ? 8% +1.6 1.63 ? 3% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
0.73 ? 3% +1.6 2.32 ? 5% perf-profile.children.cycles-pp.hrtimer_interrupt
0.74 ? 3% +1.6 2.35 ? 4% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.06 ? 11% +1.8 1.87 ? 5% perf-profile.children.cycles-pp.schedule_idle
0.24 ? 9% +1.8 2.09 ? 6% perf-profile.children.cycles-pp.unwind_next_frame
0.34 ? 2% +1.9 2.20 ? 6% perf-profile.children.cycles-pp.mmap_region
0.07 ? 10% +1.9 1.95 ? 5% perf-profile.children.cycles-pp.dequeue_entity
0.08 ? 12% +1.9 2.00 ? 4% perf-profile.children.cycles-pp.dequeue_task_fair
0.06 ? 14% +2.0 2.09 ? 13% perf-profile.children.cycles-pp.poll_idle
0.10 ? 7% +2.1 2.25 ? 3% perf-profile.children.cycles-pp.up_read
0.21 ? 32% +2.2 2.36 ? 9% perf-profile.children.cycles-pp.__cmd_record
0.10 ? 4% +2.3 2.36 ? 4% perf-profile.children.cycles-pp.perf_trace_sched_switch
0.32 ? 8% +2.4 2.68 ? 5% perf-profile.children.cycles-pp.perf_callchain_kernel
0.46 ? 3% +2.4 2.87 ? 4% perf-profile.children.cycles-pp.do_mas_align_munmap
0.12 ? 6% +2.4 2.56 ? 2% perf-profile.children.cycles-pp.up_write
0.77 ? 3% +2.5 3.26 ? 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.49 +2.6 3.08 ? 5% perf-profile.children.cycles-pp.do_mmap
0.88 ? 3% +3.0 3.90 ? 8% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.13 ? 5% +3.2 3.30 ? 2% perf-profile.children.cycles-pp.try_to_wake_up
0.13 ? 7% +3.3 3.44 ? 2% perf-profile.children.cycles-pp.wake_up_q
0.89 +3.7 4.57 ? 4% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.48 ? 6% +3.8 4.26 ? 4% perf-profile.children.cycles-pp.get_perf_callchain
0.48 ? 6% +3.8 4.28 ? 3% perf-profile.children.cycles-pp.perf_callchain
0.50 ? 6% +4.0 4.50 ? 3% perf-profile.children.cycles-pp.perf_prepare_sample
0.16 ? 7% +4.2 4.32 ? 2% perf-profile.children.cycles-pp.rwsem_wake
0.17 ? 7% +4.2 4.36 ? 3% perf-profile.children.cycles-pp.schedule_preempt_disabled
0.18 ? 7% +4.2 4.40 ? 4% perf-profile.children.cycles-pp.schedule
0.57 ? 5% +4.7 5.31 ? 3% perf-profile.children.cycles-pp.perf_event_output_forward
0.57 ? 5% +4.8 5.34 ? 3% perf-profile.children.cycles-pp.__perf_event_overflow
0.58 ? 6% +4.9 5.53 ? 3% perf-profile.children.cycles-pp.perf_tp_event
0.25 ? 7% +6.0 6.22 ? 4% perf-profile.children.cycles-pp.__schedule
1.50 ? 31% +35.0 36.50 ? 9% perf-profile.children.cycles-pp.intel_idle
1.59 ? 28% +35.4 36.95 ? 8% perf-profile.children.cycles-pp.mwait_idle_with_hints
1.66 ? 27% +41.0 42.64 ? 7% perf-profile.children.cycles-pp.cpuidle_enter_state
1.66 ? 27% +41.0 42.65 ? 7% perf-profile.children.cycles-pp.cpuidle_enter
1.67 ? 27% +42.4 44.03 ? 6% perf-profile.children.cycles-pp.cpuidle_idle_call
1.73 ? 26% +44.4 46.09 ? 6% perf-profile.children.cycles-pp.start_secondary
1.75 ? 26% +44.9 46.64 ? 6% perf-profile.children.cycles-pp.do_idle
1.75 ? 26% +44.9 46.65 ? 6% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
1.75 ? 26% +44.9 46.65 ? 6% perf-profile.children.cycles-pp.cpu_startup_entry
94.52 -66.9 27.63 ? 7% perf-profile.self.cycles-pp.osq_lock
0.02 ?141% +0.1 0.07 ? 11% perf-profile.self.cycles-pp.update_cfs_group
0.00 +0.1 0.06 ? 19% perf-profile.self.cycles-pp.error_entry
0.00 +0.1 0.06 ? 11% perf-profile.self.cycles-pp.dequeue_entity
0.00 +0.1 0.06 ? 13% perf-profile.self.cycles-pp.cpuacct_charge
0.00 +0.1 0.06 ? 14% perf-profile.self.cycles-pp.perf_iterate_sb
0.00 +0.1 0.06 ? 11% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.__intel_pmu_enable_all
0.00 +0.1 0.06 ? 7% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.00 +0.1 0.06 ? 19% perf-profile.self.cycles-pp.io_serial_in
0.00 +0.1 0.06 ? 14% perf-profile.self.cycles-pp.mas_alloc_nodes
0.00 +0.1 0.07 ? 18% perf-profile.self.cycles-pp.enqueue_task_fair
0.00 +0.1 0.07 ? 18% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.01 ?223% +0.1 0.08 ? 9% perf-profile.self.cycles-pp.ktime_get_update_offsets_now
0.00 +0.1 0.07 ? 17% perf-profile.self.cycles-pp.mas_next_entry
0.00 +0.1 0.07 ? 12% perf-profile.self.cycles-pp.kernel_text_address
0.00 +0.1 0.07 ? 15% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.00 +0.1 0.08 ? 12% perf-profile.self.cycles-pp.shmem_get_unmapped_area
0.00 +0.1 0.08 ? 19% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
0.00 +0.1 0.08 ? 15% perf-profile.self.cycles-pp.perf_output_begin_forward
0.00 +0.1 0.08 ? 10% perf-profile.self.cycles-pp.update_curr
0.00 +0.1 0.08 ? 11% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.00 +0.1 0.08 ? 8% perf-profile.self.cycles-pp.llist_reverse_order
0.00 +0.1 0.09 ? 14% perf-profile.self.cycles-pp.__update_load_avg_se
0.00 +0.1 0.09 ? 12% perf-profile.self.cycles-pp.perf_output_sample
0.00 +0.1 0.09 ? 23% perf-profile.self.cycles-pp.unmap_page_range
0.00 +0.1 0.09 ? 14% perf-profile.self.cycles-pp._raw_spin_trylock
0.00 +0.1 0.09 ? 9% perf-profile.self.cycles-pp.cmp_ex_search
0.00 +0.1 0.09 ? 18% perf-profile.self.cycles-pp.__might_sleep
0.00 +0.1 0.10 ? 8% perf-profile.self.cycles-pp.tlb_gather_mmu
0.00 +0.1 0.10 ? 10% perf-profile.self.cycles-pp.update_load_avg
0.00 +0.1 0.10 ? 18% perf-profile.self.cycles-pp.update_rq_clock
0.00 +0.1 0.10 ? 10% perf-profile.self.cycles-pp.kmem_cache_alloc
0.00 +0.1 0.10 ? 11% perf-profile.self.cycles-pp.prepare_task_switch
0.00 +0.1 0.10 ? 19% perf-profile.self.cycles-pp.__switch_to_asm
0.00 +0.1 0.10 ? 12% perf-profile.self.cycles-pp.__unwind_start
0.00 +0.1 0.10 ? 16% perf-profile.self.cycles-pp.enqueue_entity
0.00 +0.1 0.10 ? 24% perf-profile.self.cycles-pp.evsel__parse_sample
0.00 +0.1 0.11 ? 13% perf-profile.self.cycles-pp.unmap_region
0.00 +0.1 0.11 ? 17% perf-profile.self.cycles-pp.calc_global_load_tick
0.00 +0.1 0.11 ? 7% perf-profile.self.cycles-pp.mas_wr_node_store
0.00 +0.1 0.11 ? 44% perf-profile.self.cycles-pp.tick_sched_do_timer
0.00 +0.1 0.11 ? 24% perf-profile.self.cycles-pp.tick_nohz_next_event
0.00 +0.1 0.11 ? 11% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.00 +0.1 0.12 ? 14% perf-profile.self.cycles-pp.cpuidle_idle_call
0.00 +0.1 0.12 ? 10% perf-profile.self.cycles-pp.mas_leaf_max_gap
0.00 +0.1 0.12 ? 11% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.00 +0.1 0.12 ? 10% perf-profile.self.cycles-pp.kmem_cache_alloc_bulk
0.00 +0.1 0.12 ? 12% perf-profile.self.cycles-pp.mas_wr_walk
0.00 +0.1 0.12 ? 17% perf-profile.self.cycles-pp.rwsem_mark_wake
0.00 +0.1 0.12 ? 34% perf-profile.self.cycles-pp.mas_wr_store_entry
0.00 +0.1 0.13 ? 8% perf-profile.self.cycles-pp.perf_tp_event
0.00 +0.1 0.13 ? 8% perf-profile.self.cycles-pp.__might_resched
0.00 +0.1 0.13 ? 7% perf-profile.self.cycles-pp.up_read
0.00 +0.1 0.14 ? 10% perf-profile.self.cycles-pp.get_perf_callchain
0.00 +0.1 0.14 ? 14% perf-profile.self.cycles-pp.do_mmap
0.00 +0.1 0.14 ? 17% perf-profile.self.cycles-pp.read_tsc
0.00 +0.1 0.14 ? 24% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.00 +0.1 0.15 ? 8% perf-profile.self.cycles-pp.__switch_to
0.00 +0.2 0.15 ? 7% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.00 +0.2 0.15 ? 20% perf-profile.self.cycles-pp.__kmem_cache_alloc_bulk
0.00 +0.2 0.16 ? 10% perf-profile.self.cycles-pp.wake_up_q
0.00 +0.2 0.18 ? 19% perf-profile.self.cycles-pp.rwsem_wake
0.00 +0.2 0.18 ? 11% perf-profile.self.cycles-pp.kmem_cache_free
0.00 +0.2 0.18 ? 9% perf-profile.self.cycles-pp.arch_scale_freq_tick
0.00 +0.2 0.18 ? 9% perf-profile.self.cycles-pp.__output_copy
0.00 +0.2 0.19 ? 11% perf-profile.self.cycles-pp.downgrade_write
0.00 +0.2 0.19 ? 5% perf-profile.self.cycles-pp.perf_callchain_kernel
0.03 ? 70% +0.2 0.22 ? 14% perf-profile.self.cycles-pp.do_mas_align_munmap
0.00 +0.2 0.19 ? 7% perf-profile.self.cycles-pp.llist_add_batch
0.00 +0.2 0.20 ? 13% perf-profile.self.cycles-pp.lapic_next_deadline
0.05 ? 7% +0.2 0.26 ? 11% perf-profile.self.cycles-pp.mas_walk
0.00 +0.2 0.22 ? 10% perf-profile.self.cycles-pp._raw_spin_lock
0.02 ?141% +0.2 0.25 ? 9% perf-profile.self.cycles-pp.kmem_cache_free_bulk
0.05 ? 7% +0.2 0.29 ? 6% perf-profile.self.cycles-pp.mmap_region
0.00 +0.2 0.24 ? 15% perf-profile.self.cycles-pp.native_sched_clock
0.00 +0.2 0.24 ? 8% perf-profile.self.cycles-pp.stack_access_ok
0.00 +0.3 0.27 ? 10% perf-profile.self.cycles-pp.orc_find
0.05 +0.3 0.33 ? 7% perf-profile.self.cycles-pp.memset_erms
0.00 +0.3 0.29 ? 6% perf-profile.self.cycles-pp.__schedule
0.00 +0.3 0.32 ? 7% perf-profile.self.cycles-pp.rwsem_down_write_slowpath
0.02 ? 99% +0.3 0.35 ? 7% perf-profile.self.cycles-pp.up_write
0.05 +0.3 0.38 ? 9% perf-profile.self.cycles-pp.osq_unlock
0.00 +0.3 0.34 ? 27% perf-profile.self.cycles-pp.timekeeping_max_deferment
0.08 +0.4 0.44 ? 4% perf-profile.self.cycles-pp.mas_rev_awalk
0.02 ? 99% +0.4 0.39 ? 17% perf-profile.self.cycles-pp.down_write_killable
0.00 +0.4 0.37 ? 6% perf-profile.self.cycles-pp.wake_q_add
0.00 +0.4 0.39 ? 12% perf-profile.self.cycles-pp.cpuidle_enter_state
0.00 +0.4 0.40 ? 5% perf-profile.self.cycles-pp.memcpy_erms
0.00 +0.4 0.40 ? 8% perf-profile.self.cycles-pp.finish_task_switch
0.00 +0.5 0.48 ? 7% perf-profile.self.cycles-pp.menu_select
0.07 ? 20% +0.5 0.56 ? 3% perf-profile.self.cycles-pp.native_irq_return_iret
0.00 +0.5 0.50 ? 3% perf-profile.self.cycles-pp.try_to_wake_up
0.00 +0.5 0.55 ? 3% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.06 ? 11% +0.6 0.69 ? 5% perf-profile.self.cycles-pp.__orc_find
0.01 ?223% +0.6 0.66 ? 3% perf-profile.self.cycles-pp.__get_user_nocheck_8
0.10 ? 9% +0.7 0.80 ? 9% perf-profile.self.cycles-pp.unwind_next_frame
0.04 ? 71% +0.7 0.76 ? 17% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.12 ? 13% +0.7 0.86 ? 15% perf-profile.self.cycles-pp.queue_event
0.00 +0.8 0.84 ? 9% perf-profile.self.cycles-pp.ktime_get
0.18 ? 4% +0.9 1.05 ? 5% perf-profile.self.cycles-pp.rwsem_optimistic_spin
0.06 ? 11% +2.0 2.06 ? 13% perf-profile.self.cycles-pp.poll_idle
0.86 +3.6 4.43 ? 4% perf-profile.self.cycles-pp.rwsem_spin_on_owner
1.58 ? 28% +35.3 36.90 ? 8% perf-profile.self.cycles-pp.mwait_idle_with_hints

To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
sudo bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
sudo bin/lkp run generated-yaml-file

# if come across any failure that blocks the test,
# please remove ~/.lkp and /lkp dir to run from a clean state.

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests

Attachments:

(No filename) (157.00 kB)
config-6.2.0-rc3-00016-g12ea6b720a0e (163.50 kB)
job-script (8.16 kB)
job.yaml (5.70 kB)
reproduce (734.00 B)
Download all attachments