LinuxLists.cc - [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression

2023-05-31 04:29:07

Subject: [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression

With the numa scan enhancements [1], only the threads which had previously
accessed vma are allowed to scan.

While this had improved significant system time overhead, there were corner
cases, which genuinely need some relaxation. For e.g.,

1) Concern raised by PeterZ, where if there are N partition sets of vmas
belonging to tasks, then unfairness in allowing these threads to scan could
potentially amplify the side effect of some of the vmas being left
unscanned.

2) Below reports of LKP numa01 benchmark regression.

Currently this is handled by allowing first two scanning unconditional
as indicated by mm->numa_scan_seq. This is imprecise since for some
benchmark vma scanning might itself start at numa_scan_seq > 2.

Solution:
Allow unconditional scanning of vmas of tasks depending on vma size. This
is achieved by maintaining a per vma scan counter, where

f(allowed_to_scan) = f(scan_counter < vma_size / scan_size)

Result:
numa01_THREAD_ALLOC result on 6.4.0-rc2 (that has numascan enhancement)
base-numascan base base+fix
real 1m1.507s 1m23.259s 1m2.632s
user 213m51.336s 251m46.363s 220m35.528s
sys 3m3.397s 0m12.492s 2m41.393s

numa_hit 5615517 4560123 4963875
numa_local 5615505 4560024 4963700
numa_other 12 99 175
numa_pte_updates 1822797 493 1559111
numa_hint_faults 1307113 523 1469031
numa_hint_faults_local 612617 488 884829
numa_pages_migrated 694370 35 584202

Summary: Regression in base is recovered by allowing scanning as required.

[1] https://lore.kernel.org/lkml/[email protected]/T/#t

Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
regression.
Reported-by: Aithal Srikanth <[email protected]>
Reported-by: kernel test robot <[email protected]>
Closes: https://lore.kernel.org/lkml/[email protected]/T/
Signed-off-by: Raghavendra K T <[email protected]>
---
include/linux/mm_types.h | 1 +
kernel/sched/fair.c | 31 ++++++++++++++++++++++++-------
2 files changed, 25 insertions(+), 7 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 306a3d1a0fa6..992e460a713e 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -479,6 +479,7 @@ struct vma_numab_state {
unsigned long next_scan;
unsigned long next_pid_reset;
unsigned long access_pids[2];
+ unsigned int scan_counter;
};

/*
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 373ff5f55884..4e71fb58085b 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -2931,17 +2931,30 @@ static void reset_ptenuma_scan(struct task_struct *p)
static bool vma_is_accessed(struct vm_area_struct *vma)
{
unsigned long pids;
+ unsigned int vma_size;
+ unsigned int scan_threshold;
+ unsigned int scan_size;
+
+ pids = vma->numab_state->access_pids[0] | vma->numab_state->access_pids[1];
+
+ if (test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids))
+ return true;
+
+ scan_size = READ_ONCE(sysctl_numa_balancing_scan_size);
+ /* vma size in MB */
+ vma_size = (vma->vm_end - vma->vm_start) >> 20;
+
+ /* Total scans needed to cover VMA */
+ scan_threshold = vma_size / scan_size;
+
/*
- * Allow unconditional access first two times, so that all the (pages)
- * of VMAs get prot_none fault introduced irrespective of accesses.
+ * Allow the scanning of half of disjoint set's VMA to induce
+ * prot_none fault irrespective of accesses.
* This is also done to avoid any side effect of task scanning
* amplifying the unfairness of disjoint set of VMAs' access.
*/
- if (READ_ONCE(current->mm->numa_scan_seq) < 2)
- return true;
-
- pids = vma->numab_state->access_pids[0] | vma->numab_state->access_pids[1];
- return test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids);
+ scan_threshold = 1 + (scan_threshold >> 1);
+ return (vma->numab_state->scan_counter < scan_threshold);
}

#define VMA_PID_RESET_PERIOD (4 * sysctl_numa_balancing_scan_delay)
@@ -3058,6 +3071,8 @@ static void task_numa_work(struct callback_head *work)
/* Reset happens after 4 times scan delay of scan start */
vma->numab_state->next_pid_reset = vma->numab_state->next_scan +
msecs_to_jiffies(VMA_PID_RESET_PERIOD);
+
+ vma->numab_state->scan_counter = 0;
}

/*
@@ -3084,6 +3099,8 @@ static void task_numa_work(struct callback_head *work)
vma->numab_state->access_pids[1] = 0;
}

+ vma->numab_state->scan_counter++;
+
do {
start = max(start, vma->vm_start);
end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
--
2.34.1

2023-07-05 06:07:04

by Raghavendra K T

[permalink] [raw]

Subject: Re: [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression

On 5/31/2023 9:55 AM, Raghavendra K T wrote:
> With the numa scan enhancements [1], only the threads which had previously
> accessed vma are allowed to scan.
>
> While this had improved significant system time overhead, there were corner
> cases, which genuinely need some relaxation. For e.g.,
>
> 1) Concern raised by PeterZ, where if there are N partition sets of vmas
> belonging to tasks, then unfairness in allowing these threads to scan could
> potentially amplify the side effect of some of the vmas being left
> unscanned.
>
> 2) Below reports of LKP numa01 benchmark regression.
>
> Currently this is handled by allowing first two scanning unconditional
> as indicated by mm->numa_scan_seq. This is imprecise since for some
> benchmark vma scanning might itself start at numa_scan_seq > 2.
>
> Solution:
> Allow unconditional scanning of vmas of tasks depending on vma size. This
> is achieved by maintaining a per vma scan counter, where
>
> f(allowed_to_scan) = f(scan_counter < vma_size / scan_size)
>
> Result:
> numa01_THREAD_ALLOC result on 6.4.0-rc2 (that has numascan enhancement)
> base-numascan base base+fix
> real 1m1.507s 1m23.259s 1m2.632s
> user 213m51.336s 251m46.363s 220m35.528s
> sys 3m3.397s 0m12.492s 2m41.393s
>
> numa_hit 5615517 4560123 4963875
> numa_local 5615505 4560024 4963700
> numa_other 12 99 175
> numa_pte_updates 1822797 493 1559111
> numa_hint_faults 1307113 523 1469031
> numa_hint_faults_local 612617 488 884829
> numa_pages_migrated 694370 35 584202
>
> Summary: Regression in base is recovered by allowing scanning as required.
>
> [1] https://lore.kernel.org/lkml/[email protected]/T/#t
>
> Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
> regression.
> Reported-by: Aithal Srikanth <[email protected]>
> Reported-by: kernel test robot <[email protected]>
> Closes: https://lore.kernel.org/lkml/[email protected]/T/
> Signed-off-by: Raghavendra K T <[email protected]>

Hello kernel test robot,

Gentle ping to check if the patch has helped your regression report.

Thanks
- Raghu

2023-07-16 14:50:04

by kernel test robot

[permalink] [raw]

Subject: Re: [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression

hi, Raghavendra K T,

On Wed, Jul 05, 2023 > On 5/31/2023 > > With > > accessed > >
> > While this > > cases, > >
> > 1) Concern > > belonging > > potentially > > unscanned.
> >
> > 2) Below > >
> > Currently > > as indicated > > benchmark > >
> > Solution:
> > Allow unconditional > > is achieved > >
> > f(allowed_to_scan) > >
> > Result:
> > numa01_THREAD_ALLOC > > > > real > > user > > sys > >
> > numa_hit > > numa_local > > numa_other > > numa_pte_updates > > numa_hint_faults > > numa_hint_faults_local > > numa_pages_migrated > >
> > Summary: > >
> > [1] > > Fixes: > > regression.
> > Reported-by: > > Reported-by: > > Closes: > Hello kernel test robot,
>
> Gentle ping
sorry for late.

previously we found on a Cascade Lake, we cannot test your
however, we also
========================== compiler/cpufreq_governor/ gcc-11/performance/4x/x8
ef6a22b70f6d9044 ---------------- %stddev %change \ | 193.14 8.14 -0.7%

now we tested your (55fd15913b18d6a790c17d947df
========================== compiler/cpufreq_governor/ gcc-11/performance/4x/x8
ef6a22b70f6d9044 ---------------- %stddev %change \ | 193.14 8.14 -0.9%
detail comparison as below:

========================== compiler/cpufreq_governor/ gcc-11/performance/4x/x8
ef6a22b70f6d9044 ---------------- %stddev %change \ | 1.185e+10 ? 6% 12184642 ? 6% 5.00 ?100% -100.0% 189.35 ? 11% 1021 +12.6% 21665 ? 6% +50.5% 47.15 ? 16% +84.9% 42.12 ? 17% +88.2% 9448 ? 18% +60.5% 6.93 +267.2% 5.43 ? 5% +1.8 0.02 ? 15% -0.0 1.08 -0.1 0.06 ? 2% +0.0 1.99 -0.2 91.41 -1.6 0.00 -100.0% 1912287 ? 6% 2592838 ? 5% 680538 ? 6% 0.00 -100.0% 1890172 ? 6% 2436852 ? 5% 546653 ? 6% 970.73 +9.1% 970.73 +9.1% 624931 25.00 ? 21% 3150370 2514815 4096 +0.0% 20893 3931 -0.8% 198882 37974 ? 13% -21.0% 5.17 ? 7% +35.5% 2.50 ? 20% -20.0% 90.83 0.00 -100.0% 1.67 ? 44% -40.0% 4.00 +0.0% 3256207 ? 3% 2.524e+08 0.00 -100.0% 212.17 2330 ? 7% -7.8% 233033 193.14 40.49 ? 3% +27.3% 8.14 -0.9% 970.73 +9.1% 970.73 +9.1% 624931 25.00 ? 21% 3150370 2514815 4096 +0.0% 20893 3931 -0.8% 198882 37974 ? 13% -21.0% 2737 -1.8% 94.39 -1.7 2900 +0.0% 12957 ? 36% +15.9% 5540462 ? 6% 2.48 ? 5% +0.8 6453445 ? 6% 2.92 ? 5% +0.9 5.61 ? 5% +30.8% 43.17 ? 7% 2.307e+08 7.90 ? 31% +5.7 28989 ? 36% -32.3% 43.17 ? 7% 666.52 70.96 2000 +0.0% 237031 ? 49% 236935 ? 49% 96.00 ? 33% 6155451 6257767 4.00 +0.0% 3088930 ? 3% 1.318e+08 8554815 2.567e+08 10951680 ? 3% 2437198 2048 +0.0% 6294588 6294432 156.00 ? 5% 167387 39732 87203 ? 6% +12.8% 2.514e+08 2.524e+08 2.637e+08 11248567 1604 +4.3% 16758 110655 167387 430222 273371 ? 42% 597610 2815544 1.374e+13 338296 12911235 29565 ?175% +18.3% 29522 ?175% +18.3% 42.67 ?111% +49.6% 3202041 ? 2% 3256105 ? 2% 6450186 2728164 ? 2% 3266857 ? 2% 3266783 ? 2% 74.00 ?100% +10.6% 101285 ? 9% 20573 ? 3% 51364 ? 7% 1.246e+08 1.316e+08 7033196 ? 2% 1487 ? 7% -6.6% 9628 ? 12% -26.6% 101285 ? 9% 237810 ? 7% 39839 ?139% +11.9% 339095 ? 5% 2688324 ? 2% 207469 ? 53% 207416 ? 53% 53.33 ?107% +20.0% 2949624 ? 3% 2997887 ? 2% 6474808 360720 ? 40% +136.0% 3024066 ? 2% 3023983 ? 2% 82.00 ? 93% 66102 ? 14% 19151 ? 3% 35805 ? 13% +38.0% 1.278e+08 1.32e+08 4211780 ? 6% 116.67 ?100% +145.4% 7122 ? 16% +36.6% 66102 ? 14% 192408 ? 9% 233482 ? 47% 258511 ? 6% 127219 ? 57% +295.5% 7390 ?175% +18.1% 10.67 ?111% +48.4% 812998 ? 2% 1560 ? 2% -8.2% 682054 ? 2% 31146566 815598 ? 2% 18.50 ?100% 40.50 ? 14% -62.1% 20572 ? 3% 12953 ? 7% 371.33 ? 7% 2406 ? 12% -26.6% 9973 ?139% +11.7% 25321 ? 9% 59452 ? 7% 672080 ? 2% 7390 ?175% +18.1% 10.67 ?111% +48.4% 815599 ? 2% 18.50 ?100% 672080 ? 2% 2593013 ? 5% 0.00 -100.0% 1912463 ? 6% 680538 ? 6% 51877 ? 53% +52.2% 13.33 ?107% +20.0% 750017 ? 2% 1440 ? 3% +11.1% 90208 ? 40% +135.9% 31957700 756559 ? 2% 20.50 ? 93% 37.00 ? 9% -26.6% 19147 ? 3% 9012 ? 13% +37.6% 29.00 ?100% +146.6% 1779 ? 16% +36.7% 58399 ? 47% +49.0% 16525 ? 14% 48100 ? 9% 31804 ? 57% +295.5% 51877 ? 53% +52.2% 13.33 ?107% +20.0% 756560 ? 2% 20.50 ? 93% 31804 ? 57% +295.5% 2436786 ? 5% 0.00 -100.0% 1890106 ? 6% 546653 ? 6% 1159 6.17 ? 17% -2.7% 59245 ? 49% +48.0% 24.00 ? 33% +33.3% 1562608 3001 +1.1% 6272807 12560952 772234 ? 3% 63104657 1571739 39.00 ? 5% 77.67 ? 6% -50.9% 39726 21963 ? 6% +12.8% 400.67 4186 +0.3% 68343 ? 42% +43.6% 41846 107554 703885 59245 ? 49% +48.0% 24.00 ? 33% +33.3% 1571739 39.00 ? 5% 703885 200540 ? 6% 120494 ? 5% 5032020 112746 ? 2% -46.1% 0.00 -100.0% 3804791 1227192 11866841 ? 4% 57894552 ? 2% 74510 ? 14% +10.3% 0.00 -100.0% 4.537e+08 4907690 4.537e+08 86.17 ?221% -94.6% 11866841 ? 4% 2376 ? 44% -40.0% 172880 98.67 ? 18% 14.67 ? 35% 26654 0.17 ?223% -100.0% 23128 ? 4% -40.8% 14.67 ? 35% 0.00 -100.0% 1005 ? 12% 607.00 4.00 ? 14% -8.3% 3.83 ? 17% -13.0% 7602432 1459346 -100.0% 2043 ? 47% -100.0% 1755990 -100.0% 54.87 ? 10% -28.5% 4.041e+08 0.62 -0.0 1799541 29.53 ? 2% -3.0 52797131 ? 10% 1.172e+08 ? 6% 2294 ? 7% -7.8% 541.44 224169 6.06e+11 233.65 94785 ? 2% 0.07 ? 9% +0.0 224367 ? 12% 4.949e+08 0.60 -0.0 1041912 1.832e+08 1.893e+09 0.00 ? 3% -4.0% 0.11 ? 19% -22.4% 2.70 -1.7% 1516 ? 2% +0.7% 3.79 ? 2% -6.3% 5088 -5.0% 49.22 -1.9 965573 ? 3% 740879 ? 8% 5088 -5.0% 224169 59.37 ? 5% -17.5% 0.45 +0.0 44.29 ? 4% -3.9 332.08 12720 ? 10% +34.7% 0.05 ? 18% +0.0 0.57 -0.0 0.00 -1.9% 55.05 ? 6% -6.7 3.897e+08 1760709 48311108 ? 9% 1.089e+08 ? 5% 2280 ? 7% -7.7% 223740 6.088e+11 230.49 223824 ? 18% 4.8e+08 1041413 1.812e+08 1.833e+09 0.11 ? 18% -21.9% 4861 -5.2% 971538 ? 5% 795675 ? 10% 4861 -5.2% 223740 1.781e+12 32005 ? 86% +12.8% 6653945 ? 94% 0.00 +0.0% 456998 ? 91% 0.92 ? 2% -1.1% 1.90 ? 4% -6.2% 0.67 ? 18% -5.0% 0.15 ? 12% +0.4% 5652 ? 6% -3.0% 250428 ? 13% 1548 ? 26% -8.3% 18136 ? 14% -12.5% 7.26 ? 9% -0.6% 215.19 ? 20% 1.55 ? 18% -9.0% 22.20 ? 15% 32005 ? 86% +12.8% 6653945 ? 94% 0.00 +0.0% 456998 ? 91% 1.203e+08 ? 5% 1.231e+08 ? 5% 99360296 ? 6% 2529598 ? 9% 0.91 ? 2% -0.9% 1.10 ? 7% -3.8% 0.67 ? 18% -5.0% 0.08 ? 39% +16.1% 1.68 ? 48% -5.5% 76.51 ? 34% 10.52 ? 17% -6.9% 0.82 ? 50% -7.1% 37.12 ? 24% 5.08 ? 19% -7.5% 0.81 ? 52% -7.1% 35.30 ? 14% 4.96 ? 22% -7.5% 944.71 ? 2% 1881 ? 3% -6.7% 643.11 ? 13% 141.77 ? 13% 0.00 ?100% +78.5% 0.38 ?100% +78.5% 0.03 ?100% +78.5% 8911395 ? 18% 11635928 ? 17% -12145667 2510183 ? 10% 934.78 ? 2% 1278 ? 5% -6.2% 515.26 ? 12% 95.53 ? 24% 441.09 ? 8% 1272 ? 6% -10.8% 280.15 ? 9% 1308353 ? 3% 2922459 ? 2% 448661 ? 15% 412548 ? 5% 505223 ? 4% 505890 ? 4% 504409 ? 4% 415.73 ? 25% 499837 ? 4% 501130 ? 4% 478575 ? 4% 1619 ? 20% +150.3% 17029 ? 6% 19932 ? 3% 9932 ? 24% 1540 ? 44% +20.8% 591935 ? 2% 1240997 ? 5% 500000 143032 ? 7% 4295 +0.0% 4295 +0.0% 4295 +0.0% 0.00 ? 25% -4.1% 0.91 ? 3% -2.1% 1.92 ? 5% -5.3% 0.68 ? 16% -5.2% 0.16 ? 16% +4.0% 5837 ? 5% +4.8% 49744 ? 20% +13.8% 2002 ? 6% +14.9% 5880 ? 11% -1.9% 2.039e+09 ? 3% -100.0% 4.295e+09 -100.0% 0.00 -6.3e+103% 2.14e+09 -100.0% 504394 ? 4% 996147 996147 996147 4.295e+09 499768 ? 4% 0.00 ? 19% -4.0% 0.12 ? 19% -4.0% 0.01 ? 19% -4.0% 0.00 ? 19% -4.0% 0.12 ? 19% -4.0% 0.01 ? 19% -4.0% 950.00 950.00 950.00 0.00 +4.1e+97% 0.00 +9.3e+99% 0.00 +6.2e+98% 0.20 ? 55% +57.2% 44.78 ? 55% +57.2% 2.99 ? 55% +57.2% 505282 ? 3% 1.00 +0.0% 58611259 0.75 +0.0% 24.00 3.00 +0.0% 1.00 +0.0% 4.00 +0.0% 11.97 ? 60% -8.6 11.86 ? 60% -8.6 11.84 ? 60% -8.6 11.59 ? 61% -8.5 11.58 ? 64% -8.4 9.33 ? 69% -8.3 9.19 ? 69% -8.2 9.19 ? 69% -8.2 9.18 ? 69% -8.2 8.62 ? 69% -7.7 8.61 ? 69% -7.7 8.59 ? 69% -7.7 8.48 ? 69% -7.6 4.67 ?111% -3.8 4.66 ?111% -3.8 3.99 ?113% -3.3 2.20 ? 53% -2.2 2.72 ?109% -2.2 1.40 ? 28% -1.4 1.00 ?131% -0.9 0.90 ?108% -0.8 0.85 ? 53% -0.6 0.85 ? 53% -0.6 0.86 ? 52% -0.6 0.85 ? 53% -0.6 0.85 ? 53% -0.6 7.65 ? 55% -0.6 7.65 ? 55% -0.6 0.53 ? 49% -0.5 0.52 ? 49% -0.5 0.51 ? 49% -0.5 1.90 ? 23% -0.5 0.66 ? 52% -0.5 0.44 ?223% -0.4 0.43 ? 76% -0.4 0.43 ?223% -0.4 0.43 ?223% -0.4 1.50 ? 54% -0.4 1.50 ? 54% -0.4 1.49 ? 55% -0.4 0.43 ?223% -0.4 5.41 ? 57% -0.4 0.41 ? 76% -0.4 5.37 ? 56% -0.4 5.44 ? 56% -0.4 5.51 ? 57% -0.4 1.41 ? 55% -0.4 5.56 ? 57% -0.4 5.57 ? 57% -0.4 0.56 ?165% -0.4 5.58 ? 57% -0.4 5.60 ? 57% -0.4 1.94 ? 23% -0.4 0.72 ? 57% -0.3 1.40 ? 28% -0.3 0.31 ?105% -0.3 0.31 ?103% -0.3 0.30 ?103% -0.3 0.30 ?103% -0.3 0.30 ?103% -0.3 0.30 ?103% -0.3 0.30 ?103% -0.3 0.30 ?103% -0.3 0.30 ?103% -0.3 0.40 ?154% -0.3 0.28 ?100% -0.3 0.27 ?100% -0.3 1.03 ? 68% -0.3 0.26 ?100% -0.3 0.26 ?100% -0.3 1.05 ? 69% -0.3 5.76 ? 58% -0.3 0.35 ?103% -0.3 0.24 ?223% -0.2 0.24 ?150% -0.2 0.82 ? 31% -0.2 0.44 ? 75% -0.2 5.88 ? 57% -0.2 0.44 ? 75% -0.2 2.83 ? 29% -0.2 0.23 ?142% -0.2 0.23 ?142% -0.2 0.32 ?100% -0.2 0.32 ?100% -0.2 0.31 ?100% -0.2 0.22 ?142% -0.2 0.22 ?144% -0.2 0.22 ?223% -0.2 3.04 ? 37% -0.2 0.20 ?141% -0.2 0.19 ?141% -0.2 0.52 ? 80% -0.2 0.19 ?141% -0.2 0.19 ?141% -0.2 0.19 ?141% -0.2 0.19 ?141% -0.2 0.19 ?223% -0.2 0.55 ? 47% -0.2 0.19 ?223% -0.2 0.19 ?223% -0.2 2.24 ? 53% -0.2 0.18 ?223% -0.2 0.18 ?223% -0.2 0.87 ? 57% -0.2 0.26 ?145% -0.2 0.26 ?145% -0.2 0.25 ?145% -0.2 0.25 ?145% -0.2 0.16 ?223% -0.2 0.16 ?223% -0.2 0.16 ?223% -0.2 1.21 ? 30% -0.2 0.35 ?103% -0.2 1.22 ? 30% -0.2 0.97 ? 69% -0.1 1.24 ? 30% -0.1 0.14 ?223% -0.1 0.14 ?223% -0.1 0.23 ?146% -0.1 0.14 ?223% -0.1 0.14 ?223% -0.1 0.31 ?104% -0.1 6.66 ? 58% -0.1 0.14 ?223% -0.1 0.14 ?223% -0.1 0.14 ?223% -0.1 0.54 ? 57% -0.1 6.37 ? 58% -0.1 0.22 ?145% -0.1 0.13 ?223% -0.1 0.22 ?145% -0.1 0.13 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.82 ? 70% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.39 ?103% -0.1 1.30 ? 33% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 3.05 ? 37% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.38 ?103% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.55 ? 53% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.20 ?144% -0.1 0.11 ?223% -0.1 0.11 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.84 ? 70% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.48 ? 75% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.96 ? 32% -0.1 0.96 ? 32% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.09 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 0.08 ?223% -0.1 1.57 ? 30% -0.1 1.57 ? 30% -0.1 1.57 ? 30% -0.1 0.50 ? 80% -0.1 0.28 ?151% -0.1 0.70 ? 37% -0.1 0.23 ?149% -0.1 0.84 ? 40% -0.1 0.66 ? 37% -0.1 2.88 ? 30% -0.0 2.89 ? 30% -0.0 0.13 ?223% -0.0 0.85 ? 40% -0.0 2.98 ? 30% -0.0 0.87 ? 40% -0.0 2.91 ? 30% -0.0 2.91 ? 30% -0.0 0.40 ? 77% -0.0 0.33 ?107% -0.0 0.12 ?223% -0.0 0.13 ?223% -0.0 0.98 ? 39% -0.0 0.96 ? 39% -0.0 0.12 ?223% -0.0 0.12 ?223% -0.0 0.12 ?223% -0.0 0.95 ? 40% -0.0 0.96 ? 39% -0.0 0.12 ?223% -0.0 0.93 ? 39% -0.0 0.11 ?223% -0.0 0.12 ?223% -0.0 0.39 ?114% -0.0 0.24 ?143% -0.0 0.22 ?144% -0.0 1.58 ? 57% -0.0 0.23 ?144% -0.0 0.23 ?144% -0.0 0.45 ? 75% +0.0 5.79 ? 27% +0.0 0.47 ? 75% +0.0 0.64 ? 48% +0.0 5.74 ? 27% +0.0 0.68 ? 58% +0.0 0.68 ? 58% +0.0 1.45 ? 57% +0.0 0.57 ? 59% +0.0 0.50 ? 77% +0.0 0.08 ?223% +0.0 0.50 ? 77% +0.0 0.09 ?223% +0.0 0.49 ? 77% +0.0 0.50 ? 77% +0.0 0.50 ? 77% +0.0 0.50 ? 77% +0.0 0.91 ? 34% +0.0 0.55 ? 58% +0.0 3.33 ? 32% +0.0 0.78 ? 39% +0.0 0.12 ?223% +0.1 0.12 ?223% +0.1 0.12 ?223% +0.1 3.32 ? 32% +0.1 0.12 ?223% +0.1 0.12 ?223% +0.1 0.13 ?223% +0.1 7.93 ? 55% +0.1 7.92 ? 55% +0.1 7.92 ? 55% +0.1 3.54 ? 26% +0.1 0.15 ?223% +0.1 5.95 ? 27% +0.1 3.55 ? 26% +0.1 0.00 +0.1 5.87 ? 27% +0.1 0.00 +0.1 5.85 ? 27% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 2.63 ? 26% +0.1 2.61 ? 27% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.68 ? 59% +0.1 0.68 ? 59% +0.1 0.09 ?223% +0.1 0.00 +0.1 0.63 ? 55% +0.1 0.00 +0.1 1.12 ? 36% +0.1 1.12 ? 36% +0.1 1.12 ? 36% +0.1 1.12 ? 36% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.80 ? 36% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.46 ? 80% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.14 ?223% +0.1 0.14 ?223% +0.1 0.14 ?223% +0.1 0.00 +0.1 3.77 ? 23% +0.1 3.77 ? 23% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.14 ?223% +0.2 0.14 ?223% +0.2 0.11 ?223% +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.56 ? 61% +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 1.18 ? 36% +0.2 0.66 ? 28% +0.2 0.32 ?107% +0.3 0.32 ?108% +0.3 0.00 +0.3 0.00 +0.3 0.74 ? 30% +0.3 0.62 ? 59% +0.3 0.55 ? 84% +0.3 0.55 ? 84% +0.3 1.65 ? 33% +0.3 1.65 ? 34% +0.3 1.64 ? 33% +0.3 0.67 ? 59% +0.3 0.00 +0.3 0.00 +0.3 0.00 +0.3 0.00 +0.3 0.00 +0.3 0.09 ?223% +0.3 0.09 ?223% +0.3 0.09 ?223% +0.3 2.44 ? 35% +0.3 0.00 +0.3 0.00 +0.4 0.09 ?223% +0.4 0.09 ?223% +0.4 0.09 ?223% +0.4 0.30 ?146% +0.4 0.00 +0.4 0.00 +0.4 0.00 +0.5 0.00 +0.5 0.00 +0.5 0.00 +0.5 0.00 +0.5 0.11 ?223% +0.5 0.00 +0.5 3.57 ? 36% +0.5 3.55 ? 36% +0.5 3.56 ? 36% +0.6 3.56 ? 36% +0.6 3.56 ? 35% +0.6 0.00 +0.6 0.25 ?223% +0.6 0.27 ?223% +0.6 0.27 ?223% +0.6 0.27 ?223% +0.6 0.00 +0.6 0.00 +0.8 0.00 +1.0 0.00 +1.1 0.10 ?223% +1.8 0.10 ?223% +1.9 0.00 +2.0 0.11 ?223% +2.1 0.15 ?223% +2.2 0.15 ?223% +2.2 0.15 ?223% +2.2 0.16 ?223% +2.5 45.48 ? 45% +6.5 45.08 ? 45% +6.9 45.07 ? 45% +6.9 44.86 ? 45% +7.1 39.39 ? 47% +11.2 38.05 ? 48% +11.6 38.22 ? 48% +11.6 14.24 ? 54% -8.8 13.84 ? 55% -8.8 13.89 ? 55% -8.8 13.27 ? 56% -8.7 12.85 ? 57% -8.5 9.37 ? 68% -8.2 9.28 ? 68% -8.1 9.27 ? 68% -8.1 9.26 ? 68% -8.1 9.01 ? 66% -7.8 8.65 ? 68% -7.6 8.65 ? 68% -7.6 8.63 ? 68% -7.6 4.81 ?112% -3.9 4.67 ?111% -3.8 4.11 ?113% -3.4 2.40 ? 50% -2.4 2.81 ?110% -2.3 1.40 ? 28% -1.4 1.02 ?130% -0.9 0.92 ?109% -0.8 30.39 ? 33% -0.8 30.33 ? 33% -0.8 7.81 ? 54% -0.7 0.55 ? 34% -0.6 0.51 ?191% -0.5 1.90 ? 23% -0.5 0.49 ? 34% -0.5 0.44 ? 43% -0.4 0.43 ?223% -0.4 0.43 ?223% -0.4 0.43 ?223% -0.4 1.06 ? 28% -0.4 1.06 ? 28% -0.4 5.43 ? 56% -0.4 5.38 ? 56% -0.4 5.46 ? 56% -0.4 0.53 ? 49% -0.4 0.47 ? 56% -0.4 1.07 ? 28% -0.4 5.73 ? 55% -0.4 5.79 ? 55% -0.4 0.50 ?128% -0.4 5.64 ? 56% -0.4 1.94 ? 23% -0.4 0.50 ? 59% -0.3 0.34 ? 42% -0.3 0.39 ? 77% -0.3 0.41 ? 55% -0.3 0.83 ? 27% -0.3 1.40 ? 28% -0.3 0.36 ? 74% -0.3 0.34 ? 74% -0.3 0.66 ? 26% -0.3 1.43 ? 53% -0.3 0.34 ? 74% -0.3 0.57 ? 24% -0.3 1.42 ? 31% -0.3 0.71 ?119% -0.3 5.78 ? 58% -0.3 0.29 ? 70% -0.3 0.24 ?223% -0.2 0.31 ? 60% -0.2 0.28 ? 69% -0.2 0.63 ? 28% -0.2 0.24 ?148% -0.2 5.88 ? 57% -0.2 1.71 ? 31% -0.2 5.27 ? 26% -0.2 0.24 ? 93% -0.2 0.22 ?223% -0.2 0.24 ? 94% -0.2 0.22 ?223% -0.2 3.71 ? 30% -0.2 3.69 ? 30% -0.2 1.08 ? 63% -0.2 1.18 ? 32% -0.2 1.35 ? 32% -0.2 3.89 ? 30% -0.2 3.90 ? 31% -0.2 1.28 ? 32% -0.2 1.58 ? 33% -0.2 0.54 ? 56% -0.2 0.19 ?223% -0.2 0.64 ? 46% -0.2 1.67 ? 28% -0.2 1.66 ? 29% -0.2 0.24 ? 57% -0.2 0.72 ? 25% -0.2 0.72 ? 26% -0.2 0.72 ? 26% -0.2 0.44 ? 25% -0.2 0.24 ? 54% -0.2 2.24 ? 53% -0.2 6.39 ? 27% -0.2 0.73 ? 25% -0.2 0.94 ? 27% -0.2 0.72 ? 24% -0.2 0.72 ? 24% -0.2 0.72 ? 23% -0.2 0.72 ? 23% -0.2 0.72 ? 23% -0.2 0.72 ? 23% -0.2 0.72 ? 23% -0.2 0.85 ? 29% -0.2 0.70 ? 40% -0.2 0.81 ? 30% -0.2 0.82 ? 30% -0.2 0.16 ?223% -0.2 0.93 ? 22% -0.2 0.18 ? 92% -0.2 6.51 ? 27% -0.2 0.16 ? 39% -0.1 0.22 ?149% -0.1 0.15 ?223% -0.1 1.24 ? 30% -0.1 0.22 ? 66% -0.1 0.20 ? 79% -0.1 0.14 ? 27% -0.1 0.14 ? 27% -0.1 0.66 ? 22% -0.1 1.09 ? 35% -0.1 0.21 ? 48% -0.1 1.09 ? 35% -0.1 0.50 ? 48% -0.1 1.10 ? 35% -0.1 6.66 ? 58% -0.1 1.11 ? 36% -0.1 1.01 ? 37% -0.1 0.13 ?223% -0.1 0.13 ?190% -0.1 6.37 ? 58% -0.1 0.88 ? 38% -0.1 0.80 ? 30% -0.1 0.30 ?109% -0.1 0.12 ? 44% -0.1 0.48 ? 44% -0.1 0.12 ? 20% -0.1 0.49 ? 34% -0.1 0.49 ? 34% -0.1 0.12 ?223% -0.1 1.09 ? 35% -0.1 1.30 ? 33% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.12 ?223% -0.1 0.58 ? 27% -0.1 0.13 ? 40% -0.1 0.11 ?190% -0.1 0.13 ? 26% -0.1 0.14 ? 53% -0.1 0.38 ? 27% -0.1 0.11 ?185% -0.1 0.11 ?223% -0.1 0.15 ? 69% -0.1 0.16 ? 80% -0.1 0.18 ? 89% -0.1 0.38 ? 38% -0.1 0.10 ?223% -0.1 0.77 ? 35% -0.1 0.38 ? 43% -0.1 0.24 ? 60% -0.1 0.10 ?223% -0.1 0.34 ? 40% -0.1 0.22 ? 33% -0.1 0.57 ? 32% -0.1 0.55 ? 26% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.10 ?223% -0.1 0.22 ? 83% -0.1 0.12 ? 83% -0.1 0.10 ?223% -0.1 0.96 ? 32% -0.1 0.21 ? 27% -0.1 0.14 ? 35% -0.1 0.55 ? 26% -0.1 0.46 ? 50% -0.1 0.51 ? 47% -0.1 0.45 ? 41% -0.1 0.45 ? 74% -0.1 0.37 ? 24% -0.1 0.53 ? 25% -0.1 0.20 ? 95% -0.1 0.09 ?223% -0.1 0.50 ? 28% -0.1 0.09 ?189% -0.1 0.30 ? 44% -0.1 0.09 ?223% -0.1 0.21 ? 72% -0.1 0.27 ? 28% -0.1 0.27 ? 24% -0.1 0.27 ? 31% -0.1 0.78 ? 32% -0.1 0.08 ?188% -0.1 0.08 ? 27% -0.1 0.27 ? 21% -0.1 1.63 ? 31% -0.1 0.44 ? 29% -0.1 0.08 ?187% -0.1 0.42 ? 26% -0.1 0.24 ? 56% -0.1 0.24 ? 76% -0.1 0.27 ? 28% -0.1 1.63 ? 31% -0.1 0.09 ?106% -0.1 0.42 ? 29% -0.1 0.46 ? 41% -0.1 0.14 ? 42% -0.1 0.33 ? 60% -0.1 1.62 ? 31% -0.1 0.42 ? 29% -0.1 0.52 ? 35% -0.1 0.39 ? 36% -0.1 0.17 ? 33% -0.1 0.13 ? 68% -0.1 0.42 ? 29% -0.1 0.67 ? 30% -0.1 0.54 ? 34% -0.1 0.38 ? 36% -0.1 0.38 ? 36% -0.1 0.15 ? 13% -0.1 0.07 ?223% -0.1 0.20 ? 55% -0.1 0.09 ?155% -0.1 0.07 ? 83% -0.1 0.24 ? 27% -0.1 0.07 ?192% -0.1 0.27 ? 26% -0.1 0.07 ? 82% -0.1 0.07 ?223% -0.1 0.78 ? 40% -0.1 0.60 ? 37% -0.1 0.47 ? 34% -0.1 0.11 ? 30% -0.1 0.11 ? 28% -0.1 0.06 ?223% -0.1 0.16 ? 85% -0.1 0.60 ? 37% -0.1 0.38 ? 30% -0.1 0.44 ? 35% -0.1 0.06 ? 79% -0.1 0.06 ?223% -0.1 0.14 ? 33% -0.1 0.08 ? 85% -0.1 0.32 ? 49% -0.1 0.06 ?223% -0.1 0.06 ? 93% -0.1 0.06 ?223% -0.1 0.16 ? 27% -0.1 0.07 ? 63% -0.1 0.06 ? 75% -0.1 0.06 ?223% -0.1 0.16 ? 28% -0.1 0.21 ? 48% -0.1 0.16 ? 28% -0.1 0.12 ? 55% -0.1 0.76 ? 29% -0.1 0.22 ? 29% -0.1 0.25 ? 30% -0.1 0.08 ? 43% -0.1 0.22 ? 43% -0.1 0.35 ? 34% -0.1 0.34 ? 36% -0.1 0.34 ? 36% -0.1 0.08 ? 35% -0.1 0.35 ? 34% -0.1 0.10 ? 29% -0.1 0.34 ? 36% -0.1 0.27 ? 36% -0.1 0.28 ? 36% -0.0 0.32 ? 37% -0.0 0.18 ? 14% -0.0 1.39 ? 38% -0.0 0.28 ? 37% -0.0 1.44 ? 38% -0.0 0.47 ? 34% -0.0 0.30 ? 35% -0.0 0.05 ?223% -0.0 2.98 ? 30% -0.0 0.10 ? 34% -0.0 0.11 ? 57% -0.0 0.05 ?223% -0.0 0.06 ? 86% -0.0 0.13 ? 78% -0.0 0.07 ? 64% -0.0 0.34 ? 38% -0.0 3.67 ? 26% -0.0 0.56 ? 26% -0.0 0.38 ? 34% -0.0 0.11 ? 38% -0.0 0.04 ?148% -0.0 0.06 ? 84% -0.0 0.12 ? 39% -0.0 0.46 ? 26% -0.0 0.13 ? 19% -0.0 0.26 ? 28% -0.0 0.31 ? 33% -0.0 0.14 ? 27% -0.0 0.14 ? 27% -0.0 0.11 ? 35% -0.0 0.04 ?151% -0.0 0.06 ? 14% -0.0 0.16 ? 26% -0.0 0.20 ? 26% -0.0 0.19 ? 37% -0.0 0.20 ? 33% -0.0 0.46 ? 25% -0.0 0.46 ? 25% -0.0 0.46 ? 25% -0.0 0.46 ? 26% -0.0 0.46 ? 25% -0.0 0.11 ? 35% -0.0 0.06 ? 14% -0.0 0.07 ?101% -0.0 0.10 ? 36% -0.0 0.10 ? 36% -0.0 0.08 ? 29% -0.0 0.16 ? 35% -0.0 0.10 ? 43% -0.0 0.65 ? 34% -0.0 0.44 ? 26% -0.0 0.04 ?223% -0.0 0.07 ? 21% -0.0 0.07 ? 21% -0.0 0.04 ? 79% -0.0 0.06 ? 47% -0.0 0.04 ?223% -0.0 0.09 ? 23% -0.0 0.04 ?104% -0.0 0.04 ? 72% -0.0 0.06 ? 83% -0.0 0.19 ? 34% -0.0 0.07 ? 16% -0.0 0.04 ?223% -0.0 0.04 ?112% -0.0 0.05 ? 87% -0.0 0.07 ? 83% -0.0 0.08 ? 26% -0.0 0.20 ? 14% -0.0 0.22 ? 36% -0.0 0.10 ? 39% -0.0 0.65 ? 34% -0.0 0.20 ? 27% -0.0 0.11 ? 22% -0.0 0.11 ? 22% -0.0 0.04 ?109% -0.0 0.04 ?152% -0.0 0.12 ? 29% -0.0 0.08 ? 24% -0.0 0.03 ? 70% -0.0 0.16 ? 48% -0.0 0.12 ? 34% -0.0 0.65 ? 34% -0.0 0.11 ? 44% -0.0 0.03 ?223% -0.0 0.08 ? 29% -0.0 0.09 ? 66% -0.0 0.03 ?155% -0.0 0.03 ?223% -0.0 0.12 ? 58% -0.0 0.09 ? 28% -0.0 0.21 ? 31% -0.0 0.18 ? 34% -0.0 0.07 ? 10% -0.0 0.03 ?105% -0.0 0.03 ?223% -0.0 0.10 ? 23% -0.0 0.79 ? 30% -0.0 0.13 ? 83% -0.0 0.10 ? 53% -0.0 0.08 ? 26% -0.0 0.04 ?156% -0.0 0.03 ?158% -0.0 0.07 ? 56% -0.0 0.37 ? 30% -0.0 0.32 ? 29% -0.0 0.04 ?102% -0.0 0.07 ? 31% -0.0 0.63 ? 34% -0.0 0.38 ? 38% -0.0 0.13 ? 40% -0.0 0.02 ? 99% -0.0 0.02 ? 99% -0.0 0.02 ? 99% -0.0 0.06 ? 50% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.24 ? 32% -0.0 0.17 ? 24% -0.0 0.10 ? 31% -0.0 0.10 ? 46% -0.0 0.12 ? 27% -0.0 0.40 ? 30% -0.0 0.26 ? 18% -0.0 0.14 ? 58% -0.0 0.03 ?108% -0.0 0.03 ?108% -0.0 0.03 ?103% -0.0 0.02 ?143% -0.0 0.13 ? 37% -0.0 0.14 ? 38% -0.0 0.12 ? 30% -0.0 0.06 ? 28% -0.0 0.05 ? 80% -0.0 0.03 ?157% -0.0 0.32 ? 74% -0.0 0.30 ? 36% -0.0 0.20 ? 25% -0.0 0.20 ? 25% -0.0 0.38 ? 40% -0.0 0.19 ? 28% -0.0 0.18 ? 30% -0.0 0.17 ? 54% -0.0 0.02 ?223% -0.0 0.04 ?107% -0.0 0.15 ? 36% -0.0 0.02 ?146% -0.0 0.02 ?142% -0.0 0.02 ?146% -0.0 0.05 ? 80% -0.0 0.02 ?142% -0.0 0.02 ?142% -0.0 0.02 ?142% -0.0 0.50 ? 36% -0.0 0.37 ? 58% -0.0 0.13 ? 40% -0.0 0.13 ? 26% -0.0 0.12 ? 37% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.05 ? 77% -0.0 0.02 ?144% -0.0 0.02 ?223% -0.0 0.02 ?144% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.02 ?144% -0.0 0.02 ?144% -0.0 0.02 ?223% -0.0 0.03 ?101% -0.0 0.04 ?223% -0.0 0.12 ? 43% -0.0 0.11 ? 21% -0.0 0.33 ? 28% -0.0 0.29 ? 36% -0.0 0.46 ? 31% -0.0 0.11 ? 34% -0.0 0.19 ? 31% -0.0 0.09 ? 81% -0.0 0.08 ? 35% -0.0 0.08 ? 33% -0.0 0.03 ?147% -0.0 0.02 ?142% -0.0 0.06 ? 52% -0.0 0.03 ?100% -0.0 0.03 ?102% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.02 ?142% -0.0 0.02 ?142% -0.0 0.10 ? 27% -0.0 0.42 ? 31% -0.0 0.47 ? 36% -0.0 0.16 ? 38% -0.0 0.18 ? 37% -0.0 0.02 ?152% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.12 ? 39% -0.0 0.11 ? 21% -0.0 0.06 ? 94% -0.0 0.08 ? 42% -0.0 1.58 ? 57% -0.0 1.58 ? 57% -0.0 0.48 ? 36% -0.0 0.48 ? 36% -0.0 0.29 ? 44% -0.0 0.23 ? 29% -0.0 0.22 ? 36% -0.0 0.05 ?109% -0.0 0.07 ? 85% -0.0 0.02 ?152% -0.0 0.04 ?165% -0.0 0.02 ?143% -0.0 0.02 ?223% -0.0 0.05 ?111% -0.0 0.07 ? 16% -0.0 0.07 ? 33% -0.0 0.03 ?101% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.47 ? 43% -0.0 0.02 ?152% -0.0 0.18 ? 40% -0.0 0.12 ? 45% -0.0 0.04 ?116% -0.0 0.04 ?104% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.03 ?100% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.10 ? 34% -0.0 0.21 ? 35% -0.0 0.10 ? 61% -0.0 0.13 ? 43% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.02 ?144% -0.0 0.02 ?144% -0.0 0.01 ?223% -0.0 0.02 ?144% -0.0 0.01 ?223% -0.0 0.02 ?146% -0.0 0.02 ?144% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.04 ?111% -0.0 0.05 ? 91% -0.0 0.15 ? 70% -0.0 0.12 ? 26% -0.0 0.56 ? 55% -0.0 0.30 ? 28% -0.0 0.15 ? 28% -0.0 0.20 ? 41% -0.0 0.12 ? 40% -0.0 0.07 ? 48% -0.0 0.05 ? 47% -0.0 0.07 ? 60% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.02 ?142% -0.0 0.01 ?223% -0.0 0.02 ?144% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.10 ? 39% -0.0 0.16 ? 45% -0.0 0.10 ? 34% -0.0 0.06 ? 54% -0.0 0.03 ?102% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.10 ? 30% -0.0 0.10 ? 44% -0.0 0.04 ?112% -0.0 0.05 ?119% -0.0 0.24 ? 33% -0.0 0.23 ? 36% -0.0 0.20 ? 28% -0.0 0.02 ? 99% -0.0 0.09 ? 34% -0.0 0.09 ? 32% -0.0 0.11 ? 41% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.03 ?161% -0.0 0.36 ? 30% -0.0 0.20 ? 30% -0.0 0.10 ? 39% -0.0 0.06 ? 70% -0.0 0.10 ? 51% -0.0 0.07 ? 54% -0.0 0.10 ? 37% -0.0 0.07 ? 33% -0.0 0.02 ? 99% -0.0 0.03 ?103% -0.0 0.03 ?108% -0.0 0.02 ?142% -0.0 0.10 ? 14% -0.0 0.21 ? 32% -0.0 0.03 ?103% -0.0 0.02 ?144% -0.0 0.05 ? 71% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.03 ?152% -0.0 0.16 ? 32% -0.0 0.14 ? 36% -0.0 0.20 ? 32% -0.0 0.20 ? 26% -0.0 0.21 ? 36% -0.0 0.10 ? 69% -0.0 0.13 ? 56% -0.0 0.06 ? 75% -0.0 0.01 ?223% -0.0 0.02 ?144% -0.0 0.02 ?144% -0.0 0.01 ?223% -0.0 0.12 ? 43% -0.0 0.10 ? 52% -0.0 0.07 ? 58% -0.0 0.03 ?111% -0.0 0.04 ?149% -0.0 0.10 ? 34% +0.0 0.08 ? 49% +0.0 0.08 ? 47% +0.0 0.07 ? 67% +0.0 0.06 ? 92% +0.0 0.04 ? 73% +0.0 0.05 ? 74% +0.0 0.02 ?141% +0.0 0.02 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.04 ?106% +0.0 0.26 ? 51% +0.0 0.07 ? 47% +0.0 0.04 ?107% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.03 ?101% +0.0 0.02 ?145% +0.0 0.04 ?104% +0.0 0.24 ? 30% +0.0 0.21 ? 34% +0.0 0.13 ? 32% +0.0 0.56 ? 28% +0.0 0.21 ? 57% +0.0 0.14 ? 30% +0.0 0.13 ? 62% +0.0 0.04 ? 76% +0.0 0.06 ? 60% +0.0 0.02 ?141% +0.0 0.01 ?223% +0.0 0.12 ? 27% +0.0 0.09 ? 31% +0.0 0.06 ? 76% +0.0 0.10 ? 29% +0.0 0.05 ? 78% +0.0 0.05 ? 77% +0.0 0.06 ? 78% +0.0 0.55 ? 28% +0.0 0.24 ? 55% +0.0 0.14 ? 32% +0.0 0.09 ? 19% +0.0 0.04 ?104% +0.0 0.25 ? 34% +0.0 0.14 ? 38% +0.0 0.01 ?223% +0.0 0.02 ?146% +0.0 0.03 ?108% +0.0 0.02 ?144% +0.0 0.17 ? 73% +0.0 0.53 ? 45% +0.0 0.48 ? 54% +0.0 0.41 ? 25% +0.0 0.10 ? 36% +0.0 0.02 ?152% +0.0 0.07 ? 53% +0.0 0.03 ?150% +0.0 0.02 ?146% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.06 ? 61% +0.0 0.06 ? 55% +0.0 0.19 ? 17% +0.0 0.28 ? 40% +0.0 0.38 ? 46% +0.0 0.13 ? 25% +0.0 0.06 ? 58% +0.0 0.02 ?149% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.06 ? 63% +0.0 0.06 ? 58% +0.0 0.06 ? 58% +0.0 0.06 ? 58% +0.0 0.06 ? 58% +0.0 0.05 ? 49% +0.0 0.28 ? 40% +0.0 0.22 ? 33% +0.0 0.18 ? 38% +0.0 0.04 ?105% +0.0 0.02 ?141% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.02 ?223% +0.0 0.10 ? 36% +0.0 0.05 ? 52% +0.0 0.16 ? 22% +0.0 0.12 ? 34% +0.0 0.03 ?143% +0.0 0.01 ?223% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.10 ? 48% +0.0 0.01 ?223% +0.0 0.08 ? 52% +0.0 0.12 ? 22% +0.0 0.36 ? 15% +0.0 0.15 ? 33% +0.0 0.14 ? 40% +0.0 0.04 ?101% +0.0 0.06 ? 74% +0.0 0.06 ? 60% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.70 ? 37% +0.0 0.23 ? 40% +0.0 0.06 ? 86% +0.0 0.15 ? 39% +0.0 0.04 ?106% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.21 ? 42% +0.0 0.76 ? 37% +0.0 0.52 ? 52% +0.0 1.12 ? 31% +0.0 0.88 ? 31% +0.0 0.02 ?144% +0.0 0.04 ?112% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.02 ?141% +0.0 0.03 ?158% +0.0 0.15 ? 37% +0.0 0.10 ? 45% +0.0 0.13 ? 34% +0.0 0.04 ?114% +0.0 0.01 ?223% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.02 ?223% +0.0 1.31 ? 35% +0.0 0.28 ? 28% +0.0 0.79 ? 49% +0.0 0.32 ? 44% +0.0 0.18 ? 33% +0.0 0.05 ?119% +0.0 0.04 ?112% +0.0 0.03 ?100% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.03 ?143% +0.0 0.06 ? 49% +0.0 0.09 ? 33% +0.0 0.00 +0.0 0.00 +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.10 ? 34% +0.0 0.39 ? 32% +0.0 0.02 ?141% +0.0 0.02 ?149% +0.0 0.14 ? 34% +0.0 0.58 ? 34% +0.0 0.13 ? 34% +0.0 0.04 ? 71% +0.0 0.39 ? 61% +0.0 0.15 ? 37% +0.0 1.28 ? 36% +0.0 0.05 ? 84% +0.0 0.03 ?102% +0.0 0.00 +0.0 0.22 ? 36% +0.0 0.08 ? 47% +0.0 0.03 ?100% +0.0 0.06 ? 66% +0.0 0.00 +0.0 0.06 ? 50% +0.0 0.10 ? 31% +0.0 0.38 ? 34% +0.0 0.00 +0.0 0.00 +0.0 0.26 ? 37% +0.0 0.14 ? 57% +0.0 0.00 +0.0 0.31 ? 35% +0.0 0.02 ?144% +0.0 0.08 ? 81% +0.0 0.10 ? 35% +0.0 0.28 ? 24% +0.0 0.08 ? 48% +0.0 0.16 ? 62% +0.0 0.16 ? 62% +0.0 0.01 ?223% +0.0 0.23 ? 30% +0.0 0.02 ?142% +0.0 0.20 ? 32% +0.0 0.08 ? 17% +0.0 0.13 ? 17% +0.0 0.00 +0.0 0.00 +0.0 0.03 ? 70% +0.0 0.00 +0.0 0.00 +0.0 0.78 ? 39% +0.0 0.08 ? 16% +0.0 0.01 ?223% +0.0 0.10 ? 45% +0.1 0.09 ? 56% +0.1 0.24 ? 33% +0.1 0.04 ?104% +0.1 0.73 ? 31% +0.1 0.07 ? 37% +0.1 0.00 +0.1 0.00 +0.1 0.01 ?223% +0.1 0.13 ? 23% +0.1 7.93 ? 55% +0.1 7.92 ? 55% +0.1 7.92 ? 55% +0.1 0.00 +0.1 0.12 ? 33% +0.1 0.14 ? 41% +0.1 0.18 ? 27% +0.1 0.03 ?102% +0.1 0.01 ?223% +0.1 0.00 +0.1 5.97 ? 27% +0.1 0.01 ?223% +0.1 0.07 ?134% +0.1 0.00 +0.1 0.00 +0.1 0.65 ? 30% +0.1 0.00 +0.1 0.00 +0.1 1.87 ? 37% +0.1 0.14 ?112% +0.1 0.00 +0.1 1.87 ? 37% +0.1 2.63 ? 26% +0.1 2.62 ? 26% +0.1 0.75 ? 42% +0.1 1.75 ? 37% +0.1 0.00 +0.1 0.04 ?168% +0.1 0.91 ? 33% +0.1 0.00 +0.1 0.82 ? 38% +0.1 0.35 ? 34% +0.1 1.00 ? 34% +0.1 0.63 ? 39% +0.1 1.12 ? 36% +0.1 0.35 ? 65% +0.1 0.35 ? 66% +0.1 0.01 ?223% +0.1 0.04 ? 89% +0.1 0.97 ? 34% +0.1 0.79 ? 28% +0.1 0.02 ?223% +0.1 0.02 ?146% +0.1 1.87 ? 36% +0.1 0.81 ? 28% +0.1 0.12 ? 51% +0.1 0.22 ? 46% +0.1 0.47 ? 32% +0.1 1.06 ? 33% +0.1 0.00 +0.1 3.77 ? 23% +0.1 0.16 ? 43% +0.2 0.80 ? 44% +0.2 0.00 +0.2 0.66 ? 46% +0.2 0.65 ? 39% +0.2 0.71 ? 38% +0.2 0.00 +0.2 0.00 +0.2 1.77 ? 27% +0.2 1.18 ? 36% +0.2 3.79 ? 23% +0.2 0.75 ? 38% +0.2 0.00 +0.2 0.00 +0.2 0.23 ? 39% +0.2 0.01 ?223% +0.2 0.01 ?223% +0.2 0.01 ?223% +0.2 0.66 ? 28% +0.2 0.23 ? 40% +0.2 0.00 +0.3 0.00 +0.3 0.00 +0.3 0.87 ? 38% +0.3 0.74 ? 30% +0.3 0.00 +0.3 0.00 +0.3 1.66 ? 34% +0.3 1.65 ? 34% +0.3 0.29 ? 57% +0.3 0.25 ? 71% +0.3 1.64 ? 33% +0.3 0.00 +0.3 0.00 +0.3 0.33 ? 39% +0.3 2.48 ? 35% +0.3 0.28 ? 41% +0.3 0.94 ? 39% +0.4 0.00 +0.4 0.00 +0.4 0.03 ?223% +0.4 0.00 +0.5 0.92 ? 38% +0.5 1.14 ? 34% +0.5 3.57 ? 36% +0.5 3.57 ? 36% +0.5 3.58 ? 36% +0.6 0.01 ?223% +0.6 0.72 ? 36% +0.6 0.01 ?223% +0.6 0.29 ? 42% +0.6 0.02 ?223% +0.6 0.02 ?146% +0.6 1.68 ? 32% +0.7 1.43 ? 33% +0.7 0.89 ? 43% +0.8 1.54 ? 26% +0.8 1.50 ? 40% +0.9 0.42 ? 69% +0.9 0.39 ? 72% +1.0 0.52 ? 72% +1.0 0.54 ? 72% +1.1 0.54 ? 73% +1.1 0.00 +1.1 1.68 ? 41% +1.1 4.06 ? 25% +1.1 0.00 +1.1 0.63 ? 72% +1.2 0.21 ? 94% +2.0 0.00 +2.0 0.21 ? 94% +2.1 0.31 ? 89% +2.1 0.23 ? 91% +2.3 0.33 ? 87% +2.3 0.33 ? 87% +2.3 0.33 ? 87% +2.3 53.13 ? 31% +5.9 45.08 ? 45% +6.9 45.07 ? 45% +6.9 44.88 ? 45% +7.1 39.40 ? 47% +11.2 38.23 ? 48% +11.6 38.12 ? 48% +11.9 8.95 ? 66% -7.8 2.34 ? 51% -2.3 1.88 ?111% -1.4 0.92 ?109% -0.8 0.67 ?105% -0.5 0.49 ? 31% -0.5 0.44 ? 43% -0.4 0.44 ?128% -0.3 0.60 ?124% -0.3 0.70 ?120% -0.3 1.52 ? 22% -0.3 0.30 ? 47% -0.2 0.21 ?144% -0.2 0.62 ? 27% -0.2 0.17 ? 73% -0.2 0.36 ? 33% -0.1 0.17 ? 90% -0.1 0.65 ? 23% -0.1 0.15 ? 28% -0.1 0.64 ? 27% -0.1 0.12 ? 45% -0.1 0.29 ?109% -0.1 0.38 ? 27% -0.1 0.13 ? 40% -0.1 0.10 ?195% -0.1 0.14 ? 58% -0.1 0.38 ? 42% -0.1 0.23 ? 61% -0.1 0.45 ? 41% -0.1 0.50 ? 28% -0.1 0.08 ? 22% -0.1 0.09 ? 17% -0.1 0.34 ? 23% -0.1 0.40 ? 24% -0.1 0.07 ? 82% -0.1 0.06 ? 65% -0.1 0.06 ? 47% -0.1 0.06 ?133% -0.1 0.31 ? 42% -0.1 0.06 ? 94% -0.1 0.06 ?223% -0.1 0.47 ? 34% -0.1 0.22 ? 27% -0.1 0.06 ? 76% -0.1 0.08 ? 43% -0.1 0.18 ? 29% -0.1 0.73 ? 30% -0.1 0.07 ? 39% -0.1 0.08 ? 60% -0.1 0.06 ?145% -0.0 0.06 ?107% -0.0 0.21 ? 36% -0.0 0.33 ? 38% -0.0 0.04 ?151% -0.0 0.11 ? 60% -0.0 0.06 ? 14% -0.0 0.05 ? 84% -0.0 0.08 ? 29% -0.0 0.26 ? 33% -0.0 0.24 ? 30% -0.0 0.29 ? 24% -0.0 0.06 ? 84% -0.0 0.06 ? 47% -0.0 0.17 ? 32% -0.0 0.18 ? 32% -0.0 0.06 ? 49% -0.0 0.04 ? 71% -0.0 0.05 ? 75% -0.0 0.06 ? 51% -0.0 0.20 ? 14% -0.0 0.16 ? 44% -0.0 0.14 ? 29% -0.0 0.04 ?101% -0.0 0.07 ? 31% -0.0 0.35 ? 32% -0.0 0.12 ? 29% -0.0 0.03 ?223% -0.0 0.03 ?100% -0.0 0.03 ?108% -0.0 0.03 ?108% -0.0 0.09 ? 67% -0.0 0.05 ?121% -0.0 0.10 ? 23% -0.0 0.03 ?223% -0.0 0.03 ?146% -0.0 0.35 ? 35% -0.0 0.19 ? 26% -0.0 0.03 ?144% -0.0 0.03 ?100% -0.0 0.26 ? 18% -0.0 0.04 ?105% -0.0 0.03 ?100% -0.0 0.03 ?100% -0.0 0.03 ?100% -0.0 0.08 ? 56% -0.0 0.02 ? 99% -0.0 0.02 ? 99% -0.0 0.02 ?152% -0.0 0.02 ?223% -0.0 0.02 ?145% -0.0 0.10 ? 47% -0.0 0.04 ?104% -0.0 0.03 ?108% -0.0 0.02 ?149% -0.0 0.02 ?223% -0.0 0.14 ? 37% -0.0 0.06 ? 28% -0.0 0.05 ? 80% -0.0 0.03 ?157% -0.0 0.27 ? 58% -0.0 0.10 ? 68% -0.0 0.19 ? 28% -0.0 0.02 ?223% -0.0 0.10 ? 58% -0.0 0.02 ?142% -0.0 0.02 ?142% -0.0 0.02 ?142% -0.0 0.07 ? 78% -0.0 0.02 ?144% -0.0 0.13 ? 57% -0.0 0.18 ? 29% -0.0 0.12 ? 43% -0.0 0.08 ? 35% -0.0 0.03 ?147% -0.0 0.02 ?142% -0.0 0.02 ?223% -0.0 0.02 ?142% -0.0 0.17 ? 30% -0.0 0.12 ? 39% -0.0 0.03 ?155% -0.0 0.02 ?223% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.02 ?141% -0.0 0.13 ? 50% -0.0 0.07 ? 69% -0.0 0.18 ? 25% -0.0 0.02 ?223% -0.0 0.02 ?223% -0.0 0.03 ?100% -0.0 0.12 ? 21% -0.0 0.06 ? 48% -0.0 0.08 ? 35% -0.0 0.05 ? 87% -0.0 0.03 ?108% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.03 ?100% -0.0 0.02 ?142% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.04 ? 75% -0.0 0.03 ?103% -0.0 0.04 ? 72% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.02 ?144% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.04 ?111% -0.0 0.11 ? 40% -0.0 0.11 ? 28% -0.0 0.06 ? 58% -0.0 0.05 ? 47% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.02 ?144% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.12 ? 29% -0.0 0.11 ? 57% -0.0 0.01 ?223% -0.0 0.02 ?141% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.02 ?141% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.09 ? 35% -0.0 0.06 ? 82% -0.0 0.11 ? 43% -0.0 0.02 ? 99% -0.0 0.02 ?149% -0.0 0.02 ?141% -0.0 0.02 ?223% -0.0 0.04 ?105% -0.0 0.06 ? 70% -0.0 0.10 ? 39% -0.0 0.09 ? 36% -0.0 0.07 ? 53% -0.0 0.11 ? 25% -0.0 0.04 ? 79% -0.0 0.12 ? 38% -0.0 0.07 ? 35% -0.0 0.02 ?142% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.05 ? 79% -0.0 0.03 ?108% -0.0 0.02 ?144% -0.0 0.01 ?223% -0.0 0.01 ?223% -0.0 0.22 ? 54% -0.0 0.04 ?149% -0.0 0.04 ? 73% -0.0 0.13 ? 32% +0.0 0.10 ? 36% +0.0 0.04 ?105% +0.0 0.02 ?142% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.14 ? 30% +0.0 0.01 ?223% +0.0 0.04 ?109% +0.0 0.03 ?106% +0.0 0.18 ? 27% +0.0 0.02 ?141% +0.0 0.02 ?141% +0.0 0.01 ?223% +0.0 0.05 ? 77% +0.0 0.05 ? 85% +0.0 0.01 ?223% +0.0 0.33 ? 33% +0.0 0.20 ? 52% +0.0 0.10 ? 27% +0.0 0.08 ? 59% +0.0 0.01 ?223% +0.0 0.05 ?104% +0.0 0.03 ?111% +0.0 0.06 ? 58% +0.0 0.03 ?148% +0.0 0.01 ?223% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.18 ? 19% +0.0 0.13 ? 25% +0.0 0.01 ?223% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.10 ? 42% +0.0 0.03 ?101% +0.0 0.01 ?223% +0.0 0.01 ?223% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.09 ? 86% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.02 ?142% +0.0 0.03 ?103% +0.0 0.08 ? 50% +0.0 0.10 ? 17% +0.0 0.00 +0.0 0.00 +0.0 0.01 ?223% +0.0 0.09 ? 30% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.05 ?117% +0.0 0.12 ? 48% +0.0 0.10 ? 45% +0.0 0.04 ?100% +0.0 0.03 ?150% +0.0 0.01 ?223% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.02 ?141% +0.0 0.03 ?158% +0.0 0.04 ?112% +0.0 0.00 +0.0 0.00 +0.0 0.04 ? 77% +0.0 0.10 ? 34% +0.0 0.01 ?223% +0.0 0.06 ? 46% +0.0 0.06 ? 60% +0.0 0.03 ?101% +0.0 0.35 ? 34% +0.0 0.01 ?223% +0.0 0.07 ?104% +0.0 0.09 ? 34% +0.0 1.28 ? 36% +0.0 0.04 ?110% +0.0 0.09 ? 31% +0.0 0.07 ? 97% +0.0 0.04 ? 71% +0.0 0.00 +0.0 0.30 ? 30% +0.0 0.06 ? 50% +0.0 0.07 ? 83% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.10 ? 35% +0.0 0.29 ? 60% +0.0 0.01 ?223% +0.0 0.00 +0.0 0.15 ? 29% +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.00 +0.0 0.05 ? 78% +0.0 0.23 ? 34% +0.1 0.26 ? 29% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.18 ? 27% +0.1 0.13 ? 44% +0.1 1.38 ? 27% +0.1 0.00 +0.1 0.56 ? 56% +0.1 0.01 ?223% +0.1 0.01 ?223% +0.1 0.64 ? 39% +0.1 0.14 ?110% +0.1 0.00 +0.1 0.04 ?168% +0.1 0.45 ? 41% +0.1 0.33 ? 70% +0.1 0.12 ? 53% +0.1 0.00 +0.1 0.00 +0.2 0.00 +0.3 0.33 ? 39% +0.3 0.03 ?223% +0.4 0.00 +0.4 0.00 +0.6 0.02 ?223% +0.6 0.52 ? 72% +1.0 0.00 +1.1 0.00 +2.0 37.64 ? 48% +11.6

>
> Thanks
> - Raghu
>
>
>

at 11:18:37AM +0530, Raghavendra K T wrote:
9:55 AM, Raghavendra K T wrote:
the numa scan enhancements [1], only the threads which had previously
vma are allowed to scan.
had improved significant system time overhead, there were corner
which genuinely need some relaxation. For e.g.,
raised by PeterZ, where if there are N partition sets of vmas
to tasks, then unfairness in allowing these threads to scan could
amplify the side effect of some of the vmas being left
reports of LKP numa01 benchmark regression.
this is handled by allowing first two scanning unconditional
by mm->numa_scan_seq. This is imprecise since for some
vma scanning might itself start at numa_scan_seq > 2.
scanning of vmas of tasks depending on vma size. This
by maintaining a per vma scan counter, where
= f(scan_counter < vma_size / scan_size)
result on 6.4.0-rc2 (that has numascan enhancement)
base-numascan base base+fix
1m1.507s 1m23.259s 1m2.632s
213m51.336s 251m46.363s 220m35.528s
3m3.397s 0m12.492s 2m41.393s
5615517 4560123 4963875
5615505 4560024 4963700
12 99 175
1822797 493 1559111
1307113 523 1469031
612617 488 884829
694370 35 584202
Regression in base is recovered by allowing scanning as required.
href="https://lore.kernel.org/lkml/cover.1677672277.git.raghavendra.kt@amd.com/T/#t">https://lore.kernel.org/lkml/[email protected]/T/#t
fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
Aithal Srikanth <[email protected]>
kernel test robot <[email protected]>
href="https://lore.kernel.org/lkml/db995c11-08ba-9abf-812f-01407f70a5d4@amd.com/T/">https://lore.kernel.org/lkml/[email protected]/T/
Raghavendra K T <[email protected]>
to check if the patch has helped your regression report.
a 118.9% regression of autonuma-benchmark.numa01.seconds
which happened to be converted for other test purposes, so
patch on it again.
found a 39.3% regression on a Sapphire Rapids test machine:
===============================================================
iterations/kconfig/rootfs/tbox_group/test/testcase:
6_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-spr-r02/numa02_SMT/autonuma-benchmark
fc137c0ddab29b591db6a091dc6
---------------------------
%stddev
\
+39.2% 268.84 autonuma-benchmark.numa01.seconds
8.09 autonuma-benchmark.numa02.seconds
v3 patch on it, found regression mostly recovered
is just [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression)
===============================================================
iterations/kconfig/rootfs/tbox_group/test/testcase:
6_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-spr-r02/numa01_THREAD_ALLOC/autonuma-benchmark
55fd15913b18d6a790c17d947df
---------------------------
%stddev
\
+5.8% 204.37 ? 3% autonuma-benchmark.numa01.seconds
8.06 autonuma-benchmark.numa02.seconds
===============================================================
iterations/kconfig/rootfs/tbox_group/test/testcase:
6_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-spr-r02/numa01_THREAD_ALLOC/autonuma-benchmark
55fd15913b18d6a790c17d947df
---------------------------
%stddev
\
+43.9% 1.705e+10 ? 6% cpuidle..time
+44.0% 17542134 ? 6% cpuidle..usage
0.00 dmesg.bootstage:last
+32.1% 250.07 ? 20% dmesg.timestamp:last
1149 ? 3% uptime.boot
32596 ? 22% uptime.idle
87.19 ? 53% boot-time.boot
79.25 ? 58% boot-time.dhcp
15162 ? 51% boot-time.idle
25.45 ? 76% boot-time.smp_boot
7.21 ? 5% mpstat.cpu.all.idle%
0.01 ? 10% mpstat.cpu.all.iowait%
1.03 mpstat.cpu.all.irq%
0.08 ? 2% mpstat.cpu.all.soft%
1.81 ? 2% mpstat.cpu.all.sys%
89.86 mpstat.cpu.all.usr%
0.00 numa-numastat.node0.interleave_hit
-3.9% 1837158 ? 13% numa-numastat.node0.local_node
-9.8% 2338452 ? 13% numa-numastat.node0.numa_hit
-26.3% 501281 ? 30% numa-numastat.node0.other_node
0.00 numa-numastat.node1.interleave_hit
+11.2% 2101474 ? 14% numa-numastat.node1.local_node
+15.4% 2811758 ? 13% numa-numastat.node1.numa_hit
+29.9% 710290 ? 23% numa-numastat.node1.other_node
1059 ? 2% time.elapsed_time
1059 ? 2% time.elapsed_time.max
+4.9% 655561 ? 2% time.involuntary_context_switches
+8.7% 27.17 ? 19% time.major_page_faults
+0.0% 3150530 time.maximum_resident_set_size
-0.6% 2498715 ? 2% time.minor_page_faults
4096 time.page_size
-1.9% 20499 time.percent_of_cpu_this_job_got
3899 time.system_time
+7.2% 213298 ? 2% time.user_time
29982 ? 7% time.voluntary_context_switches
7.00 ? 8% vmstat.cpu.id
2.00 vmstat.cpu.sy
-1.8% 89.17 vmstat.cpu.us
0.00 vmstat.cpu.wa
1.00 ?100% vmstat.io.bi
4.00 vmstat.memory.buff
+3.6% 3373341 vmstat.memory.cache
-0.0% 2.523e+08 vmstat.memory.free
0.00 vmstat.procs.b
-1.6% 208.83 vmstat.procs.r
2148 vmstat.system.cs
-1.3% 229952 vmstat.system.in
+5.8% 204.37 ? 3% autonuma-benchmark.numa01.seconds
51.53 ? 4% autonuma-benchmark.numa01_THREAD_ALLOC.seconds
8.06 autonuma-benchmark.numa02.seconds
1059 ? 2% autonuma-benchmark.time.elapsed_time
1059 ? 2% autonuma-benchmark.time.elapsed_time.max
+4.9% 655561 ? 2% autonuma-benchmark.time.involuntary_context_switches
+8.7% 27.17 ? 19% autonuma-benchmark.time.major_page_faults
+0.0% 3150530 autonuma-benchmark.time.maximum_resident_set_size
-0.6% 2498715 ? 2% autonuma-benchmark.time.minor_page_faults
4096 autonuma-benchmark.time.page_size
-1.9% 20499 autonuma-benchmark.time.percent_of_cpu_this_job_got
3899 autonuma-benchmark.time.system_time
+7.2% 213298 ? 2% autonuma-benchmark.time.user_time
29982 ? 7% autonuma-benchmark.time.voluntary_context_switches
2687 turbostat.Avg_MHz
92.66 turbostat.Busy%
2900 turbostat.Bzy_MHz
15015 ? 25% turbostat.C1
+46.1% 8093574 ? 6% turbostat.C1E
3.30 ? 6% turbostat.C1E%
+43.6% 9265716 ? 6% turbostat.C6
3.84 ? 6% turbostat.C6%
7.34 ? 5% turbostat.CPU%c1
+6.2% 45.83 ? 5% turbostat.CoreTmp
+6.0% 2.446e+08 ? 2% turbostat.IRQ
13.56 ? 32% turbostat.PKG_%
19619 ? 11% turbostat.POLL
+6.6% 46.00 ? 6% turbostat.PkgTmp
-0.1% 665.80 turbostat.PkgWatt
-2.9% 68.89 turbostat.RAMWatt
2000 turbostat.TSC_MHz
+47.9% 350630 ? 3% meminfo.Active
+47.9% 350502 ? 3% meminfo.Active(anon)
+33.3% 128.00 meminfo.Active(file)
+1.0% 6215315 meminfo.AnonHugePages
+0.4% 6284278 meminfo.AnonPages
4.00 meminfo.Buffers
+4.0% 3211320 meminfo.Cached
-0.0% 1.318e+08 meminfo.CommitLimit
+1.6% 8690945 meminfo.Committed_AS
-0.1% 2.566e+08 meminfo.DirectMap1G
+17.0% 12811605 ? 5% meminfo.DirectMap2M
-69.1% 752036 ? 14% meminfo.DirectMap4k
2048 meminfo.Hugepagesize
+0.5% 6326486 meminfo.Inactive
+0.5% 6326322 meminfo.Inactive(anon)
+5.0% 163.83 meminfo.Inactive(file)
-3.1% 162160 meminfo.KReclaimable
+0.0% 39751 meminfo.KernelStack
98392 ? 14% meminfo.Mapped
-0.0% 2.513e+08 meminfo.MemAvailable
-0.0% 2.523e+08 meminfo.MemFree
-0.0% 2.637e+08 meminfo.MemTotal
+0.8% 11333611 meminfo.Memused
1673 meminfo.Mlocked
+0.3% 16803 meminfo.PageTables
-7.3% 102584 meminfo.Percpu
-3.1% 162160 meminfo.SReclaimable
-0.4% 428319 meminfo.SUnreclaim
+43.6% 392445 meminfo.Shmem
-1.2% 590480 meminfo.Slab
+0.1% 2818830 meminfo.Unevictable
+0.0% 1.374e+13 meminfo.VmallocTotal
-17.2% 280268 meminfo.VmallocUsed
-0.2% 12891540 meminfo.max_used_kB
34976 ?214% numa-meminfo.node0.Active
34912 ?214% numa-meminfo.node0.Active(anon)
63.83 ?100% numa-meminfo.node0.Active(file)
-8.3% 2935773 ? 10% numa-meminfo.node0.AnonHugePages
-8.9% 2967039 ? 10% numa-meminfo.node0.AnonPages
-0.6% 6408448 numa-meminfo.node0.AnonPages.max
-13.5% 2360218 ? 41% numa-meminfo.node0.FilePages
-8.9% 2976525 ? 10% numa-meminfo.node0.Inactive
-8.9% 2976443 ? 10% numa-meminfo.node0.Inactive(anon)
81.83 ? 93% numa-meminfo.node0.Inactive(file)
-9.0% 92138 ? 27% numa-meminfo.node0.KReclaimable
-2.1% 20149 ? 8% numa-meminfo.node0.KernelStack
-5.0% 48807 ? 35% numa-meminfo.node0.Mapped
+0.7% 1.254e+08 numa-meminfo.node0.MemFree
+0.0% 1.316e+08 numa-meminfo.node0.MemTotal
-11.9% 6193901 ? 15% numa-meminfo.node0.MemUsed
1388 ? 37% numa-meminfo.node0.Mlocked
7070 ? 36% numa-meminfo.node0.PageTables
-9.0% 92138 ? 27% numa-meminfo.node0.SReclaimable
-6.4% 222602 ? 5% numa-meminfo.node0.SUnreclaim
44566 ?194% numa-meminfo.node0.Shmem
-7.2% 314741 ? 10% numa-meminfo.node0.Slab
-13.9% 2315704 ? 41% numa-meminfo.node0.Unevictable
+52.2% 315756 ? 25% numa-meminfo.node1.Active
+52.2% 315692 ? 25% numa-meminfo.node1.Active(anon)
64.00 ?100% numa-meminfo.node1.Active(file)
+11.1% 3278227 ? 9% numa-meminfo.node1.AnonHugePages
+10.6% 3315853 ? 9% numa-meminfo.node1.AnonPages
+0.0% 6477523 numa-meminfo.node1.AnonPages.max
851138 ?114% numa-meminfo.node1.FilePages
+10.7% 3348368 ? 9% numa-meminfo.node1.Inactive
+10.7% 3348286 ? 9% numa-meminfo.node1.Inactive(anon)
-0.2% 81.83 ? 93% numa-meminfo.node1.Inactive(file)
+5.9% 70021 ? 36% numa-meminfo.node1.KReclaimable
+2.3% 19600 ? 8% numa-meminfo.node1.KernelStack
49423 ? 17% numa-meminfo.node1.Mapped
-0.7% 1.269e+08 numa-meminfo.node1.MemFree
-0.0% 1.32e+08 numa-meminfo.node1.MemTotal
+22.0% 5137530 ? 18% numa-meminfo.node1.MemUsed
286.33 ?183% numa-meminfo.node1.Mlocked
9729 ? 27% numa-meminfo.node1.PageTables
+5.9% 70021 ? 36% numa-meminfo.node1.SReclaimable
+6.9% 205718 ? 5% numa-meminfo.node1.SUnreclaim
+49.0% 347910 ? 24% numa-meminfo.node1.Shmem
+6.7% 275740 ? 11% numa-meminfo.node1.Slab
503125 ?189% numa-meminfo.node1.Unevictable
8730 ?214% numa-vmstat.node0.nr_active_anon
15.83 ?100% numa-vmstat.node0.nr_active_file
-8.8% 741650 ? 10% numa-vmstat.node0.nr_anon_pages
1432 ? 10% numa-vmstat.node0.nr_anon_transparent_hugepages
-13.5% 590056 ? 41% numa-vmstat.node0.nr_file_pages
+0.7% 31355317 numa-vmstat.node0.nr_free_pages
-8.8% 744048 ? 10% numa-vmstat.node0.nr_inactive_anon
+9.9% 20.33 ? 94% numa-vmstat.node0.nr_inactive_file
15.33 ? 39% numa-vmstat.node0.nr_isolated_anon
-2.1% 20150 ? 8% numa-vmstat.node0.nr_kernel_stack
-4.8% 12333 ? 35% numa-vmstat.node0.nr_mapped
-6.6% 346.67 ? 37% numa-vmstat.node0.nr_mlock
1767 ? 36% numa-vmstat.node0.nr_page_table_pages
11143 ?194% numa-vmstat.node0.nr_shmem
-9.0% 23034 ? 27% numa-vmstat.node0.nr_slab_reclaimable
-6.4% 55651 ? 5% numa-vmstat.node0.nr_slab_unreclaimable
-13.9% 578925 ? 41% numa-vmstat.node0.nr_unevictable
8730 ?214% numa-vmstat.node0.nr_zone_active_anon
15.83 ?100% numa-vmstat.node0.nr_zone_active_file
-8.8% 744047 ? 10% numa-vmstat.node0.nr_zone_inactive_anon
+9.9% 20.33 ? 94% numa-vmstat.node0.nr_zone_inactive_file
-13.9% 578925 ? 41% numa-vmstat.node0.nr_zone_unevictable
-9.8% 2338261 ? 13% numa-vmstat.node0.numa_hit
0.00 numa-vmstat.node0.numa_interleave
-3.9% 1836967 ? 13% numa-vmstat.node0.numa_local
-26.3% 501281 ? 30% numa-vmstat.node0.numa_other
78947 ? 25% numa-vmstat.node1.nr_active_anon
16.00 ?100% numa-vmstat.node1.nr_active_file
+10.6% 829613 ? 9% numa-vmstat.node1.nr_anon_pages
1601 ? 9% numa-vmstat.node1.nr_anon_transparent_hugepages
212803 ?114% numa-vmstat.node1.nr_file_pages
-0.7% 31726112 numa-vmstat.node1.nr_free_pages
+10.7% 837739 ? 9% numa-vmstat.node1.nr_inactive_anon
-0.8% 20.33 ? 93% numa-vmstat.node1.nr_inactive_file
27.17 ? 26% numa-vmstat.node1.nr_isolated_anon
+2.3% 19596 ? 8% numa-vmstat.node1.nr_kernel_stack
12401 ? 17% numa-vmstat.node1.nr_mapped
71.50 ?184% numa-vmstat.node1.nr_mlock
2433 ? 27% numa-vmstat.node1.nr_page_table_pages
86996 ? 24% numa-vmstat.node1.nr_shmem
+5.9% 17505 ? 36% numa-vmstat.node1.nr_slab_reclaimable
+6.9% 51428 ? 5% numa-vmstat.node1.nr_slab_unreclaimable
125781 ?189% numa-vmstat.node1.nr_unevictable
78946 ? 25% numa-vmstat.node1.nr_zone_active_anon
16.00 ?100% numa-vmstat.node1.nr_zone_active_file
+10.7% 837741 ? 9% numa-vmstat.node1.nr_zone_inactive_anon
-0.8% 20.33 ? 93% numa-vmstat.node1.nr_zone_inactive_file
125781 ?189% numa-vmstat.node1.nr_zone_unevictable
+15.4% 2811280 ? 13% numa-vmstat.node1.numa_hit
0.00 numa-vmstat.node1.numa_interleave
+11.2% 2100996 ? 14% numa-vmstat.node1.numa_local
+29.9% 710290 ? 23% numa-vmstat.node1.numa_other
-70.9% 336.83 ? 15% proc-vmstat.direct_map_level2_splits
6.00 ? 19% proc-vmstat.direct_map_level3_splits
87655 ? 3% proc-vmstat.nr_active_anon
32.00 proc-vmstat.nr_active_file
+0.5% 1570691 proc-vmstat.nr_anon_pages
3033 proc-vmstat.nr_anon_transparent_hugepages
-0.0% 6270545 proc-vmstat.nr_dirty_background_threshold
-0.0% 12556422 proc-vmstat.nr_dirty_threshold
+4.0% 802849 proc-vmstat.nr_file_pages
-0.0% 63082004 proc-vmstat.nr_free_pages
+0.6% 1581204 proc-vmstat.nr_inactive_anon
+4.7% 40.83 proc-vmstat.nr_inactive_file
38.17 ? 15% proc-vmstat.nr_isolated_anon
+0.1% 39748 proc-vmstat.nr_kernel_stack
24764 ? 14% proc-vmstat.nr_mapped
+4.4% 418.33 proc-vmstat.nr_mlock
4199 proc-vmstat.nr_page_table_pages
98129 proc-vmstat.nr_shmem
-3.1% 40539 proc-vmstat.nr_slab_reclaimable
-0.4% 107079 proc-vmstat.nr_slab_unreclaimable
+0.1% 704707 proc-vmstat.nr_unevictable
87655 ? 3% proc-vmstat.nr_zone_active_anon
32.00 proc-vmstat.nr_zone_active_file
+0.6% 1581204 proc-vmstat.nr_zone_inactive_anon
+4.7% 40.83 proc-vmstat.nr_zone_inactive_file
+0.1% 704707 proc-vmstat.nr_zone_unevictable
-18.2% 164021 ? 7% proc-vmstat.numa_hint_faults
-10.8% 107425 ? 10% proc-vmstat.numa_hint_faults_local
+2.4% 5152018 proc-vmstat.numa_hit
60803 ? 4% proc-vmstat.numa_huge_pte_updates
0.00 proc-vmstat.numa_interleave
+3.6% 3940440 proc-vmstat.numa_local
-1.3% 1211572 ? 2% proc-vmstat.numa_other
-40.8% 7027448 ? 2% proc-vmstat.numa_pages_migrated
-45.9% 31298933 ? 4% proc-vmstat.numa_pte_updates
82160 ? 18% proc-vmstat.pgactivate
0.00 proc-vmstat.pgalloc_dma32
-1.7% 4.462e+08 proc-vmstat.pgalloc_normal
+3.1% 5060927 proc-vmstat.pgfault
-1.7% 4.461e+08 proc-vmstat.pgfree
4.67 ? 91% proc-vmstat.pgmigrate_fail
-40.8% 7027448 ? 2% proc-vmstat.pgmigrate_success
1426 ?100% proc-vmstat.pgpgin
+8.8% 188160 ? 2% proc-vmstat.pgreuse
+24.8% 123.17 ? 9% proc-vmstat.thp_collapse_alloc
+0.0% 14.67 ? 28% proc-vmstat.thp_deferred_split_page
+0.0% 26654 proc-vmstat.thp_fault_alloc
0.00 proc-vmstat.thp_migration_fail
13682 ? 2% proc-vmstat.thp_migration_success
+0.0% 14.67 ? 28% proc-vmstat.thp_split_pmd
0.00 proc-vmstat.thp_zero_page_alloc
-12.2% 883.33 proc-vmstat.unevictable_pgs_culled
-0.1% 606.67 proc-vmstat.unevictable_pgs_mlocked
3.67 ? 20% proc-vmstat.unevictable_pgs_munlocked
3.33 ? 14% proc-vmstat.unevictable_pgs_rescued
+8.8% 8273408 ? 3% proc-vmstat.unevictable_pgs_scanned
0.00 proc-vmstat.vma_lock_abort
0.00 proc-vmstat.vma_lock_retry
0.00 proc-vmstat.vma_lock_success
39.24 ? 5% perf-stat.i.MPKI
-4.1% 3.877e+08 perf-stat.i.branch-instructions
0.61 perf-stat.i.branch-miss-rate%
-2.6% 1752689 perf-stat.i.branch-misses
26.50 ? 2% perf-stat.i.cache-miss-rate%
-27.3% 38359610 ? 4% perf-stat.i.cache-misses
-20.3% 93439529 ? 2% perf-stat.i.cache-references
2116 perf-stat.i.context-switches
-2.3% 529.18 perf-stat.i.cpi
-0.0% 224149 perf-stat.i.cpu-clock
-1.7% 5.954e+11 perf-stat.i.cpu-cycles
-1.7% 229.68 perf-stat.i.cpu-migrations
+16.9% 110768 ? 3% perf-stat.i.cycles-between-cache-misses
0.10 ? 8% perf-stat.i.dTLB-load-miss-rate%
+51.5% 339894 ? 8% perf-stat.i.dTLB-load-misses
-3.1% 4.795e+08 perf-stat.i.dTLB-loads
0.58 perf-stat.i.dTLB-store-miss-rate%
-1.8% 1022937 perf-stat.i.dTLB-store-misses
+0.6% 1.844e+08 perf-stat.i.dTLB-stores
-3.3% 1.83e+09 perf-stat.i.instructions
0.00 perf-stat.i.ipc
0.09 ? 12% perf-stat.i.major-faults
2.66 perf-stat.i.metric.GHz
1526 perf-stat.i.metric.K/sec
3.56 perf-stat.i.metric.M/sec
4834 ? 2% perf-stat.i.minor-faults
47.29 perf-stat.i.node-load-miss-rate%
-25.0% 724436 ? 3% perf-stat.i.node-load-misses
-2.0% 725851 ? 9% perf-stat.i.node-loads
4835 ? 2% perf-stat.i.page-faults
-0.0% 224149 perf-stat.i.task-clock
48.98 perf-stat.overall.MPKI
0.46 perf-stat.overall.branch-miss-rate%
40.39 ? 2% perf-stat.overall.cache-miss-rate%
+1.9% 338.53 perf-stat.overall.cpi
17135 ? 4% perf-stat.overall.cycles-between-cache-misses
0.07 ? 7% perf-stat.overall.dTLB-load-miss-rate%
0.56 perf-stat.overall.dTLB-store-miss-rate%
0.00 perf-stat.overall.ipc
48.39 ? 4% perf-stat.overall.node-load-miss-rate%
-4.4% 3.726e+08 perf-stat.ps.branch-instructions
-2.7% 1713424 perf-stat.ps.branch-misses
-27.6% 34960590 ? 4% perf-stat.ps.cache-misses
-20.5% 86529746 perf-stat.ps.cache-references
2105 perf-stat.ps.context-switches
+0.0% 223765 perf-stat.ps.cpu-clock
-1.8% 5.98e+11 perf-stat.ps.cpu-cycles
-1.6% 226.70 perf-stat.ps.cpu-migrations
+49.2% 333910 ? 7% perf-stat.ps.dTLB-load-misses
-3.4% 4.638e+08 perf-stat.ps.dTLB-loads
-2.0% 1020849 perf-stat.ps.dTLB-store-misses
+0.6% 1.823e+08 perf-stat.ps.dTLB-stores
-3.6% 1.767e+09 perf-stat.ps.instructions
0.08 ? 13% perf-stat.ps.major-faults
4608 ? 2% perf-stat.ps.minor-faults
-25.0% 728715 ? 5% perf-stat.ps.node-load-misses
-1.8% 781081 ? 10% perf-stat.ps.node-loads
4608 ? 2% perf-stat.ps.page-faults
+0.0% 223765 perf-stat.ps.task-clock
+5.2% 1.873e+12 perf-stat.total.instructions
36117 ?135% sched_debug.cfs_rq:/.MIN_vruntime.avg
+15.2% 7664854 ?135% sched_debug.cfs_rq:/.MIN_vruntime.max
0.00 sched_debug.cfs_rq:/.MIN_vruntime.min
+14.7% 524024 ?135% sched_debug.cfs_rq:/.MIN_vruntime.stddev
0.91 ? 5% sched_debug.cfs_rq:/.h_nr_running.avg
1.78 ? 3% sched_debug.cfs_rq:/.h_nr_running.max
0.64 ? 20% sched_debug.cfs_rq:/.h_nr_running.min
0.15 ? 24% sched_debug.cfs_rq:/.h_nr_running.stddev
5483 ? 13% sched_debug.cfs_rq:/.load.avg
-14.4% 214489 ? 49% sched_debug.cfs_rq:/.load.max
1420 ? 20% sched_debug.cfs_rq:/.load.min
15867 ? 46% sched_debug.cfs_rq:/.load.stddev
7.22 ? 12% sched_debug.cfs_rq:/.load_avg.avg
+3.0% 221.67 ? 22% sched_debug.cfs_rq:/.load_avg.max
1.41 ? 21% sched_debug.cfs_rq:/.load_avg.min
+1.2% 22.46 ? 13% sched_debug.cfs_rq:/.load_avg.stddev
36117 ?135% sched_debug.cfs_rq:/.max_vruntime.avg
+15.2% 7664854 ?135% sched_debug.cfs_rq:/.max_vruntime.max
0.00 sched_debug.cfs_rq:/.max_vruntime.min
+14.7% 524024 ?135% sched_debug.cfs_rq:/.max_vruntime.stddev
+11.8% 1.345e+08 ? 2% sched_debug.cfs_rq:/.min_vruntime.avg
+12.5% 1.385e+08 ? 2% sched_debug.cfs_rq:/.min_vruntime.max
+9.4% 1.087e+08 ? 4% sched_debug.cfs_rq:/.min_vruntime.min
+32.7% 3357930 ? 15% sched_debug.cfs_rq:/.min_vruntime.stddev
0.90 ? 5% sched_debug.cfs_rq:/.nr_running.avg
1.06 ? 6% sched_debug.cfs_rq:/.nr_running.max
0.64 ? 20% sched_debug.cfs_rq:/.nr_running.min
0.09 ? 51% sched_debug.cfs_rq:/.nr_running.stddev
1.59 ? 64% sched_debug.cfs_rq:/.removed.load_avg.avg
-0.4% 76.20 ? 34% sched_debug.cfs_rq:/.removed.load_avg.max
9.79 ? 26% sched_debug.cfs_rq:/.removed.load_avg.stddev
0.76 ? 65% sched_debug.cfs_rq:/.removed.runnable_avg.avg
+1.1% 37.52 ? 31% sched_debug.cfs_rq:/.removed.runnable_avg.max
4.70 ? 28% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
0.76 ? 66% sched_debug.cfs_rq:/.removed.util_avg.avg
+1.5% 35.83 ? 30% sched_debug.cfs_rq:/.removed.util_avg.max
4.59 ? 30% sched_debug.cfs_rq:/.removed.util_avg.stddev
-1.1% 934.57 ? 5% sched_debug.cfs_rq:/.runnable_avg.avg
1755 ? 6% sched_debug.cfs_rq:/.runnable_avg.max
-3.7% 619.05 ? 23% sched_debug.cfs_rq:/.runnable_avg.min
+3.3% 146.48 ? 25% sched_debug.cfs_rq:/.runnable_avg.stddev
0.00 ?153% sched_debug.cfs_rq:/.spread.avg
0.68 ?153% sched_debug.cfs_rq:/.spread.max
0.05 ?153% sched_debug.cfs_rq:/.spread.stddev
+77.5% 15821663 ? 33% sched_debug.cfs_rq:/.spread0.avg
+70.0% 19782673 ? 27% sched_debug.cfs_rq:/.spread0.max
-17.2% -10050665 sched_debug.cfs_rq:/.spread0.min
+33.3% 3345922 ? 15% sched_debug.cfs_rq:/.spread0.stddev
-1.1% 924.53 ? 5% sched_debug.cfs_rq:/.util_avg.avg
1198 ? 2% sched_debug.cfs_rq:/.util_avg.max
-5.7% 485.68 ? 22% sched_debug.cfs_rq:/.util_avg.min
+10.5% 105.53 ? 40% sched_debug.cfs_rq:/.util_avg.stddev
-21.8% 345.05 ? 11% sched_debug.cfs_rq:/.util_est_enqueued.avg
1135 ? 9% sched_debug.cfs_rq:/.util_est_enqueued.max
+14.2% 319.98 ? 9% sched_debug.cfs_rq:/.util_est_enqueued.stddev
+22.3% 1600451 ? 3% sched_debug.cpu.avg_idle.avg
+10.5% 3230277 ? 5% sched_debug.cpu.avg_idle.max
+36.4% 611757 ? 13% sched_debug.cpu.avg_idle.min
+38.1% 569757 ? 7% sched_debug.cpu.avg_idle.stddev
+20.0% 606292 ? 5% sched_debug.cpu.clock.avg
+20.0% 606955 ? 5% sched_debug.cpu.clock.max
+20.1% 605556 ? 5% sched_debug.cpu.clock.min
-4.3% 397.71 ? 28% sched_debug.cpu.clock.stddev
+19.8% 598837 ? 5% sched_debug.cpu.clock_task.avg
+19.9% 600703 ? 5% sched_debug.cpu.clock_task.max
+12.9% 540404 ? 2% sched_debug.cpu.clock_task.min
4053 ? 73% sched_debug.cpu.clock_task.stddev
+5.7% 17999 ? 7% sched_debug.cpu.curr->pid.avg
+7.9% 21502 sched_debug.cpu.curr->pid.max
+3.7% 10302 ? 27% sched_debug.cpu.curr->pid.min
1861 ? 55% sched_debug.cpu.curr->pid.stddev
+13.8% 673677 ? 4% sched_debug.cpu.max_idle_balance_cost.avg
+12.9% 1400895 ? 4% sched_debug.cpu.max_idle_balance_cost.max
+0.0% 500000 sched_debug.cpu.max_idle_balance_cost.min
+41.5% 202328 ? 5% sched_debug.cpu.max_idle_balance_cost.stddev
4295 sched_debug.cpu.next_balance.avg
4295 sched_debug.cpu.next_balance.max
4295 sched_debug.cpu.next_balance.min
0.00 ? 27% sched_debug.cpu.next_balance.stddev
0.89 ? 6% sched_debug.cpu.nr_running.avg
1.82 ? 2% sched_debug.cpu.nr_running.max
0.64 ? 19% sched_debug.cpu.nr_running.min
0.16 ? 26% sched_debug.cpu.nr_running.stddev
6118 sched_debug.cpu.nr_switches.avg
56608 ? 24% sched_debug.cpu.nr_switches.max
2301 ? 2% sched_debug.cpu.nr_switches.min
5769 ? 10% sched_debug.cpu.nr_switches.stddev
0.00 ? 76% sched_debug.cpu.nr_uninterruptible.avg
46.24 ? 39% sched_debug.cpu.nr_uninterruptible.max
-63.50 sched_debug.cpu.nr_uninterruptible.min
8.63 ? 48% sched_debug.cpu.nr_uninterruptible.stddev
+20.1% 605544 ? 5% sched_debug.cpu_clk
+0.0% 996147 sched_debug.dl_rq:.dl_bw->bw.avg
+0.0% 996147 sched_debug.dl_rq:.dl_bw->bw.max
+0.0% 996147 sched_debug.dl_rq:.dl_bw->bw.min
+0.0% 4.295e+09 sched_debug.jiffies
+20.9% 604335 ? 5% sched_debug.ktime
0.00 ? 2% sched_debug.rt_rq:.rt_nr_migratory.avg
0.11 ? 2% sched_debug.rt_rq:.rt_nr_migratory.max
0.01 ? 2% sched_debug.rt_rq:.rt_nr_migratory.stddev
0.00 ? 2% sched_debug.rt_rq:.rt_nr_running.avg
0.11 ? 2% sched_debug.rt_rq:.rt_nr_running.max
0.01 ? 2% sched_debug.rt_rq:.rt_nr_running.stddev
+0.0% 950.00 sched_debug.rt_rq:.rt_runtime.avg
+0.0% 950.00 sched_debug.rt_rq:.rt_runtime.max
+0.0% 950.00 sched_debug.rt_rq:.rt_runtime.min
0.00 ?223% sched_debug.rt_rq:.rt_throttled.avg
0.01 ?223% sched_debug.rt_rq:.rt_throttled.max
0.00 ?223% sched_debug.rt_rq:.rt_throttled.stddev
0.31 ? 35% sched_debug.rt_rq:.rt_time.avg
70.39 ? 35% sched_debug.rt_rq:.rt_time.max
4.69 ? 35% sched_debug.rt_rq:.rt_time.stddev
+11.1% 561614 ? 2% sched_debug.sched_clk
1.00 sched_debug.sched_clock_stable()
+0.0% 58611259 sched_debug.sysctl_sched.sysctl_sched_features
0.75 sched_debug.sysctl_sched.sysctl_sched_idle_min_granularity
+0.0% 24.00 sched_debug.sysctl_sched.sysctl_sched_latency
3.00 sched_debug.sysctl_sched.sysctl_sched_min_granularity
1.00 sched_debug.sysctl_sched.sysctl_sched_tunable_scaling
4.00 sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity
3.38 ? 66% perf-profile.calltrace.cycles-pp.asm_exc_page_fault
3.27 ? 68% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
3.25 ? 69% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
3.08 ? 71% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
3.17 ? 67% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
1.06 ?191% perf-profile.calltrace.cycles-pp.do_huge_pmd_numa_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
1.01 ?192% perf-profile.calltrace.cycles-pp.migrate_misplaced_page.do_huge_pmd_numa_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
1.01 ?192% perf-profile.calltrace.cycles-pp.migrate_pages.migrate_misplaced_page.do_huge_pmd_numa_page.__handle_mm_fault.handle_mm_fault
1.01 ?192% perf-profile.calltrace.cycles-pp.migrate_pages_batch.migrate_pages.migrate_misplaced_page.do_huge_pmd_numa_page.__handle_mm_fault
0.93 ?191% perf-profile.calltrace.cycles-pp.move_to_new_folio.migrate_pages_batch.migrate_pages.migrate_misplaced_page.do_huge_pmd_numa_page
0.93 ?191% perf-profile.calltrace.cycles-pp.migrate_folio_extra.move_to_new_folio.migrate_pages_batch.migrate_pages.migrate_misplaced_page
0.93 ?191% perf-profile.calltrace.cycles-pp.folio_copy.migrate_folio_extra.move_to_new_folio.migrate_pages_batch.migrate_pages
0.92 ?191% perf-profile.calltrace.cycles-pp.copy_page.folio_copy.migrate_folio_extra.move_to_new_folio.migrate_pages_batch
0.88 ?223% perf-profile.calltrace.cycles-pp.perf_session__process_user_event.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record
0.88 ?223% perf-profile.calltrace.cycles-pp.__ordered_events__flush.perf_session__process_user_event.reader__read_event.perf_session__process_events.record__finish_output
0.71 ?223% perf-profile.calltrace.cycles-pp.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event.reader__read_event.perf_session__process_events
0.00 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copy_page_from_iter_atomic.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
0.55 ?223% perf-profile.calltrace.cycles-pp.evlist__parse_sample.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event.reader__read_event
0.00 perf-profile.calltrace.cycles-pp.drm_fbdev_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread
0.14 ?223% perf-profile.calltrace.cycles-pp.build_id__mark_dso_hit.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event.reader__read_event
0.12 ?223% perf-profile.calltrace.cycles-pp.evsel__parse_sample.evlist__parse_sample.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event
0.26 ?142% perf-profile.calltrace.cycles-pp.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll.__cmd_record
0.26 ?142% perf-profile.calltrace.cycles-pp.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll
0.28 ?142% perf-profile.calltrace.cycles-pp.__poll.__cmd_record.cmd_record.run_builtin.main
0.27 ?142% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__poll.__cmd_record.cmd_record.run_builtin
0.27 ?142% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll.__cmd_record.cmd_record
7.09 ? 57% perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.main.__libc_start_main
7.09 ? 57% perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.main.__libc_start_main
0.00 perf-profile.calltrace.cycles-pp.__fxstat64
0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__fxstat64
0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__fxstat64
1.41 ? 13% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
0.21 ?144% perf-profile.calltrace.cycles-pp.do_poll.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.thread__find_map.build_id__mark_dso_hit.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event
0.00 perf-profile.calltrace.cycles-pp.drm_atomic_helper_dirtyfb.drm_fbdev_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread
0.00 perf-profile.calltrace.cycles-pp.map__load.thread__find_map.build_id__mark_dso_hit.perf_session__deliver_event.__ordered_events__flush
0.00 perf-profile.calltrace.cycles-pp.dso__load.map__load.thread__find_map.build_id__mark_dso_hit.perf_session__deliver_event
1.07 ?103% perf-profile.calltrace.cycles-pp.cmd_stat
1.07 ?103% perf-profile.calltrace.cycles-pp.dispatch_events.cmd_stat
1.06 ?103% perf-profile.calltrace.cycles-pp.process_interval.dispatch_events.cmd_stat
0.00 perf-profile.calltrace.cycles-pp.__dso__load_kallsyms.dso__load.map__load.thread__find_map.build_id__mark_dso_hit
5.00 ? 58% perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_fb_dirty.drm_fb_helper_damage_work.process_one_work
4.96 ? 58% perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
5.04 ? 58% perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.11 ? 58% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
1.01 ?104% perf-profile.calltrace.cycles-pp.read_counters.process_interval.dispatch_events.cmd_stat
5.16 ? 58% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen
5.20 ? 58% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn
0.19 ?223% perf-profile.calltrace.cycles-pp.evlist__parse_sample_timestamp.process_simple.reader__read_event.perf_session__process_events.record__finish_output
5.20 ? 58% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn.perf_mmap__push
5.24 ? 58% perf-profile.calltrace.cycles-pp.__libc_write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
1.58 ? 21% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
0.41 ? 71% perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
1.08 ? 15% perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork
0.00 perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_fb_dirty.drm_fb_helper_damage_work
0.00 perf-profile.calltrace.cycles-pp.__do_sys_newfstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__fxstat64
0.00 perf-profile.calltrace.cycles-pp.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
0.00 perf-profile.calltrace.cycles-pp.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work
0.00 perf-profile.calltrace.cycles-pp._printk.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work
0.00 perf-profile.calltrace.cycles-pp.vprintk_emit._printk.irq_work_single.irq_work_run_list.irq_work_run
0.00 perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit._printk.irq_work_single.irq_work_run_list
0.00 perf-profile.calltrace.cycles-pp.console_flush_all.console_unlock.vprintk_emit._printk.irq_work_single
0.00 perf-profile.calltrace.cycles-pp.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit._printk
0.13 ?223% perf-profile.calltrace.cycles-pp.machine__findnew_thread.build_id__mark_dso_hit.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event
0.00 perf-profile.calltrace.cycles-pp.sched_setaffinity.__x64_sys_sched_setaffinity.do_syscall_64.entry_SYSCALL_64_after_hwframe.sched_setaffinity
0.00 perf-profile.calltrace.cycles-pp.cpu_stopper_thread.smpboot_thread_fn.kthread.ret_from_fork
0.77 ? 78% perf-profile.calltrace.cycles-pp.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
0.00 perf-profile.calltrace.cycles-pp.__sched_setaffinity.sched_setaffinity.__x64_sys_sched_setaffinity.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.migration_cpu_stop.cpu_stopper_thread.smpboot_thread_fn.kthread.ret_from_fork
0.79 ? 78% perf-profile.calltrace.cycles-pp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write
5.50 ? 61% perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record
0.10 ?223% perf-profile.calltrace.cycles-pp.perf_poll.do_poll.do_sys_poll.__x64_sys_poll.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.kallsyms__parse.__dso__load_kallsyms.dso__load.map__load.thread__find_map
0.00 perf-profile.calltrace.cycles-pp.evsel__parse_sample_timestamp.process_simple.reader__read_event.perf_session__process_events.record__finish_output
0.58 ? 45% perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat
0.20 ?141% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
5.65 ? 61% perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
0.20 ?141% perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
2.60 ? 21% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt
0.00 perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.09 ?223% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sched_setaffinity.evlist_cpu_iterator__next.read_counters.process_interval
0.09 ?223% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sched_setaffinity.evlist_cpu_iterator__next.read_counters
0.09 ?223% perf-profile.calltrace.cycles-pp.__x64_sys_sched_setaffinity.do_syscall_64.entry_SYSCALL_64_after_hwframe.sched_setaffinity.evlist_cpu_iterator__next
0.00 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function
0.00 perf-profile.calltrace.cycles-pp.perf_session__delete.__cmd_record
2.84 ? 21% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
0.32 ?103% perf-profile.calltrace.cycles-pp.__percpu_counter_init.mm_init.alloc_bprm.do_execveat_common.__x64_sys_execve
0.00 perf-profile.calltrace.cycles-pp.drm_gem_vmap_unlocked.drm_client_buffer_vmap.drm_fbdev_fb_dirty.drm_fb_helper_damage_work.process_one_work
0.00 perf-profile.calltrace.cycles-pp.drm_gem_vmap.drm_gem_vmap_unlocked.drm_client_buffer_vmap.drm_fbdev_fb_dirty.drm_fb_helper_damage_work
0.00 perf-profile.calltrace.cycles-pp.drm_gem_shmem_vmap.drm_gem_vmap.drm_gem_vmap_unlocked.drm_client_buffer_vmap.drm_fbdev_fb_dirty
0.00 perf-profile.calltrace.cycles-pp.drm_client_buffer_vmap.drm_fbdev_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread
0.00 perf-profile.calltrace.cycles-pp.__open64
0.36 ? 70% perf-profile.calltrace.cycles-pp.sched_setaffinity.evlist_cpu_iterator__next.read_counters.process_interval.dispatch_events
0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__open64
0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64
2.05 ? 57% perf-profile.calltrace.cycles-pp.copy_page_from_iter_atomic.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write
0.00 perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64
0.00 perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64
0.70 ? 18% perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
0.09 ?223% perf-profile.calltrace.cycles-pp.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
0.09 ?223% perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
0.08 ?223% perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__do_softirq.run_ksoftirqd.smpboot_thread_fn
0.08 ?223% perf-profile.calltrace.cycles-pp.rcu_core.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread
0.00 perf-profile.calltrace.cycles-pp.cmd_record
0.00 perf-profile.calltrace.cycles-pp.evlist__delete.cmd_record
0.00 perf-profile.calltrace.cycles-pp.migrate_folio_unmap.migrate_pages_batch.migrate_pages.migrate_misplaced_page.do_huge_pmd_numa_page
1.05 ? 19% perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
0.19 ?141% perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
1.06 ? 20% perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group
0.82 ? 77% perf-profile.calltrace.cycles-pp.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable.generic_perform_write.__generic_file_write_iter
1.09 ? 20% perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.io__get_hex.kallsyms__parse.__dso__load_kallsyms.dso__load.map__load
0.00 perf-profile.calltrace.cycles-pp.__ordered_events__flush.perf_session__process_events.record__finish_output.__cmd_record
0.08 ?223% perf-profile.calltrace.cycles-pp.perf_evsel__read.read_counters.process_interval.dispatch_events.cmd_stat
0.00 perf-profile.calltrace.cycles-pp.select_task_rq_fair.wake_up_new_task.kernel_clone.__x64_sys_vfork.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.find_idlest_cpu.select_task_rq_fair.wake_up_new_task.kernel_clone.__x64_sys_vfork
0.17 ?141% perf-profile.calltrace.cycles-pp.__close_nocancel
6.52 ? 62% perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin.main
0.00 perf-profile.calltrace.cycles-pp.rmap_walk_anon.try_to_migrate.migrate_folio_unmap.migrate_pages_batch.migrate_pages
0.00 perf-profile.calltrace.cycles-pp.try_to_migrate.migrate_folio_unmap.migrate_pages_batch.migrate_pages.migrate_misplaced_page
0.00 perf-profile.calltrace.cycles-pp.try_to_migrate_one.rmap_walk_anon.try_to_migrate.migrate_folio_unmap.migrate_pages_batch
0.41 ? 72% perf-profile.calltrace.cycles-pp.vmstat_start.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read
6.24 ? 62% perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin
0.09 ?223% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close_nocancel
0.00 perf-profile.calltrace.cycles-pp.perf_session__do_write_header.__cmd_record
0.09 ?223% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close_nocancel
0.00 perf-profile.calltrace.cycles-pp.find_idlest_group.find_idlest_cpu.select_task_rq_fair.wake_up_new_task.kernel_clone
0.00 perf-profile.calltrace.cycles-pp.update_sg_wakeup_stats.find_idlest_group.find_idlest_cpu.select_task_rq_fair.wake_up_new_task
0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work
0.00 perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work
0.00 perf-profile.calltrace.cycles-pp.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
0.00 perf-profile.calltrace.cycles-pp.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read.io__get_hex.kallsyms__parse.__dso__load_kallsyms
0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read.io__get_hex.kallsyms__parse
0.00 perf-profile.calltrace.cycles-pp.__libc_read.io__get_hex.kallsyms__parse.__dso__load_kallsyms.dso__load
0.70 ? 79% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable
0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read.io__get_hex
0.00 perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
0.00 perf-profile.calltrace.cycles-pp.dup_mmap.dup_mm.copy_process.kernel_clone.__do_sys_clone
0.27 ?100% perf-profile.calltrace.cycles-pp.walk_component.link_path_walk.path_openat.do_filp_open.do_sys_openat2
1.18 ? 9% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
0.00 perf-profile.calltrace.cycles-pp.machine__exit.perf_session__delete.__cmd_record
0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
0.00 perf-profile.calltrace.cycles-pp.readn.perf_evsel__read.read_counters.process_interval.dispatch_events
0.00 perf-profile.calltrace.cycles-pp.__libc_read.readn.perf_evsel__read.read_counters.process_interval
0.00 perf-profile.calltrace.cycles-pp.wait_for_lsr.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit
2.93 ? 16% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.seq_read_iter.seq_read.proc_reg_read.vfs_read.ksys_read
0.00 perf-profile.calltrace.cycles-pp.seq_read.proc_reg_read.vfs_read.ksys_read.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.proc_reg_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read.readn.perf_evsel__read.read_counters
0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read.readn.perf_evsel__read
0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read.readn
0.27 ?100% perf-profile.calltrace.cycles-pp.lookup_fast.walk_component.link_path_walk.path_openat.do_filp_open
0.00 perf-profile.calltrace.cycles-pp.dso__delete.machine__exit.perf_session__delete.__cmd_record
0.00 perf-profile.calltrace.cycles-pp.gather_pte_stats.walk_pmd_range.walk_pud_range.walk_p4d_range.walk_pgd_range
0.00 perf-profile.calltrace.cycles-pp.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
0.00 perf-profile.calltrace.cycles-pp.evlist__close.evlist__delete.cmd_record
0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault
0.00 perf-profile.calltrace.cycles-pp.schedule.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault
0.00 perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare
0.00 perf-profile.calltrace.cycles-pp.io_schedule.migration_entry_wait_on_locked.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
0.00 perf-profile.calltrace.cycles-pp.commit_tail.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_fb_dirty
0.00 perf-profile.calltrace.cycles-pp.ast_mode_config_helper_atomic_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb
0.44 ? 75% perf-profile.calltrace.cycles-pp.seq_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.machine__process_fork_event.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event.reader__read_event
0.00 perf-profile.calltrace.cycles-pp.perf_session__deliver_event.__ordered_events__flush.perf_session__process_events.record__finish_output.__cmd_record
0.00 perf-profile.calltrace.cycles-pp.set_pmd_migration_entry.try_to_migrate_one.rmap_walk_anon.try_to_migrate.migrate_folio_unmap
0.00 perf-profile.calltrace.cycles-pp.rb_erase.symbols__delete.dso__delete.machine__exit.perf_session__delete
0.00 perf-profile.calltrace.cycles-pp.symbols__delete.dso__delete.machine__exit.perf_session__delete.__cmd_record
0.00 perf-profile.calltrace.cycles-pp.next_uptodate_page.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
0.00 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.io_schedule.migration_entry_wait_on_locked
0.00 perf-profile.calltrace.cycles-pp.proc_fill_cache.proc_pid_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.setlocale
0.00 perf-profile.calltrace.cycles-pp.perf_evsel__close_cpu.evlist__close.evlist__delete.cmd_record
0.00 perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.io_schedule
0.00 perf-profile.calltrace.cycles-pp.drm_fbdev_damage_blit_real.drm_fbdev_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread
0.00 perf-profile.calltrace.cycles-pp.pid_revalidate.lookup_fast.walk_component.link_path_walk.path_openat
0.10 ?223% perf-profile.calltrace.cycles-pp.__collapse_huge_page_copy.collapse_huge_page.hpage_collapse_scan_pmd.khugepaged_scan_mm_slot.khugepaged
0.00 perf-profile.calltrace.cycles-pp.__mmdrop.finish_task_switch.__schedule.schedule.exit_to_user_mode_loop
0.00 perf-profile.calltrace.cycles-pp.memcpy_erms.drm_fbdev_damage_blit_real.drm_fbdev_fb_dirty.drm_fb_helper_damage_work.process_one_work
0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close.perf_evsel__close_cpu.evlist__close.evlist__delete
0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close.perf_evsel__close_cpu.evlist__close
0.00 perf-profile.calltrace.cycles-pp.__close.perf_evsel__close_cpu.evlist__close.evlist__delete.cmd_record
0.00 perf-profile.calltrace.cycles-pp.proc_pid_cmdline_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.pmdp_invalidate.set_pmd_migration_entry.try_to_migrate_one.rmap_walk_anon
0.00 perf-profile.calltrace.cycles-pp.pmdp_invalidate.set_pmd_migration_entry.try_to_migrate_one.rmap_walk_anon.try_to_migrate
0.74 ? 78% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable.generic_perform_write
0.00 perf-profile.calltrace.cycles-pp.write_cache.perf_session__do_write_header.__cmd_record
0.00 perf-profile.calltrace.cycles-pp.get_mm_cmdline.proc_pid_cmdline_read.vfs_read.ksys_read.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.dispatch_events
0.00 perf-profile.calltrace.cycles-pp.process_interval.dispatch_events
0.00 perf-profile.calltrace.cycles-pp.smp_call_function_many_cond.on_each_cpu_cond_mask.flush_tlb_mm_range.pmdp_invalidate.set_pmd_migration_entry
0.00 perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.flush_tlb_mm_range.pmdp_invalidate.set_pmd_migration_entry.try_to_migrate_one
0.00 perf-profile.calltrace.cycles-pp.vfs_fstat.__do_sys_newfstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__fxstat64
0.00 perf-profile.calltrace.cycles-pp.all_vm_events.vmstat_start.seq_read_iter.proc_reg_read_iter.vfs_read
0.00 perf-profile.calltrace.cycles-pp.machine__delete_threads.perf_session__delete.__cmd_record
0.00 perf-profile.calltrace.cycles-pp.sysvec_call_function.asm_sysvec_call_function
0.00 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close.perf_evsel__close_cpu
0.00 perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
0.00 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close_nocancel
0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close_nocancel
0.00 perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail.commit_tail.drm_atomic_helper_commit.drm_atomic_commit
0.00 perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail.commit_tail.drm_atomic_helper_commit
0.00 perf-profile.calltrace.cycles-pp.ast_primary_plane_helper_atomic_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail.commit_tail
0.00 perf-profile.calltrace.cycles-pp.drm_fb_memcpy.ast_primary_plane_helper_atomic_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm.ast_mode_config_helper_atomic_commit_tail
0.00 perf-profile.calltrace.cycles-pp.memcpy_toio.drm_fb_memcpy.ast_primary_plane_helper_atomic_update.drm_atomic_helper_commit_planes.drm_atomic_helper_commit_tail_rpm
0.39 ? 72% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
0.00 perf-profile.calltrace.cycles-pp.thread__delete.machine__delete_threads.perf_session__delete.__cmd_record
0.00 perf-profile.calltrace.cycles-pp.read_counters.process_interval.dispatch_events
0.86 ? 21% perf-profile.calltrace.cycles-pp.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64
0.86 ? 21% perf-profile.calltrace.cycles-pp.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.wake_up_new_task.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.__folio_alloc.vma_alloc_folio.shmem_alloc_folio.shmem_alloc_and_acct_folio.shmem_get_folio_gfp
0.00 perf-profile.calltrace.cycles-pp._dl_addr
0.00 perf-profile.calltrace.cycles-pp.shmem_add_to_page_cache.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter
0.00 perf-profile.calltrace.cycles-pp.____machine__findnew_thread.machine__process_fork_event.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event
0.00 perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare
0.00 perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.exit_to_user_mode_loop
0.00 perf-profile.calltrace.cycles-pp.copy_page.__collapse_huge_page_copy.collapse_huge_page.hpage_collapse_scan_pmd.khugepaged_scan_mm_slot
0.00 perf-profile.calltrace.cycles-pp.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio.shmem_alloc_and_acct_folio
0.00 perf-profile.calltrace.cycles-pp.select_task_rq_fair.wake_up_new_task.kernel_clone.__do_sys_clone.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.find_idlest_cpu.select_task_rq_fair.wake_up_new_task.kernel_clone.__do_sys_clone
0.00 perf-profile.calltrace.cycles-pp.walk_component.path_lookupat.filename_lookup.vfs_statx.vfs_fstatat
0.00 perf-profile.calltrace.cycles-pp.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 perf-profile.calltrace.cycles-pp.rb_next.symbols__fixup_end.__dso__load_kallsyms.dso__load.map__load
0.00 perf-profile.calltrace.cycles-pp.rb_erase.__dso__load_kallsyms.dso__load.map__load.thread__find_map
0.00 perf-profile.calltrace.cycles-pp.symbols__fixup_end.__dso__load_kallsyms.dso__load.map__load.thread__find_map
0.00 perf-profile.calltrace.cycles-pp.__set_cpus_allowed_ptr.__sched_setaffinity.sched_setaffinity.__x64_sys_sched_setaffinity.do_syscall_64
0.00 perf-profile.calltrace.cycles-pp.__check_object_size.get_mm_cmdline.proc_pid_cmdline_read.vfs_read.ksys_read
0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work.__virt_addr_valid.__check_object_size.get_mm_cmdline.proc_pid_cmdline_read
0.00 perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work.__virt_addr_valid.__check_object_size.get_mm_cmdline
0.00 perf-profile.calltrace.cycles-pp.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.__virt_addr_valid.__check_object_size
0.00 perf-profile.calltrace.cycles-pp.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.__virt_addr_valid
0.00 perf-profile.calltrace.cycles-pp.__virt_addr_valid.__check_object_size.get_mm_cmdline.proc_pid_cmdline_read.vfs_read
0.00 perf-profile.calltrace.cycles-pp.diskstats_show.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read
0.00 perf-profile.calltrace.cycles-pp.move_queued_task.migration_cpu_stop.cpu_stopper_thread.smpboot_thread_fn.kthread
0.00 perf-profile.calltrace.cycles-pp.rb_free.rcu_do_batch.rcu_core.__do_softirq.run_ksoftirqd
0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work.queue_event.ordered_events__queue.process_simple.reader__read_event
0.00 perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work.queue_event.ordered_events__queue.process_simple
0.00 perf-profile.calltrace.cycles-pp.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.queue_event.ordered_events__queue
0.00 perf-profile.calltrace.cycles-pp.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.queue_event
0.00 perf-profile.calltrace.cycles-pp.task_dump_owner.pid_revalidate.lookup_fast.walk_component.link_path_walk
0.00 perf-profile.calltrace.cycles-pp.shmem_get_folio_gfp.shmem_read_mapping_page_gfp.drm_gem_get_pages.drm_gem_shmem_get_pages_locked.drm_gem_shmem_vmap
0.00 perf-profile.calltrace.cycles-pp.filemap_get_entry.shmem_get_folio_gfp.shmem_read_mapping_page_gfp.drm_gem_get_pages.drm_gem_shmem_get_pages_locked
0.00 perf-profile.calltrace.cycles-pp.drm_gem_shmem_get_pages_locked.drm_gem_shmem_vmap.drm_gem_vmap.drm_gem_vmap_unlocked.drm_client_buffer_vmap
0.00 perf-profile.calltrace.cycles-pp.drm_gem_get_pages.drm_gem_shmem_get_pages_locked.drm_gem_shmem_vmap.drm_gem_vmap.drm_gem_vmap_unlocked
0.00 perf-profile.calltrace.cycles-pp.shmem_read_mapping_page_gfp.drm_gem_get_pages.drm_gem_shmem_get_pages_locked.drm_gem_shmem_vmap.drm_gem_vmap
1.49 ? 12% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.49 ? 12% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.49 ? 12% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.42 ? 74% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.20 ?144% perf-profile.calltrace.cycles-pp.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter
0.63 ? 47% perf-profile.calltrace.cycles-pp.sched_exec.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
0.18 ?141% perf-profile.calltrace.cycles-pp.wake_up_new_task.kernel_clone.__x64_sys_vfork.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.79 ? 18% perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.vfs_statx.vfs_fstatat.__do_sys_newstat
0.61 ? 47% perf-profile.calltrace.cycles-pp.select_task_rq_fair.sched_exec.bprm_execve.do_execveat_common.__x64_sys_execve
2.84 ? 18% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
2.84 ? 18% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
0.09 ?223% perf-profile.calltrace.cycles-pp.pcpu_alloc.__percpu_counter_init.mm_init.alloc_bprm.do_execveat_common
0.80 ? 18% perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.vfs_fstatat.__do_sys_newstat.do_syscall_64
2.94 ? 18% perf-profile.calltrace.cycles-pp.open64
0.83 ? 17% perf-profile.calltrace.cycles-pp.vfs_statx.vfs_fstatat.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.88 ? 18% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64
2.87 ? 18% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
0.37 ? 71% perf-profile.calltrace.cycles-pp.show_numa_map.seq_read_iter.seq_read.vfs_read.ksys_read
0.29 ?100% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
0.09 ?223% perf-profile.calltrace.cycles-pp.step_into.path_lookupat.filename_lookup.vfs_statx.vfs_fstatat
0.09 ?223% perf-profile.calltrace.cycles-pp.shmem_alloc_folio.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write
0.94 ? 17% perf-profile.calltrace.cycles-pp.__xstat64
0.93 ? 17% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__xstat64
0.09 ?223% perf-profile.calltrace.cycles-pp.pick_link.step_into.path_lookupat.filename_lookup.vfs_statx
0.08 ?223% perf-profile.calltrace.cycles-pp.walk_p4d_range.walk_pgd_range.__walk_page_range.walk_page_vma.show_numa_map
0.08 ?223% perf-profile.calltrace.cycles-pp.walk_pud_range.walk_p4d_range.walk_pgd_range.__walk_page_range.walk_page_vma
0.92 ? 17% perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64
0.92 ? 17% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64
0.08 ?223% perf-profile.calltrace.cycles-pp.walk_pmd_range.walk_pud_range.walk_p4d_range.walk_pgd_range.__walk_page_range
0.91 ? 17% perf-profile.calltrace.cycles-pp.vfs_fstatat.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64
0.09 ?223% perf-profile.calltrace.cycles-pp.vma_alloc_folio.shmem_alloc_folio.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin
0.10 ?223% perf-profile.calltrace.cycles-pp.wp_page_copy.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
0.37 ?103% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.fault_in_readable
0.22 ?141% perf-profile.calltrace.cycles-pp.khugepaged.kthread.ret_from_fork
0.20 ?141% perf-profile.calltrace.cycles-pp.collapse_huge_page.hpage_collapse_scan_pmd.khugepaged_scan_mm_slot.khugepaged.kthread
1.56 ? 59% perf-profile.calltrace.cycles-pp.fault_in_iov_iter_readable.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write
0.21 ?141% perf-profile.calltrace.cycles-pp.hpage_collapse_scan_pmd.khugepaged_scan_mm_slot.khugepaged.kthread.ret_from_fork
0.21 ?141% perf-profile.calltrace.cycles-pp.khugepaged_scan_mm_slot.khugepaged.kthread.ret_from_fork
0.45 ? 71% perf-profile.calltrace.cycles-pp.lookup_fast.open_last_lookups.path_openat.do_filp_open.do_sys_openat2
5.80 ? 15% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
0.47 ? 71% perf-profile.calltrace.cycles-pp.open_last_lookups.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat
0.65 ? 7% perf-profile.calltrace.cycles-pp.evlist_cpu_iterator__next.read_counters.process_interval.dispatch_events.cmd_stat
5.75 ? 15% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
0.69 ? 47% perf-profile.calltrace.cycles-pp.kernel_clone.__x64_sys_vfork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__vfork
0.69 ? 47% perf-profile.calltrace.cycles-pp.__x64_sys_vfork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__vfork
1.47 ? 60% perf-profile.calltrace.cycles-pp.fault_in_readable.fault_in_iov_iter_readable.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
0.60 ? 47% perf-profile.calltrace.cycles-pp.find_idlest_cpu.select_task_rq_fair.sched_exec.bprm_execve.do_execveat_common
0.53 ? 44% perf-profile.calltrace.cycles-pp.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe.getdents64
0.12 ?223% perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork
0.54 ? 44% perf-profile.calltrace.cycles-pp.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe.getdents64
0.13 ?223% perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__mmdrop.finish_task_switch.__schedule.schedule
0.53 ? 44% perf-profile.calltrace.cycles-pp.proc_pid_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.54 ? 45% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.getdents64
0.54 ? 45% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.getdents64
0.54 ? 45% perf-profile.calltrace.cycles-pp.getdents64
0.95 ? 13% perf-profile.calltrace.cycles-pp.do_open.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat
0.60 ? 47% perf-profile.calltrace.cycles-pp.find_idlest_group.find_idlest_cpu.select_task_rq_fair.sched_exec.bprm_execve
3.38 ? 13% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
0.83 ? 20% perf-profile.calltrace.cycles-pp.__vfork
0.17 ?141% perf-profile.calltrace.cycles-pp.walk_page_vma.show_numa_map.seq_read_iter.seq_read.vfs_read
0.17 ?141% perf-profile.calltrace.cycles-pp.__walk_page_range.walk_page_vma.show_numa_map.seq_read_iter.seq_read
0.17 ?141% perf-profile.calltrace.cycles-pp.walk_pgd_range.__walk_page_range.walk_page_vma.show_numa_map.seq_read_iter
3.38 ? 13% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.17 ?141% perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exec_mmap.begin_new_exec.load_elf_binary
0.17 ?141% perf-profile.calltrace.cycles-pp.__mmput.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler
0.18 ?141% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
7.99 ? 51% perf-profile.calltrace.cycles-pp.__libc_start_main
7.98 ? 51% perf-profile.calltrace.cycles-pp.main.__libc_start_main
7.98 ? 51% perf-profile.calltrace.cycles-pp.run_builtin.main.__libc_start_main
3.60 ? 15% perf-profile.calltrace.cycles-pp.seq_read_iter.seq_read.vfs_read.ksys_read.do_syscall_64
0.22 ?146% perf-profile.calltrace.cycles-pp.shmem_write_end.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write
6.02 ? 15% perf-profile.calltrace.cycles-pp.read
3.62 ? 15% perf-profile.calltrace.cycles-pp.seq_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.08 ?223% perf-profile.calltrace.cycles-pp.copy_mc_enhanced_fast_string.__collapse_huge_page_copy.collapse_huge_page.hpage_collapse_scan_pmd.khugepaged_scan_mm_slot
5.95 ? 16% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
0.08 ?223% perf-profile.calltrace.cycles-pp.mm_init.dup_mm.copy_process.kernel_clone.__do_sys_clone
5.94 ? 16% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
0.09 ?223% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.pipe_read
0.09 ?223% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_read.vfs_read
0.09 ?223% perf-profile.calltrace.cycles-pp.show_stat.seq_read_iter.vfs_read.ksys_read.do_syscall_64
0.09 ?223% perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
0.09 ?223% perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
0.09 ?223% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains
0.09 ?223% perf-profile.calltrace.cycles-pp.__memset.pcpu_alloc.__percpu_counter_init.mm_init.alloc_bprm
2.72 ? 15% perf-profile.calltrace.cycles-pp.proc_single_show.seq_read_iter.seq_read.vfs_read.ksys_read
2.70 ? 15% perf-profile.calltrace.cycles-pp.do_task_stat.proc_single_show.seq_read_iter.seq_read.vfs_read
0.10 ?223% perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__x64_sys_vfork.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.10 ?223% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.10 ?223% perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.78 ? 22% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__vfork
0.78 ? 22% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__vfork
0.20 ?141% perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
0.11 ?223% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__do_softirq
0.74 ? 12% perf-profile.calltrace.cycles-pp.do_dentry_open.do_open.path_openat.do_filp_open.do_sys_openat2
0.11 ?223% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu
1.23 ? 19% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork
1.23 ? 19% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
1.23 ? 19% perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
1.23 ? 19% perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
0.11 ?223% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.worker_thread
0.11 ?223% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.worker_thread.kthread
0.12 ?223% perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork
0.92 ? 17% perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.12 ?223% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
0.12 ?223% perf-profile.calltrace.cycles-pp.task_mm_cid_work.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
0.12 ?223% perf-profile.calltrace.cycles-pp.__schedule.schedule.worker_thread.kthread.ret_from_fork
0.12 ?223% perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork
0.13 ?223% perf-profile.calltrace.cycles-pp.__pthread_disable_asynccancel.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
0.59 ? 47% perf-profile.calltrace.cycles-pp.update_sg_wakeup_stats.find_idlest_group.find_idlest_cpu.select_task_rq_fair.sched_exec
0.13 ?223% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
0.13 ?223% perf-profile.calltrace.cycles-pp.__mmdrop.finish_task_switch.__schedule.schedule.do_wait
0.14 ?223% perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.do_wait.kernel_wait4
0.14 ?223% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
0.28 ?100% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.28 ?100% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.28 ?101% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.14 ?223% perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
3.91 ? 35% perf-profile.calltrace.cycles-pp.ret_from_fork
3.91 ? 35% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
0.15 ?223% perf-profile.calltrace.cycles-pp.alloc_empty_file.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat
0.15 ?223% perf-profile.calltrace.cycles-pp.__alloc_file.alloc_empty_file.path_openat.do_filp_open.do_sys_openat2
0.15 ?223% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
0.15 ?223% perf-profile.calltrace.cycles-pp.__schedule.schedule.do_wait.kernel_wait4.do_syscall_64
0.15 ?223% perf-profile.calltrace.cycles-pp.schedule.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.16 ?223% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.30 ?146% perf-profile.calltrace.cycles-pp.ring_buffer_read_head.perf_mmap__read_head.perf_mmap__push.record__mmap_read_evlist.__cmd_record
0.30 ?146% perf-profile.calltrace.cycles-pp.perf_mmap__read_head.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
0.28 ?223% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.17 ?141% perf-profile.calltrace.cycles-pp.drm_atomic_helper_commit.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work
0.17 ?223% perf-profile.calltrace.cycles-pp.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.17 ?141% perf-profile.calltrace.cycles-pp.drm_atomic_commit.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work
0.17 ?223% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
0.74 ? 18% perf-profile.calltrace.cycles-pp.dup_mm.copy_process.kernel_clone.__do_sys_clone.do_syscall_64
0.18 ?223% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
0.19 ?141% perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.19 ?223% perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations
0.19 ?223% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
0.19 ?223% perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify
0.19 ?223% perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
0.19 ?223% perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
0.19 ?223% perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
0.19 ?223% perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
0.19 ?141% perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
1.37 ? 19% perf-profile.calltrace.cycles-pp.__libc_fork
0.90 ? 25% perf-profile.calltrace.cycles-pp.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm
0.58 ? 13% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.58 ? 14% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.26 ?100% perf-profile.calltrace.cycles-pp.drm_atomic_helper_dirtyfb.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread
0.27 ?148% perf-profile.calltrace.cycles-pp.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait4
1.01 ? 24% perf-profile.calltrace.cycles-pp.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
0.91 ? 24% perf-profile.calltrace.cycles-pp.mm_init.alloc_bprm.do_execveat_common.__x64_sys_execve.do_syscall_64
0.84 ? 48% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
0.84 ? 48% perf-profile.calltrace.cycles-pp.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
1.95 ? 22% perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
1.96 ? 22% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
1.94 ? 22% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
0.97 ? 24% perf-profile.calltrace.cycles-pp.alloc_bprm.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.30 ?223% perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.32 ?223% perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.32 ?151% perf-profile.calltrace.cycles-pp.queue_event
0.33 ?223% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.34 ?223% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.43 ?110% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait4
0.43 ?109% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.wait4
0.43 ?110% perf-profile.calltrace.cycles-pp.wait4
2.78 ? 21% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.34 ?223% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.36 ?110% perf-profile.calltrace.cycles-pp.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait4
0.48 ?223% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance
0.49 ?223% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule
0.49 ?223% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
0.75 ? 48% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
0.45 ?223% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
0.45 ?223% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.rcu_gp_fqs_loop
0.46 ?223% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread
0.46 ?223% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
0.46 ?223% perf-profile.calltrace.cycles-pp.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread.ret_from_fork
0.48 ?100% perf-profile.calltrace.cycles-pp.evsel__read_counter.read_counters.process_interval.dispatch_events.cmd_stat
0.49 ?223% perf-profile.calltrace.cycles-pp.rcu_gp_fqs_loop.rcu_gp_kthread.kthread.ret_from_fork
0.64 ?223% perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
0.54 ?223% perf-profile.calltrace.cycles-pp.rcu_gp_kthread.kthread.ret_from_fork
4.11 ? 22% perf-profile.calltrace.cycles-pp.execve
4.10 ? 22% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
4.11 ? 22% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
4.11 ? 22% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
4.11 ? 22% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
0.57 ?223% perf-profile.calltrace.cycles-pp.intel_idle_xstate.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.84 ?106% perf-profile.calltrace.cycles-pp.read_counters.process_interval.dispatch_events.cmd_stat.run_builtin
0.89 ?107% perf-profile.calltrace.cycles-pp.process_interval.dispatch_events.cmd_stat.run_builtin.main
0.90 ?107% perf-profile.calltrace.cycles-pp.cmd_stat.run_builtin.main.__libc_start_main
0.90 ?107% perf-profile.calltrace.cycles-pp.dispatch_events.cmd_stat.run_builtin.main.__libc_start_main
0.63 ? 14% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.queue_event.ordered_events__queue.process_simple.reader__read_event
0.80 ?223% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
0.98 ?223% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
1.08 ? 15% perf-profile.calltrace.cycles-pp.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread
1.93 ?209% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
2.00 ?210% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
2.02 ? 57% perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter_atomic.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
2.21 ?198% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
2.37 ?191% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
2.37 ?191% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
2.37 ?191% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
2.63 ?188% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
51.98 ? 8% perf-profile.calltrace.cycles-pp.__cmd_record
51.98 ? 8% perf-profile.calltrace.cycles-pp.record__finish_output.__cmd_record
51.98 ? 8% perf-profile.calltrace.cycles-pp.perf_session__process_events.record__finish_output.__cmd_record
51.95 ? 8% perf-profile.calltrace.cycles-pp.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record
50.61 ? 10% perf-profile.calltrace.cycles-pp.process_simple.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record
49.63 ? 11% perf-profile.calltrace.cycles-pp.queue_event.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events
49.86 ? 11% perf-profile.calltrace.cycles-pp.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events.record__finish_output
5.45 ? 40% perf-profile.children.cycles-pp.asm_exc_page_fault
5.07 ? 42% perf-profile.children.cycles-pp.do_user_addr_fault
5.14 ? 42% perf-profile.children.cycles-pp.exc_page_fault
4.62 ? 47% perf-profile.children.cycles-pp.handle_mm_fault
4.32 ? 51% perf-profile.children.cycles-pp.__handle_mm_fault
1.21 ?161% perf-profile.children.cycles-pp.do_huge_pmd_numa_page
1.18 ?159% perf-profile.children.cycles-pp.migrate_misplaced_page
1.17 ?159% perf-profile.children.cycles-pp.migrate_pages
1.17 ?159% perf-profile.children.cycles-pp.migrate_pages_batch
1.18 ?147% perf-profile.children.cycles-pp.copy_page
1.07 ?160% perf-profile.children.cycles-pp.move_to_new_folio
1.07 ?160% perf-profile.children.cycles-pp.migrate_folio_extra
1.07 ?160% perf-profile.children.cycles-pp.folio_copy
0.88 ?223% perf-profile.children.cycles-pp.__ordered_events__flush
0.88 ?223% perf-profile.children.cycles-pp.perf_session__process_user_event
0.71 ?223% perf-profile.children.cycles-pp.perf_session__deliver_event
0.00 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.55 ?223% perf-profile.children.cycles-pp.evlist__parse_sample
0.00 perf-profile.children.cycles-pp.drm_fbdev_fb_dirty
0.14 ?223% perf-profile.children.cycles-pp.build_id__mark_dso_hit
0.12 ?223% perf-profile.children.cycles-pp.evsel__parse_sample
29.60 ? 15% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
29.56 ? 15% perf-profile.children.cycles-pp.do_syscall_64
7.09 ? 57% perf-profile.children.cycles-pp.cmd_record
0.00 perf-profile.children.cycles-pp.memcpy_erms
0.01 ?223% perf-profile.children.cycles-pp.thread__find_map
1.41 ? 13% perf-profile.children.cycles-pp.process_one_work
0.00 perf-profile.children.cycles-pp.drm_gem_shmem_vmap
0.00 perf-profile.children.cycles-pp.memset_erms
0.00 perf-profile.children.cycles-pp.map__load
0.00 perf-profile.children.cycles-pp.dso__load
0.00 perf-profile.children.cycles-pp.__dso__load_kallsyms
0.64 ? 31% perf-profile.children.cycles-pp.do_sys_poll
0.65 ? 31% perf-profile.children.cycles-pp.__x64_sys_poll
5.02 ? 58% perf-profile.children.cycles-pp.__generic_file_write_iter
4.98 ? 58% perf-profile.children.cycles-pp.generic_perform_write
5.05 ? 58% perf-profile.children.cycles-pp.generic_file_write_iter
0.13 ? 79% perf-profile.children.cycles-pp.smp_call_function_many_cond
0.08 ?138% perf-profile.children.cycles-pp.on_each_cpu_cond_mask
0.68 ? 30% perf-profile.children.cycles-pp.__poll
5.34 ? 56% perf-profile.children.cycles-pp.vfs_write
5.40 ? 56% perf-profile.children.cycles-pp.ksys_write
0.13 ?223% perf-profile.children.cycles-pp.machine__findnew_thread
5.27 ? 58% perf-profile.children.cycles-pp.__libc_write
1.58 ? 21% perf-profile.children.cycles-pp.worker_thread
0.15 ? 82% perf-profile.children.cycles-pp.flush_tlb_mm_range
0.00 perf-profile.children.cycles-pp.drm_fbdev_damage_blit_real
0.06 ?223% perf-profile.children.cycles-pp.migrate_folio_unmap
0.08 ?106% perf-profile.children.cycles-pp.asm_sysvec_call_function
0.52 ? 31% perf-profile.children.cycles-pp.do_poll
1.08 ? 15% perf-profile.children.cycles-pp.drm_fb_helper_damage_work
0.06 ?189% perf-profile.children.cycles-pp.rmap_walk_anon
0.05 ?223% perf-profile.children.cycles-pp.try_to_migrate
0.37 ? 11% perf-profile.children.cycles-pp.__fxstat64
1.14 ? 41% perf-profile.children.cycles-pp.shmem_get_folio_gfp
0.05 ?223% perf-profile.children.cycles-pp.try_to_migrate_one
0.30 ? 11% perf-profile.children.cycles-pp.__do_sys_newfstat
1.15 ? 12% perf-profile.children.cycles-pp.link_path_walk
0.45 ? 67% perf-profile.children.cycles-pp.evlist__parse_sample_timestamp
5.52 ? 61% perf-profile.children.cycles-pp.writen
0.04 ?223% perf-profile.children.cycles-pp.set_pmd_migration_entry
0.00 perf-profile.children.cycles-pp.kallsyms__parse
0.06 ?107% perf-profile.children.cycles-pp.sysvec_call_function
0.04 ?223% perf-profile.children.cycles-pp.pmdp_invalidate
0.39 ? 36% perf-profile.children.cycles-pp.perf_poll
0.00 perf-profile.children.cycles-pp.rb_erase
5.65 ? 61% perf-profile.children.cycles-pp.record__pushfn
1.48 ? 15% perf-profile.children.cycles-pp.lookup_fast
5.05 ? 17% perf-profile.children.cycles-pp.seq_read_iter
0.02 ?223% perf-profile.children.cycles-pp.migration_entry_wait_on_locked
0.00 perf-profile.children.cycles-pp.evlist__delete
0.02 ?223% perf-profile.children.cycles-pp.io_schedule
0.00 perf-profile.children.cycles-pp.perf_session__delete
3.49 ? 15% perf-profile.children.cycles-pp.do_filp_open
3.48 ? 15% perf-profile.children.cycles-pp.path_openat
0.86 ? 62% perf-profile.children.cycles-pp.shmem_write_begin
0.97 ? 7% perf-profile.children.cycles-pp.sched_setaffinity
1.14 ? 18% perf-profile.children.cycles-pp.do_read_fault
3.69 ? 15% perf-profile.children.cycles-pp.do_sys_openat2
3.70 ? 15% perf-profile.children.cycles-pp.__x64_sys_openat
1.08 ? 13% perf-profile.children.cycles-pp.walk_component
1.38 ? 18% perf-profile.children.cycles-pp.do_fault
0.35 ? 21% perf-profile.children.cycles-pp.__fput
0.00 perf-profile.children.cycles-pp.__open64
0.45 ? 23% perf-profile.children.cycles-pp._raw_spin_lock
1.49 ? 19% perf-profile.children.cycles-pp.__mmput
1.48 ? 19% perf-profile.children.cycles-pp.exit_mmap
0.06 ?106% perf-profile.children.cycles-pp.__sysvec_call_function
0.54 ? 12% perf-profile.children.cycles-pp.serial8250_console_write
0.54 ? 12% perf-profile.children.cycles-pp.console_unlock
0.54 ? 12% perf-profile.children.cycles-pp.console_flush_all
0.26 ? 14% perf-profile.children.cycles-pp.vfs_fstat
0.06 ?104% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
2.06 ? 58% perf-profile.children.cycles-pp.copy_page_from_iter_atomic
6.22 ? 15% perf-profile.children.cycles-pp.vfs_read
0.56 ? 12% perf-profile.children.cycles-pp.vprintk_emit
0.77 ? 20% perf-profile.children.cycles-pp.unmap_vmas
0.56 ? 12% perf-profile.children.cycles-pp.irq_work_run_list
0.56 ? 12% perf-profile.children.cycles-pp.irq_work_single
0.56 ? 12% perf-profile.children.cycles-pp.sysvec_irq_work
0.56 ? 12% perf-profile.children.cycles-pp.__sysvec_irq_work
0.56 ? 12% perf-profile.children.cycles-pp.irq_work_run
0.56 ? 12% perf-profile.children.cycles-pp._printk
0.56 ? 12% perf-profile.children.cycles-pp.asm_sysvec_irq_work
0.68 ? 21% perf-profile.children.cycles-pp.unmap_page_range
0.54 ? 22% perf-profile.children.cycles-pp.__libc_read
0.65 ? 20% perf-profile.children.cycles-pp.zap_pte_range
0.66 ? 21% perf-profile.children.cycles-pp.zap_pmd_range
0.00 perf-profile.children.cycles-pp.evlist__close
0.78 ? 19% perf-profile.children.cycles-pp.filemap_map_pages
0.03 ?223% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
6.36 ? 16% perf-profile.children.cycles-pp.ksys_read
0.01 ?223% perf-profile.children.cycles-pp.security_inode_getattr
0.07 ? 49% perf-profile.children.cycles-pp.__close
0.00 perf-profile.children.cycles-pp.io__get_hex
1.09 ? 20% perf-profile.children.cycles-pp.exit_mm
0.08 ? 87% perf-profile.children.cycles-pp.task_numa_work
0.06 ? 88% perf-profile.children.cycles-pp.change_prot_numa
0.00 perf-profile.children.cycles-pp.__mem_cgroup_flush_stats
0.00 perf-profile.children.cycles-pp.cgroup_rstat_flush_irqsafe
0.52 ? 23% perf-profile.children.cycles-pp.next_uptodate_page
0.95 ? 16% perf-profile.children.cycles-pp.path_lookupat
0.07 ? 34% perf-profile.children.cycles-pp.free_unref_page
0.95 ? 17% perf-profile.children.cycles-pp.__xstat64
0.36 ? 27% perf-profile.children.cycles-pp.tlb_finish_mmu
0.96 ? 15% perf-profile.children.cycles-pp.filename_lookup
6.52 ? 62% perf-profile.children.cycles-pp.record__mmap_read_evlist
0.98 ? 16% perf-profile.children.cycles-pp.__do_sys_newstat
0.88 ? 16% perf-profile.children.cycles-pp.vfs_statx
0.00 perf-profile.children.cycles-pp.perf_session__do_write_header
0.00 perf-profile.children.cycles-pp.machine__process_fork_event
6.24 ? 62% perf-profile.children.cycles-pp.perf_mmap__push
0.76 ? 24% perf-profile.children.cycles-pp.__percpu_counter_init
0.68 ? 6% perf-profile.children.cycles-pp.evlist_cpu_iterator__next
0.18 ? 59% perf-profile.children.cycles-pp.evsel__parse_sample_timestamp
0.00 perf-profile.children.cycles-pp.common_perm_cond
0.36 ? 25% perf-profile.children.cycles-pp.perf_evsel__read
0.00 perf-profile.children.cycles-pp.update_rq_clock
0.37 ? 29% perf-profile.children.cycles-pp.drm_gem_vmap_unlocked
0.37 ? 29% perf-profile.children.cycles-pp.drm_gem_vmap
0.00 perf-profile.children.cycles-pp.perf_evsel__close_cpu
0.97 ? 16% perf-profile.children.cycles-pp.vfs_fstatat
1.18 ? 9% perf-profile.children.cycles-pp.smpboot_thread_fn
0.00 perf-profile.children.cycles-pp.machine__exit
0.00 perf-profile.children.cycles-pp.proc_reg_read
0.00 perf-profile.children.cycles-pp.dso__delete
0.47 ? 14% perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
0.01 ?223% perf-profile.children.cycles-pp.llist_add_batch
0.00 perf-profile.children.cycles-pp.____machine__findnew_thread
0.02 ?142% perf-profile.children.cycles-pp.cp_new_stat
0.03 ?111% perf-profile.children.cycles-pp.free_unref_page_prepare
0.27 ? 12% perf-profile.children.cycles-pp.__fget_light
0.00 perf-profile.children.cycles-pp.rb_next
0.00 perf-profile.children.cycles-pp.symbols__delete
0.04 ?113% perf-profile.children.cycles-pp.task_numa_fault
0.06 ? 56% perf-profile.children.cycles-pp.vmstat_shepherd
0.08 ? 46% perf-profile.children.cycles-pp.do_numa_page
0.28 ? 33% perf-profile.children.cycles-pp.drm_client_buffer_vmap
0.00 perf-profile.children.cycles-pp.write_cache
0.67 ? 16% perf-profile.children.cycles-pp.kmem_cache_alloc
0.28 ? 21% perf-profile.children.cycles-pp.__list_del_entry_valid
0.14 ? 18% perf-profile.children.cycles-pp.mutex_lock
0.00 perf-profile.children.cycles-pp.machine__delete_threads
0.24 ? 21% perf-profile.children.cycles-pp.___perf_sw_event
0.12 ? 40% perf-profile.children.cycles-pp.poll_freewait
0.47 ? 8% perf-profile.children.cycles-pp.__x64_sys_sched_setaffinity
0.45 ? 13% perf-profile.children.cycles-pp.drm_atomic_commit
0.00 perf-profile.children.cycles-pp.rb_free
0.00 perf-profile.children.cycles-pp.perf_release
0.00 perf-profile.children.cycles-pp.perf_event_release_kernel
0.12 ? 23% perf-profile.children.cycles-pp.get_mm_cmdline
0.03 ?155% perf-profile.children.cycles-pp.task_numa_migrate
0.00 perf-profile.children.cycles-pp.thread__delete
0.86 ? 21% perf-profile.children.cycles-pp.proc_reg_read_iter
0.12 ? 25% perf-profile.children.cycles-pp.task_tick_fair
0.05 ? 47% perf-profile.children.cycles-pp.prepare_task_switch
0.46 ? 21% perf-profile.children.cycles-pp.__close_nocancel
0.37 ? 27% perf-profile.children.cycles-pp.run_ksoftirqd
0.42 ? 18% perf-profile.children.cycles-pp.wait_for_lsr
0.36 ? 28% perf-profile.children.cycles-pp.native_irq_return_iret
0.36 ? 67% perf-profile.children.cycles-pp.shmem_alloc_and_acct_folio
0.28 ? 22% perf-profile.children.cycles-pp._dl_addr
0.44 ? 14% perf-profile.children.cycles-pp.drm_atomic_helper_commit
0.11 ? 19% perf-profile.children.cycles-pp.__virt_addr_valid
0.00 perf-profile.children.cycles-pp.symbols__fixup_end
0.41 ? 21% perf-profile.children.cycles-pp.fold_vm_numa_events
0.00 perf-profile.children.cycles-pp.thread__new
0.21 ? 26% perf-profile.children.cycles-pp.__perf_sw_event
0.00 perf-profile.children.cycles-pp._free_event
0.12 ? 15% perf-profile.children.cycles-pp.task_dump_owner
0.18 ? 15% perf-profile.children.cycles-pp.do_readlinkat
0.19 ? 14% perf-profile.children.cycles-pp.sysfs_kf_seq_show
0.18 ? 76% perf-profile.children.cycles-pp.io_serial_out
0.70 ? 10% perf-profile.children.cycles-pp.io_serial_in
0.00 perf-profile.children.cycles-pp.machine__process_mmap2_event
0.00 perf-profile.children.cycles-pp.__mod_memcg_state
0.18 ? 14% perf-profile.children.cycles-pp.dev_attr_show
1.55 ? 11% perf-profile.children.cycles-pp.__x64_sys_exit_group
0.36 ? 19% perf-profile.children.cycles-pp.cpu_stopper_thread
0.00 perf-profile.children.cycles-pp.nsinfo__new
0.34 ? 23% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.16 ? 24% perf-profile.children.cycles-pp.__vm_munmap
0.16 ? 23% perf-profile.children.cycles-pp.proc_pid_cmdline_read
0.19 ? 16% perf-profile.children.cycles-pp.__x64_sys_readlink
1.55 ? 11% perf-profile.children.cycles-pp.do_group_exit
0.02 ?223% perf-profile.children.cycles-pp.delay_halt
0.34 ? 14% perf-profile.children.cycles-pp.ast_mode_config_helper_atomic_commit_tail
0.39 ? 18% perf-profile.children.cycles-pp.filemap_get_entry
0.06 ? 76% perf-profile.children.cycles-pp.flush_tlb_func
0.25 ? 15% perf-profile.children.cycles-pp.__check_object_size
1.55 ? 12% perf-profile.children.cycles-pp.do_exit
0.34 ? 17% perf-profile.children.cycles-pp.migration_cpu_stop
0.44 ? 21% perf-profile.children.cycles-pp.do_vmi_align_munmap
0.31 ? 29% perf-profile.children.cycles-pp.drm_gem_shmem_get_pages_locked
0.10 ? 39% perf-profile.children.cycles-pp.remove_wait_queue
0.05 ? 76% perf-profile.children.cycles-pp.free_pcppages_bulk
0.35 ? 13% perf-profile.children.cycles-pp.commit_tail
0.60 ? 17% perf-profile.children.cycles-pp.pid_revalidate
0.46 ? 21% perf-profile.children.cycles-pp.do_vmi_munmap
0.31 ? 29% perf-profile.children.cycles-pp.drm_gem_get_pages
0.31 ? 29% perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
0.08 ? 24% perf-profile.children.cycles-pp.__lock_task_sighand
0.00 perf-profile.children.cycles-pp.cpu_cache_level__read
0.13 ? 19% perf-profile.children.cycles-pp.unmap_region
0.02 ?146% perf-profile.children.cycles-pp.__munmap
0.00 perf-profile.children.cycles-pp.task_numa_find_cpu
0.17 ? 14% perf-profile.children.cycles-pp.readlink
0.00 perf-profile.children.cycles-pp.map__new
0.20 ? 19% perf-profile.children.cycles-pp.update_load_avg
0.00 perf-profile.children.cycles-pp.task_numa_compare
0.00 perf-profile.children.cycles-pp.maps__put
0.72 ? 26% perf-profile.children.cycles-pp.wake_up_new_task
0.54 ? 23% perf-profile.children.cycles-pp.vmstat_start
0.40 ? 11% perf-profile.children.cycles-pp.__sched_setaffinity
0.05 ? 75% perf-profile.children.cycles-pp.__get_vm_area_node
0.04 ? 75% perf-profile.children.cycles-pp.alloc_vmap_area
0.00 perf-profile.children.cycles-pp.sw_perf_event_destroy
0.09 ? 33% perf-profile.children.cycles-pp.__x64_sys_munmap
0.53 ? 17% perf-profile.children.cycles-pp.step_into
0.32 ? 17% perf-profile.children.cycles-pp.move_queued_task
0.38 ? 13% perf-profile.children.cycles-pp.__set_cpus_allowed_ptr
0.00 perf-profile.children.cycles-pp.from_kuid_munged
0.00 perf-profile.children.cycles-pp.swevent_hlist_put_cpu
0.08 ? 64% perf-profile.children.cycles-pp.add_wait_queue
0.02 ?142% perf-profile.children.cycles-pp.terminate_walk
0.26 ? 57% perf-profile.children.cycles-pp.shmem_add_to_page_cache
0.00 perf-profile.children.cycles-pp.map__process_kallsym_symbol
0.00 perf-profile.children.cycles-pp.llist_reverse_order
0.00 perf-profile.children.cycles-pp.queue_delayed_work_on
0.10 ? 40% perf-profile.children.cycles-pp.drm_gem_vunmap_unlocked
0.01 ?223% perf-profile.children.cycles-pp.generic_permission
0.00 perf-profile.children.cycles-pp.map_id_up
0.00 perf-profile.children.cycles-pp.__free_pages
0.10 ? 40% perf-profile.children.cycles-pp.drm_gem_shmem_vunmap
0.15 ? 24% perf-profile.children.cycles-pp.folio_batch_move_lru
0.10 ? 39% perf-profile.children.cycles-pp.drm_gem_vunmap
0.06 ? 51% perf-profile.children.cycles-pp.folio_add_lru
0.70 ? 18% perf-profile.children.cycles-pp.__d_lookup_rcu
0.17 ? 34% perf-profile.children.cycles-pp.__count_memcg_events
0.20 ? 13% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.02 ?142% perf-profile.children.cycles-pp.folio_unlock
0.16 ? 18% perf-profile.children.cycles-pp.vsnprintf
0.29 ? 12% perf-profile.children.cycles-pp.drm_atomic_helper_commit_planes
0.28 ? 13% perf-profile.children.cycles-pp.memcpy_toio
0.28 ? 13% perf-profile.children.cycles-pp.drm_fb_memcpy
0.03 ?100% perf-profile.children.cycles-pp.irqtime_account_irq
0.30 ? 11% perf-profile.children.cycles-pp.drm_atomic_helper_commit_tail_rpm
0.05 ? 47% perf-profile.children.cycles-pp.vmap
0.29 ? 13% perf-profile.children.cycles-pp.ast_primary_plane_helper_atomic_update
0.22 ? 21% perf-profile.children.cycles-pp.mod_objcg_state
0.23 ? 18% perf-profile.children.cycles-pp.affine_move_task
0.27 ? 23% perf-profile.children.cycles-pp.tlb_batch_pages_flush
0.13 ? 29% perf-profile.children.cycles-pp.single_release
1.34 ? 23% perf-profile.children.cycles-pp.find_idlest_cpu
0.23 ? 19% perf-profile.children.cycles-pp.release_pages
1.40 ? 22% perf-profile.children.cycles-pp.select_task_rq_fair
0.42 ? 19% perf-profile.children.cycles-pp.inode_permission
0.25 ? 17% perf-profile.children.cycles-pp.__d_lookup
0.00 perf-profile.children.cycles-pp.sysfs__read_int
2.94 ? 18% perf-profile.children.cycles-pp.open64
0.05 ? 77% perf-profile.children.cycles-pp.acpi_ps_parse_loop
0.07 ? 50% perf-profile.children.cycles-pp.__mod_node_page_state
0.00 perf-profile.children.cycles-pp.s_next
0.02 ?223% perf-profile.children.cycles-pp.change_pmd_range
0.08 ? 20% perf-profile.children.cycles-pp.seq_printf
0.02 ?152% perf-profile.children.cycles-pp.change_protection_range
0.29 ? 22% perf-profile.children.cycles-pp.all_vm_events
3.62 ? 15% perf-profile.children.cycles-pp.seq_read
0.51 ? 14% perf-profile.children.cycles-pp.show_numa_map
0.33 ? 25% perf-profile.children.cycles-pp.setlocale
0.07 ? 50% perf-profile.children.cycles-pp.acpi_os_execute_deferred
0.00 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.01 ?223% perf-profile.children.cycles-pp.change_pte_range
0.07 ? 50% perf-profile.children.cycles-pp.__perf_event_read_value
0.41 ? 18% perf-profile.children.cycles-pp.walk_pud_range
0.09 ? 21% perf-profile.children.cycles-pp.node_read_vmstat
0.22 ? 27% perf-profile.children.cycles-pp.__mod_lruvec_page_state
0.27 ? 59% perf-profile.children.cycles-pp.scheduler_tick
0.10 ? 32% perf-profile.children.cycles-pp.flush_memcg_stats_dwork
0.10 ? 32% perf-profile.children.cycles-pp.cgroup_rstat_flush_locked
0.06 ? 53% perf-profile.children.cycles-pp.acpi_ns_evaluate
0.00 perf-profile.children.cycles-pp.thread__get
0.02 ?142% perf-profile.children.cycles-pp.need_update
0.12 ? 25% perf-profile.children.cycles-pp.dput
0.16 ? 27% perf-profile.children.cycles-pp.lock_vma_under_rcu
0.15 ? 17% perf-profile.children.cycles-pp.cpu_stop_queue_work
0.15 ? 24% perf-profile.children.cycles-pp.shift_arg_pages
0.42 ? 17% perf-profile.children.cycles-pp.walk_page_vma
0.42 ? 17% perf-profile.children.cycles-pp.__walk_page_range
0.42 ? 17% perf-profile.children.cycles-pp.walk_pgd_range
0.42 ? 17% perf-profile.children.cycles-pp.walk_p4d_range
0.41 ? 18% perf-profile.children.cycles-pp.walk_pmd_range
0.06 ? 52% perf-profile.children.cycles-pp.acpi_ev_asynch_execute_gpe_method
0.02 ?142% perf-profile.children.cycles-pp.memchr_inv
0.03 ?101% perf-profile.children.cycles-pp.__free_one_page
0.06 ? 52% perf-profile.children.cycles-pp.acpi_ps_execute_method
0.06 ? 52% perf-profile.children.cycles-pp.acpi_ps_parse_aml
0.04 ? 75% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.12 ? 15% perf-profile.children.cycles-pp.wake_up_q
0.06 ? 51% perf-profile.children.cycles-pp.__vmalloc_node_range
0.61 ? 11% perf-profile.children.cycles-pp.iterate_dir
0.40 ? 17% perf-profile.children.cycles-pp.gather_pte_stats
0.00 perf-profile.children.cycles-pp.update_iter
0.03 ?100% perf-profile.children.cycles-pp.drm_atomic_helper_cleanup_planes
0.03 ?100% perf-profile.children.cycles-pp.drm_gem_fb_vunmap
0.00 perf-profile.children.cycles-pp._compound_head
0.02 ?142% perf-profile.children.cycles-pp.scnprintf
0.00 perf-profile.children.cycles-pp.s_show
0.05 ? 51% perf-profile.children.cycles-pp.vfs_readlink
0.00 perf-profile.children.cycles-pp.vunmap
0.00 perf-profile.children.cycles-pp.complete_walk
0.02 ?152% perf-profile.children.cycles-pp.change_p4d_range
0.15 ? 24% perf-profile.children.cycles-pp.vma_interval_tree_remove
0.03 ?100% perf-profile.children.cycles-pp.proc_pid_readlink
0.00 perf-profile.children.cycles-pp.io__get_char
0.00 perf-profile.children.cycles-pp.rw_verify_area
0.01 ?223% perf-profile.children.cycles-pp.lru_add_fn
0.04 ? 73% perf-profile.children.cycles-pp.security_vm_enough_memory_mm
0.04 ?103% perf-profile.children.cycles-pp.check_move_unevictable_pages
0.16 ? 19% perf-profile.children.cycles-pp.sync_regs
0.18 ? 21% perf-profile.children.cycles-pp.setup_arg_pages
0.07 ? 52% perf-profile.children.cycles-pp.free_unref_page_list
0.62 ? 11% perf-profile.children.cycles-pp.__x64_sys_getdents64
0.17 ? 23% perf-profile.children.cycles-pp.vma_prepare
0.08 ? 62% perf-profile.children.cycles-pp.drm_gem_shmem_put_pages_locked
0.08 ? 62% perf-profile.children.cycles-pp.drm_gem_put_pages
0.00 perf-profile.children.cycles-pp.percpu_counter_destroy
0.00 perf-profile.children.cycles-pp.alloc_misplaced_dst_page
0.08 ? 29% perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush
0.04 ? 71% perf-profile.children.cycles-pp.dentry_kill
0.00 perf-profile.children.cycles-pp.get_cpu_idle_time_us
0.13 ? 48% perf-profile.children.cycles-pp.do_set_pte
0.09 ? 28% perf-profile.children.cycles-pp.node_read_numastat
0.62 ? 11% perf-profile.children.cycles-pp.getdents64
0.08 ? 49% perf-profile.children.cycles-pp.free_pages_and_swap_cache
0.00 perf-profile.children.cycles-pp.machine__findnew_vdso
0.05 ? 71% perf-profile.children.cycles-pp.__dentry_kill
0.06 ? 48% perf-profile.children.cycles-pp.perf_exclude_event
0.00 perf-profile.children.cycles-pp.machines__deliver_event
0.00 perf-profile.children.cycles-pp.kernfs_fop_open
0.08 ? 29% perf-profile.children.cycles-pp.lru_add_drain_cpu
0.06 ? 47% perf-profile.children.cycles-pp.fput
0.18 ? 22% perf-profile.children.cycles-pp.switch_fpu_return
0.15 ? 21% perf-profile.children.cycles-pp.kmalloc_trace
0.04 ? 73% perf-profile.children.cycles-pp.set_task_cpu
0.00 perf-profile.children.cycles-pp.get_vm_area_caller
0.00 perf-profile.children.cycles-pp.kernfs_dop_revalidate
0.07 ? 16% perf-profile.children.cycles-pp.__switch_to
0.76 ? 16% perf-profile.children.cycles-pp.open_last_lookups
0.10 ? 52% perf-profile.children.cycles-pp.finish_fault
0.07 ? 26% perf-profile.children.cycles-pp.lru_add_drain
0.05 ? 80% perf-profile.children.cycles-pp.kstat_irqs_usr
0.01 ?223% perf-profile.children.cycles-pp.page_vma_mapped_walk
0.00 perf-profile.children.cycles-pp.sysvec_call_function_single
0.04 ? 72% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.34 ? 20% perf-profile.children.cycles-pp.free_pgtables
0.29 ? 24% perf-profile.children.cycles-pp.__open64_nocancel
0.01 ?223% perf-profile.children.cycles-pp.update_numa_stats
0.04 ? 75% perf-profile.children.cycles-pp.mas_next_entry
0.60 ? 11% perf-profile.children.cycles-pp.proc_pid_readdir
0.35 ? 15% perf-profile.children.cycles-pp.proc_fill_cache
0.10 ? 23% perf-profile.children.cycles-pp.alloc_thread_stack_node
0.00 perf-profile.children.cycles-pp.alloc_fd
0.00 perf-profile.children.cycles-pp.perf_swevent_event
0.00 perf-profile.children.cycles-pp.task_prio
0.03 ?100% perf-profile.children.cycles-pp._copy_to_user
0.00 perf-profile.children.cycles-pp.proc_ns_get_link
0.00 perf-profile.children.cycles-pp.ns_get_path
0.00 perf-profile.children.cycles-pp.kallsyms_expand_symbol
0.00 perf-profile.children.cycles-pp.__ns_get_path
0.00 perf-profile.children.cycles-pp.sysfs__read_str
0.21 ? 13% perf-profile.children.cycles-pp.xas_load
0.14 ? 14% perf-profile.children.cycles-pp.user_path_at_empty
0.07 ? 50% perf-profile.children.cycles-pp.move_page_tables
0.08 ? 48% perf-profile.children.cycles-pp.free_swap_cache
0.09 ? 23% perf-profile.children.cycles-pp.__d_alloc
0.38 ? 68% perf-profile.children.cycles-pp.update_process_times
0.23 ? 21% perf-profile.children.cycles-pp.vma_interval_tree_insert
0.11 ? 47% perf-profile.children.cycles-pp.aa_file_perm
0.01 ?223% perf-profile.children.cycles-pp.acpi_ns_search_and_enter
0.01 ?223% perf-profile.children.cycles-pp.acpi_ns_lookup
0.01 ?223% perf-profile.children.cycles-pp.close_fd
0.00 perf-profile.children.cycles-pp.release_pte_folio
0.11 ? 23% perf-profile.children.cycles-pp.percpu_counter_add_batch
0.12 ? 25% perf-profile.children.cycles-pp.vm_area_dup
0.09 ? 24% perf-profile.children.cycles-pp.seq_open
0.04 ? 72% perf-profile.children.cycles-pp.arch_irq_stat_cpu
0.03 ?102% perf-profile.children.cycles-pp.vmap_pages_pud_range
0.01 ?223% perf-profile.children.cycles-pp.tlb_is_not_lazy
0.29 ? 64% perf-profile.children.cycles-pp.shmem_alloc_folio
0.28 ? 25% perf-profile.children.cycles-pp.get_user_pages_remote
0.17 ? 29% perf-profile.children.cycles-pp.blk_mq_in_flight
0.17 ? 29% perf-profile.children.cycles-pp.blk_mq_queue_tag_busy_iter
0.36 ? 18% perf-profile.children.cycles-pp.dup_mmap
0.16 ? 20% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
0.16 ? 22% perf-profile.children.cycles-pp.pid_nr_ns
0.15 ? 55% perf-profile.children.cycles-pp.perf_mmap_fault
0.00 perf-profile.children.cycles-pp.__dsos__find
0.02 ?142% perf-profile.children.cycles-pp.remove_vm_area
0.13 ? 23% perf-profile.children.cycles-pp.unlink_file_vma
0.00 perf-profile.children.cycles-pp.find_unlink_vmap_area
0.00 perf-profile.children.cycles-pp.acpi_ds_exec_end_op
0.00 perf-profile.children.cycles-pp.acpi_ps_get_next_namepath
0.03 ?101% perf-profile.children.cycles-pp.__vmap_pages_range_noflush
0.00 perf-profile.children.cycles-pp.mntput_no_expire
0.00 perf-profile.children.cycles-pp.folio_putback_lru
0.00 perf-profile.children.cycles-pp.mem_cgroup_from_task
0.48 ? 29% perf-profile.children.cycles-pp.khugepaged
0.35 ? 28% perf-profile.children.cycles-pp.rmqueue
0.11 ? 10% perf-profile.children.cycles-pp.__do_set_cpus_allowed
0.11 ? 26% perf-profile.children.cycles-pp.unlink_anon_vmas
0.10 ? 45% perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
0.00 perf-profile.children.cycles-pp.update_iter_mod
0.00 perf-profile.children.cycles-pp.module_get_kallsym
0.03 ?100% perf-profile.children.cycles-pp.handle_pte_fault
0.00 perf-profile.children.cycles-pp.readdir64
0.00 perf-profile.children.cycles-pp.__sysvec_call_function_single
0.00 perf-profile.children.cycles-pp.follow_page_pte
0.00 perf-profile.children.cycles-pp.map__put
0.00 perf-profile.children.cycles-pp.perf_trace_destroy
0.00 perf-profile.children.cycles-pp.do_pipe2
0.00 perf-profile.children.cycles-pp.__x64_sys_pipe
0.00 perf-profile.children.cycles-pp.__x2apic_send_IPI_mask
0.01 ?223% perf-profile.children.cycles-pp.__vm_enough_memory
0.02 ?223% perf-profile.children.cycles-pp.delay_halt_tpause
0.10 ? 14% perf-profile.children.cycles-pp.__x64_sys_close
0.09 ? 27% perf-profile.children.cycles-pp.drm_gem_fb_vmap
0.31 ? 20% perf-profile.children.cycles-pp.diskstats_show
0.27 ? 25% perf-profile.children.cycles-pp.__get_user_pages
0.44 ? 78% perf-profile.children.cycles-pp.tick_sched_timer
0.09 ? 26% perf-profile.children.cycles-pp.kmem_cache_alloc_lru
0.18 ? 30% perf-profile.children.cycles-pp.copy_strings
0.07 ? 81% perf-profile.children.cycles-pp.folio_mark_accessed
0.07 ? 14% perf-profile.children.cycles-pp.__switch_to_asm
0.06 ? 45% perf-profile.children.cycles-pp.unmap_single_vma
0.01 ?223% perf-profile.children.cycles-pp.cap_vm_enough_memory
0.00 perf-profile.children.cycles-pp.hrtimer_active
0.04 ? 76% perf-profile.children.cycles-pp.do_notify_parent
0.01 ?223% perf-profile.children.cycles-pp.mutex_unlock
0.01 ?223% perf-profile.children.cycles-pp.mas_store
0.00 perf-profile.children.cycles-pp.write_cpu_topology
0.00 perf-profile.children.cycles-pp.perf_trace_event_unreg
0.00 perf-profile.children.cycles-pp.mntput
0.00 perf-profile.children.cycles-pp.__tlb_remove_page_size
0.08 ? 19% perf-profile.children.cycles-pp.select_task_rq
0.40 ? 72% perf-profile.children.cycles-pp.tick_sched_handle
0.45 ? 29% perf-profile.children.cycles-pp.collapse_huge_page
0.14 ? 15% perf-profile.children.cycles-pp.__set_cpus_allowed_ptr_locked
0.16 ? 17% perf-profile.children.cycles-pp.__cond_resched
0.01 ?223% perf-profile.children.cycles-pp.events_stats__inc
0.01 ?223% perf-profile.children.cycles-pp.security_file_free
0.00 perf-profile.children.cycles-pp.__get_vma_policy
0.00 perf-profile.children.cycles-pp.folio_add_lru_vma
0.00 perf-profile.children.cycles-pp.proc_exe_link
0.00 perf-profile.children.cycles-pp.trace_event_reg
0.00 perf-profile.children.cycles-pp.tracepoint_probe_unregister
0.00 perf-profile.children.cycles-pp.text_poke_bp_batch
0.00 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.00 perf-profile.children.cycles-pp._copy_from_user
0.11 ? 18% perf-profile.children.cycles-pp.perf_event_read
0.09 ? 26% perf-profile.children.cycles-pp.drm_atomic_helper_prepare_planes
0.04 ?105% perf-profile.children.cycles-pp.workingset_activation
0.07 ? 13% perf-profile.children.cycles-pp.filp_close
1.56 ? 59% perf-profile.children.cycles-pp.fault_in_readable
1.57 ? 59% perf-profile.children.cycles-pp.fault_in_iov_iter_readable
0.47 ? 29% perf-profile.children.cycles-pp.hpage_collapse_scan_pmd
0.47 ? 29% perf-profile.children.cycles-pp.khugepaged_scan_mm_slot
0.28 ? 30% perf-profile.children.cycles-pp.__do_fault
0.21 ? 17% perf-profile.children.cycles-pp.single_open
0.20 ? 21% perf-profile.children.cycles-pp.perf_event_mmap
0.03 ?101% perf-profile.children.cycles-pp.__do_sys_brk
0.05 ? 74% perf-profile.children.cycles-pp.__mod_lruvec_state
0.01 ?223% perf-profile.children.cycles-pp.alloc_inode
0.02 ?145% perf-profile.children.cycles-pp.smp_call_function
0.01 ?223% perf-profile.children.cycles-pp.__mem_cgroup_uncharge_list
0.00 perf-profile.children.cycles-pp.new_inode_pseudo
0.03 ?100% perf-profile.children.cycles-pp.cfree
0.05 ? 51% perf-profile.children.cycles-pp.___slab_alloc
0.05 ? 51% perf-profile.children.cycles-pp.file_ra_state_init
0.02 ?223% perf-profile.children.cycles-pp.__list_add_valid
0.00 perf-profile.children.cycles-pp.sched_ttwu_pending
0.00 perf-profile.children.cycles-pp.kernfs_fop_release
0.00 perf-profile.children.cycles-pp.proc_single_open
0.46 ? 17% perf-profile.children.cycles-pp.pick_link
0.01 ?223% perf-profile.children.cycles-pp.sched_clock_cpu
0.16 ? 20% perf-profile.children.cycles-pp.proc_pid_get_link
0.11 ? 18% perf-profile.children.cycles-pp.copy_p4d_range
0.03 ?103% perf-profile.children.cycles-pp.brk
0.02 ?146% perf-profile.children.cycles-pp.string
0.00 perf-profile.children.cycles-pp.symbol__is_idle
0.00 perf-profile.children.cycles-pp.__dsos__findnew_link_by_longname_id
0.00 perf-profile.children.cycles-pp.rb_insert_color
0.00 perf-profile.children.cycles-pp.s_start
0.01 ?223% perf-profile.children.cycles-pp.native_sched_clock
0.00 perf-profile.children.cycles-pp.machines__for_each_thread
0.00 perf-profile.children.cycles-pp.rb_first
0.00 perf-profile.children.cycles-pp.kernfs_iop_permission
0.00 perf-profile.children.cycles-pp.mmap__munmap
0.00 perf-profile.children.cycles-pp.perf_mmap__munmap
0.00 perf-profile.children.cycles-pp.kfree
0.00 perf-profile.children.cycles-pp.do_huge_pmd_wp_page
0.08 ? 18% perf-profile.children.cycles-pp.xas_find
0.20 ? 21% perf-profile.children.cycles-pp.perf_event_mmap_event
0.08 ? 82% perf-profile.children.cycles-pp.__perf_mmap__read_init
0.12 ? 21% perf-profile.children.cycles-pp.copy_page_range
0.00 perf-profile.children.cycles-pp.path_put
0.00 perf-profile.children.cycles-pp.maps__clone
0.01 ?223% perf-profile.children.cycles-pp.format_decode
0.01 ?223% perf-profile.children.cycles-pp.slab_pre_alloc_hook
0.00 perf-profile.children.cycles-pp.dsos__findnew_id
0.01 ?223% perf-profile.children.cycles-pp.strlen
0.00 perf-profile.children.cycles-pp.get_kthread_comm
0.01 ?223% perf-profile.children.cycles-pp.meminfo_proc_show
0.01 ?223% perf-profile.children.cycles-pp.rcu_all_qs
0.00 perf-profile.children.cycles-pp.ZSTD_freeDCtx
0.00 perf-profile.children.cycles-pp.comm__free
0.00 perf-profile.children.cycles-pp.access_error
0.00 perf-profile.children.cycles-pp.mpol_shared_policy_lookup
0.00 perf-profile.children.cycles-pp.fclose
0.00 perf-profile.children.cycles-pp.lockref_put_or_lock
0.00 perf-profile.children.cycles-pp.folio_lruvec_lock_irqsave
0.00 perf-profile.children.cycles-pp.sched_move_task
0.00 perf-profile.children.cycles-pp.xyarray__entry
0.00 perf-profile.children.cycles-pp.get_user_cpu_mask
0.00 perf-profile.children.cycles-pp.perf_event_ctx_lock_nested
0.03 ?101% perf-profile.children.cycles-pp.__perf_session__process_decomp_events
0.04 ? 72% perf-profile.children.cycles-pp.anon_vma_clone
0.14 ? 80% perf-profile.children.cycles-pp.find_vma
0.10 ? 22% perf-profile.children.cycles-pp.__access_remote_vm
0.56 ? 22% perf-profile.children.cycles-pp.get_page_from_freelist
0.29 ? 17% perf-profile.children.cycles-pp.__kmem_cache_alloc_node
0.14 ? 28% perf-profile.children.cycles-pp.mas_walk
0.20 ? 33% perf-profile.children.cycles-pp.get_arg_page
0.11 ? 27% perf-profile.children.cycles-pp.cgroup_rstat_updated
0.06 ? 73% perf-profile.children.cycles-pp.evsel__compute_deltas
0.04 ? 71% perf-profile.children.cycles-pp.native_flush_tlb_one_user
0.06 ? 55% perf-profile.children.cycles-pp.page_add_file_rmap
0.00 perf-profile.children.cycles-pp.__rblist__findnew
0.00 perf-profile.children.cycles-pp.number
0.00 perf-profile.children.cycles-pp.save_fpregs_to_fpstate
0.00 perf-profile.children.cycles-pp.try_to_unmap_flush
0.00 perf-profile.children.cycles-pp.inode_init_always
0.00 perf-profile.children.cycles-pp.arch_tlbbatch_flush
0.01 ?223% perf-profile.children.cycles-pp.xas_store
0.00 perf-profile.children.cycles-pp.__mutex_lock
0.01 ?223% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.00 perf-profile.children.cycles-pp.apparmor_file_free_security
0.00 perf-profile.children.cycles-pp.uncharge_batch
0.00 perf-profile.children.cycles-pp.perf_env__arch
0.00 perf-profile.children.cycles-pp.cpu_topology__new
0.00 perf-profile.children.cycles-pp.check_move_unevictable_folios
0.00 perf-profile.children.cycles-pp.getopt_long
0.00 perf-profile.children.cycles-pp.__irqentry_text_end
0.00 perf-profile.children.cycles-pp.__check_heap_object
0.00 perf-profile.children.cycles-pp.check_preempt_wakeup
0.00 perf-profile.children.cycles-pp.__sysconf
0.00 perf-profile.children.cycles-pp.__vmalloc_area_node
0.00 perf-profile.children.cycles-pp.create_pipe_files
0.09 ? 15% perf-profile.children.cycles-pp._copy_to_iter
0.15 ? 74% perf-profile.children.cycles-pp.perf_mmap__read_init
0.09 ? 22% perf-profile.children.cycles-pp.pid_task
0.06 ? 50% perf-profile.children.cycles-pp.exit_notify
0.02 ?141% perf-profile.children.cycles-pp.alloc_new_pud
0.00 perf-profile.children.cycles-pp.__comm_str__findnew
0.00 perf-profile.children.cycles-pp.access
0.00 perf-profile.children.cycles-pp.__static_call_update
0.00 perf-profile.children.cycles-pp.static_key_disable
0.00 perf-profile.children.cycles-pp.arch_static_call_transform
0.00 perf-profile.children.cycles-pp.static_key_disable_cpuslocked
0.00 perf-profile.children.cycles-pp.__static_call_transform
0.00 perf-profile.children.cycles-pp.arch_jump_label_transform_apply
0.00 perf-profile.children.cycles-pp.text_poke_bp
0.00 perf-profile.children.cycles-pp.text_poke_finish
0.00 perf-profile.children.cycles-pp.perf_trace_sched_migrate_task
0.00 perf-profile.children.cycles-pp.can_change_pte_writable
0.00 perf-profile.children.cycles-pp.__get_task_ioprio
0.00 perf-profile.children.cycles-pp.maps__find
0.00 perf-profile.children.cycles-pp.set_cpus_allowed_common
0.00 perf-profile.children.cycles-pp.do_write
0.00 perf-profile.children.cycles-pp.__update_load_avg_se
0.00 perf-profile.children.cycles-pp.perf_tp_event
0.00 perf-profile.children.cycles-pp.acpi_ex_system_memory_space_handler
0.00 perf-profile.children.cycles-pp.task_h_load
0.00 perf-profile.children.cycles-pp.update_cfs_group
0.00 perf-profile.children.cycles-pp.fdarray__filter
0.00 perf-profile.children.cycles-pp.apparmor_file_alloc_security
0.00 perf-profile.children.cycles-pp.page_get_link
0.00 perf-profile.children.cycles-pp.__alloc_pages_bulk
0.00 perf-profile.children.cycles-pp.generic_file_mmap
0.00 perf-profile.children.cycles-pp.acpi_ex_field_datum_io
0.00 perf-profile.children.cycles-pp.acpi_ex_access_region
0.00 perf-profile.children.cycles-pp.acpi_ev_address_space_dispatch
0.00 perf-profile.children.cycles-pp.simple_lookup
0.00 perf-profile.children.cycles-pp.stop_one_cpu
0.00 perf-profile.children.cycles-pp.migrate_task_rq_fair
0.00 perf-profile.children.cycles-pp._IO_file_close_it
0.00 perf-profile.children.cycles-pp.cpu__max_cpu
0.00 perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited_flags
0.00 perf-profile.children.cycles-pp.security_mmap_file
0.00 perf-profile.children.cycles-pp.__put_anon_vma
0.00 perf-profile.children.cycles-pp.pcpu_memcg_post_alloc_hook
0.00 perf-profile.children.cycles-pp.perf_event_comm
0.00 perf-profile.children.cycles-pp.acpi_ds_evaluate_name_path
0.00 perf-profile.children.cycles-pp.acpi_ex_resolve_node_to_value
0.00 perf-profile.children.cycles-pp.acpi_ex_read_data_from_field
0.00 perf-profile.children.cycles-pp.copy_creds
0.00 perf-profile.children.cycles-pp.devkmsg_write
0.00 perf-profile.children.cycles-pp.devkmsg_emit
0.00 perf-profile.children.cycles-pp.lru_add_drain_per_cpu
0.00 perf-profile.children.cycles-pp.remove_migration_pte
0.00 perf-profile.children.cycles-pp.vm_memory_committed
0.00 perf-profile.children.cycles-pp.wake_affine
0.00 perf-profile.children.cycles-pp.memmove
0.00 perf-profile.children.cycles-pp.malloc@plt
0.09 ? 25% perf-profile.children.cycles-pp.page_remove_rmap
0.09 ? 14% perf-profile.children.cycles-pp.seq_puts
0.03 ?100% perf-profile.children.cycles-pp.__folio_throttle_swaprate
0.04 ? 73% perf-profile.children.cycles-pp.anon_vma_fork
0.24 ? 25% perf-profile.children.cycles-pp.enqueue_task_fair
0.22 ? 17% perf-profile.children.cycles-pp.d_hash_and_lookup
0.20 ? 18% perf-profile.children.cycles-pp.vfs_getattr_nosec
0.02 ?142% perf-profile.children.cycles-pp.rcu_report_qs_rdp
0.09 ? 21% perf-profile.children.cycles-pp.aa_get_task_label
0.08 ? 14% perf-profile.children.cycles-pp.__legitimize_path
0.11 ? 15% perf-profile.children.cycles-pp.errseq_sample
0.01 ?223% perf-profile.children.cycles-pp.__evlist__disable
0.01 ?223% perf-profile.children.cycles-pp.kthread_data
0.01 ?223% perf-profile.children.cycles-pp.kmem_cache_free_bulk
0.02 ?149% perf-profile.children.cycles-pp.zero_user_segments
0.36 ? 16% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
0.19 ? 16% perf-profile.children.cycles-pp.generic_fillattr
0.09 ? 11% perf-profile.children.cycles-pp.try_to_unlazy
0.06 ? 88% perf-profile.children.cycles-pp.perf_mmap_to_page
0.09 ? 58% perf-profile.children.cycles-pp.perf_stat_process_counter
0.06 ? 53% perf-profile.children.cycles-pp.gather_stats
0.10 ? 30% perf-profile.children.cycles-pp.create_elf_tables
0.07 ? 47% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.02 ?144% perf-profile.children.cycles-pp.prepend_path
0.03 ?100% perf-profile.children.cycles-pp.mas_push_data
0.03 ?147% perf-profile.children.cycles-pp.expand_downwards
0.02 ?141% perf-profile.children.cycles-pp.m_next
0.09 ? 17% perf-profile.children.cycles-pp.__kmem_cache_free
0.20 ? 18% perf-profile.children.cycles-pp.d_alloc_parallel
0.03 ?100% perf-profile.children.cycles-pp.remove_vma
0.02 ?141% perf-profile.children.cycles-pp._find_next_bit
0.04 ? 71% perf-profile.children.cycles-pp.fsnotify_perm
0.01 ?223% perf-profile.children.cycles-pp.perf_event_mmap_output
0.01 ?223% perf-profile.children.cycles-pp.proc_invalidate_siblings_dcache
0.01 ?223% perf-profile.children.cycles-pp.ptep_clear_flush
0.03 ?100% perf-profile.children.cycles-pp.blk_cgroup_congested
0.16 ? 19% perf-profile.children.cycles-pp.apparmor_ptrace_access_check
0.14 ? 25% perf-profile.children.cycles-pp.__wake_up_common
0.20 ? 20% perf-profile.children.cycles-pp.d_alloc
0.20 ? 18% perf-profile.children.cycles-pp.dequeue_task_fair
0.21 ? 21% perf-profile.children.cycles-pp.may_open
0.10 ? 30% perf-profile.children.cycles-pp.rmqueue_bulk
0.12 ? 28% perf-profile.children.cycles-pp.pte_alloc_one
0.06 ?139% perf-profile.children.cycles-pp.clockevents_program_event
0.01 ?223% perf-profile.children.cycles-pp.alloc_cpumask_var_node
0.02 ?142% perf-profile.children.cycles-pp.inode_to_bdi
0.02 ?142% perf-profile.children.cycles-pp.free_percpu
0.01 ?223% perf-profile.children.cycles-pp.put_prev_task_stop
0.12 ? 27% perf-profile.children.cycles-pp.copy_string_kernel
0.10 ? 59% perf-profile.children.cycles-pp.process_counters
0.07 ? 17% perf-profile.children.cycles-pp.__anon_vma_prepare
0.03 ?103% perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
0.04 ?106% perf-profile.children.cycles-pp.workingset_age_nonresident
0.10 ? 23% perf-profile.children.cycles-pp.__entry_text_start
0.08 ? 16% perf-profile.children.cycles-pp.smp_call_function_single
0.08 ? 12% perf-profile.children.cycles-pp.generic_exec_single
0.07 ? 50% perf-profile.children.cycles-pp.__pte_alloc
0.06 ? 17% perf-profile.children.cycles-pp.copy_pte_range
0.04 ? 72% perf-profile.children.cycles-pp.__pud_alloc
0.05 ? 71% perf-profile.children.cycles-pp.d_path
0.02 ?141% perf-profile.children.cycles-pp.dup_fd
0.02 ?141% perf-profile.children.cycles-pp.do_wp_page
0.01 ?223% perf-profile.children.cycles-pp.mas_next_nentry
0.01 ?223% perf-profile.children.cycles-pp.__anon_vma_interval_tree_remove
0.01 ?223% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.01 ?223% perf-profile.children.cycles-pp.mas_mab_cp
0.01 ?223% perf-profile.children.cycles-pp.perf_event_task
0.01 ?223% perf-profile.children.cycles-pp.copy_page_to_iter
0.01 ?223% perf-profile.children.cycles-pp.mas_store_b_node
0.04 ? 73% perf-profile.children.cycles-pp.do_brk_flags
0.26 ? 24% perf-profile.children.cycles-pp.perf_read
0.07 ? 49% perf-profile.children.cycles-pp.find_extend_vma
0.04 ? 72% perf-profile.children.cycles-pp.__legitimize_mnt
0.01 ?223% perf-profile.children.cycles-pp.mas_wr_store_entry
0.01 ?223% perf-profile.children.cycles-pp.mas_wmb_replace
0.03 ?102% perf-profile.children.cycles-pp._IO_setvbuf
0.03 ?100% perf-profile.children.cycles-pp.lookup_open
0.04 ? 71% perf-profile.children.cycles-pp.xas_start
0.25 ? 15% perf-profile.children.cycles-pp.next_tgid
0.21 ? 19% perf-profile.children.cycles-pp.wq_worker_comm
0.13 ? 17% perf-profile.children.cycles-pp.part_stat_read_all
0.56 ? 39% perf-profile.children.cycles-pp.alloc_empty_file
0.22 ? 29% perf-profile.children.cycles-pp.apparmor_file_permission
0.14 ? 29% perf-profile.children.cycles-pp.strnlen_user
0.13 ? 28% perf-profile.children.cycles-pp.vm_area_alloc
0.05 ? 73% perf-profile.children.cycles-pp.__p4d_alloc
0.06 ? 50% perf-profile.children.cycles-pp.current_time
0.02 ?149% perf-profile.children.cycles-pp.ui_progress__update
0.01 ?223% perf-profile.children.cycles-pp.idle_cpu
0.12 ? 26% perf-profile.children.cycles-pp.padzero
0.09 ? 20% perf-profile.children.cycles-pp.clock_gettime
0.06 ? 49% perf-profile.children.cycles-pp.folio_add_new_anon_rmap
0.11 ? 20% perf-profile.children.cycles-pp.num_to_str
0.05 ?105% perf-profile.children.cycles-pp.atomic_read
0.05 ?105% perf-profile.children.cycles-pp.refcount_read
0.06 ? 63% perf-profile.children.cycles-pp._raw_spin_trylock
0.55 ? 40% perf-profile.children.cycles-pp.__alloc_file
0.24 ? 25% perf-profile.children.cycles-pp.security_file_permission
0.14 ? 13% perf-profile.children.cycles-pp.dequeue_entity
0.09 ? 19% perf-profile.children.cycles-pp.getenv
0.04 ? 72% perf-profile.children.cycles-pp.check_preempt_curr
0.26 ? 18% perf-profile.children.cycles-pp.security_ptrace_access_check
0.14 ? 19% perf-profile.children.cycles-pp.ttwu_do_activate
0.02 ?223% perf-profile.children.cycles-pp._find_next_and_bit
0.03 ?102% perf-profile.children.cycles-pp.printk_get_next_message
0.04 ? 72% perf-profile.children.cycles-pp.prepare_creds
0.03 ?100% perf-profile.children.cycles-pp.cap_ptrace_access_check
0.18 ? 55% perf-profile.children.cycles-pp.mt_find
0.54 ? 22% perf-profile.children.cycles-pp.readn
0.49 ? 28% perf-profile.children.cycles-pp.__folio_alloc
0.42 ? 21% perf-profile.children.cycles-pp.kmem_cache_free
0.10 ? 21% perf-profile.children.cycles-pp.get_zeroed_page
0.03 ?103% perf-profile.children.cycles-pp.mas_destroy
0.08 ? 25% perf-profile.children.cycles-pp.vmf_insert_pfn_prot
0.04 ? 71% perf-profile.children.cycles-pp.__get_obj_cgroup_from_memcg
0.03 ?101% perf-profile.children.cycles-pp.vma_expand
0.02 ?141% perf-profile.children.cycles-pp.__fsnotify_parent
0.02 ?141% perf-profile.children.cycles-pp.mas_find
0.01 ?223% perf-profile.children.cycles-pp.strtoll
0.01 ?223% perf-profile.children.cycles-pp.sched_mm_cid_remote_clear
0.01 ?223% perf-profile.children.cycles-pp.hrtimer_update_next_event
0.01 ?223% perf-profile.children.cycles-pp.timerqueue_add
0.01 ?223% perf-profile.children.cycles-pp.__bitmap_and
0.01 ?223% perf-profile.children.cycles-pp.rcu_gp_cleanup
0.01 ?223% perf-profile.children.cycles-pp.flush_smp_call_function_queue
0.01 ?223% perf-profile.children.cycles-pp.pm_qos_read_value
0.01 ?223% perf-profile.children.cycles-pp.pgd_ctor
0.01 ?223% perf-profile.children.cycles-pp.strnlen
0.01 ?223% perf-profile.children.cycles-pp._IO_file_open
0.01 ?223% perf-profile.children.cycles-pp.make_child
0.01 ?223% perf-profile.children.cycles-pp.free_pgd_range
0.01 ?223% perf-profile.children.cycles-pp.lockref_get_not_dead
0.01 ?223% perf-profile.children.cycles-pp.free_p4d_range
0.01 ?223% perf-profile.children.cycles-pp.free_pud_range
0.01 ?223% perf-profile.children.cycles-pp.swake_up_one
0.01 ?223% perf-profile.children.cycles-pp.xas_create
0.01 ?223% perf-profile.children.cycles-pp.generic_write_checks
0.06 ? 47% perf-profile.children.cycles-pp._exit
0.07 ? 21% perf-profile.children.cycles-pp.security_file_alloc
0.20 ? 21% perf-profile.children.cycles-pp.update_curr
0.29 ? 22% perf-profile.children.cycles-pp.do_mprotect_pkey
0.39 ? 27% perf-profile.children.cycles-pp.__collapse_huge_page_copy
0.14 ? 13% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.07 ? 26% perf-profile.children.cycles-pp.track_pfn_insert
0.03 ?106% perf-profile.children.cycles-pp.file_update_time
0.01 ?223% perf-profile.children.cycles-pp.snprintf
0.01 ?223% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.01 ?223% perf-profile.children.cycles-pp.vfree
0.01 ?223% perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
0.01 ?223% perf-profile.children.cycles-pp.enqueue_hrtimer
0.01 ?223% perf-profile.children.cycles-pp.update_rt_rq_load_avg
0.01 ?223% perf-profile.children.cycles-pp.timerqueue_del
0.01 ?223% perf-profile.children.cycles-pp.uptime_proc_show
0.01 ?223% perf-profile.children.cycles-pp.strscpy_pad
0.01 ?223% perf-profile.children.cycles-pp.irqentry_enter_from_user_mode
0.07 ? 29% perf-profile.children.cycles-pp.thread_group_cputime
0.07 ? 23% perf-profile.children.cycles-pp.lookup_memtype
0.07 ? 23% perf-profile.children.cycles-pp.pat_pagerange_is_ram
0.07 ? 23% perf-profile.children.cycles-pp.walk_system_ram_range
0.07 ? 23% perf-profile.children.cycles-pp.find_next_iomem_res
0.06 ? 54% perf-profile.children.cycles-pp.__pmd_alloc
0.29 ? 21% perf-profile.children.cycles-pp.__x64_sys_mprotect
0.23 ? 18% perf-profile.children.cycles-pp.__lookup_slow
0.19 ? 32% perf-profile.children.cycles-pp.pipe_write
0.05 ? 74% perf-profile.children.cycles-pp.security_file_open
0.03 ?100% perf-profile.children.cycles-pp.__send_signal_locked
0.02 ?223% perf-profile.children.cycles-pp.update_irq_load_avg
0.02 ?141% perf-profile.children.cycles-pp.__vm_area_free
0.02 ?141% perf-profile.children.cycles-pp.set_next_entity
0.01 ?223% perf-profile.children.cycles-pp.irqtime_account_process_tick
0.01 ?223% perf-profile.children.cycles-pp.tcp_orphan_count_sum
0.01 ?223% perf-profile.children.cycles-pp.tcp_orphan_update
0.01 ?223% perf-profile.children.cycles-pp.irqentry_enter
0.03 ?100% perf-profile.children.cycles-pp.proc_pid_permission
0.12 ? 24% perf-profile.children.cycles-pp.mas_split
0.06 ? 23% perf-profile.children.cycles-pp.aa_may_ptrace
0.18 ? 16% perf-profile.children.cycles-pp.perf_iterate_sb
0.13 ? 22% perf-profile.children.cycles-pp.seq_put_decimal_ull_width
0.04 ? 72% perf-profile.children.cycles-pp.vma_link
0.03 ?100% perf-profile.children.cycles-pp.do_faccessat
0.01 ?223% perf-profile.children.cycles-pp.lapic_next_deadline
0.01 ?223% perf-profile.children.cycles-pp.read_tsc
0.01 ?223% perf-profile.children.cycles-pp.tick_irq_enter
0.01 ?223% perf-profile.children.cycles-pp.arch_scale_freq_tick
0.12 ? 33% perf-profile.children.cycles-pp.autoremove_wake_function
0.02 ? 99% perf-profile.children.cycles-pp.acct_collect
0.10 ? 17% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
0.13 ? 24% perf-profile.children.cycles-pp.__kmalloc_node
0.38 ? 13% perf-profile.children.cycles-pp.try_to_wake_up
0.17 ? 24% perf-profile.children.cycles-pp.__fdget_pos
0.16 ? 26% perf-profile.children.cycles-pp.enqueue_entity
0.05 ? 46% perf-profile.children.cycles-pp.arch_dup_task_struct
0.08 ? 15% perf-profile.children.cycles-pp.__perf_event_read
0.08 ? 30% perf-profile.children.cycles-pp.thread_group_cputime_adjusted
0.02 ?223% perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.02 ?223% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
0.02 ?223% perf-profile.children.cycles-pp.run_posix_cpu_timers
0.71 ? 21% perf-profile.children.cycles-pp.sched_exec
0.24 ? 19% perf-profile.children.cycles-pp.mprotect_fixup
0.08 ? 22% perf-profile.children.cycles-pp.__get_free_pages
0.17 ? 25% perf-profile.children.cycles-pp.atime_needs_update
0.06 ? 47% perf-profile.children.cycles-pp.insert_vm_struct
0.02 ?141% perf-profile.children.cycles-pp.get_sigframe
0.02 ?223% perf-profile.children.cycles-pp.get_cpu_device
0.02 ?223% perf-profile.children.cycles-pp.irq_enter_rcu
0.02 ?223% perf-profile.children.cycles-pp.tick_sched_do_timer
0.03 ?102% perf-profile.children.cycles-pp.show_map_vma
0.03 ?102% perf-profile.children.cycles-pp.show_map
0.23 ? 21% perf-profile.children.cycles-pp.do_cow_fault
0.78 ? 21% perf-profile.children.cycles-pp.__x64_sys_vfork
0.54 ? 28% perf-profile.children.cycles-pp.vma_alloc_folio
1.14 ? 10% perf-profile.children.cycles-pp.do_open
0.90 ? 9% perf-profile.children.cycles-pp.do_dentry_open
0.04 ? 71% perf-profile.children.cycles-pp.__pipe
0.06 ? 14% perf-profile.children.cycles-pp.__install_special_mapping
0.02 ?142% perf-profile.children.cycles-pp.vm_unmapped_area
0.02 ?142% perf-profile.children.cycles-pp.x64_setup_rt_frame
0.02 ?142% perf-profile.children.cycles-pp._IO_setb
0.02 ?142% perf-profile.children.cycles-pp.__exit_signal
0.02 ?142% perf-profile.children.cycles-pp.down_read_trylock
0.02 ?223% perf-profile.children.cycles-pp.timekeeping_max_deferment
0.02 ?142% perf-profile.children.cycles-pp.delayed_vfree_work
0.02 ?142% perf-profile.children.cycles-pp.strscpy
0.04 ? 71% perf-profile.children.cycles-pp._IO_link_in
0.05 ? 75% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
0.17 ? 19% perf-profile.children.cycles-pp.load_elf_interp
0.12 ? 29% perf-profile.children.cycles-pp.clear_page_erms
0.15 ? 24% perf-profile.children.cycles-pp.__wake_up_common_lock
0.06 ? 11% perf-profile.children.cycles-pp.map_vdso
0.03 ?100% perf-profile.children.cycles-pp.handle_signal
0.02 ?144% perf-profile.children.cycles-pp._IO_getc
0.02 ?223% perf-profile.children.cycles-pp.__run_timers
0.02 ?223% perf-profile.children.cycles-pp.rcu_pending
0.02 ?223% perf-profile.children.cycles-pp.call_timer_fn
0.04 ?109% perf-profile.children.cycles-pp.__kmalloc
1.33 ? 22% perf-profile.children.cycles-pp.find_idlest_group
0.30 ? 39% perf-profile.children.cycles-pp.show_stat
0.81 ? 21% perf-profile.children.cycles-pp.__alloc_pages
0.34 ? 22% perf-profile.children.cycles-pp.elf_map
0.20 ? 5% perf-profile.children.cycles-pp.malloc
0.08 ? 20% perf-profile.children.cycles-pp.__libc_calloc
0.06 ? 79% perf-profile.children.cycles-pp.kcpustat_cpu_fetch
0.05 ? 48% perf-profile.children.cycles-pp.filemap_read
0.04 ?101% perf-profile.children.cycles-pp.rcu_nocb_try_bypass
0.03 ?103% perf-profile.children.cycles-pp._IO_file_doallocate
0.03 ?101% perf-profile.children.cycles-pp.mas_store_gfp
0.05 ? 49% perf-profile.children.cycles-pp.devkmsg_read
0.08 ? 28% perf-profile.children.cycles-pp.memcg_account_kmem
0.12 ? 9% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.02 ?223% perf-profile.children.cycles-pp.tick_nohz_irq_exit
0.02 ?223% perf-profile.children.cycles-pp.force_qs_rnp
0.04 ? 71% perf-profile.children.cycles-pp.cpuacct_charge
0.04 ? 71% perf-profile.children.cycles-pp.get_pid_task
0.13 ? 22% perf-profile.children.cycles-pp.__ptrace_may_access
0.41 ? 11% perf-profile.children.cycles-pp.ptrace_may_access
0.04 ?107% perf-profile.children.cycles-pp.__perf_read_group_add
0.05 ? 80% perf-profile.children.cycles-pp.pgd_alloc
0.16 ? 24% perf-profile.children.cycles-pp.__memcg_kmem_charge_page
0.60 ? 22% perf-profile.children.cycles-pp.wp_page_copy
0.15 ? 33% perf-profile.children.cycles-pp.__call_rcu_common
0.06 ? 19% perf-profile.children.cycles-pp.put_dec_trunc8
0.42 ? 46% perf-profile.children.cycles-pp.shmem_write_end
0.18 ? 27% perf-profile.children.cycles-pp.mas_wr_bnode
1.31 ? 22% perf-profile.children.cycles-pp.update_sg_wakeup_stats
0.08 ? 22% perf-profile.children.cycles-pp.release_task
0.06 ? 49% perf-profile.children.cycles-pp.touch_atime
0.03 ?223% perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.25 ? 57% perf-profile.children.cycles-pp.dup_task_struct
0.12 ? 25% perf-profile.children.cycles-pp.do_open_execat
0.06 ? 50% perf-profile.children.cycles-pp.path_init
0.09 ? 19% perf-profile.children.cycles-pp.wait_task_zombie
0.03 ?103% perf-profile.children.cycles-pp.open_exec
0.10 ?143% perf-profile.children.cycles-pp.__intel_pmu_enable_all
0.13 ? 17% perf-profile.children.cycles-pp.down_write
0.41 ? 20% perf-profile.children.cycles-pp.__split_vma
0.03 ?100% perf-profile.children.cycles-pp.__collapse_huge_page_copy_succeeded
0.03 ?223% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.30 ? 29% perf-profile.children.cycles-pp.write
0.17 ? 24% perf-profile.children.cycles-pp.perf_evlist__poll
0.04 ? 71% perf-profile.children.cycles-pp.memset@plt
0.35 ? 19% perf-profile.children.cycles-pp.proc_task_name
0.06 ? 47% perf-profile.children.cycles-pp.get_unmapped_area
0.12 ?172% perf-profile.children.cycles-pp.ktime_get
0.14 ? 28% perf-profile.children.cycles-pp.file_free_rcu
0.32 ? 25% perf-profile.children.cycles-pp.__mmap
0.12 ?122% perf-profile.children.cycles-pp.perf_rotate_context
0.20 ? 28% perf-profile.children.cycles-pp.start_thread
0.20 ? 28% perf-profile.children.cycles-pp.perf_evlist__poll_thread
0.05 ? 49% perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
0.27 ? 17% perf-profile.children.cycles-pp.getname_flags
0.06 ?223% perf-profile.children.cycles-pp.kmem_cache_alloc_node
0.25 ? 35% perf-profile.children.cycles-pp.charge_memcg
0.12 ? 20% perf-profile.children.cycles-pp.mas_preallocate
0.18 ? 16% perf-profile.children.cycles-pp.strncpy_from_user
0.04 ? 73% perf-profile.children.cycles-pp.sscanf
0.04 ? 73% perf-profile.children.cycles-pp.ktime_get_update_offsets_now
0.08 ? 48% perf-profile.children.cycles-pp.get_idle_time
0.05 ?159% perf-profile.children.cycles-pp.wait_for_completion_state
0.05 ?223% perf-profile.children.cycles-pp.__update_blocked_fair
0.83 ? 20% perf-profile.children.cycles-pp.__vfork
0.13 ? 21% perf-profile.children.cycles-pp.mas_alloc_nodes
0.06 ?112% perf-profile.children.cycles-pp.wait_for_partner
0.14 ? 23% perf-profile.children.cycles-pp.vma_complete
0.14 ? 67% perf-profile.children.cycles-pp.__wait_for_common
0.29 ? 17% perf-profile.children.cycles-pp.__slab_free
0.09 ?160% perf-profile.children.cycles-pp.balance_fair
0.78 ? 21% perf-profile.children.cycles-pp.ksys_mmap_pgoff
0.12 ?109% perf-profile.children.cycles-pp.obj_cgroup_charge
0.06 ? 78% perf-profile.children.cycles-pp.drm_gem_check_release_pagevec
0.06 ? 65% perf-profile.children.cycles-pp.get_cpu_sleep_time_us
0.07 ? 90% perf-profile.children.cycles-pp.fifo_open
0.19 ? 59% perf-profile.children.cycles-pp.try_charge_memcg
7.99 ? 51% perf-profile.children.cycles-pp.__libc_start_main
7.98 ? 51% perf-profile.children.cycles-pp.main
7.98 ? 51% perf-profile.children.cycles-pp.run_builtin
0.06 ?223% perf-profile.children.cycles-pp.tick_nohz_next_event
0.19 ?123% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
0.20 ? 28% perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
0.25 ? 8% perf-profile.children.cycles-pp.rcu_cblist_dequeue
0.10 ? 24% perf-profile.children.cycles-pp.kmem_cache_alloc_bulk
0.08 ?141% perf-profile.children.cycles-pp.schedule_tail
0.07 ? 53% perf-profile.children.cycles-pp.__snprintf_chk
6.04 ? 15% perf-profile.children.cycles-pp.read
0.09 ? 22% perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
0.15 ? 90% perf-profile.children.cycles-pp.schedule_idle
0.08 ?168% perf-profile.children.cycles-pp.run_rebalance_domains
0.08 ? 17% perf-profile.children.cycles-pp.copyout
0.74 ? 86% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.08 ? 20% perf-profile.children.cycles-pp._IO_fread
0.08 ?193% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
1.96 ? 24% perf-profile.children.cycles-pp.process_interval
0.23 ?128% perf-profile.children.cycles-pp.__pthread_disable_asynccancel
0.09 ?165% perf-profile.children.cycles-pp.update_blocked_averages
1.96 ? 24% perf-profile.children.cycles-pp.dispatch_events
2.72 ? 15% perf-profile.children.cycles-pp.proc_single_show
2.71 ? 15% perf-profile.children.cycles-pp.do_task_stat
0.85 ? 14% perf-profile.children.cycles-pp.rcu_do_batch
1.85 ? 24% perf-profile.children.cycles-pp.read_counters
0.10 ? 32% perf-profile.children.cycles-pp.cgroup_rstat_flush
0.14 ?169% perf-profile.children.cycles-pp.find_busiest_queue
1.01 ? 20% perf-profile.children.cycles-pp.mmap_region
0.10 ? 35% perf-profile.children.cycles-pp.get_signal
0.93 ? 14% perf-profile.children.cycles-pp.rcu_core
0.46 ? 9% perf-profile.children.cycles-pp.__mem_cgroup_charge
1.11 ? 19% perf-profile.children.cycles-pp.vm_mmap_pgoff
0.74 ? 18% perf-profile.children.cycles-pp.dup_mm
1.23 ? 19% perf-profile.children.cycles-pp.__do_sys_clone
0.46 ? 84% perf-profile.children.cycles-pp.perf_mmap__read_head
0.46 ? 83% perf-profile.children.cycles-pp.ring_buffer_read_head
0.12 ?126% perf-profile.children.cycles-pp.do_task_dead
0.16 ? 16% perf-profile.children.cycles-pp.mas_wr_modify
1.09 ? 19% perf-profile.children.cycles-pp.do_mmap
0.91 ? 83% perf-profile.children.cycles-pp.hrtimer_interrupt
0.14 ? 19% perf-profile.children.cycles-pp.mas_wr_node_store
0.15 ? 33% perf-profile.children.cycles-pp.arch_do_signal_or_restart
2.00 ? 18% perf-profile.children.cycles-pp.kernel_clone
0.94 ? 82% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.25 ? 63% perf-profile.children.cycles-pp.evlist__id2evsel
0.36 ? 21% perf-profile.children.cycles-pp.mas_store_prealloc
0.61 ? 19% perf-profile.children.cycles-pp.do_anonymous_page
1.20 ? 19% perf-profile.children.cycles-pp.copy_process
0.14 ? 36% perf-profile.children.cycles-pp.update_rq_clock_task
3.91 ? 35% perf-profile.children.cycles-pp.kthread
0.32 ? 49% perf-profile.children.cycles-pp.pipe_read
0.96 ? 24% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.16 ? 18% perf-profile.children.cycles-pp.check_heap_object
0.82 ? 35% perf-profile.children.cycles-pp.finish_task_switch
0.82 ? 34% perf-profile.children.cycles-pp.__percpu_counter_sum
0.89 ? 32% perf-profile.children.cycles-pp.__mmdrop
0.18 ?185% perf-profile.children.cycles-pp.menu_select
0.19 ? 29% perf-profile.children.cycles-pp.mm_account_fault
1.96 ? 24% perf-profile.children.cycles-pp.cmd_stat
1.37 ? 19% perf-profile.children.cycles-pp.__libc_fork
4.00 ? 37% perf-profile.children.cycles-pp.ret_from_fork
0.97 ? 24% perf-profile.children.cycles-pp.alloc_bprm
0.22 ? 39% perf-profile.children.cycles-pp.mm_init_cid
0.24 ? 28% perf-profile.children.cycles-pp.__rmqueue_pcplist
0.48 ? 60% perf-profile.children.cycles-pp.kernel_wait4
0.26 ?164% perf-profile.children.cycles-pp.arch_call_rest_init
0.26 ?164% perf-profile.children.cycles-pp.start_kernel
0.26 ?164% perf-profile.children.cycles-pp.rest_init
0.90 ? 25% perf-profile.children.cycles-pp.exec_mmap
0.48 ? 60% perf-profile.children.cycles-pp.do_wait
0.26 ?164% perf-profile.children.cycles-pp.x86_64_start_kernel
0.26 ?164% perf-profile.children.cycles-pp.x86_64_start_reservations
0.26 ? 30% perf-profile.children.cycles-pp.mtree_range_walk
1.15 ? 21% perf-profile.children.cycles-pp.pcpu_alloc
1.01 ? 24% perf-profile.children.cycles-pp.begin_new_exec
0.28 ? 19% perf-profile.children.cycles-pp.drm_fbdev_generic_damage_blit_real
0.29 ? 21% perf-profile.children.cycles-pp.activate_task
1.96 ? 22% perf-profile.children.cycles-pp.exec_binprm
1.95 ? 22% perf-profile.children.cycles-pp.search_binary_handler
0.60 ? 61% perf-profile.children.cycles-pp.evsel__read_counter
0.56 ? 72% perf-profile.children.cycles-pp.rebalance_domains
1.94 ? 22% perf-profile.children.cycles-pp.load_elf_binary
0.31 ? 29% perf-profile.children.cycles-pp.drm_gem_shmem_get_pages
0.32 ? 22% perf-profile.children.cycles-pp.__wp_page_copy_user
0.66 ? 23% perf-profile.children.cycles-pp.copy_mc_enhanced_fast_string
2.82 ? 21% perf-profile.children.cycles-pp.bprm_execve
0.62 ? 50% perf-profile.children.cycles-pp.wait4
1.28 ? 22% perf-profile.children.cycles-pp.mm_init
0.36 ? 28% perf-profile.children.cycles-pp.drm_gem_shmem_vmap_locked
0.37 ? 29% perf-profile.children.cycles-pp.drm_gem_shmem_object_vmap
0.40 ?198% perf-profile.children.cycles-pp.intel_idle
0.48 ? 11% perf-profile.children.cycles-pp.__memcpy
1.41 ? 22% perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
1.66 ? 34% perf-profile.children.cycles-pp.__do_softirq
4.11 ? 22% perf-profile.children.cycles-pp.execve
4.12 ? 22% perf-profile.children.cycles-pp.do_execveat_common
4.13 ? 22% perf-profile.children.cycles-pp.__x64_sys_execve
0.60 ?175% perf-profile.children.cycles-pp.rcu_gp_fqs_loop
1.32 ? 50% perf-profile.children.cycles-pp.__irq_exit_rcu
0.62 ?169% perf-profile.children.cycles-pp.schedule_timeout
0.92 ? 20% perf-profile.children.cycles-pp.__memset
0.66 ?201% perf-profile.children.cycles-pp.intel_idle_xstate
0.67 ?172% perf-profile.children.cycles-pp.rcu_gp_kthread
2.35 ? 21% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
2.14 ? 22% perf-profile.children.cycles-pp.exit_to_user_mode_loop
1.69 ? 20% perf-profile.children.cycles-pp.task_work_run
2.35 ? 63% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
2.43 ? 67% perf-profile.children.cycles-pp.schedule
1.35 ?143% perf-profile.children.cycles-pp.pick_next_task_fair
1.35 ?154% perf-profile.children.cycles-pp.newidle_balance
1.52 ?126% perf-profile.children.cycles-pp.update_sg_lb_stats
1.59 ?126% perf-profile.children.cycles-pp.update_sd_lb_stats
1.60 ?126% perf-profile.children.cycles-pp.find_busiest_group
1.08 ? 15% perf-profile.children.cycles-pp.drm_fbdev_generic_helper_fb_dirty
2.79 ? 68% perf-profile.children.cycles-pp.__schedule
5.18 ? 27% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
1.13 ? 20% perf-profile.children.cycles-pp.task_mm_cid_work
1.84 ?126% perf-profile.children.cycles-pp.load_balance
2.26 ?192% perf-profile.children.cycles-pp.cpuidle_enter_state
2.05 ? 58% perf-profile.children.cycles-pp.copyin
2.26 ?192% perf-profile.children.cycles-pp.cpuidle_enter
2.42 ?187% perf-profile.children.cycles-pp.start_secondary
2.48 ?191% perf-profile.children.cycles-pp.cpuidle_idle_call
2.67 ?184% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
2.67 ?184% perf-profile.children.cycles-pp.cpu_startup_entry
2.67 ?184% perf-profile.children.cycles-pp.do_idle
59.07 ? 11% perf-profile.children.cycles-pp.__cmd_record
51.98 ? 8% perf-profile.children.cycles-pp.record__finish_output
51.98 ? 8% perf-profile.children.cycles-pp.perf_session__process_events
51.95 ? 8% perf-profile.children.cycles-pp.reader__read_event
50.62 ? 10% perf-profile.children.cycles-pp.process_simple
49.86 ? 11% perf-profile.children.cycles-pp.ordered_events__queue
50.04 ? 10% perf-profile.children.cycles-pp.queue_event
1.16 ?147% perf-profile.self.cycles-pp.copy_page
0.00 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.42 ?223% perf-profile.self.cycles-pp.evlist__parse_sample
0.12 ?223% perf-profile.self.cycles-pp.evsel__parse_sample
0.17 ?223% perf-profile.self.cycles-pp.__ordered_events__flush
0.00 perf-profile.self.cycles-pp.memcpy_erms
0.00 perf-profile.self.cycles-pp.memset_erms
0.12 ?223% perf-profile.self.cycles-pp.machine__findnew_thread
0.31 ? 59% perf-profile.self.cycles-pp.reader__read_event
0.44 ? 67% perf-profile.self.cycles-pp.evlist__parse_sample_timestamp
1.26 ? 17% perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
0.06 ?118% perf-profile.self.cycles-pp.smp_call_function_many_cond
0.00 perf-profile.self.cycles-pp.rb_erase
0.43 ? 12% perf-profile.self.cycles-pp.io_serial_in
0.01 ?223% perf-profile.self.cycles-pp.mt_find
0.21 ? 36% perf-profile.self.cycles-pp.perf_poll
0.03 ?223% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.52 ? 23% perf-profile.self.cycles-pp.next_uptodate_page
0.01 ?223% perf-profile.self.cycles-pp.mas_walk
0.51 ? 22% perf-profile.self.cycles-pp.zap_pte_range
0.00 perf-profile.self.cycles-pp.common_perm_cond
0.16 ? 59% perf-profile.self.cycles-pp.evsel__parse_sample_timestamp
0.26 ? 12% perf-profile.self.cycles-pp.__fget_light
0.01 ?223% perf-profile.self.cycles-pp.llist_add_batch
0.00 perf-profile.self.cycles-pp.rb_next
0.03 ?111% perf-profile.self.cycles-pp.free_unref_page_prepare
0.27 ? 19% perf-profile.self.cycles-pp.__list_del_entry_valid
0.13 ? 18% perf-profile.self.cycles-pp.mutex_lock
0.36 ? 28% perf-profile.self.cycles-pp.native_irq_return_iret
0.41 ? 21% perf-profile.self.cycles-pp.fold_vm_numa_events
0.00 perf-profile.self.cycles-pp.update_rq_clock
0.01 ?223% perf-profile.self.cycles-pp.handle_mm_fault
0.26 ? 22% perf-profile.self.cycles-pp._dl_addr
0.33 ? 23% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.00 perf-profile.self.cycles-pp.task_numa_compare
0.00 perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.00 perf-profile.self.cycles-pp.__lock_task_sighand
0.00 perf-profile.self.cycles-pp.find_idlest_cpu
0.25 ? 26% perf-profile.self.cycles-pp.filemap_get_entry
0.00 perf-profile.self.cycles-pp.llist_reverse_order
0.00 perf-profile.self.cycles-pp.__free_pages
0.41 ? 14% perf-profile.self.cycles-pp._raw_spin_lock
0.17 ? 15% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.00 perf-profile.self.cycles-pp.map_id_up
0.02 ?142% perf-profile.self.cycles-pp.folio_unlock
0.13 ? 33% perf-profile.self.cycles-pp.__count_memcg_events
0.67 ? 18% perf-profile.self.cycles-pp.__d_lookup_rcu
0.02 ?142% perf-profile.self.cycles-pp.alloc_vmap_area
0.03 ?101% perf-profile.self.cycles-pp.vsnprintf
0.01 ?223% perf-profile.self.cycles-pp.thread__find_map
0.01 ?223% perf-profile.self.cycles-pp.flush_tlb_func
0.16 ? 31% perf-profile.self.cycles-pp.___perf_sw_event
0.29 ? 22% perf-profile.self.cycles-pp.all_vm_events
0.00 perf-profile.self.cycles-pp.thread__get
0.07 ? 50% perf-profile.self.cycles-pp.__mod_node_page_state
0.02 ?142% perf-profile.self.cycles-pp.memchr_inv
0.01 ?223% perf-profile.self.cycles-pp.change_pte_range
0.04 ? 75% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.22 ? 17% perf-profile.self.cycles-pp.__alloc_file
0.20 ? 12% perf-profile.self.cycles-pp.__d_lookup
0.25 ? 19% perf-profile.self.cycles-pp.memcpy_toio
0.02 ?142% perf-profile.self.cycles-pp.vma_alloc_folio
0.02 ?146% perf-profile.self.cycles-pp.fput
0.13 ? 25% perf-profile.self.cycles-pp.update_load_avg
0.15 ? 25% perf-profile.self.cycles-pp.vma_interval_tree_remove
0.02 ?141% perf-profile.self.cycles-pp.lock_vma_under_rcu
0.00 perf-profile.self.cycles-pp.__mod_memcg_state
0.01 ?223% perf-profile.self.cycles-pp.__perf_event_read
0.03 ?102% perf-profile.self.cycles-pp.__check_object_size
0.16 ? 19% perf-profile.self.cycles-pp.sync_regs
0.12 ? 19% perf-profile.self.cycles-pp.lookup_fast
0.10 ? 15% perf-profile.self.cycles-pp.io_serial_out
0.00 perf-profile.self.cycles-pp.ttwu_do_activate
0.04 ?107% perf-profile.self.cycles-pp.kstat_irqs_usr
0.31 ? 15% perf-profile.self.cycles-pp.gather_pte_stats
0.08 ? 29% perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush
0.00 perf-profile.self.cycles-pp.io__get_char
0.00 perf-profile.self.cycles-pp.prepare_task_switch
0.00 perf-profile.self.cycles-pp._compound_head
0.00 perf-profile.self.cycles-pp.step_into
0.06 ? 47% perf-profile.self.cycles-pp.perf_exclude_event
0.02 ?141% perf-profile.self.cycles-pp.__free_one_page
0.07 ? 15% perf-profile.self.cycles-pp.__switch_to
0.00 perf-profile.self.cycles-pp.map__process_kallsym_symbol
0.00 perf-profile.self.cycles-pp.generic_permission
0.32 ? 18% perf-profile.self.cycles-pp.kmem_cache_alloc
0.16 ? 16% perf-profile.self.cycles-pp.xas_load
0.00 perf-profile.self.cycles-pp.perf_session__process_events
0.00 perf-profile.self.cycles-pp.irqtime_account_irq
0.23 ? 20% perf-profile.self.cycles-pp.vma_interval_tree_insert
0.01 ?223% perf-profile.self.cycles-pp.handle_pte_fault
0.00 perf-profile.self.cycles-pp.read
0.00 perf-profile.self.cycles-pp.proc_pid_readlink
0.00 perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.06 ? 48% perf-profile.self.cycles-pp.vfs_read
0.00 perf-profile.self.cycles-pp.perf_swevent_event
0.00 perf-profile.self.cycles-pp.get_cpu_idle_time_us
0.00 perf-profile.self.cycles-pp.events_stats__inc
0.00 perf-profile.self.cycles-pp.kallsyms_expand_symbol
0.00 perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.08 ? 48% perf-profile.self.cycles-pp.free_swap_cache
0.01 ?223% perf-profile.self.cycles-pp.update_numa_stats
0.01 ?223% perf-profile.self.cycles-pp.acpi_ns_search_and_enter
0.00 perf-profile.self.cycles-pp.lru_add_drain_cpu
0.00 perf-profile.self.cycles-pp.kernfs_dop_revalidate
0.12 ? 24% perf-profile.self.cycles-pp.mod_objcg_state
0.04 ? 72% perf-profile.self.cycles-pp.arch_irq_stat_cpu
0.03 ?102% perf-profile.self.cycles-pp.vmap_pages_pud_range
0.01 ?223% perf-profile.self.cycles-pp.tlb_is_not_lazy
0.25 ? 83% perf-profile.self.cycles-pp.record__mmap_read_evlist
0.07 ? 76% perf-profile.self.cycles-pp.perf_mmap_fault
0.16 ? 20% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
0.00 perf-profile.self.cycles-pp.__dsos__find
0.08 ? 49% perf-profile.self.cycles-pp.get_page_from_freelist
0.00 perf-profile.self.cycles-pp.security_vm_enough_memory_mm
0.00 perf-profile.self.cycles-pp.mntput_no_expire
0.00 perf-profile.self.cycles-pp.mem_cgroup_from_task
0.05 ?114% perf-profile.self.cycles-pp.generic_perform_write
0.00 perf-profile.self.cycles-pp.readdir64
0.11 ? 47% perf-profile.self.cycles-pp.aa_file_perm
0.16 ? 25% perf-profile.self.cycles-pp.__kmem_cache_alloc_node
0.10 ? 21% perf-profile.self.cycles-pp.percpu_counter_add_batch
0.07 ? 14% perf-profile.self.cycles-pp.__switch_to_asm
0.01 ?223% perf-profile.self.cycles-pp.cap_vm_enough_memory
0.00 perf-profile.self.cycles-pp.hrtimer_active
0.00 perf-profile.self.cycles-pp.map__put
0.00 perf-profile.self.cycles-pp.mntput
0.15 ? 20% perf-profile.self.cycles-pp.pid_nr_ns
0.10 ? 48% perf-profile.self.cycles-pp.do_user_addr_fault
0.01 ?223% perf-profile.self.cycles-pp.blk_cgroup_congested
0.00 perf-profile.self.cycles-pp.____machine__findnew_thread
0.00 perf-profile.self.cycles-pp.__get_vma_policy
0.00 perf-profile.self.cycles-pp.task_prio
0.00 perf-profile.self.cycles-pp.schedule
0.00 perf-profile.self.cycles-pp.task_tick_fair
0.00 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.11 ? 17% perf-profile.self.cycles-pp.proc_fill_cache
0.05 ?102% perf-profile.self.cycles-pp.__perf_mmap__read_init
0.17 ? 29% perf-profile.self.cycles-pp.blk_mq_queue_tag_busy_iter
0.00 perf-profile.self.cycles-pp.__x2apic_send_IPI_mask
0.00 perf-profile.self.cycles-pp.proc_single_open
0.01 ?223% perf-profile.self.cycles-pp.__list_add_valid
0.10 ? 24% perf-profile.self.cycles-pp.try_charge_memcg
0.05 ? 74% perf-profile.self.cycles-pp.evsel__compute_deltas
0.07 ? 50% perf-profile.self.cycles-pp.__mod_lruvec_page_state
0.03 ? 70% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.02 ?142% perf-profile.self.cycles-pp.enqueue_task_fair
0.00 perf-profile.self.cycles-pp.kallsyms__parse
0.00 perf-profile.self.cycles-pp.module_get_kallsym
0.00 perf-profile.self.cycles-pp.__dsos__findnew_link_by_longname_id
0.01 ?223% perf-profile.self.cycles-pp.native_sched_clock
0.01 ?223% perf-profile.self.cycles-pp.exit_to_user_mode_loop
0.00 perf-profile.self.cycles-pp.perf_event_read
0.00 perf-profile.self.cycles-pp.kfree
0.03 ?100% perf-profile.self.cycles-pp.charge_memcg
0.02 ?144% perf-profile.self.cycles-pp.string
0.02 ? 99% perf-profile.self.cycles-pp.unlink_anon_vmas
0.00 perf-profile.self.cycles-pp.rb_insert_color
0.00 perf-profile.self.cycles-pp.format_decode
0.01 ?223% perf-profile.self.cycles-pp.strlen
0.00 perf-profile.self.cycles-pp.get_kthread_comm
0.00 perf-profile.self.cycles-pp.rb_first
0.00 perf-profile.self.cycles-pp.perf_evsel__read
0.00 perf-profile.self.cycles-pp.__perf_sw_event
0.00 perf-profile.self.cycles-pp.__cond_resched
0.00 perf-profile.self.cycles-pp.anon_vma_clone
0.00 perf-profile.self.cycles-pp.rw_verify_area
0.00 perf-profile.self.cycles-pp.access_error
0.00 perf-profile.self.cycles-pp.mpol_shared_policy_lookup
0.00 perf-profile.self.cycles-pp.fclose
0.00 perf-profile.self.cycles-pp.xyarray__entry
0.00 perf-profile.self.cycles-pp.perf_event_ctx_lock_nested
0.03 ?101% perf-profile.self.cycles-pp.__perf_session__process_decomp_events
0.10 ? 14% perf-profile.self.cycles-pp.release_pages
0.10 ? 16% perf-profile.self.cycles-pp.update_curr
0.06 ? 45% perf-profile.self.cycles-pp.unmap_single_vma
0.04 ? 71% perf-profile.self.cycles-pp.native_flush_tlb_one_user
0.00 perf-profile.self.cycles-pp.io__get_hex
0.00 perf-profile.self.cycles-pp.map__new
0.00 perf-profile.self.cycles-pp.number
0.00 perf-profile.self.cycles-pp.save_fpregs_to_fpstate
0.01 ?223% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.00 perf-profile.self.cycles-pp.apparmor_file_free_security
0.00 perf-profile.self.cycles-pp.perf_env__arch
0.00 perf-profile.self.cycles-pp.getname_flags
0.00 perf-profile.self.cycles-pp.shmem_alloc_and_acct_folio
0.00 perf-profile.self.cycles-pp.filp_close
0.00 perf-profile.self.cycles-pp.__irqentry_text_end
0.00 perf-profile.self.cycles-pp.put_prev_task_stop
0.12 ? 16% perf-profile.self.cycles-pp.task_dump_owner
0.10 ? 33% perf-profile.self.cycles-pp.process_simple
0.00 perf-profile.self.cycles-pp.update_iter
0.01 ?223% perf-profile.self.cycles-pp.kmem_cache_alloc_lru
0.00 perf-profile.self.cycles-pp.__vm_enough_memory
0.00 perf-profile.self.cycles-pp.__comm_str__findnew
0.00 perf-profile.self.cycles-pp.may_open
0.00 perf-profile.self.cycles-pp.obj_cgroup_uncharge_pages
0.01 ?223% perf-profile.self.cycles-pp.mutex_unlock
0.00 perf-profile.self.cycles-pp.perf_trace_sched_migrate_task
0.00 perf-profile.self.cycles-pp.m_next
0.00 perf-profile.self.cycles-pp.__update_load_avg_se
0.00 perf-profile.self.cycles-pp.asm_exc_page_fault
0.00 perf-profile.self.cycles-pp.ksys_read
0.00 perf-profile.self.cycles-pp.do_mmap
0.00 perf-profile.self.cycles-pp.finish_task_switch
0.00 perf-profile.self.cycles-pp.asm_sysvec_call_function
0.00 perf-profile.self.cycles-pp.try_to_wake_up
0.00 perf-profile.self.cycles-pp.perf_event_mmap_event
0.00 perf-profile.self.cycles-pp.do_set_pte
0.00 perf-profile.self.cycles-pp.kmalloc_trace
0.00 perf-profile.self.cycles-pp.add_wait_queue
0.00 perf-profile.self.cycles-pp.__do_set_cpus_allowed
0.00 perf-profile.self.cycles-pp.select_task_rq
0.00 perf-profile.self.cycles-pp.vfs_readlink
0.00 perf-profile.self.cycles-pp.file_ra_state_init
0.00 perf-profile.self.cycles-pp.lockref_put_or_lock
0.00 perf-profile.self.cycles-pp.rcu_all_qs
0.00 perf-profile.self.cycles-pp.__check_heap_object
0.00 perf-profile.self.cycles-pp.check_preempt_wakeup
0.00 perf-profile.self.cycles-pp.acpi_ex_system_memory_space_handler
0.00 perf-profile.self.cycles-pp.task_h_load
0.00 perf-profile.self.cycles-pp.update_cfs_group
0.00 perf-profile.self.cycles-pp.fdarray__filter
0.00 perf-profile.self.cycles-pp.apparmor_file_alloc_security
0.00 perf-profile.self.cycles-pp.migrate_task_rq_fair
0.00 perf-profile.self.cycles-pp._IO_file_close_it
0.00 perf-profile.self.cycles-pp.cpu__max_cpu
0.00 perf-profile.self.cycles-pp.malloc@plt
0.08 ? 22% perf-profile.self.cycles-pp.pid_task
0.05 ? 46% perf-profile.self.cycles-pp.wq_worker_comm
0.10 ? 15% perf-profile.self.cycles-pp.errseq_sample
0.02 ?142% perf-profile.self.cycles-pp.seq_read_iter
0.02 ?141% perf-profile.self.cycles-pp.folio_batch_move_lru
0.01 ?223% perf-profile.self.cycles-pp.kthread_data
0.01 ?223% perf-profile.self.cycles-pp.shmem_get_folio_gfp
0.03 ?103% perf-profile.self.cycles-pp.page_remove_rmap
0.06 ? 88% perf-profile.self.cycles-pp.perf_mmap_to_page
0.10 ? 26% perf-profile.self.cycles-pp.cgroup_rstat_updated
0.09 ? 21% perf-profile.self.cycles-pp.aa_get_task_label
0.06 ? 55% perf-profile.self.cycles-pp.gather_stats
0.10 ? 18% perf-profile.self.cycles-pp.__virt_addr_valid
0.04 ? 75% perf-profile.self.cycles-pp.thread_group_cputime
0.11 ? 25% perf-profile.self.cycles-pp.do_syscall_64
0.07 ? 47% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.02 ?142% perf-profile.self.cycles-pp.vfs_write
0.01 ?223% perf-profile.self.cycles-pp.num_to_str
0.01 ?223% perf-profile.self.cycles-pp.___slab_alloc
0.05 ? 86% perf-profile.self.cycles-pp._raw_spin_trylock
0.03 ?100% perf-profile.self.cycles-pp.security_ptrace_access_check
0.02 ?142% perf-profile.self.cycles-pp.inode_to_bdi
0.01 ?223% perf-profile.self.cycles-pp.rcu_nocb_try_bypass
0.01 ?223% perf-profile.self.cycles-pp._find_next_bit
0.21 ? 35% perf-profile.self.cycles-pp.read_counters
0.04 ?106% perf-profile.self.cycles-pp.workingset_age_nonresident
0.04 ?102% perf-profile.self.cycles-pp.check_move_unevictable_pages
0.13 ? 15% perf-profile.self.cycles-pp.part_stat_read_all
0.10 ? 16% perf-profile.self.cycles-pp.filemap_map_pages
0.04 ? 71% perf-profile.self.cycles-pp.vm_area_dup
0.02 ?142% perf-profile.self.cycles-pp.cfree
0.01 ?223% perf-profile.self.cycles-pp.mas_next_nentry
0.01 ?223% perf-profile.self.cycles-pp.shmem_write_begin
0.01 ?223% perf-profile.self.cycles-pp.switch_fpu_return
0.01 ?223% perf-profile.self.cycles-pp.__anon_vma_interval_tree_remove
0.01 ?223% perf-profile.self.cycles-pp.kmem_cache_free_bulk
0.01 ?223% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.14 ? 31% perf-profile.self.cycles-pp.strnlen_user
0.01 ?223% perf-profile.self.cycles-pp.mmap_region
0.04 ? 73% perf-profile.self.cycles-pp.__legitimize_mnt
0.04 ? 71% perf-profile.self.cycles-pp.xas_start
0.19 ? 15% perf-profile.self.cycles-pp.generic_fillattr
0.03 ?100% perf-profile.self.cycles-pp.enqueue_entity
0.02 ?149% perf-profile.self.cycles-pp.ui_progress__update
0.01 ?223% perf-profile.self.cycles-pp.idle_cpu
0.05 ?105% perf-profile.self.cycles-pp.atomic_read
0.05 ? 13% perf-profile.self.cycles-pp.perf_read
0.02 ?141% perf-profile.self.cycles-pp.cap_ptrace_access_check
0.33 ? 19% perf-profile.self.cycles-pp.inode_permission
0.21 ? 37% perf-profile.self.cycles-pp.__handle_mm_fault
0.11 ? 4% perf-profile.self.cycles-pp.malloc
0.09 ? 47% perf-profile.self.cycles-pp.__alloc_pages
0.02 ?223% perf-profile.self.cycles-pp._find_next_and_bit
0.05 ? 71% perf-profile.self.cycles-pp.rmqueue_bulk
0.04 ? 70% perf-profile.self.cycles-pp.pick_link
0.07 ? 19% perf-profile.self.cycles-pp.find_next_iomem_res
0.04 ? 72% perf-profile.self.cycles-pp.__get_obj_cgroup_from_memcg
0.02 ?141% perf-profile.self.cycles-pp.__fsnotify_parent
0.01 ?223% perf-profile.self.cycles-pp.free_percpu
0.01 ?223% perf-profile.self.cycles-pp._IO_setb
0.01 ?223% perf-profile.self.cycles-pp.strtoll
0.01 ?223% perf-profile.self.cycles-pp.sched_mm_cid_remote_clear
0.01 ?223% perf-profile.self.cycles-pp.set_task_cpu
0.01 ?223% perf-profile.self.cycles-pp.newidle_balance
0.01 ?223% perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
0.01 ?223% perf-profile.self.cycles-pp.__do_softirq
0.01 ?223% perf-profile.self.cycles-pp.update_process_times
0.01 ?223% perf-profile.self.cycles-pp.__bitmap_and
0.01 ?223% perf-profile.self.cycles-pp.do_anonymous_page
0.01 ?223% perf-profile.self.cycles-pp.down_read_trylock
0.01 ?223% perf-profile.self.cycles-pp.pgd_ctor
0.01 ?223% perf-profile.self.cycles-pp.strnlen
0.01 ?223% perf-profile.self.cycles-pp._IO_file_open
0.01 ?223% perf-profile.self.cycles-pp.lockref_get_not_dead
0.19 ? 11% perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
0.14 ? 13% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.02 ?142% perf-profile.self.cycles-pp.strncpy_from_user
0.01 ?223% perf-profile.self.cycles-pp.open64
0.01 ?223% perf-profile.self.cycles-pp.cpuidle_idle_call
0.01 ?223% perf-profile.self.cycles-pp.__irq_exit_rcu
0.01 ?223% perf-profile.self.cycles-pp.rcu_sched_clock_irq
0.01 ?223% perf-profile.self.cycles-pp.update_rt_rq_load_avg
0.01 ?223% perf-profile.self.cycles-pp.irqentry_enter_from_user_mode
0.11 ? 28% perf-profile.self.cycles-pp.atime_needs_update
0.04 ? 71% perf-profile.self.cycles-pp.fsnotify_perm
0.02 ?223% perf-profile.self.cycles-pp.update_irq_load_avg
0.02 ?141% perf-profile.self.cycles-pp.path_init
0.01 ?223% perf-profile.self.cycles-pp.__hrtimer_run_queues
0.01 ?223% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
0.01 ?223% perf-profile.self.cycles-pp.irqtime_account_process_tick
0.01 ?223% perf-profile.self.cycles-pp.tcp_orphan_count_sum
0.10 ? 58% perf-profile.self.cycles-pp.evlist_cpu_iterator__next
0.01 ?223% perf-profile.self.cycles-pp.lapic_next_deadline
0.01 ?223% perf-profile.self.cycles-pp.read_tsc
0.01 ?223% perf-profile.self.cycles-pp.arch_scale_freq_tick
0.03 ?103% perf-profile.self.cycles-pp.smpboot_thread_fn
0.05 ? 74% perf-profile.self.cycles-pp.link_path_walk
0.09 ? 31% perf-profile.self.cycles-pp.__call_rcu_common
0.11 ? 19% perf-profile.self.cycles-pp.perf_iterate_sb
0.02 ?223% perf-profile.self.cycles-pp.tick_sched_do_timer
0.02 ?223% perf-profile.self.cycles-pp.run_posix_cpu_timers
0.03 ?100% perf-profile.self.cycles-pp.get_pid_task
0.10 ? 16% perf-profile.self.cycles-pp.next_tgid
0.02 ?141% perf-profile.self.cycles-pp._IO_link_in
0.02 ?223% perf-profile.self.cycles-pp.rcu_pending
0.02 ?223% perf-profile.self.cycles-pp.get_cpu_device
0.07 ? 19% perf-profile.self.cycles-pp.__libc_calloc
0.14 ? 71% perf-profile.self.cycles-pp.record__pushfn
0.12 ? 30% perf-profile.self.cycles-pp.clear_page_erms
0.06 ? 50% perf-profile.self.cycles-pp.__entry_text_start
0.05 ? 49% perf-profile.self.cycles-pp.__fput
0.03 ?102% perf-profile.self.cycles-pp.security_file_open
0.02 ?142% perf-profile.self.cycles-pp.vprintk_emit
0.02 ?223% perf-profile.self.cycles-pp.timekeeping_max_deferment
0.02 ?142% perf-profile.self.cycles-pp.strscpy
0.04 ? 71% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.05 ? 75% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
0.06 ? 77% perf-profile.self.cycles-pp.kcpustat_cpu_fetch
0.02 ?223% perf-profile.self.cycles-pp.rebalance_domains
0.02 ?223% perf-profile.self.cycles-pp.scheduler_tick
0.06 ? 23% perf-profile.self.cycles-pp.aa_may_ptrace
0.12 ? 21% perf-profile.self.cycles-pp.__ptrace_may_access
0.03 ?105% perf-profile.self.cycles-pp._IO_file_doallocate
0.08 ? 23% perf-profile.self.cycles-pp.getenv
0.08 ? 27% perf-profile.self.cycles-pp.__memcg_kmem_charge_page
0.05 ? 47% perf-profile.self.cycles-pp.__kmem_cache_free
0.37 ? 23% perf-profile.self.cycles-pp.pid_revalidate
0.04 ? 71% perf-profile.self.cycles-pp.cpuacct_charge
0.09 ? 32% perf-profile.self.cycles-pp.ordered_events__queue
0.12 ? 9% perf-profile.self.cycles-pp._raw_spin_lock_irq
1.30 ? 22% perf-profile.self.cycles-pp.update_sg_wakeup_stats
0.07 ? 57% perf-profile.self.cycles-pp.rmqueue
0.12 ? 17% perf-profile.self.cycles-pp.down_write
0.10 ? 30% perf-profile.self.cycles-pp.apparmor_file_permission
0.06 ? 19% perf-profile.self.cycles-pp.put_dec_trunc8
0.03 ?223% perf-profile.self.cycles-pp.tick_nohz_next_event
0.33 ? 19% perf-profile.self.cycles-pp.kmem_cache_free
0.10 ?143% perf-profile.self.cycles-pp.__intel_pmu_enable_all
0.10 ?170% perf-profile.self.cycles-pp.ktime_get
0.04 ? 71% perf-profile.self.cycles-pp.memset@plt
0.04 ? 71% perf-profile.self.cycles-pp.check_heap_object
0.04 ? 72% perf-profile.self.cycles-pp.ktime_get_update_offsets_now
0.14 ? 28% perf-profile.self.cycles-pp.file_free_rcu
0.33 ? 43% perf-profile.self.cycles-pp.shmem_write_end
0.05 ? 72% perf-profile.self.cycles-pp.exc_page_fault
0.04 ?223% perf-profile.self.cycles-pp.__update_blocked_fair
0.19 ? 20% perf-profile.self.cycles-pp.__schedule
0.04 ?105% perf-profile.self.cycles-pp.__snprintf_chk
0.04 ? 73% perf-profile.self.cycles-pp.mm_account_fault
0.04 ?128% perf-profile.self.cycles-pp.load_balance
0.04 ? 74% perf-profile.self.cycles-pp.memcg_account_kmem
0.04 ?171% perf-profile.self.cycles-pp.update_sd_lb_stats
0.10 ? 83% perf-profile.self.cycles-pp.evsel__read_counter
0.28 ? 17% perf-profile.self.cycles-pp.__slab_free
0.32 ? 17% perf-profile.self.cycles-pp.do_dentry_open
0.05 ?223% perf-profile.self.cycles-pp.menu_select
0.05 ? 8% perf-profile.self.cycles-pp.copyout
0.05 ? 84% perf-profile.self.cycles-pp.mm_init
0.06 ? 61% perf-profile.self.cycles-pp.get_cpu_sleep_time_us
0.24 ? 6% perf-profile.self.cycles-pp.rcu_cblist_dequeue
0.20 ? 28% perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
1.45 ? 16% perf-profile.self.cycles-pp.do_task_stat
0.07 ? 23% perf-profile.self.cycles-pp._IO_fread
0.63 ? 67% perf-profile.self.cycles-pp.fault_in_readable
0.09 ? 20% perf-profile.self.cycles-pp.__kmem_cache_alloc_bulk
0.09 ? 13% perf-profile.self.cycles-pp.mas_wr_node_store
0.72 ? 19% perf-profile.self.cycles-pp.__percpu_counter_sum
0.23 ?129% perf-profile.self.cycles-pp.__pthread_disable_asynccancel
0.09 ? 27% perf-profile.self.cycles-pp.update_rq_clock_task
0.13 ?169% perf-profile.self.cycles-pp.find_busiest_queue
0.55 ? 22% perf-profile.self.cycles-pp.pcpu_alloc
0.45 ? 85% perf-profile.self.cycles-pp.ring_buffer_read_head
0.24 ? 63% perf-profile.self.cycles-pp.evlist__id2evsel
0.14 ?223% perf-profile.self.cycles-pp.cpuidle_enter_state
0.22 ? 39% perf-profile.self.cycles-pp.mm_init_cid
0.25 ? 29% perf-profile.self.cycles-pp.mtree_range_walk
0.65 ? 23% perf-profile.self.cycles-pp.copy_mc_enhanced_fast_string
0.40 ?198% perf-profile.self.cycles-pp.intel_idle
0.45 ? 13% perf-profile.self.cycles-pp.__memcpy
0.60 ? 23% perf-profile.self.cycles-pp.__memset
0.66 ?200% perf-profile.self.cycles-pp.intel_idle_xstate
1.51 ?125% perf-profile.self.cycles-pp.update_sg_lb_stats
1.11 ? 21% perf-profile.self.cycles-pp.task_mm_cid_work
2.04 ? 58% perf-profile.self.cycles-pp.copyin
49.20 ? 10% perf-profile.self.cycles-pp.queue_event

2023-07-17 06:42:58

by Raghavendra K T

[permalink] [raw]

Subject: Re: [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression

On 7/16/2023 7:47 PM, Oliver Sang wrote:
> hi, Raghavendra K T,
>
> On Wed, Jul 05, 2023 at 11:18:37AM +0530, Raghavendra K T wrote:
>> On 5/31/2023 9:55 AM, Raghavendra K T wrote:
>>> With the numa scan enhancements [1], only the threads which had previously
>>> accessed vma are allowed to scan.
>>>
>>> While this had improved significant system time overhead, there were corner
>>> cases, which genuinely need some relaxation. For e.g.,
>>>
>>> 1) Concern raised by PeterZ, where if there are N partition sets of vmas
>>> belonging to tasks, then unfairness in allowing these threads to scan could
>>> potentially amplify the side effect of some of the vmas being left
>>> unscanned.
>>>
>>> 2) Below reports of LKP numa01 benchmark regression.
>>>
>>> Currently this is handled by allowing first two scanning unconditional
>>> as indicated by mm->numa_scan_seq. This is imprecise since for some
>>> benchmark vma scanning might itself start at numa_scan_seq > 2.
>>>
>>> Solution:
>>> Allow unconditional scanning of vmas of tasks depending on vma size. This
>>> is achieved by maintaining a per vma scan counter, where
>>>
>>> f(allowed_to_scan) = f(scan_counter < vma_size / scan_size)
>>>
>>> Result:
>>> numa01_THREAD_ALLOC result on 6.4.0-rc2 (that has numascan enhancement)
>>> base-numascan base base+fix
>>> real 1m1.507s 1m23.259s 1m2.632s
>>> user 213m51.336s 251m46.363s 220m35.528s
>>> sys 3m3.397s 0m12.492s 2m41.393s
>>>
>>> numa_hit 5615517 4560123 4963875
>>> numa_local 5615505 4560024 4963700
>>> numa_other 12 99 175
>>> numa_pte_updates 1822797 493 1559111
>>> numa_hint_faults 1307113 523 1469031
>>> numa_hint_faults_local 612617 488 884829
>>> numa_pages_migrated 694370 35 584202
>>>
>>> Summary: Regression in base is recovered by allowing scanning as required.
>>>
>>> [1] https://lore.kernel.org/lkml/[email protected]/T/#t
>>>
>>> Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
>>> regression.
>>> Reported-by: Aithal Srikanth <[email protected]>
>>> Reported-by: kernel test robot <[email protected]>
>>> Closes: https://lore.kernel.org/lkml/[email protected]/T/
>>> Signed-off-by: Raghavendra K T <[email protected]>
>>
>> Hello kernel test robot,
>>
>> Gentle ping to check if the patch has helped your regression report.
>
> sorry for late.
>
> previously we found a 118.9% regression of autonuma-benchmark.numa01.seconds
> on a Cascade Lake, which happened to be converted for other test purposes, so
> we cannot test your patch on it again.
>
> however, we also found a 39.3% regression on a Sapphire Rapids test machine:
>
> =========================================================================================
> compiler/cpufreq_governor/iterations/kconfig/rootfs/tbox_group/test/testcase:
> gcc-11/performance/4x/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-spr-r02/numa02_SMT/autonuma-benchmark
>
> ef6a22b70f6d9044 fc137c0ddab29b591db6a091dc6
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 193.14 +39.2% 268.84 autonuma-benchmark.numa01.seconds
> 8.14 -0.7% 8.09 autonuma-benchmark.numa02.seconds
>
>
> now we tested your v3 patch on it, found regression mostly recovered
> (55fd15913b18d6a790c17d947df is just [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression)
>
> =========================================================================================
> compiler/cpufreq_governor/iterations/kconfig/rootfs/tbox_group/test/testcase:
> gcc-11/performance/4x/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-spr-r02/numa01_THREAD_ALLOC/autonuma-benchmark
>
> ef6a22b70f6d9044 55fd15913b18d6a790c17d947df
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 193.14 +5.8% 204.37 ± 3% autonuma-benchmark.numa01.seconds
> 8.14 -0.9% 8.06 autonuma-benchmark.numa02.seconds
>
> detail comparison as below:
>

Thank you for the confirmation. So looks like we got back most of the
regression with the patch for numa01_THREAD_ALLOC case.

Andrew, could you please help by picking up this patch unless
Mel , PeterZ do not have any concern about the patch / direction.

(as we note it brings back little bit system time overhead by allowing
some scanning..)

Thanks and Regards
- Raghu

2023-07-21 15:54:34

by Mel Gorman

[permalink] [raw]

Subject: Re: [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression

On Wed, May 31, 2023 at 09:55:26AM +0530, Raghavendra K T wrote:
> With the numa scan enhancements [1], only the threads which had previously
> accessed vma are allowed to scan.
>

I know it's very late to be reading this properly for the first time. It's a
brief review as it's late but I'm not 100% convinced by the patch as-is.

To start off -- reference the specific commit e.g. Since commit fc137c0ddab2
("sched/numa: enhance vma scanning logic")...

> While this had improved significant system time overhead, there were corner
> cases, which genuinely need some relaxation. For e.g.,
>
> 1) Concern raised by PeterZ, where if there are N partition sets of vmas
> belonging to tasks, then unfairness in allowing these threads to scan could
> potentially amplify the side effect of some of the vmas being left
> unscanned.
>

This needs a much better description as there is too much guesswork
required to figure out what problem is being solved. You may be referring
to an issue where a VMA not accessed for a long time may be skipped for
scanning indefinitely. You could also be referring to an issue where a
highly active thread always traps the NUMA hinting fault first and other
tasks never pass the vma_is_accessed test. It also could be the case that
due to a malloc implementation behaviour that the thread using a particular
VMA changes and it's not detected. It's simply not clear how the VMAs are
partitioned or what exactly is unfair. I certainly can guess a number of
reasons why a patch like this is necessary but in a years time, it'll be
hard to be certain about the original intent.

> 2) Below reports of LKP numa01 benchmark regression.
>
> Currently this is handled by allowing first two scanning unconditional
> as indicated by mm->numa_scan_seq. This is imprecise since for some
> benchmark vma scanning might itself start at numa_scan_seq > 2.
>

Well, it's also not useful in many cases. There is nothing special about
the first two scans other than they happen early in the lifetime of the
process. A major change in phase behaviour that occurs after 2 scans
could miss the new access pattern.

> Solution:
> Allow unconditional scanning of vmas of tasks depending on vma size. This
> is achieved by maintaining a per vma scan counter, where
>
> f(allowed_to_scan) = f(scan_counter < vma_size / scan_size)
>

This is a vague description as well and does not mention that the
scan_counter resets and the code does not appear to match the equation.
It's also a bit arbitrary that unconditional scanning occurs for every task
after the scan_count passes a threshold. The timing of when this occurs may
vary and all threads sharing the address space may conduct the scanning
which may be overkill. It should only be necessary for at least one task
to unconditionally mark the VMA for hinting, no? I'm generally a bit wary
that the time it takes to detect a VMA needs unconditonal scanning depends
on the size as due to VMA-merging, it may be non-deterministic when a VMA
gets scanned and there may be run-to-run variance to the timing of threads
allocating address space.

> Result:
> numa01_THREAD_ALLOC result on 6.4.0-rc2 (that has numascan enhancement)
> base-numascan base base+fix
> real 1m1.507s 1m23.259s 1m2.632s
> user 213m51.336s 251m46.363s 220m35.528s
> sys 3m3.397s 0m12.492s 2m41.393s
>
> numa_hit 5615517 4560123 4963875
> numa_local 5615505 4560024 4963700
> numa_other 12 99 175
> numa_pte_updates 1822797 493 1559111
> numa_hint_faults 1307113 523 1469031
> numa_hint_faults_local 612617 488 884829
> numa_pages_migrated 694370 35 584202
>

It's not clear what these kernels are. Is base-numascan a 6.4-rc2 kernel
with at least commit fc137c0ddab2 ("sched/numa: enhance vma scanning
logic") reverted?

The numa_[hint|local|other] stats are not helpful. They are updated on
the allocation side and not related to actual behaviour.

As I'm unsure what the kernels are, I'm not sure how interesting it is
that the pte update stats and number of faults trapped are very different
for base to base+fix. Sure enough, the overall runtime is lower so it's
*probably* good but not sure as it may be also the case that disabling NUMA
balancing would be faster again given the nature of this particular test :P

The testing is a little thin and I'd worry slightly that this patch is
very specific to this particular workload.

> Summary: Regression in base is recovered by allowing scanning as required.
>
> [1] https://lore.kernel.org/lkml/[email protected]/T/#t
>
> Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
> regression.

Fixes may be a bit overkill as this patch is more of an enhancement than
something that justifies a backport to -stable but I don't feel strongly
about it.

> Reported-by: Aithal Srikanth <[email protected]>
> Reported-by: kernel test robot <[email protected]>
> Closes: https://lore.kernel.org/lkml/[email protected]/T/
> Signed-off-by: Raghavendra K T <[email protected]>
> ---
> include/linux/mm_types.h | 1 +
> kernel/sched/fair.c | 31 ++++++++++++++++++++++++-------
> 2 files changed, 25 insertions(+), 7 deletions(-)
>
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index 306a3d1a0fa6..992e460a713e 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -479,6 +479,7 @@ struct vma_numab_state {
> unsigned long next_scan;
> unsigned long next_pid_reset;
> unsigned long access_pids[2];
> + unsigned int scan_counter;
> };

Vague name as it's not counting and gets reset. It might have been clearer
to name is something like scan_selective with an initial value related to
the VMA size that decrements. When it hits 0, the scan is forced *once*
for the unlucky task. The suggested name is also bad, I'm terrible at
naming but "counter" gives no hints either.

>
> /*
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 373ff5f55884..4e71fb58085b 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -2931,17 +2931,30 @@ static void reset_ptenuma_scan(struct task_struct *p)
> static bool vma_is_accessed(struct vm_area_struct *vma)
> {
> unsigned long pids;
> + unsigned int vma_size;
> + unsigned int scan_threshold;
> + unsigned int scan_size;
> +
> + pids = vma->numab_state->access_pids[0] | vma->numab_state->access_pids[1];
> +
> + if (test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids))
> + return true;
> +
> + scan_size = READ_ONCE(sysctl_numa_balancing_scan_size);
> + /* vma size in MB */
> + vma_size = (vma->vm_end - vma->vm_start) >> 20;
> +
> + /* Total scans needed to cover VMA */
> + scan_threshold = vma_size / scan_size;
> +
> /*
> - * Allow unconditional access first two times, so that all the (pages)
> - * of VMAs get prot_none fault introduced irrespective of accesses.
> + * Allow the scanning of half of disjoint set's VMA to induce
> + * prot_none fault irrespective of accesses.
> * This is also done to avoid any side effect of task scanning
> * amplifying the unfairness of disjoint set of VMAs' access.
> */

Much better description is required. It's not stated how the VMAs are
disjoint. For example, they all belong to the same address space so from that
perspective they are not disjoint. Clarify that it's related to accesses
at the very least or be clear on how a set of VMAs can be disjoint. Also,
it's not half the total VMAs or some other critieria either, it's related
to scanning events. There is a lot of guesswork involved here.

> - if (READ_ONCE(current->mm->numa_scan_seq) < 2)
> - return true;
> -
> - pids = vma->numab_state->access_pids[0] | vma->numab_state->access_pids[1];
> - return test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids);
> + scan_threshold = 1 + (scan_threshold >> 1);
> + return (vma->numab_state->scan_counter < scan_threshold);
> }
>
> #define VMA_PID_RESET_PERIOD (4 * sysctl_numa_balancing_scan_delay)
> @@ -3058,6 +3071,8 @@ static void task_numa_work(struct callback_head *work)
> /* Reset happens after 4 times scan delay of scan start */
> vma->numab_state->next_pid_reset = vma->numab_state->next_scan +
> msecs_to_jiffies(VMA_PID_RESET_PERIOD);
> +
> + vma->numab_state->scan_counter = 0;
> }
>
> /*
> @@ -3084,6 +3099,8 @@ static void task_numa_work(struct callback_head *work)
> vma->numab_state->access_pids[1] = 0;
> }
>
> + vma->numab_state->scan_counter++;
> +
> do {
> start = max(start, vma->vm_start);
> end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);

This has a potential timing issue as only tasks that accessed the VMA
update the counter. A task could create a VMA, access is heavily in an
init phase and then go completely idle waiting on other tasks to complete
(e.g. parallelised load that uses a single thread to setup the data and then
creates worker threads to process that data). As there is no guarantee other
tasks would trap a fault, the access bits would never be set, the counter
never increments and the VMA is ignored forever. It's late on a Friday so
maybe I'm wrong but it looks like this could have weird corner cases.

The counter should at least increase if the VMA could have been scanned
except the access bits were not set with the caveat that it may incur
excessive cache misses.

--
Mel Gorman
SUSE Labs

2023-07-24 08:17:37

by Raghavendra K T

[permalink] [raw]

Subject: Re: [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression

On 7/21/2023 8:48 PM, Mel Gorman wrote:
> On Wed, May 31, 2023 at 09:55:26AM +0530, Raghavendra K T wrote:
>> With the numa scan enhancements [1], only the threads which had previously
>> accessed vma are allowed to scan.
>>
>
> I know it's very late to be reading this properly for the first time. It's a
> brief review as it's late but I'm not 100% convinced by the patch as-is.
>

Thank you Mel for the review. Expert opinion / intuition from you,
PeterZ helped, helps alot to improve to solve the problem in better way
finally.

> To start off -- reference the specific commit e.g. Since commit fc137c0ddab2
> ("sched/numa: enhance vma scanning logic")...
>

Okay.

>> While this had improved significant system time overhead, there were corner
>> cases, which genuinely need some relaxation. For e.g.,
>>
>> 1) Concern raised by PeterZ, where if there are N partition sets of vmas
>> belonging to tasks, then unfairness in allowing these threads to scan could
>> potentially amplify the side effect of some of the vmas being left
>> unscanned.
>>
>
> This needs a much better description as there is too much guesswork
> required to figure out what problem is being solved.

Agree I needed to be more descriptive.

You may be referring
> to an issue where a VMA not accessed for a long time may be skipped for
> scanning indefinitely.
I feel this is mostly a favorable case because we don't incur
overhead of scanning unneeded VMA. so not this, but below ones

You could also be referring to an issue where a
> highly active thread always traps the NUMA hinting fault first and other
> tasks never pass the vma_is_accessed test. It also could be the case that
> due to a malloc implementation behaviour that the thread using a particular
> VMA changes and it's not detected. It's simply not clear how the VMAs are
> partitioned or what exactly is unfair. I certainly can guess a number of
> reasons why a patch like this is necessary but in a years time, it'll be
> hard to be certain about the original intent.

Agree, to summarize
here there are multiple layers of issues

1) PeterZ pointed this

(Thread1/Set1)---->access(vma1) <== gets 65% opportunity

(Thread2/Set2)---->access(vma2) <== gets 35% opprtunity

Suppose Thread1 (or threads from Set1) are actively getting more
scanning opportunity, we may miss or scan less of vma2.

I did some experiment and found there was indeed a bit of unfairness in
the way set1/set2 gets opportunity to scan.

Link:
https://lore.kernel.org/lkml/[email protected]/

2) Suppose for the above case if we have
vma1, vma2 represents sum of vmas accessed by set1 and set2 respectively
in disjoint way.

and vma1 size is far less than vma2 size, effect of unfairness in
scanning gets amplified. e.g.,

(Thread1/Set1)---->access(vma1 size = 1MB) <== actively scanning

(Thread2/Set2)---->access(vma2 size = 10MB)

3) malloc behavior you pointed.

I was trying to minimize the side effect of unfairness in scanning
overall by introducing a bit of unconditional scans.

(But the unfairness amongst the threads, which thread (or set of thread)
gets opportunity to scan remains)

I may be missing some more cases here

>
>> 2) Below reports of LKP numa01 benchmark regression.
>>
>> Currently this is handled by allowing first two scanning unconditional
>> as indicated by mm->numa_scan_seq. This is imprecise since for some
>> benchmark vma scanning might itself start at numa_scan_seq > 2.
>>
>
> Well, it's also not useful in many cases. There is nothing special about
> the first two scans other than they happen early in the lifetime of the
> process. A major change in phase behaviour that occurs after 2 scans
> could miss the new access pattern.

Fully agree. That is why that condition is removed in the patch now.
What I also saw was some times when new vma gets created
mm->numa_scan_seq could be already more than 2.
(To handle that correctly I would have to have per vma start_numa_seq
kind of things etc which was messy and more space consuming)

Note:
Somewhat related, In my experiments what I saw is normal short
running benchmarks used to finish by numa_scan_seq < 16 whereas some
benchmark like NAS used to go till 100-150. (to set context for any
long running benchmark optimization in future)

>
>> Solution:
>> Allow unconditional scanning of vmas of tasks depending on vma size. This
>> is achieved by maintaining a per vma scan counter, where
>>
>> f(allowed_to_scan) = f(scan_counter < vma_size / scan_size)
>>
>
> This is a vague description as well and does not mention that the
> scan_counter resets and the code does not appear to match the equation.
> It's also a bit arbitrary that unconditional scanning occurs for every task
> after the scan_count passes a threshold. The timing of when this occurs may
> vary and all threads sharing the address space may conduct the scanning
> which may be overkill. It should only be necessary for at least one task
> to unconditionally mark the VMA for hinting, no? I'm generally a bit wary
> that the time it takes to detect a VMA needs unconditonal scanning depends
> on the size as due to VMA-merging,

I do agree that changing of vma size due to merge/split can affect
scanning. But I was assuming that it is Okay because this calculation is
done every time we hit task_numa_work which may be wrong (and imprecise).

it may be non-deterministic when a VMA
> gets scanned and there may be run-to-run variance to the timing of threads
> allocating address space >

Agree

>> Result:
>> numa01_THREAD_ALLOC result on 6.4.0-rc2 (that has numascan enhancement)
>> base-numascan base base+fix
>> real 1m1.507s 1m23.259s 1m2.632s
>> user 213m51.336s 251m46.363s 220m35.528s
>> sys 3m3.397s 0m12.492s 2m41.393s
>>
>> numa_hit 5615517 4560123 4963875
>> numa_local 5615505 4560024 4963700
>> numa_other 12 99 175
>> numa_pte_updates 1822797 493 1559111
>> numa_hint_faults 1307113 523 1469031
>> numa_hint_faults_local 612617 488 884829
>> numa_pages_migrated 694370 35 584202
>>
>
> It's not clear what these kernels are. Is base-numascan a 6.4-rc2 kernel
> with at least commit fc137c0ddab2 ("sched/numa: enhance vma scanning
> logic") reverted?

This is correct. To compare the system time overhead before commit
fc137c0ddab2 ("sched/numa: enhance vma scanning logic")

> The numa_[hint|local|other] stats are not helpful. They are updated on
> the allocation side and not related to actual behaviour.

will trim this next time.

>
> As I'm unsure what the kernels are, I'm not sure how interesting it is
> that the pte update stats and number of faults trapped are very different
> for base to base+fix.

Number of faults trapped are excessively refined for base (6.4.0-rc2
vanilla) for this particular case, because of malloc behaviour you
pointed above.

Sure enough, the overall runtime is lower so it's
> *probably* good but not sure as it may be also the case that disabling NUMA
> balancing would be faster again given the nature of this particular test :P
>
> The testing is a little thin and I'd worry slightly that this patch is
> very specific to this particular workload.

I do agree that it is mostly trying to address LKP numa01_THREAD_ALLOC
case.
But regarding the testing though I reported only above result,
I do run
1) kernbench, autonuma from mmtest,
2) some microbenchmarks and also NAS/hashjoin etc long running
benchmarks to ensure there are no surprises.

>
>> Summary: Regression in base is recovered by allowing scanning as required.
>>
>> [1] https://lore.kernel.org/lkml/[email protected]/T/#t
>>
>> Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
>> regression.
>
> Fixes may be a bit overkill as this patch is more of an enhancement than
> something that justifies a backport to -stable but I don't feel strongly
> about it.

Sure.

>
>> Reported-by: Aithal Srikanth <[email protected]>
>> Reported-by: kernel test robot <[email protected]>
>> Closes: https://lore.kernel.org/lkml/[email protected]/T/
>> Signed-off-by: Raghavendra K T <[email protected]>
>> ---
>> include/linux/mm_types.h | 1 +
>> kernel/sched/fair.c | 31 ++++++++++++++++++++++++-------
>> 2 files changed, 25 insertions(+), 7 deletions(-)
>>
>> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
>> index 306a3d1a0fa6..992e460a713e 100644
>> --- a/include/linux/mm_types.h
>> +++ b/include/linux/mm_types.h
>> @@ -479,6 +479,7 @@ struct vma_numab_state {
>> unsigned long next_scan;
>> unsigned long next_pid_reset;
>> unsigned long access_pids[2];
>> + unsigned int scan_counter;
>> };
>
> Vague name as it's not counting and gets reset. It might have been clearer
> to name is something like scan_selective with an initial value related to
> the VMA size that decrements. When it hits 0, the scan is forced *once*
> for the unlucky task. The suggested name is also bad, I'm terrible at
> naming but "counter" gives no hints either.
>

Agree that naming is bad, will use scan_selective / or try if I get more
idea on naming.
So you are suggesting to try other-way where we force scan once we hit
zero. Looks fair to me. will try the idea and see how it goes.

>>
>> /*
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index 373ff5f55884..4e71fb58085b 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -2931,17 +2931,30 @@ static void reset_ptenuma_scan(struct task_struct *p)
>> static bool vma_is_accessed(struct vm_area_struct *vma)
>> {
>> unsigned long pids;
>> + unsigned int vma_size;
>> + unsigned int scan_threshold;
>> + unsigned int scan_size;
>> +
>> + pids = vma->numab_state->access_pids[0] | vma->numab_state->access_pids[1];
>> +
>> + if (test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids))
>> + return true;
>> +
>> + scan_size = READ_ONCE(sysctl_numa_balancing_scan_size);
>> + /* vma size in MB */
>> + vma_size = (vma->vm_end - vma->vm_start) >> 20;
>> +
>> + /* Total scans needed to cover VMA */
>> + scan_threshold = vma_size / scan_size;
>> +
>> /*
>> - * Allow unconditional access first two times, so that all the (pages)
>> - * of VMAs get prot_none fault introduced irrespective of accesses.
>> + * Allow the scanning of half of disjoint set's VMA to induce
>> + * prot_none fault irrespective of accesses.
>> * This is also done to avoid any side effect of task scanning
>> * amplifying the unfairness of disjoint set of VMAs' access.
>> */
>
> Much better description is required. It's not stated how the VMAs are
> disjoint. For example, they all belong to the same address space so from that
> perspective they are not disjoint. Clarify that it's related to accesses
> at the very least or be clear on how a set of VMAs can be disjoint. Also,
> it's not half the total VMAs or some other critieria either, it's related
> to scanning events. There is a lot of guesswork involved here.

Sure. will add more details.
ALso I was not clear here where does that half came from. Idea was
allowing unconditional scanning = vma_size for a unrelated task that has
not accessed vma is too much.. since we are letting unconditional scans
let it be reduced to atleast half of that. I will experiment a bit here
to get optimal f(vma_size/scan_size)

>
>> - if (READ_ONCE(current->mm->numa_scan_seq) < 2)
>> - return true;
>> -
>> - pids = vma->numab_state->access_pids[0] | vma->numab_state->access_pids[1];
>> - return test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids);
>> + scan_threshold = 1 + (scan_threshold >> 1);
>> + return (vma->numab_state->scan_counter < scan_threshold);
>> }
>>
>> #define VMA_PID_RESET_PERIOD (4 * sysctl_numa_balancing_scan_delay)
>> @@ -3058,6 +3071,8 @@ static void task_numa_work(struct callback_head *work)
>> /* Reset happens after 4 times scan delay of scan start */
>> vma->numab_state->next_pid_reset = vma->numab_state->next_scan +
>> msecs_to_jiffies(VMA_PID_RESET_PERIOD);
>> +
>> + vma->numab_state->scan_counter = 0;
>> }
>>
>> /*
>> @@ -3084,6 +3099,8 @@ static void task_numa_work(struct callback_head *work)
>> vma->numab_state->access_pids[1] = 0;
>> }
>>
>> + vma->numab_state->scan_counter++;
>> +
>> do {
>> start = max(start, vma->vm_start);
>> end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
>
> This has a potential timing issue as only tasks that accessed the VMA
> update the counter.

I think there is a confusion because of reverted logic. we do allow
unconditionally to let the scan happen initially till scan_counter
reaches scan_threshold. So even though task did not access vma, we get
to scan initially.

So idea is update scan_counter only if we are able to successfully come
till here because we are sure that there is not condition that returns
back after this without scanning.

A task could create a VMA, access is heavily in an
> init phase and then go completely idle waiting on other tasks to complete
> (e.g. parallelised load that uses a single thread to setup the data and then
> creates worker threads to process that data). As there is no guarantee other
> tasks would trap a fault, the access bits would never be set, the counter
> never increments and the VMA is ignored forever. It's late on a Friday so
> maybe I'm wrong but it looks like this could have weird corner cases.
>
> The counter should at least increase if the VMA could have been scanned
> except the access bits were not set with the caveat that it may incur
> excessive cache misses.
>

Explained above

Thanks a lot,
Raghu

2023-08-11 14:09:32

by Raghavendra K T

[permalink] [raw]

Subject: Re: [RFC PATCH V3 1/1] sched/numa: Fix disjoint set vma scan regression

On 7/24/2023 1:11 PM, Raghavendra K T wrote:
> On 7/21/2023 8:48 PM, Mel Gorman wrote:
>> On Wed, May 31, 2023 at 09:55:26AM +0530, Raghavendra K T wrote:
>>> With the numa scan enhancements [1], only the threads which had
>>> previously
>>> accessed vma are allowed to scan.
>>>
>>
>> I know it's very late to be reading this properly for the first time.
>> It's a
>> brief review as it's late but I'm not 100% convinced by the patch as-is.
>>
>
> Thank you Mel for the review. Expert opinion / intuition from you,
> PeterZ helped, helps alot to improve to solve the problem in better way
> finally.
[...]
>>> ---
>>> include/linux/mm_types.h | 1 +
>>> kernel/sched/fair.c      | 31 ++++++++++++++++++++++++-------
>>> 2 files changed, 25 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
>>> index 306a3d1a0fa6..992e460a713e 100644
>>> --- a/include/linux/mm_types.h
>>> +++ b/include/linux/mm_types.h
>>> @@ -479,6 +479,7 @@ struct vma_numab_state {
>>>       unsigned long next_scan;
>>>       unsigned long next_pid_reset;
>>>       unsigned long access_pids[2];
>>> +    unsigned int scan_counter;
>>> };
>>
>> Vague name as it's not counting and gets reset. It might have been
>> clearer
>> to name is something like scan_selective with an initial value related to
>> the VMA size that decrements. When it hits 0, the scan is forced *once*
>> for the unlucky task. The suggested name is also bad, I'm terrible at
>> naming but "counter" gives no hints either.
>>
>
> Agree that naming is bad, will use scan_selective / or try if I get more
> idea on naming.
> So you are suggesting to try other-way where we force scan once we hit
> zero. Looks fair to me. will try the idea and see how it goes.
>

Hello Mel,
Sorry for going slow from my side. I was experimenting further with
combination patches, I will be posting set of patches that I mention
further down here.

Just want us to be on same page, posting full description of the
problem, details of solution, and how it looks like (code) etc

Also sorry for long description.

Problem statement (Disjoint VMA set):
====================
Since commit fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
vma scanning is allowed if
1) The task had accessed vma
2) Early phase of the scan where mm->numa_scan_seq is less than 2
while this works for most of the times to reduce scanning overhead
there are concerning corner case(s) associated with it.

Let's look at these corner caseswith below example of tasks and their
access pattern.
Set 1 tasks accessing vma_x (group of vmas)
Set 2 tasks accessing vma_y (group of vmas)

Set1 Set2
------------------- --------------------
| task_1..task_n/2 | | task_n/2+1..task_n |
------------------- --------------------
| |
V V
------------------- --------------------
| vma_x | | vma_y |
------------------- --------------------

Figure. (1) Tasks accessing disjoint vma

Corner cases:
(a) Out of N tasks, not all of them gets fair opportunity to scan. (PeterZ).
suppose Set1 tasks gets more opportunity to scan in the above example,
then vma_x gets scanned more number of times than vma_y.

some experiment is also done here:
Link:
https://lore.kernel.org/lkml/[email protected]/

(b) Sizes of vmas can differ.
suppose size of vma_y is far greater than size of vma_x, then bigger
portion of
vma_y can potentially be left unscanned since scanning is bounded by
scan_size

(c) highly active threads trap a few vmas frequently, and some of the
vmas not
accessed for long time can potentially get starved of scanning indefinitely
(Mel) (because task access history gets reset periodically. Not sure if this
is is a bigger issue if vma is not important but on the other hand we
will be
left with no hints for migration later if necessary)

(d) allocation of memory in some specific manner (Mel).

(e) vmas that are created after two full scans of mm (mm->numa_scan_seq
> 2),
will never get scanned. (I have not seen that occurring in some rare cases)

on top of this Combination of some of the above (e.g., (a) and (b)) can
potentially
amplify the side effect of numa scanning overall.

High level ideas to address the issue:
===================================
Idea 1) depending on vma_size populate per vma scan_select value,
decrement it and when it hits zero
do force scan (Mel)

this is how vma phases looks like after implementation:

|<--p1--->|<-----p2----->|<-----p2----->|...

p1: new vma, initial phase do not scan till scan_delay

p2: allow scanning if task has accessed vma or vma_scan_selectiv hit zero

repeat p2

pros/cons:
+ : Ratelimiting is inbuilt to the approach
+/-: scanning continues forever
- : changes in vma size is taken care after force scan

Idea 2) Have a per vma_scan_seq. allow the unconditional scan till
vma_scan_seq
reaches a value proportional (or equal) to vma_size/scan_size

this is how vma phases looks like after implementation:

|<--p1--->|<-----p2----->|<-----p3----->|<-----p4----->...||<-----p2----->|<-----p3----->|<-----p4----->
...||
RESET
RESET

p1: new vma, initial phase do not scan till scan_delay

p2: allow scanning if task has accessed vma or vma_scan_seq has reached
till
f1(vma_size)/scan_size) for e.g., number of scan to cover full or
half of vmas

p3: allow scanning if task has accessed vma or vma_scan_seq has reached till
f2(vma_size)/scan_size in rate limited manner (Optional)

p4: allow scanning iff task has accessed vma

Reset after p4 ( f(scan_delay) ??)

and repeat p2, p3 p4

+ : Ratelimiting need to be taken care separately if needed
+/-: scanning continues only if RESET of vma_scan_seq is implemented
+ : changes in vma size is taken care in every scan
- :

Additional optimizations (unconditionally allow if vma is hot
i.e., either accessed very recently or number of tasks accessing vma
is very high)

Idea 3) Take bitmask_weight of pid_access history of vma. (suggested by
Bharata)
(vma->numab_state->pid_access[1] | vma->numab_state->pid_access[0])
If number of tasks accessing vma is > THRESHOLD (say 3),
unconditionally allow scanning

Idea 4) Take bitmask_weight of latest pid_access value
(vma->numab_state->pid_access[1])
If number of tasks accessing vma is >= 1, i.e. recently accessed
unconditionally allow scanning.

Other ideas still to explore:
5) Can we maintain more pid access history (vs last two as maintained now)
The whole idea with code has been already given by PeterZ.

Some of the observations from my experiments:
-----------------------------------
- It is very critical to allow scanning after initial scan_delay (as
mentioned by Mel)
so as to capture early patterns.
- It is also important to limit (or rate limit) unconditional scans to
avoid excess
scanning overhead.

Example code for idea1:
=======================
static inline unsigned int vma_scan_ratelimit(struct vm_area_struct *vma)
{
unsigned int vma_size, ratelimit;

/* vma size */
vma_size = (vma->vm_end - vma->vm_start);

if (vma_size)
ratelimit = (1 << 15) / vma_size;

/* for 4KB vma rate limit is 1 in 9 */
/* and for > 64K it is 1 in 2 */

return 1 + ratelimit;
}

static bool task_disjoint_vma_select(struct vm_area_struct *vma)
{
if (vma->numab_state->vma_scan_seq > 0) {
vma->numab_state->vma_scan_selective--;
return false;
} else
vma->numab_state->vma_scan_selective =
vma_scan_ratelimit(vma);
}

Example code for idea 2:
=======================
static inline unsigned int disjoint_vma_scan_threshold(struct
vm_area_struct *vma)
{
unsigned int vma_size, scan_size, threshold;

scan_size = READ_ONCE(sysctl_numa_balancing_scan_size);

/* vma size in MB */
vma_size = (vma->vm_end - vma->vm_start) >> 20;

/* Total scans needed to cover VMA. Add minimum of one */
threshold = 1 + (vma_size / scan_size);

return threshold;
}

static bool task_disjoint_vma_select(struct vm_area_struct *vma)
{
return (vma->numab_state->vma_scan_seq <=
disjoint_vma_scan_threshold(vma));
}

Example code for idea 3,4 combined:
================================

#define HOT_VMA_NR_TASK_THRESH 3

static bool vma_is_accessed(struct vm_area_struct *vma)
{
unsigned long pids, recent_hist;
unsigned int hashval;

recent_hist = READ_ONCE(vma->numab_state->access_pids[1]);
pids = vma->numab_state->access_pids[0] | recent_hist ;

/*
* Check if vma was recently accessed OR
* vma is being accessed by more than 2 tasks
*/
if (recent_hist || bitmap_wight(&pids, BITS_PER_LONG))
return true;

hashval = hash_32(current->pid, ilog2(BITS_PER_LONG));

return test_bit(hashval, &pids);
}

Additionally, to Implement vma_scan_seq reset have used something like

#define VMA_SCANSEQ_RESET_PERIOD (4 * VMA_PID_RESET_PERIOD)
/* this is to recent per vma scan_seq */

I will be posting combination of idea1 and idea3, idea4 by next week
mostly as a new enhancement series.

(OR idea2 and idea3, idea4 in case you have any positive thought about
idea2)

initial results are looking good for both the ideas, I am taking more
time to
experiment further and fine tune.

Thanks
- Raghu