Greeting,
FYI, we noticed a -5.6% regression of aim7.jobs-per-min due to commit:
commit: 32678f151338b9a321e9e27139a63c81f353acb7 ("[PATCH 1/4] xfs: detect self referencing btree sibling pointers")
url: https://github.com/intel-lab-lkp/linux/commits/Dave-Chinner/xfs-fix-random-format-verification-issues/20220502-162206
base: https://git.kernel.org/cgit/fs/xfs/xfs-linux.git for-next
patch link: https://lore.kernel.org/linux-xfs/[email protected]
in testcase: aim7
on test machine: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
with following parameters:
disk: 4BRD_12G
md: RAID0
fs: xfs
test: disk_wrt
load: 3000
cpufreq_governor: performance
ucode: 0x500320a
test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
sudo bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
sudo bin/lkp run generated-yaml-file
# if come across any failure that blocks the test,
# please remove ~/.lkp and /lkp dir to run from a clean state.
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
gcc-11/performance/4BRD_12G/xfs/x86_64-rhel-8.3/3000/RAID0/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/disk_wrt/aim7/0x500320a
commit:
a44a027a8b ("Merge tag 'large-extent-counters-v9' of https://github.com/chandanr/linux into xfs-5.19-for-next")
32678f1513 ("xfs: detect self referencing btree sibling pointers")
a44a027a8b2a20fe 32678f151338b9a321e9e27139a
---------------- ---------------------------
%stddev %change %stddev
\ | \
464232 -5.6% 438315 aim7.jobs-per-min
702.64 +2.9% 723.04 aim7.time.system_time
20.61 -3.7% 19.84 iostat.cpu.system
2257 ? 14% +35.0% 3047 ? 21% sched_debug.cpu.avg_idle.min
45360 -5.7% 42793 vmstat.system.cs
5130 ? 5% -13.7% 4427 ? 5% meminfo.Dirty
5972 ? 5% -10.4% 5351 ? 4% meminfo.Inactive(file)
2778 ? 53% -44.2% 1549 ? 77% numa-meminfo.node0.Active(anon)
6262 ?156% -76.5% 1473 ? 15% numa-meminfo.node1.Mapped
694.17 ? 53% -44.3% 386.83 ? 77% numa-vmstat.node0.nr_active_anon
694.17 ? 53% -44.3% 386.83 ? 77% numa-vmstat.node0.nr_zone_active_anon
1676 ?151% -75.8% 405.33 ? 10% numa-vmstat.node1.nr_mapped
8.246e+09 -3.9% 7.924e+09 perf-stat.i.branch-instructions
23053543 ? 2% -6.0% 21668177 perf-stat.i.cache-misses
46885 -4.6% 44714 perf-stat.i.context-switches
5.425e+10 -2.6% 5.285e+10 perf-stat.i.cpu-cycles
1424 -8.9% 1296 perf-stat.i.cpu-migrations
1.175e+10 -4.1% 1.127e+10 perf-stat.i.dTLB-loads
5.701e+09 -5.3% 5.396e+09 perf-stat.i.dTLB-stores
17300716 -12.4% 15151342 perf-stat.i.iTLB-load-misses
4.151e+10 -3.8% 3.994e+10 perf-stat.i.instructions
2341 ? 5% +10.1% 2578 perf-stat.i.instructions-per-iTLB-miss
0.62 -2.6% 0.60 perf-stat.i.metric.GHz
292.90 -4.3% 280.20 perf-stat.i.metric.M/sec
1997742 ? 2% -6.2% 1874006 perf-stat.i.node-loads
3534663 -5.9% 3326105 perf-stat.i.node-stores
2354 ? 2% +3.6% 2440 ? 2% perf-stat.overall.cycles-between-cache-misses
80.54 -1.7 78.83 perf-stat.overall.iTLB-load-miss-rate%
2400 +9.8% 2636 perf-stat.overall.instructions-per-iTLB-miss
8.05e+09 -3.7% 7.749e+09 perf-stat.ps.branch-instructions
22503328 ? 2% -5.9% 21186670 perf-stat.ps.cache-misses
45772 -4.5% 43725 perf-stat.ps.context-switches
5.296e+10 -2.4% 5.168e+10 perf-stat.ps.cpu-cycles
1390 -8.8% 1267 perf-stat.ps.cpu-migrations
1.147e+10 -4.0% 1.102e+10 perf-stat.ps.dTLB-loads
5.565e+09 -5.2% 5.277e+09 perf-stat.ps.dTLB-stores
16890063 -12.3% 14816058 perf-stat.ps.iTLB-load-misses
4.053e+10 -3.6% 3.905e+10 perf-stat.ps.instructions
1950358 ? 2% -6.0% 1832583 perf-stat.ps.node-loads
3450878 -5.7% 3252498 perf-stat.ps.node-stores
1.625e+12 +2.8% 1.67e+12 perf-stat.total.instructions
0.67 ? 12% -0.1 0.57 ? 4% perf-profile.calltrace.cycles-pp.file_update_time.xfs_file_write_checks.xfs_file_buffered_write.new_sync_write.vfs_write
1.33 -0.1 1.26 ? 3% perf-profile.calltrace.cycles-pp.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill.dentry_kill
1.29 -0.1 1.22 ? 3% perf-profile.calltrace.cycles-pp.release_pages.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill
1.03 ? 2% -0.1 0.98 ? 4% perf-profile.calltrace.cycles-pp.__folio_mark_dirty.filemap_dirty_folio.iomap_write_end.iomap_write_iter.iomap_file_buffered_write
0.53 ? 3% +0.1 0.64 ? 4% perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work
0.88 ? 3% +0.1 1.00 ? 3% perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work.worker_thread
0.95 ? 3% +0.1 1.08 ? 3% perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_inodegc_worker.process_one_work.worker_thread.kthread
0.96 ? 3% +0.1 1.09 ? 3% perf-profile.calltrace.cycles-pp.xfs_inodegc_worker.process_one_work.worker_thread.kthread.ret_from_fork
0.98 ? 3% +0.1 1.11 ? 3% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
0.99 ? 3% +0.1 1.12 ? 3% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
0.99 ? 3% +0.1 1.13 ? 3% perf-profile.calltrace.cycles-pp.ret_from_fork
0.99 ? 3% +0.1 1.13 ? 3% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
1.11 ? 3% +0.2 1.26 ? 6% perf-profile.calltrace.cycles-pp.lookup_open.open_last_lookups.path_openat.do_filp_open.do_sys_openat2
0.95 ? 4% +0.2 1.10 ? 6% perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.lookup_open.open_last_lookups.path_openat
0.96 ? 4% +0.2 1.12 ? 6% perf-profile.calltrace.cycles-pp.xfs_generic_create.lookup_open.open_last_lookups.path_openat.do_filp_open
0.08 ?223% +0.5 0.61 ? 8% perf-profile.calltrace.cycles-pp.xfs_dialloc.xfs_create.xfs_generic_create.lookup_open.open_last_lookups
0.00 +0.5 0.55 ? 4% perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker
0.39 ? 9% -0.1 0.31 ? 5% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.33 -0.1 1.26 ? 3% perf-profile.children.cycles-pp.__pagevec_release
1.38 -0.1 1.32 ? 3% perf-profile.children.cycles-pp.release_pages
0.59 ? 3% -0.1 0.53 ? 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.00 ? 2% -0.1 0.95 ? 3% perf-profile.children.cycles-pp.folio_alloc
0.36 ? 5% -0.0 0.31 ? 4% perf-profile.children.cycles-pp.folio_lruvec_lock_irqsave
0.38 ? 4% -0.0 0.36 ? 3% perf-profile.children.cycles-pp.__might_sleep
0.02 ?141% +0.0 0.06 ? 9% perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
0.28 ? 5% +0.1 0.37 ? 4% perf-profile.children.cycles-pp.xfs_difree_inobt
0.19 ? 3% +0.1 0.29 ? 7% perf-profile.children.cycles-pp.xfs_btree_get_rec
0.44 ? 4% +0.1 0.55 ? 4% perf-profile.children.cycles-pp.xfs_difree
0.20 ? 9% +0.1 0.30 ? 7% perf-profile.children.cycles-pp.xfs_btree_increment
0.53 ? 3% +0.1 0.64 ? 4% perf-profile.children.cycles-pp.xfs_ifree
0.88 ? 3% +0.1 1.00 ? 3% perf-profile.children.cycles-pp.xfs_inactive_ifree
0.38 ? 4% +0.1 0.50 ? 6% perf-profile.children.cycles-pp.xfs_inobt_get_rec
0.95 ? 3% +0.1 1.08 ? 3% perf-profile.children.cycles-pp.xfs_inactive
0.96 ? 3% +0.1 1.09 ? 3% perf-profile.children.cycles-pp.xfs_inodegc_worker
0.42 ? 3% +0.1 0.55 ? 7% perf-profile.children.cycles-pp.xfs_dialloc_ag
0.98 ? 3% +0.1 1.11 ? 3% perf-profile.children.cycles-pp.process_one_work
0.48 ? 3% +0.1 0.61 ? 8% perf-profile.children.cycles-pp.xfs_dialloc
0.99 ? 3% +0.1 1.12 ? 3% perf-profile.children.cycles-pp.worker_thread
0.99 ? 3% +0.1 1.13 ? 3% perf-profile.children.cycles-pp.kthread
0.99 ? 3% +0.1 1.14 ? 3% perf-profile.children.cycles-pp.ret_from_fork
1.11 ? 3% +0.2 1.26 ? 6% perf-profile.children.cycles-pp.lookup_open
0.95 ? 4% +0.2 1.10 ? 6% perf-profile.children.cycles-pp.xfs_create
0.96 ? 4% +0.2 1.12 ? 6% perf-profile.children.cycles-pp.xfs_generic_create
0.13 ? 5% +0.2 0.33 ? 6% perf-profile.children.cycles-pp.__xfs_btree_check_sblock
0.22 ? 6% +0.2 0.43 ? 6% perf-profile.children.cycles-pp.xfs_btree_check_sblock
0.68 ? 4% +0.2 0.91 ? 5% perf-profile.children.cycles-pp.xfs_check_agi_freecount
0.39 ? 9% -0.1 0.31 ? 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.43 ? 4% -0.0 0.40 ? 3% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.21 ? 4% -0.0 0.19 ? 5% perf-profile.self.cycles-pp.file_update_time
0.11 ? 4% +0.2 0.30 ? 5% perf-profile.self.cycles-pp.__xfs_btree_check_sblock
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://01.org/lkp
On Fri, May 06, 2022 at 05:22:50PM +0800, kernel test robot wrote:
>
>
> Greeting,
>
> FYI, we noticed a -5.6% regression of aim7.jobs-per-min due to commit:
>
>
> commit: 32678f151338b9a321e9e27139a63c81f353acb7 ("[PATCH 1/4] xfs: detect self referencing btree sibling pointers")
> url: https://github.com/intel-lab-lkp/linux/commits/Dave-Chinner/xfs-fix-random-format-verification-issues/20220502-162206
> base: https://git.kernel.org/cgit/fs/xfs/xfs-linux.git for-next
> patch link: https://lore.kernel.org/linux-xfs/[email protected]
Well, that answers the concern I had about the impact of
changing the way endian conversions were done in that patch.
> a44a027a8b2a20fe 32678f151338b9a321e9e27139a
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 464232 -5.6% 438315 aim7.jobs-per-min
....
> 0.13 ? 5% +0.2 0.33 ? 6% perf-profile.children.cycles-pp.__xfs_btree_check_sblock
....
> 0.11 ? 4% +0.2 0.30 ? 5% perf-profile.self.cycles-pp.__xfs_btree_check_sblock
Because there is it, right at the bottom of the profile.
Can you try the patch below and see if that fixes the issue?
Cheers,
Dave.
--
Dave Chinner
[email protected]
xfs: avoid unnecessary runtime sibling pointer endian conversions
From: Dave Chinner <[email protected]>
Commit dc04db2aa7c9 has caused a small aim7 regression, showing a
small increase in CPU usage in __xfs_btree_check_sblock() as a
result of the extra checking.
This is likely due to the endian conversion of the sibling poitners
being unconditional instead of relying on the compiler to endian
convert the NULL pointer at compile time and avoiding the runtime
conversion for this common case.
Rework the checks so that endian conversion of the sibling pointers
is only done if they are not null as the original code did.
Fixes: dc04db2aa7c9 ("xfs: detect self referencing btree sibling pointers")
Reported-by: kernel test robot <[email protected]>
Signed-off-by: Dave Chinner <[email protected]>
---
fs/xfs/libxfs/xfs_btree.c | 37 +++++++++++++++++++++++++------------
1 file changed, 25 insertions(+), 12 deletions(-)
diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c
index 2aa300f7461f..4d673e943317 100644
--- a/fs/xfs/libxfs/xfs_btree.c
+++ b/fs/xfs/libxfs/xfs_btree.c
@@ -51,16 +51,25 @@ xfs_btree_magic(
return magic;
}
+/*
+ * These sibling pointer checks are optimised for null sibling pointers. This
+ * happens a lot, and we don't need to byte swap at runtime if the sibling
+ * pointer is NULL.
+ */
static xfs_failaddr_t
xfs_btree_check_lblock_siblings(
struct xfs_mount *mp,
struct xfs_btree_cur *cur,
int level,
xfs_fsblock_t fsb,
- xfs_fsblock_t sibling)
+ __be64 dsibling)
{
- if (sibling == NULLFSBLOCK)
+ xfs_fsblock_t sibling;
+
+ if (dsibling == cpu_to_be64(NULLFSBLOCK))
return NULL;
+
+ sibling = be64_to_cpu(dsibling);
if (sibling == fsb)
return __this_address;
if (level >= 0) {
@@ -81,10 +90,14 @@ xfs_btree_check_sblock_siblings(
int level,
xfs_agnumber_t agno,
xfs_agblock_t agbno,
- xfs_agblock_t sibling)
+ __be32 dsibling)
{
- if (sibling == NULLAGBLOCK)
+ xfs_agblock_t sibling;
+
+ if (dsibling == cpu_to_be32(NULLAGBLOCK))
return NULL;
+
+ sibling = be32_to_cpu(dsibling);
if (sibling == agbno)
return __this_address;
if (level >= 0) {
@@ -136,10 +149,10 @@ __xfs_btree_check_lblock(
fsb = XFS_DADDR_TO_FSB(mp, xfs_buf_daddr(bp));
fa = xfs_btree_check_lblock_siblings(mp, cur, level, fsb,
- be64_to_cpu(block->bb_u.l.bb_leftsib));
+ block->bb_u.l.bb_leftsib);
if (!fa)
fa = xfs_btree_check_lblock_siblings(mp, cur, level, fsb,
- be64_to_cpu(block->bb_u.l.bb_rightsib));
+ block->bb_u.l.bb_rightsib);
return fa;
}
@@ -204,10 +217,10 @@ __xfs_btree_check_sblock(
}
fa = xfs_btree_check_sblock_siblings(mp, cur, level, agno, agbno,
- be32_to_cpu(block->bb_u.s.bb_leftsib));
+ block->bb_u.s.bb_leftsib);
if (!fa)
fa = xfs_btree_check_sblock_siblings(mp, cur, level, agno,
- agbno, be32_to_cpu(block->bb_u.s.bb_rightsib));
+ agbno, block->bb_u.s.bb_rightsib);
return fa;
}
@@ -4523,10 +4536,10 @@ xfs_btree_lblock_verify(
/* sibling pointer verification */
fsb = XFS_DADDR_TO_FSB(mp, xfs_buf_daddr(bp));
fa = xfs_btree_check_lblock_siblings(mp, NULL, -1, fsb,
- be64_to_cpu(block->bb_u.l.bb_leftsib));
+ block->bb_u.l.bb_leftsib);
if (!fa)
fa = xfs_btree_check_lblock_siblings(mp, NULL, -1, fsb,
- be64_to_cpu(block->bb_u.l.bb_rightsib));
+ block->bb_u.l.bb_rightsib);
return fa;
}
@@ -4580,10 +4593,10 @@ xfs_btree_sblock_verify(
agno = xfs_daddr_to_agno(mp, xfs_buf_daddr(bp));
agbno = xfs_daddr_to_agbno(mp, xfs_buf_daddr(bp));
fa = xfs_btree_check_sblock_siblings(mp, NULL, -1, agno, agbno,
- be32_to_cpu(block->bb_u.s.bb_leftsib));
+ block->bb_u.s.bb_leftsib);
if (!fa)
fa = xfs_btree_check_sblock_siblings(mp, NULL, -1, agno, agbno,
- be32_to_cpu(block->bb_u.s.bb_rightsib));
+ block->bb_u.s.bb_rightsib);
return fa;
}
Hi Dave,
On Sat, May 07, 2022 at 07:29:24AM +1000, Dave Chinner wrote:
> On Fri, May 06, 2022 at 05:22:50PM +0800, kernel test robot wrote:
> >
> >
> > Greeting,
> >
> > FYI, we noticed a -5.6% regression of aim7.jobs-per-min due to commit:
> >
> >
> > commit: 32678f151338b9a321e9e27139a63c81f353acb7 ("[PATCH 1/4] xfs: detect self referencing btree sibling pointers")
> > url: https://github.com/intel-lab-lkp/linux/commits/Dave-Chinner/xfs-fix-random-format-verification-issues/20220502-162206
> > base: https://git.kernel.org/cgit/fs/xfs/xfs-linux.git for-next
> > patch link: https://lore.kernel.org/linux-xfs/[email protected]
>
> Well, that answers the concern I had about the impact of
> changing the way endian conversions were done in that patch.
>
> > a44a027a8b2a20fe 32678f151338b9a321e9e27139a
> > ---------------- ---------------------------
> > %stddev %change %stddev
> > \ | \
> > 464232 -5.6% 438315 aim7.jobs-per-min
> ....
> > 0.13 ? 5% +0.2 0.33 ? 6% perf-profile.children.cycles-pp.__xfs_btree_check_sblock
> ....
> > 0.11 ? 4% +0.2 0.30 ? 5% perf-profile.self.cycles-pp.__xfs_btree_check_sblock
>
> Because there is it, right at the bottom of the profile.
>
> Can you try the patch below and see if that fixes the issue?
We tested below patch, it didn't fix the issue, still has -6.4% regression [1]
comparing to a44a027a8b ("Merge tag 'large-extent-counters-v9' of
https://github.com/chandanr/linux into xfs-5.19-for-next").
Dmesg file is attached.
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
gcc-11/performance/4BRD_12G/xfs/x86_64-rhel-8.3/3000/RAID0/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/disk_wrt/aim7/0x500320a
commit:
a44a027a8b ("Merge tag 'large-extent-counters-v9' of https://github.com/chandanr/linux into xfs-5.19-for-next")
32678f1513 ("xfs: detect self referencing btree sibling pointers")
92a8f471fa ("fixup-for-32678f1513")
a44a027a8b2a20fe 32678f151338b9a321e9e27139a 92a8f471fa024e1a303b0595fb9
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
464232 -5.6% 438315 -6.4% 434563 aim7.jobs-per-min
...
0.13 ? 5% +0.2 0.33 ? 6% +0.2 0.32 ? 2% perf-profile.children.cycles-pp.__xfs_btree_check_sblock
...
0.11 ? 4% +0.2 0.30 ? 5% +0.2 0.29 perf-profile.self.cycles-pp.__xfs_btree_check_sblock
[1]
a44a027a8b2a20fe 32678f151338b9a321e9e27139a 92a8f471fa024e1a303b0595fb9
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
464232 -5.6% 438315 -6.4% 434563 aim7.jobs-per-min
702.64 +2.9% 723.04 +3.9% 730.36 aim7.time.system_time
20.61 -3.7% 19.84 -2.9% 20.00 iostat.cpu.system
10.13 ?111% -13.0% 8.81 ? 97% -1.0% 10.03 ? 74% perf-stat.i.MPKI
8.246e+09 -3.9% 7.924e+09 -4.1% 7.907e+09 perf-stat.i.branch-instructions
1.44 ? 64% -0.1 1.34 ? 54% +0.1 1.52 ? 46% perf-stat.i.branch-miss-rate%
34854794 ? 5% -4.2% 33387833 ? 3% -1.9% 34209787 ? 3% perf-stat.i.branch-misses
23.30 ? 3% +1.6 24.86 ? 8% +0.1 23.36 ? 4% perf-stat.i.cache-miss-rate%
23053543 ? 2% -6.0% 21668177 -4.9% 21931373 ? 3% perf-stat.i.cache-misses
86799897 ? 8% -6.0% 81632955 ? 8% -2.2% 84874629 ? 8% perf-stat.i.cache-references
46885 -4.6% 44714 -5.4% 44356 perf-stat.i.context-switches
1.99 ? 24% -2.2% 1.95 ? 17% -0.5% 1.98 ? 18% perf-stat.i.cpi
88032 -0.0% 88025 -0.0% 88021 perf-stat.i.cpu-clock
5.425e+10 -2.6% 5.285e+10 -2.1% 5.311e+10 ? 2% perf-stat.i.cpu-cycles
1424 -8.9% 1296 -9.1% 1294 perf-stat.i.cpu-migrations
3054 ? 17% -12.8% 2664 ? 12% -5.5% 2886 ? 13% perf-stat.i.cycles-between-cache-misses
0.08 ?133% +0.0 0.10 ? 83% +0.0 0.09 ? 91% perf-stat.i.dTLB-load-miss-rate%
1644013 ?101% -39.1% 1001092 ? 20% -40.4% 979021 ? 21% perf-stat.i.dTLB-load-misses
1.175e+10 -4.1% 1.127e+10 -4.7% 1.12e+10 perf-stat.i.dTLB-loads
0.02 ?119% +0.0 0.03 ? 75% +0.0 0.02 ? 84% perf-stat.i.dTLB-store-miss-rate%
120834 ? 26% +2.8% 124168 ? 20% +1.9% 123169 ? 21% perf-stat.i.dTLB-store-misses
5.701e+09 -5.3% 5.396e+09 -6.0% 5.356e+09 perf-stat.i.dTLB-stores
72.95 ? 4% -0.5 72.49 ? 3% -0.2 72.71 ? 2% perf-stat.i.iTLB-load-miss-rate%
17300716 -12.4% 15151342 -10.8% 15425181 ? 3% perf-stat.i.iTLB-load-misses
4184079 ? 4% -2.7% 4071696 ? 3% -7.2% 3882733 perf-stat.i.iTLB-loads
4.151e+10 -3.8% 3.994e+10 -4.0% 3.986e+10 perf-stat.i.instructions
2341 ? 5% +10.1% 2578 +9.7% 2567 ? 2% perf-stat.i.instructions-per-iTLB-miss
0.68 ? 4% +0.3% 0.68 ? 2% -0.0% 0.68 ? 4% perf-stat.i.ipc
37.92 ? 40% +21.1% 45.90 ? 29% +25.1% 47.43 ? 43% perf-stat.i.major-faults
0.62 -2.6% 0.60 -2.1% 0.60 ? 2% perf-stat.i.metric.GHz
311.41 ? 42% -2.6% 303.44 ? 32% -1.7% 306.21 ? 27% perf-stat.i.metric.K/sec
292.90 -4.3% 280.20 -4.8% 278.81 perf-stat.i.metric.M/sec
7947 ? 2% -3.4% 7677 ? 2% -2.4% 7758 ? 3% perf-stat.i.minor-faults
69.35 ? 3% +0.3 69.68 ? 2% +1.3 70.64 perf-stat.i.node-load-miss-rate%
4245573 ? 3% -4.9% 4035543 ? 2% -4.7% 4043985 ? 2% perf-stat.i.node-load-misses
1997742 ? 2% -6.2% 1874006 -7.5% 1847909 perf-stat.i.node-loads
44.58 ? 4% +0.7 45.29 ? 5% -0.0 44.58 ? 5% perf-stat.i.node-store-miss-rate%
2317846 ? 3% -4.7% 2208172 ? 2% -5.6% 2188128 ? 3% perf-stat.i.node-store-misses
3534663 -5.9% 3326105 -6.3% 3312606 perf-stat.i.node-stores
7985 ? 2% -3.3% 7723 ? 2% -2.2% 7806 ? 3% perf-stat.i.page-faults
88032 -0.0% 88025 -0.0% 88021 perf-stat.i.task-clock
2.09 ? 7% -2.2% 2.04 ? 8% +1.8% 2.13 ? 8% perf-stat.overall.MPKI
0.42 ? 5% -0.0 0.42 ? 3% +0.0 0.43 ? 3% perf-stat.overall.branch-miss-rate%
26.70 ? 6% +0.0 26.70 ? 7% -0.7 26.01 ? 7% perf-stat.overall.cache-miss-rate%
1.31 +1.3% 1.32 +2.0% 1.33 perf-stat.overall.cpi
2354 ? 2% +3.6% 2440 ? 2% +2.9% 2423 ? 2% perf-stat.overall.cycles-between-cache-misses
0.01 ?101% -0.0 0.01 ? 20% -0.0 0.01 ? 21% perf-stat.overall.dTLB-load-miss-rate%
0.00 ? 25% +0.0 0.00 ? 20% +0.0 0.00 ? 21% perf-stat.overall.dTLB-store-miss-rate%
80.54 -1.7 78.83 -0.7 79.88 perf-stat.overall.iTLB-load-miss-rate%
2400 +9.8% 2636 +7.8% 2586 ? 3% perf-stat.overall.instructions-per-iTLB-miss
0.77 -1.3% 0.76 -1.9% 0.75 perf-stat.overall.ipc
67.98 +0.3 68.28 +0.6 68.63 perf-stat.overall.node-load-miss-rate%
39.59 ? 2% +0.3 39.90 +0.2 39.76 perf-stat.overall.node-store-miss-rate%
8.05e+09 -3.7% 7.749e+09 -4.0% 7.729e+09 perf-stat.ps.branch-instructions
33999070 ? 5% -4.1% 32614324 ? 3% -1.7% 33414902 ? 3% perf-stat.ps.branch-misses
22503328 ? 2% -5.9% 21186670 -4.8% 21434254 ? 2% perf-stat.ps.cache-misses
84713531 ? 7% -5.8% 79798101 ? 8% -2.1% 82935444 ? 8% perf-stat.ps.cache-references
45772 -4.5% 43725 -5.3% 43356 perf-stat.ps.context-switches
85806 +0.2% 85944 +0.2% 85955 perf-stat.ps.cpu-clock
5.296e+10 -2.4% 5.168e+10 -2.0% 5.191e+10 ? 2% perf-stat.ps.cpu-cycles
1390 -8.8% 1267 -9.0% 1264 perf-stat.ps.cpu-migrations
1603543 ?101% -39.0% 977805 ? 20% -40.4% 956223 ? 21% perf-stat.ps.dTLB-load-misses
1.147e+10 -4.0% 1.102e+10 -4.6% 1.094e+10 perf-stat.ps.dTLB-loads
117801 ? 26% +2.9% 121212 ? 19% +2.1% 120248 ? 21% perf-stat.ps.dTLB-store-misses
5.565e+09 -5.2% 5.277e+09 -5.9% 5.236e+09 perf-stat.ps.dTLB-stores
16890063 -12.3% 14816058 -10.7% 15077842 ? 3% perf-stat.ps.iTLB-load-misses
4083261 ? 4% -2.5% 3980087 ? 3% -7.1% 3794317 perf-stat.ps.iTLB-loads
4.053e+10 -3.6% 3.905e+10 -3.9% 3.896e+10 perf-stat.ps.instructions
36.83 ? 40% +21.4% 44.70 ? 29% +25.1% 46.07 ? 42% perf-stat.ps.major-faults
7724 ? 2% -3.2% 7476 ? 2% -2.2% 7551 ? 3% perf-stat.ps.minor-faults
4144187 ? 3% -4.8% 3946230 ? 2% -4.6% 3952384 ? 2% perf-stat.ps.node-load-misses
1950358 ? 2% -6.0% 1832583 -7.4% 1806333 perf-stat.ps.node-loads
2262554 ? 3% -4.6% 2159317 ? 2% -5.5% 2138598 ? 3% perf-stat.ps.node-store-misses
3450878 -5.7% 3252498 -6.2% 3237962 perf-stat.ps.node-stores
7761 ? 2% -3.1% 7521 ? 2% -2.1% 7597 ? 3% perf-stat.ps.page-faults
85806 +0.2% 85944 +0.2% 85955 perf-stat.ps.task-clock
1.625e+12 +2.8% 1.67e+12 +3.0% 1.673e+12 perf-stat.total.instructions
29.46 ? 3% -1.1 28.33 ? 3% -1.2 28.25 ? 2% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
28.94 ? 3% -1.1 27.84 ? 3% -1.2 27.76 ? 2% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
27.28 ? 3% -1.0 26.25 ? 3% -1.1 26.20 ? 2% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
26.46 ? 3% -1.0 25.46 ? 3% -1.0 25.42 ? 2% perf-profile.calltrace.cycles-pp.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
23.17 ? 3% -0.8 22.36 ? 3% -0.9 22.30 ? 2% perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write
18.02 ? 3% -0.6 17.41 ? 3% -0.6 17.37 ? 2% perf-profile.calltrace.cycles-pp.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write.vfs_write
22.03 ? 12% -0.4 21.60 ? 10% -0.6 21.46 ? 9% perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
22.14 ? 12% -0.3 21.79 ? 9% -0.6 21.51 ? 9% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
9.93 ? 3% -0.3 9.61 ? 2% -0.3 9.60 ? 2% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write
23.55 ? 12% -0.3 23.23 ? 8% -0.6 22.94 ? 9% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
22.88 ? 12% -0.3 22.62 ? 9% -0.6 22.31 ? 9% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
22.93 ? 12% -0.3 22.68 ? 9% -0.6 22.36 ? 9% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
23.08 ? 12% -0.2 22.84 ? 8% -0.6 22.52 ? 9% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
23.21 ? 12% -0.2 22.98 ? 8% -0.6 22.65 ? 9% perf-profile.calltrace.cycles-pp.cpu_startup_entry.secondary_startup_64_no_verify
23.20 ? 12% -0.2 22.98 ? 8% -0.6 22.65 ? 9% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.secondary_startup_64_no_verify
4.49 ? 2% -0.2 4.31 ? 3% -0.2 4.29 ? 3% perf-profile.calltrace.cycles-pp.iomap_iter.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write.vfs_write
5.22 ? 2% -0.2 5.05 ? 3% -0.2 5.01 ? 2% perf-profile.calltrace.cycles-pp.__filemap_get_folio.iomap_write_begin.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write
0.17 ?141% -0.2 0.00 -0.2 0.00 perf-profile.calltrace.cycles-pp.xfs_break_layouts.xfs_file_write_checks.xfs_file_buffered_write.new_sync_write.vfs_write
3.36 -0.1 3.22 ? 3% -0.2 3.18 ? 2% perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
3.38 -0.1 3.24 ? 3% -0.2 3.20 ? 2% perf-profile.calltrace.cycles-pp.__close
3.35 ? 2% -0.1 3.21 ? 3% -0.2 3.18 ? 2% perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
3.37 -0.1 3.23 ? 3% -0.2 3.20 ? 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close
3.37 -0.1 3.23 ? 3% -0.2 3.20 ? 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
3.34 -0.1 3.20 ? 3% -0.2 3.16 ? 2% perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare
3.36 -0.1 3.22 ? 3% -0.2 3.19 ? 2% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
3.36 -0.1 3.22 ? 3% -0.2 3.19 ? 2% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close
3.36 -0.1 3.22 ? 3% -0.2 3.19 ? 2% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.33 ? 2% -0.1 3.19 ? 3% -0.2 3.16 ? 2% perf-profile.calltrace.cycles-pp.dentry_kill.dput.__fput.task_work_run.exit_to_user_mode_loop
3.32 -0.1 3.19 ? 3% -0.2 3.15 ? 2% perf-profile.calltrace.cycles-pp.__dentry_kill.dentry_kill.dput.__fput.task_work_run
3.29 ? 2% -0.1 3.16 ? 3% -0.2 3.12 ? 2% perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dentry_kill.dput.__fput
3.27 ? 2% -0.1 3.14 ? 3% -0.2 3.10 ? 2% perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.__dentry_kill.dentry_kill.dput
2.96 ? 2% -0.1 2.84 ? 3% -0.1 2.82 ? 4% perf-profile.calltrace.cycles-pp.xfs_buffered_write_iomap_begin.iomap_iter.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write
4.28 ? 3% -0.1 4.16 ? 2% -0.1 4.18 ? 2% perf-profile.calltrace.cycles-pp.__iomap_write_begin.iomap_write_begin.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write
1.75 ? 6% -0.1 1.64 ? 2% -0.1 1.63 perf-profile.calltrace.cycles-pp.xfs_file_write_checks.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write
1.08 ? 25% -0.1 0.96 ? 21% -0.2 0.87 ? 5% perf-profile.calltrace.cycles-pp.__entry_text_start.write
3.94 ? 3% -0.1 3.84 ? 3% -0.1 3.83 ? 2% perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write
0.67 ? 12% -0.1 0.57 ? 4% -0.1 0.59 ? 5% perf-profile.calltrace.cycles-pp.file_update_time.xfs_file_write_checks.xfs_file_buffered_write.new_sync_write.vfs_write
2.63 ? 3% -0.1 2.54 ? 3% -0.1 2.53 ? 2% perf-profile.calltrace.cycles-pp.filemap_add_folio.__filemap_get_folio.iomap_write_begin.iomap_write_iter.iomap_file_buffered_write
1.69 ? 3% -0.1 1.62 ? 3% -0.1 1.63 ? 2% perf-profile.calltrace.cycles-pp.filemap_dirty_folio.iomap_write_end.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write
1.33 -0.1 1.26 ? 3% -0.1 1.24 ? 2% perf-profile.calltrace.cycles-pp.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill.dentry_kill
1.29 -0.1 1.22 ? 3% -0.1 1.20 ? 2% perf-profile.calltrace.cycles-pp.release_pages.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill
1.93 ? 3% -0.1 1.86 ? 3% -0.1 1.85 ? 2% perf-profile.calltrace.cycles-pp.copy_page_from_iter_atomic.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write
1.21 ? 4% -0.1 1.15 ? 4% -0.1 1.15 perf-profile.calltrace.cycles-pp.zero_user_segments.__iomap_write_begin.iomap_write_begin.iomap_write_iter.iomap_file_buffered_write
1.00 ? 2% -0.1 0.94 ? 3% -0.1 0.94 ? 3% perf-profile.calltrace.cycles-pp.folio_alloc.__filemap_get_folio.iomap_write_begin.iomap_write_iter.iomap_file_buffered_write
1.14 ? 4% -0.1 1.09 ? 4% -0.0 1.10 ? 2% perf-profile.calltrace.cycles-pp.memset_erms.zero_user_segments.__iomap_write_begin.iomap_write_begin.iomap_write_iter
1.50 ? 3% -0.1 1.44 ? 3% -0.1 1.42 ? 2% perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter_atomic.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write
0.89 ? 4% -0.1 0.84 ? 4% -0.1 0.83 ? 5% perf-profile.calltrace.cycles-pp.__pagevec_lru_add.folio_add_lru.filemap_add_folio.__filemap_get_folio.iomap_write_begin
1.03 ? 2% -0.1 0.98 ? 4% -0.0 0.99 ? 3% perf-profile.calltrace.cycles-pp.__folio_mark_dirty.filemap_dirty_folio.iomap_write_end.iomap_write_iter.iomap_file_buffered_write
1.36 ? 3% -0.1 1.31 ? 3% -0.1 1.30 ? 2% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter_atomic.iomap_write_iter.iomap_file_buffered_write
0.98 ? 3% -0.1 0.92 ? 3% -0.1 0.92 ? 4% perf-profile.calltrace.cycles-pp.folio_add_lru.filemap_add_folio.__filemap_get_folio.iomap_write_begin.iomap_write_iter
1.61 ? 3% -0.0 1.57 ? 2% -0.1 1.56 perf-profile.calltrace.cycles-pp.__filemap_add_folio.filemap_add_folio.__filemap_get_folio.iomap_write_begin.iomap_write_iter
0.83 ? 3% -0.0 0.79 ? 4% -0.1 0.78 ? 2% perf-profile.calltrace.cycles-pp.__alloc_pages.folio_alloc.__filemap_get_folio.iomap_write_begin.iomap_write_iter
0.74 ? 2% -0.0 0.70 ? 2% -0.0 0.71 ? 4% perf-profile.calltrace.cycles-pp.truncate_cleanup_folio.truncate_inode_pages_range.evict.__dentry_kill.dentry_kill
0.69 ? 2% -0.0 0.66 ? 4% -0.0 0.66 ? 2% perf-profile.calltrace.cycles-pp.folio_account_dirtied.__folio_mark_dirty.filemap_dirty_folio.iomap_write_end.iomap_write_iter
0.58 ? 4% -0.0 0.55 ? 4% -0.1 0.46 ? 44% perf-profile.calltrace.cycles-pp.__pagevec_lru_add_fn.__pagevec_lru_add.folio_add_lru.filemap_add_folio.__filemap_get_folio
0.62 ? 3% -0.0 0.60 -0.0 0.60 ? 4% perf-profile.calltrace.cycles-pp.__folio_cancel_dirty.truncate_cleanup_folio.truncate_inode_pages_range.evict.__dentry_kill
0.65 ? 4% -0.0 0.62 ? 3% -0.0 0.63 ? 3% perf-profile.calltrace.cycles-pp.__mem_cgroup_charge.__filemap_add_folio.filemap_add_folio.__filemap_get_folio.iomap_write_begin
0.80 ? 3% -0.0 0.78 ? 4% -0.0 0.78 ? 3% perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_buffered_write_iomap_begin.iomap_iter.iomap_file_buffered_write.xfs_file_buffered_write
0.77 ? 2% -0.0 0.75 ? 3% -0.0 0.74 ? 3% perf-profile.calltrace.cycles-pp.delete_from_page_cache_batch.truncate_inode_pages_range.evict.__dentry_kill.dentry_kill
0.54 ? 5% -0.0 0.53 ? 4% -0.2 0.36 ? 70% perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64
0.59 ? 3% -0.0 0.58 ? 3% -0.0 0.56 ? 2% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages.folio_alloc.__filemap_get_folio.iomap_write_begin
0.67 ? 2% -0.0 0.66 ? 4% -0.0 0.65 ? 4% perf-profile.calltrace.cycles-pp.down_write.xfs_ilock.xfs_buffered_write_iomap_begin.iomap_iter.iomap_file_buffered_write
0.64 ? 4% -0.0 0.63 ? 2% -0.0 0.62 ? 3% perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_file_buffered_write.new_sync_write.vfs_write.ksys_write
0.62 ? 5% -0.0 0.60 ? 4% -0.0 0.60 ? 4% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.82 ? 2% -0.0 0.81 ? 5% -0.0 0.81 ? 3% perf-profile.calltrace.cycles-pp.fault_in_iov_iter_readable.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.new_sync_write
0.44 ? 44% -0.0 0.43 ? 44% -0.1 0.35 ? 70% perf-profile.calltrace.cycles-pp.down_write.xfs_ilock.xfs_file_buffered_write.new_sync_write.vfs_write
0.68 ? 3% -0.0 0.68 ? 6% -0.0 0.68 ? 4% perf-profile.calltrace.cycles-pp.fault_in_readable.fault_in_iov_iter_readable.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write
0.81 ? 4% +0.0 0.82 ? 3% -0.0 0.81 ? 3% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink
0.74 ? 4% +0.0 0.76 +0.0 0.75 ? 3% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.open_last_lookups.path_openat
0.68 ? 2% +0.0 0.71 ? 9% -0.0 0.65 ? 2% perf-profile.calltrace.cycles-pp.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.__x64_sys_unlink
0.70 +0.0 0.72 ? 9% -0.0 0.66 ? 2% perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.68 ? 2% +0.0 0.71 ? 9% -0.0 0.65 ? 2% perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.__x64_sys_unlink.do_syscall_64
0.63 ? 5% +0.1 0.70 ? 12% +0.0 0.68 ? 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
0.68 ? 5% +0.1 0.76 ? 12% +0.1 0.73 ? 4% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.53 ? 3% +0.1 0.64 ? 4% +0.1 0.64 ? 4% perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work
0.88 ? 3% +0.1 1.00 ? 3% +0.1 0.99 ? 4% perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work.worker_thread
0.95 ? 3% +0.1 1.08 ? 3% +0.1 1.05 ? 4% perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_inodegc_worker.process_one_work.worker_thread.kthread
0.96 ? 3% +0.1 1.09 ? 3% +0.1 1.06 ? 4% perf-profile.calltrace.cycles-pp.xfs_inodegc_worker.process_one_work.worker_thread.kthread.ret_from_fork
0.98 ? 3% +0.1 1.11 ? 3% +0.1 1.09 ? 4% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
0.99 ? 3% +0.1 1.12 ? 3% +0.1 1.09 ? 3% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
0.99 ? 3% +0.1 1.13 ? 3% +0.1 1.10 ? 3% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
0.99 ? 3% +0.1 1.13 ? 3% +0.1 1.10 ? 3% perf-profile.calltrace.cycles-pp.ret_from_fork
1.11 ? 3% +0.2 1.26 ? 6% +0.1 1.19 perf-profile.calltrace.cycles-pp.lookup_open.open_last_lookups.path_openat.do_filp_open.do_sys_openat2
0.95 ? 4% +0.2 1.10 ? 6% +0.1 1.04 perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.lookup_open.open_last_lookups.path_openat
0.96 ? 4% +0.2 1.12 ? 6% +0.1 1.05 perf-profile.calltrace.cycles-pp.xfs_generic_create.lookup_open.open_last_lookups.path_openat.do_filp_open
0.00 +0.3 0.26 ?100% +0.0 0.00 perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc.xfs_create.xfs_generic_create
0.00 +0.3 0.32 ?223% +0.0 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter
0.00 +0.5 0.47 ? 45% +0.5 0.53 ? 3% perf-profile.calltrace.cycles-pp.xfs_dialloc_ag.xfs_dialloc.xfs_create.xfs_generic_create.lookup_open
0.08 ?223% +0.5 0.61 ? 8% +0.5 0.58 ? 2% perf-profile.calltrace.cycles-pp.xfs_dialloc.xfs_create.xfs_generic_create.lookup_open.open_last_lookups
0.00 +0.5 0.55 ? 4% +0.6 0.55 ? 4% perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker
17.48 ? 4% +0.6 18.12 ? 3% +1.0 18.48 ? 3% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink
18.32 ? 4% +0.6 18.95 ? 3% +1.0 19.31 ? 3% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink.do_syscall_64
17.80 ? 5% +0.7 18.46 ? 3% +0.9 18.75 ? 3% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.open_last_lookups.path_openat
18.58 ? 4% +0.7 19.24 ? 2% +1.0 19.57 ? 3% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe
18.56 ? 5% +0.7 19.24 ? 2% +1.0 19.52 ? 3% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.open_last_lookups.path_openat.do_filp_open
19.36 ? 4% +0.7 20.06 ? 2% +1.0 20.32 ? 3% perf-profile.calltrace.cycles-pp.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
19.37 ? 4% +0.7 20.07 ? 2% +1.0 20.33 ? 3% perf-profile.calltrace.cycles-pp.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
19.38 ? 4% +0.7 20.08 ? 2% +1.0 20.34 ? 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
19.38 ? 4% +0.7 20.08 ? 2% +1.0 20.34 ? 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
19.40 ? 4% +0.7 20.10 ? 2% +1.0 20.36 ? 3% perf-profile.calltrace.cycles-pp.unlink
18.82 ? 5% +0.7 19.53 ? 2% +1.0 19.78 ? 3% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.open_last_lookups.path_openat.do_filp_open.do_sys_openat2
20.10 ? 4% +0.9 20.96 ? 3% +1.0 21.13 ? 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat64
20.09 ? 4% +0.9 20.95 ? 3% +1.0 21.12 ? 3% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_creat.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
20.09 ? 4% +0.9 20.95 ? 3% +1.0 21.12 ? 3% perf-profile.calltrace.cycles-pp.__x64_sys_creat.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
20.07 ? 4% +0.9 20.93 ? 3% +1.0 21.10 ? 3% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_creat.do_syscall_64.entry_SYSCALL_64_after_hwframe
20.00 ? 4% +0.9 20.86 ? 3% +1.0 21.04 ? 3% perf-profile.calltrace.cycles-pp.open_last_lookups.path_openat.do_filp_open.do_sys_openat2.__x64_sys_creat
20.06 ? 4% +0.9 20.92 ? 3% +1.0 21.10 ? 3% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_creat.do_syscall_64
20.10 ? 4% +0.9 20.96 ? 3% +1.0 21.13 ? 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
20.11 ? 4% +0.9 20.98 ? 3% +1.0 21.15 ? 3% perf-profile.calltrace.cycles-pp.creat64
32.16 ? 3% -1.2 30.92 ? 2% -1.3 30.86 ? 2% perf-profile.children.cycles-pp.write
29.48 ? 3% -1.1 28.34 ? 3% -1.2 28.26 ? 2% perf-profile.children.cycles-pp.ksys_write
28.97 ? 3% -1.1 27.86 ? 3% -1.2 27.79 ? 2% perf-profile.children.cycles-pp.vfs_write
27.30 ? 3% -1.0 26.26 ? 3% -1.1 26.23 ? 2% perf-profile.children.cycles-pp.new_sync_write
26.50 ? 3% -1.0 25.51 ? 3% -1.0 25.46 ? 2% perf-profile.children.cycles-pp.xfs_file_buffered_write
23.21 ? 3% -0.8 22.39 ? 3% -0.9 22.33 ? 2% perf-profile.children.cycles-pp.iomap_file_buffered_write
18.05 ? 3% -0.6 17.45 ? 3% -0.6 17.41 ? 2% perf-profile.children.cycles-pp.iomap_write_iter
22.46 ? 12% -0.4 22.03 ? 9% -0.7 21.77 ? 9% perf-profile.children.cycles-pp.intel_idle
22.45 ? 12% -0.4 22.02 ? 9% -0.7 21.77 ? 9% perf-profile.children.cycles-pp.mwait_idle_with_hints
23.27 ? 12% -0.3 22.92 ? 9% -0.6 22.64 ? 9% perf-profile.children.cycles-pp.cpuidle_enter_state
23.27 ? 12% -0.3 22.93 ? 9% -0.6 22.65 ? 9% perf-profile.children.cycles-pp.cpuidle_enter
9.95 ? 3% -0.3 9.63 ? 2% -0.3 9.61 ? 2% perf-profile.children.cycles-pp.iomap_write_begin
23.55 ? 12% -0.3 23.23 ? 8% -0.6 22.94 ? 9% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
23.55 ? 12% -0.3 23.23 ? 8% -0.6 22.94 ? 9% perf-profile.children.cycles-pp.cpu_startup_entry
23.55 ? 12% -0.3 23.23 ? 8% -0.6 22.94 ? 9% perf-profile.children.cycles-pp.do_idle
23.41 ? 12% -0.3 23.09 ? 8% -0.6 22.81 ? 9% perf-profile.children.cycles-pp.cpuidle_idle_call
5.30 ? 3% -0.2 5.11 ? 2% -0.2 5.08 ? 2% perf-profile.children.cycles-pp.__filemap_get_folio
4.52 ? 2% -0.2 4.34 ? 3% -0.2 4.32 ? 3% perf-profile.children.cycles-pp.iomap_iter
3.38 -0.1 3.24 ? 3% -0.2 3.20 ? 2% perf-profile.children.cycles-pp.__close
3.56 -0.1 3.42 ? 3% -0.2 3.38 ? 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
3.36 -0.1 3.22 ? 3% -0.2 3.18 ? 2% perf-profile.children.cycles-pp.task_work_run
3.35 ? 2% -0.1 3.21 ? 3% -0.2 3.18 ? 2% perf-profile.children.cycles-pp.__fput
3.36 -0.1 3.22 ? 3% -0.2 3.19 ? 2% perf-profile.children.cycles-pp.dput
3.33 ? 2% -0.1 3.19 ? 3% -0.2 3.16 ? 2% perf-profile.children.cycles-pp.dentry_kill
3.36 -0.1 3.22 ? 3% -0.2 3.19 ? 2% perf-profile.children.cycles-pp.exit_to_user_mode_loop
3.46 -0.1 3.32 ? 3% -0.2 3.29 ? 2% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
3.32 -0.1 3.19 ? 3% -0.2 3.15 ? 2% perf-profile.children.cycles-pp.__dentry_kill
3.29 ? 2% -0.1 3.16 ? 3% -0.2 3.12 ? 2% perf-profile.children.cycles-pp.evict
3.27 -0.1 3.14 ? 3% -0.2 3.10 ? 2% perf-profile.children.cycles-pp.truncate_inode_pages_range
4.31 ? 3% -0.1 4.18 ? 2% -0.1 4.20 ? 2% perf-profile.children.cycles-pp.__iomap_write_begin
3.02 ? 2% -0.1 2.90 ? 3% -0.1 2.88 ? 3% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
1.79 ? 6% -0.1 1.66 ? 3% -0.1 1.66 perf-profile.children.cycles-pp.xfs_file_write_checks
3.96 ? 3% -0.1 3.85 ? 3% -0.1 3.85 ? 2% perf-profile.children.cycles-pp.iomap_write_end
0.68 ? 12% -0.1 0.58 ? 4% -0.1 0.60 ? 5% perf-profile.children.cycles-pp.file_update_time
2.64 ? 3% -0.1 2.55 ? 2% -0.1 2.53 ? 2% perf-profile.children.cycles-pp.filemap_add_folio
0.34 ? 23% -0.1 0.25 ? 23% -0.0 0.29 ? 26% perf-profile.children.cycles-pp.start_kernel
1.71 ? 3% -0.1 1.63 ? 4% -0.1 1.65 ? 2% perf-profile.children.cycles-pp.filemap_dirty_folio
0.39 ? 9% -0.1 0.31 ? 5% -0.1 0.31 ? 6% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.33 -0.1 1.26 ? 3% -0.1 1.24 ? 2% perf-profile.children.cycles-pp.__pagevec_release
1.94 ? 3% -0.1 1.87 ? 3% -0.1 1.86 ? 2% perf-profile.children.cycles-pp.copy_page_from_iter_atomic
1.38 -0.1 1.32 ? 3% -0.1 1.30 ? 2% perf-profile.children.cycles-pp.release_pages
0.26 ? 31% -0.1 0.20 ? 7% -0.1 0.21 ? 19% perf-profile.children.cycles-pp.xfs_vn_update_time
0.59 ? 3% -0.1 0.53 ? 3% -0.1 0.52 ? 5% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.51 ? 3% -0.1 1.45 ? 3% -0.1 1.44 ? 2% perf-profile.children.cycles-pp.copyin
1.21 ? 4% -0.1 1.15 ? 4% -0.1 1.16 perf-profile.children.cycles-pp.zero_user_segments
1.16 ? 4% -0.1 1.11 ? 4% -0.0 1.12 ? 2% perf-profile.children.cycles-pp.memset_erms
1.00 ? 2% -0.1 0.95 ? 3% -0.1 0.94 ? 3% perf-profile.children.cycles-pp.folio_alloc
0.91 ? 4% -0.1 0.86 ? 3% -0.1 0.86 ? 5% perf-profile.children.cycles-pp.__pagevec_lru_add
1.04 ? 2% -0.0 0.98 ? 4% -0.0 1.00 ? 2% perf-profile.children.cycles-pp.__folio_mark_dirty
1.45 ? 3% -0.0 1.40 ? 3% -0.1 1.38 ? 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.98 ? 3% -0.0 0.93 ? 3% -0.1 0.92 ? 4% perf-profile.children.cycles-pp.folio_add_lru
0.36 ? 5% -0.0 0.31 ? 4% -0.1 0.30 ? 6% perf-profile.children.cycles-pp.folio_lruvec_lock_irqsave
0.47 ? 7% -0.0 0.43 ? 8% -0.0 0.42 ? 7% perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
1.64 ? 3% -0.0 1.59 ? 2% -0.1 1.58 perf-profile.children.cycles-pp.__filemap_add_folio
0.84 ? 2% -0.0 0.80 ? 4% -0.1 0.79 ? 2% perf-profile.children.cycles-pp.__alloc_pages
0.82 ? 3% -0.0 0.78 ? 2% -0.0 0.79 ? 3% perf-profile.children.cycles-pp.xfs_iunlock
0.27 ? 14% -0.0 0.23 ? 7% -0.0 0.24 ? 9% perf-profile.children.cycles-pp._raw_spin_lock
0.05 ? 45% -0.0 0.01 ?223% -0.0 0.02 ?141% perf-profile.children.cycles-pp.__x64_sys_write
1.47 ? 2% -0.0 1.44 ? 3% -0.0 1.43 ? 3% perf-profile.children.cycles-pp.xfs_ilock
0.74 ? 2% -0.0 0.71 ? 2% -0.0 0.71 ? 4% perf-profile.children.cycles-pp.truncate_cleanup_folio
0.70 ? 2% -0.0 0.67 ? 5% -0.0 0.68 ? 2% perf-profile.children.cycles-pp.folio_account_dirtied
0.46 ? 5% -0.0 0.43 ? 4% -0.0 0.43 ? 4% perf-profile.children.cycles-pp.percpu_counter_add_batch
0.60 ? 4% -0.0 0.56 ? 4% -0.0 0.57 ? 5% perf-profile.children.cycles-pp.__pagevec_lru_add_fn
0.38 ? 4% -0.0 0.36 ? 3% -0.0 0.36 ? 3% perf-profile.children.cycles-pp.__might_sleep
0.91 ? 4% -0.0 0.88 ? 3% -0.0 0.88 ? 4% perf-profile.children.cycles-pp.__entry_text_start
0.36 ? 6% -0.0 0.33 ? 3% -0.0 0.33 ? 5% perf-profile.children.cycles-pp.__fdget_pos
0.28 ? 7% -0.0 0.25 ? 3% -0.0 0.26 ? 7% perf-profile.children.cycles-pp.__fget_light
0.51 ? 3% -0.0 0.48 ? 5% -0.0 0.49 ? 3% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.63 ? 3% -0.0 0.60 ? 2% -0.0 0.61 ? 4% perf-profile.children.cycles-pp.__folio_cancel_dirty
0.59 ? 3% -0.0 0.56 ? 3% -0.0 0.58 ? 3% perf-profile.children.cycles-pp.up_write
0.02 ? 99% -0.0 0.00 -0.0 0.01 ?223% perf-profile.children.cycles-pp.__x64_sys_times
0.03 ? 70% -0.0 0.01 ?223% -0.0 0.00 perf-profile.children.cycles-pp.propagate_protected_usage
1.11 ? 7% -0.0 1.08 ? 4% -0.1 1.04 ? 3% perf-profile.children.cycles-pp.__xfs_trans_commit
0.50 ? 4% -0.0 0.48 ? 3% -0.0 0.47 ? 4% perf-profile.children.cycles-pp.xfs_break_layouts
1.28 ? 2% -0.0 1.26 ? 2% -0.0 1.25 ? 4% perf-profile.children.cycles-pp.down_write
0.46 ? 3% -0.0 0.44 ? 3% -0.0 0.44 ? 5% perf-profile.children.cycles-pp.folio_account_cleaned
0.42 ? 3% -0.0 0.40 ? 3% -0.0 0.40 ? 3% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_end
0.34 ? 3% -0.0 0.32 ? 5% -0.0 0.31 ? 6% perf-profile.children.cycles-pp.xfs_iext_lookup_extent
0.43 ? 3% -0.0 0.41 ? 5% -0.0 0.40 perf-profile.children.cycles-pp.xfs_file_write_iter
0.30 ? 4% -0.0 0.28 ? 2% -0.0 0.28 ? 4% perf-profile.children.cycles-pp.xfs_break_leased_layouts
0.78 ? 5% -0.0 0.76 ? 2% -0.0 0.76 ? 4% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.65 ? 4% -0.0 0.63 ? 3% -0.0 0.63 ? 2% perf-profile.children.cycles-pp.__mem_cgroup_charge
0.79 -0.0 0.77 ? 4% -0.0 0.78 ? 3% perf-profile.children.cycles-pp.__mod_lruvec_page_state
1.08 ? 6% -0.0 1.06 ? 4% -0.1 1.01 ? 3% perf-profile.children.cycles-pp.xlog_cil_commit
0.77 ? 2% -0.0 0.75 ? 3% -0.0 0.75 ? 3% perf-profile.children.cycles-pp.delete_from_page_cache_batch
0.41 ? 6% -0.0 0.39 ? 5% -0.0 0.40 ? 5% perf-profile.children.cycles-pp.__cond_resched
0.60 ? 3% -0.0 0.58 ? 4% -0.0 0.57 perf-profile.children.cycles-pp.get_page_from_freelist
0.09 ? 13% -0.0 0.08 ? 9% -0.0 0.08 ? 7% perf-profile.children.cycles-pp.xfs_get_extsz_hint
0.61 ? 3% -0.0 0.59 ? 2% -0.0 0.58 ? 2% perf-profile.children.cycles-pp.xas_store
0.56 ? 5% -0.0 0.54 ? 4% -0.0 0.54 ? 5% perf-profile.children.cycles-pp.apparmor_file_permission
0.63 ? 5% -0.0 0.61 ? 4% -0.0 0.61 ? 5% perf-profile.children.cycles-pp.security_file_permission
0.12 ? 6% -0.0 0.11 ? 10% -0.0 0.11 ? 8% perf-profile.children.cycles-pp.disk_wrt
0.10 ? 7% -0.0 0.09 ? 7% -0.0 0.09 ? 9% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.45 ? 2% -0.0 0.43 ? 4% -0.0 0.43 ? 5% perf-profile.children.cycles-pp.charge_memcg
0.24 ? 3% -0.0 0.23 ? 3% -0.0 0.23 ? 4% perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
0.40 ? 3% -0.0 0.38 ? 5% -0.0 0.37 ? 4% perf-profile.children.cycles-pp.__mem_cgroup_uncharge_list
0.30 ? 8% -0.0 0.28 ? 6% +0.0 0.30 ? 6% perf-profile.children.cycles-pp.xfs_errortag_test
0.05 ? 8% -0.0 0.04 ? 72% -0.0 0.04 ? 45% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.15 ? 7% -0.0 0.14 ? 8% -0.0 0.14 ? 7% perf-profile.children.cycles-pp.folio_memcg_lock
0.84 ? 3% -0.0 0.83 ? 5% -0.0 0.83 ? 3% perf-profile.children.cycles-pp.fault_in_iov_iter_readable
0.02 ?144% -0.0 0.01 ?223% -0.0 0.00 perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
0.28 ? 4% -0.0 0.27 ? 6% -0.0 0.26 ? 5% perf-profile.children.cycles-pp.page_counter_uncharge
0.12 ? 6% -0.0 0.10 ? 4% -0.0 0.11 ? 10% perf-profile.children.cycles-pp.aa_file_perm
0.73 ? 6% -0.0 0.72 ? 4% -0.0 0.70 ? 3% perf-profile.children.cycles-pp.xlog_cil_insert_items
0.34 ? 3% -0.0 0.33 ? 4% -0.0 0.32 ? 4% perf-profile.children.cycles-pp.find_lock_entries
0.07 ? 6% -0.0 0.06 ? 11% -0.0 0.07 ? 7% perf-profile.children.cycles-pp.filemap_free_folio
0.05 ? 8% -0.0 0.04 ? 45% -0.0 0.05 ? 46% perf-profile.children.cycles-pp.schedule_idle
0.01 ?223% -0.0 0.00 -0.0 0.01 ?223% perf-profile.children.cycles-pp.xlog_grant_add_space
0.21 ? 4% -0.0 0.20 ? 5% -0.0 0.20 ? 3% perf-profile.children.cycles-pp.current_time
0.05 -0.0 0.04 ? 44% -0.0 0.03 ? 70% perf-profile.children.cycles-pp.xfs_buf_item_size
0.12 ? 3% -0.0 0.12 ? 4% -0.0 0.12 ? 8% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.02 ?141% -0.0 0.01 ?223% -0.0 0.01 ?223% perf-profile.children.cycles-pp.xfs_buf_item_size_segment
0.01 ?223% -0.0 0.00 -0.0 0.00 perf-profile.children.cycles-pp.xfs_buf_rele
0.01 ?223% -0.0 0.00 -0.0 0.00 perf-profile.children.cycles-pp.xfs_trans_alloc_dir
0.01 ?223% -0.0 0.00 +0.0 0.01 ?223% perf-profile.children.cycles-pp.xfs_inode_to_log_dinode
0.18 ? 10% -0.0 0.17 ? 10% -0.0 0.17 ? 4% perf-profile.children.cycles-pp.inode_to_bdi
0.13 ? 6% -0.0 0.12 ? 6% -0.0 0.12 ? 6% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.05 ? 46% -0.0 0.04 ? 44% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.fprintf
0.08 -0.0 0.07 ? 5% -0.0 0.08 ? 9% perf-profile.children.cycles-pp.PageHeadHuge
0.10 ? 8% -0.0 0.09 ? 7% -0.0 0.09 ? 6% perf-profile.children.cycles-pp.__list_add_valid
0.66 ? 3% -0.0 0.65 ? 3% -0.0 0.64 ? 2% perf-profile.children.cycles-pp.xas_load
0.30 ? 3% -0.0 0.30 ? 6% -0.0 0.29 ? 3% perf-profile.children.cycles-pp.uncharge_batch
0.23 ? 5% -0.0 0.22 ? 3% -0.0 0.22 ? 6% perf-profile.children.cycles-pp.free_unref_page_list
0.71 ? 3% -0.0 0.70 ? 5% -0.0 0.70 ? 4% perf-profile.children.cycles-pp.fault_in_readable
0.38 ? 2% -0.0 0.37 ? 7% -0.0 0.38 ? 4% perf-profile.children.cycles-pp.__mod_lruvec_state
0.33 ? 3% -0.0 0.32 ? 7% +0.0 0.33 ? 5% perf-profile.children.cycles-pp.__mod_node_page_state
0.15 ? 6% -0.0 0.14 ? 6% -0.0 0.14 ? 9% perf-profile.children.cycles-pp.alloc_pages
0.20 ? 4% -0.0 0.19 ? 8% -0.0 0.19 ? 5% perf-profile.children.cycles-pp.node_dirty_ok
0.08 ? 10% -0.0 0.08 ? 10% -0.0 0.08 ? 6% perf-profile.children.cycles-pp.mem_cgroup_update_lru_size
0.09 ? 8% -0.0 0.08 ? 7% +0.0 0.09 perf-profile.children.cycles-pp.xa_get_order
0.05 ? 50% -0.0 0.04 ? 72% +0.0 0.05 ? 47% perf-profile.children.cycles-pp.memcg_slab_free_hook
0.26 ? 2% -0.0 0.25 ? 4% -0.0 0.25 ? 5% perf-profile.children.cycles-pp.try_charge_memcg
0.10 ? 5% -0.0 0.09 ? 10% -0.0 0.09 ? 4% perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
0.07 ? 16% -0.0 0.06 ? 7% +0.0 0.07 ? 8% perf-profile.children.cycles-pp.mem_cgroup_track_foreign_dirty_slowpath
0.94 ? 3% -0.0 0.94 ? 2% -0.0 0.91 ? 3% perf-profile.children.cycles-pp.__might_resched
0.30 ? 3% -0.0 0.30 ? 2% -0.0 0.28 ? 3% perf-profile.children.cycles-pp.rmqueue
0.15 ? 4% -0.0 0.15 ? 5% -0.0 0.14 ? 8% perf-profile.children.cycles-pp.__xa_set_mark
0.08 ? 8% -0.0 0.08 ? 12% -0.0 0.08 ? 8% perf-profile.children.cycles-pp.iomap_adjust_read_range
0.27 ? 4% -0.0 0.27 ? 5% -0.0 0.27 ? 2% perf-profile.children.cycles-pp.__list_del_entry_valid
0.15 ? 3% -0.0 0.14 ? 6% -0.0 0.15 ? 6% perf-profile.children.cycles-pp.page_counter_try_charge
0.17 ? 9% -0.0 0.16 ? 4% +0.0 0.17 ? 9% perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
0.09 ? 15% -0.0 0.09 ? 12% -0.0 0.09 ? 12% perf-profile.children.cycles-pp.rcu_do_batch
0.06 ? 13% -0.0 0.05 ? 8% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.xfs_mod_freecounter
0.08 ? 4% -0.0 0.08 ? 8% -0.0 0.07 ? 12% perf-profile.children.cycles-pp.xas_find
0.08 ? 12% -0.0 0.08 ? 14% -0.0 0.07 ? 14% perf-profile.children.cycles-pp.xfs_buf_lock
0.22 ? 3% -0.0 0.22 ? 4% -0.0 0.22 ? 5% perf-profile.children.cycles-pp.filemap_unaccount_folio
0.12 ? 6% -0.0 0.11 ? 6% -0.0 0.11 ? 9% perf-profile.children.cycles-pp.xfs_ialloc_read_agi
0.08 ? 12% -0.0 0.08 ? 9% -0.0 0.08 ? 12% perf-profile.children.cycles-pp.xfs_log_reserve
0.09 ? 12% -0.0 0.08 ? 8% -0.0 0.08 ? 11% perf-profile.children.cycles-pp.xfs_trans_reserve
0.07 ? 7% -0.0 0.06 ? 7% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.folio_mapping
0.15 ? 4% -0.0 0.15 ? 4% -0.0 0.14 ? 8% perf-profile.children.cycles-pp.xfs_iread_extents
0.08 ? 14% -0.0 0.08 ? 16% -0.0 0.07 ? 15% perf-profile.children.cycles-pp.down
0.07 ? 10% -0.0 0.07 ? 7% -0.0 0.07 ? 7% perf-profile.children.cycles-pp.xfs_iunlink_update_inode
0.06 ? 14% -0.0 0.06 ? 9% -0.0 0.06 ? 9% perf-profile.children.cycles-pp.xfs_bmap_add_extent_hole_delay
0.06 ? 9% -0.0 0.06 ? 6% -0.0 0.06 ? 6% perf-profile.children.cycles-pp.iomap_iter_done
0.06 ? 13% -0.0 0.06 ? 11% +0.0 0.06 ? 11% perf-profile.children.cycles-pp.times
0.12 -0.0 0.12 ? 11% -0.0 0.11 ? 7% perf-profile.children.cycles-pp.xlog_cil_alloc_shadow_bufs
0.08 ? 10% -0.0 0.07 ? 12% -0.0 0.07 ? 14% perf-profile.children.cycles-pp.__down_common
0.11 ? 8% -0.0 0.11 ? 9% -0.0 0.11 ? 3% perf-profile.children.cycles-pp.xas_clear_mark
0.12 ? 4% -0.0 0.11 ? 4% -0.0 0.11 ? 7% perf-profile.children.cycles-pp.xfs_bmapi_reserve_delalloc
0.11 ? 9% -0.0 0.11 ? 8% -0.0 0.10 ? 9% perf-profile.children.cycles-pp.xfs_trans_alloc
0.10 ? 3% -0.0 0.10 ? 10% -0.0 0.09 perf-profile.children.cycles-pp.xfs_lookup
0.10 ? 7% -0.0 0.10 ? 11% -0.0 0.09 perf-profile.children.cycles-pp.xfs_dir_lookup
0.04 ? 45% +0.0 0.04 ? 45% -0.0 0.02 ? 99% perf-profile.children.cycles-pp.vfprintf
0.02 ? 99% +0.0 0.02 ? 99% -0.0 0.02 ?141% perf-profile.children.cycles-pp.xas_set_mark
0.10 ? 7% +0.0 0.10 ? 10% -0.0 0.10 ? 4% perf-profile.children.cycles-pp.xfs_vn_lookup
0.08 ? 6% +0.0 0.08 ? 12% -0.0 0.07 perf-profile.children.cycles-pp.xfs_dir2_node_lookup
0.18 ? 6% +0.0 0.18 ? 6% -0.0 0.17 ? 5% perf-profile.children.cycles-pp.rcu_all_qs
0.07 ? 11% +0.0 0.07 ? 11% -0.0 0.06 ? 7% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
0.06 ? 11% +0.0 0.06 ? 11% -0.0 0.06 ? 13% perf-profile.children.cycles-pp.node_page_state
0.06 ? 6% +0.0 0.06 ? 17% +0.0 0.06 ? 6% perf-profile.children.cycles-pp.rebalance_domains
0.05 ? 46% +0.0 0.05 ? 46% +0.0 0.05 ? 53% perf-profile.children.cycles-pp.xfs_next_bit
0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% perf-profile.children.cycles-pp.free_pcp_prepare
0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% perf-profile.children.cycles-pp.serial8250_console_write
0.02 ? 99% +0.0 0.02 ? 99% +0.0 0.02 ? 99% perf-profile.children.cycles-pp.balance_dirty_pages
0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% perf-profile.children.cycles-pp.uart_console_write
0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% perf-profile.children.cycles-pp.wait_for_xmitr
0.02 ?141% +0.0 0.02 ?141% +0.0 0.02 ?141% perf-profile.children.cycles-pp.xfs_dir3_data_read
0.01 ?223% +0.0 0.01 ?223% +0.0 0.01 ?223% perf-profile.children.cycles-pp.xfs_verify_agino
0.08 ? 13% +0.0 0.08 ? 11% +0.0 0.09 ? 12% perf-profile.children.cycles-pp.kmem_cache_free
0.02 ? 99% +0.0 0.02 ? 99% +0.0 0.03 ?100% perf-profile.children.cycles-pp.xfs_iextents_copy
0.09 ? 6% +0.0 0.09 ? 14% +0.0 0.09 ? 10% perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
0.08 ? 6% +0.0 0.08 ? 14% +0.0 0.08 ? 12% perf-profile.children.cycles-pp.xfs_btree_read_buf_block
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ?141% perf-profile.children.cycles-pp.asm_sysvec_irq_work
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ?141% perf-profile.children.cycles-pp.sysvec_irq_work
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ?141% perf-profile.children.cycles-pp.__sysvec_irq_work
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ?141% perf-profile.children.cycles-pp.irq_work_run
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ?141% perf-profile.children.cycles-pp.irq_work_single
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ?141% perf-profile.children.cycles-pp._printk
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ?141% perf-profile.children.cycles-pp.vprintk_emit
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ?141% perf-profile.children.cycles-pp.console_unlock
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ?141% perf-profile.children.cycles-pp.call_console_drivers
0.00 +0.0 0.00 +0.0 0.01 ?223% perf-profile.children.cycles-pp.update_curr
0.00 +0.0 0.00 +0.0 0.01 ?223% perf-profile.children.cycles-pp.policy_node
0.01 ?223% +0.0 0.01 ?223% +0.0 0.02 ? 99% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
0.16 ? 4% +0.0 0.16 ? 10% -0.0 0.15 ? 9% perf-profile.children.cycles-pp.xfs_read_agi
0.10 ? 4% +0.0 0.10 ? 6% -0.0 0.09 ? 9% perf-profile.children.cycles-pp.free_unref_page_commit
0.07 ? 6% +0.0 0.08 ? 6% -0.0 0.07 ? 10% perf-profile.children.cycles-pp.folio_memcg_unlock
0.29 ? 7% +0.0 0.30 ? 3% -0.0 0.29 ? 3% perf-profile.children.cycles-pp.generic_write_checks
0.10 ? 10% +0.0 0.10 ? 7% +0.0 0.10 ? 14% perf-profile.children.cycles-pp.xfs_inode_item_format
0.09 ? 7% +0.0 0.09 ? 14% -0.0 0.09 ? 4% perf-profile.children.cycles-pp.rwsem_wake
0.08 ? 4% +0.0 0.08 ? 7% -0.0 0.08 ? 6% perf-profile.children.cycles-pp.uncharge_folio
0.06 ? 11% +0.0 0.07 ? 11% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.schedule_timeout
0.11 ? 4% +0.0 0.12 ? 8% -0.0 0.11 ? 5% perf-profile.children.cycles-pp.iomap_page_create
0.07 ? 9% +0.0 0.07 ? 15% -0.0 0.07 ? 10% perf-profile.children.cycles-pp.wake_up_q
0.12 ? 12% +0.0 0.12 ? 11% +0.0 0.12 ? 12% perf-profile.children.cycles-pp.rcu_core
0.06 ? 9% +0.0 0.06 ? 8% +0.0 0.06 ? 9% perf-profile.children.cycles-pp.iov_iter_init
0.10 ? 5% +0.0 0.10 ? 4% +0.0 0.10 ? 7% perf-profile.children.cycles-pp.xas_create
0.06 ? 9% +0.0 0.06 ? 13% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.xfs_imap_to_bp
0.06 ? 6% +0.0 0.06 ? 9% +0.0 0.06 ? 6% perf-profile.children.cycles-pp.xfs_buf_item_init
0.03 ? 70% +0.0 0.04 ? 71% +0.0 0.04 ? 44% perf-profile.children.cycles-pp.rw_verify_area
0.28 ? 5% +0.0 0.29 ? 8% -0.0 0.27 ? 3% perf-profile.children.cycles-pp.xfs_buf_read_map
0.06 ? 13% +0.0 0.06 ? 11% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.xfs_isilocked
0.08 +0.0 0.08 ? 8% -0.0 0.08 ? 8% perf-profile.children.cycles-pp.xfs_iunlink_remove
0.14 ? 9% +0.0 0.14 ? 8% +0.0 0.14 ? 6% perf-profile.children.cycles-pp.xfs_btree_lookup
0.05 +0.0 0.05 ? 8% +0.0 0.05 ? 8% perf-profile.children.cycles-pp.xas_alloc
0.10 ? 6% +0.0 0.10 ? 9% +0.0 0.11 ? 6% perf-profile.children.cycles-pp.__mark_inode_dirty
0.05 ? 8% +0.0 0.06 ? 13% -0.0 0.04 ? 45% perf-profile.children.cycles-pp.xfs_init_new_inode
0.06 ? 6% +0.0 0.06 ? 7% -0.0 0.06 ? 9% perf-profile.children.cycles-pp.kmem_cache_alloc_lru
0.11 ? 6% +0.0 0.11 ? 12% -0.0 0.10 ? 6% perf-profile.children.cycles-pp.xfs_iunlink
0.12 ? 8% +0.0 0.12 ? 9% -0.0 0.11 ? 6% perf-profile.children.cycles-pp.try_to_wake_up
0.08 ? 8% +0.0 0.08 ? 8% -0.0 0.08 ? 8% perf-profile.children.cycles-pp.xas_find_conflict
0.10 ? 6% +0.0 0.10 ? 13% +0.0 0.10 ? 10% perf-profile.children.cycles-pp.update_sd_lb_stats
0.07 ? 10% +0.0 0.08 ? 9% +0.0 0.07 ? 17% perf-profile.children.cycles-pp.__mod_zone_page_state
0.09 ? 8% +0.0 0.09 ? 15% +0.0 0.09 ? 12% perf-profile.children.cycles-pp.update_sg_lb_stats
0.06 +0.0 0.06 ? 7% +0.0 0.06 ? 6% perf-profile.children.cycles-pp.kmem_cache_alloc
0.15 ? 11% +0.0 0.15 ? 4% +0.0 0.15 ? 7% perf-profile.children.cycles-pp.generic_write_check_limits
0.26 ? 4% +0.0 0.27 ? 9% -0.0 0.26 ? 4% perf-profile.children.cycles-pp.xfs_buf_get_map
0.18 ? 2% +0.0 0.18 ? 9% -0.0 0.17 ? 3% perf-profile.children.cycles-pp.xfs_da_read_buf
0.14 ? 7% +0.0 0.14 ? 7% -0.0 0.14 ? 3% perf-profile.children.cycles-pp.xfs_dir_createname
0.07 ? 9% +0.0 0.08 ? 9% -0.0 0.07 perf-profile.children.cycles-pp.xfs_dir2_leafn_remove
0.07 ? 7% +0.0 0.07 ? 9% -0.0 0.06 ? 7% perf-profile.children.cycles-pp.up
0.13 ? 10% +0.0 0.14 ? 4% +0.0 0.13 ? 5% perf-profile.children.cycles-pp.file_modified
0.10 ? 10% +0.0 0.10 ? 4% +0.0 0.10 ? 6% perf-profile.children.cycles-pp.file_remove_privs
0.42 ? 2% +0.0 0.42 ? 7% -0.0 0.40 ? 2% perf-profile.children.cycles-pp.xfs_trans_read_buf_map
0.24 ? 5% +0.0 0.25 ? 9% -0.0 0.24 ? 5% perf-profile.children.cycles-pp.xfs_buf_find
0.11 ? 3% +0.0 0.12 ? 7% -0.0 0.11 ? 7% perf-profile.children.cycles-pp._xfs_trans_bjoin
0.08 ? 9% +0.0 0.08 ? 11% +0.0 0.08 ? 7% perf-profile.children.cycles-pp.xfs_dir2_node_addname_int
0.06 ? 9% +0.0 0.06 ? 11% +0.0 0.06 ? 13% perf-profile.children.cycles-pp.xlog_prepare_iovec
0.10 ? 6% +0.0 0.10 ? 16% +0.0 0.10 ? 11% perf-profile.children.cycles-pp.find_busiest_group
0.24 ? 5% +0.0 0.24 ? 5% -0.0 0.24 ? 4% perf-profile.children.cycles-pp.xas_start
0.13 ? 8% +0.0 0.14 ? 7% -0.0 0.13 ? 5% perf-profile.children.cycles-pp.xfs_dir2_node_addname
0.12 ? 10% +0.0 0.12 ? 7% +0.0 0.12 ? 4% perf-profile.children.cycles-pp.pick_next_task_fair
0.32 ? 2% +0.0 0.33 ? 5% +0.0 0.32 ? 3% perf-profile.children.cycles-pp.folio_unlock
0.02 ? 99% +0.0 0.03 ? 70% +0.0 0.03 ?100% perf-profile.children.cycles-pp.xfs_inode_item_format_data_fork
0.07 +0.0 0.08 ? 8% -0.0 0.07 ? 7% perf-profile.children.cycles-pp.xfs_buf_unlock
0.08 ? 6% +0.0 0.08 ? 5% -0.0 0.07 ? 6% perf-profile.children.cycles-pp.xfs_buf_item_release
0.01 ?223% +0.0 0.02 ?141% -0.0 0.00 perf-profile.children.cycles-pp.xfs_verify_dir_ino
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.children.cycles-pp.ttwu_do_activate
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.children.cycles-pp.enqueue_task_fair
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.children.cycles-pp.sched_ttwu_pending
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.children.cycles-pp.xfs_dabuf_map
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.children.cycles-pp.calc_global_load_tick
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.children.cycles-pp.xfs_iget
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.children.cycles-pp.dequeue_entity
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.children.cycles-pp.xfs_defer_trans_roll
0.01 ?223% +0.0 0.02 ?141% +0.0 0.02 ?141% perf-profile.children.cycles-pp.irq_work_run_list
0.02 ?141% +0.0 0.02 ? 99% -0.0 0.01 ?223% perf-profile.children.cycles-pp.tick_nohz_irq_exit
0.02 ?141% +0.0 0.02 ? 99% +0.0 0.02 ? 99% perf-profile.children.cycles-pp.generic_file_write_iter
0.10 ? 12% +0.0 0.11 ? 11% +0.0 0.10 ? 5% perf-profile.children.cycles-pp.newidle_balance
0.10 ? 16% +0.0 0.11 ? 8% +0.0 0.10 ? 10% perf-profile.children.cycles-pp.cgroup_rstat_updated
0.12 ? 5% +0.0 0.13 ? 13% +0.0 0.12 ? 7% perf-profile.children.cycles-pp.load_balance
0.27 ? 8% +0.0 0.28 ? 11% +0.0 0.28 ? 3% perf-profile.children.cycles-pp.__irq_exit_rcu
0.09 ? 4% +0.0 0.10 ? 10% -0.0 0.08 ? 8% perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
0.07 ? 8% +0.0 0.08 ? 7% -0.0 0.06 ? 7% perf-profile.children.cycles-pp.xfs_inactive_truncate
0.01 ?223% +0.0 0.02 ?142% -0.0 0.00 perf-profile.children.cycles-pp.tick_sched_do_timer
0.04 ? 71% +0.0 0.04 ? 45% -0.0 0.03 ? 70% perf-profile.children.cycles-pp.xfs_perag_get
0.05 ? 7% +0.0 0.06 ? 14% +0.0 0.06 ? 6% perf-profile.children.cycles-pp.__rhashtable_lookup
0.19 ? 8% +0.0 0.20 ? 8% -0.0 0.18 ? 7% perf-profile.children.cycles-pp.schedule
0.24 ? 7% +0.0 0.25 ? 9% +0.0 0.24 ? 6% perf-profile.children.cycles-pp.__schedule
0.10 ? 13% +0.0 0.11 ? 29% +0.0 0.10 ? 11% perf-profile.children.cycles-pp.update_load_avg
0.06 ? 11% +0.0 0.07 ? 34% +0.0 0.06 ? 9% perf-profile.children.cycles-pp.update_cfs_group
0.02 ?141% +0.0 0.03 ?102% +0.0 0.03 ? 70% perf-profile.children.cycles-pp.flush_smp_call_function_from_idle
0.04 ? 71% +0.0 0.05 ? 7% +0.0 0.04 ? 44% perf-profile.children.cycles-pp.workingset_update_node
0.22 ? 10% +0.0 0.24 ? 11% +0.0 0.23 ? 11% perf-profile.children.cycles-pp.ktime_get
0.09 ? 4% +0.0 0.11 ? 11% +0.0 0.09 ? 7% perf-profile.children.cycles-pp.xfs_dir3_data_check
0.20 ? 8% +0.0 0.22 ? 13% -0.0 0.20 ? 12% perf-profile.children.cycles-pp.clockevents_program_event
0.24 ? 6% +0.0 0.26 ? 11% +0.0 0.25 ? 2% perf-profile.children.cycles-pp.__softirqentry_text_start
0.22 +0.0 0.24 ? 11% -0.0 0.21 ? 5% perf-profile.children.cycles-pp.xfs_dir_removename
0.30 ? 5% +0.0 0.32 ? 4% +0.0 0.30 ? 5% perf-profile.children.cycles-pp.memcpy_erms
0.09 +0.0 0.11 ? 11% +0.0 0.09 ? 7% perf-profile.children.cycles-pp.__xfs_dir3_data_check
0.00 +0.0 0.02 ?141% +0.0 0.01 ?223% perf-profile.children.cycles-pp.native_sched_clock
0.00 +0.0 0.02 ?141% +0.0 0.01 ?223% perf-profile.children.cycles-pp.slab_pre_alloc_hook
0.02 ?141% +0.0 0.03 ? 70% +0.0 0.03 ? 70% perf-profile.children.cycles-pp.native_irq_return_iret
0.00 +0.0 0.02 ?141% +0.0 0.02 ?141% perf-profile.children.cycles-pp.dequeue_task_fair
0.02 ?141% +0.0 0.03 ? 70% +0.0 0.04 ? 71% perf-profile.children.cycles-pp.memcg_check_events
0.22 +0.0 0.24 ? 10% -0.0 0.21 ? 3% perf-profile.children.cycles-pp.xfs_da3_node_lookup_int
0.21 ? 2% +0.0 0.23 ? 10% -0.0 0.20 ? 4% perf-profile.children.cycles-pp.xfs_dir2_node_removename
0.02 ?141% +0.0 0.04 ? 71% +0.0 0.04 ? 71% perf-profile.children.cycles-pp.sched_clock_cpu
0.00 +0.0 0.02 ?144% +0.0 0.04 ? 44% perf-profile.children.cycles-pp.update_rq_clock
0.03 ? 70% +0.0 0.05 ? 8% -0.0 0.02 ? 99% perf-profile.children.cycles-pp.lapic_next_deadline
0.11 ? 6% +0.0 0.13 ? 11% +0.0 0.12 ? 7% perf-profile.children.cycles-pp.menu_select
0.03 ?101% +0.0 0.05 ? 50% +0.0 0.05 ? 54% perf-profile.children.cycles-pp.xfs_inobt_irec_to_allocmask
0.10 ? 3% +0.0 0.12 ? 10% +0.0 0.12 ? 9% perf-profile.children.cycles-pp.xfs_difree_finobt
0.41 ? 4% +0.0 0.44 ? 3% -0.0 0.41 ? 3% perf-profile.children.cycles-pp.xfs_buf_item_format_segment
0.42 ? 3% +0.0 0.44 ? 3% -0.0 0.41 ? 4% perf-profile.children.cycles-pp.xfs_buf_item_format
0.14 ? 11% +0.0 0.16 ? 40% +0.0 0.15 ? 10% perf-profile.children.cycles-pp.task_tick_fair
0.68 ? 2% +0.0 0.71 ? 9% -0.0 0.65 ? 2% perf-profile.children.cycles-pp.xfs_remove
0.70 +0.0 0.72 ? 9% -0.0 0.66 ? 2% perf-profile.children.cycles-pp.vfs_unlink
0.68 ? 2% +0.0 0.71 ? 9% -0.0 0.65 ? 2% perf-profile.children.cycles-pp.xfs_vn_unlink
0.57 ? 3% +0.0 0.60 ? 4% +0.0 0.57 ? 4% perf-profile.children.cycles-pp.xlog_cil_insert_format_items
0.02 ?141% +0.0 0.06 ? 9% -0.0 0.01 ?223% perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
0.24 ? 8% +0.0 0.28 ? 25% +0.0 0.25 ? 7% perf-profile.children.cycles-pp.scheduler_tick
0.01 ?223% +0.0 0.05 ? 7% +0.0 0.04 ? 44% perf-profile.children.cycles-pp.update_blocked_averages
0.00 +0.0 0.04 ? 45% +0.0 0.04 ? 44% perf-profile.children.cycles-pp.run_rebalance_domains
0.32 ? 7% +0.0 0.37 ? 22% +0.0 0.34 ? 8% perf-profile.children.cycles-pp.update_process_times
0.33 ? 7% +0.0 0.38 ? 22% +0.0 0.35 ? 8% perf-profile.children.cycles-pp.tick_sched_handle
0.38 ? 6% +0.1 0.44 ? 21% +0.0 0.40 ? 8% perf-profile.children.cycles-pp.tick_sched_timer
0.46 ? 5% +0.1 0.53 ? 18% +0.0 0.48 ? 7% perf-profile.children.cycles-pp.__hrtimer_run_queues
1.90 ? 3% +0.1 1.98 ? 4% +0.0 1.92 ? 2% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.74 ? 6% +0.1 0.83 ? 15% +0.0 0.76 ? 7% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.73 ? 6% +0.1 0.82 ? 15% +0.0 0.75 ? 6% perf-profile.children.cycles-pp.hrtimer_interrupt
0.28 ? 5% +0.1 0.37 ? 4% +0.1 0.36 ? 4% perf-profile.children.cycles-pp.xfs_difree_inobt
0.19 ? 3% +0.1 0.29 ? 7% +0.1 0.29 ? 3% perf-profile.children.cycles-pp.xfs_btree_get_rec
0.44 ? 4% +0.1 0.55 ? 4% +0.1 0.55 ? 4% perf-profile.children.cycles-pp.xfs_difree
0.20 ? 9% +0.1 0.30 ? 7% +0.1 0.29 ? 3% perf-profile.children.cycles-pp.xfs_btree_increment
1.10 ? 6% +0.1 1.21 ? 12% +0.0 1.14 ? 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.53 ? 3% +0.1 0.64 ? 4% +0.1 0.64 ? 4% perf-profile.children.cycles-pp.xfs_ifree
0.88 ? 3% +0.1 1.00 ? 3% +0.1 0.99 ? 4% perf-profile.children.cycles-pp.xfs_inactive_ifree
0.38 ? 4% +0.1 0.50 ? 6% +0.1 0.50 ? 6% perf-profile.children.cycles-pp.xfs_inobt_get_rec
0.95 ? 3% +0.1 1.08 ? 3% +0.1 1.05 ? 4% perf-profile.children.cycles-pp.xfs_inactive
0.96 ? 3% +0.1 1.09 ? 3% +0.1 1.06 ? 4% perf-profile.children.cycles-pp.xfs_inodegc_worker
0.42 ? 3% +0.1 0.55 ? 7% +0.1 0.53 ? 3% perf-profile.children.cycles-pp.xfs_dialloc_ag
0.98 ? 3% +0.1 1.11 ? 3% +0.1 1.09 ? 4% perf-profile.children.cycles-pp.process_one_work
0.48 ? 3% +0.1 0.61 ? 8% +0.1 0.58 ? 2% perf-profile.children.cycles-pp.xfs_dialloc
0.99 ? 3% +0.1 1.12 ? 3% +0.1 1.09 ? 3% perf-profile.children.cycles-pp.worker_thread
0.99 ? 3% +0.1 1.13 ? 3% +0.1 1.10 ? 3% perf-profile.children.cycles-pp.kthread
0.99 ? 3% +0.1 1.14 ? 3% +0.1 1.10 ? 3% perf-profile.children.cycles-pp.ret_from_fork
1.11 ? 3% +0.2 1.26 ? 6% +0.1 1.19 perf-profile.children.cycles-pp.lookup_open
0.95 ? 4% +0.2 1.10 ? 6% +0.1 1.04 perf-profile.children.cycles-pp.xfs_create
0.96 ? 4% +0.2 1.12 ? 6% +0.1 1.05 perf-profile.children.cycles-pp.xfs_generic_create
0.13 ? 5% +0.2 0.33 ? 6% +0.2 0.32 ? 2% perf-profile.children.cycles-pp.__xfs_btree_check_sblock
1.30 ? 7% +0.2 1.51 ? 21% -0.0 1.28 ? 5% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.22 ? 6% +0.2 0.43 ? 6% +0.2 0.42 perf-profile.children.cycles-pp.xfs_btree_check_sblock
0.68 ? 4% +0.2 0.91 ? 5% +0.2 0.90 ? 3% perf-profile.children.cycles-pp.xfs_check_agi_freecount
72.85 ? 3% +0.3 73.10 ? 2% +0.6 73.43 ? 2% perf-profile.children.cycles-pp.do_syscall_64
73.11 ? 3% +0.3 73.36 ? 2% +0.6 73.69 ? 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
19.36 ? 4% +0.7 20.06 ? 2% +1.0 20.32 ? 3% perf-profile.children.cycles-pp.do_unlinkat
19.37 ? 4% +0.7 20.07 ? 2% +1.0 20.33 ? 3% perf-profile.children.cycles-pp.__x64_sys_unlink
19.41 ? 4% +0.7 20.11 ? 2% +1.0 20.36 ? 3% perf-profile.children.cycles-pp.unlink
20.09 ? 4% +0.9 20.95 ? 3% +1.0 21.12 ? 3% perf-profile.children.cycles-pp.__x64_sys_creat
20.10 ? 4% +0.9 20.96 ? 3% +1.0 21.13 ? 3% perf-profile.children.cycles-pp.do_sys_openat2
20.12 ? 4% +0.9 20.98 ? 3% +1.0 21.16 ? 3% perf-profile.children.cycles-pp.creat64
20.00 ? 4% +0.9 20.87 ? 3% +1.0 21.04 ? 3% perf-profile.children.cycles-pp.open_last_lookups
20.07 ? 4% +0.9 20.94 ? 3% +1.0 21.11 ? 3% perf-profile.children.cycles-pp.do_filp_open
20.07 ? 4% +0.9 20.93 ? 3% +1.0 21.11 ? 3% perf-profile.children.cycles-pp.path_openat
35.30 ? 4% +1.3 36.59 ? 3% +1.9 37.24 ? 3% perf-profile.children.cycles-pp.osq_lock
36.88 ? 4% +1.3 38.19 ? 2% +1.9 38.83 ? 3% perf-profile.children.cycles-pp.rwsem_optimistic_spin
37.40 ? 4% +1.4 38.77 ? 2% +2.0 39.35 ? 3% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
22.44 ? 12% -0.4 22.00 ? 9% -0.7 21.75 ? 9% perf-profile.self.cycles-pp.mwait_idle_with_hints
0.39 ? 9% -0.1 0.31 ? 4% -0.1 0.31 ? 6% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
2.89 ? 2% -0.1 2.82 ? 3% -0.0 2.85 ? 3% perf-profile.self.cycles-pp.__iomap_write_begin
1.15 ? 4% -0.1 1.10 ? 4% -0.0 1.10 ? 2% perf-profile.self.cycles-pp.memset_erms
0.72 ? 4% -0.0 0.68 ? 5% -0.1 0.65 ? 3% perf-profile.self.cycles-pp.vfs_write
1.43 ? 3% -0.0 1.39 ? 3% -0.1 1.37 ? 2% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.36 ? 7% -0.0 0.32 ? 9% -0.0 0.32 ? 8% perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
1.99 ? 5% -0.0 1.95 ? 4% -0.1 1.93 ? 2% perf-profile.self.cycles-pp.iomap_write_end
0.52 ? 5% -0.0 0.49 ? 2% -0.0 0.48 ? 3% perf-profile.self.cycles-pp.iomap_write_iter
0.66 ? 2% -0.0 0.62 ? 2% -0.0 0.63 ? 2% perf-profile.self.cycles-pp.iomap_file_buffered_write
0.92 ? 4% -0.0 0.89 ? 4% -0.0 0.91 ? 3% perf-profile.self.cycles-pp.__filemap_get_folio
0.43 ? 4% -0.0 0.40 ? 3% -0.0 0.41 ? 6% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
1.03 ? 4% -0.0 1.00 ? 4% -0.0 0.98 ? 4% perf-profile.self.cycles-pp.iomap_iter
0.56 ? 3% -0.0 0.54 ? 3% -0.0 0.55 ? 3% perf-profile.self.cycles-pp.up_write
0.27 ? 8% -0.0 0.24 ? 4% -0.0 0.25 ? 6% perf-profile.self.cycles-pp.__fget_light
0.21 ? 4% -0.0 0.19 ? 5% -0.0 0.19 ? 3% perf-profile.self.cycles-pp.file_update_time
0.67 ? 4% -0.0 0.64 ? 4% -0.0 0.63 ? 4% perf-profile.self.cycles-pp.write
0.47 ? 5% -0.0 0.44 ? 4% -0.0 0.46 ? 3% perf-profile.self.cycles-pp.xfs_file_buffered_write
0.32 ? 3% -0.0 0.30 ? 3% -0.0 0.30 ? 3% perf-profile.self.cycles-pp.__might_sleep
0.37 ? 5% -0.0 0.35 ? 5% -0.0 0.34 ? 3% perf-profile.self.cycles-pp.percpu_counter_add_batch
0.30 ? 4% -0.0 0.28 -0.0 0.27 ? 4% perf-profile.self.cycles-pp.xfs_break_leased_layouts
0.42 ? 5% -0.0 0.40 ? 4% -0.0 0.41 ? 6% perf-profile.self.cycles-pp.__entry_text_start
0.77 ? 5% -0.0 0.75 ? 2% -0.0 0.75 ? 4% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.32 ? 3% -0.0 0.30 ? 5% -0.0 0.30 ? 5% perf-profile.self.cycles-pp.xfs_iext_lookup_extent
0.27 ? 9% -0.0 0.25 ? 5% -0.0 0.27 ? 5% perf-profile.self.cycles-pp.xfs_errortag_test
0.28 ? 3% -0.0 0.26 ? 4% -0.0 0.28 ? 4% perf-profile.self.cycles-pp.new_sync_write
0.14 ? 6% -0.0 0.12 ? 4% -0.0 0.13 ? 8% perf-profile.self.cycles-pp.__alloc_pages
0.42 ? 3% -0.0 0.40 ? 5% -0.0 0.40 ? 2% perf-profile.self.cycles-pp.xfs_file_write_iter
0.30 ? 4% -0.0 0.29 ? 2% -0.0 0.28 ? 3% perf-profile.self.cycles-pp.xas_store
0.25 ? 6% -0.0 0.23 ? 3% -0.0 0.22 ? 4% perf-profile.self.cycles-pp.xfs_iunlock
0.21 ? 5% -0.0 0.19 ? 6% -0.0 0.19 ? 3% perf-profile.self.cycles-pp.xfs_ilock
0.02 ? 99% -0.0 0.01 ?223% -0.0 0.00 perf-profile.self.cycles-pp.vfprintf
0.38 ? 5% -0.0 0.36 ? 5% -0.0 0.36 ? 3% perf-profile.self.cycles-pp.filemap_dirty_folio
0.43 ? 6% -0.0 0.41 ? 4% -0.0 0.41 ? 3% perf-profile.self.cycles-pp.copy_page_from_iter_atomic
0.21 ? 5% -0.0 0.20 ? 8% -0.0 0.20 ? 8% perf-profile.self.cycles-pp.__cond_resched
0.04 ? 44% -0.0 0.03 ?100% -0.0 0.02 ? 99% perf-profile.self.cycles-pp.__pagevec_lru_add
0.21 ? 7% -0.0 0.19 ? 2% -0.0 0.19 ? 7% perf-profile.self.cycles-pp.__mod_lruvec_page_state
0.24 ? 3% -0.0 0.22 ? 5% -0.0 0.23 ? 4% perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
0.27 ? 4% -0.0 0.26 ? 4% -0.0 0.26 ? 5% perf-profile.self.cycles-pp.xfs_buffered_write_iomap_end
0.08 ? 13% -0.0 0.07 ? 13% -0.0 0.07 ? 11% perf-profile.self.cycles-pp.xfs_get_extsz_hint
0.78 ? 3% -0.0 0.77 ? 4% -0.0 0.76 ? 4% perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
0.06 ? 9% -0.0 0.04 ? 45% -0.0 0.04 ? 71% perf-profile.self.cycles-pp.zero_user_segments
0.30 ? 4% -0.0 0.29 ? 3% -0.0 0.29 ? 5% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.93 ? 3% -0.0 0.92 ? 3% -0.0 0.90 ? 3% perf-profile.self.cycles-pp.__might_resched
0.09 ? 7% -0.0 0.08 ? 5% -0.0 0.08 ? 10% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.10 ? 10% -0.0 0.08 ? 5% -0.0 0.08 ? 5% perf-profile.self.cycles-pp.__list_add_valid
0.36 ? 4% -0.0 0.35 ? 4% -0.0 0.35 ? 4% perf-profile.self.cycles-pp.iomap_write_begin
0.07 ? 6% -0.0 0.06 ? 11% -0.0 0.07 ? 7% perf-profile.self.cycles-pp.filemap_free_folio
0.01 ?223% -0.0 0.00 -0.0 0.00 perf-profile.self.cycles-pp.xfs_log_ticket_ungrant
0.01 ?223% -0.0 0.00 -0.0 0.01 ?223% perf-profile.self.cycles-pp.xlog_grant_add_space
0.12 ? 4% -0.0 0.10 ? 4% -0.0 0.10 ? 4% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.09 ? 8% -0.0 0.08 ? 6% -0.0 0.08 ? 8% perf-profile.self.cycles-pp.truncate_cleanup_folio
0.10 ? 4% -0.0 0.09 ? 7% -0.0 0.10 ? 10% perf-profile.self.cycles-pp.aa_file_perm
0.15 ? 13% -0.0 0.14 ? 11% -0.0 0.14 ? 4% perf-profile.self.cycles-pp.inode_to_bdi
0.12 ? 3% -0.0 0.12 ? 4% -0.0 0.12 ? 8% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.02 ? 99% -0.0 0.02 ?141% -0.0 0.02 ?141% perf-profile.self.cycles-pp.xfs_isilocked
0.02 ?141% -0.0 0.01 ?223% -0.0 0.00 perf-profile.self.cycles-pp.propagate_protected_usage
0.01 ?223% -0.0 0.00 -0.0 0.00 perf-profile.self.cycles-pp.free_pcp_prepare
0.01 ?223% -0.0 0.00 -0.0 0.00 perf-profile.self.cycles-pp.__x64_sys_write
0.01 ?223% -0.0 0.00 +0.0 0.01 ?223% perf-profile.self.cycles-pp.xfs_verify_agino
0.01 ?223% -0.0 0.00 +0.0 0.01 ?223% perf-profile.self.cycles-pp.xfs_btree_get_rec
0.17 ? 5% -0.0 0.16 ? 7% -0.0 0.16 ? 7% perf-profile.self.cycles-pp._raw_spin_lock
0.03 ? 70% -0.0 0.02 ? 99% -0.0 0.02 ? 99% perf-profile.self.cycles-pp.iomap_iter_done
0.69 ? 3% -0.0 0.68 ? 6% -0.0 0.68 ? 3% perf-profile.self.cycles-pp.fault_in_readable
0.32 ? 4% -0.0 0.31 ? 7% +0.0 0.32 ? 4% perf-profile.self.cycles-pp.__mod_node_page_state
0.14 ? 4% -0.0 0.14 ? 7% -0.0 0.13 ? 3% perf-profile.self.cycles-pp.fault_in_iov_iter_readable
0.11 ? 4% -0.0 0.10 ? 5% -0.0 0.10 ? 10% perf-profile.self.cycles-pp.do_syscall_64
0.07 ? 5% -0.0 0.06 ? 6% -0.0 0.06 ? 11% perf-profile.self.cycles-pp.PageHeadHuge
0.06 ? 14% -0.0 0.06 ? 11% -0.0 0.06 ? 7% perf-profile.self.cycles-pp.charge_memcg
0.13 ? 6% -0.0 0.12 ? 7% +0.0 0.13 ? 8% perf-profile.self.cycles-pp.folio_memcg_lock
0.44 ? 4% -0.0 0.44 ? 4% -0.0 0.42 ? 2% perf-profile.self.cycles-pp.xas_load
0.16 ? 8% -0.0 0.15 ? 4% -0.0 0.14 ? 9% perf-profile.self.cycles-pp.ksys_write
0.44 ? 5% -0.0 0.44 ? 4% -0.0 0.43 ? 6% perf-profile.self.cycles-pp.apparmor_file_permission
0.15 ? 6% -0.0 0.14 ? 5% -0.0 0.14 ? 5% perf-profile.self.cycles-pp.current_time
0.27 ? 3% -0.0 0.27 ? 4% -0.0 0.26 ? 4% perf-profile.self.cycles-pp.find_lock_entries
0.25 ? 4% -0.0 0.24 ? 6% -0.0 0.24 ? 4% perf-profile.self.cycles-pp.page_counter_uncharge
0.08 ? 8% -0.0 0.07 ? 10% -0.0 0.07 ? 8% perf-profile.self.cycles-pp.mem_cgroup_charge_statistics
0.12 ? 6% -0.0 0.12 ? 6% -0.0 0.11 ? 9% perf-profile.self.cycles-pp.__folio_cancel_dirty
0.08 ? 4% -0.0 0.07 ? 10% -0.0 0.07 ? 5% perf-profile.self.cycles-pp.__mod_lruvec_state
0.08 ? 10% -0.0 0.08 ? 6% -0.0 0.07 ? 10% perf-profile.self.cycles-pp.disk_wrt
0.14 ? 6% -0.0 0.14 ? 7% -0.0 0.14 ? 2% perf-profile.self.cycles-pp.node_dirty_ok
0.14 ? 3% -0.0 0.13 ? 6% -0.0 0.13 ? 9% perf-profile.self.cycles-pp.page_counter_try_charge
0.09 ? 5% -0.0 0.08 ? 8% +0.0 0.09 ? 7% perf-profile.self.cycles-pp.free_unref_page_list
0.09 ? 7% -0.0 0.09 ? 4% +0.0 0.10 ? 8% perf-profile.self.cycles-pp.get_page_from_freelist
0.12 ? 3% -0.0 0.12 ? 13% -0.0 0.12 ? 7% perf-profile.self.cycles-pp.folio_account_dirtied
0.10 ? 6% -0.0 0.09 ? 10% -0.0 0.10 ? 4% perf-profile.self.cycles-pp.__mark_inode_dirty
0.08 ? 12% -0.0 0.07 ? 11% +0.0 0.08 ? 6% perf-profile.self.cycles-pp.mem_cgroup_update_lru_size
0.31 ? 3% -0.0 0.31 ? 3% -0.0 0.30 ? 4% perf-profile.self.cycles-pp.release_pages
0.06 ? 11% -0.0 0.06 -0.0 0.06 ? 9% perf-profile.self.cycles-pp.folio_mapping
0.11 ? 9% -0.0 0.10 ? 7% -0.0 0.10 ? 4% perf-profile.self.cycles-pp.xas_clear_mark
0.11 ? 8% -0.0 0.10 ? 4% -0.0 0.10 ? 8% perf-profile.self.cycles-pp.try_charge_memcg
0.13 ? 5% -0.0 0.12 ? 6% -0.0 0.12 ? 7% perf-profile.self.cycles-pp.folio_account_cleaned
0.06 ? 19% -0.0 0.06 ? 9% -0.0 0.06 ? 14% perf-profile.self.cycles-pp.mem_cgroup_track_foreign_dirty_slowpath
0.06 -0.0 0.06 ? 13% -0.0 0.06 ? 9% perf-profile.self.cycles-pp.xas_find
0.07 ? 5% -0.0 0.07 ? 5% -0.0 0.07 ? 11% perf-profile.self.cycles-pp.security_file_permission
0.19 ? 2% -0.0 0.18 ? 6% -0.0 0.18 ? 5% perf-profile.self.cycles-pp.xfs_file_write_checks
0.16 ? 11% -0.0 0.16 ? 5% +0.0 0.16 ? 8% perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
0.61 ? 3% -0.0 0.60 -0.0 0.60 ? 2% perf-profile.self.cycles-pp.down_write
0.32 ? 3% -0.0 0.31 ? 5% -0.0 0.30 ? 7% perf-profile.self.cycles-pp.__pagevec_lru_add_fn
0.27 ? 3% -0.0 0.26 ? 4% -0.0 0.26 ? 3% perf-profile.self.cycles-pp.__list_del_entry_valid
0.07 ? 6% -0.0 0.07 ? 14% +0.0 0.07 ? 6% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
0.08 ? 8% -0.0 0.08 ? 5% +0.0 0.09 ? 8% perf-profile.self.cycles-pp.delete_from_page_cache_batch
0.04 ? 72% -0.0 0.04 ? 71% +0.0 0.05 ? 50% perf-profile.self.cycles-pp.xfs_next_bit
0.12 ? 6% -0.0 0.12 ? 4% -0.0 0.10 ? 7% perf-profile.self.cycles-pp.rmqueue
0.08 ? 10% -0.0 0.08 ? 8% -0.0 0.08 ? 6% perf-profile.self.cycles-pp.iomap_adjust_read_range
0.08 -0.0 0.08 ? 11% +0.0 0.08 perf-profile.self.cycles-pp.__fdget_pos
0.14 ? 6% -0.0 0.14 ? 7% -0.0 0.14 ? 5% perf-profile.self.cycles-pp.xfs_break_layouts
0.08 ? 14% -0.0 0.07 ? 6% +0.0 0.08 ? 8% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.23 ? 6% -0.0 0.23 ? 2% -0.0 0.22 ? 4% perf-profile.self.cycles-pp.__filemap_add_folio
0.04 ? 71% +0.0 0.04 ? 71% -0.0 0.02 ? 99% perf-profile.self.cycles-pp.xfs_btree_check_sblock
0.15 ? 8% +0.0 0.15 ? 8% -0.0 0.14 ? 5% perf-profile.self.cycles-pp.generic_write_checks
0.08 ? 6% +0.0 0.08 ? 9% -0.0 0.07 perf-profile.self.cycles-pp.uncharge_folio
0.09 ? 5% +0.0 0.09 ? 7% -0.0 0.09 ? 9% perf-profile.self.cycles-pp.alloc_pages
0.07 ? 5% +0.0 0.07 ? 9% -0.0 0.07 ? 10% perf-profile.self.cycles-pp.folio_memcg_unlock
0.07 ? 10% +0.0 0.07 ? 10% -0.0 0.07 ? 7% perf-profile.self.cycles-pp.xfs_btree_increment
0.14 ? 11% +0.0 0.14 ? 3% +0.0 0.14 ? 7% perf-profile.self.cycles-pp.generic_write_check_limits
0.02 ?141% +0.0 0.02 ?141% +0.0 0.02 ?141% perf-profile.self.cycles-pp.xas_set_mark
0.07 ? 11% +0.0 0.07 ? 11% +0.0 0.07 ? 19% perf-profile.self.cycles-pp.__mod_zone_page_state
0.25 ? 4% +0.0 0.25 ? 4% +0.0 0.26 ? 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.00 +0.0 0.00 +0.0 0.01 ?223% perf-profile.self.cycles-pp.filemap_unaccount_folio
0.00 +0.0 0.00 +0.0 0.01 ?223% perf-profile.self.cycles-pp.xas_create
0.10 ? 3% +0.0 0.10 ? 9% -0.0 0.10 ? 6% perf-profile.self.cycles-pp.iomap_page_create
0.05 ? 7% +0.0 0.05 ? 8% -0.0 0.04 ? 45% perf-profile.self.cycles-pp._xfs_trans_bjoin
0.06 ? 6% +0.0 0.06 ? 9% -0.0 0.06 ? 8% perf-profile.self.cycles-pp.xfs_buf_item_init
0.05 ? 46% +0.0 0.05 ? 7% +0.0 0.05 ? 8% perf-profile.self.cycles-pp.copyin
0.13 ? 6% +0.0 0.13 ? 5% -0.0 0.13 ? 5% perf-profile.self.cycles-pp.rcu_all_qs
0.08 ? 5% +0.0 0.08 ? 11% +0.0 0.08 ? 11% perf-profile.self.cycles-pp.folio_add_lru
0.14 ? 6% +0.0 0.15 ? 6% -0.0 0.13 ? 9% perf-profile.self.cycles-pp.xfs_iread_extents
0.06 ? 7% +0.0 0.07 ? 11% -0.0 0.06 ? 11% perf-profile.self.cycles-pp.xlog_cil_alloc_shadow_bufs
0.21 ? 7% +0.0 0.22 ? 5% -0.0 0.21 ? 3% perf-profile.self.cycles-pp.xas_start
0.06 ? 7% +0.0 0.07 ? 16% +0.0 0.06 ? 14% perf-profile.self.cycles-pp.update_sg_lb_stats
0.05 ? 45% +0.0 0.05 -0.0 0.04 ? 71% perf-profile.self.cycles-pp.node_page_state
0.09 ? 15% +0.0 0.10 ? 11% +0.0 0.10 ? 13% perf-profile.self.cycles-pp.cgroup_rstat_updated
0.04 ? 71% +0.0 0.04 ? 44% -0.0 0.03 ? 70% perf-profile.self.cycles-pp.workingset_update_node
0.05 ? 46% +0.0 0.05 ? 8% +0.0 0.05 ? 7% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
0.07 ? 5% +0.0 0.07 ? 6% -0.0 0.07 ? 11% perf-profile.self.cycles-pp.free_unref_page_commit
0.06 ? 9% +0.0 0.06 ? 7% +0.0 0.06 ? 9% perf-profile.self.cycles-pp.xas_find_conflict
0.10 ? 4% +0.0 0.11 ? 8% +0.0 0.11 ? 6% perf-profile.self.cycles-pp.xfs_inobt_get_rec
0.05 +0.0 0.06 ? 13% +0.0 0.06 ? 13% perf-profile.self.cycles-pp.xlog_cil_insert_format_items
0.31 ? 2% +0.0 0.32 ? 5% +0.0 0.32 ? 3% perf-profile.self.cycles-pp.folio_unlock
0.05 ? 7% +0.0 0.06 ? 9% +0.0 0.06 ? 13% perf-profile.self.cycles-pp.xlog_prepare_iovec
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.self.cycles-pp.calc_global_load_tick
0.00 +0.0 0.01 ?223% +0.0 0.00 perf-profile.self.cycles-pp.tick_sched_do_timer
0.02 ?141% +0.0 0.02 ? 99% +0.0 0.02 ?141% perf-profile.self.cycles-pp.iov_iter_init
0.02 ?141% +0.0 0.02 ? 99% +0.0 0.02 ?141% perf-profile.self.cycles-pp.xfs_buf_find
0.09 ? 15% +0.0 0.10 ? 8% +0.0 0.10 ? 5% perf-profile.self.cycles-pp.file_remove_privs
0.05 +0.0 0.06 ? 13% +0.0 0.06 ? 8% perf-profile.self.cycles-pp.__rhashtable_lookup
0.07 ? 10% +0.0 0.08 ? 20% +0.0 0.09 ? 10% perf-profile.self.cycles-pp.cpuidle_enter_state
0.02 ? 99% +0.0 0.04 ? 71% +0.0 0.02 ? 99% perf-profile.self.cycles-pp.rw_verify_area
0.05 ? 7% +0.0 0.06 ? 14% -0.0 0.04 ? 71% perf-profile.self.cycles-pp.__xfs_dir3_data_check
0.20 ? 10% +0.0 0.22 ? 11% +0.0 0.20 ? 12% perf-profile.self.cycles-pp.ktime_get
0.06 ? 11% +0.0 0.07 ? 34% +0.0 0.06 ? 9% perf-profile.self.cycles-pp.update_cfs_group
0.04 ? 44% +0.0 0.05 ? 8% -0.0 0.03 ?100% perf-profile.self.cycles-pp.xfs_buf_item_format_segment
0.07 ? 10% +0.0 0.09 ? 10% +0.0 0.08 ? 8% perf-profile.self.cycles-pp.menu_select
0.30 ? 5% +0.0 0.31 ? 3% -0.0 0.29 ? 5% perf-profile.self.cycles-pp.memcpy_erms
0.06 ? 45% +0.0 0.07 ? 37% +0.0 0.07 ? 7% perf-profile.self.cycles-pp.update_load_avg
0.00 +0.0 0.02 ?141% +0.0 0.00 perf-profile.self.cycles-pp.xfs_verify_dir_ino
0.00 +0.0 0.02 ?141% +0.0 0.00 perf-profile.self.cycles-pp.native_sched_clock
0.02 ?141% +0.0 0.03 ? 70% +0.0 0.03 ? 70% perf-profile.self.cycles-pp.native_irq_return_iret
0.01 ?223% +0.0 0.02 ? 99% -0.0 0.00 perf-profile.self.cycles-pp.xfs_perag_get
0.03 ? 70% +0.0 0.05 ? 8% -0.0 0.02 ? 99% perf-profile.self.cycles-pp.lapic_next_deadline
1.89 ? 3% +0.1 1.96 ? 4% +0.0 1.90 ? 2% perf-profile.self.cycles-pp.rwsem_spin_on_owner
0.11 ? 4% +0.2 0.30 ? 5% +0.2 0.29 perf-profile.self.cycles-pp.__xfs_btree_check_sblock
35.08 ? 4% +1.3 36.33 ? 2% +1.9 37.01 ? 3% perf-profile.self.cycles-pp.osq_lock
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> [email protected]
>
>
> xfs: avoid unnecessary runtime sibling pointer endian conversions
>
> From: Dave Chinner <[email protected]>
>
> Commit dc04db2aa7c9 has caused a small aim7 regression, showing a
> small increase in CPU usage in __xfs_btree_check_sblock() as a
> result of the extra checking.
>
> This is likely due to the endian conversion of the sibling poitners
> being unconditional instead of relying on the compiler to endian
> convert the NULL pointer at compile time and avoiding the runtime
> conversion for this common case.
>
> Rework the checks so that endian conversion of the sibling pointers
> is only done if they are not null as the original code did.
>
> Fixes: dc04db2aa7c9 ("xfs: detect self referencing btree sibling pointers")
> Reported-by: kernel test robot <[email protected]>
> Signed-off-by: Dave Chinner <[email protected]>
> ---
> fs/xfs/libxfs/xfs_btree.c | 37 +++++++++++++++++++++++++------------
> 1 file changed, 25 insertions(+), 12 deletions(-)
>
> diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c
> index 2aa300f7461f..4d673e943317 100644
> --- a/fs/xfs/libxfs/xfs_btree.c
> +++ b/fs/xfs/libxfs/xfs_btree.c
> @@ -51,16 +51,25 @@ xfs_btree_magic(
> return magic;
> }
>
> +/*
> + * These sibling pointer checks are optimised for null sibling pointers. This
> + * happens a lot, and we don't need to byte swap at runtime if the sibling
> + * pointer is NULL.
> + */
> static xfs_failaddr_t
> xfs_btree_check_lblock_siblings(
> struct xfs_mount *mp,
> struct xfs_btree_cur *cur,
> int level,
> xfs_fsblock_t fsb,
> - xfs_fsblock_t sibling)
> + __be64 dsibling)
> {
> - if (sibling == NULLFSBLOCK)
> + xfs_fsblock_t sibling;
> +
> + if (dsibling == cpu_to_be64(NULLFSBLOCK))
> return NULL;
> +
> + sibling = be64_to_cpu(dsibling);
> if (sibling == fsb)
> return __this_address;
> if (level >= 0) {
> @@ -81,10 +90,14 @@ xfs_btree_check_sblock_siblings(
> int level,
> xfs_agnumber_t agno,
> xfs_agblock_t agbno,
> - xfs_agblock_t sibling)
> + __be32 dsibling)
> {
> - if (sibling == NULLAGBLOCK)
> + xfs_agblock_t sibling;
> +
> + if (dsibling == cpu_to_be32(NULLAGBLOCK))
> return NULL;
> +
> + sibling = be32_to_cpu(dsibling);
> if (sibling == agbno)
> return __this_address;
> if (level >= 0) {
> @@ -136,10 +149,10 @@ __xfs_btree_check_lblock(
> fsb = XFS_DADDR_TO_FSB(mp, xfs_buf_daddr(bp));
>
> fa = xfs_btree_check_lblock_siblings(mp, cur, level, fsb,
> - be64_to_cpu(block->bb_u.l.bb_leftsib));
> + block->bb_u.l.bb_leftsib);
> if (!fa)
> fa = xfs_btree_check_lblock_siblings(mp, cur, level, fsb,
> - be64_to_cpu(block->bb_u.l.bb_rightsib));
> + block->bb_u.l.bb_rightsib);
> return fa;
> }
>
> @@ -204,10 +217,10 @@ __xfs_btree_check_sblock(
> }
>
> fa = xfs_btree_check_sblock_siblings(mp, cur, level, agno, agbno,
> - be32_to_cpu(block->bb_u.s.bb_leftsib));
> + block->bb_u.s.bb_leftsib);
> if (!fa)
> fa = xfs_btree_check_sblock_siblings(mp, cur, level, agno,
> - agbno, be32_to_cpu(block->bb_u.s.bb_rightsib));
> + agbno, block->bb_u.s.bb_rightsib);
> return fa;
> }
>
> @@ -4523,10 +4536,10 @@ xfs_btree_lblock_verify(
> /* sibling pointer verification */
> fsb = XFS_DADDR_TO_FSB(mp, xfs_buf_daddr(bp));
> fa = xfs_btree_check_lblock_siblings(mp, NULL, -1, fsb,
> - be64_to_cpu(block->bb_u.l.bb_leftsib));
> + block->bb_u.l.bb_leftsib);
> if (!fa)
> fa = xfs_btree_check_lblock_siblings(mp, NULL, -1, fsb,
> - be64_to_cpu(block->bb_u.l.bb_rightsib));
> + block->bb_u.l.bb_rightsib);
> return fa;
> }
>
> @@ -4580,10 +4593,10 @@ xfs_btree_sblock_verify(
> agno = xfs_daddr_to_agno(mp, xfs_buf_daddr(bp));
> agbno = xfs_daddr_to_agbno(mp, xfs_buf_daddr(bp));
> fa = xfs_btree_check_sblock_siblings(mp, NULL, -1, agno, agbno,
> - be32_to_cpu(block->bb_u.s.bb_leftsib));
> + block->bb_u.s.bb_leftsib);
> if (!fa)
> fa = xfs_btree_check_sblock_siblings(mp, NULL, -1, agno, agbno,
> - be32_to_cpu(block->bb_u.s.bb_rightsib));
> + block->bb_u.s.bb_rightsib);
> return fa;
> }
>
> _______________________________________________
> LKP mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
On Sat, May 07, 2022 at 07:09:46PM +0800, Carel Si wrote:
> Hi Dave,
>
> On Sat, May 07, 2022 at 07:29:24AM +1000, Dave Chinner wrote:
> > On Fri, May 06, 2022 at 05:22:50PM +0800, kernel test robot wrote:
> > >
> > >
> > > Greeting,
> > >
> > > FYI, we noticed a -5.6% regression of aim7.jobs-per-min due to commit:
> > >
> > >
> > > commit: 32678f151338b9a321e9e27139a63c81f353acb7 ("[PATCH 1/4] xfs: detect self referencing btree sibling pointers")
> > > url: https://github.com/intel-lab-lkp/linux/commits/Dave-Chinner/xfs-fix-random-format-verification-issues/20220502-162206
> > > base: https://git.kernel.org/cgit/fs/xfs/xfs-linux.git for-next
> > > patch link: https://lore.kernel.org/linux-xfs/[email protected]
> >
> > Well, that answers the concern I had about the impact of
> > changing the way endian conversions were done in that patch.
> >
> > > a44a027a8b2a20fe 32678f151338b9a321e9e27139a
> > > ---------------- ---------------------------
> > > %stddev %change %stddev
> > > \ | \
> > > 464232 -5.6% 438315 aim7.jobs-per-min
> > ....
> > > 0.13 ? 5% +0.2 0.33 ? 6% perf-profile.children.cycles-pp.__xfs_btree_check_sblock
> > ....
> > > 0.11 ? 4% +0.2 0.30 ? 5% perf-profile.self.cycles-pp.__xfs_btree_check_sblock
> >
> > Because there is it, right at the bottom of the profile.
> >
> > Can you try the patch below and see if that fixes the issue?
>
> We tested below patch, it didn't fix the issue, still has -6.4% regression [1]
> comparing to a44a027a8b ("Merge tag 'large-extent-counters-v9' of
> https://github.com/chandanr/linux into xfs-5.19-for-next").
Really? It made the regression *worse*?
Oh, wait, *that* wasn't in the last set of profiles:
....
> 35.30 ? 4% +1.3 36.59 ? 3% +1.9 37.24 ? 3% perf-profile.children.cycles-pp.osq_lock
> 36.88 ? 4% +1.3 38.19 ? 2% +1.9 38.83 ? 3% perf-profile.children.cycles-pp.rwsem_optimistic_spin
> 37.40 ? 4% +1.4 38.77 ? 2% +2.0 39.35 ? 3% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
....
> 1.89 ? 3% +0.1 1.96 ? 4% +0.0 1.90 ? 2% perf-profile.self.cycles-pp.rwsem_spin_on_owner
> 0.11 ? 4% +0.2 0.30 ? 5% +0.2 0.29 perf-profile.self.cycles-pp.__xfs_btree_check_sblock
> 35.08 ? 4% +1.3 36.33 ? 2% +1.9 37.01 ? 3% perf-profile.self.cycles-pp.osq_lock
This test is hammering an rwsem which is them spinning on exclusive
write locks somewhere. That's where all the increase in system time
has come from, and that's what's causing the difference in
perofrmance. I bet this test is hammering single file write IO from
all 96 CPUs at once, and that's where all the problems start.
IOWs, this likley has nothing to with the btree validation change,
and everything to do with the system being driven into a lock and
cacheline contention corner. When there is lock contention like this
on a large CPU count system, we can't infer anything at all from any
other measurement because the cacheline contention skews
everything...
I'm not going to look at this any further, and I think I'm going to
ignore AIM7 write test regressions in future until the test is
reconfigured to avoid this sort of "100 cpus all trying to do
buffered writes to the same file" contention issue.
Cheers,
Dave.
--
Dave Chinner
[email protected]