LinuxLists.cc - [REPOST PATCH v4 0/5] kernfs: proposed locking and concurrency improvement

2021-05-28 08:58:34

Subject: [REPOST PATCH v4 0/5] kernfs: proposed locking and concurrency improvement

There have been a few instances of contention on the kernfs_mutex during
path walks, a case on very large IBM systems seen by myself, a report by
Brice Goglin and followed up by Fox Chen, and I've since seen a couple
of other reports by CoreOS users.

The common thread is a large number of kernfs path walks leading to
slowness of path walks due to kernfs_mutex contention.

The problem being that changes to the VFS over some time have increased
it's concurrency capabilities to an extent that kernfs's use of a mutex
is no longer appropriate. There's also an issue of walks for non-existent
paths causing contention if there are quite a few of them which is a less
common problem.

This patch series is relatively straight forward.

All it does is add the ability to take advantage of VFS negative dentry
caching to avoid needless dentry alloc/free cycles for lookups of paths
that don't exit and change the kernfs_mutex to a read/write semaphore.

The patch that tried to stay in VFS rcu-walk mode during path walks has
been dropped for two reasons. First, it doesn't actually give very much
improvement and, second, if there's a place where mistakes could go
unnoticed it would be in that path. This makes the patch series simpler
to review and reduces the likelihood of problems going unnoticed and
popping up later.

The patch to use a revision to identify if a directory has changed has
also been dropped. If the directory has changed the dentry revision
needs to be updated to avoid subsequent rb tree searches and after
changing to use a read/write semaphore the update also requires a lock.
But the d_lock is the only lock available at this point which might
itself be contended.

Changes since v3:
- remove unneeded indirection when referencing the super block.
- check if inode attribute update is actually needed.

Changes since v2:
- actually fix the inode attribute update locking.
- drop the patch that tried to stay in rcu-walk mode.
- drop the use a revision to identify if a directory has changed patch.

Changes since v1:
- fix locking in .permission() and .getattr() by re-factoring the attribute
handling code.
---

Ian Kent (5):
kernfs: move revalidate to be near lookup
kernfs: use VFS negative dentry caching
kernfs: switch kernfs to use an rwsem
kernfs: use i_lock to protect concurrent inode updates
kernfs: add kernfs_need_inode_refresh()

fs/kernfs/dir.c | 170 ++++++++++++++++++++----------------
fs/kernfs/file.c | 4 +-
fs/kernfs/inode.c | 45 ++++++++--
fs/kernfs/kernfs-internal.h | 5 +-
fs/kernfs/mount.c | 12 +--
fs/kernfs/symlink.c | 4 +-
include/linux/kernfs.h | 2 +-
7 files changed, 147 insertions(+), 95 deletions(-)

--
Ian

2021-05-28 08:58:42

by Ian Kent

[permalink] [raw]

Subject: [REPOST PATCH v4 4/5] kernfs: use i_lock to protect concurrent inode updates

The inode operations .permission() and .getattr() use the kernfs node
write lock but all that's needed is to keep the rb tree stable while
updating the inode attributes as well as protecting the update itself
against concurrent changes.

And .permission() is called frequently during path walks and can cause
quite a bit of contention between kernfs node operations and path
walks when the number of concurrent walks is high.

To change kernfs_iop_getattr() and kernfs_iop_permission() to take
the rw sem read lock instead of the write lock an additional lock is
needed to protect against multiple processes concurrently updating
the inode attributes and link count in kernfs_refresh_inode().

The inode i_lock seems like the sensible thing to use to protect these
inode attribute updates so use it in kernfs_refresh_inode().

Signed-off-by: Ian Kent <[email protected]>
---
fs/kernfs/inode.c | 10 ++++++----
fs/kernfs/mount.c | 4 ++--
2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
index 3b01e9e61f14e..6728ecd81eb37 100644
--- a/fs/kernfs/inode.c
+++ b/fs/kernfs/inode.c
@@ -172,6 +172,7 @@ static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
{
struct kernfs_iattrs *attrs = kn->iattr;

+ spin_lock(&inode->i_lock);
inode->i_mode = kn->mode;
if (attrs)
/*
@@ -182,6 +183,7 @@ static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)

if (kernfs_type(kn) == KERNFS_DIR)
set_nlink(inode, kn->dir.subdirs + 2);
+ spin_unlock(&inode->i_lock);
}

int kernfs_iop_getattr(struct user_namespace *mnt_userns,
@@ -191,9 +193,9 @@ int kernfs_iop_getattr(struct user_namespace *mnt_userns,
struct inode *inode = d_inode(path->dentry);
struct kernfs_node *kn = inode->i_private;

- down_write(&kernfs_rwsem);
+ down_read(&kernfs_rwsem);
kernfs_refresh_inode(kn, inode);
- up_write(&kernfs_rwsem);
+ up_read(&kernfs_rwsem);

generic_fillattr(&init_user_ns, inode, stat);
return 0;
@@ -284,9 +286,9 @@ int kernfs_iop_permission(struct user_namespace *mnt_userns,

kn = inode->i_private;

- down_write(&kernfs_rwsem);
+ down_read(&kernfs_rwsem);
kernfs_refresh_inode(kn, inode);
- up_write(&kernfs_rwsem);
+ up_read(&kernfs_rwsem);

return generic_permission(&init_user_ns, inode, mask);
}
diff --git a/fs/kernfs/mount.c b/fs/kernfs/mount.c
index baa4155ba2edf..f2f909d09f522 100644
--- a/fs/kernfs/mount.c
+++ b/fs/kernfs/mount.c
@@ -255,9 +255,9 @@ static int kernfs_fill_super(struct super_block *sb, struct kernfs_fs_context *k
sb->s_shrink.seeks = 0;

/* get root inode, initialize and unlock it */
- down_write(&kernfs_rwsem);
+ down_read(&kernfs_rwsem);
inode = kernfs_get_inode(sb, info->root->kn);
- up_write(&kernfs_rwsem);
+ up_read(&kernfs_rwsem);
if (!inode) {
pr_debug("kernfs: could not get root inode\n");
return -ENOMEM;

2021-05-28 11:37:50

by Greg KH

[permalink] [raw]

Subject: Re: [REPOST PATCH v4 0/5] kernfs: proposed locking and concurrency improvement

On Fri, May 28, 2021 at 02:33:42PM +0800, Ian Kent wrote:
> There have been a few instances of contention on the kernfs_mutex during
> path walks, a case on very large IBM systems seen by myself, a report by
> Brice Goglin and followed up by Fox Chen, and I've since seen a couple
> of other reports by CoreOS users.
>
> The common thread is a large number of kernfs path walks leading to
> slowness of path walks due to kernfs_mutex contention.
>
> The problem being that changes to the VFS over some time have increased
> it's concurrency capabilities to an extent that kernfs's use of a mutex
> is no longer appropriate. There's also an issue of walks for non-existent
> paths causing contention if there are quite a few of them which is a less
> common problem.
>
> This patch series is relatively straight forward.
>
> All it does is add the ability to take advantage of VFS negative dentry
> caching to avoid needless dentry alloc/free cycles for lookups of paths
> that don't exit and change the kernfs_mutex to a read/write semaphore.
>
> The patch that tried to stay in VFS rcu-walk mode during path walks has
> been dropped for two reasons. First, it doesn't actually give very much
> improvement and, second, if there's a place where mistakes could go
> unnoticed it would be in that path. This makes the patch series simpler
> to review and reduces the likelihood of problems going unnoticed and
> popping up later.
>
> The patch to use a revision to identify if a directory has changed has
> also been dropped. If the directory has changed the dentry revision
> needs to be updated to avoid subsequent rb tree searches and after
> changing to use a read/write semaphore the update also requires a lock.
> But the d_lock is the only lock available at this point which might
> itself be contended.

Fox, can you take some time and test these to verify it all still works
properly with your benchmarks?

thanks,

greg k-h

2021-05-28 12:39:51

by Fox Chen

[permalink] [raw]

Subject: Re: [REPOST PATCH v4 0/5] kernfs: proposed locking and concurrency improvement

On Fri, May 28, 2021 at 4:56 PM Greg Kroah-Hartman
<[email protected]> wrote:
>
> On Fri, May 28, 2021 at 02:33:42PM +0800, Ian Kent wrote:
> > There have been a few instances of contention on the kernfs_mutex during
> > path walks, a case on very large IBM systems seen by myself, a report by
> > Brice Goglin and followed up by Fox Chen, and I've since seen a couple
> > of other reports by CoreOS users.
> >
> > The common thread is a large number of kernfs path walks leading to
> > slowness of path walks due to kernfs_mutex contention.
> >
> > The problem being that changes to the VFS over some time have increased
> > it's concurrency capabilities to an extent that kernfs's use of a mutex
> > is no longer appropriate. There's also an issue of walks for non-existent
> > paths causing contention if there are quite a few of them which is a less
> > common problem.
> >
> > This patch series is relatively straight forward.
> >
> > All it does is add the ability to take advantage of VFS negative dentry
> > caching to avoid needless dentry alloc/free cycles for lookups of paths
> > that don't exit and change the kernfs_mutex to a read/write semaphore.
> >
> > The patch that tried to stay in VFS rcu-walk mode during path walks has
> > been dropped for two reasons. First, it doesn't actually give very much
> > improvement and, second, if there's a place where mistakes could go
> > unnoticed it would be in that path. This makes the patch series simpler
> > to review and reduces the likelihood of problems going unnoticed and
> > popping up later.
> >
> > The patch to use a revision to identify if a directory has changed has
> > also been dropped. If the directory has changed the dentry revision
> > needs to be updated to avoid subsequent rb tree searches and after
> > changing to use a read/write semaphore the update also requires a lock.
> > But the d_lock is the only lock available at this point which might
> > itself be contended.
>
> Fox, can you take some time and test these to verify it all still works
> properly with your benchmarks?

Sure, I will take a look.
Actually, I've tested it before, but I will test it again to confirm it.

> thanks,
>
> greg k-h

thanks,
fox

2021-05-30 04:46:02

by Fox Chen

[permalink] [raw]

Subject: Re: [REPOST PATCH v4 0/5] kernfs: proposed locking and concurrency improvement

On Fri, May 28, 2021 at 4:56 PM Greg Kroah-Hartman
<[email protected]> wrote:
>
> Fox, can you take some time and test these to verify it all still works
> properly with your benchmarks?
>

I've tested it on an AWS C5a (amd, 96 logical cores):
Before, mutex_locks in kernfs_iop_permission(), kernfs_dop_revalidate
take significant time.
With the patchset, there is no mutex_lock issue. (see flamegraph
before.png/after.png)

On AWS C5 (intel, also 96 logical cores), the benchmark runs slower
than on c5a. But I don't think it's related, because
running the benchmark on ext4 is slower too, and the perf report,
which is no different from running on kernfs with this patchset,
shows the pressure is on the VFS side.

My conclusion: It works well with my benchmark.

I've attached:
flame graphs -- before.png/after.png
benchmark outputs -- result.before/result.after
perf reports -- report.before/report.after
perf report on ext4 -- report.baremetal.ext4
for you reference.

thanks,
fox

Attachments:

after.png (90.41 kB)
before.png (84.93 kB)
result.before (4.94 kB)
result.after (4.81 kB)
report.after (207.81 kB)
report.before (162.71 kB)
report.baremetal.ext4 (155.12 kB)
Download all attachments

2021-05-31 16:32:28

by kernel test robot

[permalink] [raw]

Subject: [kernfs] 9a658329cd: stress-ng.get.ops_per_sec 191.4% improvement

Greeting,

FYI, we noticed a

commit: 9a658329cda84c0916a url: in testcase: stress-ng
on test machine: 96 with following parameters:

nr_threads: 10%
disk: 1HDD
testtime: 60s
fs: ext4
class: os
test: get
cpufreq_governor: performance
ucode: 0x5003006

Details are as below:
---------------------------

To reproduce:

git clone bin/lkp install bin/lkp split-job bin/lkp run
=========================== class/compiler/cpufreq_gove os/gcc-9/performance/1HDD
commit:
dbc68beb5a ("kernfs: 9a658329cd ("kernfs:
dbc68beb5a70364f 9a658329cd ---------------- ---------- %stddev %change \ | 318098 ? 5% +191.4% 5301 ? 5% +191.4% 38975 ? 4% -94.1% 805.50 491.11 9.13 ? 4% +116.7% 9069575 -100.0% 5611023 ? 3% -93.7% 8.78 +4.8% 0.09 +0.1 0.19 ? 3% +0.2 26907 ? 15% +35.9% 26907 ? 15% +35.9% 166.34 84.19 +2.7% 282176 -99.3% 207323 6725 ? 15% +36.2% 514724 ? 14% +82.8% 466993 ? 14% +94.6% 175792 ? 22% +392.1% 225225 ? 16% +311.9% 148702 ? 25% +385.8% 185788 ? 19% +301.6% 0.27 ?143% -100.0% 16.33 ? 71% -100.0% 2.82 ?108% -100.0% 0.10 ?191% -100.0% 0.02 ?153% -100.0% 1.26 ?204% -100.0% 0.04 ?177% -100.0% 10168 +2.0% 6781 +3.0% 22147 +7.9% 45841 +9.8% 444471 +281.9% 357937 +350.0% 525864 ? 2% +480.4% 396561 ? 3% +636.6% 2926 -18.0% 2926 -18.0% 122528 +17.0% 2935 +17.2% 123316 +17.2% 2935 +17.2% 30529 +190.0% 961.17 +188.5% 30769 +188.4% 961.17 +188.5% 9308 +107.2% 291.00 ? 2% +109.3% 9325 ? 2% +109.2% 291.00 ? 2% +109.3% 4640 +138.0% 144.67 +138.8% 4640 +138.5% 144.67 +138.8% 4045 -12.2% 4045 -12.2% 6051 -25.8% 6051 -25.8% 11.74 ? 3% +30.8% 2.247e+09 ? 2% 0.96 -0.1 21090920 ? 2% 25783788 ? 6% 1.249e+08 ? 6% 292463 -99.4% 2.52 ? 2% -33.0% 219.80 -49.5% 1253 ? 12% -49.8% 2.792e+09 ? 2% 0.01 ? 6% +0.0 60929 ? 9% +152.5% 1.133e+09 ? 3% 75.05 +11.3 7531434 ? 2% 1.047e+10 ? 2% 1473 -24.7% 0.40 ? 2% +47.9% 11.05 +4.9% 184.97 ? 5% +37.2% 65.58 ? 2% +65.3% 90.65 -1.9 8639400 ? 6% 781197 ? 5% 96.61 -2.3 4553793 ? 6% 41459 ? 9% +438.9% 13312 ? 3% +145.3% 11.92 ? 3% +30.1% 0.94 -0.1 2.54 ? 2% -33.6% 1036 ? 6% -48.0% 76.14 +11.5 1390 -26.2% 0.39 ? 2% +50.9% 91.69 -1.5 99.09 -1.8 2.211e+09 ? 2% 20754755 ? 2% 25369981 ? 6% 1.229e+08 ? 6% 287786 -99.4% 216.30 -49.5% 2.748e+09 ? 2% 59958 ? 9% +152.5% 1.115e+09 ? 3% 7411100 ? 2% 1.03e+10 ? 2% 3119 -0.8% 8500724 ? 6% 768670 ? 5% 4480680 ? 6% 40808 ? 9% +438.7% 13100 ? 3% +145.3% 6.511e+11 ? 2% 19879 ? 5% -81.1% 14745 ? 4% -21.7% 20773 ? 11% -85.4% 13361 ? 6% -34.3% 19242 ? 12% -88.2% 11621 ? 3% -23.4% 20518 ? 8% -88.5% 11882 -28.0% 19804 ? 11% -88.3% 11886 ? 3% -21.5% 19546 ? 6% -87.4% 11649 -24.9% 20062 ? 6% -88.9% 19471 ? 9% -87.7% 11662 ? 2% -30.2% 21044 ? 9% -87.2% 20706 ? 12% -85.8% 19892 ? 11% -86.0% 11724 ? 3% -24.2% 20594 ? 9% -88.7% 11876 ? 3% -18.4% 20272 ? 9% -88.4% 20147 ? 8% -88.0% 11597 ? 2% -22.5% 21232 ? 8% -89.3% 11941 ? 2% -24.5% 21322 ? 10% -88.7% 21544 ? 11% -89.2% 11983 ? 3% -29.5% 13707 ? 6% -79.3% 10804 ? 3% -39.4% 13754 ? 4% -81.0% 10662 ? 2% -25.8% 13168 ? 8% -81.3% 10474 ? 3% -18.2% 14032 ? 9% -80.7% 13950 ? 8% -83.9% 10658 ? 2% -14.3% 13273 ? 6% -84.1% 20047 ? 9% -85.7% 12083 ? 2% -33.2% 13881 ? 6% -84.5% 13523 ? 7% -84.2% 12672 ? 5% -82.1% 13126 ? 7% -84.5% 12706 ? 10% -82.9% 10497 ? 2% -17.6% 12699 ? 7% -83.0% 10613 ? 3% -10.5% 12472 ? 8% -82.9% 10450 ? 3% -12.6% 12641 ? 8% -82.8% 12374 ? 7% -83.0% 12532 ? 7% -82.3% 10407 ? 3% -19.9% 20339 ? 15% -87.5% 12051 ? 4% -26.5% 13264 ? 7% -84.1% 13351 ? 7% -83.9% 13317 ? 8% -84.1% 10595 ? 3% -21.0% 12939 ? 10% -83.9% 10415 ? 4% -17.8% 12572 ? 7% -83.8% 10484 ? 2% -16.5% 13022 ? 7% -84.2% 10552 ? 3% -17.6% 12842 ? 5% -83.4% 12593 ? 4% -82.3% 19365 ? 15% -89.3% 11060 ? 7% -24.5% 19953 ? 12% -89.5% 11517 ? 4% -27.7% 20599 ? 12% -87.8% 19806 ? 10% -88.2% 11157 ? 5% -23.4% 19892 ? 11% -89.3% 19758 ? 15% -86.8% 19122 ? 11% -86.8% 20141 ? 12% -89.0% 11311 ? 9% -24.4% 18913 ? 10% -87.0% 11165 ? 6% -28.3% 18701 ? 13% -87.7% 11247 ? 3% -21.8% 19742 ? 10% -88.0% 11543 ? 3% -28.6% 19075 ? 9% -88.5% 18777 ? 9% -86.9% 19641 ? 8% -88.2% 11577 ? 4% -24.9% 18783 ? 16% -87.2% 11459 ? 5% -20.9% 19265 ? 17% -87.8% 11501 ? 3% -32.0% 18970 ? 8% -87.8% 11106 ? 5% -21.6% 19648 ? 11% -87.2% 11679 ? 3% -24.9% 20771 ? 7% -89.6% 11725 ? 6% -21.3% 20874 ? 11% -88.6% 11801 ? 3% -28.6% 21376 ? 5% -89.0% 11815 ? 2% -23.2% 20018 ? 9% -88.7% 11509 ? 3% -25.3% 19886 ? 11% -87.2% 11531 ? 4% -20.8% 19427 ? 6% -88.7% 11508 ? 2% -26.3% 20383 ? 9% -86.6% 12005 ? 3% -34.3% 20153 ? 11% -88.2% 11692 ? 3% -28.1% 21687 ? 10% -88.1% 11998 ? 4% -25.1% 13726 ? 7% -81.7% 10651 ? 3% -21.6% 12261 ? 7% -82.8% 10278 ? 2% -19.2% 12825 ? 4% -83.0% 10326 -14.9% 12875 ? 9% -84.5% 10516 ? 2% -17.2% 12981 ? 7% -84.4% 10425 ? 2% -13.1% 13111 ? 7% -84.8% 10498 ? 2% -12.1% 12867 ? 8% -83.7% 10435 ? 2% -11.3% 12742 ? 4% -84.0% 10248 ? 2% -13.5% 19762 ? 9% -87.5% 11712 ? 2% -22.9% 12359 ? 9% -84.4% 10233 ? 3% -9.8% 12142 ? 5% -84.1% 12698 ? 8% -84.2% 10555 ? 3% -13.2% 12563 ? 8% -84.6% 10406 ? 2% -11.7% 12341 ? 8% -83.8% 10226 ? 3% -11.3% 12409 ? 4% -84.0% 10146 ? 2% -15.5% 12025 ? 8% -83.4% 10220 ? 3% -16.9% 12530 ? 5% -83.7% 10223 -19.6% 11948 ? 10% -83.3% 10373 ? 3% -15.1% 12595 ? 10% -83.1% 10569 ? 3% -10.4% 20331 ? 10% -88.1% 11652 ? 2% -34.4% 13229 ? 7% -84.7% 10554 ? 3% -19.7% 12425 ? 11% -83.9% 10182 ? 4% -12.0% 12145 ? 6% -83.5% 10348 -12.5% 12686 ? 10% -83.8% 10320 ? 2% -14.1% 12096 ? 6% -83.2% 10247 ? 2% -10.1% 12522 ? 4% -82.4% 1577826 ? 3% -86.0% 1058063 -19.7% 1045243 ? 6% -92.4% 14348 ? 8% -95.5% 117.83 ? 16% -76.7% 16484 ? 11% -96.9% 126.17 ? 15% -91.3% 16068 ? 21% -96.6% 1103 ? 30% -83.2% 1103 ? 30% -83.2% 118.00 ? 24% -97.0% 16994 ? 11% -96.9% 1361 ? 13% -78.7% 1361 ? 13% -78.7% 123.50 ? 13% -95.3% 15303 ? 13% -94.4% 1126 ? 35% -91.3% 1126 ? 35% -91.3% 108.83 ? 15% -97.5% 16337 ? 17% -88.4% 98.00 ? 23% -94.9% 16067 ? 12% -83.3% 110.33 ? 10% -96.2% 15390 ? 16% -96.6% 115.33 ? 21% -98.0% 17398 ? 12% -82.1% 1490 ? 13% -82.3% 1490 ? 13% -82.3% 136.00 ? 12% -98.4% 16726 ? 18% -95.7% 125.17 ? 26% -95.3% 15301 ? 17% -95.3% 123.67 ? 20% -96.6% 17143 ? 17% -85.9% 114.83 ? 16% -96.2% 15882 ? 15% -96.1% 1411 ? 19% -85.7% 1411 ? 19% -85.7% 121.33 ? 18% -92.0% 15669 ? 11% -96.7% 1434 ? 17% -93.4% 1434 ? 17% -93.4% 108.33 ? 16% -98.0% 17038 ? 18% -95.9% 126.00 ? 28% -98.0% 17323 ? 23% -96.7% 119.67 ? 25% -92.9% 17451 ? 20% -96.9% 133.33 ? 19% -97.9% 5645 ? 29% -90.7% 6121 ? 22% -91.4% 5667 ? 33% -86.8% 6011 ? 32% -91.3% 981.50 ? 19% -87.3% 981.50 ? 19% -87.3% 841.33 ? 34% -86.9% 841.33 ? 34% -86.9% 16647 ? 14% -96.8% 1419 ? 20% -80.3% 1419 ? 20% -80.3% 127.17 ? 15% -97.6% 5643 ? 23% -79.8% 5844 ? 30% -75.1% 820.00 ? 20% -87.4% 820.00 ? 20% -87.4% 794.83 ? 21% -87.1% 794.83 ? 21% -87.1% 5306 ? 26% -82.1% 5519 ? 29% -77.0% 5379 ? 29% -84.7% 666.67 ? 45% -83.9% 666.67 ? 45% -83.9% 5609 ? 33% -75.8% 6058 ? 37% -91.4% 16672 ? 19% -96.0% 124.17 ? 22% -90.1% 5562 ? 31% -79.9% 558.33 ? 40% -80.9% 558.33 ? 40% -80.9% 5934 ? 31% -91.2% 5945 ? 33% -83.9% 5361 ? 32% -90.1% 5511 ? 24% -90.7% 5339 ? 25% -75.9% 5086 ? 23% -89.8% 15990 ? 21% -96.8% 101.00 ? 22% -94.6% 17628 ? 21% -96.9% 101.83 ? 23% -93.8% 16369 ? 19% -94.0% 1405 ? 17% -93.1% 1405 ? 17% -93.1% 113.00 ? 19% -98.4% 17436 ? 17% -96.8% 1250 ? 31% -76.1% 1250 ? 31% -76.1% 103.17 ? 20% -91.8% 15642 ? 16% -96.6% 1377 ? 15% -88.8% 1377 ? 15% -88.8% 107.33 ? 23% -92.9% 16130 ? 17% -96.8% 100.67 ? 18% -95.7% 15003 ? 21% -96.6% 1411 ? 21% -93.2% 1411 ? 21% -93.2% 95.00 ? 27% -92.8% 16791 ? 22% -96.7% 102.17 ? 41% -94.9% 15999 ? 16% -96.7% 93.00 ? 17% -92.7% 15850 ? 22% -96.9% 87.33 ? 21% -93.3% 16309 ? 17% -96.8% 104.50 ? 15% -94.9% 14718 ? 19% -96.5% 1386 ? 23% -90.4% 1386 ? 23% -90.4% 100.00 ? 25% -96.0% 14707 ? 10% -96.5% 1408 ? 20% -80.7% 1408 ? 20% -80.7% 103.33 ? 7% -95.0% 16059 ? 15% -96.6% 136.00 ? 21% -95.8% 15466 ? 26% -96.5% 1216 ? 25% -91.2% 1216 ? 25% -91.2% 90.00 ? 32% -96.5% 16533 ? 29% -96.9% 97.33 ? 41% -87.0% 15917 ? 15% -96.5% 85.17 ? 15% -91.6% 16431 ? 18% -96.9% 107.67 ? 24% -96.4% 17023 ? 14% -96.4% 102.33 ? 15% -90.4% 17217 ? 18% -97.0% 121.33 ? 19% -94.5% 17319 ? 10% -97.0% 100.33 ? 17% -95.0% 16534 ? 17% -96.9% 88.67 ? 24% -91.2% 15719 ? 21% -96.7% 1225 ? 28% -92.5% 1225 ? 28% -92.5% 96.50 ? 25% -97.2% 15215 ? 15% -96.7% 84.67 ? 25% -95.1% 16512 ? 14% -96.8% 116.83 ? 23% -92.2% 14889 ? 22% -96.5% 99.17 ? 28% -95.6% 17056 ? 13% -97.0% 91.83 ? 19% -92.6% 6314 ? 31% -91.7% 4920 ? 31% -89.5% 5577 ? 19% -90.8% 5454 ? 30% -89.8% 5443 ? 28% -90.2% 760.00 ? 23% -82.8% 760.00 ? 23% -82.8% 5486 ? 28% -90.4% 778.17 ? 37% -87.3% 778.17 ? 37% -87.3% 5303 ? 33% -89.4% 5484 ? 20% -88.7% 16625 ? 15% -93.7% 107.33 ? 22% -93.9% 5364 ? 33% -89.5% 731.67 ? 27% -87.0% 731.67 ? 27% -87.0% 5111 ? 21% -89.4% 717.33 ? 29% -86.8% 717.33 ? 29% -86.8% 5566 ? 23% -90.7% 5540 ? 29% -90.4% 5199 ? 30% -90.1% 737.17 ? 33% -87.5% 737.17 ? 33% -87.5% 5501 ? 27% -90.5% 5142 ? 31% -89.8% 5427 ? 24% -90.5% 4763 ? 31% -89.1% 675.50 ? 26% -84.2% 675.50 ? 26% -84.2% 5099 ? 30% -89.8% 16196 ? 14% -96.3% 117.67 ? 18% -95.9% 5940 ? 25% -91.1% 5525 ? 32% -87.8% 5040 ? 25% -89.6% 5392 ? 33% -90.5% 4800 ? 24% -89.3% 6775 ? 13% -92.4% 7137 ? 7% -93.8% 29.06 ? 7% -29.1 23.13 ? 7% -23.1 26.22 ? 7% -19.4 26.67 ? 7% -17.9 30.59 ? 7% -12.4 11.52 ? 7% -11.5 13.03 ? 7% -9.7 13.28 ? 7% -8.8 32.52 ? 7% -8.8 32.59 ? 7% -8.7 15.28 ? 7% -5.8 4.45 ? 6% -3.7 4.50 ? 6% -3.6 4.55 ? 6% -3.5 5.18 ? 6% -1.2 0.72 ? 7% +0.6 0.00 +0.7 0.00 +0.7 0.00 +0.7 0.00 +0.7 0.00 +0.7 0.00 +0.7 0.00 +0.7 0.00 +0.8 1.35 ? 6% +0.8 0.00 +0.8 0.00 +0.8 0.00 +0.8 0.00 +0.9 0.00 +0.9 0.00 +0.9 0.00 +0.9 0.00 +0.9 0.00 +0.9 0.00 +0.9 0.00 +1.0 0.00 +1.0 0.00 +1.0 0.00 +1.0 0.00 +1.1 0.00 +1.1 0.00 +1.2 0.00 +1.2 0.00 +1.2 0.47 ? 44% +1.3 0.00 +1.3 0.36 ? 71% +1.3 2.68 ? 8% +1.3 0.49 ? 45% +1.3 0.50 ? 45% +1.3 0.00 +1.3 0.00 +1.3 0.00 +1.4 0.00 +1.4 0.00 +1.4 0.50 ? 44% +1.4 0.09 ?223% +1.4 0.00 +1.4 0.00 +1.4 0.47 ? 45% +1.5 0.75 ? 8% +1.5 0.00 +1.5 0.00 +1.5 0.09 ?223% +1.5 0.00 +1.5 0.82 ? 8% +1.6 0.00 +1.6 0.00 +1.7 0.00 +1.8 0.00 +1.9 0.00 +1.9 1.75 ? 7% +1.9 0.27 ?100% +1.9 1.30 ? 9% +2.0 0.00 +2.1 0.75 ? 13% +2.2 1.27 ? 10% +2.2 1.06 ? 9% +2.3 1.08 ? 9% +2.3 0.00 +2.4 0.00 +2.7 3.44 ? 9% +3.5 0.00 +3.6 38.79 ? 7% -38.8 43.72 ? 7% -32.8 32.43 ? 7% -32.4 44.46 ? 7% -30.2 45.89 ? 7% -18.2 32.54 ? 7% -8.7 32.61 ? 7% -8.6 4.55 ? 6% -3.5 5.19 ? 6% -1.2 0.68 ? 6% -0.6 1.24 ? 8% -0.6 0.30 ? 10% -0.2 0.30 ? 10% -0.2 0.29 ? 7% -0.2 0.25 ? 13% -0.2 0.24 ? 12% -0.2 0.18 ? 12% -0.1 0.22 ? 6% -0.1 0.12 ? 15% -0.0 0.04 ? 71% +0.0 0.10 ? 11% +0.0 0.03 ?100% +0.1 0.03 ? 70% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.06 ? 11% +0.1 0.12 ? 19% +0.1 0.08 ? 23% +0.1 0.06 ? 14% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.07 ? 17% +0.1 0.00 +0.1 0.00 +0.1 0.05 ? 46% +0.1 0.00 +0.1 0.00 +0.1 0.04 ? 71% +0.1 0.05 ? 56% +0.1 0.06 ? 11% +0.1 0.02 ?141% +0.1 0.00 +0.1 0.00 +0.1 0.01 ?223% +0.1 0.01 ?223% +0.1 0.00 +0.1 0.05 ? 47% +0.1 0.05 ? 47% +0.1 0.04 ? 73% +0.1 0.00 +0.1 0.05 ? 46% +0.1 0.06 ? 11% +0.1 0.01 ?223% +0.1 0.04 ? 72% +0.1 0.00 +0.1 0.06 ? 7% +0.1 0.01 ?223% +0.1 0.06 ? 47% +0.1 0.00 +0.1 0.05 ? 45% +0.1 0.05 ? 48% +0.1 0.06 ? 11% +0.1 0.03 ?100% +0.1 0.02 ?142% +0.1 0.07 ? 11% +0.1 0.09 ? 9% +0.1 0.04 ? 72% +0.1 0.04 ? 71% +0.1 0.00 +0.1 0.08 ? 6% +0.1 0.16 ? 9% +0.1 0.09 ? 20% +0.2 0.09 ? 9% +0.2 0.11 ? 12% +0.2 0.08 ? 21% +0.2 0.06 ? 18% +0.2 0.09 ? 9% +0.2 0.00 +0.2 0.00 +0.2 0.12 ? 25% +0.2 0.08 ? 14% +0.2 0.11 ? 13% +0.2 0.05 ? 49% +0.2 0.00 +0.2 0.23 ? 9% +0.2 0.10 ? 14% +0.2 0.13 ? 18% +0.2 0.11 ? 9% +0.2 0.13 ? 25% +0.2 0.04 ? 71% +0.2 0.08 ? 5% +0.2 0.09 ? 16% +0.2 0.12 ? 16% +0.2 0.12 ? 14% +0.3 0.11 ? 10% +0.3 0.14 ? 12% +0.3 0.17 ? 15% +0.3 0.00 +0.3 0.00 +0.3 0.00 +0.3 0.20 ? 9% +0.3 0.16 ? 12% +0.3 0.08 ? 13% +0.3 0.19 ? 10% +0.3 0.05 ? 77% +0.4 0.09 ? 11% +0.4 0.25 ? 12% +0.4 0.11 ? 12% +0.4 0.11 ? 10% +0.4 0.23 ? 11% +0.4 0.07 ? 56% +0.4 0.22 ? 10% +0.4 0.07 ? 14% +0.4 0.11 ? 27% +0.4 0.06 ? 13% +0.4 0.11 ? 14% +0.4 0.05 ? 47% +0.4 0.11 ? 14% +0.5 0.07 ? 20% +0.5 0.31 ? 12% +0.5 0.12 ? 8% +0.5 0.30 ? 7% +0.5 0.13 ? 6% +0.5 0.33 ? 11% +0.5 0.22 ? 6% +0.5 0.21 ? 7% +0.6 0.24 ? 7% +0.6 0.33 ? 13% +0.6 0.32 ? 12% +0.6 0.28 ? 14% +0.6 0.27 ? 6% +0.6 0.89 ? 8% +0.7 0.43 ? 10% +0.8 0.22 ? 14% +0.8 0.43 ? 9% +0.8 0.54 ? 12% +0.8 0.30 ? 11% +0.9 0.73 ? 8% +0.9 0.49 ? 13% +0.9 0.50 ? 11% +0.9 0.43 ? 7% +0.9 0.50 ? 12% +1.0 3.33 ? 8% +1.0 3.06 ? 8% +1.0 0.43 ? 10% +1.0 0.59 ? 10% +1.1 0.51 ? 11% +1.1 0.57 ? 13% +1.2 0.57 ? 11% +1.2 0.58 ? 10% +1.2 0.43 ? 6% +1.2 0.55 ? 8% +1.4 0.77 ? 8% +1.5 0.61 ? 10% +1.5 0.84 ? 8% +1.6 0.72 ? 9% +1.6 0.59 ? 8% +1.7 0.52 ? 6% +1.7 0.88 ? 10% +1.9 1.28 ? 10% +2.2 0.49 ? 5% +2.3 0.81 ? 10% +2.6 1.21 ? 10% +2.6 0.70 ? 9% +2.9 0.69 ? 8% +2.9 5.04 ? 7% +2.9 1.47 ? 9% +3.2 1.50 ? 9% +3.3 1.19 ? 6% +3.6 1.22 ? 8% +3.8 0.45 ? 8% +4.1 0.00 +4.2 0.84 ? 7% +4.5 1.92 ? 12% +5.2 1.68 ? 9% +5.6 0.91 ? 11% +7.3 6.52 ? 8% +7.4 32.12 ? 7% -32.1 0.52 ? 6% -0.2 0.28 ? 7% -0.2 0.21 ? 11% -0.2 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.05 ? 45% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.03 ? 70% +0.1 0.00 +0.1 0.06 ? 45% +0.1 0.05 ? 47% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.05 ? 47% +0.1 0.05 ? 45% +0.1 0.00 +0.1 0.04 ? 45% +0.1 0.05 ? 45% +0.1 0.06 ? 17% +0.1 0.05 ? 8% +0.1 0.00 +0.1 0.01 ?223% +0.1 0.06 ? 13% +0.1 0.00 +0.1 0.01 ?223% +0.1 0.00 +0.1 0.04 ? 71% +0.1 0.02 ?142% +0.1 0.09 ? 5% +0.1 0.06 ? 11% +0.1 0.04 ? 71% +0.1 0.09 ? 15% +0.1 0.04 ? 71% +0.1 0.04 ? 71% +0.1 0.14 ? 9% +0.1 0.07 ? 10% +0.1 0.04 ? 75% +0.1 0.01 ?223% +0.1 0.02 ?142% +0.1 0.03 ?101% +0.2 0.00 +0.2 0.09 ? 11% +0.2 0.21 ? 8% +0.2 0.07 ? 11% +0.2 0.11 ? 13% +0.2 0.00 +0.2 0.13 ? 14% +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.08 ? 8% +0.2 0.04 ?110% +0.2 0.07 ? 26% +0.2 0.09 ? 20% +0.2 0.09 ? 12% +0.3 0.09 ? 14% +0.3 0.19 ? 11% +0.3 0.00 +0.3 0.21 ? 13% +0.3 0.17 ? 5% +0.3 0.08 ? 13% +0.3 0.24 ? 10% +0.3 0.19 ? 12% +0.3 0.19 ? 10% +0.3 0.18 ? 9% +0.3 0.05 ? 78% +0.3 0.11 ? 13% +0.4 0.26 ? 8% +0.4 0.20 ? 41% +0.4 0.29 ? 13% +0.5 0.29 ? 9% +0.5 0.12 ? 6% +0.5 0.20 ? 15% +0.5 0.20 ? 6% +0.5 0.32 ? 11% +0.6 0.27 ? 14% +0.6 0.17 ? 45% +0.6 0.15 ? 16% +0.6 0.22 ? 22% +0.8 0.70 ? 11% +1.3 0.46 ? 12% +1.4 0.61 ? 9% +1.4 0.94 ? 10% +1.6 0.48 ? 6% +2.2 0.86 ? 11% +2.7 1.17 ? 7% +3.5 1.22 ? 8% +3.5 0.45 ? 8% +4.0 0.69 ? 8% +4.3

1.2e+06 +---------------- 1.1e+06 |-O O O O | O O 1e+06 |-+ O 900000 |-+ | 800000 |-+ 700000 |-+ 600000 |-+ |.+.+. .+.+. 500000 |-+ + 400000 |-+ | 300000 |-+ 200000 +---------------- 20000 +------------------ | O O O O 18000 |-+ O O O O O 16000 |-+ | O 14000 |-+ | 12000 |-+ | 10000 |-+.+. .+. 8000 |.+ + +.+.+.+.+ | 6000 |-+ | 4000 +------------------ [*] bisect-good sample
[O] bisect-bad sample

Disclaimer:
Results have been for informational design or configuration

---
0DAY/LKP+ Test Infrastructure https://github.com/0day-ci/linux/commits/Ian-Kent/kernfs-proposed-locking-and-concurrency-improvement/20210528-143519
.kernel.org/cgit/linux/kernel/git/gregkh/driver-core.git">https://git.kernel.org/cgit/linux/kernel/git/gregkh/driver-core.git 39b27e89a76f3827ad93aed9213a6daf2b91f819
threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
----------------------------------------------------------------------->
href="https://github.com/intel/lkp-tests.git">https://github.com/intel/lkp-tests.git
job.yaml # job file is attached in this email
--compatible job.yaml # generate the yaml file for lkp run
generated-yaml-file
==============================================================
rnor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/get/stress-ng/60s/0x5003006
switch kernfs to use an rwsem")
use i_lock to protect concurrent inode updates")
a84c0916a85fec1cd
-----------------
%stddev
\
926882 ? 4% stress-ng.get.ops
15448 ? 4% stress-ng.get.ops_per_sec
2306 ? 5% stress-ng.time.involuntary_context_switches
+4.8% 843.83 stress-ng.time.percent_of_cpu_this_job_got
+2.7% 504.29 stress-ng.time.system_time
19.79 ? 10% stress-ng.time.user_time
902.67 ? 18% stress-ng.time.voluntary_context_switches
355240 ?213% cpuidle.POLL.usage
9.20 iostat.cpu.system
0.20 ? 3% mpstat.cpu.all.soft%
0.35 ? 9% mpstat.cpu.all.usr%
36572 ? 14% numa-meminfo.node1.KReclaimable
36572 ? 14% numa-meminfo.node1.SReclaimable
+3.0% 171.40 turbostat.PkgWatt
86.46 turbostat.RAMWatt
1950 vmstat.system.cs
-7.6% 191467 vmstat.system.in
9161 ? 14% numa-vmstat.node1.nr_slab_reclaimable
940837 ? 24% numa-vmstat.node1.numa_hit
908580 ? 24% numa-vmstat.node1.numa_local
864994 ? 11% numa-numastat.node0.local_node
927760 ? 11% numa-numastat.node0.numa_hit
722426 ? 12% numa-numastat.node1.local_node
746175 ? 12% numa-numastat.node1.numa_hit
0.00 perf-sched.wait_and_delay.avg.ms.rwsem_down_read_slowpath.kernfs_dop_revalidate.lookup_fast.walk_component
0.00 perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.kernfs_dop_revalidate.lookup_fast.walk_component
0.00 perf-sched.wait_and_delay.max.ms.rwsem_down_read_slowpath.kernfs_dop_revalidate.lookup_fast.walk_component
0.00 perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.kernfs_dop_revalidate.lookup_fast.walk_component
0.00 perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.kernfs_iop_permission.inode_permission.link_path_walk.part
0.00 perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.kernfs_dop_revalidate.lookup_fast.walk_component
0.00 perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.kernfs_iop_permission.inode_permission.link_path_walk.part
10369 proc-vmstat.nr_mapped
6983 proc-vmstat.nr_shmem
23908 proc-vmstat.nr_slab_reclaimable
50328 proc-vmstat.nr_slab_unreclaimable
1697425 ? 2% proc-vmstat.numa_hit
1610894 ? 3% proc-vmstat.numa_local
3052021 ? 3% proc-vmstat.pgalloc_normal
2921129 ? 3% proc-vmstat.pgfree
2399 ? 8% slabinfo.PING.active_objs
2399 ? 8% slabinfo.PING.num_objs
143418 slabinfo.dentry.active_objs
3439 slabinfo.dentry.active_slabs
144486 slabinfo.dentry.num_objs
3439 slabinfo.dentry.num_slabs
88521 ? 4% slabinfo.filp.active_objs
2773 ? 4% slabinfo.filp.active_slabs
88751 ? 4% slabinfo.filp.num_objs
2773 ? 4% slabinfo.filp.num_slabs
19285 ? 8% slabinfo.kmalloc-256.active_objs
609.17 ? 8% slabinfo.kmalloc-256.active_slabs
19506 ? 8% slabinfo.kmalloc-256.num_objs
609.17 ? 8% slabinfo.kmalloc-256.num_slabs
11043 ? 5% slabinfo.kmalloc-rcl-512.active_objs
345.50 ? 5% slabinfo.kmalloc-rcl-512.active_slabs
11069 ? 5% slabinfo.kmalloc-rcl-512.num_objs
345.50 ? 5% slabinfo.kmalloc-rcl-512.num_slabs
3551 ? 6% slabinfo.sock_inode_cache.active_objs
3551 ? 6% slabinfo.sock_inode_cache.num_objs
4488 ? 2% slabinfo.trace_event_file.active_objs
4488 ? 2% slabinfo.trace_event_file.num_objs
15.35 ? 4% perf-stat.i.MPKI
+53.1% 3.44e+09 ? 3% perf-stat.i.branch-instructions
0.85 ? 3% perf-stat.i.branch-miss-rate%
+31.8% 27799658 ? 3% perf-stat.i.branch-misses
+94.6% 50163044 ? 3% perf-stat.i.cache-misses
+98.8% 2.482e+08 perf-stat.i.cache-references
1759 perf-stat.i.context-switches
1.69 ? 4% perf-stat.i.cpi
110.96 perf-stat.i.cpu-migrations
628.55 ? 3% perf-stat.i.cycles-between-cache-misses
+55.5% 4.341e+09 ? 3% perf-stat.i.dTLB-loads
0.01 ? 14% perf-stat.i.dTLB-store-miss-rate%
153866 ? 16% perf-stat.i.dTLB-store-misses
+110.2% 2.381e+09 ? 3% perf-stat.i.dTLB-stores
86.38 perf-stat.i.iTLB-load-miss-rate%
+107.5% 15628073 ? 3% perf-stat.i.iTLB-load-misses
+53.2% 1.604e+10 ? 3% perf-stat.i.instructions
1109 ? 3% perf-stat.i.instructions-per-iTLB-miss
0.59 ? 4% perf-stat.i.ipc
11.60 ? 3% perf-stat.i.major-faults
253.81 ? 2% perf-stat.i.metric.K/sec
108.41 ? 3% perf-stat.i.metric.M/sec
88.79 perf-stat.i.node-load-miss-rate%
+24.0% 10708698 ? 4% perf-stat.i.node-load-misses
+49.5% 1167934 ? 14% perf-stat.i.node-loads
94.33 perf-stat.i.node-store-miss-rate%
+79.7% 8184022 perf-stat.i.node-store-misses
223414 ? 8% perf-stat.i.node-stores
32654 ? 3% perf-stat.i.page-faults
15.50 ? 4% perf-stat.overall.MPKI
0.81 perf-stat.overall.branch-miss-rate%
1.69 ? 4% perf-stat.overall.cpi
539.37 ? 3% perf-stat.overall.cycles-between-cache-misses
87.59 perf-stat.overall.iTLB-load-miss-rate%
1026 ? 3% perf-stat.overall.instructions-per-iTLB-miss
0.59 ? 4% perf-stat.overall.ipc
90.21 perf-stat.overall.node-load-miss-rate%
97.34 perf-stat.overall.node-store-miss-rate%
+53.1% 3.386e+09 ? 3% perf-stat.ps.branch-instructions
+31.8% 27355760 ? 3% perf-stat.ps.branch-misses
+94.6% 49380619 ? 3% perf-stat.ps.cache-misses
+98.9% 2.443e+08 perf-stat.ps.cache-references
1730 perf-stat.ps.context-switches
109.24 perf-stat.ps.cpu-migrations
+55.5% 4.272e+09 ? 3% perf-stat.ps.dTLB-loads
151372 ? 16% perf-stat.ps.dTLB-store-misses
+110.2% 2.343e+09 ? 3% perf-stat.ps.dTLB-stores
+107.6% 15381874 ? 3% perf-stat.ps.iTLB-load-misses
+53.3% 1.579e+10 ? 3% perf-stat.ps.instructions
3094 perf-stat.ps.minor-faults
+24.0% 10541253 ? 4% perf-stat.ps.node-load-misses
+49.6% 1149632 ? 14% perf-stat.ps.node-loads
+79.8% 8056175 perf-stat.ps.node-store-misses
219828 ? 8% perf-stat.ps.node-stores
32137 ? 3% perf-stat.ps.page-faults
+52.6% 9.935e+11 ? 3% perf-stat.total.instructions
3757 ? 8% softirqs.CPU0.RCU
11542 ? 12% softirqs.CPU0.SCHED
3028 ? 18% softirqs.CPU1.RCU
8778 ? 11% softirqs.CPU1.SCHED
2276 ? 7% softirqs.CPU10.RCU
8899 ? 3% softirqs.CPU10.SCHED
2366 ? 18% softirqs.CPU11.RCU
8559 ? 15% softirqs.CPU11.SCHED
2311 ? 10% softirqs.CPU12.RCU
9331 ? 3% softirqs.CPU12.SCHED
2464 ? 16% softirqs.CPU13.RCU
8748 ? 18% softirqs.CPU13.SCHED
2225 ? 6% softirqs.CPU14.RCU
2388 ? 17% softirqs.CPU15.RCU
8135 ? 21% softirqs.CPU15.SCHED
2691 ? 18% softirqs.CPU16.RCU
2937 ? 19% softirqs.CPU17.RCU
2792 ? 20% softirqs.CPU18.RCU
8884 ? 11% softirqs.CPU18.SCHED
2337 ? 11% softirqs.CPU19.RCU
9695 ? 7% softirqs.CPU19.SCHED
2341 ? 8% softirqs.CPU2.RCU
2416 ? 12% softirqs.CPU20.RCU
8991 ? 4% softirqs.CPU20.SCHED
2278 ? 11% softirqs.CPU21.RCU
9014 ? 8% softirqs.CPU21.SCHED
2412 ? 11% softirqs.CPU22.RCU
2330 ? 6% softirqs.CPU23.RCU
8449 ? 16% softirqs.CPU23.SCHED
2832 ? 12% softirqs.CPU24.RCU
6545 ? 26% softirqs.CPU24.SCHED
2609 ? 19% softirqs.CPU25.RCU
7907 ? 14% softirqs.CPU25.SCHED
2458 ? 14% softirqs.CPU26.RCU
8564 ? 14% softirqs.CPU26.SCHED
2705 ? 16% softirqs.CPU27.RCU
2242 ? 7% softirqs.CPU28.RCU
9132 ? 5% softirqs.CPU28.SCHED
2104 ? 4% softirqs.CPU29.RCU
2861 ? 32% softirqs.CPU3.RCU
8073 ? 24% softirqs.CPU3.SCHED
2146 ? 7% softirqs.CPU30.RCU
2133 ? 6% softirqs.CPU31.RCU
2267 ? 25% softirqs.CPU32.RCU
2034 ? 11% softirqs.CPU33.RCU
2171 ? 17% softirqs.CPU34.RCU
8647 ? 21% softirqs.CPU34.SCHED
2164 ? 14% softirqs.CPU35.RCU
9501 ? 9% softirqs.CPU35.SCHED
2137 ? 8% softirqs.CPU36.RCU
9129 ? 8% softirqs.CPU36.SCHED
2172 ? 4% softirqs.CPU37.RCU
2104 ? 7% softirqs.CPU38.RCU
2214 ? 21% softirqs.CPU39.RCU
8338 ? 10% softirqs.CPU39.SCHED
2544 ? 26% softirqs.CPU4.RCU
8856 ? 6% softirqs.CPU4.SCHED
2108 ? 12% softirqs.CPU40.RCU
2149 ? 6% softirqs.CPU41.RCU
2122 ? 10% softirqs.CPU42.RCU
8375 ? 12% softirqs.CPU42.SCHED
2079 ? 13% softirqs.CPU43.RCU
8566 ? 9% softirqs.CPU43.SCHED
2039 ? 10% softirqs.CPU44.RCU
8756 ? 5% softirqs.CPU44.SCHED
2059 ? 6% softirqs.CPU45.RCU
8692 ? 11% softirqs.CPU45.SCHED
2129 ? 9% softirqs.CPU46.RCU
2232 ? 19% softirqs.CPU47.RCU
2066 ? 11% softirqs.CPU48.RCU
8348 ? 8% softirqs.CPU48.SCHED
2089 ? 3% softirqs.CPU49.RCU
8325 ? 3% softirqs.CPU49.SCHED
2521 ? 20% softirqs.CPU5.RCU
2334 ? 14% softirqs.CPU50.RCU
8548 ? 8% softirqs.CPU50.SCHED
2138 ? 6% softirqs.CPU51.RCU
2602 ? 21% softirqs.CPU52.RCU
2518 ? 25% softirqs.CPU53.RCU
2206 ? 8% softirqs.CPU54.RCU
8550 ? 6% softirqs.CPU54.SCHED
2449 ? 38% softirqs.CPU55.RCU
8010 ? 12% softirqs.CPU55.SCHED
2293 ? 17% softirqs.CPU56.RCU
8791 ? 4% softirqs.CPU56.SCHED
2368 ? 15% softirqs.CPU57.RCU
8244 ? 6% softirqs.CPU57.SCHED
2198 ? 6% softirqs.CPU58.RCU
2454 ? 23% softirqs.CPU59.RCU
2324 ? 8% softirqs.CPU6.RCU
8700 ? 7% softirqs.CPU6.SCHED
2409 ? 9% softirqs.CPU60.RCU
9063 ? 8% softirqs.CPU60.SCHED
2359 ? 15% softirqs.CPU61.RCU
7821 ? 12% softirqs.CPU61.SCHED
2323 ? 16% softirqs.CPU62.RCU
8702 ? 9% softirqs.CPU62.SCHED
2524 ? 34% softirqs.CPU63.RCU
8776 ? 12% softirqs.CPU63.SCHED
2166 ? 11% softirqs.CPU64.RCU
9227 ? 3% softirqs.CPU64.SCHED
2389 ? 15% softirqs.CPU65.RCU
8431 ? 7% softirqs.CPU65.SCHED
2361 ? 14% softirqs.CPU66.RCU
9072 ? 6% softirqs.CPU66.SCHED
2266 ? 4% softirqs.CPU67.RCU
8595 ? 10% softirqs.CPU67.SCHED
2537 ? 21% softirqs.CPU68.RCU
9134 ? 4% softirqs.CPU68.SCHED
2198 ? 6% softirqs.CPU69.RCU
8483 ? 15% softirqs.CPU69.SCHED
2733 ? 19% softirqs.CPU7.RCU
7888 ? 15% softirqs.CPU7.SCHED
2387 ? 12% softirqs.CPU70.RCU
8401 ? 16% softirqs.CPU70.SCHED
2580 ? 18% softirqs.CPU71.RCU
8981 ? 6% softirqs.CPU71.SCHED
2518 ? 28% softirqs.CPU72.RCU
8347 ? 8% softirqs.CPU72.SCHED
2114 ? 15% softirqs.CPU73.RCU
8304 ? 5% softirqs.CPU73.SCHED
2185 ? 18% softirqs.CPU74.RCU
8790 ? 6% softirqs.CPU74.SCHED
2001 ? 4% softirqs.CPU75.RCU
8710 ? 12% softirqs.CPU75.SCHED
2020 ? 7% softirqs.CPU76.RCU
9061 ? 6% softirqs.CPU76.SCHED
1994 ? 5% softirqs.CPU77.RCU
9228 ? 5% softirqs.CPU77.SCHED
2091 ? 14% softirqs.CPU78.RCU
9261 softirqs.CPU78.SCHED
2036 ? 6% softirqs.CPU79.RCU
8870 ? 6% softirqs.CPU79.SCHED
2470 ? 21% softirqs.CPU8.RCU
9024 ? 11% softirqs.CPU8.SCHED
1928 ? 4% softirqs.CPU80.RCU
9235 ? 2% softirqs.CPU80.SCHED
1935 ? 3% softirqs.CPU81.RCU
2000 ? 11% softirqs.CPU82.RCU
9163 ? 9% softirqs.CPU82.SCHED
1929 ? 6% softirqs.CPU83.RCU
9189 ? 5% softirqs.CPU83.SCHED
2001 ? 10% softirqs.CPU84.RCU
9071 ? 3% softirqs.CPU84.SCHED
1981 ? 9% softirqs.CPU85.RCU
8569 ? 7% softirqs.CPU85.SCHED
1990 ? 12% softirqs.CPU86.RCU
8497 ? 11% softirqs.CPU86.SCHED
2043 ? 6% softirqs.CPU87.RCU
8222 ? 9% softirqs.CPU87.SCHED
1991 ? 11% softirqs.CPU88.RCU
8811 ? 9% softirqs.CPU88.SCHED
2131 ? 11% softirqs.CPU89.RCU
9472 ? 5% softirqs.CPU89.SCHED
2429 ? 10% softirqs.CPU9.RCU
7638 ? 12% softirqs.CPU9.SCHED
2022 ? 9% softirqs.CPU90.RCU
8478 ? 10% softirqs.CPU90.SCHED
1995 ? 6% softirqs.CPU91.RCU
8959 ? 2% softirqs.CPU91.SCHED
1999 ? 14% softirqs.CPU92.RCU
9052 ? 5% softirqs.CPU92.SCHED
2054 ? 18% softirqs.CPU93.RCU
8869 ? 7% softirqs.CPU93.SCHED
2031 ? 17% softirqs.CPU94.RCU
9213 softirqs.CPU94.SCHED
2207 ? 13% softirqs.CPU95.RCU
220116 ? 3% softirqs.RCU
849975 softirqs.SCHED
79317 ? 10% interrupts.CAL:Function_call_interrupts
648.00 ? 31% interrupts.CPU0.CAL:Function_call_interrupts
27.50 ? 75% interrupts.CPU0.RES:Rescheduling_interrupts
510.00 ? 26% interrupts.CPU1.CAL:Function_call_interrupts
11.00 ? 50% interrupts.CPU1.RES:Rescheduling_interrupts
545.17 ? 7% interrupts.CPU10.CAL:Function_call_interrupts
185.33 ?106% interrupts.CPU10.NMI:Non-maskable_interrupts
185.33 ?106% interrupts.CPU10.PMI:Performance_monitoring_interrupts
3.50 ? 76% interrupts.CPU10.RES:Rescheduling_interrupts
523.33 ? 3% interrupts.CPU11.CAL:Function_call_interrupts
289.67 ?151% interrupts.CPU11.NMI:Non-maskable_interrupts
289.67 ?151% interrupts.CPU11.PMI:Performance_monitoring_interrupts
5.83 ?129% interrupts.CPU11.RES:Rescheduling_interrupts
860.67 ? 66% interrupts.CPU12.CAL:Function_call_interrupts
97.67 ? 31% interrupts.CPU12.NMI:Non-maskable_interrupts
97.67 ? 31% interrupts.CPU12.PMI:Performance_monitoring_interrupts
2.67 ?141% interrupts.CPU12.RES:Rescheduling_interrupts
1896 ?104% interrupts.CPU13.CAL:Function_call_interrupts
5.00 ? 56% interrupts.CPU13.RES:Rescheduling_interrupts
2678 ?157% interrupts.CPU14.CAL:Function_call_interrupts
4.17 ? 88% interrupts.CPU14.RES:Rescheduling_interrupts
517.17 ? 4% interrupts.CPU15.CAL:Function_call_interrupts
2.33 ?141% interrupts.CPU15.RES:Rescheduling_interrupts
3120 ?159% interrupts.CPU16.CAL:Function_call_interrupts
263.83 ?154% interrupts.CPU16.NMI:Non-maskable_interrupts
263.83 ?154% interrupts.CPU16.PMI:Performance_monitoring_interrupts
2.17 ? 81% interrupts.CPU16.RES:Rescheduling_interrupts
716.33 ? 57% interrupts.CPU17.CAL:Function_call_interrupts
5.83 ?144% interrupts.CPU17.RES:Rescheduling_interrupts
717.00 ? 39% interrupts.CPU18.CAL:Function_call_interrupts
4.17 ? 86% interrupts.CPU18.RES:Rescheduling_interrupts
2421 ? 84% interrupts.CPU19.CAL:Function_call_interrupts
4.33 ? 98% interrupts.CPU19.RES:Rescheduling_interrupts
613.17 ? 26% interrupts.CPU2.CAL:Function_call_interrupts
201.83 ? 79% interrupts.CPU2.NMI:Non-maskable_interrupts
201.83 ? 79% interrupts.CPU2.PMI:Performance_monitoring_interrupts
9.67 ?122% interrupts.CPU2.RES:Rescheduling_interrupts
510.83 interrupts.CPU20.CAL:Function_call_interrupts
94.50 ? 31% interrupts.CPU20.NMI:Non-maskable_interrupts
94.50 ? 31% interrupts.CPU20.PMI:Performance_monitoring_interrupts
2.17 ? 81% interrupts.CPU20.RES:Rescheduling_interrupts
704.00 ? 38% interrupts.CPU21.CAL:Function_call_interrupts
2.50 ? 91% interrupts.CPU21.RES:Rescheduling_interrupts
578.00 ? 19% interrupts.CPU22.CAL:Function_call_interrupts
8.50 ?109% interrupts.CPU22.RES:Rescheduling_interrupts
535.17 ? 6% interrupts.CPU23.CAL:Function_call_interrupts
2.83 ? 62% interrupts.CPU23.RES:Rescheduling_interrupts
527.33 ? 3% interrupts.CPU24.CAL:Function_call_interrupts
526.83 ? 4% interrupts.CPU25.CAL:Function_call_interrupts
749.83 ? 69% interrupts.CPU26.CAL:Function_call_interrupts
523.50 ? 5% interrupts.CPU28.CAL:Function_call_interrupts
124.67 ? 71% interrupts.CPU28.NMI:Non-maskable_interrupts
124.67 ? 71% interrupts.CPU28.PMI:Performance_monitoring_interrupts
110.50 ? 23% interrupts.CPU29.NMI:Non-maskable_interrupts
110.50 ? 23% interrupts.CPU29.PMI:Performance_monitoring_interrupts
527.33 ? 5% interrupts.CPU3.CAL:Function_call_interrupts
279.83 ?147% interrupts.CPU3.NMI:Non-maskable_interrupts
279.83 ?147% interrupts.CPU3.PMI:Performance_monitoring_interrupts
3.00 ? 54% interrupts.CPU3.RES:Rescheduling_interrupts
1142 ?108% interrupts.CPU30.CAL:Function_call_interrupts
1457 ?143% interrupts.CPU31.CAL:Function_call_interrupts
103.67 ? 34% interrupts.CPU32.NMI:Non-maskable_interrupts
103.67 ? 34% interrupts.CPU32.PMI:Performance_monitoring_interrupts
102.50 ? 34% interrupts.CPU33.NMI:Non-maskable_interrupts
102.50 ? 34% interrupts.CPU33.PMI:Performance_monitoring_interrupts
949.17 ?101% interrupts.CPU34.CAL:Function_call_interrupts
1272 ? 86% interrupts.CPU35.CAL:Function_call_interrupts
824.67 ? 89% interrupts.CPU36.CAL:Function_call_interrupts
107.33 ? 23% interrupts.CPU36.NMI:Non-maskable_interrupts
107.33 ? 23% interrupts.CPU36.PMI:Performance_monitoring_interrupts
1355 ? 88% interrupts.CPU37.CAL:Function_call_interrupts
518.67 ? 2% interrupts.CPU39.CAL:Function_call_interrupts
665.83 ? 26% interrupts.CPU4.CAL:Function_call_interrupts
12.33 ?107% interrupts.CPU4.RES:Rescheduling_interrupts
1119 ?129% interrupts.CPU40.CAL:Function_call_interrupts
106.83 ? 34% interrupts.CPU40.NMI:Non-maskable_interrupts
106.83 ? 34% interrupts.CPU40.PMI:Performance_monitoring_interrupts
521.50 ? 2% interrupts.CPU42.CAL:Function_call_interrupts
957.00 ?103% interrupts.CPU43.CAL:Function_call_interrupts
533.17 ? 5% interrupts.CPU44.CAL:Function_call_interrupts
515.17 interrupts.CPU45.CAL:Function_call_interrupts
1286 ?134% interrupts.CPU46.CAL:Function_call_interrupts
520.17 ? 2% interrupts.CPU47.CAL:Function_call_interrupts
514.67 interrupts.CPU48.CAL:Function_call_interrupts
5.50 ? 81% interrupts.CPU48.RES:Rescheduling_interrupts
541.33 ? 10% interrupts.CPU49.CAL:Function_call_interrupts
6.33 ? 58% interrupts.CPU49.RES:Rescheduling_interrupts
980.17 ? 83% interrupts.CPU5.CAL:Function_call_interrupts
96.50 ? 32% interrupts.CPU5.NMI:Non-maskable_interrupts
96.50 ? 32% interrupts.CPU5.PMI:Performance_monitoring_interrupts
1.83 ? 73% interrupts.CPU5.RES:Rescheduling_interrupts
561.50 ? 16% interrupts.CPU50.CAL:Function_call_interrupts
298.83 ?129% interrupts.CPU50.NMI:Non-maskable_interrupts
298.83 ?129% interrupts.CPU50.PMI:Performance_monitoring_interrupts
8.50 ?116% interrupts.CPU50.RES:Rescheduling_interrupts
533.33 ? 4% interrupts.CPU51.CAL:Function_call_interrupts
154.33 ?100% interrupts.CPU51.NMI:Non-maskable_interrupts
154.33 ?100% interrupts.CPU51.PMI:Performance_monitoring_interrupts
7.67 ?132% interrupts.CPU51.RES:Rescheduling_interrupts
515.33 ? 3% interrupts.CPU52.CAL:Function_call_interrupts
4.33 ? 51% interrupts.CPU52.RES:Rescheduling_interrupts
515.50 interrupts.CPU53.CAL:Function_call_interrupts
95.67 ? 31% interrupts.CPU53.NMI:Non-maskable_interrupts
95.67 ? 31% interrupts.CPU53.PMI:Performance_monitoring_interrupts
6.83 ? 82% interrupts.CPU53.RES:Rescheduling_interrupts
547.83 ? 9% interrupts.CPU54.CAL:Function_call_interrupts
5.17 ? 54% interrupts.CPU54.RES:Rescheduling_interrupts
525.83 ? 3% interrupts.CPU55.CAL:Function_call_interrupts
6.83 ? 39% interrupts.CPU55.RES:Rescheduling_interrupts
498.50 ? 4% interrupts.CPU56.CAL:Function_call_interrupts
5.83 ? 84% interrupts.CPU56.RES:Rescheduling_interrupts
519.67 ? 2% interrupts.CPU57.CAL:Function_call_interrupts
5.33 ? 38% interrupts.CPU57.RES:Rescheduling_interrupts
518.17 ? 3% interrupts.CPU58.CAL:Function_call_interrupts
132.50 ? 63% interrupts.CPU58.NMI:Non-maskable_interrupts
132.50 ? 63% interrupts.CPU58.PMI:Performance_monitoring_interrupts
4.00 ? 50% interrupts.CPU58.RES:Rescheduling_interrupts
517.67 ? 3% interrupts.CPU59.CAL:Function_call_interrupts
271.17 ?138% interrupts.CPU59.NMI:Non-maskable_interrupts
271.17 ?138% interrupts.CPU59.PMI:Performance_monitoring_interrupts
5.17 ?131% interrupts.CPU59.RES:Rescheduling_interrupts
539.33 ? 4% interrupts.CPU6.CAL:Function_call_interrupts
5.67 ? 57% interrupts.CPU6.RES:Rescheduling_interrupts
536.17 ? 8% interrupts.CPU60.CAL:Function_call_interrupts
106.67 ? 24% interrupts.CPU60.NMI:Non-maskable_interrupts
106.67 ? 24% interrupts.CPU60.PMI:Performance_monitoring_interrupts
3.17 ? 69% interrupts.CPU60.RES:Rescheduling_interrupts
507.83 ? 17% interrupts.CPU61.CAL:Function_call_interrupts
12.67 ? 76% interrupts.CPU61.RES:Rescheduling_interrupts
555.67 ? 13% interrupts.CPU62.CAL:Function_call_interrupts
7.17 ? 85% interrupts.CPU62.RES:Rescheduling_interrupts
515.50 ? 3% interrupts.CPU63.CAL:Function_call_interrupts
3.83 ?106% interrupts.CPU63.RES:Rescheduling_interrupts
610.83 ? 35% interrupts.CPU64.CAL:Function_call_interrupts
9.83 ?106% interrupts.CPU64.RES:Rescheduling_interrupts
514.67 ? 7% interrupts.CPU65.CAL:Function_call_interrupts
6.67 ? 69% interrupts.CPU65.RES:Rescheduling_interrupts
527.33 ? 22% interrupts.CPU66.CAL:Function_call_interrupts
5.00 ? 60% interrupts.CPU66.RES:Rescheduling_interrupts
520.00 ? 2% interrupts.CPU67.CAL:Function_call_interrupts
7.83 ? 73% interrupts.CPU67.RES:Rescheduling_interrupts
516.33 ? 2% interrupts.CPU68.CAL:Function_call_interrupts
91.33 ? 11% interrupts.CPU68.NMI:Non-maskable_interrupts
91.33 ? 11% interrupts.CPU68.PMI:Performance_monitoring_interrupts
2.67 ? 41% interrupts.CPU68.RES:Rescheduling_interrupts
509.33 interrupts.CPU69.CAL:Function_call_interrupts
4.17 ? 44% interrupts.CPU69.RES:Rescheduling_interrupts
531.67 ? 3% interrupts.CPU7.CAL:Function_call_interrupts
9.17 ? 81% interrupts.CPU7.RES:Rescheduling_interrupts
522.33 ? 5% interrupts.CPU70.CAL:Function_call_interrupts
4.33 ? 83% interrupts.CPU70.RES:Rescheduling_interrupts
516.83 ? 2% interrupts.CPU71.CAL:Function_call_interrupts
6.83 ? 99% interrupts.CPU71.RES:Rescheduling_interrupts
522.33 ? 2% interrupts.CPU72.CAL:Function_call_interrupts
517.17 ? 2% interrupts.CPU73.CAL:Function_call_interrupts
511.33 interrupts.CPU74.CAL:Function_call_interrupts
554.33 ? 9% interrupts.CPU75.CAL:Function_call_interrupts
531.83 ? 3% interrupts.CPU76.CAL:Function_call_interrupts
130.83 ? 59% interrupts.CPU76.NMI:Non-maskable_interrupts
130.83 ? 59% interrupts.CPU76.PMI:Performance_monitoring_interrupts
527.67 ? 5% interrupts.CPU77.CAL:Function_call_interrupts
98.50 ? 10% interrupts.CPU77.NMI:Non-maskable_interrupts
98.50 ? 10% interrupts.CPU77.PMI:Performance_monitoring_interrupts
564.50 ? 22% interrupts.CPU78.CAL:Function_call_interrupts
619.00 ? 38% interrupts.CPU79.CAL:Function_call_interrupts
1054 ?104% interrupts.CPU8.CAL:Function_call_interrupts
6.50 ? 62% interrupts.CPU8.RES:Rescheduling_interrupts
563.67 ? 20% interrupts.CPU80.CAL:Function_call_interrupts
95.17 ? 9% interrupts.CPU80.NMI:Non-maskable_interrupts
95.17 ? 9% interrupts.CPU80.PMI:Performance_monitoring_interrupts
544.00 ? 9% interrupts.CPU81.CAL:Function_call_interrupts
94.50 ? 8% interrupts.CPU81.NMI:Non-maskable_interrupts
94.50 ? 8% interrupts.CPU81.PMI:Performance_monitoring_interrupts
515.00 interrupts.CPU82.CAL:Function_call_interrupts
530.00 ? 9% interrupts.CPU83.CAL:Function_call_interrupts
514.50 ? 3% interrupts.CPU84.CAL:Function_call_interrupts
92.00 ? 7% interrupts.CPU84.NMI:Non-maskable_interrupts
92.00 ? 7% interrupts.CPU84.PMI:Performance_monitoring_interrupts
524.17 ? 3% interrupts.CPU85.CAL:Function_call_interrupts
523.33 ? 3% interrupts.CPU86.CAL:Function_call_interrupts
515.00 interrupts.CPU87.CAL:Function_call_interrupts
519.17 ? 8% interrupts.CPU88.CAL:Function_call_interrupts
106.67 ? 32% interrupts.CPU88.NMI:Non-maskable_interrupts
106.67 ? 32% interrupts.CPU88.PMI:Performance_monitoring_interrupts
519.17 interrupts.CPU89.CAL:Function_call_interrupts
591.83 ? 16% interrupts.CPU9.CAL:Function_call_interrupts
4.83 ? 60% interrupts.CPU9.RES:Rescheduling_interrupts
529.33 ? 4% interrupts.CPU90.CAL:Function_call_interrupts
675.50 ? 36% interrupts.CPU91.CAL:Function_call_interrupts
525.67 ? 9% interrupts.CPU92.CAL:Function_call_interrupts
511.83 interrupts.CPU93.CAL:Function_call_interrupts
512.50 interrupts.CPU94.CAL:Function_call_interrupts
517.17 ? 3% interrupts.CPU95.CAL:Function_call_interrupts
442.33 ? 28% interrupts.RES:Rescheduling_interrupts
0.00 perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.kernfs_iop_permission.inode_permission.link_path_walk
0.00 perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.kernfs_iop_permission.inode_permission.link_path_walk.path_lookupat
6.80 ? 10% perf-profile.calltrace.cycles-pp.kernfs_iop_permission.inode_permission.link_path_walk.path_lookupat.filename_lookup
8.80 ? 10% perf-profile.calltrace.cycles-pp.inode_permission.link_path_walk.path_lookupat.filename_lookup.user_statfs
18.18 ? 9% perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.user_statfs.__do_sys_statfs
0.00 perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.kernfs_iop_permission.inode_permission.link_path_walk.path_openat
3.38 ? 10% perf-profile.calltrace.cycles-pp.kernfs_iop_permission.inode_permission.link_path_walk.path_openat.do_filp_open
4.52 ? 10% perf-profile.calltrace.cycles-pp.inode_permission.link_path_walk.path_openat.do_filp_open.do_sys_openat2
23.77 ? 8% perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.user_statfs.__do_sys_statfs.do_syscall_64
23.93 ? 8% perf-profile.calltrace.cycles-pp.filename_lookup.user_statfs.__do_sys_statfs.do_syscall_64.entry_SYSCALL_64_after_hwframe
9.47 ? 8% perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_openat2.do_sys_open
0.73 ? 13% perf-profile.calltrace.cycles-pp.kernfs_iop_permission.inode_permission.may_open.do_open.path_openat
0.89 ? 13% perf-profile.calltrace.cycles-pp.inode_permission.may_open.do_open.path_openat.do_filp_open
1.01 ? 13% perf-profile.calltrace.cycles-pp.may_open.do_open.path_openat.do_filp_open.do_sys_openat2
3.97 ? 13% perf-profile.calltrace.cycles-pp.do_open.path_openat.do_filp_open.do_sys_openat2.do_sys_open
1.29 ? 16% perf-profile.calltrace.cycles-pp.lookup_fast.walk_component.path_lookupat.filename_lookup.user_statfs
0.69 ? 8% perf-profile.calltrace.cycles-pp.lockref_put_return.dput.terminate_walk.path_openat.do_filp_open
0.71 ? 13% perf-profile.calltrace.cycles-pp.kmem_cache_alloc.__alloc_file.alloc_empty_file.path_openat.do_filp_open
0.71 ? 7% perf-profile.calltrace.cycles-pp.lockref_get_not_dead.__legitimize_path.try_to_unlazy.link_path_walk.path_openat
0.72 ? 13% perf-profile.calltrace.cycles-pp.lockref_get.__traverse_mounts.step_into.walk_component.path_lookupat
0.72 ? 11% perf-profile.calltrace.cycles-pp.__task_pid_nr_ns.__x64_sys_getpriority.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.74 ? 14% perf-profile.calltrace.cycles-pp.__d_lookup.lookup_fast.walk_component.link_path_walk.path_openat
0.75 ? 15% perf-profile.calltrace.cycles-pp.apparmor_file_alloc_security.security_file_alloc.__alloc_file.alloc_empty_file.path_openat
0.77 ? 12% perf-profile.calltrace.cycles-pp.step_into.path_openat.do_filp_open.do_sys_openat2.do_sys_open
2.12 ? 11% perf-profile.calltrace.cycles-pp.lookup_fast.walk_component.link_path_walk.path_openat.do_filp_open
0.81 ? 13% perf-profile.calltrace.cycles-pp.fs_index.__x64_sys_sysfs.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.81 ? 15% perf-profile.calltrace.cycles-pp.__d_lookup.lookup_fast.walk_component.path_lookupat.filename_lookup
0.85 ? 16% perf-profile.calltrace.cycles-pp.getname_flags.user_path_at_empty.user_statfs.__do_sys_statfs.do_syscall_64
0.87 ? 11% perf-profile.calltrace.cycles-pp.dput.step_into.walk_component.link_path_walk.path_openat
0.88 ? 16% perf-profile.calltrace.cycles-pp.user_path_at_empty.user_statfs.__do_sys_statfs.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.89 ? 12% perf-profile.calltrace.cycles-pp.__legitimize_path.try_to_unlazy.link_path_walk.path_openat.do_filp_open
0.90 ? 15% perf-profile.calltrace.cycles-pp.security_file_alloc.__alloc_file.alloc_empty_file.path_openat.do_filp_open
0.92 ? 18% perf-profile.calltrace.cycles-pp.__traverse_mounts.step_into.walk_component.link_path_walk.path_lookupat
0.92 ? 13% perf-profile.calltrace.cycles-pp.try_to_unlazy.link_path_walk.path_openat.do_filp_open.do_sys_openat2
0.93 ? 4% perf-profile.calltrace.cycles-pp.dput.terminate_walk.path_openat.do_filp_open.do_sys_openat2
0.97 ? 6% perf-profile.calltrace.cycles-pp.lockref_put_return.dput.terminate_walk.path_lookupat.filename_lookup
0.98 ? 15% perf-profile.calltrace.cycles-pp.down_read.kernfs_iop_permission.inode_permission.link_path_walk.path_openat
1.00 ? 2% perf-profile.calltrace.cycles-pp.terminate_walk.path_openat.do_filp_open.do_sys_openat2.do_sys_open
1.04 ? 9% perf-profile.calltrace.cycles-pp.up_read.kernfs_iop_permission.inode_permission.link_path_walk.path_openat
1.11 ? 13% perf-profile.calltrace.cycles-pp.__traverse_mounts.step_into.walk_component.path_lookupat.filename_lookup
1.15 ? 15% perf-profile.calltrace.cycles-pp.__x64_sys_prlimit64.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.17 ? 12% perf-profile.calltrace.cycles-pp.__percpu_counter_sum.ext4_statfs.statfs_by_dentry.vfs_statfs.user_statfs
1.21 ? 8% perf-profile.calltrace.cycles-pp.dput.terminate_walk.path_lookupat.filename_lookup.user_statfs
1.22 ? 12% perf-profile.calltrace.cycles-pp.ext4_statfs.statfs_by_dentry.vfs_statfs.user_statfs.__do_sys_statfs
1.72 ? 13% perf-profile.calltrace.cycles-pp.shmem_statfs.statfs_by_dentry.vfs_statfs.user_statfs.__do_sys_statfs
1.26 ? 8% perf-profile.calltrace.cycles-pp.terminate_walk.path_lookupat.filename_lookup.user_statfs.__do_sys_statfs
1.63 ? 12% perf-profile.calltrace.cycles-pp.__do_sys_fstatfs.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.96 ? 14% perf-profile.calltrace.cycles-pp.lookup_fast.walk_component.link_path_walk.path_lookupat.filename_lookup
1.78 ? 13% perf-profile.calltrace.cycles-pp.__alloc_file.alloc_empty_file.path_openat.do_filp_open.do_sys_openat2
1.80 ? 13% perf-profile.calltrace.cycles-pp.alloc_empty_file.path_openat.do_filp_open.do_sys_openat2.do_sys_open
1.30 ? 35% perf-profile.calltrace.cycles-pp._raw_spin_lock.__d_lookup.lookup_fast.walk_component.link_path_walk
1.31 ? 8% perf-profile.calltrace.cycles-pp.kernfs_refresh_inode.kernfs_iop_permission.inode_permission.link_path_walk.path_openat
1.35 ? 11% perf-profile.calltrace.cycles-pp.statfs_by_dentry.vfs_statfs.fd_statfs.__do_sys_fstatfs.do_syscall_64
1.36 ? 12% perf-profile.calltrace.cycles-pp.__x64_sys_sysfs.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.39 ? 12% perf-profile.calltrace.cycles-pp.vfs_statfs.fd_statfs.__do_sys_fstatfs.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.92 ? 13% perf-profile.calltrace.cycles-pp.step_into.walk_component.path_lookupat.filename_lookup.user_statfs
1.51 ? 13% perf-profile.calltrace.cycles-pp.__percpu_counter_sum.shmem_statfs.statfs_by_dentry.vfs_statfs.user_statfs
1.43 ? 12% perf-profile.calltrace.cycles-pp.fd_statfs.__do_sys_fstatfs.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.45 ? 5% perf-profile.calltrace.cycles-pp.lockref_get_not_dead.__legitimize_path.try_to_unlazy.link_path_walk.path_lookupat
1.93 ? 13% perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.24 ? 13% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.51 ? 13% perf-profile.calltrace.cycles-pp.step_into.walk_component.link_path_walk.path_openat.do_filp_open
1.51 ? 18% perf-profile.calltrace.cycles-pp.__d_lookup.lookup_fast.walk_component.link_path_walk.path_lookupat
1.62 ? 29% perf-profile.calltrace.cycles-pp.down_read.kernfs_dop_revalidate.lookup_fast.walk_component.link_path_walk
1.54 ? 31% perf-profile.calltrace.cycles-pp.lockref_put_return.dput.step_into.walk_component.link_path_walk
2.38 ? 14% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.63 ? 14% perf-profile.calltrace.cycles-pp.dput.step_into.walk_component.link_path_walk.path_lookupat
1.65 ? 13% perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
1.79 ? 12% perf-profile.calltrace.cycles-pp.__legitimize_path.try_to_unlazy.link_path_walk.path_lookupat.filename_lookup
1.85 ? 12% perf-profile.calltrace.cycles-pp.try_to_unlazy.link_path_walk.path_lookupat.filename_lookup.user_statfs
1.89 ? 14% perf-profile.calltrace.cycles-pp.down_read.kernfs_iop_permission.inode_permission.link_path_walk.path_lookupat
3.67 ? 12% perf-profile.calltrace.cycles-pp.walk_component.link_path_walk.path_openat.do_filp_open.do_sys_openat2
2.21 ? 14% perf-profile.calltrace.cycles-pp.do_dentry_open.do_open.path_openat.do_filp_open.do_sys_openat2
3.28 ? 13% perf-profile.calltrace.cycles-pp.walk_component.path_lookupat.filename_lookup.user_statfs.__do_sys_statfs
2.11 ? 11% perf-profile.calltrace.cycles-pp.up_read.kernfs_iop_permission.inode_permission.link_path_walk.path_lookupat
2.90 ? 13% perf-profile.calltrace.cycles-pp.step_into.walk_component.link_path_walk.path_lookupat.filename_lookup
3.48 ? 12% perf-profile.calltrace.cycles-pp.__x64_sys_getpriority.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.35 ? 13% perf-profile.calltrace.cycles-pp.statfs_by_dentry.vfs_statfs.user_statfs.__do_sys_statfs.do_syscall_64
3.41 ? 13% perf-profile.calltrace.cycles-pp.vfs_statfs.user_statfs.__do_sys_statfs.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.44 ? 8% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.kernfs_refresh_inode.kernfs_iop_permission.inode_permission
2.70 ? 8% perf-profile.calltrace.cycles-pp.kernfs_refresh_inode.kernfs_iop_permission.inode_permission.link_path_walk.path_lookupat
6.92 ? 13% perf-profile.calltrace.cycles-pp.walk_component.link_path_walk.path_lookupat.filename_lookup.user_statfs
3.65 ? 9% perf-profile.calltrace.cycles-pp._raw_spin_lock.kernfs_refresh_inode.kernfs_iop_permission.inode_permission.link_path_walk
0.00 perf-profile.children.cycles-pp.rwsem_down_write_slowpath
10.93 ? 11% perf-profile.children.cycles-pp.kernfs_iop_permission
0.00 perf-profile.children.cycles-pp.osq_lock
14.24 ? 10% perf-profile.children.cycles-pp.inode_permission
27.71 ? 9% perf-profile.children.cycles-pp.link_path_walk
23.83 ? 8% perf-profile.children.cycles-pp.path_lookupat
24.00 ? 8% perf-profile.children.cycles-pp.filename_lookup
1.02 ? 13% perf-profile.children.cycles-pp.may_open
3.98 ? 13% perf-profile.children.cycles-pp.do_open
0.05 ? 47% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.61 ? 15% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.11 ? 29% perf-profile.children.cycles-pp.ret_from_fork
0.11 ? 29% perf-profile.children.cycles-pp.kthread
0.12 ? 16% perf-profile.children.cycles-pp.native_sched_clock
0.08 ? 57% perf-profile.children.cycles-pp.smpboot_thread_fn
0.07 ? 57% perf-profile.children.cycles-pp.run_ksoftirqd
0.07 ? 16% perf-profile.children.cycles-pp.update_rq_clock
0.13 ? 17% perf-profile.children.cycles-pp.sched_clock_cpu
0.09 ? 14% perf-profile.children.cycles-pp.irq_enter_rcu
0.08 ? 16% perf-profile.children.cycles-pp.task_tick_fair
0.15 ? 17% perf-profile.children.cycles-pp.rcu_all_qs
0.08 ? 20% perf-profile.children.cycles-pp.exc_page_fault
0.09 ? 12% perf-profile.children.cycles-pp.task_work_add
0.06 ? 11% perf-profile.children.cycles-pp.rcu_segcblist_enqueue
0.06 ? 18% perf-profile.children.cycles-pp.simple_statfs
0.06 ? 15% perf-profile.children.cycles-pp.fs_maxindex
0.06 ? 14% perf-profile.children.cycles-pp.shmem_file_read_iter
0.06 ? 11% perf-profile.children.cycles-pp.pick_link
0.13 ? 16% perf-profile.children.cycles-pp.fput_many
0.18 ? 13% perf-profile.children.cycles-pp.call_rcu
0.15 ? 13% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.13 ? 13% perf-profile.children.cycles-pp.__dentry_kill
0.07 ? 9% perf-profile.children.cycles-pp.__do_proc_dointvec
0.07 ? 18% perf-profile.children.cycles-pp.pick_file
0.07 ? 19% perf-profile.children.cycles-pp.do_adjtimex
0.08 ? 16% perf-profile.children.cycles-pp.__radix_tree_lookup
0.08 ? 12% perf-profile.children.cycles-pp.file_free_rcu
0.08 ? 12% perf-profile.children.cycles-pp.proc_dointvec_minmax
0.15 ? 18% perf-profile.children.cycles-pp.asm_exc_page_fault
0.08 ? 22% perf-profile.children.cycles-pp.apparmor_task_setrlimit
0.08 ? 24% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
0.13 ? 16% perf-profile.children.cycles-pp.__lookup_mnt
0.08 ? 27% perf-profile.children.cycles-pp.proc_sys_open
0.08 ? 23% perf-profile.children.cycles-pp.security_task_setrlimit
0.12 ? 15% perf-profile.children.cycles-pp.obj_cgroup_charge_pages
0.14 ? 16% perf-profile.children.cycles-pp.alloc_fd
0.14 ? 14% perf-profile.children.cycles-pp.find_task_by_vpid
0.10 ? 19% perf-profile.children.cycles-pp.__do_sys_getrusage
0.09 ? 18% perf-profile.children.cycles-pp.getrusage
0.09 ? 19% perf-profile.children.cycles-pp.proc_sys_compare
0.10 ? 11% perf-profile.children.cycles-pp.__do_sys_newstat
0.10 ? 11% perf-profile.children.cycles-pp.fsnotify
0.09 ? 9% perf-profile.children.cycles-pp.vfs_statx
0.14 ? 13% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.14 ? 21% perf-profile.children.cycles-pp.__virt_addr_valid
0.13 ? 19% perf-profile.children.cycles-pp.fsnotify_grab_connector
0.09 ? 14% perf-profile.children.cycles-pp.close_fd
0.14 ? 16% perf-profile.children.cycles-pp.copy_user_generic_unrolled
0.16 ? 18% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.10 ? 17% perf-profile.children.cycles-pp.page_counter_try_charge
0.14 ? 18% perf-profile.children.cycles-pp.fsnotify_find_mark
0.10 ? 12% perf-profile.children.cycles-pp._copy_from_user
0.16 ? 16% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.11 ? 13% perf-profile.children.cycles-pp.page_counter_cancel
0.16 ? 17% perf-profile.children.cycles-pp.obj_cgroup_charge
0.11 ? 16% perf-profile.children.cycles-pp.__do_sys_adjtimex
0.16 ? 25% perf-profile.children.cycles-pp.btrfs_statfs
0.17 ? 17% perf-profile.children.cycles-pp.dnotify_flush
0.18 ? 16% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.14 ? 11% perf-profile.children.cycles-pp.page_counter_uncharge
0.13 ? 14% perf-profile.children.cycles-pp.memset_erms
0.18 ? 11% perf-profile.children.cycles-pp.mntput_no_expire
0.21 ? 8% perf-profile.children.cycles-pp.strcmp
0.17 ? 11% perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
0.16 ? 11% perf-profile.children.cycles-pp.__slab_free
0.13 ? 16% perf-profile.children.cycles-pp.allocate_slab
0.21 ? 12% perf-profile.children.cycles-pp.map_id_range_down
0.30 ? 16% perf-profile.children.cycles-pp.__cond_resched
0.25 ? 17% perf-profile.children.cycles-pp.__do_sys_newuname
0.25 ? 14% perf-profile.children.cycles-pp.make_kuid
0.27 ? 14% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.24 ? 11% perf-profile.children.cycles-pp.drain_obj_stock
0.22 ? 14% perf-profile.children.cycles-pp.proc_sys_call_handler
0.26 ? 15% perf-profile.children.cycles-pp.__check_heap_object
0.17 ? 13% perf-profile.children.cycles-pp.___slab_alloc
0.18 ? 13% perf-profile.children.cycles-pp.__slab_alloc
0.30 ? 21% perf-profile.children.cycles-pp.legitimize_mnt
0.26 ? 19% perf-profile.children.cycles-pp.__d_alloc
0.30 ? 16% perf-profile.children.cycles-pp.__d_lookup_rcu
0.24 ? 9% perf-profile.children.cycles-pp.try_to_unlazy_next
0.20 ? 25% perf-profile.children.cycles-pp.security_inode_permission
0.43 ? 14% perf-profile.children.cycles-pp.__might_sleep
0.31 ? 12% perf-profile.children.cycles-pp.new_sync_read
0.34 ? 14% perf-profile.children.cycles-pp.filp_close
0.33 ? 17% perf-profile.children.cycles-pp.__might_fault
0.35 ? 20% perf-profile.children.cycles-pp.lookup_mnt
0.26 ? 18% perf-profile.children.cycles-pp.lockref_put_or_lock
0.31 ? 33% perf-profile.children.cycles-pp.set_nlink
0.33 ? 10% perf-profile.children.cycles-pp.path_put
0.36 ? 15% perf-profile.children.cycles-pp.vfs_read
0.38 ? 15% perf-profile.children.cycles-pp.ksys_read
0.37 ? 11% perf-profile.children.cycles-pp.dcache_dir_close
0.40 ? 12% perf-profile.children.cycles-pp.refill_obj_stock
0.44 ? 15% perf-profile.children.cycles-pp.__x64_sys_close
0.29 ? 20% perf-profile.children.cycles-pp.sysctl_head_finish
0.31 ? 20% perf-profile.children.cycles-pp.sysctl_head_grab
0.31 ? 14% perf-profile.children.cycles-pp.aa_get_task_label
0.53 ? 12% perf-profile.children.cycles-pp.do_statfs_native
0.49 ? 12% perf-profile.children.cycles-pp.fs_name
0.41 ? 15% perf-profile.children.cycles-pp._raw_read_lock
0.52 ? 16% perf-profile.children.cycles-pp.__entry_text_start
0.41 ? 70% perf-profile.children.cycles-pp.set_root
0.46 ? 16% perf-profile.children.cycles-pp.do_prlimit
0.61 ? 14% perf-profile.children.cycles-pp.___might_sleep
0.49 ? 15% perf-profile.children.cycles-pp.d_alloc_cursor
0.49 ? 15% perf-profile.children.cycles-pp.dcache_dir_open
0.61 ? 14% perf-profile.children.cycles-pp.__check_object_size
0.46 ? 61% perf-profile.children.cycles-pp.nd_jump_root
0.62 ? 13% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.46 ? 19% perf-profile.children.cycles-pp.proc_sys_permission
0.53 ? 52% perf-profile.children.cycles-pp.path_init
0.50 ? 16% perf-profile.children.cycles-pp.security_task_getsecid_subj
0.56 ? 15% perf-profile.children.cycles-pp.security_file_free
0.50 ? 16% perf-profile.children.cycles-pp.apparmor_task_getsecid
0.56 ? 15% perf-profile.children.cycles-pp.apparmor_file_free_security
0.53 ? 15% perf-profile.children.cycles-pp.ima_file_check
0.80 ? 10% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.61 ? 14% perf-profile.children.cycles-pp.apparmor_file_open
0.82 ? 14% perf-profile.children.cycles-pp._find_next_bit
0.66 ? 14% perf-profile.children.cycles-pp.security_file_open
0.88 ? 14% perf-profile.children.cycles-pp._copy_to_user
0.77 ? 15% perf-profile.children.cycles-pp.apparmor_file_alloc_security
0.77 ? 12% perf-profile.children.cycles-pp.complete_walk
0.81 ? 13% perf-profile.children.cycles-pp.fs_index
0.90 ? 15% perf-profile.children.cycles-pp.user_path_at_empty
0.91 ? 14% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.90 ? 9% perf-profile.children.cycles-pp.__legitimize_mnt
0.90 ? 15% perf-profile.children.cycles-pp.security_file_alloc
1.62 ? 11% perf-profile.children.cycles-pp.__softirqentry_text_start
1.18 ? 15% perf-profile.children.cycles-pp.strncpy_from_user
1.00 ? 14% perf-profile.children.cycles-pp.generic_permission
1.23 ? 14% perf-profile.children.cycles-pp.cpumask_next
1.36 ? 12% perf-profile.children.cycles-pp.rcu_core
1.16 ? 14% perf-profile.children.cycles-pp.__x64_sys_prlimit64
1.60 ? 10% perf-profile.children.cycles-pp.irq_exit_rcu
1.36 ? 12% perf-profile.children.cycles-pp.rcu_do_batch
1.43 ? 13% perf-profile.children.cycles-pp.kmem_cache_free
1.36 ? 13% perf-profile.children.cycles-pp.__x64_sys_sysfs
1.45 ? 13% perf-profile.children.cycles-pp.kmem_cache_alloc
4.30 ? 8% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
4.03 ? 8% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
1.43 ? 12% perf-profile.children.cycles-pp.fd_statfs
1.65 ? 14% perf-profile.children.cycles-pp.getname_flags
1.64 ? 12% perf-profile.children.cycles-pp.__do_sys_fstatfs
1.78 ? 12% perf-profile.children.cycles-pp.ext4_statfs
1.79 ? 13% perf-profile.children.cycles-pp.__alloc_file
1.81 ? 13% perf-profile.children.cycles-pp.alloc_empty_file
1.67 ? 13% perf-profile.children.cycles-pp.__fput
1.94 ? 13% perf-profile.children.cycles-pp.task_work_run
2.27 ? 13% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
2.12 ? 14% perf-profile.children.cycles-pp.lockref_get
2.43 ? 14% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
2.34 ? 12% perf-profile.children.cycles-pp.shmem_statfs
2.28 ? 5% perf-profile.children.cycles-pp.terminate_walk
2.23 ? 13% perf-profile.children.cycles-pp.do_dentry_open
2.80 ? 16% perf-profile.children.cycles-pp.__traverse_mounts
3.52 ? 12% perf-profile.children.cycles-pp.__x64_sys_getpriority
2.79 ? 4% perf-profile.children.cycles-pp.lockref_get_not_dead
3.39 ? 14% perf-profile.children.cycles-pp.__d_lookup
3.83 ? 12% perf-profile.children.cycles-pp.__percpu_counter_sum
3.56 ? 8% perf-profile.children.cycles-pp.try_to_unlazy
3.58 ? 8% perf-profile.children.cycles-pp.__legitimize_path
7.99 ? 13% perf-profile.children.cycles-pp.lookup_fast
4.71 ? 12% perf-profile.children.cycles-pp.statfs_by_dentry
4.80 ? 12% perf-profile.children.cycles-pp.vfs_statfs
4.80 ? 10% perf-profile.children.cycles-pp.lockref_put_return
4.98 ? 10% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
4.60 ? 12% perf-profile.children.cycles-pp.up_read
4.16 ? 8% perf-profile.children.cycles-pp.kernfs_refresh_inode
5.32 ? 14% perf-profile.children.cycles-pp.down_read
7.13 ? 13% perf-profile.children.cycles-pp.step_into
7.27 ? 7% perf-profile.children.cycles-pp.dput
8.19 ? 10% perf-profile.children.cycles-pp._raw_spin_lock
13.92 ? 12% perf-profile.children.cycles-pp.walk_component
0.00 perf-profile.self.cycles-pp.osq_lock
0.32 ? 17% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.11 ? 14% perf-profile.self.cycles-pp.native_sched_clock
0.05 ? 47% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.06 ? 13% perf-profile.self.cycles-pp.simple_statfs
0.06 ? 11% perf-profile.self.cycles-pp.rcu_segcblist_enqueue
0.06 ? 18% perf-profile.self.cycles-pp._copy_to_user
0.11 ? 16% perf-profile.self.cycles-pp.may_open
0.06 ? 17% perf-profile.self.cycles-pp.path_openat
0.07 ? 20% perf-profile.self.cycles-pp.kernfs_refresh_inode
0.07 ? 18% perf-profile.self.cycles-pp.task_work_run
0.10 ? 14% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.07 ? 20% perf-profile.self.cycles-pp.filename_lookup
0.13 ? 17% perf-profile.self.cycles-pp.do_syscall_64
0.12 ? 19% perf-profile.self.cycles-pp.__lookup_mnt
0.07 ? 19% perf-profile.self.cycles-pp.apparmor_task_setrlimit
0.08 ? 16% perf-profile.self.cycles-pp.__radix_tree_lookup
0.08 ? 12% perf-profile.self.cycles-pp.file_free_rcu
0.08 ? 17% perf-profile.self.cycles-pp.drain_obj_stock
0.08 ? 15% perf-profile.self.cycles-pp.do_statfs_native
0.08 ? 12% perf-profile.self.cycles-pp.path_init
0.08 ? 17% perf-profile.self.cycles-pp.__might_fault
0.08 ? 17% perf-profile.self.cycles-pp.task_work_add
0.13 ? 23% perf-profile.self.cycles-pp.__virt_addr_valid
0.13 ? 12% perf-profile.self.cycles-pp.getname_flags
0.09 ? 7% perf-profile.self.cycles-pp.fsnotify
0.13 ? 16% perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.14 ? 17% perf-profile.self.cycles-pp.copy_user_generic_unrolled
0.15 ? 15% perf-profile.self.cycles-pp.rcu_read_unlock_strict
0.14 ? 15% perf-profile.self.cycles-pp.refill_obj_stock
0.09 ? 18% perf-profile.self.cycles-pp.vfs_statfs
0.10 ? 15% perf-profile.self.cycles-pp.page_counter_try_charge
0.16 ? 11% perf-profile.self.cycles-pp.mntput_no_expire
0.10 ? 19% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
0.11 ? 13% perf-profile.self.cycles-pp.page_counter_cancel
0.11 ? 20% perf-profile.self.cycles-pp.do_sys_openat2
0.14 ? 16% perf-profile.self.cycles-pp.__cond_resched
0.13 ? 14% perf-profile.self.cycles-pp.memset_erms
0.20 ? 9% perf-profile.self.cycles-pp.strcmp
0.18 ? 16% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.15 ? 14% perf-profile.self.cycles-pp.__slab_free
0.21 ? 15% perf-profile.self.cycles-pp.__check_object_size
0.16 ? 13% perf-profile.self.cycles-pp.__alloc_file
0.16 ? 11% perf-profile.self.cycles-pp.walk_component
0.27 ? 17% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.20 ? 15% perf-profile.self.cycles-pp.map_id_range_down
0.18 ? 40% perf-profile.self.cycles-pp.__traverse_mounts
0.15 ? 13% perf-profile.self.cycles-pp.fs_name
0.17 ? 40% perf-profile.self.cycles-pp.kernfs_iop_permission
0.18 ? 12% perf-profile.self.cycles-pp.__fput
0.16 ? 16% perf-profile.self.cycles-pp.fs_index
0.25 ? 16% perf-profile.self.cycles-pp.__check_heap_object
0.38 ? 14% perf-profile.self.cycles-pp.__might_sleep
0.24 ? 13% perf-profile.self.cycles-pp.shmem_statfs
0.30 ? 15% perf-profile.self.cycles-pp.__d_lookup_rcu
0.19 ? 19% perf-profile.self.cycles-pp.apparmor_task_getsecid
0.32 ? 14% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.19 ? 16% perf-profile.self.cycles-pp.lockref_put_or_lock
0.19 ? 25% perf-profile.self.cycles-pp.security_inode_permission
0.20 ? 19% perf-profile.self.cycles-pp.do_prlimit
0.30 ? 33% perf-profile.self.cycles-pp.set_nlink
0.26 ? 65% perf-profile.self.cycles-pp.__legitimize_path
0.30 ? 13% perf-profile.self.cycles-pp.lookup_fast
0.32 ? 11% perf-profile.self.cycles-pp.statfs_by_dentry
0.36 ? 30% perf-profile.self.cycles-pp.kernfs_dop_revalidate
0.38 ? 15% perf-profile.self.cycles-pp.__x64_sys_prlimit64
0.49 ? 16% perf-profile.self.cycles-pp.strncpy_from_user
0.30 ? 16% perf-profile.self.cycles-pp.aa_get_task_label
0.52 ? 13% perf-profile.self.cycles-pp.kmem_cache_alloc
0.48 ? 9% perf-profile.self.cycles-pp.do_dentry_open
0.40 ? 15% perf-profile.self.cycles-pp._raw_read_lock
0.57 ? 14% perf-profile.self.cycles-pp.___might_sleep
0.53 ? 13% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.52 ? 16% perf-profile.self.cycles-pp.__entry_text_start
0.53 ? 14% perf-profile.self.cycles-pp.cpumask_next
0.40 ? 69% perf-profile.self.cycles-pp.set_root
0.55 ? 15% perf-profile.self.cycles-pp.apparmor_file_free_security
0.70 ? 14% perf-profile.self.cycles-pp._find_next_bit
0.64 ? 22% perf-profile.self.cycles-pp.step_into
0.75 ? 9% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.77 ? 13% perf-profile.self.cycles-pp.kmem_cache_free
0.60 ? 15% perf-profile.self.cycles-pp.apparmor_file_open
0.71 ? 13% perf-profile.self.cycles-pp.link_path_walk
0.72 ? 14% perf-profile.self.cycles-pp.apparmor_file_alloc_security
0.89 ? 14% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.88 ? 10% perf-profile.self.cycles-pp.__legitimize_mnt
0.79 ? 22% perf-profile.self.cycles-pp.dput
0.79 ? 15% perf-profile.self.cycles-pp.generic_permission
1.00 ? 17% perf-profile.self.cycles-pp.__d_lookup
2.01 ? 12% perf-profile.self.cycles-pp.__percpu_counter_sum
1.81 ? 11% perf-profile.self.cycles-pp.inode_permission
2.04 ? 13% perf-profile.self.cycles-pp.lockref_get
2.58 ? 12% perf-profile.self.cycles-pp.__x64_sys_getpriority
2.64 ? 4% perf-profile.self.cycles-pp.lockref_get_not_dead
3.52 ? 12% perf-profile.self.cycles-pp._raw_spin_lock
4.67 ? 11% perf-profile.self.cycles-pp.lockref_put_return
4.74 ? 10% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
4.49 ? 12% perf-profile.self.cycles-pp.up_read
4.99 ? 14% perf-profile.self.cycles-pp.down_read

stress-ng.get.ops

-------------------------------------------------+
O O O O |
O O O O O O O O O O O O O |
O O O O O |
O O O |
O |
|
|
.+.+. .+ |
.+.+ .+.+. .+.+. .+.+.+.+ +.+.+ : |
+ + + + : |
: |
+.+. .+.+. |
+ + |
-------------------------------------------------+

stress-ng.get.ops_per_sec

-------------------------------------------------+
O O O O |
O O O O O O O O O O |
O O O O O |
O O O |
O |
|
|
|
.+.+. .+.+. .+.+. .+.+.+.+. .+ |
+ + + +.+ : |
: |
: .+. |
+.+.+.+ + |
-------------------------------------------------+

estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
may affect actual performance.
Open Source Technology Center
org/hyperkitty/list/lkp@lists.01.org">https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation

Attachments:

(No filename) (81.99 kB)
config-5.13.0-rc2-00023-g9a658329cda8 (176.78 kB)
job-script (8.50 kB)
job.yaml (5.88 kB)
reproduce (548.00 B)
Download all attachments

2021-06-01 13:20:48

by Miklos Szeredi

[permalink] [raw]

Subject: Re: [REPOST PATCH v4 4/5] kernfs: use i_lock to protect concurrent inode updates

On Fri, 28 May 2021 at 08:34, Ian Kent <[email protected]> wrote:
>
> The inode operations .permission() and .getattr() use the kernfs node
> write lock but all that's needed is to keep the rb tree stable while
> updating the inode attributes as well as protecting the update itself
> against concurrent changes.
>
> And .permission() is called frequently during path walks and can cause
> quite a bit of contention between kernfs node operations and path
> walks when the number of concurrent walks is high.
>
> To change kernfs_iop_getattr() and kernfs_iop_permission() to take
> the rw sem read lock instead of the write lock an additional lock is
> needed to protect against multiple processes concurrently updating
> the inode attributes and link count in kernfs_refresh_inode().
>
> The inode i_lock seems like the sensible thing to use to protect these
> inode attribute updates so use it in kernfs_refresh_inode().
>
> Signed-off-by: Ian Kent <[email protected]>
> ---
> fs/kernfs/inode.c | 10 ++++++----
> fs/kernfs/mount.c | 4 ++--
> 2 files changed, 8 insertions(+), 6 deletions(-)
>
> diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
> index 3b01e9e61f14e..6728ecd81eb37 100644
> --- a/fs/kernfs/inode.c
> +++ b/fs/kernfs/inode.c
> @@ -172,6 +172,7 @@ static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
> {
> struct kernfs_iattrs *attrs = kn->iattr;
>
> + spin_lock(&inode->i_lock);
> inode->i_mode = kn->mode;
> if (attrs)
> /*
> @@ -182,6 +183,7 @@ static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
>
> if (kernfs_type(kn) == KERNFS_DIR)
> set_nlink(inode, kn->dir.subdirs + 2);
> + spin_unlock(&inode->i_lock);
> }
>
> int kernfs_iop_getattr(struct user_namespace *mnt_userns,
> @@ -191,9 +193,9 @@ int kernfs_iop_getattr(struct user_namespace *mnt_userns,
> struct inode *inode = d_inode(path->dentry);
> struct kernfs_node *kn = inode->i_private;
>
> - down_write(&kernfs_rwsem);
> + down_read(&kernfs_rwsem);
> kernfs_refresh_inode(kn, inode);
> - up_write(&kernfs_rwsem);
> + up_read(&kernfs_rwsem);
>
> generic_fillattr(&init_user_ns, inode, stat);
> return 0;
> @@ -284,9 +286,9 @@ int kernfs_iop_permission(struct user_namespace *mnt_userns,
>
> kn = inode->i_private;
>
> - down_write(&kernfs_rwsem);
> + down_read(&kernfs_rwsem);
> kernfs_refresh_inode(kn, inode);
> - up_write(&kernfs_rwsem);
> + up_read(&kernfs_rwsem);
>
> return generic_permission(&init_user_ns, inode, mask);
> }
> diff --git a/fs/kernfs/mount.c b/fs/kernfs/mount.c
> index baa4155ba2edf..f2f909d09f522 100644
> --- a/fs/kernfs/mount.c
> +++ b/fs/kernfs/mount.c
> @@ -255,9 +255,9 @@ static int kernfs_fill_super(struct super_block *sb, struct kernfs_fs_context *k
> sb->s_shrink.seeks = 0;
>
> /* get root inode, initialize and unlock it */
> - down_write(&kernfs_rwsem);
> + down_read(&kernfs_rwsem);
> inode = kernfs_get_inode(sb, info->root->kn);
> - up_write(&kernfs_rwsem);
> + up_read(&kernfs_rwsem);
> if (!inode) {
> pr_debug("kernfs: could not get root inode\n");
> return -ENOMEM;
>

This last hunk is not mentioned in the patch header. Why is this needed?

Otherwise looks good.

Thanks,
Miklos

2021-06-02 06:43:02

by Ian Kent

[permalink] [raw]

Subject: Re: [REPOST PATCH v4 4/5] kernfs: use i_lock to protect concurrent inode updates

On Tue, 2021-06-01 at 15:18 +0200, Miklos Szeredi wrote:
> On Fri, 28 May 2021 at 08:34, Ian Kent <[email protected]> wrote:
> >
> > The inode operations .permission() and .getattr() use the kernfs
> > node
> > write lock but all that's needed is to keep the rb tree stable
> > while
> > updating the inode attributes as well as protecting the update
> > itself
> > against concurrent changes.
> >
> > And .permission() is called frequently during path walks and can
> > cause
> > quite a bit of contention between kernfs node operations and path
> > walks when the number of concurrent walks is high.
> >
> > To change kernfs_iop_getattr() and kernfs_iop_permission() to take
> > the rw sem read lock instead of the write lock an additional lock
> > is
> > needed to protect against multiple processes concurrently updating
> > the inode attributes and link count in kernfs_refresh_inode().
> >
> > The inode i_lock seems like the sensible thing to use to protect
> > these
> > inode attribute updates so use it in kernfs_refresh_inode().
> >
> > Signed-off-by: Ian Kent <[email protected]>
> > ---
> > fs/kernfs/inode.c |   10 ++++++----
> > fs/kernfs/mount.c |    4 ++--
> > 2 files changed, 8 insertions(+), 6 deletions(-)
> >
> > diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
> > index 3b01e9e61f14e..6728ecd81eb37 100644
> > --- a/fs/kernfs/inode.c
> > +++ b/fs/kernfs/inode.c
> > @@ -172,6 +172,7 @@ static void kernfs_refresh_inode(struct
> > kernfs_node *kn, struct inode *inode)
> > {
> >         struct kernfs_iattrs *attrs = kn->iattr;
> >
> > +       spin_lock(&inode->i_lock);
> >         inode->i_mode = kn->mode;
> >         if (attrs)
> >                 /*
> > @@ -182,6 +183,7 @@ static void kernfs_refresh_inode(struct
> > kernfs_node *kn, struct inode *inode)
> >
> >         if (kernfs_type(kn) == KERNFS_DIR)
> >                 set_nlink(inode, kn->dir.subdirs + 2);
> > +       spin_unlock(&inode->i_lock);
> > }
> >
> > int kernfs_iop_getattr(struct user_namespace *mnt_userns,
> > @@ -191,9 +193,9 @@ int kernfs_iop_getattr(struct user_namespace
> > *mnt_userns,
> >         struct inode *inode = d_inode(path->dentry);
> >         struct kernfs_node *kn = inode->i_private;
> >
> > -       down_write(&kernfs_rwsem);
> > +       down_read(&kernfs_rwsem);
> >         kernfs_refresh_inode(kn, inode);
> > -       up_write(&kernfs_rwsem);
> > +       up_read(&kernfs_rwsem);
> >
> >         generic_fillattr(&init_user_ns, inode, stat);
> >         return 0;
> > @@ -284,9 +286,9 @@ int kernfs_iop_permission(struct user_namespace
> > *mnt_userns,
> >
> >         kn = inode->i_private;
> >
> > -       down_write(&kernfs_rwsem);
> > +       down_read(&kernfs_rwsem);
> >         kernfs_refresh_inode(kn, inode);
> > -       up_write(&kernfs_rwsem);
> > +       up_read(&kernfs_rwsem);
> >
> >         return generic_permission(&init_user_ns, inode, mask);
> > }
> > diff --git a/fs/kernfs/mount.c b/fs/kernfs/mount.c
> > index baa4155ba2edf..f2f909d09f522 100644
> > --- a/fs/kernfs/mount.c
> > +++ b/fs/kernfs/mount.c
> > @@ -255,9 +255,9 @@ static int kernfs_fill_super(struct super_block
> > *sb, struct kernfs_fs_context *k
> >         sb->s_shrink.seeks = 0;
> >
> >         /* get root inode, initialize and unlock it */
> > -       down_write(&kernfs_rwsem);
> > +       down_read(&kernfs_rwsem);
> >         inode = kernfs_get_inode(sb, info->root->kn);
> > -       up_write(&kernfs_rwsem);
> > +       up_read(&kernfs_rwsem);
> >         if (!inode) {
> >                 pr_debug("kernfs: could not get root inode\n");
> >                 return -ENOMEM;
> >
>
> This last hunk is not mentioned in the patch header. Why is this
> needed?

Yes, that's right.

The lock is needed to keep the node rb tree stable.

kernfs_get_inode() calls kernfs_refresh_inode() indirectly so
since the i_lock is probably not needed here this hunk could
just as well have gone into the rwsem change but because of
that kernfs_refresh_inode() call it also makes sense to put
it here.

I'd prefer to keep it here and clearly what's going on isn't
as obvious as I thought so I can add this reasoning to the
description if you still think it's worth while?

>
> Otherwise looks good.
>
> Thanks,
> Miklos