2021-02-22 10:00:27

by Alexey Gladkov

[permalink] [raw]

Subject: [PATCH v7 0/7] Count rlimits in each user namespace

---

Preface
-------
These patches are for binding the rlimit counters to a user in user namespace.
This patch set can be applied on top of:

git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git v5.11

Problem
-------
The RLIMIT_NPROC, RLIMIT_MEMLOCK, RLIMIT_SIGPENDING, RLIMIT_MSGQUEUE rlimits
implementation places the counters in user_struct [1]. These limits are global
between processes and persists for the lifetime of the process, even if
processes are in different user namespaces.

To illustrate the impact of rlimits, let's say there is a program that does not
fork. Some service-A wants to run this program as user X in multiple containers.
Since the program never fork the service wants to set RLIMIT_NPROC=1.

service-A
\- program (uid=1000, container1, rlimit_nproc=1)
\- program (uid=1000, container2, rlimit_nproc=1)

The service-A sets RLIMIT_NPROC=1 and runs the program in container1. When the
service-A tries to run a program with RLIMIT_NPROC=1 in container2 it fails
since user X already has one running process.

The problem is not that the limit from container1 affects container2. The
problem is that limit is verified against the global counter that reflects
the number of processes in all containers.

This problem can be worked around by using different users for each container
but in this case we face a different problem of uid mapping when transferring
files from one container to another.

Eric W. Biederman mentioned this issue [2][3].

Introduced changes
------------------
To address the problem, we bind rlimit counters to user namespace. Each counter
reflects the number of processes in a given uid in a given user namespace. The
result is a tree of rlimit counters with the biggest value at the root (aka
init_user_ns). The limit is considered exceeded if it's exceeded up in the tree.

[1] https://lore.kernel.org/containers/[email protected]/
[2] https://lists.linuxfoundation.org/pipermail/containers/2020-August/042096.html
[3] https://lists.linuxfoundation.org/pipermail/containers/2020-October/042524.html

Changelog
---------
v7:
* Fixed issues found by lkp-tests project in the patch that Reimplements
RLIMIT_MEMLOCK on top of ucounts.

v6:
* Fixed issues found by lkp-tests project.
* Rebased onto v5.11.

v5:
* Split the first commit into two commits: change ucounts.count type to atomic_long_t
and add ucounts to cred. These commits were merged by mistake during the rebase.
* The __get_ucounts() renamed to alloc_ucounts().
* The cred.ucounts update has been moved from commit_creds() as it did not allow
to handle errors.
* Added error handling of set_cred_ucounts().

v4:
* Reverted the type change of ucounts.count to refcount_t.
* Fixed typo in the kernel/cred.c

v3:
* Added get_ucounts() function to increase the reference count. The existing
get_counts() function renamed to __get_ucounts().
* The type of ucounts.count changed from atomic_t to refcount_t.
* Dropped 'const' from set_cred_ucounts() arguments.
* Fixed a bug with freeing the cred structure after calling cred_alloc_blank().
* Commit messages have been updated.
* Added selftest.

v2:
* RLIMIT_MEMLOCK, RLIMIT_SIGPENDING and RLIMIT_MSGQUEUE are migrated to ucounts.
* Added ucounts for pair uid and user namespace into cred.
* Added the ability to increase ucount by more than 1.

v1:
* After discussion with Eric W. Biederman, I increased the size of ucounts to
atomic_long_t.
* Added ucount_max to avoid the fork bomb.

--

Alexey Gladkov (7):
Increase size of ucounts to atomic_long_t
Add a reference to ucounts for each cred
Reimplement RLIMIT_NPROC on top of ucounts
Reimplement RLIMIT_MSGQUEUE on top of ucounts
Reimplement RLIMIT_SIGPENDING on top of ucounts
Reimplement RLIMIT_MEMLOCK on top of ucounts
kselftests: Add test to check for rlimit changes in different user
namespaces

fs/exec.c | 6 +-
fs/hugetlbfs/inode.c | 16 +-
fs/io-wq.c | 22 ++-
fs/io-wq.h | 2 +-
fs/io_uring.c | 2 +-
fs/proc/array.c | 2 +-
include/linux/cred.h | 4 +
include/linux/hugetlb.h | 4 +-
include/linux/mm.h | 4 +-
include/linux/sched/user.h | 7 -
include/linux/shmem_fs.h | 2 +-
include/linux/signal_types.h | 4 +-
include/linux/user_namespace.h | 24 ++-
ipc/mqueue.c | 41 ++---
ipc/shm.c | 26 +--
kernel/cred.c | 50 +++++-
kernel/exit.c | 2 +-
kernel/fork.c | 18 +-
kernel/signal.c | 57 +++----
kernel/sys.c | 14 +-
kernel/ucount.c | 120 +++++++++++--
kernel/user.c | 3 -
kernel/user_namespace.c | 9 +-
mm/memfd.c | 4 +-
mm/mlock.c | 20 ++-
mm/mmap.c | 4 +-
mm/shmem.c | 8 +-
tools/testing/selftests/Makefile | 1 +
tools/testing/selftests/rlimits/.gitignore | 2 +
tools/testing/selftests/rlimits/Makefile | 6 +
tools/testing/selftests/rlimits/config | 1 +
.../selftests/rlimits/rlimits-per-userns.c | 161 ++++++++++++++++++
32 files changed, 502 insertions(+), 144 deletions(-)
create mode 100644 tools/testing/selftests/rlimits/.gitignore
create mode 100644 tools/testing/selftests/rlimits/Makefile
create mode 100644 tools/testing/selftests/rlimits/config
create mode 100644 tools/testing/selftests/rlimits/rlimits-per-userns.c

--
2.29.2

2021-02-22 10:01:44

by Alexey Gladkov

[permalink] [raw]

Subject: [PATCH v7 4/7] Reimplement RLIMIT_MSGQUEUE on top of ucounts

---

The rlimit counter is tied to uid in the user_namespace. This allows
rlimit values to be specified in userns even if they are already
globally exceeded by the user. However, the value of the previous
user_namespaces cannot be exceeded.

Signed-off-by: Alexey Gladkov <[email protected]>
---
include/linux/sched/user.h | 4 ----
include/linux/user_namespace.h | 1 +
ipc/mqueue.c | 41 ++++++++++++++++++----------------
kernel/fork.c | 1 +
kernel/ucount.c | 1 +
kernel/user_namespace.c | 1 +
6 files changed, 26 insertions(+), 23 deletions(-)

diff --git a/include/linux/sched/user.h b/include/linux/sched/user.h
index d33d867ad6c1..8a34446681aa 100644
--- a/include/linux/sched/user.h
+++ b/include/linux/sched/user.h
@@ -18,10 +18,6 @@ struct user_struct {
#endif
#ifdef CONFIG_EPOLL
atomic_long_t epoll_watches; /* The number of file descriptors currently watched */
-#endif
-#ifdef CONFIG_POSIX_MQUEUE
- /* protected by mq_lock */
- unsigned long mq_bytes; /* How many bytes can be allocated to mqueue? */
#endif
unsigned long locked_shm; /* How many pages of mlocked shm ? */
unsigned long unix_inflight; /* How many files in flight in unix sockets */
diff --git a/include/linux/user_namespace.h b/include/linux/user_namespace.h
index 0a27cd049404..52453143fe23 100644
--- a/include/linux/user_namespace.h
+++ b/include/linux/user_namespace.h
@@ -51,6 +51,7 @@ enum ucount_type {
UCOUNT_INOTIFY_WATCHES,
#endif
UCOUNT_RLIMIT_NPROC,
+ UCOUNT_RLIMIT_MSGQUEUE,
UCOUNT_COUNTS,
};

diff --git a/ipc/mqueue.c b/ipc/mqueue.c
index beff0cfcd1e8..75dba8780c80 100644
--- a/ipc/mqueue.c
+++ b/ipc/mqueue.c
@@ -144,7 +144,7 @@ struct mqueue_inode_info {
struct pid *notify_owner;
u32 notify_self_exec_id;
struct user_namespace *notify_user_ns;
- struct user_struct *user; /* user who created, for accounting */
+ struct ucounts *ucounts; /* user who created, for accounting */
struct sock *notify_sock;
struct sk_buff *notify_cookie;

@@ -292,7 +292,6 @@ static struct inode *mqueue_get_inode(struct super_block *sb,
struct ipc_namespace *ipc_ns, umode_t mode,
struct mq_attr *attr)
{
- struct user_struct *u = current_user();
struct inode *inode;
int ret = -ENOMEM;

@@ -321,7 +320,7 @@ static struct inode *mqueue_get_inode(struct super_block *sb,
info->notify_owner = NULL;
info->notify_user_ns = NULL;
info->qsize = 0;
- info->user = NULL; /* set when all is ok */
+ info->ucounts = NULL; /* set when all is ok */
info->msg_tree = RB_ROOT;
info->msg_tree_rightmost = NULL;
info->node_cache = NULL;
@@ -371,19 +370,24 @@ static struct inode *mqueue_get_inode(struct super_block *sb,
if (mq_bytes + mq_treesize < mq_bytes)
goto out_inode;
mq_bytes += mq_treesize;
- spin_lock(&mq_lock);
- if (u->mq_bytes + mq_bytes < u->mq_bytes ||
- u->mq_bytes + mq_bytes > rlimit(RLIMIT_MSGQUEUE)) {
+ info->ucounts = get_ucounts(current_ucounts());
+ if (info->ucounts) {
+ bool overlimit;
+
+ spin_lock(&mq_lock);
+ overlimit = inc_rlimit_ucounts_and_test(info->ucounts, UCOUNT_RLIMIT_MSGQUEUE,
+ mq_bytes, rlimit(RLIMIT_MSGQUEUE));
+ if (overlimit) {
+ dec_rlimit_ucounts(info->ucounts, UCOUNT_RLIMIT_MSGQUEUE, mq_bytes);
+ spin_unlock(&mq_lock);
+ put_ucounts(info->ucounts);
+ info->ucounts = NULL;
+ /* mqueue_evict_inode() releases info->messages */
+ ret = -EMFILE;
+ goto out_inode;
+ }
spin_unlock(&mq_lock);
- /* mqueue_evict_inode() releases info->messages */
- ret = -EMFILE;
- goto out_inode;
}
- u->mq_bytes += mq_bytes;
- spin_unlock(&mq_lock);
-
- /* all is ok */
- info->user = get_uid(u);
} else if (S_ISDIR(mode)) {
inc_nlink(inode);
/* Some things misbehave if size == 0 on a directory */
@@ -497,7 +501,6 @@ static void mqueue_free_inode(struct inode *inode)
static void mqueue_evict_inode(struct inode *inode)
{
struct mqueue_inode_info *info;
- struct user_struct *user;
struct ipc_namespace *ipc_ns;
struct msg_msg *msg, *nmsg;
LIST_HEAD(tmp_msg);
@@ -520,8 +523,7 @@ static void mqueue_evict_inode(struct inode *inode)
free_msg(msg);
}

- user = info->user;
- if (user) {
+ if (info->ucounts) {
unsigned long mq_bytes, mq_treesize;

/* Total amount of bytes accounted for the mqueue */
@@ -533,7 +535,7 @@ static void mqueue_evict_inode(struct inode *inode)
info->attr.mq_msgsize);

spin_lock(&mq_lock);
- user->mq_bytes -= mq_bytes;
+ dec_rlimit_ucounts(info->ucounts, UCOUNT_RLIMIT_MSGQUEUE, mq_bytes);
/*
* get_ns_from_inode() ensures that the
* (ipc_ns = sb->s_fs_info) is either a valid ipc_ns
@@ -543,7 +545,8 @@ static void mqueue_evict_inode(struct inode *inode)
if (ipc_ns)
ipc_ns->mq_queues_count--;
spin_unlock(&mq_lock);
- free_uid(user);
+ put_ucounts(info->ucounts);
+ info->ucounts = NULL;
}
if (ipc_ns)
put_ipc_ns(ipc_ns);
diff --git a/kernel/fork.c b/kernel/fork.c
index 812b023ecdce..0a939332efcc 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -823,6 +823,7 @@ void __init fork_init(void)
init_user_ns.ucount_max[i] = max_threads/2;

init_user_ns.ucount_max[UCOUNT_RLIMIT_NPROC] = task_rlimit(&init_task, RLIMIT_NPROC);
+ init_user_ns.ucount_max[UCOUNT_RLIMIT_MSGQUEUE] = task_rlimit(&init_task, RLIMIT_MSGQUEUE);

#ifdef CONFIG_VMAP_STACK
cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "fork:vm_stack_cache",
diff --git a/kernel/ucount.c b/kernel/ucount.c
index 2f42d2ee6e27..6fb2ebdef0bc 100644
--- a/kernel/ucount.c
+++ b/kernel/ucount.c
@@ -81,6 +81,7 @@ static struct ctl_table user_table[] = {
UCOUNT_ENTRY("max_inotify_instances"),
UCOUNT_ENTRY("max_inotify_watches"),
#endif
+ { },
{ },
{ }
};
diff --git a/kernel/user_namespace.c b/kernel/user_namespace.c
index 2434b13b02e5..cc90d5203acf 100644
--- a/kernel/user_namespace.c
+++ b/kernel/user_namespace.c
@@ -122,6 +122,7 @@ int create_user_ns(struct cred *new)
ns->ucount_max[i] = INT_MAX;
}
ns->ucount_max[UCOUNT_RLIMIT_NPROC] = rlimit(RLIMIT_NPROC);
+ ns->ucount_max[UCOUNT_RLIMIT_MSGQUEUE] = rlimit(RLIMIT_MSGQUEUE);
ns->ucounts = ucounts;

/* Inherit USERNS_SETGROUPS_ALLOWED from our parent */
--
2.29.2

2021-02-22 10:02:03

by Alexey Gladkov

[permalink] [raw]

Subject: [PATCH v7 3/7] Reimplement RLIMIT_NPROC on top of ucounts

---

The rlimit counter is tied to uid in the user_namespace. This allows
rlimit values to be specified in userns even if they are already
globally exceeded by the user. However, the value of the previous
user_namespaces cannot be exceeded.

To illustrate the impact of rlimits, let's say there is a program that
does not fork. Some service-A wants to run this program as user X in
multiple containers. Since the program never fork the service wants to
set RLIMIT_NPROC=1.

service-A
\- program (uid=1000, container1, rlimit_nproc=1)
\- program (uid=1000, container2, rlimit_nproc=1)

The service-A sets RLIMIT_NPROC=1 and runs the program in container1.
When the service-A tries to run a program with RLIMIT_NPROC=1 in
container2 it fails since user X already has one running process.

We cannot use existing inc_ucounts / dec_ucounts because they do not
allow us to exceed the maximum for the counter. Some rlimits can be
overlimited by root or if the user has the appropriate capability.

Signed-off-by: Alexey Gladkov <[email protected]>
---
fs/exec.c | 2 +-
fs/io-wq.c | 22 ++++++------
fs/io-wq.h | 2 +-
fs/io_uring.c | 2 +-
include/linux/cred.h | 2 ++
include/linux/sched/user.h | 1 -
include/linux/user_namespace.h | 13 ++++++++
kernel/cred.c | 10 +++---
kernel/exit.c | 2 +-
kernel/fork.c | 9 ++---
kernel/sys.c | 2 +-
kernel/ucount.c | 61 ++++++++++++++++++++++++++++++++++
kernel/user.c | 1 -
kernel/user_namespace.c | 3 +-
14 files changed, 103 insertions(+), 29 deletions(-)

diff --git a/fs/exec.c b/fs/exec.c
index 0371a3400be5..e6d7f186f33c 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1874,7 +1874,7 @@ static int do_execveat_common(int fd, struct filename *filename,
* whether NPROC limit is still exceeded.
*/
if ((current->flags & PF_NPROC_EXCEEDED) &&
- atomic_read(&current_user()->processes) > rlimit(RLIMIT_NPROC)) {
+ is_ucounts_overlimit(current_ucounts(), UCOUNT_RLIMIT_NPROC, rlimit(RLIMIT_NPROC))) {
retval = -EAGAIN;
goto out_ret;
}
diff --git a/fs/io-wq.c b/fs/io-wq.c
index a564f36e260c..5b6940c90c61 100644
--- a/fs/io-wq.c
+++ b/fs/io-wq.c
@@ -20,6 +20,7 @@
#include <linux/blk-cgroup.h>
#include <linux/audit.h>
#include <linux/cpu.h>
+#include <linux/user_namespace.h>

#include "../kernel/sched/sched.h"
#include "io-wq.h"
@@ -120,7 +121,7 @@ struct io_wq {
io_wq_work_fn *do_work;

struct task_struct *manager;
- struct user_struct *user;
+ const struct cred *cred;
refcount_t refs;
struct completion done;

@@ -234,7 +235,7 @@ static void io_worker_exit(struct io_worker *worker)
if (worker->flags & IO_WORKER_F_RUNNING)
atomic_dec(&acct->nr_running);
if (!(worker->flags & IO_WORKER_F_BOUND))
- atomic_dec(&wqe->wq->user->processes);
+ dec_rlimit_ucounts(wqe->wq->cred->ucounts, UCOUNT_RLIMIT_NPROC, 1);
worker->flags = 0;
preempt_enable();

@@ -364,15 +365,15 @@ static void __io_worker_busy(struct io_wqe *wqe, struct io_worker *worker,
worker->flags |= IO_WORKER_F_BOUND;
wqe->acct[IO_WQ_ACCT_UNBOUND].nr_workers--;
wqe->acct[IO_WQ_ACCT_BOUND].nr_workers++;
- atomic_dec(&wqe->wq->user->processes);
+ dec_rlimit_ucounts(wqe->wq->cred->ucounts, UCOUNT_RLIMIT_NPROC, 1);
} else {
worker->flags &= ~IO_WORKER_F_BOUND;
wqe->acct[IO_WQ_ACCT_UNBOUND].nr_workers++;
wqe->acct[IO_WQ_ACCT_BOUND].nr_workers--;
- atomic_inc(&wqe->wq->user->processes);
+ inc_rlimit_ucounts(wqe->wq->cred->ucounts, UCOUNT_RLIMIT_NPROC, 1);
}
io_wqe_inc_running(wqe, worker);
- }
+ }
}

/*
@@ -707,7 +708,7 @@ static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index)
raw_spin_unlock_irq(&wqe->lock);

if (index == IO_WQ_ACCT_UNBOUND)
- atomic_inc(&wq->user->processes);
+ inc_rlimit_ucounts(wq->cred->ucounts, UCOUNT_RLIMIT_NPROC, 1);

refcount_inc(&wq->refs);
wake_up_process(worker->task);
@@ -838,7 +839,7 @@ static bool io_wq_can_queue(struct io_wqe *wqe, struct io_wqe_acct *acct,
if (free_worker)
return true;

- if (atomic_read(&wqe->wq->user->processes) >= acct->max_workers &&
+ if (is_ucounts_overlimit(wqe->wq->cred->ucounts, UCOUNT_RLIMIT_NPROC, acct->max_workers) &&
!(capable(CAP_SYS_RESOURCE) || capable(CAP_SYS_ADMIN)))
return false;

@@ -1074,7 +1075,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)
wq->do_work = data->do_work;

/* caller must already hold a reference to this */
- wq->user = data->user;
+ wq->cred = data->cred;

ret = -ENOMEM;
for_each_node(node) {
@@ -1090,10 +1091,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)
wqe->node = alloc_node;
wqe->acct[IO_WQ_ACCT_BOUND].max_workers = bounded;
atomic_set(&wqe->acct[IO_WQ_ACCT_BOUND].nr_running, 0);
- if (wq->user) {
- wqe->acct[IO_WQ_ACCT_UNBOUND].max_workers =
- task_rlimit(current, RLIMIT_NPROC);
- }
+ wqe->acct[IO_WQ_ACCT_UNBOUND].max_workers = task_rlimit(current, RLIMIT_NPROC);
atomic_set(&wqe->acct[IO_WQ_ACCT_UNBOUND].nr_running, 0);
wqe->wq = wq;
raw_spin_lock_init(&wqe->lock);
diff --git a/fs/io-wq.h b/fs/io-wq.h
index b158f8addcf3..4130e247c556 100644
--- a/fs/io-wq.h
+++ b/fs/io-wq.h
@@ -111,7 +111,7 @@ typedef void (free_work_fn)(struct io_wq_work *);
typedef struct io_wq_work *(io_wq_work_fn)(struct io_wq_work *);

struct io_wq_data {
- struct user_struct *user;
+ const struct cred *cred;

io_wq_work_fn *do_work;
free_work_fn *free_work;
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 931671082e61..389998f39843 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -8084,7 +8084,7 @@ static int io_init_wq_offload(struct io_ring_ctx *ctx,
unsigned int concurrency;
int ret = 0;

- data.user = ctx->user;
+ data.cred = ctx->creds;
data.free_work = io_free_work;
data.do_work = io_wq_submit_work;

diff --git a/include/linux/cred.h b/include/linux/cred.h
index ad160e5fe5c6..8025fe48198f 100644
--- a/include/linux/cred.h
+++ b/include/linux/cred.h
@@ -372,6 +372,7 @@ static inline void put_cred(const struct cred *_cred)

#define task_uid(task) (task_cred_xxx((task), uid))
#define task_euid(task) (task_cred_xxx((task), euid))
+#define task_ucounts(task) (task_cred_xxx((task), ucounts))

#define current_cred_xxx(xxx) \
({ \
@@ -388,6 +389,7 @@ static inline void put_cred(const struct cred *_cred)
#define current_fsgid() (current_cred_xxx(fsgid))
#define current_cap() (current_cred_xxx(cap_effective))
#define current_user() (current_cred_xxx(user))
+#define current_ucounts() (current_cred_xxx(ucounts))

extern struct user_namespace init_user_ns;
#ifdef CONFIG_USER_NS
diff --git a/include/linux/sched/user.h b/include/linux/sched/user.h
index a8ec3b6093fc..d33d867ad6c1 100644
--- a/include/linux/sched/user.h
+++ b/include/linux/sched/user.h
@@ -12,7 +12,6 @@
*/
struct user_struct {
refcount_t __count; /* reference count */
- atomic_t processes; /* How many processes does this user have? */
atomic_t sigpending; /* How many pending signals does this user have? */
#ifdef CONFIG_FANOTIFY
atomic_t fanotify_listeners;
diff --git a/include/linux/user_namespace.h b/include/linux/user_namespace.h
index f71b5a4a3e74..0a27cd049404 100644
--- a/include/linux/user_namespace.h
+++ b/include/linux/user_namespace.h
@@ -50,9 +50,12 @@ enum ucount_type {
UCOUNT_INOTIFY_INSTANCES,
UCOUNT_INOTIFY_WATCHES,
#endif
+ UCOUNT_RLIMIT_NPROC,
UCOUNT_COUNTS,
};

+#define MAX_PER_NAMESPACE_UCOUNTS UCOUNT_RLIMIT_NPROC
+
struct user_namespace {
struct uid_gid_map uid_map;
struct uid_gid_map gid_map;
@@ -107,6 +110,16 @@ struct ucounts *alloc_ucounts(struct user_namespace *ns, kuid_t uid);
struct ucounts *get_ucounts(struct ucounts *ucounts);
void put_ucounts(struct ucounts *ucounts);

+static inline long get_ucounts_value(struct ucounts *ucounts, enum ucount_type type)
+{
+ return atomic_long_read(&ucounts->ucount[type]);
+}
+
+bool inc_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
+bool inc_rlimit_ucounts_and_test(struct ucounts *ucounts, enum ucount_type type, long v, long max);
+void dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v);
+bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, long max);
+
#ifdef CONFIG_USER_NS

static inline struct user_namespace *get_user_ns(struct user_namespace *ns)
diff --git a/kernel/cred.c b/kernel/cred.c
index 58a8a9e24347..dcfa30b337c5 100644
--- a/kernel/cred.c
+++ b/kernel/cred.c
@@ -360,7 +360,7 @@ int copy_creds(struct task_struct *p, unsigned long clone_flags)
kdebug("share_creds(%p{%d,%d})",
p->cred, atomic_read(&p->cred->usage),
read_cred_subscribers(p->cred));
- atomic_inc(&p->cred->user->processes);
+ inc_rlimit_ucounts(task_ucounts(p), UCOUNT_RLIMIT_NPROC, 1);
return 0;
}

@@ -395,8 +395,8 @@ int copy_creds(struct task_struct *p, unsigned long clone_flags)
}
#endif

- atomic_inc(&new->user->processes);
p->cred = p->real_cred = get_cred(new);
+ inc_rlimit_ucounts(task_ucounts(p), UCOUNT_RLIMIT_NPROC, 1);
alter_cred_subscribers(new, 2);
validate_creds(new);
return 0;
@@ -496,12 +496,12 @@ int commit_creds(struct cred *new)
* in set_user().
*/
alter_cred_subscribers(new, 2);
- if (new->user != old->user)
- atomic_inc(&new->user->processes);
+ if (new->user != old->user || new->user_ns != old->user_ns)
+ inc_rlimit_ucounts(new->ucounts, UCOUNT_RLIMIT_NPROC, 1);
rcu_assign_pointer(task->real_cred, new);
rcu_assign_pointer(task->cred, new);
if (new->user != old->user)
- atomic_dec(&old->user->processes);
+ dec_rlimit_ucounts(old->ucounts, UCOUNT_RLIMIT_NPROC, 1);
alter_cred_subscribers(old, -2);

/* send notifications */
diff --git a/kernel/exit.c b/kernel/exit.c
index 04029e35e69a..61c0fe902b50 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -188,7 +188,7 @@ void release_task(struct task_struct *p)
/* don't need to get the RCU readlock here - the process is dead and
* can't be modifying its own credentials. But shut RCU-lockdep up */
rcu_read_lock();
- atomic_dec(&__task_cred(p)->user->processes);
+ dec_rlimit_ucounts(task_ucounts(p), UCOUNT_RLIMIT_NPROC, 1);
rcu_read_unlock();

cgroup_release(p);
diff --git a/kernel/fork.c b/kernel/fork.c
index 40a5da7d3d70..812b023ecdce 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -819,9 +819,11 @@ void __init fork_init(void)
init_task.signal->rlim[RLIMIT_SIGPENDING] =
init_task.signal->rlim[RLIMIT_NPROC];

- for (i = 0; i < UCOUNT_COUNTS; i++)
+ for (i = 0; i < MAX_PER_NAMESPACE_UCOUNTS; i++)
init_user_ns.ucount_max[i] = max_threads/2;

+ init_user_ns.ucount_max[UCOUNT_RLIMIT_NPROC] = task_rlimit(&init_task, RLIMIT_NPROC);
+
#ifdef CONFIG_VMAP_STACK
cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "fork:vm_stack_cache",
NULL, free_vm_stack_cache);
@@ -1962,8 +1964,7 @@ static __latent_entropy struct task_struct *copy_process(
DEBUG_LOCKS_WARN_ON(!p->softirqs_enabled);
#endif
retval = -EAGAIN;
- if (atomic_read(&p->real_cred->user->processes) >=
- task_rlimit(p, RLIMIT_NPROC)) {
+ if (is_ucounts_overlimit(task_ucounts(p), UCOUNT_RLIMIT_NPROC, rlimit(RLIMIT_NPROC))) {
if (p->real_cred->user != INIT_USER &&
!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
goto bad_fork_free;
@@ -2366,7 +2367,7 @@ static __latent_entropy struct task_struct *copy_process(
#endif
delayacct_tsk_free(p);
bad_fork_cleanup_count:
- atomic_dec(&p->cred->user->processes);
+ dec_rlimit_ucounts(task_ucounts(p), UCOUNT_RLIMIT_NPROC, 1);
exit_creds(p);
bad_fork_free:
p->state = TASK_DEAD;
diff --git a/kernel/sys.c b/kernel/sys.c
index 373def7debe8..304b6b5e5942 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -474,7 +474,7 @@ static int set_user(struct cred *new)
* for programs doing set*uid()+execve() by harmlessly deferring the
* failure to the execve() stage.
*/
- if (atomic_read(&new_user->processes) >= rlimit(RLIMIT_NPROC) &&
+ if (is_ucounts_overlimit(new->ucounts, UCOUNT_RLIMIT_NPROC, rlimit(RLIMIT_NPROC)) &&
new_user != INIT_USER)
current->flags |= PF_NPROC_EXCEEDED;
else
diff --git a/kernel/ucount.c b/kernel/ucount.c
index 50cc1dfb7d28..2f42d2ee6e27 100644
--- a/kernel/ucount.c
+++ b/kernel/ucount.c
@@ -7,6 +7,7 @@
#include <linux/hash.h>
#include <linux/kmemleak.h>
#include <linux/user_namespace.h>
+#include <linux/security.h>

struct ucounts init_ucounts = {
.ns = &init_user_ns,
@@ -80,6 +81,7 @@ static struct ctl_table user_table[] = {
UCOUNT_ENTRY("max_inotify_instances"),
UCOUNT_ENTRY("max_inotify_watches"),
#endif
+ { },
{ }
};
#endif /* CONFIG_SYSCTL */
@@ -222,6 +224,19 @@ static inline bool atomic_long_inc_below(atomic_long_t *v, int u)
}
}

+static inline long atomic_long_dec_value(atomic_long_t *v, long n)
+{
+ long c, old;
+ c = atomic_long_read(v);
+ for (;;) {
+ old = atomic_long_cmpxchg(v, c, c - n);
+ if (likely(old == c))
+ return c;
+ c = old;
+ }
+ return c;
+}
+
struct ucounts *inc_ucount(struct user_namespace *ns, kuid_t uid,
enum ucount_type type)
{
@@ -255,6 +270,51 @@ void dec_ucount(struct ucounts *ucounts, enum ucount_type type)
put_ucounts(ucounts);
}

+bool inc_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v)
+{
+ struct ucounts *iter;
+ bool overlimit = false;
+
+ for (iter = ucounts; iter; iter = iter->ns->ucounts) {
+ long max = READ_ONCE(iter->ns->ucount_max[type]);
+ if (atomic_long_add_return(v, &iter->ucount[type]) > max)
+ overlimit = true;
+ }
+
+ return overlimit;
+}
+
+bool inc_rlimit_ucounts_and_test(struct ucounts *ucounts, enum ucount_type type,
+ long v, long max)
+{
+ bool overlimit = inc_rlimit_ucounts(ucounts, type, v);
+ if (!overlimit && get_ucounts_value(ucounts, type) > max)
+ overlimit = true;
+ return overlimit;
+}
+
+void dec_rlimit_ucounts(struct ucounts *ucounts, enum ucount_type type, long v)
+{
+ struct ucounts *iter;
+ for (iter = ucounts; iter; iter = iter->ns->ucounts) {
+ long dec = atomic_long_dec_value(&iter->ucount[type], v);
+ WARN_ON_ONCE(dec < 0);
+ }
+}
+
+bool is_ucounts_overlimit(struct ucounts *ucounts, enum ucount_type type, long max)
+{
+ struct ucounts *iter;
+ if (get_ucounts_value(ucounts, type) > max)
+ return true;
+ for (iter = ucounts; iter; iter = iter->ns->ucounts) {
+ max = READ_ONCE(iter->ns->ucount_max[type]);
+ if (get_ucounts_value(iter, type) > max)
+ return true;
+ }
+ return false;
+}
+
static __init int user_namespace_sysctl_init(void)
{
#ifdef CONFIG_SYSCTL
@@ -271,6 +331,7 @@ static __init int user_namespace_sysctl_init(void)
BUG_ON(!setup_userns_sysctls(&init_user_ns));
#endif
hlist_add_ucounts(&init_ucounts);
+ inc_rlimit_ucounts(&init_ucounts, UCOUNT_RLIMIT_NPROC, 1);
return 0;
}
subsys_initcall(user_namespace_sysctl_init);
diff --git a/kernel/user.c b/kernel/user.c
index a2478cddf536..7f5ff498207a 100644
--- a/kernel/user.c
+++ b/kernel/user.c
@@ -98,7 +98,6 @@ static DEFINE_SPINLOCK(uidhash_lock);
/* root_user.__count is 1, for init task cred */
struct user_struct root_user = {
.__count = REFCOUNT_INIT(1),
- .processes = ATOMIC_INIT(1),
.sigpending = ATOMIC_INIT(0),
.locked_shm = 0,
.uid = GLOBAL_ROOT_UID,
diff --git a/kernel/user_namespace.c b/kernel/user_namespace.c
index 516db53166ab..2434b13b02e5 100644
--- a/kernel/user_namespace.c
+++ b/kernel/user_namespace.c
@@ -118,9 +118,10 @@ int create_user_ns(struct cred *new)
ns->owner = owner;
ns->group = group;
INIT_WORK(&ns->work, free_user_ns);
- for (i = 0; i < UCOUNT_COUNTS; i++) {
+ for (i = 0; i < MAX_PER_NAMESPACE_UCOUNTS; i++) {
ns->ucount_max[i] = INT_MAX;
}
+ ns->ucount_max[UCOUNT_RLIMIT_NPROC] = rlimit(RLIMIT_NPROC);
ns->ucounts = ucounts;

/* Inherit USERNS_SETGROUPS_ALLOWED from our parent */
--
2.29.2

2021-02-22 10:03:15

by Alexey Gladkov

[permalink] [raw]

Subject: [PATCH v7 5/7] Reimplement RLIMIT_SIGPENDING on top of ucounts

---

The rlimit counter is tied to uid in the user_namespace. This allows
rlimit values to be specified in userns even if they are already
globally exceeded by the user. However, the value of the previous
user_namespaces cannot be exceeded.

Signed-off-by: Alexey Gladkov <[email protected]>
---
fs/proc/array.c | 2 +-
include/linux/sched/user.h | 1 -
include/linux/signal_types.h | 4 ++-
include/linux/user_namespace.h | 1 +
kernel/fork.c | 1 +
kernel/signal.c | 57 ++++++++++++++++------------------
kernel/ucount.c | 1 +
kernel/user.c | 1 -
kernel/user_namespace.c | 1 +
9 files changed, 34 insertions(+), 35 deletions(-)

diff --git a/fs/proc/array.c b/fs/proc/array.c
index bb87e4d89cd8..74b0ea4b7e38 100644
--- a/fs/proc/array.c
+++ b/fs/proc/array.c
@@ -284,7 +284,7 @@ static inline void task_sig(struct seq_file *m, struct task_struct *p)
collect_sigign_sigcatch(p, &ignored, &caught);
num_threads = get_nr_threads(p);
rcu_read_lock(); /* FIXME: is this correct? */
- qsize = atomic_read(&__task_cred(p)->user->sigpending);
+ qsize = get_ucounts_value(task_ucounts(p), UCOUNT_RLIMIT_SIGPENDING);
rcu_read_unlock();
qlim = task_rlimit(p, RLIMIT_SIGPENDING);
unlock_task_sighand(p, &flags);
diff --git a/include/linux/sched/user.h b/include/linux/sched/user.h
index 8a34446681aa..8ba9cec4fb99 100644
--- a/include/linux/sched/user.h
+++ b/include/linux/sched/user.h
@@ -12,7 +12,6 @@
*/
struct user_struct {
refcount_t __count; /* reference count */
- atomic_t sigpending; /* How many pending signals does this user have? */
#ifdef CONFIG_FANOTIFY
atomic_t fanotify_listeners;
#endif
diff --git a/include/linux/signal_types.h b/include/linux/signal_types.h
index 68e06c75c5b2..34cb28b8f16c 100644
--- a/include/linux/signal_types.h
+++ b/include/linux/signal_types.h
@@ -13,6 +13,8 @@ typedef struct kernel_siginfo {
__SIGINFO;
} kernel_siginfo_t;

+struct ucounts;
+
/*
* Real Time signals may be queued.
*/
@@ -21,7 +23,7 @@ struct sigqueue {
struct list_head list;
int flags;
kernel_siginfo_t info;
- struct user_struct *user;
+ struct ucounts *ucounts;
};

/* flags values. */
diff --git a/include/linux/user_namespace.h b/include/linux/user_namespace.h
index 52453143fe23..f84b68832c56 100644
--- a/include/linux/user_namespace.h
+++ b/include/linux/user_namespace.h
@@ -52,6 +52,7 @@ enum ucount_type {
#endif
UCOUNT_RLIMIT_NPROC,
UCOUNT_RLIMIT_MSGQUEUE,
+ UCOUNT_RLIMIT_SIGPENDING,
UCOUNT_COUNTS,
};

diff --git a/kernel/fork.c b/kernel/fork.c
index 0a939332efcc..99b10b9fe4b6 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -824,6 +824,7 @@ void __init fork_init(void)

init_user_ns.ucount_max[UCOUNT_RLIMIT_NPROC] = task_rlimit(&init_task, RLIMIT_NPROC);
init_user_ns.ucount_max[UCOUNT_RLIMIT_MSGQUEUE] = task_rlimit(&init_task, RLIMIT_MSGQUEUE);
+ init_user_ns.ucount_max[UCOUNT_RLIMIT_SIGPENDING] = task_rlimit(&init_task, RLIMIT_SIGPENDING);

#ifdef CONFIG_VMAP_STACK
cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "fork:vm_stack_cache",
diff --git a/kernel/signal.c b/kernel/signal.c
index 5ad8566534e7..a515e36a8a11 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -412,49 +412,44 @@ void task_join_group_stop(struct task_struct *task)
static struct sigqueue *
__sigqueue_alloc(int sig, struct task_struct *t, gfp_t flags, int override_rlimit)
{
- struct sigqueue *q = NULL;
- struct user_struct *user;
- int sigpending;
+ struct sigqueue *q = kmem_cache_alloc(sigqueue_cachep, flags);

- /*
- * Protect access to @t credentials. This can go away when all
- * callers hold rcu read lock.
- *
- * NOTE! A pending signal will hold on to the user refcount,
- * and we get/put the refcount only when the sigpending count
- * changes from/to zero.
- */
- rcu_read_lock();
- user = __task_cred(t)->user;
- sigpending = atomic_inc_return(&user->sigpending);
- if (sigpending == 1)
- get_uid(user);
- rcu_read_unlock();
+ if (likely(q != NULL)) {
+ bool overlimit;

- if (override_rlimit || likely(sigpending <= task_rlimit(t, RLIMIT_SIGPENDING))) {
- q = kmem_cache_alloc(sigqueue_cachep, flags);
- } else {
- print_dropped_signal(sig);
- }
-
- if (unlikely(q == NULL)) {
- if (atomic_dec_and_test(&user->sigpending))
- free_uid(user);
- } else {
INIT_LIST_HEAD(&q->list);
q->flags = 0;
- q->user = user;
+
+ /*
+ * Protect access to @t credentials. This can go away when all
+ * callers hold rcu read lock.
+ */
+ rcu_read_lock();
+ q->ucounts = get_ucounts(task_ucounts(t));
+ if (q->ucounts) {
+ overlimit = inc_rlimit_ucounts_and_test(q->ucounts, UCOUNT_RLIMIT_SIGPENDING,
+ 1, task_rlimit(t, RLIMIT_SIGPENDING));
+
+ if (override_rlimit || likely(!overlimit)) {
+ rcu_read_unlock();
+ return q;
+ }
+ }
+ rcu_read_unlock();
}

- return q;
+ print_dropped_signal(sig);
+ return NULL;
}

static void __sigqueue_free(struct sigqueue *q)
{
if (q->flags & SIGQUEUE_PREALLOC)
return;
- if (atomic_dec_and_test(&q->user->sigpending))
- free_uid(q->user);
+ if (q->ucounts) {
+ dec_rlimit_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING, 1);
+ put_ucounts(q->ucounts);
+ }
kmem_cache_free(sigqueue_cachep, q);
}

diff --git a/kernel/ucount.c b/kernel/ucount.c
index 6fb2ebdef0bc..2ac969fba668 100644
--- a/kernel/ucount.c
+++ b/kernel/ucount.c
@@ -81,6 +81,7 @@ static struct ctl_table user_table[] = {
UCOUNT_ENTRY("max_inotify_instances"),
UCOUNT_ENTRY("max_inotify_watches"),
#endif
+ { },
{ },
{ },
{ }
diff --git a/kernel/user.c b/kernel/user.c
index 7f5ff498207a..6737327f83be 100644
--- a/kernel/user.c
+++ b/kernel/user.c
@@ -98,7 +98,6 @@ static DEFINE_SPINLOCK(uidhash_lock);
/* root_user.__count is 1, for init task cred */
struct user_struct root_user = {
.__count = REFCOUNT_INIT(1),
- .sigpending = ATOMIC_INIT(0),
.locked_shm = 0,
.uid = GLOBAL_ROOT_UID,
.ratelimit = RATELIMIT_STATE_INIT(root_user.ratelimit, 0, 0),
diff --git a/kernel/user_namespace.c b/kernel/user_namespace.c
index cc90d5203acf..df1bed32dd48 100644
--- a/kernel/user_namespace.c
+++ b/kernel/user_namespace.c
@@ -123,6 +123,7 @@ int create_user_ns(struct cred *new)
}
ns->ucount_max[UCOUNT_RLIMIT_NPROC] = rlimit(RLIMIT_NPROC);
ns->ucount_max[UCOUNT_RLIMIT_MSGQUEUE] = rlimit(RLIMIT_MSGQUEUE);
+ ns->ucount_max[UCOUNT_RLIMIT_SIGPENDING] = rlimit(RLIMIT_SIGPENDING);
ns->ucounts = ucounts;

/* Inherit USERNS_SETGROUPS_ALLOWED from our parent */
--
2.29.2

2021-02-22 10:03:17

by Alexey Gladkov

[permalink] [raw]

Subject: [PATCH v7 2/7] Add a reference to ucounts for each cred

---

For RLIMIT_NPROC and some other rlimits the user_struct that holds the
global limit is kept alive for the lifetime of a process by keeping it
in struct cred. Adding a pointer to ucounts in the struct cred will
allow to track RLIMIT_NPROC not only for user in the system, but for
user in the user_namespace.

Updating ucounts may require memory allocation which may fail. So, we
cannot change cred.ucounts in the commit_creds() because this function
cannot fail and it should always return 0. For this reason, we modify
cred.ucounts before calling the commit_creds().

Changelog

v6:
* Fix null-ptr-deref in is_ucounts_overlimit() detected by trinity. This
error was caused by the fact that cred_alloc_blank() left the ucounts
pointer empty.

Reported-by: kernel test robot <[email protected]>
Signed-off-by: Alexey Gladkov <[email protected]>
---
fs/exec.c | 4 ++++
include/linux/cred.h | 2 ++
include/linux/user_namespace.h | 4 ++++
kernel/cred.c | 40 ++++++++++++++++++++++++++++++++++
kernel/fork.c | 6 +++++
kernel/sys.c | 12 ++++++++++
kernel/ucount.c | 40 +++++++++++++++++++++++++++++++---
kernel/user_namespace.c | 3 +++
8 files changed, 108 insertions(+), 3 deletions(-)

diff --git a/fs/exec.c b/fs/exec.c
index 5d4d52039105..0371a3400be5 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1360,6 +1360,10 @@ int begin_new_exec(struct linux_binprm * bprm)
WRITE_ONCE(me->self_exec_id, me->self_exec_id + 1);
flush_signal_handlers(me, 0);

+ retval = set_cred_ucounts(bprm->cred);
+ if (retval < 0)
+ goto out_unlock;
+
/*
* install the new credentials for this executable
*/
diff --git a/include/linux/cred.h b/include/linux/cred.h
index 18639c069263..ad160e5fe5c6 100644
--- a/include/linux/cred.h
+++ b/include/linux/cred.h
@@ -144,6 +144,7 @@ struct cred {
#endif
struct user_struct *user; /* real user ID subscription */
struct user_namespace *user_ns; /* user_ns the caps and keyrings are relative to. */
+ struct ucounts *ucounts;
struct group_info *group_info; /* supplementary groups for euid/fsgid */
/* RCU deletion */
union {
@@ -170,6 +171,7 @@ extern int set_security_override_from_ctx(struct cred *, const char *);
extern int set_create_files_as(struct cred *, struct inode *);
extern int cred_fscmp(const struct cred *, const struct cred *);
extern void __init cred_init(void);
+extern int set_cred_ucounts(struct cred *);

/*
* check for validity of credentials
diff --git a/include/linux/user_namespace.h b/include/linux/user_namespace.h
index 0bb833fd41f4..f71b5a4a3e74 100644
--- a/include/linux/user_namespace.h
+++ b/include/linux/user_namespace.h
@@ -97,11 +97,15 @@ struct ucounts {
};

extern struct user_namespace init_user_ns;
+extern struct ucounts init_ucounts;

bool setup_userns_sysctls(struct user_namespace *ns);
void retire_userns_sysctls(struct user_namespace *ns);
struct ucounts *inc_ucount(struct user_namespace *ns, kuid_t uid, enum ucount_type type);
void dec_ucount(struct ucounts *ucounts, enum ucount_type type);
+struct ucounts *alloc_ucounts(struct user_namespace *ns, kuid_t uid);
+struct ucounts *get_ucounts(struct ucounts *ucounts);
+void put_ucounts(struct ucounts *ucounts);

#ifdef CONFIG_USER_NS

diff --git a/kernel/cred.c b/kernel/cred.c
index 421b1149c651..58a8a9e24347 100644
--- a/kernel/cred.c
+++ b/kernel/cred.c
@@ -60,6 +60,7 @@ struct cred init_cred = {
.user = INIT_USER,
.user_ns = &init_user_ns,
.group_info = &init_groups,
+ .ucounts = &init_ucounts,
};

static inline void set_cred_subscribers(struct cred *cred, int n)
@@ -119,6 +120,8 @@ static void put_cred_rcu(struct rcu_head *rcu)
if (cred->group_info)
put_group_info(cred->group_info);
free_uid(cred->user);
+ if (cred->ucounts)
+ put_ucounts(cred->ucounts);
put_user_ns(cred->user_ns);
kmem_cache_free(cred_jar, cred);
}
@@ -222,6 +225,7 @@ struct cred *cred_alloc_blank(void)
#ifdef CONFIG_DEBUG_CREDENTIALS
new->magic = CRED_MAGIC;
#endif
+ new->ucounts = get_ucounts(&init_ucounts);

if (security_cred_alloc_blank(new, GFP_KERNEL_ACCOUNT) < 0)
goto error;
@@ -284,6 +288,11 @@ struct cred *prepare_creds(void)

if (security_prepare_creds(new, old, GFP_KERNEL_ACCOUNT) < 0)
goto error;
+
+ new->ucounts = get_ucounts(new->ucounts);
+ if (!new->ucounts)
+ goto error;
+
validate_creds(new);
return new;

@@ -363,6 +372,8 @@ int copy_creds(struct task_struct *p, unsigned long clone_flags)
ret = create_user_ns(new);
if (ret < 0)
goto error_put;
+ if (set_cred_ucounts(new) < 0)
+ goto error_put;
}

#ifdef CONFIG_KEYS
@@ -653,6 +664,31 @@ int cred_fscmp(const struct cred *a, const struct cred *b)
}
EXPORT_SYMBOL(cred_fscmp);

+int set_cred_ucounts(struct cred *new)
+{
+ struct task_struct *task = current;
+ const struct cred *old = task->real_cred;
+ struct ucounts *old_ucounts = new->ucounts;
+
+ if (new->user == old->user && new->user_ns == old->user_ns)
+ return 0;
+
+ /*
+ * This optimization is needed because alloc_ucounts() uses locks
+ * for table lookups.
+ */
+ if (old_ucounts && old_ucounts->ns == new->user_ns && uid_eq(old_ucounts->uid, new->euid))
+ return 0;
+
+ if (!(new->ucounts = alloc_ucounts(new->user_ns, new->euid)))
+ return -EAGAIN;
+
+ if (old_ucounts)
+ put_ucounts(old_ucounts);
+
+ return 0;
+}
+
/*
* initialise the credentials stuff
*/
@@ -719,6 +755,10 @@ struct cred *prepare_kernel_cred(struct task_struct *daemon)
if (security_prepare_creds(new, old, GFP_KERNEL_ACCOUNT) < 0)
goto error;

+ new->ucounts = get_ucounts(new->ucounts);
+ if (!new->ucounts)
+ goto error;
+
put_cred(old);
validate_creds(new);
return new;
diff --git a/kernel/fork.c b/kernel/fork.c
index d66cd1014211..40a5da7d3d70 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -2957,6 +2957,12 @@ int ksys_unshare(unsigned long unshare_flags)
if (err)
goto bad_unshare_cleanup_cred;

+ if (new_cred) {
+ err = set_cred_ucounts(new_cred);
+ if (err)
+ goto bad_unshare_cleanup_cred;
+ }
+
if (new_fs || new_fd || do_sysvsem || new_cred || new_nsproxy) {
if (do_sysvsem) {
/*
diff --git a/kernel/sys.c b/kernel/sys.c
index 51f00fe20e4d..373def7debe8 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -553,6 +553,10 @@ long __sys_setreuid(uid_t ruid, uid_t euid)
if (retval < 0)
goto error;

+ retval = set_cred_ucounts(new);
+ if (retval < 0)
+ goto error;
+
return commit_creds(new);

error:
@@ -611,6 +615,10 @@ long __sys_setuid(uid_t uid)
if (retval < 0)
goto error;

+ retval = set_cred_ucounts(new);
+ if (retval < 0)
+ goto error;
+
return commit_creds(new);

error:
@@ -686,6 +694,10 @@ long __sys_setresuid(uid_t ruid, uid_t euid, uid_t suid)
if (retval < 0)
goto error;

+ retval = set_cred_ucounts(new);
+ if (retval < 0)
+ goto error;
+
return commit_creds(new);

error:
diff --git a/kernel/ucount.c b/kernel/ucount.c
index 04c561751af1..50cc1dfb7d28 100644
--- a/kernel/ucount.c
+++ b/kernel/ucount.c
@@ -8,6 +8,12 @@
#include <linux/kmemleak.h>
#include <linux/user_namespace.h>

+struct ucounts init_ucounts = {
+ .ns = &init_user_ns,
+ .uid = GLOBAL_ROOT_UID,
+ .count = 1,
+};
+
#define UCOUNTS_HASHTABLE_BITS 10
static struct hlist_head ucounts_hashtable[(1 << UCOUNTS_HASHTABLE_BITS)];
static DEFINE_SPINLOCK(ucounts_lock);
@@ -125,7 +131,15 @@ static struct ucounts *find_ucounts(struct user_namespace *ns, kuid_t uid, struc
return NULL;
}

-static struct ucounts *get_ucounts(struct user_namespace *ns, kuid_t uid)
+static void hlist_add_ucounts(struct ucounts *ucounts)
+{
+ struct hlist_head *hashent = ucounts_hashentry(ucounts->ns, ucounts->uid);
+ spin_lock_irq(&ucounts_lock);
+ hlist_add_head(&ucounts->node, hashent);
+ spin_unlock_irq(&ucounts_lock);
+}
+
+struct ucounts *alloc_ucounts(struct user_namespace *ns, kuid_t uid)
{
struct hlist_head *hashent = ucounts_hashentry(ns, uid);
struct ucounts *ucounts, *new;
@@ -160,7 +174,26 @@ static struct ucounts *get_ucounts(struct user_namespace *ns, kuid_t uid)
return ucounts;
}

-static void put_ucounts(struct ucounts *ucounts)
+struct ucounts *get_ucounts(struct ucounts *ucounts)
+{
+ unsigned long flags;
+
+ if (!ucounts)
+ return NULL;
+
+ spin_lock_irqsave(&ucounts_lock, flags);
+ if (ucounts->count == INT_MAX) {
+ WARN_ONCE(1, "ucounts: counter has reached its maximum value");
+ ucounts = NULL;
+ } else {
+ ucounts->count += 1;
+ }
+ spin_unlock_irqrestore(&ucounts_lock, flags);
+
+ return ucounts;
+}
+
+void put_ucounts(struct ucounts *ucounts)
{
unsigned long flags;

@@ -194,7 +227,7 @@ struct ucounts *inc_ucount(struct user_namespace *ns, kuid_t uid,
{
struct ucounts *ucounts, *iter, *bad;
struct user_namespace *tns;
- ucounts = get_ucounts(ns, uid);
+ ucounts = alloc_ucounts(ns, uid);
for (iter = ucounts; iter; iter = tns->ucounts) {
long max;
tns = iter->ns;
@@ -237,6 +270,7 @@ static __init int user_namespace_sysctl_init(void)
BUG_ON(!user_header);
BUG_ON(!setup_userns_sysctls(&init_user_ns));
#endif
+ hlist_add_ucounts(&init_ucounts);
return 0;
}
subsys_initcall(user_namespace_sysctl_init);
diff --git a/kernel/user_namespace.c b/kernel/user_namespace.c
index af612945a4d0..516db53166ab 100644
--- a/kernel/user_namespace.c
+++ b/kernel/user_namespace.c
@@ -1281,6 +1281,9 @@ static int userns_install(struct nsset *nsset, struct ns_common *ns)
put_user_ns(cred->user_ns);
set_cred_user_ns(cred, get_user_ns(user_ns));

+ if (set_cred_ucounts(cred) < 0)
+ return -EINVAL;
+
return 0;
}

--
2.29.2

2021-02-22 10:04:08

by Alexey Gladkov

[permalink] [raw]

Subject: [PATCH v7 7/7] kselftests: Add test to check for rlimit changes in different user namespaces

---

The testcase runs few instances of the program with RLIMIT_NPROC=1 from
user uid=60000, in different user namespaces.

Signed-off-by: Alexey Gladkov <[email protected]>
---
tools/testing/selftests/Makefile | 1 +
tools/testing/selftests/rlimits/.gitignore | 2 +
tools/testing/selftests/rlimits/Makefile | 6 +
tools/testing/selftests/rlimits/config | 1 +
.../selftests/rlimits/rlimits-per-userns.c | 161 ++++++++++++++++++
5 files changed, 171 insertions(+)
create mode 100644 tools/testing/selftests/rlimits/.gitignore
create mode 100644 tools/testing/selftests/rlimits/Makefile
create mode 100644 tools/testing/selftests/rlimits/config
create mode 100644 tools/testing/selftests/rlimits/rlimits-per-userns.c

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 8a917cb4426a..a6d3fde4a617 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -46,6 +46,7 @@ TARGETS += proc
TARGETS += pstore
TARGETS += ptrace
TARGETS += openat2
+TARGETS += rlimits
TARGETS += rseq
TARGETS += rtc
TARGETS += seccomp
diff --git a/tools/testing/selftests/rlimits/.gitignore b/tools/testing/selftests/rlimits/.gitignore
new file mode 100644
index 000000000000..091021f255b3
--- /dev/null
+++ b/tools/testing/selftests/rlimits/.gitignore
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+rlimits-per-userns
diff --git a/tools/testing/selftests/rlimits/Makefile b/tools/testing/selftests/rlimits/Makefile
new file mode 100644
index 000000000000..03aadb406212
--- /dev/null
+++ b/tools/testing/selftests/rlimits/Makefile
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+CFLAGS += -Wall -O2 -g
+TEST_GEN_PROGS := rlimits-per-userns
+
+include ../lib.mk
diff --git a/tools/testing/selftests/rlimits/config b/tools/testing/selftests/rlimits/config
new file mode 100644
index 000000000000..416bd53ce982
--- /dev/null
+++ b/tools/testing/selftests/rlimits/config
@@ -0,0 +1 @@
+CONFIG_USER_NS=y
diff --git a/tools/testing/selftests/rlimits/rlimits-per-userns.c b/tools/testing/selftests/rlimits/rlimits-per-userns.c
new file mode 100644
index 000000000000..26dc949e93ea
--- /dev/null
+++ b/tools/testing/selftests/rlimits/rlimits-per-userns.c
@@ -0,0 +1,161 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Author: Alexey Gladkov <[email protected]>
+ */
+#define _GNU_SOURCE
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <sys/time.h>
+#include <sys/resource.h>
+#include <sys/prctl.h>
+#include <sys/stat.h>
+
+#include <unistd.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <sched.h>
+#include <signal.h>
+#include <limits.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <err.h>
+
+#define NR_CHILDS 2
+
+static char *service_prog;
+static uid_t user = 60000;
+static uid_t group = 60000;
+
+static void setrlimit_nproc(rlim_t n)
+{
+ pid_t pid = getpid();
+ struct rlimit limit = {
+ .rlim_cur = n,
+ .rlim_max = n
+ };
+
+ warnx("(pid=%d): Setting RLIMIT_NPROC=%ld", pid, n);
+
+ if (setrlimit(RLIMIT_NPROC, &limit) < 0)
+ err(EXIT_FAILURE, "(pid=%d): setrlimit(RLIMIT_NPROC)", pid);
+}
+
+static pid_t fork_child(void)
+{
+ pid_t pid = fork();
+
+ if (pid < 0)
+ err(EXIT_FAILURE, "fork");
+
+ if (pid > 0)
+ return pid;
+
+ pid = getpid();
+
+ warnx("(pid=%d): New process starting ...", pid);
+
+ if (prctl(PR_SET_PDEATHSIG, SIGKILL) < 0)
+ err(EXIT_FAILURE, "(pid=%d): prctl(PR_SET_PDEATHSIG)", pid);
+
+ signal(SIGUSR1, SIG_DFL);
+
+ warnx("(pid=%d): Changing to uid=%d, gid=%d", pid, user, group);
+
+ if (setgid(group) < 0)
+ err(EXIT_FAILURE, "(pid=%d): setgid(%d)", pid, group);
+ if (setuid(user) < 0)
+ err(EXIT_FAILURE, "(pid=%d): setuid(%d)", pid, user);
+
+ warnx("(pid=%d): Service running ...", pid);
+
+ warnx("(pid=%d): Unshare user namespace", pid);
+ if (unshare(CLONE_NEWUSER) < 0)
+ err(EXIT_FAILURE, "unshare(CLONE_NEWUSER)");
+
+ char *const argv[] = { "service", NULL };
+ char *const envp[] = { "I_AM_SERVICE=1", NULL };
+
+ warnx("(pid=%d): Executing real service ...", pid);
+
+ execve(service_prog, argv, envp);
+ err(EXIT_FAILURE, "(pid=%d): execve", pid);
+}
+
+int main(int argc, char **argv)
+{
+ size_t i;
+ pid_t child[NR_CHILDS];
+ int wstatus[NR_CHILDS];
+ int childs = NR_CHILDS;
+ pid_t pid;
+
+ if (getenv("I_AM_SERVICE")) {
+ pause();
+ exit(EXIT_SUCCESS);
+ }
+
+ service_prog = argv[0];
+ pid = getpid();
+
+ warnx("(pid=%d) Starting testcase", pid);
+
+ /*
+ * This rlimit is not a problem for root because it can be exceeded.
+ */
+ setrlimit_nproc(1);
+
+ for (i = 0; i < NR_CHILDS; i++) {
+ child[i] = fork_child();
+ wstatus[i] = 0;
+ usleep(250000);
+ }
+
+ while (1) {
+ for (i = 0; i < NR_CHILDS; i++) {
+ if (child[i] <= 0)
+ continue;
+
+ errno = 0;
+ pid_t ret = waitpid(child[i], &wstatus[i], WNOHANG);
+
+ if (!ret || (!WIFEXITED(wstatus[i]) && !WIFSIGNALED(wstatus[i])))
+ continue;
+
+ if (ret < 0 && errno != ECHILD)
+ warn("(pid=%d): waitpid(%d)", pid, child[i]);
+
+ child[i] *= -1;
+ childs -= 1;
+ }
+
+ if (!childs)
+ break;
+
+ usleep(250000);
+
+ for (i = 0; i < NR_CHILDS; i++) {
+ if (child[i] <= 0)
+ continue;
+ kill(child[i], SIGUSR1);
+ }
+ }
+
+ for (i = 0; i < NR_CHILDS; i++) {
+ if (WIFEXITED(wstatus[i]))
+ warnx("(pid=%d): pid %d exited, status=%d",
+ pid, -child[i], WEXITSTATUS(wstatus[i]));
+ else if (WIFSIGNALED(wstatus[i]))
+ warnx("(pid=%d): pid %d killed by signal %d",
+ pid, -child[i], WTERMSIG(wstatus[i]));
+
+ if (WIFSIGNALED(wstatus[i]) && WTERMSIG(wstatus[i]) == SIGUSR1)
+ continue;
+
+ warnx("(pid=%d): Test failed", pid);
+ exit(EXIT_FAILURE);
+ }
+
+ warnx("(pid=%d): Test passed", pid);
+ exit(EXIT_SUCCESS);
+}
--
2.29.2

2021-02-22 10:04:50

by Alexey Gladkov

[permalink] [raw]

Subject: [PATCH v7 6/7] Reimplement RLIMIT_MEMLOCK on top of ucounts

---

The rlimit counter is tied to uid in the user_namespace. This allows
rlimit values to be specified in userns even if they are already
globally exceeded by the user. However, the value of the previous
user_namespaces cannot be exceeded.

Changelog

v7:
* Keep only ucounts for RLIMIT_MEMLOCK checks instead of struct cred.

v6:
* Fix bug in hugetlb_file_setup() detected by trinity.

Reported-by: kernel test robot <[email protected]>
Signed-off-by: Alexey Gladkov <[email protected]>
---
fs/hugetlbfs/inode.c | 16 ++++++++--------
include/linux/hugetlb.h | 4 ++--
include/linux/mm.h | 4 ++--
include/linux/sched/user.h | 1 -
include/linux/shmem_fs.h | 2 +-
include/linux/user_namespace.h | 1 +
ipc/shm.c | 26 +++++++++++++-------------
kernel/fork.c | 1 +
kernel/ucount.c | 1 +
kernel/user.c | 1 -
kernel/user_namespace.c | 1 +
mm/memfd.c | 4 ++--
mm/mlock.c | 20 ++++++++++++--------
mm/mmap.c | 4 ++--
mm/shmem.c | 8 ++++----
15 files changed, 50 insertions(+), 44 deletions(-)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 21c20fd5f9ee..cea98b68f271 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -1452,7 +1452,7 @@ static int get_hstate_idx(int page_size_log)
* otherwise hugetlb_reserve_pages reserves one less hugepages than intended.
*/
struct file *hugetlb_file_setup(const char *name, size_t size,
- vm_flags_t acctflag, struct user_struct **user,
+ vm_flags_t acctflag, struct ucounts **ucounts,
int creat_flags, int page_size_log)
{
struct inode *inode;
@@ -1464,20 +1464,20 @@ struct file *hugetlb_file_setup(const char *name, size_t size,
if (hstate_idx < 0)
return ERR_PTR(-ENODEV);

- *user = NULL;
+ *ucounts = NULL;
mnt = hugetlbfs_vfsmount[hstate_idx];
if (!mnt)
return ERR_PTR(-ENOENT);

if (creat_flags == HUGETLB_SHMFS_INODE && !can_do_hugetlb_shm()) {
- *user = current_user();
- if (user_shm_lock(size, *user)) {
+ *ucounts = current_ucounts();
+ if (user_shm_lock(size, *ucounts)) {
task_lock(current);
pr_warn_once("%s (%d): Using mlock ulimits for SHM_HUGETLB is deprecated\n",
current->comm, current->pid);
task_unlock(current);
} else {
- *user = NULL;
+ *ucounts = NULL;
return ERR_PTR(-EPERM);
}
}
@@ -1504,9 +1504,9 @@ struct file *hugetlb_file_setup(const char *name, size_t size,

iput(inode);
out:
- if (*user) {
- user_shm_unlock(size, *user);
- *user = NULL;
+ if (*ucounts) {
+ user_shm_unlock(size, *ucounts);
+ *ucounts = NULL;
}
return file;
}
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index b5807f23caf8..12b78ae587a2 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -434,7 +434,7 @@ static inline struct hugetlbfs_inode_info *HUGETLBFS_I(struct inode *inode)
extern const struct file_operations hugetlbfs_file_operations;
extern const struct vm_operations_struct hugetlb_vm_ops;
struct file *hugetlb_file_setup(const char *name, size_t size, vm_flags_t acct,
- struct user_struct **user, int creat_flags,
+ struct ucounts **ucounts, int creat_flags,
int page_size_log);

static inline bool is_file_hugepages(struct file *file)
@@ -454,7 +454,7 @@ static inline struct hstate *hstate_inode(struct inode *i)
#define is_file_hugepages(file) false
static inline struct file *
hugetlb_file_setup(const char *name, size_t size, vm_flags_t acctflag,
- struct user_struct **user, int creat_flags,
+ struct ucounts **ucounts, int creat_flags,
int page_size_log)
{
return ERR_PTR(-ENOSYS);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index ecdf8a8cd6ae..64927c5492f2 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1628,8 +1628,8 @@ extern bool can_do_mlock(void);
#else
static inline bool can_do_mlock(void) { return false; }
#endif
-extern int user_shm_lock(size_t, struct user_struct *);
-extern void user_shm_unlock(size_t, struct user_struct *);
+extern int user_shm_lock(size_t, struct ucounts *);
+extern void user_shm_unlock(size_t, struct ucounts *);

/*
* Parameter block passed down to zap_pte_range in exceptional cases.
diff --git a/include/linux/sched/user.h b/include/linux/sched/user.h
index 8ba9cec4fb99..82bd2532da6b 100644
--- a/include/linux/sched/user.h
+++ b/include/linux/sched/user.h
@@ -18,7 +18,6 @@ struct user_struct {
#ifdef CONFIG_EPOLL
atomic_long_t epoll_watches; /* The number of file descriptors currently watched */
#endif
- unsigned long locked_shm; /* How many pages of mlocked shm ? */
unsigned long unix_inflight; /* How many files in flight in unix sockets */
atomic_long_t pipe_bufs; /* how many pages are allocated in pipe buffers */

diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h
index d82b6f396588..aa77dcd1646f 100644
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -65,7 +65,7 @@ extern struct file *shmem_file_setup_with_mnt(struct vfsmount *mnt,
extern int shmem_zero_setup(struct vm_area_struct *);
extern unsigned long shmem_get_unmapped_area(struct file *, unsigned long addr,
unsigned long len, unsigned long pgoff, unsigned long flags);
-extern int shmem_lock(struct file *file, int lock, struct user_struct *user);
+extern int shmem_lock(struct file *file, int lock, struct ucounts *ucounts);
#ifdef CONFIG_SHMEM
extern const struct address_space_operations shmem_aops;
static inline bool shmem_mapping(struct address_space *mapping)
diff --git a/include/linux/user_namespace.h b/include/linux/user_namespace.h
index f84b68832c56..966b0d733bb8 100644
--- a/include/linux/user_namespace.h
+++ b/include/linux/user_namespace.h
@@ -53,6 +53,7 @@ enum ucount_type {
UCOUNT_RLIMIT_NPROC,
UCOUNT_RLIMIT_MSGQUEUE,
UCOUNT_RLIMIT_SIGPENDING,
+ UCOUNT_RLIMIT_MEMLOCK,
UCOUNT_COUNTS,
};

diff --git a/ipc/shm.c b/ipc/shm.c
index febd88daba8c..003234fbbd17 100644
--- a/ipc/shm.c
+++ b/ipc/shm.c
@@ -60,7 +60,7 @@ struct shmid_kernel /* private to the kernel */
time64_t shm_ctim;
struct pid *shm_cprid;
struct pid *shm_lprid;
- struct user_struct *mlock_user;
+ struct ucounts *mlock_ucounts;

/* The task created the shm object. NULL if the task is dead. */
struct task_struct *shm_creator;
@@ -286,10 +286,10 @@ static void shm_destroy(struct ipc_namespace *ns, struct shmid_kernel *shp)
shm_rmid(ns, shp);
shm_unlock(shp);
if (!is_file_hugepages(shm_file))
- shmem_lock(shm_file, 0, shp->mlock_user);
- else if (shp->mlock_user)
+ shmem_lock(shm_file, 0, shp->mlock_ucounts);
+ else if (shp->mlock_ucounts)
user_shm_unlock(i_size_read(file_inode(shm_file)),
- shp->mlock_user);
+ shp->mlock_ucounts);
fput(shm_file);
ipc_update_pid(&shp->shm_cprid, NULL);
ipc_update_pid(&shp->shm_lprid, NULL);
@@ -625,7 +625,7 @@ static int newseg(struct ipc_namespace *ns, struct ipc_params *params)

shp->shm_perm.key = key;
shp->shm_perm.mode = (shmflg & S_IRWXUGO);
- shp->mlock_user = NULL;
+ shp->mlock_ucounts = NULL;

shp->shm_perm.security = NULL;
error = security_shm_alloc(&shp->shm_perm);
@@ -650,7 +650,7 @@ static int newseg(struct ipc_namespace *ns, struct ipc_params *params)
if (shmflg & SHM_NORESERVE)
acctflag = VM_NORESERVE;
file = hugetlb_file_setup(name, hugesize, acctflag,
- &shp->mlock_user, HUGETLB_SHMFS_INODE,
+ &shp->mlock_ucounts, HUGETLB_SHMFS_INODE,
(shmflg >> SHM_HUGE_SHIFT) & SHM_HUGE_MASK);
} else {
/*
@@ -698,8 +698,8 @@ static int newseg(struct ipc_namespace *ns, struct ipc_params *params)
no_id:
ipc_update_pid(&shp->shm_cprid, NULL);
ipc_update_pid(&shp->shm_lprid, NULL);
- if (is_file_hugepages(file) && shp->mlock_user)
- user_shm_unlock(size, shp->mlock_user);
+ if (is_file_hugepages(file) && shp->mlock_ucounts)
+ user_shm_unlock(size, shp->mlock_ucounts);
fput(file);
ipc_rcu_putref(&shp->shm_perm, shm_rcu_free);
return error;
@@ -1105,12 +1105,12 @@ static int shmctl_do_lock(struct ipc_namespace *ns, int shmid, int cmd)
goto out_unlock0;

if (cmd == SHM_LOCK) {
- struct user_struct *user = current_user();
+ struct ucounts *ucounts = current_ucounts();

- err = shmem_lock(shm_file, 1, user);
+ err = shmem_lock(shm_file, 1, ucounts);
if (!err && !(shp->shm_perm.mode & SHM_LOCKED)) {
shp->shm_perm.mode |= SHM_LOCKED;
- shp->mlock_user = user;
+ shp->mlock_ucounts = ucounts;
}
goto out_unlock0;
}
@@ -1118,9 +1118,9 @@ static int shmctl_do_lock(struct ipc_namespace *ns, int shmid, int cmd)
/* SHM_UNLOCK */
if (!(shp->shm_perm.mode & SHM_LOCKED))
goto out_unlock0;
- shmem_lock(shm_file, 0, shp->mlock_user);
+ shmem_lock(shm_file, 0, shp->mlock_ucounts);
shp->shm_perm.mode &= ~SHM_LOCKED;
- shp->mlock_user = NULL;
+ shp->mlock_ucounts = NULL;
get_file(shm_file);
ipc_unlock_object(&shp->shm_perm);
rcu_read_unlock();
diff --git a/kernel/fork.c b/kernel/fork.c
index 99b10b9fe4b6..76ccb000856c 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -825,6 +825,7 @@ void __init fork_init(void)
init_user_ns.ucount_max[UCOUNT_RLIMIT_NPROC] = task_rlimit(&init_task, RLIMIT_NPROC);
init_user_ns.ucount_max[UCOUNT_RLIMIT_MSGQUEUE] = task_rlimit(&init_task, RLIMIT_MSGQUEUE);
init_user_ns.ucount_max[UCOUNT_RLIMIT_SIGPENDING] = task_rlimit(&init_task, RLIMIT_SIGPENDING);
+ init_user_ns.ucount_max[UCOUNT_RLIMIT_MEMLOCK] = task_rlimit(&init_task, RLIMIT_MEMLOCK);

#ifdef CONFIG_VMAP_STACK
cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "fork:vm_stack_cache",
diff --git a/kernel/ucount.c b/kernel/ucount.c
index 2ac969fba668..b6242b77eb89 100644
--- a/kernel/ucount.c
+++ b/kernel/ucount.c
@@ -84,6 +84,7 @@ static struct ctl_table user_table[] = {
{ },
{ },
{ },
+ { },
{ }
};
#endif /* CONFIG_SYSCTL */
diff --git a/kernel/user.c b/kernel/user.c
index 6737327f83be..c82399c1618a 100644
--- a/kernel/user.c
+++ b/kernel/user.c
@@ -98,7 +98,6 @@ static DEFINE_SPINLOCK(uidhash_lock);
/* root_user.__count is 1, for init task cred */
struct user_struct root_user = {
.__count = REFCOUNT_INIT(1),
- .locked_shm = 0,
.uid = GLOBAL_ROOT_UID,
.ratelimit = RATELIMIT_STATE_INIT(root_user.ratelimit, 0, 0),
};
diff --git a/kernel/user_namespace.c b/kernel/user_namespace.c
index df1bed32dd48..5ef0d4b182ba 100644
--- a/kernel/user_namespace.c
+++ b/kernel/user_namespace.c
@@ -124,6 +124,7 @@ int create_user_ns(struct cred *new)
ns->ucount_max[UCOUNT_RLIMIT_NPROC] = rlimit(RLIMIT_NPROC);
ns->ucount_max[UCOUNT_RLIMIT_MSGQUEUE] = rlimit(RLIMIT_MSGQUEUE);
ns->ucount_max[UCOUNT_RLIMIT_SIGPENDING] = rlimit(RLIMIT_SIGPENDING);
+ ns->ucount_max[UCOUNT_RLIMIT_MEMLOCK] = rlimit(RLIMIT_MEMLOCK);
ns->ucounts = ucounts;

/* Inherit USERNS_SETGROUPS_ALLOWED from our parent */
diff --git a/mm/memfd.c b/mm/memfd.c
index 2647c898990c..081dd33e6a61 100644
--- a/mm/memfd.c
+++ b/mm/memfd.c
@@ -297,9 +297,9 @@ SYSCALL_DEFINE2(memfd_create,
}

if (flags & MFD_HUGETLB) {
- struct user_struct *user = NULL;
+ struct ucounts *ucounts = NULL;

- file = hugetlb_file_setup(name, 0, VM_NORESERVE, &user,
+ file = hugetlb_file_setup(name, 0, VM_NORESERVE, &ucounts,
HUGETLB_ANONHUGE_INODE,
(flags >> MFD_HUGE_SHIFT) &
MFD_HUGE_MASK);
diff --git a/mm/mlock.c b/mm/mlock.c
index 55b3b3672977..293b6c1deceb 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -818,9 +818,10 @@ SYSCALL_DEFINE0(munlockall)
*/
static DEFINE_SPINLOCK(shmlock_user_lock);

-int user_shm_lock(size_t size, struct user_struct *user)
+int user_shm_lock(size_t size, struct ucounts *ucounts)
{
unsigned long lock_limit, locked;
+ bool overlimit;
int allowed = 0;

locked = (size + PAGE_SIZE - 1) >> PAGE_SHIFT;
@@ -829,21 +830,24 @@ int user_shm_lock(size_t size, struct user_struct *user)
allowed = 1;
lock_limit >>= PAGE_SHIFT;
spin_lock(&shmlock_user_lock);
- if (!allowed &&
- locked + user->locked_shm > lock_limit && !capable(CAP_IPC_LOCK))
+ overlimit = inc_rlimit_ucounts_and_test(ucounts, UCOUNT_RLIMIT_MEMLOCK,
+ locked, lock_limit);
+
+ if (!allowed && overlimit && !capable(CAP_IPC_LOCK)) {
+ dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_MEMLOCK, locked);
goto out;
- get_uid(user);
- user->locked_shm += locked;
+ }
+ get_ucounts(ucounts);
allowed = 1;
out:
spin_unlock(&shmlock_user_lock);
return allowed;
}

-void user_shm_unlock(size_t size, struct user_struct *user)
+void user_shm_unlock(size_t size, struct ucounts *ucounts)
{
spin_lock(&shmlock_user_lock);
- user->locked_shm -= (size + PAGE_SIZE - 1) >> PAGE_SHIFT;
+ dec_rlimit_ucounts(ucounts, UCOUNT_RLIMIT_MEMLOCK, (size + PAGE_SIZE - 1) >> PAGE_SHIFT);
spin_unlock(&shmlock_user_lock);
- free_uid(user);
+ put_ucounts(ucounts);
}
diff --git a/mm/mmap.c b/mm/mmap.c
index dc7206032387..773baa8c82ff 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1607,7 +1607,7 @@ unsigned long ksys_mmap_pgoff(unsigned long addr, unsigned long len,
goto out_fput;
}
} else if (flags & MAP_HUGETLB) {
- struct user_struct *user = NULL;
+ struct ucounts *ucounts = NULL;
struct hstate *hs;

hs = hstate_sizelog((flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK);
@@ -1623,7 +1623,7 @@ unsigned long ksys_mmap_pgoff(unsigned long addr, unsigned long len,
*/
file = hugetlb_file_setup(HUGETLB_ANON_FILE, len,
VM_NORESERVE,
- &user, HUGETLB_ANONHUGE_INODE,
+ &ucounts, HUGETLB_ANONHUGE_INODE,
(flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK);
if (IS_ERR(file))
return PTR_ERR(file);
diff --git a/mm/shmem.c b/mm/shmem.c
index 7c6b6d8f6c39..efd195da364e 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2225,7 +2225,7 @@ static struct mempolicy *shmem_get_policy(struct vm_area_struct *vma,
}
#endif

-int shmem_lock(struct file *file, int lock, struct user_struct *user)
+int shmem_lock(struct file *file, int lock, struct ucounts *ucounts)
{
struct inode *inode = file_inode(file);
struct shmem_inode_info *info = SHMEM_I(inode);
@@ -2237,13 +2237,13 @@ int shmem_lock(struct file *file, int lock, struct user_struct *user)
* no serialization needed when called from shm_destroy().
*/
if (lock && !(info->flags & VM_LOCKED)) {
- if (!user_shm_lock(inode->i_size, user))
+ if (!user_shm_lock(inode->i_size, ucounts))
goto out_nomem;
info->flags |= VM_LOCKED;
mapping_set_unevictable(file->f_mapping);
}
- if (!lock && (info->flags & VM_LOCKED) && user) {
- user_shm_unlock(inode->i_size, user);
+ if (!lock && (info->flags & VM_LOCKED) && ucounts) {
+ user_shm_unlock(inode->i_size, ucounts);
info->flags &= ~VM_LOCKED;
mapping_clear_unevictable(file->f_mapping);
}
--
2.29.2

2021-02-24 12:46:23

by kernel test robot

[permalink] [raw]

Subject: d28296d248: stress-ng.sigsegv.ops_per_sec -82.7% regression

---

Greeting,

FYI, we noticed a -82.7% regression of stress-ng.sigsegv.ops_per_sec due to commit:


commit: d28296d2484fa11e94dff65e93eb25802a443d47 ("[PATCH v7 5/7] Reimplement RLIMIT_SIGPENDING on top of ucounts")
url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next

in testcase: stress-ng
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
with following parameters:

nr_threads: 100%
disk: 1HDD
testtime: 60s
class: interrupt
test: sigsegv
cpufreq_governor: performance
ucode: 0x42e


In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.sigq.ops_per_sec -56.1% regression |
| test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory |
| test parameters | class=interrupt |
| | cpufreq_governor=performance |
| | disk=1HDD |
| | nr_threads=100% |
| | test=sigq |
| | testtime=60s |
| | ucode=0x42e |
+------------------+-----------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml
bin/lkp run compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
interrupt/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/sigsegv/stress-ng/60s/0x42e

commit:
4660d663b4 ("Reimplement RLIMIT_MSGQUEUE on top of ucounts")
d28296d248 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")

4660d663b4207ce6 d28296d2484fa11e94dff65e93e
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
14:6 -217% 1:6 perf-profile.children.cycles-pp.error_entry
12:6 -179% 1:6 perf-profile.self.cycles-pp.error_entry
%stddev %change %stddev
\ | \
4.766e+08 -82.7% 82358308 stress-ng.sigsegv.ops
7942807 -82.7% 1372639 stress-ng.sigsegv.ops_per_sec
29408 -77.5% 6621 ? 61% stress-ng.time.file_system_inputs
1566 +69.4% 2653 stress-ng.time.system_time
1274 -84.8% 193.22 ? 8% stress-ng.time.user_time
12458 ? 5% +37.2% 17097 ? 5% numa-meminfo.node1.Active(anon)
51.41 +66.5% 85.59 iostat.cpu.system
41.17 -84.2% 6.50 ? 7% iostat.cpu.user
3040 ? 4% +37.9% 4193 ? 4% numa-vmstat.node1.nr_active_anon
3040 ? 4% +37.9% 4193 ? 4% numa-vmstat.node1.nr_zone_active_anon
50.83 +67.2% 85.00 vmstat.cpu.sy
40.50 -85.6% 5.83 ? 11% vmstat.cpu.us
225.33 -77.7% 50.33 ? 62% vmstat.io.bi
7.00 -100.0% 0.00 vmstat.memory.buff
20735 ? 2% -14.1% 17812 ? 5% meminfo.Active
13506 ? 3% +31.9% 17812 ? 5% meminfo.Active(anon)
7228 -100.0% 0.00 meminfo.Active(file)
29308 +18.4% 34687 ? 2% meminfo.Shmem
202067 -9.5% 182899 meminfo.VmallocUsed
0.01 ? 17% -0.0 0.00 ? 10% mpstat.cpu.all.iowait%
1.04 -0.1 0.92 mpstat.cpu.all.irq%
0.03 ? 8% -0.0 0.02 ? 4% mpstat.cpu.all.soft%
51.54 +35.6 87.17 mpstat.cpu.all.sys%
42.22 -35.6 6.66 ? 8% mpstat.cpu.all.usr%
0.00 ? 70% +191.7% 0.01 ? 26% perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
0.00 ? 47% +158.8% 0.01 ? 43% perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
0.27 ? 25% -55.4% 0.12 ? 37% perf-sched.total_wait_and_delay.average.ms
202.17 ? 23% -29.5% 142.50 ? 13% perf-sched.total_wait_and_delay.count.ms
0.21 ? 18% -69.3% 0.06 ? 58% perf-sched.total_wait_time.average.ms
0.00 ? 70% +191.7% 0.01 ? 26% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
8.17 ? 29% -85.7% 1.17 ?125% perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
0.00 ? 47% +158.8% 0.01 ? 43% perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
3.11 ? 11% -76.1% 0.74 ?142% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
1790 ? 15% -30.1% 1250 ? 13% slabinfo.dmaengine-unmap-16.active_objs
1790 ? 15% -30.1% 1250 ? 13% slabinfo.dmaengine-unmap-16.num_objs
123.00 -100.0% 0.00 slabinfo.ext4_pending_reservation.active_objs
123.00 -100.0% 0.00 slabinfo.ext4_pending_reservation.num_objs
3619 -100.0% 0.00 slabinfo.f2fs_free_nid.active_objs
3619 -100.0% 0.00 slabinfo.f2fs_free_nid.num_objs
62865 -44.1% 35155 slabinfo.kmalloc-64.active_objs
984.33 -43.9% 552.50 slabinfo.kmalloc-64.active_slabs
63031 -43.9% 35389 slabinfo.kmalloc-64.num_objs
984.33 -43.9% 552.50 slabinfo.kmalloc-64.num_slabs
161.00 ? 9% +67.1% 269.00 ? 7% slabinfo.xfs_buf.active_objs
161.00 ? 9% +67.1% 269.00 ? 7% slabinfo.xfs_buf.num_objs
3399 ? 3% +32.9% 4519 ? 4% proc-vmstat.nr_active_anon
1806 -100.0% 0.00 proc-vmstat.nr_active_file
9333 +3.0% 9610 proc-vmstat.nr_mapped
7344 +18.4% 8698 ? 2% proc-vmstat.nr_shmem
16319 -0.9% 16176 proc-vmstat.nr_slab_reclaimable
24399 -6.2% 22882 proc-vmstat.nr_slab_unreclaimable
3399 ? 3% +32.9% 4519 ? 4% proc-vmstat.nr_zone_active_anon
1806 -100.0% 0.00 proc-vmstat.nr_zone_active_file
3693 ? 80% -80.1% 736.17 ? 61% proc-vmstat.numa_hint_faults
293002 -2.7% 284991 proc-vmstat.numa_hit
249530 -3.3% 241180 proc-vmstat.numa_local
5007 ?111% -90.5% 478.00 ? 81% proc-vmstat.numa_pages_migrated
11443 ? 2% -7.0% 10636 ? 4% proc-vmstat.pgactivate
332528 -3.9% 319693 proc-vmstat.pgalloc_normal
249148 ? 2% -4.1% 239053 ? 2% proc-vmstat.pgfree
5007 ?111% -90.5% 478.00 ? 81% proc-vmstat.pgmigrate_success
14704 -77.5% 3310 ? 61% proc-vmstat.pgpgin
0.00 +2.1e+105% 2095 proc-vmstat.pgpgout
13870 ? 10% -55.1% 6227 ? 28% softirqs.CPU0.RCU
9989 ? 3% -62.2% 3775 ? 23% softirqs.CPU1.RCU
8625 ? 13% -76.1% 2061 ? 10% softirqs.CPU10.RCU
7954 ? 15% -65.9% 2709 ? 18% softirqs.CPU14.RCU
9075 ? 14% -78.7% 1929 ? 11% softirqs.CPU17.RCU
8522 ? 13% -76.7% 1985 ? 22% softirqs.CPU18.RCU
9595 ? 7% -63.3% 3522 ? 22% softirqs.CPU2.RCU
8455 ? 11% -74.5% 2152 ? 45% softirqs.CPU20.RCU
8320 ? 12% -76.7% 1939 ? 14% softirqs.CPU21.RCU
8338 ? 13% -71.7% 2359 ? 32% softirqs.CPU23.RCU
8541 ? 12% -75.5% 2089 ? 32% softirqs.CPU26.RCU
9639 ? 20% -79.5% 1976 ? 17% softirqs.CPU28.RCU
9232 ? 13% -78.0% 2026 ? 6% softirqs.CPU30.RCU
7857 ? 17% -68.9% 2446 ? 27% softirqs.CPU34.RCU
8619 ? 11% -75.8% 2081 ? 30% softirqs.CPU36.RCU
9614 ? 3% -74.3% 2469 ? 15% softirqs.CPU4.RCU
8962 ? 10% -77.9% 1981 ? 12% softirqs.CPU41.RCU
9027 ? 12% -78.6% 1932 ? 8% softirqs.CPU42.RCU
9364 ? 12% -76.5% 2197 ? 8% softirqs.CPU44.RCU
8774 ? 13% -75.5% 2147 ? 21% softirqs.CPU47.RCU
8783 ? 12% -76.0% 2105 ? 12% softirqs.CPU5.RCU
9007 ? 9% -75.8% 2177 ? 8% softirqs.CPU6.RCU
417664 ? 7% -72.8% 113621 ? 8% softirqs.RCU
12708 ? 4% +13.0% 14362 ? 2% softirqs.TIMER
60500 -27.7% 43751 interrupts.CAL:Function_call_interrupts
1121 -26.9% 819.17 ? 3% interrupts.CPU10.CAL:Function_call_interrupts
1561 ? 43% -48.8% 800.00 ? 5% interrupts.CPU11.CAL:Function_call_interrupts
1425 ? 6% -25.3% 1065 ? 6% interrupts.CPU12.CAL:Function_call_interrupts
166.17 ? 13% -26.4% 122.33 ? 21% interrupts.CPU13.RES:Rescheduling_interrupts
1402 ? 18% -25.6% 1043 ? 22% interrupts.CPU15.CAL:Function_call_interrupts
129.17 ? 50% -42.5% 74.33 ? 4% interrupts.CPU17.RES:Rescheduling_interrupts
1182 ? 9% -31.0% 815.00 ? 2% interrupts.CPU20.CAL:Function_call_interrupts
1120 -29.7% 787.17 ? 4% interrupts.CPU21.CAL:Function_call_interrupts
1115 ? 3% -28.2% 801.17 interrupts.CPU23.CAL:Function_call_interrupts
1169 ? 7% -27.2% 851.33 ? 5% interrupts.CPU24.CAL:Function_call_interrupts
177.33 ? 98% -55.9% 78.17 ? 6% interrupts.CPU25.RES:Rescheduling_interrupts
1142 ? 16% -28.8% 813.00 ? 3% interrupts.CPU27.CAL:Function_call_interrupts
1229 ? 18% -33.3% 820.33 ? 4% interrupts.CPU28.CAL:Function_call_interrupts
1124 ? 4% -28.3% 806.17 interrupts.CPU29.CAL:Function_call_interrupts
1123 ? 3% -28.5% 803.00 interrupts.CPU30.CAL:Function_call_interrupts
1127 ? 2% -32.0% 766.67 ? 19% interrupts.CPU31.CAL:Function_call_interrupts
1066 ? 8% -22.3% 829.33 ? 7% interrupts.CPU32.CAL:Function_call_interrupts
1109 -26.0% 820.50 ? 4% interrupts.CPU34.CAL:Function_call_interrupts
1315 ? 22% -37.7% 818.83 ? 2% interrupts.CPU38.CAL:Function_call_interrupts
1164 ? 4% -29.0% 827.00 ? 3% interrupts.CPU39.CAL:Function_call_interrupts
5513 ? 35% +13.1% 6237 ? 33% interrupts.CPU39.NMI:Non-maskable_interrupts
5513 ? 35% +13.1% 6237 ? 33% interrupts.CPU39.PMI:Performance_monitoring_interrupts
1277 ? 23% -36.1% 815.83 interrupts.CPU40.CAL:Function_call_interrupts
97.33 ? 28% -25.7% 72.33 ? 6% interrupts.CPU40.RES:Rescheduling_interrupts
1116 -24.8% 839.00 ? 11% interrupts.CPU42.CAL:Function_call_interrupts
1130 ? 3% -28.5% 808.67 ? 3% interrupts.CPU43.CAL:Function_call_interrupts
1121 -29.8% 787.50 ? 4% interrupts.CPU45.CAL:Function_call_interrupts
1119 -27.3% 813.83 interrupts.CPU46.CAL:Function_call_interrupts
1167 ? 6% -28.0% 840.67 interrupts.CPU47.CAL:Function_call_interrupts
1667 ? 41% -44.3% 928.67 ? 24% interrupts.CPU5.CAL:Function_call_interrupts
1369 ? 24% -39.5% 827.67 ? 3% interrupts.CPU6.CAL:Function_call_interrupts
96.83 ? 25% -23.1% 74.50 ? 2% interrupts.CPU7.RES:Rescheduling_interrupts
1123 -28.1% 807.00 interrupts.CPU9.CAL:Function_call_interrupts
0.72 ? 5% +107.4% 1.50 ? 2% perf-stat.i.MPKI
8.023e+09 -13.5% 6.943e+09 perf-stat.i.branch-instructions
1.08 ? 2% -0.5 0.55 ? 3% perf-stat.i.branch-miss-rate%
71073978 ? 2% -67.4% 23149119 ? 3% perf-stat.i.branch-misses
29.33 ? 2% +5.9 35.23 perf-stat.i.cache-miss-rate%
6189596 +120.2% 13628525 perf-stat.i.cache-misses
21228048 +82.4% 38714786 perf-stat.i.cache-references
3.36 +34.8% 4.53 perf-stat.i.cpi
109.52 ? 2% -22.5% 84.92 perf-stat.i.cpu-migrations
22695 -56.5% 9882 perf-stat.i.cycles-between-cache-misses
1.15 ? 7% -0.9 0.30 ? 4% perf-stat.i.dTLB-load-miss-rate%
1.398e+08 ? 7% -83.4% 23247225 ? 4% perf-stat.i.dTLB-load-misses
1.154e+10 -34.4% 7.564e+09 perf-stat.i.dTLB-loads
1.17 -0.0 1.13 perf-stat.i.dTLB-store-miss-rate%
1.321e+08 -82.1% 23679743 perf-stat.i.dTLB-store-misses
1.071e+10 -81.3% 2.005e+09 perf-stat.i.dTLB-stores
41869658 -74.5% 10693569 ? 56% perf-stat.i.iTLB-load-misses
19932113 ? 38% -88.3% 2325708 ? 64% perf-stat.i.iTLB-loads
3.945e+10 -25.9% 2.924e+10 perf-stat.i.instructions
1199 ? 4% +182.7% 3389 ? 26% perf-stat.i.instructions-per-iTLB-miss
0.31 -23.7% 0.24 perf-stat.i.ipc
634.71 -45.5% 345.81 perf-stat.i.metric.M/sec
166710 ? 10% +3369.3% 5783625 perf-stat.i.node-load-misses
227268 ? 6% +3063.1% 7188743 perf-stat.i.node-loads
48.41 -4.5 43.91 perf-stat.i.node-store-miss-rate%
5425859 -11.8% 4783945 perf-stat.i.node-store-misses
5599687 +6.1% 5943407 perf-stat.i.node-stores
7532204 -82.7% 1305118 perf-stat.i.page-faults
0.54 +146.0% 1.32 perf-stat.overall.MPKI
0.89 ? 2% -0.6 0.33 ? 3% perf-stat.overall.branch-miss-rate%
29.17 ? 2% +6.0 35.20 perf-stat.overall.cache-miss-rate%
3.45 +35.0% 4.65 perf-stat.overall.cpi
21953 -54.5% 9979 perf-stat.overall.cycles-between-cache-misses
1.20 ? 7% -0.9 0.31 ? 4% perf-stat.overall.dTLB-load-miss-rate%
1.22 -0.1 1.17 perf-stat.overall.dTLB-store-miss-rate%
942.39 +245.5% 3255 ? 28% perf-stat.overall.instructions-per-iTLB-miss
0.29 -25.9% 0.21 perf-stat.overall.ipc
42.24 ? 2% +2.3 44.58 perf-stat.overall.node-load-miss-rate%
49.21 -4.6 44.59 perf-stat.overall.node-store-miss-rate%
7.894e+09 -13.5% 6.83e+09 perf-stat.ps.branch-instructions
69952381 ? 2% -67.4% 22781972 ? 3% perf-stat.ps.branch-misses
6093197 +120.1% 13409962 perf-stat.ps.cache-misses
20897937 +82.3% 38097787 perf-stat.ps.cache-references
107.78 ? 2% -22.4% 83.62 perf-stat.ps.cpu-migrations
1.375e+08 ? 7% -83.4% 22871450 ? 4% perf-stat.ps.dTLB-load-misses
1.135e+10 -34.4% 7.442e+09 perf-stat.ps.dTLB-loads
1.3e+08 -82.1% 23295591 perf-stat.ps.dTLB-store-misses
1.054e+10 -81.3% 1.973e+09 perf-stat.ps.dTLB-stores
41193894 -74.5% 10519305 ? 56% perf-stat.ps.iTLB-load-misses
19610606 ? 38% -88.3% 2288293 ? 64% perf-stat.ps.iTLB-loads
3.882e+10 -25.9% 2.876e+10 perf-stat.ps.instructions
164152 ? 10% +3366.2% 5689843 perf-stat.ps.node-load-misses
223940 ? 6% +3058.2% 7072454 perf-stat.ps.node-loads
5338769 -11.8% 4706549 perf-stat.ps.node-store-misses
5510338 +6.1% 5847491 perf-stat.ps.node-stores
7410609 -82.7% 1283937 perf-stat.ps.page-faults
2.454e+12 -25.9% 1.817e+12 perf-stat.total.instructions
33.68 -29.6 4.04 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
18.94 -16.5 2.46 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
13.67 -12.3 1.42 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
12.27 -10.9 1.36 ? 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
8.04 -7.0 1.05 ? 2% perf-profile.calltrace.cycles-pp.__entry_text_start
6.06 -6.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack
6.02 ? 2% -5.4 0.66 ? 2% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigaction.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.51 -4.8 0.70 perf-profile.calltrace.cycles-pp.__setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
0.00 +0.6 0.65 perf-profile.calltrace.cycles-pp.inc_rlimit_ucounts_and_test.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault
18.91 +27.0 45.89 perf-profile.calltrace.cycles-pp.irqentry_exit_to_user_mode.asm_exc_page_fault
15.00 +30.4 45.42 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
14.66 +30.7 45.38 perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
11.73 +34.2 45.97 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
8.85 +36.7 45.54 perf-profile.calltrace.cycles-pp.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
8.16 +37.2 45.40 perf-profile.calltrace.cycles-pp.force_sig_fault.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
8.04 +37.3 45.39 perf-profile.calltrace.cycles-pp.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
6.53 +37.8 44.36 perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
7.06 +38.0 45.05 perf-profile.calltrace.cycles-pp.__send_signal.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore.exc_page_fault
4.28 ? 2% +39.2 43.46 perf-profile.calltrace.cycles-pp.__sigqueue_free.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
4.87 ? 2% +39.9 44.81 perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore
0.00 +42.7 42.72 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.get_signal
0.00 +43.0 42.99 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.get_signal.arch_do_signal_or_restart
0.00 +43.1 43.08 perf-profile.calltrace.cycles-pp.put_ucounts.__sigqueue_free.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare
0.00 +43.5 43.52 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal
0.00 +44.0 43.97 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal.force_sig_info_to_task
0.00 +44.1 44.05 perf-profile.calltrace.cycles-pp.get_ucounts.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault
31.89 +60.0 91.94 perf-profile.calltrace.cycles-pp.asm_exc_page_fault
33.84 -29.7 4.10 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
19.01 -16.5 2.48 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
14.93 -13.3 1.59 perf-profile.children.cycles-pp.syscall_return_via_sysret
13.14 -11.7 1.43 ? 2% perf-profile.children.cycles-pp.do_syscall_64
10.60 -9.3 1.27 perf-profile.children.cycles-pp.__entry_text_start
6.17 ? 2% -5.5 0.68 ? 2% perf-profile.children.cycles-pp.__x64_sys_rt_sigaction
5.55 -4.8 0.71 ? 2% perf-profile.children.cycles-pp.__setup_rt_frame
3.63 ? 2% -3.2 0.40 ? 2% perf-profile.children.cycles-pp.__x64_sys_rt_sigprocmask
3.54 -3.2 0.32 ? 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
3.11 ? 3% -2.8 0.35 ? 3% perf-profile.children.cycles-pp._copy_from_user
2.91 -2.5 0.37 ? 2% perf-profile.children.cycles-pp.copy_fpstate_to_sigframe
2.38 -2.1 0.28 perf-profile.children.cycles-pp.__irqentry_text_end
2.40 ? 4% -2.1 0.30 ? 2% perf-profile.children.cycles-pp.__might_fault
2.23 -1.9 0.32 perf-profile.children.cycles-pp.native_irq_return_iret
2.26 ? 2% -1.9 0.35 ? 5% perf-profile.children.cycles-pp.do_user_addr_fault
1.86 -1.6 0.27 ? 3% perf-profile.children.cycles-pp.do_sigaction
1.75 -1.5 0.21 ? 3% perf-profile.children.cycles-pp.copy_user_generic_unrolled
1.67 ? 3% -1.5 0.19 ? 3% perf-profile.children.cycles-pp.__set_current_blocked
1.51 ? 2% -1.3 0.18 ? 2% perf-profile.children.cycles-pp._raw_spin_lock_irq
1.33 ? 3% -1.1 0.19 perf-profile.children.cycles-pp.copy_siginfo_to_user
1.24 ? 3% -1.1 0.13 ? 3% perf-profile.children.cycles-pp.recalc_sigpending
1.30 ? 4% -1.1 0.18 ? 3% perf-profile.children.cycles-pp._copy_to_user
1.23 ? 3% -1.1 0.15 ? 3% perf-profile.children.cycles-pp.___might_sleep
1.07 ? 4% -0.9 0.12 ? 3% perf-profile.children.cycles-pp.signal_setup_done
0.98 ? 4% -0.9 0.11 ? 4% perf-profile.children.cycles-pp.fpu__clear
0.93 ? 4% -0.8 0.12 ? 8% perf-profile.children.cycles-pp.__might_sleep
0.89 -0.8 0.14 ? 3% perf-profile.children.cycles-pp.__clear_user
0.85 ? 4% -0.7 0.10 ? 3% perf-profile.children.cycles-pp.__set_task_blocked
0.81 ? 2% -0.7 0.10 ? 8% perf-profile.children.cycles-pp.sigprocmask
0.76 ? 11% -0.7 0.07 ? 10% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.70 ? 2% -0.6 0.06 ? 9% perf-profile.children.cycles-pp.signal_wake_up_state
0.73 -0.6 0.13 ? 7% perf-profile.children.cycles-pp.__perf_sw_event
0.50 ? 2% -0.5 0.04 ? 44% perf-profile.children.cycles-pp.complete_signal
0.50 ? 4% -0.4 0.08 ? 11% perf-profile.children.cycles-pp.___perf_sw_event
0.45 -0.4 0.06 ? 6% perf-profile.children.cycles-pp.sync_regs
0.44 -0.4 0.07 ? 10% perf-profile.children.cycles-pp.fixup_vdso_exception
0.35 ? 2% -0.3 0.05 perf-profile.children.cycles-pp.prepare_signal
0.35 ? 3% -0.3 0.08 ? 5% perf-profile.children.cycles-pp.is_prefetch
0.33 ? 11% -0.3 0.07 ? 6% perf-profile.children.cycles-pp.kmem_cache_alloc
0.19 ? 4% -0.1 0.05 ? 45% perf-profile.children.cycles-pp.copy_from_kernel_nofault
0.33 +0.0 0.37 ? 3% perf-profile.children.cycles-pp.kmem_cache_free
0.00 +0.4 0.36 ? 4% perf-profile.children.cycles-pp.dec_rlimit_ucounts
0.00 +0.6 0.65 perf-profile.children.cycles-pp.inc_rlimit_ucounts_and_test
19.11 +26.8 45.91 perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
15.69 +29.8 45.51 perf-profile.children.cycles-pp.exit_to_user_mode_prepare
14.70 +30.7 45.38 perf-profile.children.cycles-pp.arch_do_signal_or_restart
11.77 +34.2 45.99 perf-profile.children.cycles-pp.exc_page_fault
8.90 +36.7 45.55 perf-profile.children.cycles-pp.__bad_area_nosemaphore
8.17 +37.2 45.40 perf-profile.children.cycles-pp.force_sig_fault
8.10 +37.3 45.39 perf-profile.children.cycles-pp.force_sig_info_to_task
6.57 +37.8 44.38 perf-profile.children.cycles-pp.get_signal
7.11 +38.0 45.06 perf-profile.children.cycles-pp.__send_signal
4.28 ? 2% +39.2 43.46 perf-profile.children.cycles-pp.__sigqueue_free
4.89 ? 2% +39.9 44.82 perf-profile.children.cycles-pp.__sigqueue_alloc
0.00 +43.1 43.09 perf-profile.children.cycles-pp.put_ucounts
0.00 +44.1 44.05 perf-profile.children.cycles-pp.get_ucounts
31.95 +60.0 91.95 perf-profile.children.cycles-pp.asm_exc_page_fault
0.00 +86.2 86.24 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.32 ? 4% +86.7 87.01 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
18.17 -15.8 2.36 perf-profile.self.cycles-pp.syscall_exit_to_user_mode
14.90 -13.3 1.58 perf-profile.self.cycles-pp.syscall_return_via_sysret
10.60 -9.3 1.27 perf-profile.self.cycles-pp.__entry_text_start
4.55 ? 2% -4.5 0.04 ? 72% perf-profile.self.cycles-pp.__sigqueue_alloc
4.09 ? 2% -3.6 0.49 ? 2% perf-profile.self.cycles-pp.irqentry_exit_to_user_mode
2.30 ? 5% -2.1 0.19 ? 4% perf-profile.self.cycles-pp.do_syscall_64
2.38 -2.1 0.28 perf-profile.self.cycles-pp.__irqentry_text_end
2.22 -1.9 0.32 perf-profile.self.cycles-pp.native_irq_return_iret
1.92 ? 8% -1.8 0.16 ? 2% perf-profile.self.cycles-pp.__x64_sys_rt_sigaction
1.72 ? 3% -1.5 0.20 ? 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
1.71 -1.5 0.21 ? 4% perf-profile.self.cycles-pp.copy_fpstate_to_sigframe
1.68 -1.5 0.20 ? 3% perf-profile.self.cycles-pp.copy_user_generic_unrolled
1.46 ? 4% -1.3 0.12 ? 4% perf-profile.self.cycles-pp.__x64_sys_rt_sigprocmask
1.46 ? 2% -1.3 0.17 ? 3% perf-profile.self.cycles-pp._raw_spin_lock_irq
1.21 ? 6% -1.1 0.14 ? 5% perf-profile.self.cycles-pp.__setup_rt_frame
1.21 ? 2% -1.1 0.14 ? 3% perf-profile.self.cycles-pp.___might_sleep
1.06 ? 4% -1.0 0.09 ? 5% perf-profile.self.cycles-pp.recalc_sigpending
1.01 ? 6% -1.0 0.05 perf-profile.self.cycles-pp.asm_exc_page_fault
0.97 ? 4% -0.9 0.10 ? 7% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.95 ? 2% -0.8 0.12 ? 4% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.98 -0.8 0.15 ? 4% perf-profile.self.cycles-pp.do_sigaction
0.84 ? 4% -0.7 0.10 ? 6% perf-profile.self.cycles-pp.__might_sleep
0.72 ? 8% -0.6 0.09 ? 4% perf-profile.self.cycles-pp._copy_from_user
0.71 ? 2% -0.6 0.10 ? 7% perf-profile.self.cycles-pp.do_user_addr_fault
0.65 ? 14% -0.6 0.06 ? 11% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.68 ? 6% -0.6 0.10 ? 3% perf-profile.self.cycles-pp.__might_fault
0.80 -0.6 0.25 ? 3% perf-profile.self.cycles-pp.get_signal
0.64 ? 4% -0.6 0.08 ? 5% perf-profile.self.cycles-pp.fpu__clear
0.57 ? 3% -0.5 0.07 ? 7% perf-profile.self.cycles-pp.__send_signal
0.53 ? 2% -0.5 0.08 ? 6% perf-profile.self.cycles-pp.__clear_user
0.49 ? 5% -0.4 0.04 ? 45% perf-profile.self.cycles-pp.arch_do_signal_or_restart
0.43 -0.4 0.06 ? 7% perf-profile.self.cycles-pp.fixup_vdso_exception
0.42 ? 4% -0.4 0.06 ? 7% perf-profile.self.cycles-pp.___perf_sw_event
0.41 -0.4 0.05 ? 9% perf-profile.self.cycles-pp.sync_regs
0.33 ? 3% -0.3 0.04 ? 44% perf-profile.self.cycles-pp.prepare_signal
0.31 ? 12% -0.2 0.07 ? 7% perf-profile.self.cycles-pp.kmem_cache_alloc
0.22 ? 3% -0.1 0.10 ? 7% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.33 +0.0 0.36 ? 3% perf-profile.self.cycles-pp.kmem_cache_free
0.00 +0.1 0.06 ? 9% perf-profile.self.cycles-pp.get_ucounts
0.00 +0.1 0.06 ? 6% perf-profile.self.cycles-pp.put_ucounts
0.00 +0.4 0.36 ? 4% perf-profile.self.cycles-pp.dec_rlimit_ucounts
0.27 ? 4% +0.5 0.77 ? 2% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.00 +0.6 0.65 perf-profile.self.cycles-pp.inc_rlimit_ucounts_and_test
0.00 +86.2 86.24 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



stress-ng.time.user_time

1400 +--------------------------------------------------------------------+
|.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.++.+.+.++.+.++.+.+. +.+.+.++.|
1200 |-+ + |
| |
1000 |-+ |
| |
800 |-+ |
| |
600 |-+ |
| |
400 |-+ O |
| |
200 |-OO O O O O O OO O OO O O OO O O OO O OO O O OO O OO |
| |
0 +--------------------------------------------------------------------+


stress-ng.time.system_time

2800 +--------------------------------------------------------------------+
| O O O O OO O O O O O |
2600 |-OO O O O O OO O OO OO O O O O O O |
| |
2400 |-+ O |
| |
2200 |-+ |
| |
2000 |-+ |
| |
1800 |-+ |
| |
1600 |.++.+.+.++.+. .+.++.+.+.++.+.+.++.+.++.+.+.++.+.+ .+.+.++.+.+.++.|
| +.++ + |
1400 +--------------------------------------------------------------------+


stress-ng.sigsegv.ops

5e+08 +-----------------------------------------------------------------+
|.++.+.++.+.++.+.++.++.+.++.+.++.+.++.+.++.+.++.+.++.++.+.++.+.++.|
4.5e+08 |-+ |
4e+08 |-+ |
| |
3.5e+08 |-+ |
3e+08 |-+ |
| |
2.5e+08 |-+ |
2e+08 |-+ |
| |
1.5e+08 |-+ |
1e+08 |-+ |
| OO O OO O OO O OO OO O OO O OO O OO O OO O OO O OO |
5e+07 +-----------------------------------------------------------------+


stress-ng.sigsegv.ops_per_sec

8e+06 +-------------------------------------------------------------------+
| |
7e+06 |-+ |
| |
6e+06 |-+ |
| |
5e+06 |-+ |
| |
4e+06 |-+ |
| |
3e+06 |-+ |
| |
2e+06 |-+ |
| OO O O OO O OO O OO O O OO O OO O OO O O OO O OO O O |
1e+06 +-------------------------------------------------------------------+


[*] bisect-good sample
[O] bisect-bad sample

***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
interrupt/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/sigq/stress-ng/60s/0x42e

commit:
4660d663b4 ("Reimplement RLIMIT_MSGQUEUE on top of ucounts")
d28296d248 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")

4660d663b4207ce6 d28296d2484fa11e94dff65e93e
---------------- ---------------------------
%stddev %change %stddev
\ | \
4.176e+08 -56.1% 1.831e+08 ? 21% stress-ng.sigq.ops
6959423 -56.1% 3051857 ? 21% stress-ng.sigq.ops_per_sec
2.157e+08 ? 8% -66.1% 73100172 ? 9% stress-ng.time.involuntary_context_switches
14467 +1.4% 14674 stress-ng.time.minor_page_faults
4513 +1.2% 4569 stress-ng.time.percent_of_cpu_this_job_got
2150 +17.9% 2535 ? 3% stress-ng.time.system_time
660.96 -53.1% 309.96 ? 30% stress-ng.time.user_time
2.478e+08 ? 5% -70.5% 73168836 ? 9% stress-ng.time.voluntary_context_switches
7.70 ? 8% -6.8% 7.18 iostat.cpu.idle
70.60 +16.6% 82.32 ? 3% iostat.cpu.system
21.70 -51.6% 10.49 ? 29% iostat.cpu.user
15660 ? 3% +13.7% 17806 ? 3% meminfo.Active
15660 ? 3% +13.7% 17806 ? 3% meminfo.Active(anon)
31878 ? 2% +8.8% 34688 ? 2% meminfo.Shmem
123900 ? 66% -66.9% 41021 ? 30% cpuidle.C1.usage
1.507e+08 ? 6% +9.5% 1.65e+08 ? 4% cpuidle.C6.time
681599 ? 38% -72.8% 185137 ?113% cpuidle.POLL.time
645871 ? 39% -98.5% 9580 ? 75% cpuidle.POLL.usage
1.78 ? 3% -0.8 0.95 mpstat.cpu.all.irq%
0.03 ? 4% -0.0 0.02 ? 27% mpstat.cpu.all.soft%
70.90 +12.5 83.42 ? 3% mpstat.cpu.all.sys%
22.35 -11.6 10.75 ? 29% mpstat.cpu.all.usr%
825.33 ? 2% -11.6% 729.67 ? 5% slabinfo.file_lock_cache.active_objs
825.33 ? 2% -11.6% 729.67 ? 5% slabinfo.file_lock_cache.num_objs
1455 ? 11% -32.4% 983.00 ? 13% slabinfo.khugepaged_mm_slot.active_objs
1455 ? 11% -32.4% 983.00 ? 13% slabinfo.khugepaged_mm_slot.num_objs
69.67 +17.2% 81.67 ? 3% vmstat.cpu.sy
21.00 -52.4% 10.00 ? 29% vmstat.cpu.us
7139282 ? 7% -68.5% 2251603 ? 9% vmstat.system.cs
518916 ? 6% -80.5% 101139 ? 4% vmstat.system.in
8082 ? 51% -68.3% 2565 ? 17% softirqs.CPU10.SCHED
6261 ? 72% -56.4% 2729 ? 14% softirqs.CPU17.SCHED
6147 ? 66% -64.3% 2195 ? 3% softirqs.CPU25.SCHED
16334 ? 41% -82.6% 2846 ? 24% softirqs.CPU27.SCHED
6280 ? 56% -61.9% 2394 ? 9% softirqs.CPU39.SCHED
8248 ? 50% -73.2% 2209 ? 8% softirqs.CPU40.SCHED
228327 ? 9% -46.3% 122665 ? 2% softirqs.SCHED
3851 ? 4% +13.8% 4381 ? 4% proc-vmstat.nr_active_anon
9587 +2.8% 9855 proc-vmstat.nr_mapped
7943 ? 2% +8.7% 8630 ? 2% proc-vmstat.nr_shmem
3851 ? 4% +13.8% 4381 ? 4% proc-vmstat.nr_zone_active_anon
438.33 ?122% +1463.6% 6853 ? 69% proc-vmstat.numa_pages_migrated
8816 ? 4% +11.5% 9827 ? 6% proc-vmstat.pgactivate
291848 +1.7% 296689 proc-vmstat.pgalloc_normal
438.33 ?122% +1463.6% 6853 ? 69% proc-vmstat.pgmigrate_success
252.00 ? 19% +30.8% 329.67 ? 7% numa-vmstat.node0.nr_active_anon
44373 ? 23% -64.0% 15967 ? 49% numa-vmstat.node0.nr_anon_pages
46179 ? 21% -61.8% 17619 ? 38% numa-vmstat.node0.nr_inactive_anon
252.00 ? 19% +30.8% 329.67 ? 7% numa-vmstat.node0.nr_zone_active_anon
46179 ? 21% -61.8% 17619 ? 38% numa-vmstat.node0.nr_zone_inactive_anon
3678 ? 4% +11.5% 4100 ? 4% numa-vmstat.node1.nr_active_anon
13880 ? 74% +206.7% 42565 ? 17% numa-vmstat.node1.nr_anon_pages
16004 ? 59% +181.1% 44988 ? 14% numa-vmstat.node1.nr_inactive_anon
3678 ? 4% +11.5% 4100 ? 4% numa-vmstat.node1.nr_zone_active_anon
16004 ? 59% +181.1% 44988 ? 14% numa-vmstat.node1.nr_zone_inactive_anon
1007 ? 19% +31.3% 1322 ? 7% numa-meminfo.node0.Active
1007 ? 19% +31.3% 1322 ? 7% numa-meminfo.node0.Active(anon)
39280 ? 35% -66.1% 13314 ? 74% numa-meminfo.node0.AnonHugePages
177224 ? 24% -63.9% 63933 ? 49% numa-meminfo.node0.AnonPages
182948 ? 22% -52.3% 87286 ? 59% numa-meminfo.node0.AnonPages.max
184457 ? 21% -61.8% 70532 ? 38% numa-meminfo.node0.Inactive
184457 ? 21% -61.8% 70532 ? 38% numa-meminfo.node0.Inactive(anon)
1080630 ? 5% -12.3% 947332 ? 3% numa-meminfo.node0.MemUsed
14267 ? 4% +17.0% 16695 ? 5% numa-meminfo.node1.Active
14267 ? 4% +17.0% 16695 ? 5% numa-meminfo.node1.Active(anon)
56124 ? 74% +202.8% 169963 ? 17% numa-meminfo.node1.AnonPages
75025 ? 56% +139.7% 179816 ? 14% numa-meminfo.node1.AnonPages.max
64951 ? 59% +176.3% 179452 ? 14% numa-meminfo.node1.Inactive
64951 ? 59% +176.3% 179452 ? 14% numa-meminfo.node1.Inactive(anon)
941178 ? 5% +14.6% 1078500 ? 3% numa-meminfo.node1.MemUsed
51.14 ? 15% +47.4% 75.37 ? 13% sched_debug.cfs_rq:/.load_avg.avg
114.91 ? 14% +38.2% 158.79 ? 13% sched_debug.cfs_rq:/.load_avg.stddev
6.99 ? 70% +352.8% 31.65 ? 27% sched_debug.cfs_rq:/.removed.load_avg.avg
47.92 ? 70% +153.1% 121.29 ? 12% sched_debug.cfs_rq:/.removed.load_avg.stddev
2.38 ? 97% +182.8% 6.74 ? 36% sched_debug.cfs_rq:/.removed.runnable_avg.avg
2.38 ? 97% +182.7% 6.73 ? 36% sched_debug.cfs_rq:/.removed.util_avg.avg
356.50 ? 3% -24.1% 270.50 ? 31% sched_debug.cfs_rq:/.util_avg.min
290.54 ? 2% +10.6% 321.30 ? 5% sched_debug.cfs_rq:/.util_est_enqueued.avg
446696 ? 4% +50.1% 670473 ? 7% sched_debug.cpu.avg_idle.avg
1120651 ? 11% +20.5% 1350114 ? 10% sched_debug.cpu.avg_idle.max
2124 ? 10% +43.0% 3038 ? 15% sched_debug.cpu.avg_idle.min
708.23 ? 3% -14.4% 606.42 ? 4% sched_debug.cpu.clock_task.stddev
989.50 ? 8% +12.0% 1108 sched_debug.cpu.curr->pid.min
21503 ? 71% +114.0% 46014 ? 39% sched_debug.cpu.max_idle_balance_cost.stddev
4583665 ? 7% -68.5% 1443748 ? 9% sched_debug.cpu.nr_switches.avg
6508139 ? 3% -71.5% 1854831 sched_debug.cpu.nr_switches.max
2183394 ? 28% -83.2% 367194 ? 87% sched_debug.cpu.nr_switches.min
1135976 ? 11% -69.9% 342489 ? 34% sched_debug.cpu.nr_switches.stddev
0.03 ? 59% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
0.01 ? 12% -36.4% 0.00 ? 26% perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
0.05 ? 57% -100.0% 0.00 perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
0.19 ? 77% -61.0% 0.07 ? 5% perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
7.83 ?140% -99.8% 0.01 ? 38% perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
0.36 ? 18% -64.4% 0.13 ? 75% perf-sched.total_wait_and_delay.average.ms
0.24 ? 5% -72.2% 0.07 ? 66% perf-sched.total_wait_time.average.ms
1.38 ? 6% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
0.03 ? 41% -61.6% 0.01 ? 80% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
0.01 ? 12% -36.4% 0.00 ? 26% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
0.08 ? 17% -30.3% 0.05 ? 3% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
2.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.do_syslog.part.0
9.00 -81.5% 1.67 ? 28% perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
2.72 ? 8% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
0.80 ? 45% -89.3% 0.09 ? 10% perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
7.83 ?140% -99.8% 0.01 ? 38% perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
1.35 ? 8% -100.0% 0.00 perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
0.03 ? 42% -66.3% 0.01 ? 82% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
0.07 ? 24% -27.6% 0.05 ? 3% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
3.21 ? 2% -58.6% 1.33 ? 70% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
2.69 ? 8% -100.0% 0.00 perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
0.77 ? 50% -89.7% 0.08 ? 10% perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
6.12 ? 3% -27.9% 4.42 ? 3% perf-stat.i.MPKI
1.008e+10 ? 4% -20.8% 7.989e+09 ? 2% perf-stat.i.branch-instructions
1.55 ? 2% -0.7 0.89 perf-stat.i.branch-miss-rate%
1.372e+08 ? 2% -60.4% 54290819 perf-stat.i.branch-misses
5.24 ? 7% +6.4 11.66 ? 5% perf-stat.i.cache-miss-rate%
12610881 ? 8% +30.8% 16498692 ? 4% perf-stat.i.cache-misses
2.994e+08 -50.1% 1.495e+08 ? 2% perf-stat.i.cache-references
7370989 ? 7% -68.6% 2316288 ? 9% perf-stat.i.context-switches
2.75 ? 4% +37.7% 3.78 ? 2% perf-stat.i.cpi
37935 ? 20% -99.1% 344.33 ? 39% perf-stat.i.cpu-migrations
10857 ? 9% -24.6% 8184 ? 4% perf-stat.i.cycles-between-cache-misses
1.33 ? 4% -0.5 0.81 ? 15% perf-stat.i.dTLB-load-miss-rate%
1.921e+08 ? 3% -59.4% 78058438 ? 14% perf-stat.i.dTLB-load-misses
1.375e+10 ? 4% -32.6% 9.26e+09 perf-stat.i.dTLB-loads
0.51 ? 5% +0.1 0.61 ? 9% perf-stat.i.dTLB-store-miss-rate%
52874010 -55.6% 23456497 ? 13% perf-stat.i.dTLB-store-misses
9.955e+09 ? 3% -63.0% 3.687e+09 ? 4% perf-stat.i.dTLB-stores
62946854 ? 4% -59.7% 25392611 ? 3% perf-stat.i.iTLB-load-misses
4.88e+10 ? 4% -28.0% 3.514e+10 perf-stat.i.instructions
1071 +54.5% 1655 ? 10% perf-stat.i.instructions-per-iTLB-miss
0.38 ? 3% -26.8% 0.28 ? 3% perf-stat.i.ipc
0.52 ? 7% -70.6% 0.15 ? 11% perf-stat.i.metric.K/sec
711.88 ? 3% -38.1% 440.48 perf-stat.i.metric.M/sec
47.42 -1.7 45.75 perf-stat.i.node-load-miss-rate%
3468825 ? 20% +118.3% 7571866 ? 5% perf-stat.i.node-load-misses
3747717 ? 20% +133.9% 8764955 ? 4% perf-stat.i.node-loads
47.67 -2.6 45.11 perf-stat.i.node-store-miss-rate%
8290895 ? 4% -28.2% 5954593 ? 6% perf-stat.i.node-store-misses
8800840 -19.9% 7053271 ? 5% perf-stat.i.node-stores
6.14 ? 3% -30.8% 4.25 perf-stat.overall.MPKI
1.36 ? 2% -0.7 0.68 perf-stat.overall.branch-miss-rate%
4.22 ? 10% +6.8 11.05 ? 6% perf-stat.overall.cache-miss-rate%
2.79 ? 4% +38.7% 3.87 perf-stat.overall.cpi
10868 ? 8% -23.9% 8267 ? 4% perf-stat.overall.cycles-between-cache-misses
1.38 ? 4% -0.5 0.84 ? 15% perf-stat.overall.dTLB-load-miss-rate%
0.53 ? 5% +0.1 0.63 ? 8% perf-stat.overall.dTLB-store-miss-rate%
775.35 +78.9% 1386 ? 5% perf-stat.overall.instructions-per-iTLB-miss
0.36 ? 4% -28.0% 0.26 perf-stat.overall.ipc
48.07 -1.7 46.34 perf-stat.overall.node-load-miss-rate%
48.49 -2.7 45.76 perf-stat.overall.node-store-miss-rate%
9.922e+09 ? 4% -20.8% 7.859e+09 ? 2% perf-stat.ps.branch-instructions
1.351e+08 ? 2% -60.4% 53438494 perf-stat.ps.branch-misses
12410866 ? 8% +30.8% 16234283 ? 4% perf-stat.ps.cache-misses
2.946e+08 -50.1% 1.47e+08 ? 2% perf-stat.ps.cache-references
7252050 ? 7% -68.6% 2278311 ? 9% perf-stat.ps.context-switches
37323 ? 20% -99.1% 338.80 ? 39% perf-stat.ps.cpu-migrations
1.89e+08 ? 3% -59.4% 76787292 ? 14% perf-stat.ps.dTLB-load-misses
1.352e+10 ? 4% -32.6% 9.11e+09 perf-stat.ps.dTLB-loads
52020930 -55.6% 23075080 ? 13% perf-stat.ps.dTLB-store-misses
9.794e+09 ? 3% -63.0% 3.627e+09 ? 4% perf-stat.ps.dTLB-stores
61931461 ? 4% -59.7% 24978175 ? 3% perf-stat.ps.iTLB-load-misses
4.802e+10 ? 4% -28.0% 3.457e+10 perf-stat.ps.instructions
3412948 ? 20% +118.2% 7448345 ? 5% perf-stat.ps.node-load-misses
3687448 ? 20% +133.8% 8622075 ? 4% perf-stat.ps.node-loads
8157655 ? 4% -28.2% 5857930 ? 6% perf-stat.ps.node-store-misses
8659916 -19.9% 6939197 ? 5% perf-stat.ps.node-stores
3.035e+12 ? 4% -27.9% 2.189e+12 perf-stat.total.instructions
82.33 ? 64% +1322.3% 1171 ?127% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
17959753 ? 6% -99.7% 45889 ? 2% interrupts.CAL:Function_call_interrupts
103593 ?125% -99.5% 569.33 ? 9% interrupts.CPU0.CAL:Function_call_interrupts
45339 ?104% -94.7% 2414 ? 30% interrupts.CPU0.RES:Rescheduling_interrupts
1078794 ?112% -99.9% 1002 ? 12% interrupts.CPU1.CAL:Function_call_interrupts
6507 ? 26% +21.7% 7922 ? 6% interrupts.CPU1.NMI:Non-maskable_interrupts
6507 ? 26% +21.7% 7922 ? 6% interrupts.CPU1.PMI:Performance_monitoring_interrupts
505526 ?101% -99.8% 1223 ? 46% interrupts.CPU1.RES:Rescheduling_interrupts
791738 ? 80% -99.9% 898.67 ? 13% interrupts.CPU10.CAL:Function_call_interrupts
406363 ? 71% -99.4% 2378 ? 80% interrupts.CPU10.RES:Rescheduling_interrupts
46401 ?115% -98.3% 789.00 ? 21% interrupts.CPU11.CAL:Function_call_interrupts
5509 ? 35% +25.8% 6931 ? 28% interrupts.CPU11.NMI:Non-maskable_interrupts
5509 ? 35% +25.8% 6931 ? 28% interrupts.CPU11.PMI:Performance_monitoring_interrupts
27829 ?104% -87.3% 3526 ? 77% interrupts.CPU11.RES:Rescheduling_interrupts
290182 ?130% -99.6% 1299 ? 24% interrupts.CPU12.CAL:Function_call_interrupts
170757 ?136% -98.3% 2886 ? 40% interrupts.CPU12.RES:Rescheduling_interrupts
131940 ? 43% -99.1% 1206 ? 37% interrupts.CPU13.CAL:Function_call_interrupts
68303 ? 32% -97.9% 1445 ?126% interrupts.CPU13.RES:Rescheduling_interrupts
241043 ?116% -99.5% 1319 ? 32% interrupts.CPU14.CAL:Function_call_interrupts
98573 ? 93% -97.0% 3005 ? 47% interrupts.CPU14.RES:Rescheduling_interrupts
874074 ? 72% -99.9% 875.33 ? 6% interrupts.CPU15.CAL:Function_call_interrupts
350120 ? 49% -99.9% 412.67 ? 52% interrupts.CPU15.RES:Rescheduling_interrupts
577696 ? 23% -99.9% 855.33 ? 3% interrupts.CPU16.CAL:Function_call_interrupts
8263 -24.9% 6202 ? 23% interrupts.CPU16.NMI:Non-maskable_interrupts
8263 -24.9% 6202 ? 23% interrupts.CPU16.PMI:Performance_monitoring_interrupts
237815 ? 15% -99.5% 1220 ? 53% interrupts.CPU16.RES:Rescheduling_interrupts
686797 ?112% -99.9% 933.67 ? 13% interrupts.CPU17.CAL:Function_call_interrupts
317108 ?105% -99.8% 536.00 ? 49% interrupts.CPU17.RES:Rescheduling_interrupts
748163 ?122% -99.9% 863.00 ? 5% interrupts.CPU18.CAL:Function_call_interrupts
320920 ?112% -99.5% 1469 ? 95% interrupts.CPU18.RES:Rescheduling_interrupts
286835 ?121% -99.7% 943.33 ? 15% interrupts.CPU19.CAL:Function_call_interrupts
174455 ?113% -98.7% 2326 ? 69% interrupts.CPU19.RES:Rescheduling_interrupts
443377 ? 83% -99.8% 944.33 ? 12% interrupts.CPU2.CAL:Function_call_interrupts
57297 ? 99% -98.5% 855.00 ? 6% interrupts.CPU20.CAL:Function_call_interrupts
54090 ? 67% -99.0% 535.33 ? 74% interrupts.CPU20.RES:Rescheduling_interrupts
27584 ? 80% -96.8% 890.33 ? 6% interrupts.CPU21.CAL:Function_call_interrupts
27052 ? 82% -94.3% 1539 ? 58% interrupts.CPU21.RES:Rescheduling_interrupts
62804 ?104% -97.8% 1362 ? 47% interrupts.CPU22.CAL:Function_call_interrupts
27230 ? 84% -92.6% 2002 ? 86% interrupts.CPU22.RES:Rescheduling_interrupts
351930 ? 72% -99.7% 966.00 ? 16% interrupts.CPU23.CAL:Function_call_interrupts
136149 ? 62% -98.1% 2565 ? 93% interrupts.CPU23.RES:Rescheduling_interrupts
366644 ?138% -99.8% 877.67 ? 5% interrupts.CPU24.CAL:Function_call_interrupts
499203 ? 82% -99.8% 862.33 ? 3% interrupts.CPU25.CAL:Function_call_interrupts
298172 ? 70% -99.2% 2317 ? 19% interrupts.CPU25.RES:Rescheduling_interrupts
11844 ? 61% -92.9% 838.33 ? 2% interrupts.CPU26.CAL:Function_call_interrupts
82.33 ? 64% +1322.3% 1171 ?127% interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
1841110 ? 49% -100.0% 746.33 ? 26% interrupts.CPU27.CAL:Function_call_interrupts
5501 ? 35% +25.6% 6909 ? 28% interrupts.CPU27.NMI:Non-maskable_interrupts
5501 ? 35% +25.6% 6909 ? 28% interrupts.CPU27.PMI:Performance_monitoring_interrupts
1089819 ? 55% -99.8% 1727 ? 80% interrupts.CPU27.RES:Rescheduling_interrupts
74718 ?118% -98.7% 942.00 interrupts.CPU28.CAL:Function_call_interrupts
5504 ? 35% +25.6% 6912 ? 28% interrupts.CPU28.NMI:Non-maskable_interrupts
5504 ? 35% +25.6% 6912 ? 28% interrupts.CPU28.PMI:Performance_monitoring_interrupts
67380 ?125% -98.5% 1034 ? 37% interrupts.CPU28.RES:Rescheduling_interrupts
367428 ? 77% -99.7% 1054 ? 22% interrupts.CPU29.CAL:Function_call_interrupts
239038 ? 82% -99.3% 1767 ? 84% interrupts.CPU29.RES:Rescheduling_interrupts
220744 ?117% -99.5% 1046 ? 21% interrupts.CPU3.CAL:Function_call_interrupts
191269 ?119% -98.4% 3122 ? 15% interrupts.CPU3.RES:Rescheduling_interrupts
247241 ? 81% -99.6% 874.33 ? 5% interrupts.CPU30.CAL:Function_call_interrupts
161878 ? 84% -98.1% 3063 ? 37% interrupts.CPU30.RES:Rescheduling_interrupts
144713 ?110% -99.3% 1077 ? 23% interrupts.CPU31.CAL:Function_call_interrupts
5502 ? 35% +25.6% 6911 ? 28% interrupts.CPU31.NMI:Non-maskable_interrupts
5502 ? 35% +25.6% 6911 ? 28% interrupts.CPU31.PMI:Performance_monitoring_interrupts
84227 ?100% -96.4% 3013 ? 72% interrupts.CPU31.RES:Rescheduling_interrupts
104510 ?109% -99.2% 848.00 ? 4% interrupts.CPU32.CAL:Function_call_interrupts
5515 ? 35% +31.4% 7249 ? 20% interrupts.CPU32.NMI:Non-maskable_interrupts
5515 ? 35% +31.4% 7249 ? 20% interrupts.CPU32.PMI:Performance_monitoring_interrupts
70068 ?101% -96.7% 2320 ?126% interrupts.CPU32.RES:Rescheduling_interrupts
87671 ?131% -99.0% 857.33 ? 2% interrupts.CPU33.CAL:Function_call_interrupts
35054 ?127% -96.5% 1242 ? 80% interrupts.CPU33.RES:Rescheduling_interrupts
77336 ?124% -98.7% 988.33 ? 21% interrupts.CPU34.CAL:Function_call_interrupts
66106 ?123% -96.0% 2630 ? 52% interrupts.CPU34.RES:Rescheduling_interrupts
136901 ?122% -99.1% 1278 ? 30% interrupts.CPU35.CAL:Function_call_interrupts
731054 ? 69% -99.9% 1046 ? 15% interrupts.CPU36.CAL:Function_call_interrupts
376136 ? 70% -99.6% 1429 ? 37% interrupts.CPU36.RES:Rescheduling_interrupts
941442 ?105% -99.9% 851.33 ? 7% interrupts.CPU37.CAL:Function_call_interrupts
300885 ? 80% -99.2% 2379 ? 57% interrupts.CPU37.RES:Rescheduling_interrupts
242878 ?117% -99.6% 995.00 ? 16% interrupts.CPU38.CAL:Function_call_interrupts
702836 ? 69% -99.9% 855.00 ? 3% interrupts.CPU39.CAL:Function_call_interrupts
286450 ? 74% -99.5% 1448 ? 70% interrupts.CPU39.RES:Rescheduling_interrupts
37564 ? 55% -96.0% 1505 ? 62% interrupts.CPU4.CAL:Function_call_interrupts
6346 ? 26% +24.0% 7870 ? 6% interrupts.CPU4.NMI:Non-maskable_interrupts
6346 ? 26% +24.0% 7870 ? 6% interrupts.CPU4.PMI:Performance_monitoring_interrupts
24154 ? 65% -94.0% 1444 ? 83% interrupts.CPU4.RES:Rescheduling_interrupts
977103 ? 53% -99.9% 837.33 ? 3% interrupts.CPU40.CAL:Function_call_interrupts
512103 ? 71% -99.7% 1579 ? 66% interrupts.CPU40.RES:Rescheduling_interrupts
190023 ?117% -99.6% 818.67 ? 2% interrupts.CPU41.CAL:Function_call_interrupts
346179 ? 54% -99.8% 829.00 ? 2% interrupts.CPU42.CAL:Function_call_interrupts
183103 ? 48% -99.5% 850.00 ? 54% interrupts.CPU42.RES:Rescheduling_interrupts
371814 ?131% -99.8% 863.33 ? 5% interrupts.CPU43.CAL:Function_call_interrupts
158123 ?124% -99.2% 1224 ? 71% interrupts.CPU43.RES:Rescheduling_interrupts
578077 ? 61% -99.9% 850.67 ? 3% interrupts.CPU44.CAL:Function_call_interrupts
331471 ? 53% -99.7% 1120 ? 91% interrupts.CPU44.RES:Rescheduling_interrupts
172794 ?114% -99.5% 838.33 ? 4% interrupts.CPU45.CAL:Function_call_interrupts
89408 ?111% -99.6% 326.00 ? 23% interrupts.CPU45.RES:Rescheduling_interrupts
34858 ? 59% -97.3% 947.00 ? 13% interrupts.CPU46.CAL:Function_call_interrupts
22612 ? 53% -87.6% 2794 ? 27% interrupts.CPU46.RES:Rescheduling_interrupts
41349 ? 57% -97.6% 996.67 ? 10% interrupts.CPU47.CAL:Function_call_interrupts
27498 ? 54% -93.7% 1725 ? 89% interrupts.CPU47.RES:Rescheduling_interrupts
45388 ? 61% -97.5% 1141 ? 57% interrupts.CPU5.CAL:Function_call_interrupts
32500 ? 63% -96.7% 1063 ? 95% interrupts.CPU5.RES:Rescheduling_interrupts
534772 ?105% -99.8% 843.33 ? 3% interrupts.CPU6.CAL:Function_call_interrupts
259978 ? 94% -98.5% 4008 ? 4% interrupts.CPU6.RES:Rescheduling_interrupts
589336 ? 71% -99.8% 1042 ? 25% interrupts.CPU7.CAL:Function_call_interrupts
311886 ? 69% -99.4% 1966 ? 66% interrupts.CPU7.RES:Rescheduling_interrupts
363153 ? 95% -99.8% 858.00 ? 5% interrupts.CPU8.CAL:Function_call_interrupts
190915 ? 83% -98.1% 3681 ? 94% interrupts.CPU8.RES:Rescheduling_interrupts
78806 ? 46% -98.9% 905.33 ? 7% interrupts.CPU9.CAL:Function_call_interrupts
47384 ? 29% -95.9% 1963 ? 58% interrupts.CPU9.RES:Rescheduling_interrupts
9209745 ? 15% -99.0% 95188 ? 9% interrupts.RES:Rescheduling_interrupts
20.15 ? 4% -14.2 5.98 ? 15% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
15.92 ? 4% -11.9 3.98 ? 7% perf-profile.calltrace.cycles-pp.signal_wake_up_state.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
15.71 ? 4% -11.8 3.95 ? 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info.kill_pid_info
13.68 ? 6% -10.8 2.93 ? 10% perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
12.23 ? 26% -10.4 1.80 ? 38% perf-profile.calltrace.cycles-pp.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64
11.55 ? 27% -10.0 1.56 ? 39% perf-profile.calltrace.cycles-pp.security_task_kill.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
11.33 ? 27% -9.8 1.49 ? 40% perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
11.78 ? 5% -9.3 2.52 ? 9% perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
11.53 ? 5% -9.1 2.47 ? 9% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait
8.27 ? 13% -6.3 1.99 ? 7% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
7.32 ? 2% -4.9 2.43 ? 25% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
6.20 ? 13% -4.7 1.53 ? 10% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
5.94 ? 13% -4.5 1.46 ? 10% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
6.32 ? 8% -4.3 2.07 ? 6% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info
4.57 ? 30% -4.1 0.44 ? 74% perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.group_send_sig_info.kill_pid_info
5.55 ? 8% -3.7 1.90 ? 6% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.signal_wake_up_state.__send_signal
4.43 ? 7% -3.4 0.99 ? 6% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info
4.29 ? 7% -3.3 0.95 ? 6% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.signal_wake_up_state.__send_signal
4.19 ? 6% -3.3 0.92 ? 8% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait
4.60 -2.9 1.69 ? 28% perf-profile.calltrace.cycles-pp.__entry_text_start
1.93 ? 16% -1.6 0.37 ? 70% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode
2.40 ? 6% -1.5 0.86 ? 6% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.signal_wake_up_state
0.00 +0.7 0.67 perf-profile.calltrace.cycles-pp.inc_rlimit_ucounts_and_test.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info
0.67 ? 16% +2.4 3.11 ? 53% perf-profile.calltrace.cycles-pp.__lock_task_sighand.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
0.42 ? 71% +2.7 3.08 ? 53% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__lock_task_sighand.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
0.00 +3.0 2.98 ? 54% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__lock_task_sighand.do_send_sig_info.kill_pid_info
0.00 +3.2 3.23 ? 51% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
0.00 +3.3 3.29 ? 51% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
35.47 ? 7% +9.5 45.02 perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
34.17 ? 7% +10.4 44.54 perf-profile.calltrace.cycles-pp.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
33.33 ? 7% +10.9 44.22 perf-profile.calltrace.cycles-pp.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
81.67 +12.6 94.25 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
19.16 ? 4% +19.5 38.64 ? 8% perf-profile.calltrace.cycles-pp.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
21.31 ? 6% +20.5 41.77 ? 4% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
20.83 ? 3% +21.5 42.33 ? 3% perf-profile.calltrace.cycles-pp.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64
19.46 ? 5% +21.9 41.31 ? 4% perf-profile.calltrace.cycles-pp.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
59.72 ? 2% +28.1 87.78 ? 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.78 ? 8% +30.8 34.58 ? 9% perf-profile.calltrace.cycles-pp.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.07 ? 8% +31.3 34.40 ? 9% perf-profile.calltrace.cycles-pp.__dequeue_signal.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
1.74 ? 9% +32.1 33.86 ? 9% perf-profile.calltrace.cycles-pp.__sigqueue_free.__dequeue_signal.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait
1.94 ? 8% +32.5 34.40 ? 8% perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
0.00 +32.8 32.81 ? 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.__dequeue_signal
0.00 +33.0 33.02 ? 8% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal
0.00 +33.2 33.22 ? 9% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.__dequeue_signal.dequeue_signal
0.00 +33.3 33.32 ? 9% perf-profile.calltrace.cycles-pp.put_ucounts.__sigqueue_free.__dequeue_signal.dequeue_signal.do_sigtimedwait
0.00 +33.5 33.48 ? 8% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal.do_send_sig_info
0.00 +33.6 33.57 ? 8% perf-profile.calltrace.cycles-pp.get_ucounts.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info
20.22 ? 4% -14.2 6.00 ? 15% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
18.14 ? 8% -14.1 4.06 ? 10% perf-profile.children.cycles-pp.schedule
17.92 ? 7% -14.0 3.96 ? 9% perf-profile.children.cycles-pp.__sched_text_start
15.92 ? 4% -11.9 3.98 ? 7% perf-profile.children.cycles-pp.signal_wake_up_state
15.73 ? 4% -11.8 3.96 ? 7% perf-profile.children.cycles-pp.try_to_wake_up
13.71 ? 6% -10.8 2.93 ? 10% perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
12.24 ? 26% -10.4 1.80 ? 38% perf-profile.children.cycles-pp.group_send_sig_info
11.56 ? 27% -10.0 1.56 ? 39% perf-profile.children.cycles-pp.security_task_kill
11.35 ? 27% -9.9 1.49 ? 40% perf-profile.children.cycles-pp.apparmor_task_kill
8.45 ? 12% -6.5 2.00 ? 7% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
8.04 -5.3 2.71 ? 25% perf-profile.children.cycles-pp.syscall_return_via_sysret
6.34 ? 8% -4.3 2.07 ? 6% perf-profile.children.cycles-pp.select_task_rq_fair
4.58 ? 30% -4.0 0.55 ? 36% perf-profile.children.cycles-pp.aa_get_task_label
6.03 -4.0 2.05 ? 28% perf-profile.children.cycles-pp.__entry_text_start
4.70 ? 6% -3.7 0.99 ? 6% perf-profile.children.cycles-pp.ttwu_do_activate
5.63 ? 8% -3.7 1.93 ? 6% perf-profile.children.cycles-pp.select_idle_sibling
4.57 ? 6% -3.6 0.96 ? 6% perf-profile.children.cycles-pp.enqueue_task_fair
4.23 ? 7% -3.3 0.93 ? 8% perf-profile.children.cycles-pp.dequeue_task_fair
3.78 ? 12% -2.9 0.91 ? 11% perf-profile.children.cycles-pp.pick_next_task_fair
3.44 ? 9% -2.7 0.78 ? 9% perf-profile.children.cycles-pp.switch_mm_irqs_off
3.07 ? 10% -2.4 0.68 ? 9% perf-profile.children.cycles-pp.update_load_avg
2.96 ? 3% -2.3 0.69 ? 9% perf-profile.children.cycles-pp.update_curr
2.35 ? 3% -1.9 0.48 ? 6% perf-profile.children.cycles-pp.enqueue_entity
2.26 ? 13% -1.7 0.56 ? 10% perf-profile.children.cycles-pp.load_new_mm_cr3
2.44 ? 6% -1.6 0.86 ? 5% perf-profile.children.cycles-pp.available_idle_cpu
1.82 ? 4% -1.4 0.41 ? 7% perf-profile.children.cycles-pp.dequeue_entity
1.93 ? 3% -1.4 0.52 ? 32% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
1.63 ? 10% -1.3 0.28 ? 8% perf-profile.children.cycles-pp._raw_spin_lock
1.59 ? 13% -1.3 0.30 ? 10% perf-profile.children.cycles-pp.switch_fpu_return
1.63 ? 11% -1.2 0.39 ? 10% perf-profile.children.cycles-pp.reweight_entity
1.47 ? 7% -1.2 0.31 ? 11% perf-profile.children.cycles-pp.__switch_to
1.25 ? 12% -1.0 0.27 ? 13% perf-profile.children.cycles-pp.__switch_to_asm
1.21 ? 10% -0.9 0.27 ? 13% perf-profile.children.cycles-pp.set_next_entity
1.07 ? 7% -0.9 0.13 ? 25% perf-profile.children.cycles-pp.finish_task_switch
1.30 ? 4% -0.9 0.41 ? 10% perf-profile.children.cycles-pp._copy_from_user
1.11 ? 7% -0.8 0.27 ? 12% perf-profile.children.cycles-pp.update_rq_clock
1.04 ? 4% -0.8 0.24 ? 15% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.99 ? 13% -0.8 0.21 ? 14% perf-profile.children.cycles-pp.hrtimer_start_range_ns
0.98 ? 4% -0.8 0.22 ? 13% perf-profile.children.cycles-pp.check_preempt_curr
1.03 -0.8 0.28 ? 4% perf-profile.children.cycles-pp.recalc_sigpending
0.97 ? 12% -0.7 0.22 ? 10% perf-profile.children.cycles-pp.__update_load_avg_se
0.90 ? 8% -0.7 0.18 ? 12% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
0.95 ? 10% -0.7 0.23 ? 10% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.94 ? 33% -0.7 0.23 ? 40% perf-profile.children.cycles-pp.aa_may_signal
0.94 ? 6% -0.7 0.23 ? 2% perf-profile.children.cycles-pp.copy_siginfo_to_user
0.95 ? 6% -0.7 0.27 ? 9% perf-profile.children.cycles-pp.__might_fault
0.85 ? 5% -0.7 0.19 ? 11% perf-profile.children.cycles-pp.check_preempt_wakeup
0.83 ? 9% -0.6 0.19 ? 7% perf-profile.children.cycles-pp.___perf_sw_event
1.00 ? 2% -0.6 0.36 ? 20% perf-profile.children.cycles-pp.__copy_siginfo_from_user
0.79 ? 17% -0.6 0.19 ? 11% perf-profile.children.cycles-pp.put_prev_entity
0.80 ? 12% -0.6 0.20 ? 8% perf-profile.children.cycles-pp.pick_next_entity
0.82 ? 2% -0.5 0.29 ? 24% perf-profile.children.cycles-pp.__x64_sys_getpid
0.68 ? 2% -0.5 0.16 ? 3% perf-profile.children.cycles-pp.update_cfs_group
0.87 ? 17% -0.5 0.38 perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
0.53 ? 6% -0.5 0.06 ? 16% perf-profile.children.cycles-pp.complete_signal
0.58 ? 6% -0.5 0.12 ? 10% perf-profile.children.cycles-pp.sched_clock_cpu
0.55 ? 5% -0.4 0.10 ? 12% perf-profile.children.cycles-pp.hrtimer_try_to_cancel
0.79 ? 12% -0.4 0.35 ? 7% perf-profile.children.cycles-pp.cpumask_next_wrap
0.62 ? 18% -0.4 0.19 ? 10% perf-profile.children.cycles-pp._find_next_bit
0.71 ? 2% -0.4 0.28 ? 34% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.62 ? 3% -0.4 0.21 ? 30% perf-profile.children.cycles-pp.__x64_sys_getuid
0.53 ? 14% -0.4 0.14 ? 17% perf-profile.children.cycles-pp.__calc_delta
0.59 ? 8% -0.4 0.20 ? 32% perf-profile.children.cycles-pp.check_kill_permission
0.50 ? 6% -0.4 0.11 ? 11% perf-profile.children.cycles-pp.sched_clock
0.50 ? 13% -0.4 0.12 ? 10% perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
0.47 ? 6% -0.4 0.09 ? 18% perf-profile.children.cycles-pp.perf_tp_event
0.48 ? 6% -0.4 0.10 ? 9% perf-profile.children.cycles-pp.native_sched_clock
0.48 ? 3% -0.4 0.11 ? 4% perf-profile.children.cycles-pp.___might_sleep
0.74 ? 12% -0.4 0.39 perf-profile.children.cycles-pp.kmem_cache_free
0.45 ? 8% -0.3 0.13 ? 13% perf-profile.children.cycles-pp._copy_to_user
0.36 ? 5% -0.3 0.06 ? 8% perf-profile.children.cycles-pp.__set_task_blocked
0.35 ? 22% -0.3 0.07 ? 11% perf-profile.children.cycles-pp.cpumask_next
0.38 ? 7% -0.3 0.10 ? 14% perf-profile.children.cycles-pp.ktime_get
0.46 ? 10% -0.3 0.19 ? 36% perf-profile.children.cycles-pp.__radix_tree_lookup
0.64 ? 4% -0.3 0.37 ? 6% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.33 ? 16% -0.3 0.06 ? 14% perf-profile.children.cycles-pp.cpuacct_charge
0.53 -0.3 0.26 ? 33% perf-profile.children.cycles-pp.send_signal
0.33 ? 16% -0.3 0.07 ? 12% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.34 ? 8% -0.2 0.09 ? 5% perf-profile.children.cycles-pp.update_min_vruntime
0.39 ? 3% -0.2 0.15 ? 12% perf-profile.children.cycles-pp.copy_user_generic_unrolled
0.32 ? 6% -0.2 0.07 ? 6% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.28 ? 2% -0.2 0.04 ? 71% perf-profile.children.cycles-pp.lock_hrtimer_base
0.30 ? 9% -0.2 0.06 ? 7% perf-profile.children.cycles-pp.set_next_buddy
0.35 ? 5% -0.2 0.11 ? 36% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.30 ? 5% -0.2 0.07 ? 17% perf-profile.children.cycles-pp.rb_erase
0.32 ? 2% -0.2 0.10 ? 9% perf-profile.children.cycles-pp.__x86_retpoline_rax
0.30 ? 5% -0.2 0.08 ? 14% perf-profile.children.cycles-pp.__might_sleep
0.29 ? 7% -0.2 0.07 ? 12% perf-profile.children.cycles-pp.__clear_user
0.31 ? 14% -0.2 0.09 ? 9% perf-profile.children.cycles-pp.clear_buddies
0.25 ? 3% -0.2 0.05 perf-profile.children.cycles-pp.recalc_sigpending_tsk
0.29 ? 12% -0.2 0.09 ? 22% perf-profile.children.cycles-pp.prepare_signal
0.30 ? 4% -0.2 0.11 ? 34% perf-profile.children.cycles-pp.from_kuid_munged
0.27 ? 4% -0.2 0.09 ? 36% perf-profile.children.cycles-pp.map_id_up
0.21 ? 21% -0.2 0.03 ? 70% perf-profile.children.cycles-pp.enqueue_hrtimer
0.22 ? 13% -0.2 0.06 ? 8% perf-profile.children.cycles-pp.get_timespec64
0.22 ? 17% -0.2 0.06 ? 86% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.20 ? 4% -0.2 0.04 ? 76% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
0.20 ? 6% -0.2 0.05 perf-profile.children.cycles-pp.__list_del_entry_valid
0.19 ? 7% -0.2 0.04 ? 71% perf-profile.children.cycles-pp.perf_trace_sched_switch
0.23 ? 8% -0.1 0.08 ? 31% perf-profile.children.cycles-pp.audit_signal_info
0.22 ? 5% -0.1 0.08 ? 16% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.17 ? 9% -0.1 0.03 ? 70% perf-profile.children.cycles-pp.read_tsc
0.18 ? 7% -0.1 0.05 ? 72% perf-profile.children.cycles-pp.audit_signal_info_syscall
0.17 ? 5% -0.1 0.11 ? 4% perf-profile.children.cycles-pp.kmem_cache_alloc
0.00 +0.5 0.51 ? 3% perf-profile.children.cycles-pp.dec_rlimit_ucounts
0.00 +0.7 0.67 ? 2% perf-profile.children.cycles-pp.inc_rlimit_ucounts_and_test
0.68 ? 16% +2.4 3.11 ? 53% perf-profile.children.cycles-pp.__lock_task_sighand
0.42 ? 6% +2.9 3.30 ? 51% perf-profile.children.cycles-pp._raw_spin_lock_irq
35.50 ? 7% +9.5 45.03 perf-profile.children.cycles-pp.__x64_sys_rt_sigqueueinfo
34.19 ? 7% +10.4 44.55 perf-profile.children.cycles-pp.do_rt_sigqueueinfo
33.35 ? 7% +10.9 44.22 perf-profile.children.cycles-pp.kill_pid_info
81.79 +12.5 94.33 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
19.21 ? 4% +19.4 38.65 ? 8% perf-profile.children.cycles-pp.__send_signal
21.36 ? 6% +20.4 41.78 ? 4% perf-profile.children.cycles-pp.__x64_sys_rt_sigtimedwait
20.86 ? 3% +21.5 42.34 ? 3% perf-profile.children.cycles-pp.do_send_sig_info
19.50 ? 5% +21.8 41.33 ? 4% perf-profile.children.cycles-pp.do_sigtimedwait
59.88 ? 2% +28.0 87.85 ? 2% perf-profile.children.cycles-pp.do_syscall_64
3.81 ? 8% +30.9 34.66 ? 9% perf-profile.children.cycles-pp.dequeue_signal
3.09 ? 8% +31.3 34.41 ? 9% perf-profile.children.cycles-pp.__dequeue_signal
1.74 ? 9% +32.1 33.87 ? 9% perf-profile.children.cycles-pp.__sigqueue_free
1.95 ? 8% +32.5 34.40 ? 8% perf-profile.children.cycles-pp.__sigqueue_alloc
0.00 +33.3 33.32 ? 9% perf-profile.children.cycles-pp.put_ucounts
0.00 +33.6 33.58 ? 8% perf-profile.children.cycles-pp.get_ucounts
1.07 ? 11% +68.8 69.87 ? 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.74 ? 39% +71.3 72.04 ? 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
11.75 ? 2% -7.8 3.95 ? 26% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
8.02 -5.3 2.70 ? 25% perf-profile.self.cycles-pp.syscall_return_via_sysret
5.78 ? 25% -5.1 0.71 ? 43% perf-profile.self.cycles-pp.apparmor_task_kill
4.55 ? 30% -4.0 0.53 ? 34% perf-profile.self.cycles-pp.aa_get_task_label
6.02 -4.0 2.05 ? 28% perf-profile.self.cycles-pp.__entry_text_start
1.79 ? 9% -1.7 0.06 ? 13% perf-profile.self.cycles-pp.__sigqueue_alloc
2.22 ? 9% -1.7 0.50 ? 9% perf-profile.self.cycles-pp.__sched_text_start
2.25 ? 13% -1.7 0.55 ? 11% perf-profile.self.cycles-pp.load_new_mm_cr3
2.40 ? 6% -1.5 0.85 ? 6% perf-profile.self.cycles-pp.available_idle_cpu
1.57 ? 12% -1.3 0.30 ? 10% perf-profile.self.cycles-pp.switch_fpu_return
1.74 ? 6% -1.2 0.50 ? 11% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
1.79 ? 8% -1.2 0.60 ? 6% perf-profile.self.cycles-pp.select_idle_sibling
1.41 ? 7% -1.1 0.30 ? 11% perf-profile.self.cycles-pp.__switch_to
1.25 ? 3% -1.0 0.27 ? 8% perf-profile.self.cycles-pp.update_curr
1.23 ? 12% -1.0 0.27 ? 13% perf-profile.self.cycles-pp.__switch_to_asm
1.17 ? 7% -0.9 0.23 ? 9% perf-profile.self.cycles-pp.update_load_avg
1.15 ? 2% -0.9 0.21 ? 5% perf-profile.self.cycles-pp.switch_mm_irqs_off
1.13 ? 3% -0.8 0.28 ? 8% perf-profile.self.cycles-pp._raw_spin_lock
0.94 ? 13% -0.8 0.14 ? 9% perf-profile.self.cycles-pp.try_to_wake_up
1.06 -0.7 0.32 ? 28% perf-profile.self.cycles-pp.do_syscall_64
0.93 ? 12% -0.7 0.22 ? 12% perf-profile.self.cycles-pp.__update_load_avg_se
0.93 ? 33% -0.7 0.23 ? 40% perf-profile.self.cycles-pp.aa_may_signal
0.90 ? 10% -0.7 0.22 ? 9% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.77 -0.6 0.15 ? 5% perf-profile.self.cycles-pp.recalc_sigpending
0.77 ? 13% -0.6 0.20 ? 7% perf-profile.self.cycles-pp.reweight_entity
0.74 ? 7% -0.6 0.19 ? 11% perf-profile.self.cycles-pp.update_rq_clock
0.67 ? 7% -0.5 0.14 ? 6% perf-profile.self.cycles-pp.___perf_sw_event
0.63 ? 7% -0.5 0.11 ? 11% perf-profile.self.cycles-pp.enqueue_entity
0.67 ? 3% -0.5 0.15 ? 3% perf-profile.self.cycles-pp.update_cfs_group
0.66 ? 9% -0.5 0.16 ? 12% perf-profile.self.cycles-pp.pick_next_task_fair
0.61 ? 13% -0.5 0.12 ? 4% perf-profile.self.cycles-pp.enqueue_task_fair
0.60 ? 5% -0.5 0.11 ? 14% perf-profile.self.cycles-pp.dequeue_task_fair
0.62 ? 8% -0.5 0.14 ? 5% perf-profile.self.cycles-pp.select_task_rq_fair
0.63 ? 6% -0.5 0.16 ? 10% perf-profile.self.cycles-pp.do_sigtimedwait
0.61 ? 18% -0.4 0.19 ? 10% perf-profile.self.cycles-pp._find_next_bit
0.68 ? 3% -0.4 0.27 ? 33% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.54 ? 3% -0.4 0.14 ? 6% perf-profile.self.cycles-pp.__dequeue_signal
0.51 -0.4 0.12 ? 20% perf-profile.self.cycles-pp.__send_signal
0.53 ? 14% -0.4 0.14 ? 15% perf-profile.self.cycles-pp.__calc_delta
0.49 ? 13% -0.4 0.12 ? 10% perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
0.46 ? 7% -0.4 0.09 ? 10% perf-profile.self.cycles-pp.native_sched_clock
0.46 ? 3% -0.4 0.10 ? 4% perf-profile.self.cycles-pp.___might_sleep
0.50 ? 7% -0.3 0.16 ? 23% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.50 ? 3% -0.3 0.17 ? 29% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.44 ? 14% -0.3 0.10 ? 16% perf-profile.self.cycles-pp.schedule
0.42 ? 5% -0.3 0.09 ? 9% perf-profile.self.cycles-pp.finish_task_switch
0.43 ? 11% -0.3 0.10 ? 9% perf-profile.self.cycles-pp.pick_next_entity
0.41 ? 10% -0.3 0.08 ? 5% perf-profile.self.cycles-pp.__x64_sys_rt_sigtimedwait
0.44 ? 6% -0.3 0.13 ? 6% perf-profile.self.cycles-pp._copy_from_user
0.50 ? 3% -0.3 0.20 ? 11% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.63 ? 20% -0.3 0.35 ? 6% perf-profile.self.cycles-pp.kmem_cache_free
0.37 ? 3% -0.3 0.10 ? 12% perf-profile.self.cycles-pp.check_preempt_wakeup
0.37 ? 7% -0.3 0.10 ? 4% perf-profile.self.cycles-pp.dequeue_entity
0.33 ? 15% -0.3 0.06 ? 13% perf-profile.self.cycles-pp.cpuacct_charge
0.46 ? 11% -0.3 0.19 ? 36% perf-profile.self.cycles-pp.__radix_tree_lookup
0.34 ? 3% -0.3 0.07 ? 11% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.29 ? 7% -0.3 0.04 ? 71% perf-profile.self.cycles-pp.perf_tp_event
0.36 ? 11% -0.3 0.10 ? 4% perf-profile.self.cycles-pp.__might_fault
0.32 ? 7% -0.2 0.08 ? 5% perf-profile.self.cycles-pp.update_min_vruntime
0.31 ? 17% -0.2 0.07 ? 12% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.37 ? 2% -0.2 0.14 ? 13% perf-profile.self.cycles-pp.copy_user_generic_unrolled
0.30 ? 4% -0.2 0.07 ? 7% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.26 ? 8% -0.2 0.04 ? 71% perf-profile.self.cycles-pp.set_next_buddy
0.29 ? 5% -0.2 0.07 ? 17% perf-profile.self.cycles-pp.rb_erase
0.28 ? 11% -0.2 0.06 ? 14% perf-profile.self.cycles-pp.set_next_entity
0.30 ? 4% -0.2 0.09 ? 30% perf-profile.self.cycles-pp.__x64_sys_getuid
0.39 ? 13% -0.2 0.18 ? 9% perf-profile.self.cycles-pp.cpumask_next_wrap
0.27 ? 5% -0.2 0.07 ? 17% perf-profile.self.cycles-pp.__might_sleep
0.31 ? 3% -0.2 0.12 ? 28% perf-profile.self.cycles-pp.__x64_sys_rt_sigqueueinfo
0.24 -0.2 0.05 perf-profile.self.cycles-pp.recalc_sigpending_tsk
0.28 ? 5% -0.2 0.09 ? 36% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.27 ? 12% -0.2 0.08 ? 24% perf-profile.self.cycles-pp.prepare_signal
0.27 ? 13% -0.2 0.08 ? 10% perf-profile.self.cycles-pp.clear_buddies
0.26 ? 3% -0.2 0.08 ? 12% perf-profile.self.cycles-pp.__x86_retpoline_rax
0.21 ? 17% -0.2 0.04 ? 71% perf-profile.self.cycles-pp.put_prev_entity
0.26 ? 4% -0.2 0.09 ? 36% perf-profile.self.cycles-pp.map_id_up
0.23 ? 13% -0.2 0.07 ? 11% perf-profile.self.cycles-pp.schedule_hrtimeout_range_clock
0.19 ? 8% -0.2 0.04 ? 70% perf-profile.self.cycles-pp.ktime_get
0.19 ? 9% -0.1 0.04 ? 71% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.18 ? 4% -0.1 0.03 ? 70% perf-profile.self.cycles-pp.__list_del_entry_valid
0.19 ? 8% -0.1 0.05 ? 78% perf-profile.self.cycles-pp.check_kill_permission
0.26 -0.1 0.13 ? 26% perf-profile.self.cycles-pp.send_signal
0.21 ? 4% -0.1 0.08 ? 26% perf-profile.self.cycles-pp.__x64_sys_getpid
0.20 ? 2% -0.1 0.06 ? 19% perf-profile.self.cycles-pp.security_task_kill
0.17 ? 4% -0.1 0.05 ? 78% perf-profile.self.cycles-pp.kill_pid_info
0.16 ? 8% -0.1 0.04 ? 71% perf-profile.self.cycles-pp.rcu_read_unlock_strict
0.16 ? 10% -0.1 0.05 ? 72% perf-profile.self.cycles-pp.audit_signal_info_syscall
0.24 ? 8% -0.1 0.14 ? 3% perf-profile.self.cycles-pp.dequeue_signal
0.12 ? 4% -0.1 0.04 ? 71% perf-profile.self.cycles-pp.do_rt_sigqueueinfo
0.12 ? 11% -0.1 0.04 ? 70% perf-profile.self.cycles-pp.do_send_sig_info
0.15 ? 6% -0.0 0.10 perf-profile.self.cycles-pp.kmem_cache_alloc
0.00 +0.1 0.07 ? 11% perf-profile.self.cycles-pp.get_ucounts
0.00 +0.1 0.07 ? 6% perf-profile.self.cycles-pp.put_ucounts
0.88 ? 7% +0.2 1.04 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.00 +0.5 0.51 ? 3% perf-profile.self.cycles-pp.dec_rlimit_ucounts
0.00 +0.7 0.67 perf-profile.self.cycles-pp.inc_rlimit_ucounts_and_test
0.74 ? 39% +71.3 72.04 ? 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang

---

Attachments:

(No filename) (90.55 kB)
config-5.11.0-rc7-00016-gd28296d2484f (175.11 kB)
job-script (8.19 kB)
job.yaml (5.62 kB)
reproduce (421.00 B)
Download all attachments

2021-02-24 18:42:17

by Alexey Gladkov

[permalink] [raw]

Subject: Re: d28296d248: stress-ng.sigsegv.ops_per_sec -82.7% regression

---

On Wed, Feb 24, 2021 at 10:54:17AM -0600, Eric W. Biederman wrote:
> kernel test robot <[email protected]> writes:
>
> > Greeting,
> >
> > FYI, we noticed a -82.7% regression of stress-ng.sigsegv.ops_per_sec due to commit:
> >
> >
> > commit: d28296d2484fa11e94dff65e93eb25802a443d47 ("[PATCH v7 5/7] Reimplement RLIMIT_SIGPENDING on top of ucounts")
> > url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
> > base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next
> >
> > in testcase: stress-ng
> > on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
> > with following parameters:
> >
> > nr_threads: 100%
> > disk: 1HDD
> > testtime: 60s
> > class: interrupt
> > test: sigsegv
> > cpufreq_governor: performance
> > ucode: 0x42e
> >
> >
> > In addition to that, the commit also has significant impact on the
> > following tests:
>
> Thank you. Now we have a sense of where we need to test the performance
> of these changes carefully.

One of the reasons for this is that I rolled back the patch that changed
the ucounts.count type to atomic_t. Now get_ucounts() is forced to use a
spin_lock to increase the reference count.

--
Rgrds, legion

2021-02-24 18:54:23

by Eric W. Biederman

[permalink] [raw]

Subject: Re: d28296d248: stress-ng.sigsegv.ops_per_sec -82.7% regression

---

Alexey Gladkov <[email protected]> writes:

> On Wed, Feb 24, 2021 at 10:54:17AM -0600, Eric W. Biederman wrote:
>> kernel test robot <[email protected]> writes:
>>
>> > Greeting,
>> >
>> > FYI, we noticed a -82.7% regression of stress-ng.sigsegv.ops_per_sec due to commit:
>> >
>> >
>> > commit: d28296d2484fa11e94dff65e93eb25802a443d47 ("[PATCH v7 5/7] Reimplement RLIMIT_SIGPENDING on top of ucounts")
>> > url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
>> > base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next
>> >
>> > in testcase: stress-ng
>> > on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
>> > with following parameters:
>> >
>> > nr_threads: 100%
>> > disk: 1HDD
>> > testtime: 60s
>> > class: interrupt
>> > test: sigsegv
>> > cpufreq_governor: performance
>> > ucode: 0x42e
>> >
>> >
>> > In addition to that, the commit also has significant impact on the
>> > following tests:
>>
>> Thank you. Now we have a sense of where we need to test the performance
>> of these changes carefully.
>
> One of the reasons for this is that I rolled back the patch that changed
> the ucounts.count type to atomic_t. Now get_ucounts() is forced to use a
> spin_lock to increase the reference count.

Which given the hickups with getting a working version seems justified.

Now we can add incremental patches on top to improve the performance.


Eric

2021-02-24 19:22:21

by Linus Torvalds

[permalink] [raw]

Subject: Re: d28296d248: stress-ng.sigsegv.ops_per_sec -82.7% regression

---

On Wed, Feb 24, 2021 at 10:38 AM Alexey Gladkov
<[email protected]> wrote:
>
> One of the reasons for this is that I rolled back the patch that changed
> the ucounts.count type to atomic_t. Now get_ucounts() is forced to use a
> spin_lock to increase the reference count.

Yeah, that definitely should be an atomic type, since the extended use
of ucounts clearly puts way too much pressure on that ucount lock.

I remember complaining about one version of that patch, but my
complaint wasabout it changing semantics of the saturation logic (and
I think it was also wrong because it still kept the spinlock for
get_ucounts(), so it didn't even take advantage of the atomic
refcount).

Side note: I don't think a refcount_t" is necessarily the right thing
to do, since the ucount reference counter does its own saturation
logic, and the refcount_t version is imho not great.

So it probably just needs to use an atomic_t, and do the saturation
thing manually.

Side note: look at try_get_page(). That one actually does refcounting
with overflow protection better than refcount_t, in my opinion. But I
am obviously biased, since I wrote it ;)

See commits

88b1a17dfc3e mm: add 'try_get_page()' helper function
f958d7b528b1 mm: make page ref count overflow check tighter and
more explicit

with that "page->_recount" being just a regular atomic_t.

Linus

2021-02-25 01:14:26

by Eric W. Biederman

[permalink] [raw]

Subject: Re: d28296d248: stress-ng.sigsegv.ops_per_sec -82.7% regression

---

kernel test robot <[email protected]> writes:

> Greeting,
>
> FYI, we noticed a -82.7% regression of stress-ng.sigsegv.ops_per_sec due to commit:
>
>
> commit: d28296d2484fa11e94dff65e93eb25802a443d47 ("[PATCH v7 5/7] Reimplement RLIMIT_SIGPENDING on top of ucounts")
> url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
> base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next
>
> in testcase: stress-ng
> on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
> with following parameters:
>
> nr_threads: 100%
> disk: 1HDD
> testtime: 60s
> class: interrupt
> test: sigsegv
> cpufreq_governor: performance
> ucode: 0x42e
>
>
> In addition to that, the commit also has significant impact on the
> following tests:

Thank you. Now we have a sense of where we need to test the performance
of these changes carefully.

Eric


> +------------------+-----------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.sigq.ops_per_sec -56.1% regression |
> | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory |
> | test parameters | class=interrupt |
> | | cpufreq_governor=performance |
> | | disk=1HDD |
> | | nr_threads=100% |
> | | test=sigq |
> | | testtime=60s |
> | | ucode=0x42e |
> +------------------+-----------------------------------------------------------------------+
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <[email protected]>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp split-job --compatible job.yaml
> bin/lkp run compatible-job.yaml
>
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
> interrupt/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/sigsegv/stress-ng/60s/0x42e
>
> commit:
> 4660d663b4 ("Reimplement RLIMIT_MSGQUEUE on top of ucounts")
> d28296d248 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")
>
> 4660d663b4207ce6 d28296d2484fa11e94dff65e93e
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> 14:6 -217% 1:6 perf-profile.children.cycles-pp.error_entry
> 12:6 -179% 1:6 perf-profile.self.cycles-pp.error_entry
> %stddev %change %stddev
> \ | \
> 4.766e+08 -82.7% 82358308 stress-ng.sigsegv.ops
> 7942807 -82.7% 1372639 stress-ng.sigsegv.ops_per_sec
> 29408 -77.5% 6621 ± 61% stress-ng.time.file_system_inputs
> 1566 +69.4% 2653 stress-ng.time.system_time
> 1274 -84.8% 193.22 ± 8% stress-ng.time.user_time
> 12458 ± 5% +37.2% 17097 ± 5% numa-meminfo.node1.Active(anon)
> 51.41 +66.5% 85.59 iostat.cpu.system
> 41.17 -84.2% 6.50 ± 7% iostat.cpu.user
> 3040 ± 4% +37.9% 4193 ± 4% numa-vmstat.node1.nr_active_anon
> 3040 ± 4% +37.9% 4193 ± 4% numa-vmstat.node1.nr_zone_active_anon
> 50.83 +67.2% 85.00 vmstat.cpu.sy
> 40.50 -85.6% 5.83 ± 11% vmstat.cpu.us
> 225.33 -77.7% 50.33 ± 62% vmstat.io.bi
> 7.00 -100.0% 0.00 vmstat.memory.buff
> 20735 ± 2% -14.1% 17812 ± 5% meminfo.Active
> 13506 ± 3% +31.9% 17812 ± 5% meminfo.Active(anon)
> 7228 -100.0% 0.00 meminfo.Active(file)
> 29308 +18.4% 34687 ± 2% meminfo.Shmem
> 202067 -9.5% 182899 meminfo.VmallocUsed
> 0.01 ± 17% -0.0 0.00 ± 10% mpstat.cpu.all.iowait%
> 1.04 -0.1 0.92 mpstat.cpu.all.irq%
> 0.03 ± 8% -0.0 0.02 ± 4% mpstat.cpu.all.soft%
> 51.54 +35.6 87.17 mpstat.cpu.all.sys%
> 42.22 -35.6 6.66 ± 8% mpstat.cpu.all.usr%
> 0.00 ± 70% +191.7% 0.01 ± 26% perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
> 0.00 ± 47% +158.8% 0.01 ± 43% perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
> 0.27 ± 25% -55.4% 0.12 ± 37% perf-sched.total_wait_and_delay.average.ms
> 202.17 ± 23% -29.5% 142.50 ± 13% perf-sched.total_wait_and_delay.count.ms
> 0.21 ± 18% -69.3% 0.06 ± 58% perf-sched.total_wait_time.average.ms
> 0.00 ± 70% +191.7% 0.01 ± 26% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
> 8.17 ± 29% -85.7% 1.17 ±125% perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
> 0.00 ± 47% +158.8% 0.01 ± 43% perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
> 3.11 ± 11% -76.1% 0.74 ±142% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
> 1790 ± 15% -30.1% 1250 ± 13% slabinfo.dmaengine-unmap-16.active_objs
> 1790 ± 15% -30.1% 1250 ± 13% slabinfo.dmaengine-unmap-16.num_objs
> 123.00 -100.0% 0.00 slabinfo.ext4_pending_reservation.active_objs
> 123.00 -100.0% 0.00 slabinfo.ext4_pending_reservation.num_objs
> 3619 -100.0% 0.00 slabinfo.f2fs_free_nid.active_objs
> 3619 -100.0% 0.00 slabinfo.f2fs_free_nid.num_objs
> 62865 -44.1% 35155 slabinfo.kmalloc-64.active_objs
> 984.33 -43.9% 552.50 slabinfo.kmalloc-64.active_slabs
> 63031 -43.9% 35389 slabinfo.kmalloc-64.num_objs
> 984.33 -43.9% 552.50 slabinfo.kmalloc-64.num_slabs
> 161.00 ± 9% +67.1% 269.00 ± 7% slabinfo.xfs_buf.active_objs
> 161.00 ± 9% +67.1% 269.00 ± 7% slabinfo.xfs_buf.num_objs
> 3399 ± 3% +32.9% 4519 ± 4% proc-vmstat.nr_active_anon
> 1806 -100.0% 0.00 proc-vmstat.nr_active_file
> 9333 +3.0% 9610 proc-vmstat.nr_mapped
> 7344 +18.4% 8698 ± 2% proc-vmstat.nr_shmem
> 16319 -0.9% 16176 proc-vmstat.nr_slab_reclaimable
> 24399 -6.2% 22882 proc-vmstat.nr_slab_unreclaimable
> 3399 ± 3% +32.9% 4519 ± 4% proc-vmstat.nr_zone_active_anon
> 1806 -100.0% 0.00 proc-vmstat.nr_zone_active_file
> 3693 ± 80% -80.1% 736.17 ± 61% proc-vmstat.numa_hint_faults
> 293002 -2.7% 284991 proc-vmstat.numa_hit
> 249530 -3.3% 241180 proc-vmstat.numa_local
> 5007 ±111% -90.5% 478.00 ± 81% proc-vmstat.numa_pages_migrated
> 11443 ± 2% -7.0% 10636 ± 4% proc-vmstat.pgactivate
> 332528 -3.9% 319693 proc-vmstat.pgalloc_normal
> 249148 ± 2% -4.1% 239053 ± 2% proc-vmstat.pgfree
> 5007 ±111% -90.5% 478.00 ± 81% proc-vmstat.pgmigrate_success
> 14704 -77.5% 3310 ± 61% proc-vmstat.pgpgin
> 0.00 +2.1e+105% 2095 proc-vmstat.pgpgout
> 13870 ± 10% -55.1% 6227 ± 28% softirqs.CPU0.RCU
> 9989 ± 3% -62.2% 3775 ± 23% softirqs.CPU1.RCU
> 8625 ± 13% -76.1% 2061 ± 10% softirqs.CPU10.RCU
> 7954 ± 15% -65.9% 2709 ± 18% softirqs.CPU14.RCU
> 9075 ± 14% -78.7% 1929 ± 11% softirqs.CPU17.RCU
> 8522 ± 13% -76.7% 1985 ± 22% softirqs.CPU18.RCU
> 9595 ± 7% -63.3% 3522 ± 22% softirqs.CPU2.RCU
> 8455 ± 11% -74.5% 2152 ± 45% softirqs.CPU20.RCU
> 8320 ± 12% -76.7% 1939 ± 14% softirqs.CPU21.RCU
> 8338 ± 13% -71.7% 2359 ± 32% softirqs.CPU23.RCU
> 8541 ± 12% -75.5% 2089 ± 32% softirqs.CPU26.RCU
> 9639 ± 20% -79.5% 1976 ± 17% softirqs.CPU28.RCU
> 9232 ± 13% -78.0% 2026 ± 6% softirqs.CPU30.RCU
> 7857 ± 17% -68.9% 2446 ± 27% softirqs.CPU34.RCU
> 8619 ± 11% -75.8% 2081 ± 30% softirqs.CPU36.RCU
> 9614 ± 3% -74.3% 2469 ± 15% softirqs.CPU4.RCU
> 8962 ± 10% -77.9% 1981 ± 12% softirqs.CPU41.RCU
> 9027 ± 12% -78.6% 1932 ± 8% softirqs.CPU42.RCU
> 9364 ± 12% -76.5% 2197 ± 8% softirqs.CPU44.RCU
> 8774 ± 13% -75.5% 2147 ± 21% softirqs.CPU47.RCU
> 8783 ± 12% -76.0% 2105 ± 12% softirqs.CPU5.RCU
> 9007 ± 9% -75.8% 2177 ± 8% softirqs.CPU6.RCU
> 417664 ± 7% -72.8% 113621 ± 8% softirqs.RCU
> 12708 ± 4% +13.0% 14362 ± 2% softirqs.TIMER
> 60500 -27.7% 43751 interrupts.CAL:Function_call_interrupts
> 1121 -26.9% 819.17 ± 3% interrupts.CPU10.CAL:Function_call_interrupts
> 1561 ± 43% -48.8% 800.00 ± 5% interrupts.CPU11.CAL:Function_call_interrupts
> 1425 ± 6% -25.3% 1065 ± 6% interrupts.CPU12.CAL:Function_call_interrupts
> 166.17 ± 13% -26.4% 122.33 ± 21% interrupts.CPU13.RES:Rescheduling_interrupts
> 1402 ± 18% -25.6% 1043 ± 22% interrupts.CPU15.CAL:Function_call_interrupts
> 129.17 ± 50% -42.5% 74.33 ± 4% interrupts.CPU17.RES:Rescheduling_interrupts
> 1182 ± 9% -31.0% 815.00 ± 2% interrupts.CPU20.CAL:Function_call_interrupts
> 1120 -29.7% 787.17 ± 4% interrupts.CPU21.CAL:Function_call_interrupts
> 1115 ± 3% -28.2% 801.17 interrupts.CPU23.CAL:Function_call_interrupts
> 1169 ± 7% -27.2% 851.33 ± 5% interrupts.CPU24.CAL:Function_call_interrupts
> 177.33 ± 98% -55.9% 78.17 ± 6% interrupts.CPU25.RES:Rescheduling_interrupts
> 1142 ± 16% -28.8% 813.00 ± 3% interrupts.CPU27.CAL:Function_call_interrupts
> 1229 ± 18% -33.3% 820.33 ± 4% interrupts.CPU28.CAL:Function_call_interrupts
> 1124 ± 4% -28.3% 806.17 interrupts.CPU29.CAL:Function_call_interrupts
> 1123 ± 3% -28.5% 803.00 interrupts.CPU30.CAL:Function_call_interrupts
> 1127 ± 2% -32.0% 766.67 ± 19% interrupts.CPU31.CAL:Function_call_interrupts
> 1066 ± 8% -22.3% 829.33 ± 7% interrupts.CPU32.CAL:Function_call_interrupts
> 1109 -26.0% 820.50 ± 4% interrupts.CPU34.CAL:Function_call_interrupts
> 1315 ± 22% -37.7% 818.83 ± 2% interrupts.CPU38.CAL:Function_call_interrupts
> 1164 ± 4% -29.0% 827.00 ± 3% interrupts.CPU39.CAL:Function_call_interrupts
> 5513 ± 35% +13.1% 6237 ± 33% interrupts.CPU39.NMI:Non-maskable_interrupts
> 5513 ± 35% +13.1% 6237 ± 33% interrupts.CPU39.PMI:Performance_monitoring_interrupts
> 1277 ± 23% -36.1% 815.83 interrupts.CPU40.CAL:Function_call_interrupts
> 97.33 ± 28% -25.7% 72.33 ± 6% interrupts.CPU40.RES:Rescheduling_interrupts
> 1116 -24.8% 839.00 ± 11% interrupts.CPU42.CAL:Function_call_interrupts
> 1130 ± 3% -28.5% 808.67 ± 3% interrupts.CPU43.CAL:Function_call_interrupts
> 1121 -29.8% 787.50 ± 4% interrupts.CPU45.CAL:Function_call_interrupts
> 1119 -27.3% 813.83 interrupts.CPU46.CAL:Function_call_interrupts
> 1167 ± 6% -28.0% 840.67 interrupts.CPU47.CAL:Function_call_interrupts
> 1667 ± 41% -44.3% 928.67 ± 24% interrupts.CPU5.CAL:Function_call_interrupts
> 1369 ± 24% -39.5% 827.67 ± 3% interrupts.CPU6.CAL:Function_call_interrupts
> 96.83 ± 25% -23.1% 74.50 ± 2% interrupts.CPU7.RES:Rescheduling_interrupts
> 1123 -28.1% 807.00 interrupts.CPU9.CAL:Function_call_interrupts
> 0.72 ± 5% +107.4% 1.50 ± 2% perf-stat.i.MPKI
> 8.023e+09 -13.5% 6.943e+09 perf-stat.i.branch-instructions
> 1.08 ± 2% -0.5 0.55 ± 3% perf-stat.i.branch-miss-rate%
> 71073978 ± 2% -67.4% 23149119 ± 3% perf-stat.i.branch-misses
> 29.33 ± 2% +5.9 35.23 perf-stat.i.cache-miss-rate%
> 6189596 +120.2% 13628525 perf-stat.i.cache-misses
> 21228048 +82.4% 38714786 perf-stat.i.cache-references
> 3.36 +34.8% 4.53 perf-stat.i.cpi
> 109.52 ± 2% -22.5% 84.92 perf-stat.i.cpu-migrations
> 22695 -56.5% 9882 perf-stat.i.cycles-between-cache-misses
> 1.15 ± 7% -0.9 0.30 ± 4% perf-stat.i.dTLB-load-miss-rate%
> 1.398e+08 ± 7% -83.4% 23247225 ± 4% perf-stat.i.dTLB-load-misses
> 1.154e+10 -34.4% 7.564e+09 perf-stat.i.dTLB-loads
> 1.17 -0.0 1.13 perf-stat.i.dTLB-store-miss-rate%
> 1.321e+08 -82.1% 23679743 perf-stat.i.dTLB-store-misses
> 1.071e+10 -81.3% 2.005e+09 perf-stat.i.dTLB-stores
> 41869658 -74.5% 10693569 ± 56% perf-stat.i.iTLB-load-misses
> 19932113 ± 38% -88.3% 2325708 ± 64% perf-stat.i.iTLB-loads
> 3.945e+10 -25.9% 2.924e+10 perf-stat.i.instructions
> 1199 ± 4% +182.7% 3389 ± 26% perf-stat.i.instructions-per-iTLB-miss
> 0.31 -23.7% 0.24 perf-stat.i.ipc
> 634.71 -45.5% 345.81 perf-stat.i.metric.M/sec
> 166710 ± 10% +3369.3% 5783625 perf-stat.i.node-load-misses
> 227268 ± 6% +3063.1% 7188743 perf-stat.i.node-loads
> 48.41 -4.5 43.91 perf-stat.i.node-store-miss-rate%
> 5425859 -11.8% 4783945 perf-stat.i.node-store-misses
> 5599687 +6.1% 5943407 perf-stat.i.node-stores
> 7532204 -82.7% 1305118 perf-stat.i.page-faults
> 0.54 +146.0% 1.32 perf-stat.overall.MPKI
> 0.89 ± 2% -0.6 0.33 ± 3% perf-stat.overall.branch-miss-rate%
> 29.17 ± 2% +6.0 35.20 perf-stat.overall.cache-miss-rate%
> 3.45 +35.0% 4.65 perf-stat.overall.cpi
> 21953 -54.5% 9979 perf-stat.overall.cycles-between-cache-misses
> 1.20 ± 7% -0.9 0.31 ± 4% perf-stat.overall.dTLB-load-miss-rate%
> 1.22 -0.1 1.17 perf-stat.overall.dTLB-store-miss-rate%
> 942.39 +245.5% 3255 ± 28% perf-stat.overall.instructions-per-iTLB-miss
> 0.29 -25.9% 0.21 perf-stat.overall.ipc
> 42.24 ± 2% +2.3 44.58 perf-stat.overall.node-load-miss-rate%
> 49.21 -4.6 44.59 perf-stat.overall.node-store-miss-rate%
> 7.894e+09 -13.5% 6.83e+09 perf-stat.ps.branch-instructions
> 69952381 ± 2% -67.4% 22781972 ± 3% perf-stat.ps.branch-misses
> 6093197 +120.1% 13409962 perf-stat.ps.cache-misses
> 20897937 +82.3% 38097787 perf-stat.ps.cache-references
> 107.78 ± 2% -22.4% 83.62 perf-stat.ps.cpu-migrations
> 1.375e+08 ± 7% -83.4% 22871450 ± 4% perf-stat.ps.dTLB-load-misses
> 1.135e+10 -34.4% 7.442e+09 perf-stat.ps.dTLB-loads
> 1.3e+08 -82.1% 23295591 perf-stat.ps.dTLB-store-misses
> 1.054e+10 -81.3% 1.973e+09 perf-stat.ps.dTLB-stores
> 41193894 -74.5% 10519305 ± 56% perf-stat.ps.iTLB-load-misses
> 19610606 ± 38% -88.3% 2288293 ± 64% perf-stat.ps.iTLB-loads
> 3.882e+10 -25.9% 2.876e+10 perf-stat.ps.instructions
> 164152 ± 10% +3366.2% 5689843 perf-stat.ps.node-load-misses
> 223940 ± 6% +3058.2% 7072454 perf-stat.ps.node-loads
> 5338769 -11.8% 4706549 perf-stat.ps.node-store-misses
> 5510338 +6.1% 5847491 perf-stat.ps.node-stores
> 7410609 -82.7% 1283937 perf-stat.ps.page-faults
> 2.454e+12 -25.9% 1.817e+12 perf-stat.total.instructions
> 33.68 -29.6 4.04 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 18.94 -16.5 2.46 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 13.67 -12.3 1.42 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
> 12.27 -10.9 1.36 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 8.04 -7.0 1.05 ± 2% perf-profile.calltrace.cycles-pp.__entry_text_start
> 6.06 -6.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack
> 6.02 ± 2% -5.4 0.66 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigaction.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 5.51 -4.8 0.70 perf-profile.calltrace.cycles-pp.__setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
> 0.00 +0.6 0.65 perf-profile.calltrace.cycles-pp.inc_rlimit_ucounts_and_test.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault
> 18.91 +27.0 45.89 perf-profile.calltrace.cycles-pp.irqentry_exit_to_user_mode.asm_exc_page_fault
> 15.00 +30.4 45.42 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
> 14.66 +30.7 45.38 perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
> 11.73 +34.2 45.97 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
> 8.85 +36.7 45.54 perf-profile.calltrace.cycles-pp.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
> 8.16 +37.2 45.40 perf-profile.calltrace.cycles-pp.force_sig_fault.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
> 8.04 +37.3 45.39 perf-profile.calltrace.cycles-pp.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
> 6.53 +37.8 44.36 perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
> 7.06 +38.0 45.05 perf-profile.calltrace.cycles-pp.__send_signal.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore.exc_page_fault
> 4.28 ± 2% +39.2 43.46 perf-profile.calltrace.cycles-pp.__sigqueue_free.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
> 4.87 ± 2% +39.9 44.81 perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore
> 0.00 +42.7 42.72 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.get_signal
> 0.00 +43.0 42.99 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.get_signal.arch_do_signal_or_restart
> 0.00 +43.1 43.08 perf-profile.calltrace.cycles-pp.put_ucounts.__sigqueue_free.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare
> 0.00 +43.5 43.52 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal
> 0.00 +44.0 43.97 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal.force_sig_info_to_task
> 0.00 +44.1 44.05 perf-profile.calltrace.cycles-pp.get_ucounts.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault
> 31.89 +60.0 91.94 perf-profile.calltrace.cycles-pp.asm_exc_page_fault
> 33.84 -29.7 4.10 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 19.01 -16.5 2.48 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
> 14.93 -13.3 1.59 perf-profile.children.cycles-pp.syscall_return_via_sysret
> 13.14 -11.7 1.43 ± 2% perf-profile.children.cycles-pp.do_syscall_64
> 10.60 -9.3 1.27 perf-profile.children.cycles-pp.__entry_text_start
> 6.17 ± 2% -5.5 0.68 ± 2% perf-profile.children.cycles-pp.__x64_sys_rt_sigaction
> 5.55 -4.8 0.71 ± 2% perf-profile.children.cycles-pp.__setup_rt_frame
> 3.63 ± 2% -3.2 0.40 ± 2% perf-profile.children.cycles-pp.__x64_sys_rt_sigprocmask
> 3.54 -3.2 0.32 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
> 3.11 ± 3% -2.8 0.35 ± 3% perf-profile.children.cycles-pp._copy_from_user
> 2.91 -2.5 0.37 ± 2% perf-profile.children.cycles-pp.copy_fpstate_to_sigframe
> 2.38 -2.1 0.28 perf-profile.children.cycles-pp.__irqentry_text_end
> 2.40 ± 4% -2.1 0.30 ± 2% perf-profile.children.cycles-pp.__might_fault
> 2.23 -1.9 0.32 perf-profile.children.cycles-pp.native_irq_return_iret
> 2.26 ± 2% -1.9 0.35 ± 5% perf-profile.children.cycles-pp.do_user_addr_fault
> 1.86 -1.6 0.27 ± 3% perf-profile.children.cycles-pp.do_sigaction
> 1.75 -1.5 0.21 ± 3% perf-profile.children.cycles-pp.copy_user_generic_unrolled
> 1.67 ± 3% -1.5 0.19 ± 3% perf-profile.children.cycles-pp.__set_current_blocked
> 1.51 ± 2% -1.3 0.18 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irq
> 1.33 ± 3% -1.1 0.19 perf-profile.children.cycles-pp.copy_siginfo_to_user
> 1.24 ± 3% -1.1 0.13 ± 3% perf-profile.children.cycles-pp.recalc_sigpending
> 1.30 ± 4% -1.1 0.18 ± 3% perf-profile.children.cycles-pp._copy_to_user
> 1.23 ± 3% -1.1 0.15 ± 3% perf-profile.children.cycles-pp.___might_sleep
> 1.07 ± 4% -0.9 0.12 ± 3% perf-profile.children.cycles-pp.signal_setup_done
> 0.98 ± 4% -0.9 0.11 ± 4% perf-profile.children.cycles-pp.fpu__clear
> 0.93 ± 4% -0.8 0.12 ± 8% perf-profile.children.cycles-pp.__might_sleep
> 0.89 -0.8 0.14 ± 3% perf-profile.children.cycles-pp.__clear_user
> 0.85 ± 4% -0.7 0.10 ± 3% perf-profile.children.cycles-pp.__set_task_blocked
> 0.81 ± 2% -0.7 0.10 ± 8% perf-profile.children.cycles-pp.sigprocmask
> 0.76 ± 11% -0.7 0.07 ± 10% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
> 0.70 ± 2% -0.6 0.06 ± 9% perf-profile.children.cycles-pp.signal_wake_up_state
> 0.73 -0.6 0.13 ± 7% perf-profile.children.cycles-pp.__perf_sw_event
> 0.50 ± 2% -0.5 0.04 ± 44% perf-profile.children.cycles-pp.complete_signal
> 0.50 ± 4% -0.4 0.08 ± 11% perf-profile.children.cycles-pp.___perf_sw_event
> 0.45 -0.4 0.06 ± 6% perf-profile.children.cycles-pp.sync_regs
> 0.44 -0.4 0.07 ± 10% perf-profile.children.cycles-pp.fixup_vdso_exception
> 0.35 ± 2% -0.3 0.05 perf-profile.children.cycles-pp.prepare_signal
> 0.35 ± 3% -0.3 0.08 ± 5% perf-profile.children.cycles-pp.is_prefetch
> 0.33 ± 11% -0.3 0.07 ± 6% perf-profile.children.cycles-pp.kmem_cache_alloc
> 0.19 ± 4% -0.1 0.05 ± 45% perf-profile.children.cycles-pp.copy_from_kernel_nofault
> 0.33 +0.0 0.37 ± 3% perf-profile.children.cycles-pp.kmem_cache_free
> 0.00 +0.4 0.36 ± 4% perf-profile.children.cycles-pp.dec_rlimit_ucounts
> 0.00 +0.6 0.65 perf-profile.children.cycles-pp.inc_rlimit_ucounts_and_test
> 19.11 +26.8 45.91 perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
> 15.69 +29.8 45.51 perf-profile.children.cycles-pp.exit_to_user_mode_prepare
> 14.70 +30.7 45.38 perf-profile.children.cycles-pp.arch_do_signal_or_restart
> 11.77 +34.2 45.99 perf-profile.children.cycles-pp.exc_page_fault
> 8.90 +36.7 45.55 perf-profile.children.cycles-pp.__bad_area_nosemaphore
> 8.17 +37.2 45.40 perf-profile.children.cycles-pp.force_sig_fault
> 8.10 +37.3 45.39 perf-profile.children.cycles-pp.force_sig_info_to_task
> 6.57 +37.8 44.38 perf-profile.children.cycles-pp.get_signal
> 7.11 +38.0 45.06 perf-profile.children.cycles-pp.__send_signal
> 4.28 ± 2% +39.2 43.46 perf-profile.children.cycles-pp.__sigqueue_free
> 4.89 ± 2% +39.9 44.82 perf-profile.children.cycles-pp.__sigqueue_alloc
> 0.00 +43.1 43.09 perf-profile.children.cycles-pp.put_ucounts
> 0.00 +44.1 44.05 perf-profile.children.cycles-pp.get_ucounts
> 31.95 +60.0 91.95 perf-profile.children.cycles-pp.asm_exc_page_fault
> 0.00 +86.2 86.24 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
> 0.32 ± 4% +86.7 87.01 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 18.17 -15.8 2.36 perf-profile.self.cycles-pp.syscall_exit_to_user_mode
> 14.90 -13.3 1.58 perf-profile.self.cycles-pp.syscall_return_via_sysret
> 10.60 -9.3 1.27 perf-profile.self.cycles-pp.__entry_text_start
> 4.55 ± 2% -4.5 0.04 ± 72% perf-profile.self.cycles-pp.__sigqueue_alloc
> 4.09 ± 2% -3.6 0.49 ± 2% perf-profile.self.cycles-pp.irqentry_exit_to_user_mode
> 2.30 ± 5% -2.1 0.19 ± 4% perf-profile.self.cycles-pp.do_syscall_64
> 2.38 -2.1 0.28 perf-profile.self.cycles-pp.__irqentry_text_end
> 2.22 -1.9 0.32 perf-profile.self.cycles-pp.native_irq_return_iret
> 1.92 ± 8% -1.8 0.16 ± 2% perf-profile.self.cycles-pp.__x64_sys_rt_sigaction
> 1.72 ± 3% -1.5 0.20 ± 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
> 1.71 -1.5 0.21 ± 4% perf-profile.self.cycles-pp.copy_fpstate_to_sigframe
> 1.68 -1.5 0.20 ± 3% perf-profile.self.cycles-pp.copy_user_generic_unrolled
> 1.46 ± 4% -1.3 0.12 ± 4% perf-profile.self.cycles-pp.__x64_sys_rt_sigprocmask
> 1.46 ± 2% -1.3 0.17 ± 3% perf-profile.self.cycles-pp._raw_spin_lock_irq
> 1.21 ± 6% -1.1 0.14 ± 5% perf-profile.self.cycles-pp.__setup_rt_frame
> 1.21 ± 2% -1.1 0.14 ± 3% perf-profile.self.cycles-pp.___might_sleep
> 1.06 ± 4% -1.0 0.09 ± 5% perf-profile.self.cycles-pp.recalc_sigpending
> 1.01 ± 6% -1.0 0.05 perf-profile.self.cycles-pp.asm_exc_page_fault
> 0.97 ± 4% -0.9 0.10 ± 7% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
> 0.95 ± 2% -0.8 0.12 ± 4% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
> 0.98 -0.8 0.15 ± 4% perf-profile.self.cycles-pp.do_sigaction
> 0.84 ± 4% -0.7 0.10 ± 6% perf-profile.self.cycles-pp.__might_sleep
> 0.72 ± 8% -0.6 0.09 ± 4% perf-profile.self.cycles-pp._copy_from_user
> 0.71 ± 2% -0.6 0.10 ± 7% perf-profile.self.cycles-pp.do_user_addr_fault
> 0.65 ± 14% -0.6 0.06 ± 11% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
> 0.68 ± 6% -0.6 0.10 ± 3% perf-profile.self.cycles-pp.__might_fault
> 0.80 -0.6 0.25 ± 3% perf-profile.self.cycles-pp.get_signal
> 0.64 ± 4% -0.6 0.08 ± 5% perf-profile.self.cycles-pp.fpu__clear
> 0.57 ± 3% -0.5 0.07 ± 7% perf-profile.self.cycles-pp.__send_signal
> 0.53 ± 2% -0.5 0.08 ± 6% perf-profile.self.cycles-pp.__clear_user
> 0.49 ± 5% -0.4 0.04 ± 45% perf-profile.self.cycles-pp.arch_do_signal_or_restart
> 0.43 -0.4 0.06 ± 7% perf-profile.self.cycles-pp.fixup_vdso_exception
> 0.42 ± 4% -0.4 0.06 ± 7% perf-profile.self.cycles-pp.___perf_sw_event
> 0.41 -0.4 0.05 ± 9% perf-profile.self.cycles-pp.sync_regs
> 0.33 ± 3% -0.3 0.04 ± 44% perf-profile.self.cycles-pp.prepare_signal
> 0.31 ± 12% -0.2 0.07 ± 7% perf-profile.self.cycles-pp.kmem_cache_alloc
> 0.22 ± 3% -0.1 0.10 ± 7% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> 0.33 +0.0 0.36 ± 3% perf-profile.self.cycles-pp.kmem_cache_free
> 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.get_ucounts
> 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.put_ucounts
> 0.00 +0.4 0.36 ± 4% perf-profile.self.cycles-pp.dec_rlimit_ucounts
> 0.27 ± 4% +0.5 0.77 ± 2% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.00 +0.6 0.65 perf-profile.self.cycles-pp.inc_rlimit_ucounts_and_test
> 0.00 +86.2 86.24 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>
>
>
> stress-ng.time.user_time
>
> 1400 +--------------------------------------------------------------------+
> |.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.++.+.+.++.+.++.+.+. +.+.+.++.|
> 1200 |-+ + |
> | |
> 1000 |-+ |
> | |
> 800 |-+ |
> | |
> 600 |-+ |
> | |
> 400 |-+ O |
> | |
> 200 |-OO O O O O O OO O OO O O OO O O OO O OO O O OO O OO |
> | |
> 0 +--------------------------------------------------------------------+
>
>
> stress-ng.time.system_time
>
> 2800 +--------------------------------------------------------------------+
> | O O O O OO O O O O O |
> 2600 |-OO O O O O OO O OO OO O O O O O O |
> | |
> 2400 |-+ O |
> | |
> 2200 |-+ |
> | |
> 2000 |-+ |
> | |
> 1800 |-+ |
> | |
> 1600 |.++.+.+.++.+. .+.++.+.+.++.+.+.++.+.++.+.+.++.+.+ .+.+.++.+.+.++.|
> | +.++ + |
> 1400 +--------------------------------------------------------------------+
>
>
> stress-ng.sigsegv.ops
>
> 5e+08 +-----------------------------------------------------------------+
> |.++.+.++.+.++.+.++.++.+.++.+.++.+.++.+.++.+.++.+.++.++.+.++.+.++.|
> 4.5e+08 |-+ |
> 4e+08 |-+ |
> | |
> 3.5e+08 |-+ |
> 3e+08 |-+ |
> | |
> 2.5e+08 |-+ |
> 2e+08 |-+ |
> | |
> 1.5e+08 |-+ |
> 1e+08 |-+ |
> | OO O OO O OO O OO OO O OO O OO O OO O OO O OO O OO |
> 5e+07 +-----------------------------------------------------------------+
>
>
> stress-ng.sigsegv.ops_per_sec
>
> 8e+06 +-------------------------------------------------------------------+
> | |
> 7e+06 |-+ |
> | |
> 6e+06 |-+ |
> | |
> 5e+06 |-+ |
> | |
> 4e+06 |-+ |
> | |
> 3e+06 |-+ |
> | |
> 2e+06 |-+ |
> | OO O O OO O OO O OO O O OO O OO O OO O O OO O OO O O |
> 1e+06 +-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
> ***************************************************************************************************
> lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
> interrupt/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/sigq/stress-ng/60s/0x42e
>
> commit:
> 4660d663b4 ("Reimplement RLIMIT_MSGQUEUE on top of ucounts")
> d28296d248 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")
>
> 4660d663b4207ce6 d28296d2484fa11e94dff65e93e
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 4.176e+08 -56.1% 1.831e+08 ± 21% stress-ng.sigq.ops
> 6959423 -56.1% 3051857 ± 21% stress-ng.sigq.ops_per_sec
> 2.157e+08 ± 8% -66.1% 73100172 ± 9% stress-ng.time.involuntary_context_switches
> 14467 +1.4% 14674 stress-ng.time.minor_page_faults
> 4513 +1.2% 4569 stress-ng.time.percent_of_cpu_this_job_got
> 2150 +17.9% 2535 ± 3% stress-ng.time.system_time
> 660.96 -53.1% 309.96 ± 30% stress-ng.time.user_time
> 2.478e+08 ± 5% -70.5% 73168836 ± 9% stress-ng.time.voluntary_context_switches
> 7.70 ± 8% -6.8% 7.18 iostat.cpu.idle
> 70.60 +16.6% 82.32 ± 3% iostat.cpu.system
> 21.70 -51.6% 10.49 ± 29% iostat.cpu.user
> 15660 ± 3% +13.7% 17806 ± 3% meminfo.Active
> 15660 ± 3% +13.7% 17806 ± 3% meminfo.Active(anon)
> 31878 ± 2% +8.8% 34688 ± 2% meminfo.Shmem
> 123900 ± 66% -66.9% 41021 ± 30% cpuidle.C1.usage
> 1.507e+08 ± 6% +9.5% 1.65e+08 ± 4% cpuidle.C6.time
> 681599 ± 38% -72.8% 185137 ±113% cpuidle.POLL.time
> 645871 ± 39% -98.5% 9580 ± 75% cpuidle.POLL.usage
> 1.78 ± 3% -0.8 0.95 mpstat.cpu.all.irq%
> 0.03 ± 4% -0.0 0.02 ± 27% mpstat.cpu.all.soft%
> 70.90 +12.5 83.42 ± 3% mpstat.cpu.all.sys%
> 22.35 -11.6 10.75 ± 29% mpstat.cpu.all.usr%
> 825.33 ± 2% -11.6% 729.67 ± 5% slabinfo.file_lock_cache.active_objs
> 825.33 ± 2% -11.6% 729.67 ± 5% slabinfo.file_lock_cache.num_objs
> 1455 ± 11% -32.4% 983.00 ± 13% slabinfo.khugepaged_mm_slot.active_objs
> 1455 ± 11% -32.4% 983.00 ± 13% slabinfo.khugepaged_mm_slot.num_objs
> 69.67 +17.2% 81.67 ± 3% vmstat.cpu.sy
> 21.00 -52.4% 10.00 ± 29% vmstat.cpu.us
> 7139282 ± 7% -68.5% 2251603 ± 9% vmstat.system.cs
> 518916 ± 6% -80.5% 101139 ± 4% vmstat.system.in
> 8082 ± 51% -68.3% 2565 ± 17% softirqs.CPU10.SCHED
> 6261 ± 72% -56.4% 2729 ± 14% softirqs.CPU17.SCHED
> 6147 ± 66% -64.3% 2195 ± 3% softirqs.CPU25.SCHED
> 16334 ± 41% -82.6% 2846 ± 24% softirqs.CPU27.SCHED
> 6280 ± 56% -61.9% 2394 ± 9% softirqs.CPU39.SCHED
> 8248 ± 50% -73.2% 2209 ± 8% softirqs.CPU40.SCHED
> 228327 ± 9% -46.3% 122665 ± 2% softirqs.SCHED
> 3851 ± 4% +13.8% 4381 ± 4% proc-vmstat.nr_active_anon
> 9587 +2.8% 9855 proc-vmstat.nr_mapped
> 7943 ± 2% +8.7% 8630 ± 2% proc-vmstat.nr_shmem
> 3851 ± 4% +13.8% 4381 ± 4% proc-vmstat.nr_zone_active_anon
> 438.33 ±122% +1463.6% 6853 ± 69% proc-vmstat.numa_pages_migrated
> 8816 ± 4% +11.5% 9827 ± 6% proc-vmstat.pgactivate
> 291848 +1.7% 296689 proc-vmstat.pgalloc_normal
> 438.33 ±122% +1463.6% 6853 ± 69% proc-vmstat.pgmigrate_success
> 252.00 ± 19% +30.8% 329.67 ± 7% numa-vmstat.node0.nr_active_anon
> 44373 ± 23% -64.0% 15967 ± 49% numa-vmstat.node0.nr_anon_pages
> 46179 ± 21% -61.8% 17619 ± 38% numa-vmstat.node0.nr_inactive_anon
> 252.00 ± 19% +30.8% 329.67 ± 7% numa-vmstat.node0.nr_zone_active_anon
> 46179 ± 21% -61.8% 17619 ± 38% numa-vmstat.node0.nr_zone_inactive_anon
> 3678 ± 4% +11.5% 4100 ± 4% numa-vmstat.node1.nr_active_anon
> 13880 ± 74% +206.7% 42565 ± 17% numa-vmstat.node1.nr_anon_pages
> 16004 ± 59% +181.1% 44988 ± 14% numa-vmstat.node1.nr_inactive_anon
> 3678 ± 4% +11.5% 4100 ± 4% numa-vmstat.node1.nr_zone_active_anon
> 16004 ± 59% +181.1% 44988 ± 14% numa-vmstat.node1.nr_zone_inactive_anon
> 1007 ± 19% +31.3% 1322 ± 7% numa-meminfo.node0.Active
> 1007 ± 19% +31.3% 1322 ± 7% numa-meminfo.node0.Active(anon)
> 39280 ± 35% -66.1% 13314 ± 74% numa-meminfo.node0.AnonHugePages
> 177224 ± 24% -63.9% 63933 ± 49% numa-meminfo.node0.AnonPages
> 182948 ± 22% -52.3% 87286 ± 59% numa-meminfo.node0.AnonPages.max
> 184457 ± 21% -61.8% 70532 ± 38% numa-meminfo.node0.Inactive
> 184457 ± 21% -61.8% 70532 ± 38% numa-meminfo.node0.Inactive(anon)
> 1080630 ± 5% -12.3% 947332 ± 3% numa-meminfo.node0.MemUsed
> 14267 ± 4% +17.0% 16695 ± 5% numa-meminfo.node1.Active
> 14267 ± 4% +17.0% 16695 ± 5% numa-meminfo.node1.Active(anon)
> 56124 ± 74% +202.8% 169963 ± 17% numa-meminfo.node1.AnonPages
> 75025 ± 56% +139.7% 179816 ± 14% numa-meminfo.node1.AnonPages.max
> 64951 ± 59% +176.3% 179452 ± 14% numa-meminfo.node1.Inactive
> 64951 ± 59% +176.3% 179452 ± 14% numa-meminfo.node1.Inactive(anon)
> 941178 ± 5% +14.6% 1078500 ± 3% numa-meminfo.node1.MemUsed
> 51.14 ± 15% +47.4% 75.37 ± 13% sched_debug.cfs_rq:/.load_avg.avg
> 114.91 ± 14% +38.2% 158.79 ± 13% sched_debug.cfs_rq:/.load_avg.stddev
> 6.99 ± 70% +352.8% 31.65 ± 27% sched_debug.cfs_rq:/.removed.load_avg.avg
> 47.92 ± 70% +153.1% 121.29 ± 12% sched_debug.cfs_rq:/.removed.load_avg.stddev
> 2.38 ± 97% +182.8% 6.74 ± 36% sched_debug.cfs_rq:/.removed.runnable_avg.avg
> 2.38 ± 97% +182.7% 6.73 ± 36% sched_debug.cfs_rq:/.removed.util_avg.avg
> 356.50 ± 3% -24.1% 270.50 ± 31% sched_debug.cfs_rq:/.util_avg.min
> 290.54 ± 2% +10.6% 321.30 ± 5% sched_debug.cfs_rq:/.util_est_enqueued.avg
> 446696 ± 4% +50.1% 670473 ± 7% sched_debug.cpu.avg_idle.avg
> 1120651 ± 11% +20.5% 1350114 ± 10% sched_debug.cpu.avg_idle.max
> 2124 ± 10% +43.0% 3038 ± 15% sched_debug.cpu.avg_idle.min
> 708.23 ± 3% -14.4% 606.42 ± 4% sched_debug.cpu.clock_task.stddev
> 989.50 ± 8% +12.0% 1108 sched_debug.cpu.curr->pid.min
> 21503 ± 71% +114.0% 46014 ± 39% sched_debug.cpu.max_idle_balance_cost.stddev
> 4583665 ± 7% -68.5% 1443748 ± 9% sched_debug.cpu.nr_switches.avg
> 6508139 ± 3% -71.5% 1854831 sched_debug.cpu.nr_switches.max
> 2183394 ± 28% -83.2% 367194 ± 87% sched_debug.cpu.nr_switches.min
> 1135976 ± 11% -69.9% 342489 ± 34% sched_debug.cpu.nr_switches.stddev
> 0.03 ± 59% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.01 ± 12% -36.4% 0.00 ± 26% perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 0.05 ± 57% -100.0% 0.00 perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.19 ± 77% -61.0% 0.07 ± 5% perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 7.83 ±140% -99.8% 0.01 ± 38% perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
> 0.36 ± 18% -64.4% 0.13 ± 75% perf-sched.total_wait_and_delay.average.ms
> 0.24 ± 5% -72.2% 0.07 ± 66% perf-sched.total_wait_time.average.ms
> 1.38 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.03 ± 41% -61.6% 0.01 ± 80% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
> 0.01 ± 12% -36.4% 0.00 ± 26% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 0.08 ± 17% -30.3% 0.05 ± 3% perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 2.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.do_syslog.part.0
> 9.00 -81.5% 1.67 ± 28% perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
> 2.72 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.80 ± 45% -89.3% 0.09 ± 10% perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 7.83 ±140% -99.8% 0.01 ± 38% perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
> 1.35 ± 8% -100.0% 0.00 perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.03 ± 42% -66.3% 0.01 ± 82% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
> 0.07 ± 24% -27.6% 0.05 ± 3% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 3.21 ± 2% -58.6% 1.33 ± 70% perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
> 2.69 ± 8% -100.0% 0.00 perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
> 0.77 ± 50% -89.7% 0.08 ± 10% perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
> 6.12 ± 3% -27.9% 4.42 ± 3% perf-stat.i.MPKI
> 1.008e+10 ± 4% -20.8% 7.989e+09 ± 2% perf-stat.i.branch-instructions
> 1.55 ± 2% -0.7 0.89 perf-stat.i.branch-miss-rate%
> 1.372e+08 ± 2% -60.4% 54290819 perf-stat.i.branch-misses
> 5.24 ± 7% +6.4 11.66 ± 5% perf-stat.i.cache-miss-rate%
> 12610881 ± 8% +30.8% 16498692 ± 4% perf-stat.i.cache-misses
> 2.994e+08 -50.1% 1.495e+08 ± 2% perf-stat.i.cache-references
> 7370989 ± 7% -68.6% 2316288 ± 9% perf-stat.i.context-switches
> 2.75 ± 4% +37.7% 3.78 ± 2% perf-stat.i.cpi
> 37935 ± 20% -99.1% 344.33 ± 39% perf-stat.i.cpu-migrations
> 10857 ± 9% -24.6% 8184 ± 4% perf-stat.i.cycles-between-cache-misses
> 1.33 ± 4% -0.5 0.81 ± 15% perf-stat.i.dTLB-load-miss-rate%
> 1.921e+08 ± 3% -59.4% 78058438 ± 14% perf-stat.i.dTLB-load-misses
> 1.375e+10 ± 4% -32.6% 9.26e+09 perf-stat.i.dTLB-loads
> 0.51 ± 5% +0.1 0.61 ± 9% perf-stat.i.dTLB-store-miss-rate%
> 52874010 -55.6% 23456497 ± 13% perf-stat.i.dTLB-store-misses
> 9.955e+09 ± 3% -63.0% 3.687e+09 ± 4% perf-stat.i.dTLB-stores
> 62946854 ± 4% -59.7% 25392611 ± 3% perf-stat.i.iTLB-load-misses
> 4.88e+10 ± 4% -28.0% 3.514e+10 perf-stat.i.instructions
> 1071 +54.5% 1655 ± 10% perf-stat.i.instructions-per-iTLB-miss
> 0.38 ± 3% -26.8% 0.28 ± 3% perf-stat.i.ipc
> 0.52 ± 7% -70.6% 0.15 ± 11% perf-stat.i.metric.K/sec
> 711.88 ± 3% -38.1% 440.48 perf-stat.i.metric.M/sec
> 47.42 -1.7 45.75 perf-stat.i.node-load-miss-rate%
> 3468825 ± 20% +118.3% 7571866 ± 5% perf-stat.i.node-load-misses
> 3747717 ± 20% +133.9% 8764955 ± 4% perf-stat.i.node-loads
> 47.67 -2.6 45.11 perf-stat.i.node-store-miss-rate%
> 8290895 ± 4% -28.2% 5954593 ± 6% perf-stat.i.node-store-misses
> 8800840 -19.9% 7053271 ± 5% perf-stat.i.node-stores
> 6.14 ± 3% -30.8% 4.25 perf-stat.overall.MPKI
> 1.36 ± 2% -0.7 0.68 perf-stat.overall.branch-miss-rate%
> 4.22 ± 10% +6.8 11.05 ± 6% perf-stat.overall.cache-miss-rate%
> 2.79 ± 4% +38.7% 3.87 perf-stat.overall.cpi
> 10868 ± 8% -23.9% 8267 ± 4% perf-stat.overall.cycles-between-cache-misses
> 1.38 ± 4% -0.5 0.84 ± 15% perf-stat.overall.dTLB-load-miss-rate%
> 0.53 ± 5% +0.1 0.63 ± 8% perf-stat.overall.dTLB-store-miss-rate%
> 775.35 +78.9% 1386 ± 5% perf-stat.overall.instructions-per-iTLB-miss
> 0.36 ± 4% -28.0% 0.26 perf-stat.overall.ipc
> 48.07 -1.7 46.34 perf-stat.overall.node-load-miss-rate%
> 48.49 -2.7 45.76 perf-stat.overall.node-store-miss-rate%
> 9.922e+09 ± 4% -20.8% 7.859e+09 ± 2% perf-stat.ps.branch-instructions
> 1.351e+08 ± 2% -60.4% 53438494 perf-stat.ps.branch-misses
> 12410866 ± 8% +30.8% 16234283 ± 4% perf-stat.ps.cache-misses
> 2.946e+08 -50.1% 1.47e+08 ± 2% perf-stat.ps.cache-references
> 7252050 ± 7% -68.6% 2278311 ± 9% perf-stat.ps.context-switches
> 37323 ± 20% -99.1% 338.80 ± 39% perf-stat.ps.cpu-migrations
> 1.89e+08 ± 3% -59.4% 76787292 ± 14% perf-stat.ps.dTLB-load-misses
> 1.352e+10 ± 4% -32.6% 9.11e+09 perf-stat.ps.dTLB-loads
> 52020930 -55.6% 23075080 ± 13% perf-stat.ps.dTLB-store-misses
> 9.794e+09 ± 3% -63.0% 3.627e+09 ± 4% perf-stat.ps.dTLB-stores
> 61931461 ± 4% -59.7% 24978175 ± 3% perf-stat.ps.iTLB-load-misses
> 4.802e+10 ± 4% -28.0% 3.457e+10 perf-stat.ps.instructions
> 3412948 ± 20% +118.2% 7448345 ± 5% perf-stat.ps.node-load-misses
> 3687448 ± 20% +133.8% 8622075 ± 4% perf-stat.ps.node-loads
> 8157655 ± 4% -28.2% 5857930 ± 6% perf-stat.ps.node-store-misses
> 8659916 -19.9% 6939197 ± 5% perf-stat.ps.node-stores
> 3.035e+12 ± 4% -27.9% 2.189e+12 perf-stat.total.instructions
> 82.33 ± 64% +1322.3% 1171 ±127% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> 17959753 ± 6% -99.7% 45889 ± 2% interrupts.CAL:Function_call_interrupts
> 103593 ±125% -99.5% 569.33 ± 9% interrupts.CPU0.CAL:Function_call_interrupts
> 45339 ±104% -94.7% 2414 ± 30% interrupts.CPU0.RES:Rescheduling_interrupts
> 1078794 ±112% -99.9% 1002 ± 12% interrupts.CPU1.CAL:Function_call_interrupts
> 6507 ± 26% +21.7% 7922 ± 6% interrupts.CPU1.NMI:Non-maskable_interrupts
> 6507 ± 26% +21.7% 7922 ± 6% interrupts.CPU1.PMI:Performance_monitoring_interrupts
> 505526 ±101% -99.8% 1223 ± 46% interrupts.CPU1.RES:Rescheduling_interrupts
> 791738 ± 80% -99.9% 898.67 ± 13% interrupts.CPU10.CAL:Function_call_interrupts
> 406363 ± 71% -99.4% 2378 ± 80% interrupts.CPU10.RES:Rescheduling_interrupts
> 46401 ±115% -98.3% 789.00 ± 21% interrupts.CPU11.CAL:Function_call_interrupts
> 5509 ± 35% +25.8% 6931 ± 28% interrupts.CPU11.NMI:Non-maskable_interrupts
> 5509 ± 35% +25.8% 6931 ± 28% interrupts.CPU11.PMI:Performance_monitoring_interrupts
> 27829 ±104% -87.3% 3526 ± 77% interrupts.CPU11.RES:Rescheduling_interrupts
> 290182 ±130% -99.6% 1299 ± 24% interrupts.CPU12.CAL:Function_call_interrupts
> 170757 ±136% -98.3% 2886 ± 40% interrupts.CPU12.RES:Rescheduling_interrupts
> 131940 ± 43% -99.1% 1206 ± 37% interrupts.CPU13.CAL:Function_call_interrupts
> 68303 ± 32% -97.9% 1445 ±126% interrupts.CPU13.RES:Rescheduling_interrupts
> 241043 ±116% -99.5% 1319 ± 32% interrupts.CPU14.CAL:Function_call_interrupts
> 98573 ± 93% -97.0% 3005 ± 47% interrupts.CPU14.RES:Rescheduling_interrupts
> 874074 ± 72% -99.9% 875.33 ± 6% interrupts.CPU15.CAL:Function_call_interrupts
> 350120 ± 49% -99.9% 412.67 ± 52% interrupts.CPU15.RES:Rescheduling_interrupts
> 577696 ± 23% -99.9% 855.33 ± 3% interrupts.CPU16.CAL:Function_call_interrupts
> 8263 -24.9% 6202 ± 23% interrupts.CPU16.NMI:Non-maskable_interrupts
> 8263 -24.9% 6202 ± 23% interrupts.CPU16.PMI:Performance_monitoring_interrupts
> 237815 ± 15% -99.5% 1220 ± 53% interrupts.CPU16.RES:Rescheduling_interrupts
> 686797 ±112% -99.9% 933.67 ± 13% interrupts.CPU17.CAL:Function_call_interrupts
> 317108 ±105% -99.8% 536.00 ± 49% interrupts.CPU17.RES:Rescheduling_interrupts
> 748163 ±122% -99.9% 863.00 ± 5% interrupts.CPU18.CAL:Function_call_interrupts
> 320920 ±112% -99.5% 1469 ± 95% interrupts.CPU18.RES:Rescheduling_interrupts
> 286835 ±121% -99.7% 943.33 ± 15% interrupts.CPU19.CAL:Function_call_interrupts
> 174455 ±113% -98.7% 2326 ± 69% interrupts.CPU19.RES:Rescheduling_interrupts
> 443377 ± 83% -99.8% 944.33 ± 12% interrupts.CPU2.CAL:Function_call_interrupts
> 57297 ± 99% -98.5% 855.00 ± 6% interrupts.CPU20.CAL:Function_call_interrupts
> 54090 ± 67% -99.0% 535.33 ± 74% interrupts.CPU20.RES:Rescheduling_interrupts
> 27584 ± 80% -96.8% 890.33 ± 6% interrupts.CPU21.CAL:Function_call_interrupts
> 27052 ± 82% -94.3% 1539 ± 58% interrupts.CPU21.RES:Rescheduling_interrupts
> 62804 ±104% -97.8% 1362 ± 47% interrupts.CPU22.CAL:Function_call_interrupts
> 27230 ± 84% -92.6% 2002 ± 86% interrupts.CPU22.RES:Rescheduling_interrupts
> 351930 ± 72% -99.7% 966.00 ± 16% interrupts.CPU23.CAL:Function_call_interrupts
> 136149 ± 62% -98.1% 2565 ± 93% interrupts.CPU23.RES:Rescheduling_interrupts
> 366644 ±138% -99.8% 877.67 ± 5% interrupts.CPU24.CAL:Function_call_interrupts
> 499203 ± 82% -99.8% 862.33 ± 3% interrupts.CPU25.CAL:Function_call_interrupts
> 298172 ± 70% -99.2% 2317 ± 19% interrupts.CPU25.RES:Rescheduling_interrupts
> 11844 ± 61% -92.9% 838.33 ± 2% interrupts.CPU26.CAL:Function_call_interrupts
> 82.33 ± 64% +1322.3% 1171 ±127% interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> 1841110 ± 49% -100.0% 746.33 ± 26% interrupts.CPU27.CAL:Function_call_interrupts
> 5501 ± 35% +25.6% 6909 ± 28% interrupts.CPU27.NMI:Non-maskable_interrupts
> 5501 ± 35% +25.6% 6909 ± 28% interrupts.CPU27.PMI:Performance_monitoring_interrupts
> 1089819 ± 55% -99.8% 1727 ± 80% interrupts.CPU27.RES:Rescheduling_interrupts
> 74718 ±118% -98.7% 942.00 interrupts.CPU28.CAL:Function_call_interrupts
> 5504 ± 35% +25.6% 6912 ± 28% interrupts.CPU28.NMI:Non-maskable_interrupts
> 5504 ± 35% +25.6% 6912 ± 28% interrupts.CPU28.PMI:Performance_monitoring_interrupts
> 67380 ±125% -98.5% 1034 ± 37% interrupts.CPU28.RES:Rescheduling_interrupts
> 367428 ± 77% -99.7% 1054 ± 22% interrupts.CPU29.CAL:Function_call_interrupts
> 239038 ± 82% -99.3% 1767 ± 84% interrupts.CPU29.RES:Rescheduling_interrupts
> 220744 ±117% -99.5% 1046 ± 21% interrupts.CPU3.CAL:Function_call_interrupts
> 191269 ±119% -98.4% 3122 ± 15% interrupts.CPU3.RES:Rescheduling_interrupts
> 247241 ± 81% -99.6% 874.33 ± 5% interrupts.CPU30.CAL:Function_call_interrupts
> 161878 ± 84% -98.1% 3063 ± 37% interrupts.CPU30.RES:Rescheduling_interrupts
> 144713 ±110% -99.3% 1077 ± 23% interrupts.CPU31.CAL:Function_call_interrupts
> 5502 ± 35% +25.6% 6911 ± 28% interrupts.CPU31.NMI:Non-maskable_interrupts
> 5502 ± 35% +25.6% 6911 ± 28% interrupts.CPU31.PMI:Performance_monitoring_interrupts
> 84227 ±100% -96.4% 3013 ± 72% interrupts.CPU31.RES:Rescheduling_interrupts
> 104510 ±109% -99.2% 848.00 ± 4% interrupts.CPU32.CAL:Function_call_interrupts
> 5515 ± 35% +31.4% 7249 ± 20% interrupts.CPU32.NMI:Non-maskable_interrupts
> 5515 ± 35% +31.4% 7249 ± 20% interrupts.CPU32.PMI:Performance_monitoring_interrupts
> 70068 ±101% -96.7% 2320 ±126% interrupts.CPU32.RES:Rescheduling_interrupts
> 87671 ±131% -99.0% 857.33 ± 2% interrupts.CPU33.CAL:Function_call_interrupts
> 35054 ±127% -96.5% 1242 ± 80% interrupts.CPU33.RES:Rescheduling_interrupts
> 77336 ±124% -98.7% 988.33 ± 21% interrupts.CPU34.CAL:Function_call_interrupts
> 66106 ±123% -96.0% 2630 ± 52% interrupts.CPU34.RES:Rescheduling_interrupts
> 136901 ±122% -99.1% 1278 ± 30% interrupts.CPU35.CAL:Function_call_interrupts
> 731054 ± 69% -99.9% 1046 ± 15% interrupts.CPU36.CAL:Function_call_interrupts
> 376136 ± 70% -99.6% 1429 ± 37% interrupts.CPU36.RES:Rescheduling_interrupts
> 941442 ±105% -99.9% 851.33 ± 7% interrupts.CPU37.CAL:Function_call_interrupts
> 300885 ± 80% -99.2% 2379 ± 57% interrupts.CPU37.RES:Rescheduling_interrupts
> 242878 ±117% -99.6% 995.00 ± 16% interrupts.CPU38.CAL:Function_call_interrupts
> 702836 ± 69% -99.9% 855.00 ± 3% interrupts.CPU39.CAL:Function_call_interrupts
> 286450 ± 74% -99.5% 1448 ± 70% interrupts.CPU39.RES:Rescheduling_interrupts
> 37564 ± 55% -96.0% 1505 ± 62% interrupts.CPU4.CAL:Function_call_interrupts
> 6346 ± 26% +24.0% 7870 ± 6% interrupts.CPU4.NMI:Non-maskable_interrupts
> 6346 ± 26% +24.0% 7870 ± 6% interrupts.CPU4.PMI:Performance_monitoring_interrupts
> 24154 ± 65% -94.0% 1444 ± 83% interrupts.CPU4.RES:Rescheduling_interrupts
> 977103 ± 53% -99.9% 837.33 ± 3% interrupts.CPU40.CAL:Function_call_interrupts
> 512103 ± 71% -99.7% 1579 ± 66% interrupts.CPU40.RES:Rescheduling_interrupts
> 190023 ±117% -99.6% 818.67 ± 2% interrupts.CPU41.CAL:Function_call_interrupts
> 346179 ± 54% -99.8% 829.00 ± 2% interrupts.CPU42.CAL:Function_call_interrupts
> 183103 ± 48% -99.5% 850.00 ± 54% interrupts.CPU42.RES:Rescheduling_interrupts
> 371814 ±131% -99.8% 863.33 ± 5% interrupts.CPU43.CAL:Function_call_interrupts
> 158123 ±124% -99.2% 1224 ± 71% interrupts.CPU43.RES:Rescheduling_interrupts
> 578077 ± 61% -99.9% 850.67 ± 3% interrupts.CPU44.CAL:Function_call_interrupts
> 331471 ± 53% -99.7% 1120 ± 91% interrupts.CPU44.RES:Rescheduling_interrupts
> 172794 ±114% -99.5% 838.33 ± 4% interrupts.CPU45.CAL:Function_call_interrupts
> 89408 ±111% -99.6% 326.00 ± 23% interrupts.CPU45.RES:Rescheduling_interrupts
> 34858 ± 59% -97.3% 947.00 ± 13% interrupts.CPU46.CAL:Function_call_interrupts
> 22612 ± 53% -87.6% 2794 ± 27% interrupts.CPU46.RES:Rescheduling_interrupts
> 41349 ± 57% -97.6% 996.67 ± 10% interrupts.CPU47.CAL:Function_call_interrupts
> 27498 ± 54% -93.7% 1725 ± 89% interrupts.CPU47.RES:Rescheduling_interrupts
> 45388 ± 61% -97.5% 1141 ± 57% interrupts.CPU5.CAL:Function_call_interrupts
> 32500 ± 63% -96.7% 1063 ± 95% interrupts.CPU5.RES:Rescheduling_interrupts
> 534772 ±105% -99.8% 843.33 ± 3% interrupts.CPU6.CAL:Function_call_interrupts
> 259978 ± 94% -98.5% 4008 ± 4% interrupts.CPU6.RES:Rescheduling_interrupts
> 589336 ± 71% -99.8% 1042 ± 25% interrupts.CPU7.CAL:Function_call_interrupts
> 311886 ± 69% -99.4% 1966 ± 66% interrupts.CPU7.RES:Rescheduling_interrupts
> 363153 ± 95% -99.8% 858.00 ± 5% interrupts.CPU8.CAL:Function_call_interrupts
> 190915 ± 83% -98.1% 3681 ± 94% interrupts.CPU8.RES:Rescheduling_interrupts
> 78806 ± 46% -98.9% 905.33 ± 7% interrupts.CPU9.CAL:Function_call_interrupts
> 47384 ± 29% -95.9% 1963 ± 58% interrupts.CPU9.RES:Rescheduling_interrupts
> 9209745 ± 15% -99.0% 95188 ± 9% interrupts.RES:Rescheduling_interrupts
> 20.15 ± 4% -14.2 5.98 ± 15% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 15.92 ± 4% -11.9 3.98 ± 7% perf-profile.calltrace.cycles-pp.signal_wake_up_state.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
> 15.71 ± 4% -11.8 3.95 ± 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info.kill_pid_info
> 13.68 ± 6% -10.8 2.93 ± 10% perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 12.23 ± 26% -10.4 1.80 ± 38% perf-profile.calltrace.cycles-pp.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64
> 11.55 ± 27% -10.0 1.56 ± 39% perf-profile.calltrace.cycles-pp.security_task_kill.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
> 11.33 ± 27% -9.8 1.49 ± 40% perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
> 11.78 ± 5% -9.3 2.52 ± 9% perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
> 11.53 ± 5% -9.1 2.47 ± 9% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait
> 8.27 ± 13% -6.3 1.99 ± 7% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 7.32 ± 2% -4.9 2.43 ± 25% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
> 6.20 ± 13% -4.7 1.53 ± 10% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 5.94 ± 13% -4.5 1.46 ± 10% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 6.32 ± 8% -4.3 2.07 ± 6% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info
> 4.57 ± 30% -4.1 0.44 ± 74% perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.group_send_sig_info.kill_pid_info
> 5.55 ± 8% -3.7 1.90 ± 6% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.signal_wake_up_state.__send_signal
> 4.43 ± 7% -3.4 0.99 ± 6% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info
> 4.29 ± 7% -3.3 0.95 ± 6% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.signal_wake_up_state.__send_signal
> 4.19 ± 6% -3.3 0.92 ± 8% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait
> 4.60 -2.9 1.69 ± 28% perf-profile.calltrace.cycles-pp.__entry_text_start
> 1.93 ± 16% -1.6 0.37 ± 70% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode
> 2.40 ± 6% -1.5 0.86 ± 6% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.signal_wake_up_state
> 0.00 +0.7 0.67 perf-profile.calltrace.cycles-pp.inc_rlimit_ucounts_and_test.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info
> 0.67 ± 16% +2.4 3.11 ± 53% perf-profile.calltrace.cycles-pp.__lock_task_sighand.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
> 0.42 ± 71% +2.7 3.08 ± 53% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__lock_task_sighand.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
> 0.00 +3.0 2.98 ± 54% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__lock_task_sighand.do_send_sig_info.kill_pid_info
> 0.00 +3.2 3.23 ± 51% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
> 0.00 +3.3 3.29 ± 51% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 35.47 ± 7% +9.5 45.02 perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 34.17 ± 7% +10.4 44.54 perf-profile.calltrace.cycles-pp.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 33.33 ± 7% +10.9 44.22 perf-profile.calltrace.cycles-pp.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 81.67 +12.6 94.25 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 19.16 ± 4% +19.5 38.64 ± 8% perf-profile.calltrace.cycles-pp.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
> 21.31 ± 6% +20.5 41.77 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 20.83 ± 3% +21.5 42.33 ± 3% perf-profile.calltrace.cycles-pp.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64
> 19.46 ± 5% +21.9 41.31 ± 4% perf-profile.calltrace.cycles-pp.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 59.72 ± 2% +28.1 87.78 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 3.78 ± 8% +30.8 34.58 ± 9% perf-profile.calltrace.cycles-pp.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 3.07 ± 8% +31.3 34.40 ± 9% perf-profile.calltrace.cycles-pp.__dequeue_signal.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
> 1.74 ± 9% +32.1 33.86 ± 9% perf-profile.calltrace.cycles-pp.__sigqueue_free.__dequeue_signal.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait
> 1.94 ± 8% +32.5 34.40 ± 8% perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
> 0.00 +32.8 32.81 ± 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.__dequeue_signal
> 0.00 +33.0 33.02 ± 8% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal
> 0.00 +33.2 33.22 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.__dequeue_signal.dequeue_signal
> 0.00 +33.3 33.32 ± 9% perf-profile.calltrace.cycles-pp.put_ucounts.__sigqueue_free.__dequeue_signal.dequeue_signal.do_sigtimedwait
> 0.00 +33.5 33.48 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal.do_send_sig_info
> 0.00 +33.6 33.57 ± 8% perf-profile.calltrace.cycles-pp.get_ucounts.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info
> 20.22 ± 4% -14.2 6.00 ± 15% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
> 18.14 ± 8% -14.1 4.06 ± 10% perf-profile.children.cycles-pp.schedule
> 17.92 ± 7% -14.0 3.96 ± 9% perf-profile.children.cycles-pp.__sched_text_start
> 15.92 ± 4% -11.9 3.98 ± 7% perf-profile.children.cycles-pp.signal_wake_up_state
> 15.73 ± 4% -11.8 3.96 ± 7% perf-profile.children.cycles-pp.try_to_wake_up
> 13.71 ± 6% -10.8 2.93 ± 10% perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
> 12.24 ± 26% -10.4 1.80 ± 38% perf-profile.children.cycles-pp.group_send_sig_info
> 11.56 ± 27% -10.0 1.56 ± 39% perf-profile.children.cycles-pp.security_task_kill
> 11.35 ± 27% -9.9 1.49 ± 40% perf-profile.children.cycles-pp.apparmor_task_kill
> 8.45 ± 12% -6.5 2.00 ± 7% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
> 8.04 -5.3 2.71 ± 25% perf-profile.children.cycles-pp.syscall_return_via_sysret
> 6.34 ± 8% -4.3 2.07 ± 6% perf-profile.children.cycles-pp.select_task_rq_fair
> 4.58 ± 30% -4.0 0.55 ± 36% perf-profile.children.cycles-pp.aa_get_task_label
> 6.03 -4.0 2.05 ± 28% perf-profile.children.cycles-pp.__entry_text_start
> 4.70 ± 6% -3.7 0.99 ± 6% perf-profile.children.cycles-pp.ttwu_do_activate
> 5.63 ± 8% -3.7 1.93 ± 6% perf-profile.children.cycles-pp.select_idle_sibling
> 4.57 ± 6% -3.6 0.96 ± 6% perf-profile.children.cycles-pp.enqueue_task_fair
> 4.23 ± 7% -3.3 0.93 ± 8% perf-profile.children.cycles-pp.dequeue_task_fair
> 3.78 ± 12% -2.9 0.91 ± 11% perf-profile.children.cycles-pp.pick_next_task_fair
> 3.44 ± 9% -2.7 0.78 ± 9% perf-profile.children.cycles-pp.switch_mm_irqs_off
> 3.07 ± 10% -2.4 0.68 ± 9% perf-profile.children.cycles-pp.update_load_avg
> 2.96 ± 3% -2.3 0.69 ± 9% perf-profile.children.cycles-pp.update_curr
> 2.35 ± 3% -1.9 0.48 ± 6% perf-profile.children.cycles-pp.enqueue_entity
> 2.26 ± 13% -1.7 0.56 ± 10% perf-profile.children.cycles-pp.load_new_mm_cr3
> 2.44 ± 6% -1.6 0.86 ± 5% perf-profile.children.cycles-pp.available_idle_cpu
> 1.82 ± 4% -1.4 0.41 ± 7% perf-profile.children.cycles-pp.dequeue_entity
> 1.93 ± 3% -1.4 0.52 ± 32% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
> 1.63 ± 10% -1.3 0.28 ± 8% perf-profile.children.cycles-pp._raw_spin_lock
> 1.59 ± 13% -1.3 0.30 ± 10% perf-profile.children.cycles-pp.switch_fpu_return
> 1.63 ± 11% -1.2 0.39 ± 10% perf-profile.children.cycles-pp.reweight_entity
> 1.47 ± 7% -1.2 0.31 ± 11% perf-profile.children.cycles-pp.__switch_to
> 1.25 ± 12% -1.0 0.27 ± 13% perf-profile.children.cycles-pp.__switch_to_asm
> 1.21 ± 10% -0.9 0.27 ± 13% perf-profile.children.cycles-pp.set_next_entity
> 1.07 ± 7% -0.9 0.13 ± 25% perf-profile.children.cycles-pp.finish_task_switch
> 1.30 ± 4% -0.9 0.41 ± 10% perf-profile.children.cycles-pp._copy_from_user
> 1.11 ± 7% -0.8 0.27 ± 12% perf-profile.children.cycles-pp.update_rq_clock
> 1.04 ± 4% -0.8 0.24 ± 15% perf-profile.children.cycles-pp.ttwu_do_wakeup
> 0.99 ± 13% -0.8 0.21 ± 14% perf-profile.children.cycles-pp.hrtimer_start_range_ns
> 0.98 ± 4% -0.8 0.22 ± 13% perf-profile.children.cycles-pp.check_preempt_curr
> 1.03 -0.8 0.28 ± 4% perf-profile.children.cycles-pp.recalc_sigpending
> 0.97 ± 12% -0.7 0.22 ± 10% perf-profile.children.cycles-pp.__update_load_avg_se
> 0.90 ± 8% -0.7 0.18 ± 12% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
> 0.95 ± 10% -0.7 0.23 ± 10% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
> 0.94 ± 33% -0.7 0.23 ± 40% perf-profile.children.cycles-pp.aa_may_signal
> 0.94 ± 6% -0.7 0.23 ± 2% perf-profile.children.cycles-pp.copy_siginfo_to_user
> 0.95 ± 6% -0.7 0.27 ± 9% perf-profile.children.cycles-pp.__might_fault
> 0.85 ± 5% -0.7 0.19 ± 11% perf-profile.children.cycles-pp.check_preempt_wakeup
> 0.83 ± 9% -0.6 0.19 ± 7% perf-profile.children.cycles-pp.___perf_sw_event
> 1.00 ± 2% -0.6 0.36 ± 20% perf-profile.children.cycles-pp.__copy_siginfo_from_user
> 0.79 ± 17% -0.6 0.19 ± 11% perf-profile.children.cycles-pp.put_prev_entity
> 0.80 ± 12% -0.6 0.20 ± 8% perf-profile.children.cycles-pp.pick_next_entity
> 0.82 ± 2% -0.5 0.29 ± 24% perf-profile.children.cycles-pp.__x64_sys_getpid
> 0.68 ± 2% -0.5 0.16 ± 3% perf-profile.children.cycles-pp.update_cfs_group
> 0.87 ± 17% -0.5 0.38 perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
> 0.53 ± 6% -0.5 0.06 ± 16% perf-profile.children.cycles-pp.complete_signal
> 0.58 ± 6% -0.5 0.12 ± 10% perf-profile.children.cycles-pp.sched_clock_cpu
> 0.55 ± 5% -0.4 0.10 ± 12% perf-profile.children.cycles-pp.hrtimer_try_to_cancel
> 0.79 ± 12% -0.4 0.35 ± 7% perf-profile.children.cycles-pp.cpumask_next_wrap
> 0.62 ± 18% -0.4 0.19 ± 10% perf-profile.children.cycles-pp._find_next_bit
> 0.71 ± 2% -0.4 0.28 ± 34% perf-profile.children.cycles-pp.__task_pid_nr_ns
> 0.62 ± 3% -0.4 0.21 ± 30% perf-profile.children.cycles-pp.__x64_sys_getuid
> 0.53 ± 14% -0.4 0.14 ± 17% perf-profile.children.cycles-pp.__calc_delta
> 0.59 ± 8% -0.4 0.20 ± 32% perf-profile.children.cycles-pp.check_kill_permission
> 0.50 ± 6% -0.4 0.11 ± 11% perf-profile.children.cycles-pp.sched_clock
> 0.50 ± 13% -0.4 0.12 ± 10% perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
> 0.47 ± 6% -0.4 0.09 ± 18% perf-profile.children.cycles-pp.perf_tp_event
> 0.48 ± 6% -0.4 0.10 ± 9% perf-profile.children.cycles-pp.native_sched_clock
> 0.48 ± 3% -0.4 0.11 ± 4% perf-profile.children.cycles-pp.___might_sleep
> 0.74 ± 12% -0.4 0.39 perf-profile.children.cycles-pp.kmem_cache_free
> 0.45 ± 8% -0.3 0.13 ± 13% perf-profile.children.cycles-pp._copy_to_user
> 0.36 ± 5% -0.3 0.06 ± 8% perf-profile.children.cycles-pp.__set_task_blocked
> 0.35 ± 22% -0.3 0.07 ± 11% perf-profile.children.cycles-pp.cpumask_next
> 0.38 ± 7% -0.3 0.10 ± 14% perf-profile.children.cycles-pp.ktime_get
> 0.46 ± 10% -0.3 0.19 ± 36% perf-profile.children.cycles-pp.__radix_tree_lookup
> 0.64 ± 4% -0.3 0.37 ± 6% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> 0.33 ± 16% -0.3 0.06 ± 14% perf-profile.children.cycles-pp.cpuacct_charge
> 0.53 -0.3 0.26 ± 33% perf-profile.children.cycles-pp.send_signal
> 0.33 ± 16% -0.3 0.07 ± 12% perf-profile.children.cycles-pp.__wrgsbase_inactive
> 0.34 ± 8% -0.2 0.09 ± 5% perf-profile.children.cycles-pp.update_min_vruntime
> 0.39 ± 3% -0.2 0.15 ± 12% perf-profile.children.cycles-pp.copy_user_generic_unrolled
> 0.32 ± 6% -0.2 0.07 ± 6% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
> 0.28 ± 2% -0.2 0.04 ± 71% perf-profile.children.cycles-pp.lock_hrtimer_base
> 0.30 ± 9% -0.2 0.06 ± 7% perf-profile.children.cycles-pp.set_next_buddy
> 0.35 ± 5% -0.2 0.11 ± 36% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
> 0.30 ± 5% -0.2 0.07 ± 17% perf-profile.children.cycles-pp.rb_erase
> 0.32 ± 2% -0.2 0.10 ± 9% perf-profile.children.cycles-pp.__x86_retpoline_rax
> 0.30 ± 5% -0.2 0.08 ± 14% perf-profile.children.cycles-pp.__might_sleep
> 0.29 ± 7% -0.2 0.07 ± 12% perf-profile.children.cycles-pp.__clear_user
> 0.31 ± 14% -0.2 0.09 ± 9% perf-profile.children.cycles-pp.clear_buddies
> 0.25 ± 3% -0.2 0.05 perf-profile.children.cycles-pp.recalc_sigpending_tsk
> 0.29 ± 12% -0.2 0.09 ± 22% perf-profile.children.cycles-pp.prepare_signal
> 0.30 ± 4% -0.2 0.11 ± 34% perf-profile.children.cycles-pp.from_kuid_munged
> 0.27 ± 4% -0.2 0.09 ± 36% perf-profile.children.cycles-pp.map_id_up
> 0.21 ± 21% -0.2 0.03 ± 70% perf-profile.children.cycles-pp.enqueue_hrtimer
> 0.22 ± 13% -0.2 0.06 ± 8% perf-profile.children.cycles-pp.get_timespec64
> 0.22 ± 17% -0.2 0.06 ± 86% perf-profile.children.cycles-pp.__cgroup_account_cputime
> 0.20 ± 4% -0.2 0.04 ± 76% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
> 0.20 ± 6% -0.2 0.05 perf-profile.children.cycles-pp.__list_del_entry_valid
> 0.19 ± 7% -0.2 0.04 ± 71% perf-profile.children.cycles-pp.perf_trace_sched_switch
> 0.23 ± 8% -0.1 0.08 ± 31% perf-profile.children.cycles-pp.audit_signal_info
> 0.22 ± 5% -0.1 0.08 ± 16% perf-profile.children.cycles-pp.rcu_read_unlock_strict
> 0.17 ± 9% -0.1 0.03 ± 70% perf-profile.children.cycles-pp.read_tsc
> 0.18 ± 7% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.audit_signal_info_syscall
> 0.17 ± 5% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.kmem_cache_alloc
> 0.00 +0.5 0.51 ± 3% perf-profile.children.cycles-pp.dec_rlimit_ucounts
> 0.00 +0.7 0.67 ± 2% perf-profile.children.cycles-pp.inc_rlimit_ucounts_and_test
> 0.68 ± 16% +2.4 3.11 ± 53% perf-profile.children.cycles-pp.__lock_task_sighand
> 0.42 ± 6% +2.9 3.30 ± 51% perf-profile.children.cycles-pp._raw_spin_lock_irq
> 35.50 ± 7% +9.5 45.03 perf-profile.children.cycles-pp.__x64_sys_rt_sigqueueinfo
> 34.19 ± 7% +10.4 44.55 perf-profile.children.cycles-pp.do_rt_sigqueueinfo
> 33.35 ± 7% +10.9 44.22 perf-profile.children.cycles-pp.kill_pid_info
> 81.79 +12.5 94.33 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 19.21 ± 4% +19.4 38.65 ± 8% perf-profile.children.cycles-pp.__send_signal
> 21.36 ± 6% +20.4 41.78 ± 4% perf-profile.children.cycles-pp.__x64_sys_rt_sigtimedwait
> 20.86 ± 3% +21.5 42.34 ± 3% perf-profile.children.cycles-pp.do_send_sig_info
> 19.50 ± 5% +21.8 41.33 ± 4% perf-profile.children.cycles-pp.do_sigtimedwait
> 59.88 ± 2% +28.0 87.85 ± 2% perf-profile.children.cycles-pp.do_syscall_64
> 3.81 ± 8% +30.9 34.66 ± 9% perf-profile.children.cycles-pp.dequeue_signal
> 3.09 ± 8% +31.3 34.41 ± 9% perf-profile.children.cycles-pp.__dequeue_signal
> 1.74 ± 9% +32.1 33.87 ± 9% perf-profile.children.cycles-pp.__sigqueue_free
> 1.95 ± 8% +32.5 34.40 ± 8% perf-profile.children.cycles-pp.__sigqueue_alloc
> 0.00 +33.3 33.32 ± 9% perf-profile.children.cycles-pp.put_ucounts
> 0.00 +33.6 33.58 ± 8% perf-profile.children.cycles-pp.get_ucounts
> 1.07 ± 11% +68.8 69.87 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.74 ± 39% +71.3 72.04 ± 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
> 11.75 ± 2% -7.8 3.95 ± 26% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
> 8.02 -5.3 2.70 ± 25% perf-profile.self.cycles-pp.syscall_return_via_sysret
> 5.78 ± 25% -5.1 0.71 ± 43% perf-profile.self.cycles-pp.apparmor_task_kill
> 4.55 ± 30% -4.0 0.53 ± 34% perf-profile.self.cycles-pp.aa_get_task_label
> 6.02 -4.0 2.05 ± 28% perf-profile.self.cycles-pp.__entry_text_start
> 1.79 ± 9% -1.7 0.06 ± 13% perf-profile.self.cycles-pp.__sigqueue_alloc
> 2.22 ± 9% -1.7 0.50 ± 9% perf-profile.self.cycles-pp.__sched_text_start
> 2.25 ± 13% -1.7 0.55 ± 11% perf-profile.self.cycles-pp.load_new_mm_cr3
> 2.40 ± 6% -1.5 0.85 ± 6% perf-profile.self.cycles-pp.available_idle_cpu
> 1.57 ± 12% -1.3 0.30 ± 10% perf-profile.self.cycles-pp.switch_fpu_return
> 1.74 ± 6% -1.2 0.50 ± 11% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
> 1.79 ± 8% -1.2 0.60 ± 6% perf-profile.self.cycles-pp.select_idle_sibling
> 1.41 ± 7% -1.1 0.30 ± 11% perf-profile.self.cycles-pp.__switch_to
> 1.25 ± 3% -1.0 0.27 ± 8% perf-profile.self.cycles-pp.update_curr
> 1.23 ± 12% -1.0 0.27 ± 13% perf-profile.self.cycles-pp.__switch_to_asm
> 1.17 ± 7% -0.9 0.23 ± 9% perf-profile.self.cycles-pp.update_load_avg
> 1.15 ± 2% -0.9 0.21 ± 5% perf-profile.self.cycles-pp.switch_mm_irqs_off
> 1.13 ± 3% -0.8 0.28 ± 8% perf-profile.self.cycles-pp._raw_spin_lock
> 0.94 ± 13% -0.8 0.14 ± 9% perf-profile.self.cycles-pp.try_to_wake_up
> 1.06 -0.7 0.32 ± 28% perf-profile.self.cycles-pp.do_syscall_64
> 0.93 ± 12% -0.7 0.22 ± 12% perf-profile.self.cycles-pp.__update_load_avg_se
> 0.93 ± 33% -0.7 0.23 ± 40% perf-profile.self.cycles-pp.aa_may_signal
> 0.90 ± 10% -0.7 0.22 ± 9% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
> 0.77 -0.6 0.15 ± 5% perf-profile.self.cycles-pp.recalc_sigpending
> 0.77 ± 13% -0.6 0.20 ± 7% perf-profile.self.cycles-pp.reweight_entity
> 0.74 ± 7% -0.6 0.19 ± 11% perf-profile.self.cycles-pp.update_rq_clock
> 0.67 ± 7% -0.5 0.14 ± 6% perf-profile.self.cycles-pp.___perf_sw_event
> 0.63 ± 7% -0.5 0.11 ± 11% perf-profile.self.cycles-pp.enqueue_entity
> 0.67 ± 3% -0.5 0.15 ± 3% perf-profile.self.cycles-pp.update_cfs_group
> 0.66 ± 9% -0.5 0.16 ± 12% perf-profile.self.cycles-pp.pick_next_task_fair
> 0.61 ± 13% -0.5 0.12 ± 4% perf-profile.self.cycles-pp.enqueue_task_fair
> 0.60 ± 5% -0.5 0.11 ± 14% perf-profile.self.cycles-pp.dequeue_task_fair
> 0.62 ± 8% -0.5 0.14 ± 5% perf-profile.self.cycles-pp.select_task_rq_fair
> 0.63 ± 6% -0.5 0.16 ± 10% perf-profile.self.cycles-pp.do_sigtimedwait
> 0.61 ± 18% -0.4 0.19 ± 10% perf-profile.self.cycles-pp._find_next_bit
> 0.68 ± 3% -0.4 0.27 ± 33% perf-profile.self.cycles-pp.__task_pid_nr_ns
> 0.54 ± 3% -0.4 0.14 ± 6% perf-profile.self.cycles-pp.__dequeue_signal
> 0.51 -0.4 0.12 ± 20% perf-profile.self.cycles-pp.__send_signal
> 0.53 ± 14% -0.4 0.14 ± 15% perf-profile.self.cycles-pp.__calc_delta
> 0.49 ± 13% -0.4 0.12 ± 10% perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
> 0.46 ± 7% -0.4 0.09 ± 10% perf-profile.self.cycles-pp.native_sched_clock
> 0.46 ± 3% -0.4 0.10 ± 4% perf-profile.self.cycles-pp.___might_sleep
> 0.50 ± 7% -0.3 0.16 ± 23% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
> 0.50 ± 3% -0.3 0.17 ± 29% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
> 0.44 ± 14% -0.3 0.10 ± 16% perf-profile.self.cycles-pp.schedule
> 0.42 ± 5% -0.3 0.09 ± 9% perf-profile.self.cycles-pp.finish_task_switch
> 0.43 ± 11% -0.3 0.10 ± 9% perf-profile.self.cycles-pp.pick_next_entity
> 0.41 ± 10% -0.3 0.08 ± 5% perf-profile.self.cycles-pp.__x64_sys_rt_sigtimedwait
> 0.44 ± 6% -0.3 0.13 ± 6% perf-profile.self.cycles-pp._copy_from_user
> 0.50 ± 3% -0.3 0.20 ± 11% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> 0.63 ± 20% -0.3 0.35 ± 6% perf-profile.self.cycles-pp.kmem_cache_free
> 0.37 ± 3% -0.3 0.10 ± 12% perf-profile.self.cycles-pp.check_preempt_wakeup
> 0.37 ± 7% -0.3 0.10 ± 4% perf-profile.self.cycles-pp.dequeue_entity
> 0.33 ± 15% -0.3 0.06 ± 13% perf-profile.self.cycles-pp.cpuacct_charge
> 0.46 ± 11% -0.3 0.19 ± 36% perf-profile.self.cycles-pp.__radix_tree_lookup
> 0.34 ± 3% -0.3 0.07 ± 11% perf-profile.self.cycles-pp._raw_spin_lock_irq
> 0.29 ± 7% -0.3 0.04 ± 71% perf-profile.self.cycles-pp.perf_tp_event
> 0.36 ± 11% -0.3 0.10 ± 4% perf-profile.self.cycles-pp.__might_fault
> 0.32 ± 7% -0.2 0.08 ± 5% perf-profile.self.cycles-pp.update_min_vruntime
> 0.31 ± 17% -0.2 0.07 ± 12% perf-profile.self.cycles-pp.__wrgsbase_inactive
> 0.37 ± 2% -0.2 0.14 ± 13% perf-profile.self.cycles-pp.copy_user_generic_unrolled
> 0.30 ± 4% -0.2 0.07 ± 7% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
> 0.26 ± 8% -0.2 0.04 ± 71% perf-profile.self.cycles-pp.set_next_buddy
> 0.29 ± 5% -0.2 0.07 ± 17% perf-profile.self.cycles-pp.rb_erase
> 0.28 ± 11% -0.2 0.06 ± 14% perf-profile.self.cycles-pp.set_next_entity
> 0.30 ± 4% -0.2 0.09 ± 30% perf-profile.self.cycles-pp.__x64_sys_getuid
> 0.39 ± 13% -0.2 0.18 ± 9% perf-profile.self.cycles-pp.cpumask_next_wrap
> 0.27 ± 5% -0.2 0.07 ± 17% perf-profile.self.cycles-pp.__might_sleep
> 0.31 ± 3% -0.2 0.12 ± 28% perf-profile.self.cycles-pp.__x64_sys_rt_sigqueueinfo
> 0.24 -0.2 0.05 perf-profile.self.cycles-pp.recalc_sigpending_tsk
> 0.28 ± 5% -0.2 0.09 ± 36% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
> 0.27 ± 12% -0.2 0.08 ± 24% perf-profile.self.cycles-pp.prepare_signal
> 0.27 ± 13% -0.2 0.08 ± 10% perf-profile.self.cycles-pp.clear_buddies
> 0.26 ± 3% -0.2 0.08 ± 12% perf-profile.self.cycles-pp.__x86_retpoline_rax
> 0.21 ± 17% -0.2 0.04 ± 71% perf-profile.self.cycles-pp.put_prev_entity
> 0.26 ± 4% -0.2 0.09 ± 36% perf-profile.self.cycles-pp.map_id_up
> 0.23 ± 13% -0.2 0.07 ± 11% perf-profile.self.cycles-pp.schedule_hrtimeout_range_clock
> 0.19 ± 8% -0.2 0.04 ± 70% perf-profile.self.cycles-pp.ktime_get
> 0.19 ± 9% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.perf_trace_sched_switch
> 0.18 ± 4% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.__list_del_entry_valid
> 0.19 ± 8% -0.1 0.05 ± 78% perf-profile.self.cycles-pp.check_kill_permission
> 0.26 -0.1 0.13 ± 26% perf-profile.self.cycles-pp.send_signal
> 0.21 ± 4% -0.1 0.08 ± 26% perf-profile.self.cycles-pp.__x64_sys_getpid
> 0.20 ± 2% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.security_task_kill
> 0.17 ± 4% -0.1 0.05 ± 78% perf-profile.self.cycles-pp.kill_pid_info
> 0.16 ± 8% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.rcu_read_unlock_strict
> 0.16 ± 10% -0.1 0.05 ± 72% perf-profile.self.cycles-pp.audit_signal_info_syscall
> 0.24 ± 8% -0.1 0.14 ± 3% perf-profile.self.cycles-pp.dequeue_signal
> 0.12 ± 4% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.do_rt_sigqueueinfo
> 0.12 ± 11% -0.1 0.04 ± 70% perf-profile.self.cycles-pp.do_send_sig_info
> 0.15 ± 6% -0.0 0.10 perf-profile.self.cycles-pp.kmem_cache_alloc
> 0.00 +0.1 0.07 ± 11% perf-profile.self.cycles-pp.get_ucounts
> 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.put_ucounts
> 0.88 ± 7% +0.2 1.04 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.00 +0.5 0.51 ± 3% perf-profile.self.cycles-pp.dec_rlimit_ucounts
> 0.00 +0.7 0.67 perf-profile.self.cycles-pp.inc_rlimit_ucounts_and_test
> 0.74 ± 39% +71.3 72.04 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>
>
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Oliver Sang

2021-02-25 20:40:19

by Alexey Gladkov

[permalink] [raw]

Subject: Re: d28296d248: stress-ng.sigsegv.ops_per_sec -82.7% regression

---

On Wed, Feb 24, 2021 at 12:50:21PM -0600, Eric W. Biederman wrote:
> Alexey Gladkov <[email protected]> writes:
>
> > On Wed, Feb 24, 2021 at 10:54:17AM -0600, Eric W. Biederman wrote:
> >> kernel test robot <[email protected]> writes:
> >>
> >> > Greeting,
> >> >
> >> > FYI, we noticed a -82.7% regression of stress-ng.sigsegv.ops_per_sec due to commit:
> >> >
> >> >
> >> > commit: d28296d2484fa11e94dff65e93eb25802a443d47 ("[PATCH v7 5/7] Reimplement RLIMIT_SIGPENDING on top of ucounts")
> >> > url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
> >> > base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next
> >> >
> >> > in testcase: stress-ng
> >> > on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
> >> > with following parameters:
> >> >
> >> > nr_threads: 100%
> >> > disk: 1HDD
> >> > testtime: 60s
> >> > class: interrupt
> >> > test: sigsegv
> >> > cpufreq_governor: performance
> >> > ucode: 0x42e
> >> >
> >> >
> >> > In addition to that, the commit also has significant impact on the
> >> > following tests:
> >>
> >> Thank you. Now we have a sense of where we need to test the performance
> >> of these changes carefully.
> >
> > One of the reasons for this is that I rolled back the patch that changed
> > the ucounts.count type to atomic_t. Now get_ucounts() is forced to use a
> > spin_lock to increase the reference count.
>
> Which given the hickups with getting a working version seems justified.
>
> Now we can add incremental patches on top to improve the performance.

I'm not sure that get_ucounts() should be used in __sigqueue_alloc() [1].
I tried removing it and running KASAN tests that were failing before. So
far, I have not found any problems.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/legion/linux.git/tree/kernel/signal.c?h=patchset/per-userspace-rlimit/v7.1&id=2d4a2e2be7db42c95acb98abfc2a9b370ddd0604#n428

--
Rgrds, legion

2021-03-03 00:01:22

by kernel test robot

[permalink] [raw]

Subject: 5b5c35b757: BUG:KASAN:use-after-free_in_dec_rlimit_ucounts

---

Greeting,

FYI, we noticed the following commit (built with gcc-9):

commit: 5b5c35b757a192cc54eb96137761da67e7ce0520 ("[PATCH v7 6/7] Reimplement RLIMIT_MEMLOCK on top of ucounts")
url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next

in testcase: trinity
version: trinity-static-x86_64-x86_64-f93256fb_2019-08-28
with following parameters:

group: ["group-00", "group-01", "group-02", "group-03", "group-04"]

test-description: Trinity is a linux system call fuzz tester.
test-url: http://codemonkey.org.uk/projects/trinity/


on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G

caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):


+------------------------------------------------+------------+------------+
| | d28296d248 | 5b5c35b757 |
+------------------------------------------------+------------+------------+
| boot_failures | 0 | 5 |
| BUG:KASAN:use-after-free_in_dec_rlimit_ucounts | 0 | 5 |
| canonical_address#:#[##] | 0 | 1 |
| RIP:dec_rlimit_ucounts | 0 | 1 |
| Kernel_panic-not_syncing:Fatal_exception | 0 | 1 |
+------------------------------------------------+------------+------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>


[ 235.817305] BUG: KASAN: use-after-free in dec_rlimit_ucounts (kbuild/src/consumer/kernel/ucount.c:302 (discriminator 3))
[ 235.818278] Read of size 8 at addr ffff88810687b1d0 by task trinity-c2/4730
[ 235.819266]
[ 235.819585] CPU: 0 PID: 4730 Comm: trinity-c2 Not tainted 5.11.0-rc7-00017-g5b5c35b757a1 #1
[ 235.820944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[ 235.822206] Call Trace:
[ 235.822646] dump_stack (kbuild/src/consumer/lib/dump_stack.c:131)
[ 235.823195] print_address_description+0x21/0x140
[ 235.824066] ? dec_rlimit_ucounts (kbuild/src/consumer/kernel/ucount.c:302 (discriminator 3))
[ 235.824815] kasan_report.cold (kbuild/src/consumer/mm/kasan/report.c:397 kbuild/src/consumer/mm/kasan/report.c:413)
[ 235.825530] ? dec_rlimit_ucounts (kbuild/src/consumer/kernel/ucount.c:302 (discriminator 3))
[ 235.826260] __asan_load8 (kbuild/src/consumer/mm/kasan/generic.c:252)
[ 235.826848] dec_rlimit_ucounts (kbuild/src/consumer/kernel/ucount.c:302 (discriminator 3))
[ 235.827549] user_shm_unlock (kbuild/src/consumer/include/linux/spinlock.h:394 kbuild/src/consumer/mm/mlock.c:851)
[ 235.828237] shmem_lock (kbuild/src/consumer/mm/shmem.c:2247)
[ 235.828867] ksys_shmctl+0xc1b/0xe70
[ 235.829658] ? __fsnotify_parent (kbuild/src/consumer/fs/notify/fsnotify.c:200)
[ 235.830391] ? shm_mmap (kbuild/src/consumer/ipc/shm.c:1139)
[ 235.831035] ? ftrace_likely_update (kbuild/src/consumer/kernel/trace/trace_branch.c:225)
[ 235.831765] ? ftrace_likely_update (kbuild/src/consumer/kernel/trace/trace_branch.c:225)
[ 235.832545] ? pvclock_clocksource_read (kbuild/src/consumer/arch/x86/kernel/pvclock.c:80)
[ 235.833390] ? ftrace_likely_update (kbuild/src/consumer/kernel/trace/trace_branch.c:227)
[ 235.834154] ? ftrace_likely_update (kbuild/src/consumer/kernel/trace/trace_branch.c:225)
[ 235.834866] ? get_vtime_delta (kbuild/src/consumer/kernel/sched/cputime.c:658 (discriminator 3))
[ 235.835497] ? ftrace_likely_update (kbuild/src/consumer/kernel/trace/trace_branch.c:227)
[ 235.836217] __x64_sys_shmctl (kbuild/src/consumer/ipc/shm.c:1193)
[ 235.836912] do_syscall_64 (kbuild/src/consumer/arch/x86/entry/common.c:46)
[ 235.837540] entry_SYSCALL_64_after_hwframe (kbuild/src/consumer/arch/x86/entry/entry_64.S:127)
[ 235.838348] RIP: 0033:0x453b29
[ 235.838867] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 3b 84 00 00 c3 66 2e 0f 1f 84 00 00 00 00
All code
========
0: 00 f3 add %dh,%bl
2: c3 retq
3: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
a: 00 00 00
d: 0f 1f 40 00 nopl 0x0(%rax)
11: 48 89 f8 mov %rdi,%rax
14: 48 89 f7 mov %rsi,%rdi
17: 48 89 d6 mov %rdx,%rsi
1a: 48 89 ca mov %rcx,%rdx
1d: 4d 89 c2 mov %r8,%r10
20: 4d 89 c8 mov %r9,%r8
23: 4c 8b 4c 24 08 mov 0x8(%rsp),%r9
28: 0f 05 syscall
2a:* 48 3d 01 f0 ff ff cmp $0xfffffffffffff001,%rax <-- trapping instruction
30: 0f 83 3b 84 00 00 jae 0x8471
36: c3 retq
37: 66 data16
38: 2e cs
39: 0f .byte 0xf
3a: 1f (bad)
3b: 84 00 test %al,(%rax)
3d: 00 00 add %al,(%rax)
...

Code starting with the faulting instruction
===========================================
0: 48 3d 01 f0 ff ff cmp $0xfffffffffffff001,%rax
6: 0f 83 3b 84 00 00 jae 0x8447
c: c3 retq
d: 66 data16
e: 2e cs
f: 0f .byte 0xf
10: 1f (bad)
11: 84 00 test %al,(%rax)
13: 00 00 add %al,(%rax)
...
[ 235.841605] RSP: 002b:00007ffd5b1195a8 EFLAGS: 00000246 ORIG_RAX: 000000000000001f
[ 235.842731] RAX: ffffffffffffffda RBX: 000000000000001f RCX: 0000000000453b29
[ 235.843745] RDX: 00007f903f38e000 RSI: 000000000000000c RDI: 0000000000000000
[ 235.844880] RBP: 00007ffd5b119650 R08: 00000000000000de R09: ffffffffffffffff
[ 235.845958] R10: 0000000000000200 R11: 0000000000000246 R12: 0000000000000002
[ 235.847062] R13: 00007f903f899058 R14: 00000000010a2830 R15: 00007f903f899000
[ 235.848178]
[ 235.848538] Allocated by task 4043:
[ 235.849196] kasan_save_stack (kbuild/src/consumer/mm/kasan/common.c:39)
[ 235.849892] ____kasan_kmalloc+0x87/0xb0
[ 235.850711] __kasan_slab_alloc (kbuild/src/consumer/mm/kasan/common.c:438)
[ 235.851394] kmem_cache_alloc (kbuild/src/consumer/include/linux/kasan.h:209 kbuild/src/consumer/mm/slab.h:512 kbuild/src/consumer/mm/slub.c:2892 kbuild/src/consumer/mm/slub.c:2900 kbuild/src/consumer/mm/slub.c:2905)
[ 235.852114] create_user_ns (kbuild/src/consumer/include/linux/slab.h:672 kbuild/src/consumer/kernel/user_namespace.c:105)
[ 235.852801] unshare_userns (kbuild/src/consumer/kernel/user_namespace.c:168)
[ 235.853464] ksys_unshare (kbuild/src/consumer/kernel/fork.c:2956)
[ 235.854145] __x64_sys_unshare (kbuild/src/consumer/kernel/fork.c:3031)
[ 235.854827] do_syscall_64 (kbuild/src/consumer/arch/x86/entry/common.c:46)
[ 235.855485] entry_SYSCALL_64_after_hwframe (kbuild/src/consumer/arch/x86/entry/entry_64.S:127)
[ 235.856368]
[ 235.856733] Freed by task 5:
[ 235.857292] kasan_save_stack (kbuild/src/consumer/mm/kasan/common.c:39)
[ 235.857967] kasan_set_track (kbuild/src/consumer/mm/kasan/common.c:46)
[ 235.858627] kasan_set_free_info (kbuild/src/consumer/mm/kasan/generic.c:358)
[ 235.859329] ____kasan_slab_free (kbuild/src/consumer/mm/kasan/common.c:364)
[ 235.860068] __kasan_slab_free (kbuild/src/consumer/mm/kasan/common.c:370)
[ 235.860761] kmem_cache_free (kbuild/src/consumer/mm/slub.c:1580 kbuild/src/consumer/mm/slub.c:3143 kbuild/src/consumer/mm/slub.c:3159)
[ 235.861439] free_user_ns (kbuild/src/consumer/kernel/user_namespace.c:39 kbuild/src/consumer/kernel/user_namespace.c:202)
[ 235.862059] process_one_work (kbuild/src/consumer/arch/x86/include/asm/jump_label.h:25 kbuild/src/consumer/include/linux/jump_label.h:200 kbuild/src/consumer/include/trace/events/workqueue.h:108 kbuild/src/consumer/kernel/workqueue.c:2280)
[ 235.862754] worker_thread (kbuild/src/consumer/include/linux/list.h:282 kbuild/src/consumer/kernel/workqueue.c:2422)
[ 235.863378] kthread (kbuild/src/consumer/kernel/kthread.c:292)
[ 235.863912] ret_from_fork (kbuild/src/consumer/arch/x86/entry/entry_64.S:302)
[ 235.864576]
[ 235.864940] Last potentially related work creation:
[ 235.865717] kasan_save_stack (kbuild/src/consumer/mm/kasan/common.c:39)
[ 235.866382] kasan_record_aux_stack (kbuild/src/consumer/mm/kasan/generic.c:344)
[ 235.867124] insert_work (kbuild/src/consumer/include/linux/instrumented.h:71 kbuild/src/consumer/include/asm-generic/bitops/instrumented-non-atomic.h:134 kbuild/src/consumer/kernel/workqueue.c:615 kbuild/src/consumer/kernel/workqueue.c:622 kbuild/src/consumer/kernel/workqueue.c:1334)
[ 235.867769] __queue_work (kbuild/src/consumer/kernel/workqueue.c:1500)
[ 235.868448] queue_work_on (kbuild/src/consumer/kernel/workqueue.c:1525)
[ 235.869116] __put_user_ns (kbuild/src/consumer/kernel/user_namespace.c:210)
[ 235.869752] cleanup_net (kbuild/src/consumer/include/linux/user_namespace.h:142 kbuild/src/consumer/include/linux/user_namespace.h:139 kbuild/src/consumer/net/core/net_namespace.c:622)
[ 235.870370] process_one_work (kbuild/src/consumer/arch/x86/include/asm/jump_label.h:25 kbuild/src/consumer/include/linux/jump_label.h:200 kbuild/src/consumer/include/trace/events/workqueue.h:108 kbuild/src/consumer/kernel/workqueue.c:2280)
[ 235.871057] worker_thread (kbuild/src/consumer/include/linux/list.h:282 kbuild/src/consumer/kernel/workqueue.c:2422)
[ 235.871706] kthread (kbuild/src/consumer/kernel/kthread.c:292)
[ 235.872321] ret_from_fork (kbuild/src/consumer/arch/x86/entry/entry_64.S:302)
[ 235.872974]
[ 235.873343] Second to last potentially related work creation:
[ 235.874266] kasan_save_stack (kbuild/src/consumer/mm/kasan/common.c:39)
[ 235.874934] kasan_record_aux_stack (kbuild/src/consumer/mm/kasan/generic.c:344)
[ 235.875695] insert_work (kbuild/src/consumer/include/linux/instrumented.h:71 kbuild/src/consumer/include/asm-generic/bitops/instrumented-non-atomic.h:134 kbuild/src/consumer/kernel/workqueue.c:615 kbuild/src/consumer/kernel/workqueue.c:622 kbuild/src/consumer/kernel/workqueue.c:1334)
[ 235.876369] __queue_work (kbuild/src/consumer/kernel/workqueue.c:1500)
[ 235.877033] queue_work_on (kbuild/src/consumer/kernel/workqueue.c:1525)
[ 235.877677] __put_user_ns (kbuild/src/consumer/kernel/user_namespace.c:210)
[ 235.878286] put_cred_rcu (kbuild/src/consumer/include/linux/user_namespace.h:142 kbuild/src/consumer/kernel/cred.c:125)
[ 235.878875] rcu_do_batch+0x1e2/0x940
[ 235.879591] rcu_core (kbuild/src/consumer/kernel/rcu/tree.c:2723)
[ 235.880212] rcu_core_si (kbuild/src/consumer/kernel/rcu/tree.c:2737)
[ 235.880832] __do_softirq (kbuild/src/consumer/arch/x86/include/asm/jump_label.h:25 kbuild/src/consumer/include/linux/jump_label.h:200 kbuild/src/consumer/include/trace/events/irq.h:142 kbuild/src/consumer/kernel/softirq.c:344)
[ 235.881520]
[ 235.881867] The buggy address belongs to the object at ffff88810687aff8
[ 235.881867] which belongs to the cache user_namespace of size 592
[ 235.883687] The buggy address is located 472 bytes inside of
[ 235.883687] 592-byte region [ffff88810687aff8, ffff88810687b248)
[ 235.885560] The buggy address belongs to the page:
[ 235.886343] page:0000000066c321d7 refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff88810687b3a8 pfn:0x106878
[ 235.887924] head:0000000066c321d7 order:2 compound_mapcount:0 compound_pincount:0
[ 235.889214] flags: 0x8000000000010200(slab|head)
[ 235.889993] raw: 8000000000010200 ffff888100c25648 ffff888100c25648 ffff888100c8ccc0
[ 235.891261] raw: ffff88810687b3a8 000000000011000a 00000001ffffffff 0000000000000000
[ 235.892557] page dumped because: kasan: bad access detected
[ 235.893490]
[ 235.893838] Memory state around the buggy address:
[ 235.894636] ffff88810687b080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 235.895858] ffff88810687b100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 235.897103] >ffff88810687b180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 235.898294] ^
[ 235.899208] ffff88810687b200: fb fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc
[ 235.900359] ffff88810687b280: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 235.902866] ==================================================================
[ 235.904088] Disabling lock debugging due to kernel taint

Kboot worker: lkp-worker52
Elapsed time: 240



To reproduce:

# build kernel
cd linux
cp config-5.11.0-rc7-00017-g5b5c35b757a1 .config
make HOSTCC=gcc-9 CC=gcc-9 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email



Thanks,
Oliver Sang

---

Attachments:

(No filename) (12.95 kB)
config-5.11.0-rc7-00017-g5b5c35b757a1 (128.35 kB)
job-script (4.37 kB)
dmesg.xz (14.62 kB)
Download all attachments

2021-03-03 00:21:11

by kernel test robot

[permalink] [raw]

Subject: e1e57d56fe: stress-ng.access.ops_per_sec -41.6% regression

---

Greeting,

FYI, we noticed a -41.6% regression of stress-ng.access.ops_per_sec due to commit:


commit: e1e57d56fef0dd06daf3743f0948da86dfaf2f6f ("[PATCH v7 2/7] Add a reference to ucounts for each cred")
url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next

in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

nr_threads: 10%
disk: 1HDD
testtime: 60s
fs: ext4
class: filesystem
test: access
cpufreq_governor: performance
ucode: 0x5003006


In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.access.ops_per_sec -42.3% regression |
| test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory |
| test parameters | class=filesystem |
| | cpufreq_governor=performance |
| | disk=1HDD |
| | fs=xfs |
| | nr_threads=10% |
| | test=access |
| | testtime=60s |
| | ucode=0x5003006 |
+------------------+----------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.access.ops_per_sec -44.5% regression |
| test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory |
| test parameters | class=filesystem |
| | cpufreq_governor=performance |
| | disk=1HDD |
| | fs=f2fs |
| | nr_threads=10% |
| | test=access |
| | testtime=60s |
| | ucode=0x5003006 |
+------------------+----------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml
bin/lkp run compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
filesystem/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/access/stress-ng/60s/0x5003006

commit:
a88f997967 ("Increase size of ucounts to atomic_long_t")
e1e57d56fe ("Add a reference to ucounts for each cred")

a88f9979677c6f55 e1e57d56fef0dd06daf3743f094
---------------- ---------------------------
%stddev %change %stddev
\ | \
2158451 -41.6% 1260149 ? 3% stress-ng.access.ops
35974 -41.6% 21002 ? 3% stress-ng.access.ops_per_sec
10.85 ? 10% -46.4% 5.82 stress-ng.time.user_time
0.21 ? 8% -0.1 0.12 mpstat.cpu.all.usr%
0.16 ? 4% -18.5% 0.13 ? 12% sched_debug.cfs_rq:/.nr_running.avg
884086 ? 13% -22.7% 683381 ? 8% sched_debug.cpu.max_idle_balance_cost.max
46438 ? 21% -55.1% 20855 ? 32% sched_debug.cpu.max_idle_balance_cost.stddev
81265 +4.1% 84595 ? 2% interrupts.CAL:Function_call_interrupts
83.33 ? 34% +4735.4% 4029 ? 96% interrupts.CPU42.NMI:Non-maskable_interrupts
83.33 ? 34% +4735.4% 4029 ? 96% interrupts.CPU42.PMI:Performance_monitoring_interrupts
93.17 ? 26% +4031.1% 3848 ? 96% interrupts.CPU43.NMI:Non-maskable_interrupts
93.17 ? 26% +4031.1% 3848 ? 96% interrupts.CPU43.PMI:Performance_monitoring_interrupts
93.33 ? 27% +2585.9% 2506 ?133% interrupts.CPU44.NMI:Non-maskable_interrupts
93.33 ? 27% +2585.9% 2506 ?133% interrupts.CPU44.PMI:Performance_monitoring_interrupts
93.33 ? 26% +2141.6% 2092 ?126% interrupts.CPU45.NMI:Non-maskable_interrupts
93.33 ? 26% +2141.6% 2092 ?126% interrupts.CPU45.PMI:Performance_monitoring_interrupts
106.00 ? 23% +2191.5% 2429 ?132% interrupts.CPU46.NMI:Non-maskable_interrupts
106.00 ? 23% +2191.5% 2429 ?132% interrupts.CPU46.PMI:Performance_monitoring_interrupts
115.33 ? 23% +1392.6% 1721 ?144% interrupts.CPU47.NMI:Non-maskable_interrupts
115.33 ? 23% +1392.6% 1721 ?144% interrupts.CPU47.PMI:Performance_monitoring_interrupts
1585 ?133% +201.4% 4778 ? 69% interrupts.CPU48.NMI:Non-maskable_interrupts
1585 ?133% +201.4% 4778 ? 69% interrupts.CPU48.PMI:Performance_monitoring_interrupts
101.17 ? 18% +53.7% 155.50 ? 6% interrupts.CPU51.NMI:Non-maskable_interrupts
101.17 ? 18% +53.7% 155.50 ? 6% interrupts.CPU51.PMI:Performance_monitoring_interrupts
112.83 ? 6% +883.3% 1109 ?193% interrupts.CPU54.NMI:Non-maskable_interrupts
112.83 ? 6% +883.3% 1109 ?193% interrupts.CPU54.PMI:Performance_monitoring_interrupts
101.00 ? 22% +1390.3% 1505 ?139% interrupts.CPU79.NMI:Non-maskable_interrupts
101.00 ? 22% +1390.3% 1505 ?139% interrupts.CPU79.PMI:Performance_monitoring_interrupts
111.00 ? 25% +344.6% 493.50 ?155% interrupts.CPU80.NMI:Non-maskable_interrupts
111.00 ? 25% +344.6% 493.50 ?155% interrupts.CPU80.PMI:Performance_monitoring_interrupts
8.32 ? 7% +54.7% 12.88 perf-stat.i.MPKI
2.865e+09 -30.8% 1.984e+09 ? 2% perf-stat.i.branch-instructions
0.76 ? 2% +0.1 0.84 perf-stat.i.branch-miss-rate%
20772880 -21.8% 16252845 perf-stat.i.branch-misses
1.85 ? 2% +52.3% 2.81 ? 2% perf-stat.i.cpi
2.734e+10 +3.6% 2.832e+10 perf-stat.i.cpu-cycles
4.122e+09 -33.7% 2.733e+09 perf-stat.i.dTLB-loads
2.299e+09 -36.4% 1.463e+09 ? 2% perf-stat.i.dTLB-stores
86.57 -5.3 81.28 perf-stat.i.iTLB-load-miss-rate%
15778062 ? 3% -39.1% 9607502 ? 3% perf-stat.i.iTLB-load-misses
1.479e+10 -32.4% 9.991e+09 ? 2% perf-stat.i.instructions
977.00 ? 2% +9.0% 1064 ? 3% perf-stat.i.instructions-per-iTLB-miss
0.55 ? 2% -33.4% 0.36 ? 2% perf-stat.i.ipc
0.28 +3.6% 0.29 perf-stat.i.metric.GHz
0.41 ? 42% -28.7% 0.29 ? 2% perf-stat.i.metric.K/sec
98.22 -32.9% 65.92 ? 2% perf-stat.i.metric.M/sec
88.10 ? 2% +7.9 96.03 perf-stat.i.node-load-miss-rate%
3379054 ? 14% +82.1% 6154210 ? 7% perf-stat.i.node-load-misses
441950 ? 29% -54.3% 202107 ? 15% perf-stat.i.node-loads
12643063 ? 9% -23.6% 9657658 ? 7% perf-stat.i.node-store-misses
8.36 ? 7% +55.9% 13.04 perf-stat.overall.MPKI
0.73 +0.1 0.82 perf-stat.overall.branch-miss-rate%
1.85 ? 2% +53.3% 2.84 ? 2% perf-stat.overall.cpi
87.69 -5.6 82.07 perf-stat.overall.iTLB-load-miss-rate%
937.91 ? 2% +11.0% 1041 ? 3% perf-stat.overall.instructions-per-iTLB-miss
0.54 ? 2% -34.8% 0.35 ? 2% perf-stat.overall.ipc
88.63 ? 2% +8.2 96.83 perf-stat.overall.node-load-miss-rate%
2.819e+09 -30.8% 1.952e+09 ? 2% perf-stat.ps.branch-instructions
20439060 -21.8% 15990109 perf-stat.ps.branch-misses
2.69e+10 +3.6% 2.786e+10 perf-stat.ps.cpu-cycles
4.057e+09 -33.7% 2.689e+09 perf-stat.ps.dTLB-loads
2.263e+09 -36.4% 1.439e+09 ? 2% perf-stat.ps.dTLB-stores
15526322 ? 3% -39.1% 9453069 ? 3% perf-stat.ps.iTLB-load-misses
1.455e+10 -32.4% 9.83e+09 ? 2% perf-stat.ps.instructions
3325252 ? 14% +82.1% 6055830 ? 7% perf-stat.ps.node-load-misses
434905 ? 29% -54.3% 198876 ? 15% perf-stat.ps.node-loads
12441875 ? 9% -23.6% 9503199 ? 7% perf-stat.ps.node-store-misses
9.193e+11 -32.7% 6.187e+11 ? 2% perf-stat.total.instructions
6.30 ? 10% -5.6 0.74 ? 12% perf-profile.calltrace.cycles-pp.free_uid.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.25 ? 10% -5.5 0.71 ? 13% perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat.do_syscall_64
6.23 ? 10% -5.5 0.71 ? 13% perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat
5.64 ? 12% -4.1 1.52 ? 11% perf-profile.calltrace.cycles-pp.key_put.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.59 ? 19% -3.7 2.89 ? 12% perf-profile.calltrace.cycles-pp.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.50 ? 19% -3.7 2.84 ? 12% perf-profile.calltrace.cycles-pp.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.83 ? 20% -3.3 2.50 ? 13% perf-profile.calltrace.cycles-pp.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.63 ? 21% -3.2 2.38 ? 13% perf-profile.calltrace.cycles-pp.ext4_setattr.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64
5.39 ? 19% -3.1 2.29 ? 13% perf-profile.calltrace.cycles-pp.__mark_inode_dirty.ext4_setattr.notify_change.chmod_common.__x64_sys_fchmod
4.48 ? 21% -2.6 1.87 ? 13% perf-profile.calltrace.cycles-pp.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr.notify_change.chmod_common
4.53 ? 18% -2.4 2.09 ? 10% perf-profile.calltrace.cycles-pp.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.98 ? 22% -1.8 1.22 ? 9% perf-profile.calltrace.cycles-pp.apparmor_cred_prepare.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
2.65 ? 17% -1.6 1.05 ? 14% perf-profile.calltrace.cycles-pp.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr.notify_change
3.02 ? 12% -1.5 1.50 ? 8% perf-profile.calltrace.cycles-pp.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.19 ? 14% -1.3 0.85 ? 13% perf-profile.calltrace.cycles-pp.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr
2.03 ? 15% -1.3 0.76 ? 13% perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty
2.77 ? 11% -1.2 1.57 ? 8% perf-profile.calltrace.cycles-pp.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.38 ? 11% -1.1 1.29 ? 7% perf-profile.calltrace.cycles-pp.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.44 ? 11% -1.1 1.39 ? 8% perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.01 ? 14% -1.0 0.96 ? 6% perf-profile.calltrace.cycles-pp.apparmor_cred_free.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
1.78 ? 31% -1.0 0.78 ? 12% perf-profile.calltrace.cycles-pp.__ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr.notify_change
2.08 ? 12% -1.0 1.12 ? 7% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.55 ? 11% -0.7 0.88 ? 11% perf-profile.calltrace.cycles-pp.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.52 ? 11% -0.7 0.87 ? 11% perf-profile.calltrace.cycles-pp.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.94 ? 9% -0.7 0.29 ?100% perf-profile.calltrace.cycles-pp.kfree.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
1.43 ? 10% -0.6 0.80 ? 11% perf-profile.calltrace.cycles-pp.kmem_cache_alloc.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.44 ? 9% -0.6 0.81 ? 13% perf-profile.calltrace.cycles-pp.__kmalloc.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
0.88 ? 9% -0.6 0.27 ?100% perf-profile.calltrace.cycles-pp.kmem_cache_free.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.33 ? 10% -0.6 0.74 ? 10% perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.do_faccessat.do_syscall_64
1.20 ? 11% -0.5 0.66 ? 12% perf-profile.calltrace.cycles-pp.strncpy_from_user.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64
1.33 ? 27% +1.3 2.65 ? 12% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +1.4 1.38 ? 10% perf-profile.calltrace.cycles-pp.override_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
20.13 ? 10% +3.3 23.38 ? 9% perf-profile.calltrace.cycles-pp.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
19.93 ? 11% +3.7 23.61 ? 9% perf-profile.calltrace.cycles-pp.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +14.5 14.53 ? 8% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat
0.00 +15.0 15.01 ? 8% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat
0.00 +16.1 16.09 ? 8% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat.do_syscall_64
0.00 +16.4 16.42 ? 8% perf-profile.calltrace.cycles-pp.get_ucounts.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +16.5 16.55 ? 8% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64
0.00 +16.9 16.88 ? 8% perf-profile.calltrace.cycles-pp.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.30 ? 10% -5.6 0.74 ? 12% perf-profile.children.cycles-pp.free_uid
6.26 ? 10% -5.5 0.72 ? 13% perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
6.23 ? 10% -5.5 0.71 ? 13% perf-profile.children.cycles-pp.refcount_dec_not_one
5.66 ? 12% -4.1 1.54 ? 11% perf-profile.children.cycles-pp.key_put
6.59 ? 19% -3.7 2.89 ? 12% perf-profile.children.cycles-pp.__x64_sys_fchmod
6.50 ? 19% -3.7 2.84 ? 12% perf-profile.children.cycles-pp.chmod_common
5.83 ? 20% -3.3 2.50 ? 12% perf-profile.children.cycles-pp.notify_change
5.64 ? 21% -3.3 2.38 ? 13% perf-profile.children.cycles-pp.ext4_setattr
5.39 ? 19% -3.1 2.29 ? 13% perf-profile.children.cycles-pp.__mark_inode_dirty
4.48 ? 21% -2.6 1.87 ? 13% perf-profile.children.cycles-pp.ext4_dirty_inode
4.54 ? 18% -2.4 2.09 ? 10% perf-profile.children.cycles-pp.security_prepare_creds
2.98 ? 22% -1.8 1.22 ? 9% perf-profile.children.cycles-pp.apparmor_cred_prepare
2.65 ? 17% -1.6 1.05 ? 14% perf-profile.children.cycles-pp.__ext4_journal_start_sb
3.64 ? 11% -1.6 2.05 ? 8% perf-profile.children.cycles-pp.filename_lookup
3.02 ? 12% -1.5 1.50 ? 8% perf-profile.children.cycles-pp.security_cred_free
3.24 ? 11% -1.4 1.83 ? 8% perf-profile.children.cycles-pp.path_lookupat
2.19 ? 14% -1.3 0.85 ? 13% perf-profile.children.cycles-pp.jbd2__journal_start
2.04 ? 15% -1.3 0.77 ? 14% perf-profile.children.cycles-pp.start_this_handle
2.38 ? 11% -1.1 1.29 ? 6% perf-profile.children.cycles-pp.__do_sys_newfstatat
2.04 ? 14% -1.1 0.96 ? 6% perf-profile.children.cycles-pp.apparmor_cred_free
1.79 ? 31% -1.0 0.78 ? 13% perf-profile.children.cycles-pp.__ext4_mark_inode_dirty
2.08 ? 12% -1.0 1.12 ? 8% perf-profile.children.cycles-pp.vfs_statx
2.02 ? 12% -0.9 1.14 ? 10% perf-profile.children.cycles-pp.user_path_at_empty
1.98 ? 12% -0.9 1.12 ? 10% perf-profile.children.cycles-pp.getname_flags
1.89 ? 10% -0.8 1.08 ? 11% perf-profile.children.cycles-pp.kmem_cache_alloc
1.79 ? 10% -0.8 1.00 ? 10% perf-profile.children.cycles-pp.link_path_walk
1.53 ? 11% -0.7 0.85 ? 11% perf-profile.children.cycles-pp.strncpy_from_user
1.49 ? 9% -0.7 0.83 ? 12% perf-profile.children.cycles-pp.__kmalloc
1.05 ? 26% -0.6 0.48 ? 13% perf-profile.children.cycles-pp.ext4_mark_iloc_dirty
1.19 ? 10% -0.5 0.67 ? 8% perf-profile.children.cycles-pp.kmem_cache_free
0.90 ? 23% -0.5 0.42 ? 12% perf-profile.children.cycles-pp.ext4_do_update_inode
0.86 ? 13% -0.5 0.40 ? 14% perf-profile.children.cycles-pp.__ext4_journal_stop
0.76 ? 14% -0.4 0.34 ? 15% perf-profile.children.cycles-pp.jbd2_journal_stop
0.95 ? 9% -0.4 0.54 ? 12% perf-profile.children.cycles-pp.kfree
0.84 ? 12% -0.4 0.43 ? 12% perf-profile.children.cycles-pp.common_perm_cond
0.68 ? 37% -0.4 0.29 ? 13% perf-profile.children.cycles-pp.ext4_reserve_inode_write
0.74 ? 15% -0.4 0.35 ? 11% perf-profile.children.cycles-pp._raw_read_lock
0.79 ? 12% -0.4 0.40 ? 12% perf-profile.children.cycles-pp.common_perm
0.77 ? 12% -0.4 0.39 ? 11% perf-profile.children.cycles-pp.inode_permission
0.83 ? 12% -0.4 0.48 ? 10% perf-profile.children.cycles-pp.walk_component
0.45 ? 15% -0.3 0.13 ? 20% perf-profile.children.cycles-pp.add_transaction_credits
0.58 ? 12% -0.3 0.27 ? 8% perf-profile.children.cycles-pp.generic_permission
0.67 ? 11% -0.3 0.37 ? 12% perf-profile.children.cycles-pp.__check_object_size
0.59 ? 13% -0.3 0.34 ? 10% perf-profile.children.cycles-pp.complete_walk
0.56 ? 9% -0.2 0.31 ? 11% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.50 ? 10% -0.2 0.26 ? 9% perf-profile.children.cycles-pp.vfs_getattr
0.47 ? 12% -0.2 0.23 ? 15% perf-profile.children.cycles-pp.stop_this_handle
0.56 ? 13% -0.2 0.32 ? 10% perf-profile.children.cycles-pp.unlazy_walk
0.49 ? 11% -0.2 0.25 ? 9% perf-profile.children.cycles-pp.security_inode_getattr
0.42 ? 28% -0.2 0.19 ? 18% perf-profile.children.cycles-pp.ext4_journal_check_start
0.43 ? 12% -0.2 0.22 ? 14% perf-profile.children.cycles-pp.security_path_chmod
0.45 ? 10% -0.2 0.24 ? 13% perf-profile.children.cycles-pp.obj_cgroup_charge
0.38 ? 36% -0.2 0.17 ? 10% perf-profile.children.cycles-pp.ext4_get_inode_loc
0.44 ? 6% -0.2 0.24 ? 16% perf-profile.children.cycles-pp.refill_obj_stock
0.42 ? 10% -0.2 0.22 ? 9% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.37 ? 36% -0.2 0.17 ? 12% perf-profile.children.cycles-pp.__ext4_get_inode_loc
0.46 ? 10% -0.2 0.27 ? 10% perf-profile.children.cycles-pp.lookup_fast
0.44 ? 13% -0.2 0.25 ? 13% perf-profile.children.cycles-pp.__legitimize_path
0.41 ? 11% -0.2 0.24 ? 17% perf-profile.children.cycles-pp.___might_sleep
0.41 ? 9% -0.2 0.23 ? 6% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.26 ? 57% -0.2 0.09 ? 9% perf-profile.children.cycles-pp.ext4_inode_csum_set
0.27 ? 17% -0.2 0.10 ? 6% perf-profile.children.cycles-pp.capable_wrt_inode_uidgid
0.32 ? 36% -0.2 0.16 ? 12% perf-profile.children.cycles-pp.revert_creds
0.27 ? 33% -0.2 0.10 ? 14% perf-profile.children.cycles-pp.__ext4_journal_get_write_access
0.43 ? 10% -0.2 0.27 ? 12% perf-profile.children.cycles-pp.crc32c_pcl_intel_update
0.26 ? 17% -0.2 0.10 ? 6% perf-profile.children.cycles-pp.ns_capable_common
0.26 ? 16% -0.2 0.10 ? 6% perf-profile.children.cycles-pp.security_capable
0.23 ? 18% -0.2 0.08 ? 8% perf-profile.children.cycles-pp.apparmor_capable
0.36 ? 11% -0.2 0.21 ? 9% perf-profile.children.cycles-pp.path_put
0.32 ? 14% -0.1 0.17 ? 9% perf-profile.children.cycles-pp.__check_heap_object
0.35 ? 12% -0.1 0.21 ? 9% perf-profile.children.cycles-pp.dput
0.34 ? 11% -0.1 0.19 ? 9% perf-profile.children.cycles-pp.__entry_text_start
0.32 ? 10% -0.1 0.18 ? 12% perf-profile.children.cycles-pp.__d_lookup_rcu
0.19 ? 14% -0.1 0.07 ? 11% perf-profile.children.cycles-pp.__ext4_handle_dirty_metadata
0.30 ? 10% -0.1 0.18 ? 7% perf-profile.children.cycles-pp.path_init
0.28 ? 13% -0.1 0.16 ? 6% perf-profile.children.cycles-pp.cp_new_stat
0.27 ? 11% -0.1 0.16 ? 15% perf-profile.children.cycles-pp.__might_sleep
0.19 ? 34% -0.1 0.08 ? 10% perf-profile.children.cycles-pp.ext4_inode_csum
0.26 ? 27% -0.1 0.16 ? 17% perf-profile.children.cycles-pp.map_id_up
0.22 ? 14% -0.1 0.12 ? 14% perf-profile.children.cycles-pp._cond_resched
0.15 ? 17% -0.1 0.05 ? 45% perf-profile.children.cycles-pp.jbd2_journal_get_write_access
0.22 ? 15% -0.1 0.13 ? 15% perf-profile.children.cycles-pp.__legitimize_mnt
0.22 ? 11% -0.1 0.13 ? 14% perf-profile.children.cycles-pp.__wake_up_common_lock
0.19 ? 13% -0.1 0.10 ? 13% perf-profile.children.cycles-pp.__mod_memcg_state
0.17 ? 12% -0.1 0.09 ? 13% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.18 ? 11% -0.1 0.10 ? 15% perf-profile.children.cycles-pp.step_into
0.17 ? 13% -0.1 0.09 ? 10% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.17 ? 11% -0.1 0.10 ? 16% perf-profile.children.cycles-pp.lockref_get_not_dead
0.17 ? 13% -0.1 0.10 ? 17% perf-profile.children.cycles-pp.memset_erms
0.15 ? 15% -0.1 0.08 ? 13% perf-profile.children.cycles-pp._copy_to_user
0.16 ? 12% -0.1 0.10 ? 15% perf-profile.children.cycles-pp.__might_fault
0.10 ? 16% -0.1 0.05 ? 47% perf-profile.children.cycles-pp.__put_cred
0.13 ? 17% -0.1 0.08 ? 9% perf-profile.children.cycles-pp.lockref_put_return
0.08 ? 13% -0.1 0.03 ? 99% perf-profile.children.cycles-pp.__find_get_block
0.12 ? 14% -0.1 0.06 ? 14% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.11 ? 16% -0.1 0.06 ? 11% perf-profile.children.cycles-pp.rcu_all_qs
0.28 ? 7% -0.1 0.23 ? 5% perf-profile.children.cycles-pp.scheduler_tick
0.09 ? 19% -0.0 0.05 ? 45% perf-profile.children.cycles-pp.security_inode_permission
0.07 ? 9% -0.0 0.03 ? 99% perf-profile.children.cycles-pp.terminate_walk
0.11 ? 12% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.__virt_addr_valid
0.11 ? 9% -0.0 0.07 ? 18% perf-profile.children.cycles-pp.__getblk_gfp
0.10 ? 13% -0.0 0.06 ? 13% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.11 ? 9% -0.0 0.08 ? 9% perf-profile.children.cycles-pp.calc_global_load_tick
0.09 ? 12% -0.0 0.06 ? 13% perf-profile.children.cycles-pp.__fget_light
0.08 ? 8% +0.3 0.39 ? 9% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.14 ? 12% +1.2 1.39 ? 10% perf-profile.children.cycles-pp.override_creds
1.34 ? 27% +1.3 2.66 ? 12% perf-profile.children.cycles-pp.menu_select
20.16 ? 10% +3.2 23.40 ? 9% perf-profile.children.cycles-pp.put_cred_rcu
19.94 ? 11% +3.7 23.62 ? 9% perf-profile.children.cycles-pp.prepare_creds
0.00 +16.4 16.43 ? 8% perf-profile.children.cycles-pp.get_ucounts
0.00 +16.9 16.88 ? 8% perf-profile.children.cycles-pp.put_ucounts
0.00 +29.5 29.55 ? 8% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.22 ? 8% +32.6 32.80 ? 8% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
13.87 ? 11% -9.6 4.27 ? 13% perf-profile.self.cycles-pp.prepare_creds
6.20 ? 10% -5.5 0.70 ? 13% perf-profile.self.cycles-pp.refcount_dec_not_one
5.62 ? 12% -4.1 1.52 ? 11% perf-profile.self.cycles-pp.key_put
5.18 ? 9% -2.4 2.73 ? 10% perf-profile.self.cycles-pp.put_cred_rcu
2.96 ? 22% -1.7 1.22 ? 10% perf-profile.self.cycles-pp.apparmor_cred_prepare
2.02 ? 15% -1.1 0.96 ? 6% perf-profile.self.cycles-pp.apparmor_cred_free
0.85 ? 16% -0.6 0.28 ? 18% perf-profile.self.cycles-pp.start_this_handle
0.96 ? 9% -0.4 0.51 ? 9% perf-profile.self.cycles-pp.link_path_walk
0.96 ? 10% -0.4 0.57 ? 12% perf-profile.self.cycles-pp.kmem_cache_alloc
0.78 ? 12% -0.4 0.40 ? 12% perf-profile.self.cycles-pp.common_perm
0.73 ? 15% -0.4 0.35 ? 11% perf-profile.self.cycles-pp._raw_read_lock
0.83 ? 10% -0.4 0.48 ? 8% perf-profile.self.cycles-pp.kmem_cache_free
0.45 ? 15% -0.3 0.13 ? 18% perf-profile.self.cycles-pp.add_transaction_credits
0.71 ? 11% -0.3 0.39 ? 8% perf-profile.self.cycles-pp.strncpy_from_user
0.73 ? 11% -0.3 0.43 ? 14% perf-profile.self.cycles-pp.__kmalloc
0.62 ? 11% -0.3 0.36 ? 14% perf-profile.self.cycles-pp.kfree
0.56 ? 9% -0.2 0.31 ? 11% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.38 ? 30% -0.2 0.17 ? 19% perf-profile.self.cycles-pp.ext4_journal_check_start
0.44 ? 11% -0.2 0.23 ? 14% perf-profile.self.cycles-pp.obj_cgroup_charge
0.43 ? 6% -0.2 0.23 ? 14% perf-profile.self.cycles-pp.refill_obj_stock
0.41 ? 11% -0.2 0.23 ? 17% perf-profile.self.cycles-pp.___might_sleep
0.36 ? 8% -0.2 0.19 ? 8% perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.32 ? 36% -0.2 0.15 ? 11% perf-profile.self.cycles-pp.revert_creds
0.24 ? 18% -0.2 0.07 ? 18% perf-profile.self.cycles-pp.jbd2_journal_stop
0.42 ? 11% -0.2 0.26 ? 12% perf-profile.self.cycles-pp.crc32c_pcl_intel_update
0.39 ? 13% -0.2 0.23 ? 12% perf-profile.self.cycles-pp.do_faccessat
0.23 ? 18% -0.2 0.08 ? 8% perf-profile.self.cycles-pp.apparmor_capable
0.25 ? 16% -0.2 0.10 ? 23% perf-profile.self.cycles-pp.stop_this_handle
0.31 ? 14% -0.1 0.16 ? 10% perf-profile.self.cycles-pp.__check_heap_object
0.34 ? 11% -0.1 0.19 ? 10% perf-profile.self.cycles-pp.__entry_text_start
0.31 ? 9% -0.1 0.18 ? 12% perf-profile.self.cycles-pp.__d_lookup_rcu
0.28 ? 9% -0.1 0.15 ? 11% perf-profile.self.cycles-pp.generic_permission
0.26 ? 11% -0.1 0.15 ? 9% perf-profile.self.cycles-pp.path_init
0.24 ? 13% -0.1 0.13 ? 16% perf-profile.self.cycles-pp.__check_object_size
0.22 ? 21% -0.1 0.12 ? 19% perf-profile.self.cycles-pp.ext4_do_update_inode
0.19 ? 39% -0.1 0.09 ? 13% perf-profile.self.cycles-pp.__ext4_get_inode_loc
0.26 ? 29% -0.1 0.16 ? 18% perf-profile.self.cycles-pp.map_id_up
0.25 ? 11% -0.1 0.15 ? 14% perf-profile.self.cycles-pp.__might_sleep
0.22 ? 14% -0.1 0.13 ? 15% perf-profile.self.cycles-pp.__legitimize_mnt
0.20 ? 15% -0.1 0.12 ? 18% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.19 ? 13% -0.1 0.10 ? 13% perf-profile.self.cycles-pp.__mod_memcg_state
0.19 ? 16% -0.1 0.11 ? 12% perf-profile.self.cycles-pp.walk_component
0.20 ? 7% -0.1 0.12 ? 14% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.21 ? 13% -0.1 0.13 ? 23% perf-profile.self.cycles-pp.inode_permission
0.17 ? 11% -0.1 0.10 ? 16% perf-profile.self.cycles-pp.step_into
0.17 ? 11% -0.1 0.10 ? 16% perf-profile.self.cycles-pp.lockref_get_not_dead
0.14 ? 17% -0.1 0.08 ? 14% perf-profile.self.cycles-pp.lookup_fast
0.16 ? 14% -0.1 0.09 ? 19% perf-profile.self.cycles-pp.memset_erms
0.11 ? 12% -0.1 0.05 ? 70% perf-profile.self.cycles-pp._cond_resched
0.10 ? 19% -0.1 0.04 ? 72% perf-profile.self.cycles-pp.__put_cred
0.13 ? 16% -0.1 0.07 ? 6% perf-profile.self.cycles-pp.getname_flags
0.13 ? 19% -0.1 0.08 ? 22% perf-profile.self.cycles-pp.filename_lookup
0.12 ? 12% -0.1 0.07 ? 14% perf-profile.self.cycles-pp.rcu_read_unlock_strict
0.12 ? 14% -0.1 0.06 ? 14% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.12 ? 18% -0.1 0.07 ? 5% perf-profile.self.cycles-pp.lockref_put_return
0.10 ? 10% -0.0 0.05 ? 46% perf-profile.self.cycles-pp.__virt_addr_valid
0.09 ? 19% -0.0 0.05 ? 45% perf-profile.self.cycles-pp.security_inode_permission
0.08 ? 13% -0.0 0.04 ? 73% perf-profile.self.cycles-pp.__fget_light
0.10 ? 18% -0.0 0.06 ? 18% perf-profile.self.cycles-pp.notify_change
0.10 ? 13% -0.0 0.06 ? 13% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.10 ? 18% -0.0 0.06 ? 11% perf-profile.self.cycles-pp.path_lookupat
0.11 ? 9% -0.0 0.08 ? 9% perf-profile.self.cycles-pp.calc_global_load_tick
0.07 ? 10% +0.2 0.23 ? 11% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.00 +0.2 0.18 ? 16% perf-profile.self.cycles-pp.get_ucounts
0.00 +0.2 0.18 ? 16% perf-profile.self.cycles-pp.put_ucounts
0.65 ? 23% +0.8 1.48 ? 14% perf-profile.self.cycles-pp.cpuidle_enter_state
0.13 ? 11% +1.2 1.38 ? 10% perf-profile.self.cycles-pp.override_creds
0.79 ? 43% +1.3 2.12 ? 16% perf-profile.self.cycles-pp.menu_select
0.22 ? 7% +3.0 3.25 ? 10% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.00 +29.5 29.55 ? 8% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



stress-ng.access.ops

2.6e+06 +-----------------------------------------------------------------+
| +. |
2.4e+06 |-+ : ++. +. +.++.+ .+ |
| + : ++.+ .+.+ + + +.++. .++. +. |
2.2e+06 |.++.++. :+: + + + ++. +.++.++.++.|
2e+06 |-+ + + + |
| |
1.8e+06 |-+ |
| |
1.6e+06 |-+ |
1.4e+06 |-+ |
| O O O O OO |
1.2e+06 |-OO O O OO OO OO OO OO OO O OO OO OO O O |
| |
1e+06 +-----------------------------------------------------------------+


stress-ng.access.ops_per_sec

45000 +-------------------------------------------------------------------+
| |
40000 |-+ ++.+ |
| : : .++.++.+.++.++.+.++. .+ |
|.+ +. +. : +.+.++ ++ +.+.++.+ .+.++. .++.|
35000 |-++.+ + + + ++ |
| |
30000 |-+ |
| |
25000 |-+ |
| |
| OO OO OO O OO OO O O O O O OO OO OO |
20000 |-+ O O OO O OO O O |
| |
15000 +-------------------------------------------------------------------+


[*] bisect-good sample
[O] bisect-bad sample

***************************************************************************************************
lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
filesystem/gcc-9/performance/1HDD/xfs/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/access/stress-ng/60s/0x5003006

commit:
a88f997967 ("Increase size of ucounts to atomic_long_t")
e1e57d56fe ("Add a reference to ucounts for each cred")

a88f9979677c6f55 e1e57d56fef0dd06daf3743f094
---------------- ---------------------------
%stddev %change %stddev
\ | \
2101125 -42.3% 1213160 ? 4% stress-ng.access.ops
35018 -42.3% 20219 ? 4% stress-ng.access.ops_per_sec
10.19 -41.9% 5.92 ? 4% stress-ng.time.user_time
0.27 ? 6% -0.1 0.20 ? 2% mpstat.cpu.all.usr%
303.99 -1.3% 300.00 pmeter.Average_Active_Power
1136 ? 10% +35.9% 1545 ? 19% sched_debug.cfs_rq:/.runnable_avg.max
1135 ? 10% +36.0% 1545 ? 19% sched_debug.cfs_rq:/.util_avg.max
31.83 ? 9% -15.7% 26.83 ? 7% sched_debug.cpu.nr_uninterruptible.max
16112 ? 11% +23.7% 19924 ? 10% numa-meminfo.node0.Mapped
2090 ? 19% +38.8% 2900 ? 17% numa-meminfo.node0.PageTables
3068 ? 98% +180.5% 8607 ? 42% numa-meminfo.node0.Shmem
21447 ? 9% -18.2% 17535 ? 11% numa-meminfo.node1.Mapped
16587 ? 20% -33.9% 10971 ? 31% numa-meminfo.node1.Shmem
4093 ? 11% +23.8% 5069 ? 10% numa-vmstat.node0.nr_mapped
521.00 ? 19% +37.5% 716.17 ? 16% numa-vmstat.node0.nr_page_table_pages
766.83 ? 99% +180.5% 2151 ? 42% numa-vmstat.node0.nr_shmem
5432 ? 9% -18.4% 4433 ? 11% numa-vmstat.node1.nr_mapped
4147 ? 20% -33.8% 2744 ? 31% numa-vmstat.node1.nr_shmem
106.33 ? 24% +1398.7% 1593 ?178% interrupts.CPU15.NMI:Non-maskable_interrupts
106.33 ? 24% +1398.7% 1593 ?178% interrupts.CPU15.PMI:Performance_monitoring_interrupts
101.33 ? 32% +1158.9% 1275 ?196% interrupts.CPU16.NMI:Non-maskable_interrupts
101.33 ? 32% +1158.9% 1275 ?196% interrupts.CPU16.PMI:Performance_monitoring_interrupts
105.00 ? 23% +1277.5% 1446 ?199% interrupts.CPU17.NMI:Non-maskable_interrupts
105.00 ? 23% +1277.5% 1446 ?199% interrupts.CPU17.PMI:Performance_monitoring_interrupts
117.33 ? 7% +960.7% 1244 ?195% interrupts.CPU21.NMI:Non-maskable_interrupts
117.33 ? 7% +960.7% 1244 ?195% interrupts.CPU21.PMI:Performance_monitoring_interrupts
114.33 ? 9% +2296.1% 2739 ?133% interrupts.CPU22.NMI:Non-maskable_interrupts
114.33 ? 9% +2296.1% 2739 ?133% interrupts.CPU22.PMI:Performance_monitoring_interrupts
122.33 ? 4% +1530.1% 1994 ?139% interrupts.CPU3.NMI:Non-maskable_interrupts
122.33 ? 4% +1530.1% 1994 ?139% interrupts.CPU3.PMI:Performance_monitoring_interrupts
118.00 ? 9% +1711.9% 2138 ?130% interrupts.CPU46.NMI:Non-maskable_interrupts
118.00 ? 9% +1711.9% 2138 ?130% interrupts.CPU46.PMI:Performance_monitoring_interrupts
120.17 ? 23% +964.5% 1279 ?182% interrupts.CPU52.NMI:Non-maskable_interrupts
120.17 ? 23% +964.5% 1279 ?182% interrupts.CPU52.PMI:Performance_monitoring_interrupts
86.50 ? 36% +1641.4% 1506 ?190% interrupts.CPU58.NMI:Non-maskable_interrupts
86.50 ? 36% +1641.4% 1506 ?190% interrupts.CPU58.PMI:Performance_monitoring_interrupts
109.83 ? 21% +873.1% 1068 ?191% interrupts.CPU69.NMI:Non-maskable_interrupts
109.83 ? 21% +873.1% 1068 ?191% interrupts.CPU69.PMI:Performance_monitoring_interrupts
103.33 ? 22% +490.3% 610.00 ?164% interrupts.CPU87.NMI:Non-maskable_interrupts
103.33 ? 22% +490.3% 610.00 ?164% interrupts.CPU87.PMI:Performance_monitoring_interrupts
109.17 ? 27% +1859.8% 2139 ?139% interrupts.CPU94.NMI:Non-maskable_interrupts
109.17 ? 27% +1859.8% 2139 ?139% interrupts.CPU94.PMI:Performance_monitoring_interrupts
8.31 ? 7% +53.2% 12.74 ? 2% perf-stat.i.MPKI
2.845e+09 -30.2% 1.985e+09 ? 2% perf-stat.i.branch-instructions
0.84 ? 3% +0.1 0.95 perf-stat.i.branch-miss-rate%
22369243 ? 4% -16.3% 18726571 ? 2% perf-stat.i.branch-misses
1.86 +53.6% 2.86 ? 3% perf-stat.i.cpi
2.766e+10 +3.9% 2.873e+10 perf-stat.i.cpu-cycles
4.076e+09 ? 2% -33.0% 2.732e+09 ? 4% perf-stat.i.dTLB-loads
2.322e+09 -36.8% 1.467e+09 ? 3% perf-stat.i.dTLB-stores
83.49 -7.8 75.66 perf-stat.i.iTLB-load-miss-rate%
14193692 -41.6% 8290248 ? 2% perf-stat.i.iTLB-load-misses
1.479e+10 -32.0% 1.005e+10 ? 2% perf-stat.i.instructions
1146 +12.8% 1293 perf-stat.i.instructions-per-iTLB-miss
0.55 -33.1% 0.37 ? 3% perf-stat.i.ipc
0.29 +3.9% 0.30 perf-stat.i.metric.GHz
97.79 -32.5% 65.96 ? 3% perf-stat.i.metric.M/sec
93.66 +3.2 96.82 perf-stat.i.node-load-miss-rate%
5224551 ? 8% +42.9% 7464161 ? 5% perf-stat.i.node-load-misses
319707 ? 17% -42.2% 184671 ? 9% perf-stat.i.node-loads
13233767 ? 9% -23.4% 10143303 ? 8% perf-stat.i.node-store-misses
8.40 ? 7% +52.0% 12.77 ? 2% perf-stat.overall.MPKI
0.79 ? 3% +0.2 0.94 perf-stat.overall.branch-miss-rate%
1.87 +52.9% 2.86 ? 3% perf-stat.overall.cpi
0.00 ? 41% +0.0 0.00 ? 30% perf-stat.overall.dTLB-load-miss-rate%
0.00 ? 12% +0.0 0.00 ? 16% perf-stat.overall.dTLB-store-miss-rate%
84.65 -8.3 76.34 perf-stat.overall.iTLB-load-miss-rate%
1041 +16.4% 1212 perf-stat.overall.instructions-per-iTLB-miss
0.53 -34.5% 0.35 ? 3% perf-stat.overall.ipc
94.24 +3.3 97.58 perf-stat.overall.node-load-miss-rate%
2.8e+09 -30.2% 1.953e+09 ? 2% perf-stat.ps.branch-instructions
21989609 ? 4% -16.2% 18425367 ? 2% perf-stat.ps.branch-misses
2.722e+10 +3.8% 2.827e+10 perf-stat.ps.cpu-cycles
4.01e+09 ? 2% -33.0% 2.688e+09 ? 4% perf-stat.ps.dTLB-loads
2.285e+09 -36.8% 1.443e+09 ? 3% perf-stat.ps.dTLB-stores
13967002 -41.6% 8156895 ? 2% perf-stat.ps.iTLB-load-misses
1.455e+10 -32.0% 9.888e+09 ? 2% perf-stat.ps.instructions
5142072 ? 8% +42.8% 7344021 ? 5% perf-stat.ps.node-load-misses
314656 ? 17% -42.2% 181724 ? 9% perf-stat.ps.node-loads
13025851 ? 9% -23.4% 9980421 ? 8% perf-stat.ps.node-store-misses
9.19e+11 -32.1% 6.244e+11 ? 3% perf-stat.total.instructions
5.77 ? 6% -5.0 0.73 ? 10% perf-profile.calltrace.cycles-pp.free_uid.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.73 ? 6% -5.0 0.71 ? 10% perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat.do_syscall_64
5.71 ? 6% -5.0 0.70 ? 10% perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat
8.88 ? 11% -4.7 4.16 ? 8% perf-profile.calltrace.cycles-pp.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
8.78 ? 11% -4.7 4.10 ? 8% perf-profile.calltrace.cycles-pp.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
8.05 ? 11% -4.3 3.74 ? 8% perf-profile.calltrace.cycles-pp.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.84 ? 11% -4.2 3.60 ? 8% perf-profile.calltrace.cycles-pp.xfs_vn_setattr.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64
7.77 ? 11% -4.2 3.56 ? 8% perf-profile.calltrace.cycles-pp.xfs_setattr_nonsize.xfs_vn_setattr.notify_change.chmod_common.__x64_sys_fchmod
5.72 ? 11% -4.2 1.56 ? 12% perf-profile.calltrace.cycles-pp.key_put.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.83 ? 13% -2.8 1.99 ? 10% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_setattr_nonsize.xfs_vn_setattr.notify_change.chmod_common
4.70 ? 13% -2.8 1.92 ? 10% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize.xfs_vn_setattr.notify_change
4.90 ? 14% -2.7 2.19 ? 9% perf-profile.calltrace.cycles-pp.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.29 ? 16% -2.0 1.32 ? 10% perf-profile.calltrace.cycles-pp.apparmor_cred_prepare.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
3.16 ? 9% -1.5 1.62 ? 10% perf-profile.calltrace.cycles-pp.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.14 ? 12% -1.3 0.82 ? 11% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize.xfs_vn_setattr
2.85 ? 10% -1.3 1.59 ? 8% perf-profile.calltrace.cycles-pp.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.46 ? 8% -1.2 1.30 ? 10% perf-profile.calltrace.cycles-pp.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.53 ? 10% -1.1 1.40 ? 8% perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.14 ? 8% -1.1 1.09 ? 11% perf-profile.calltrace.cycles-pp.apparmor_cred_free.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
2.17 ? 9% -1.0 1.13 ? 10% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.66 ? 13% -0.8 0.84 ? 6% perf-profile.calltrace.cycles-pp.xfs_trans_alloc.xfs_setattr_nonsize.xfs_vn_setattr.notify_change.chmod_common
1.43 ? 13% -0.7 0.71 ? 7% perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize.xfs_vn_setattr.notify_change
1.39 ? 14% -0.7 0.69 ? 7% perf-profile.calltrace.cycles-pp.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize.xfs_vn_setattr
1.50 ? 10% -0.7 0.81 ? 10% perf-profile.calltrace.cycles-pp.__kmalloc.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
1.46 ? 10% -0.7 0.80 ? 10% perf-profile.calltrace.cycles-pp.kmem_cache_alloc.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.92 ? 10% -0.7 0.26 ?100% perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.55 ? 10% -0.6 0.91 ? 8% perf-profile.calltrace.cycles-pp.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.52 ? 10% -0.6 0.89 ? 8% perf-profile.calltrace.cycles-pp.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.38 ? 9% -0.6 0.75 ? 9% perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.do_faccessat.do_syscall_64
0.96 ? 12% -0.6 0.37 ? 70% perf-profile.calltrace.cycles-pp.kfree.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
0.89 ? 9% -0.5 0.36 ? 70% perf-profile.calltrace.cycles-pp.kmem_cache_free.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.18 ? 10% -0.5 0.67 ? 9% perf-profile.calltrace.cycles-pp.strncpy_from_user.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64
0.00 +1.5 1.52 ? 17% perf-profile.calltrace.cycles-pp.override_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.47 ? 38% +2.4 3.89 ? 57% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +15.2 15.20 ? 12% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat
0.00 +15.6 15.63 ? 12% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat
0.00 +16.8 16.78 ? 12% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat.do_syscall_64
0.00 +17.1 17.12 ? 12% perf-profile.calltrace.cycles-pp.get_ucounts.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +17.2 17.23 ? 12% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64
0.00 +17.6 17.57 ? 12% perf-profile.calltrace.cycles-pp.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.77 ? 6% -5.0 0.73 ? 10% perf-profile.children.cycles-pp.free_uid
5.74 ? 6% -5.0 0.72 ? 9% perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
5.71 ? 6% -5.0 0.70 ? 10% perf-profile.children.cycles-pp.refcount_dec_not_one
8.88 ? 11% -4.7 4.16 ? 8% perf-profile.children.cycles-pp.__x64_sys_fchmod
8.78 ? 11% -4.7 4.10 ? 8% perf-profile.children.cycles-pp.chmod_common
8.05 ? 11% -4.3 3.74 ? 8% perf-profile.children.cycles-pp.notify_change
7.85 ? 11% -4.2 3.60 ? 8% perf-profile.children.cycles-pp.xfs_vn_setattr
7.78 ? 11% -4.2 3.56 ? 8% perf-profile.children.cycles-pp.xfs_setattr_nonsize
5.74 ? 11% -4.2 1.57 ? 13% perf-profile.children.cycles-pp.key_put
4.83 ? 13% -2.8 1.99 ? 10% perf-profile.children.cycles-pp.__xfs_trans_commit
4.71 ? 13% -2.8 1.92 ? 10% perf-profile.children.cycles-pp.xfs_log_commit_cil
4.90 ? 14% -2.7 2.19 ? 10% perf-profile.children.cycles-pp.security_prepare_creds
3.28 ? 48% -2.6 0.71 ?113% perf-profile.children.cycles-pp.start_kernel
3.29 ? 16% -2.0 1.32 ? 10% perf-profile.children.cycles-pp.apparmor_cred_prepare
3.79 ? 10% -1.7 2.08 ? 8% perf-profile.children.cycles-pp.filename_lookup
3.39 ? 10% -1.5 1.85 ? 9% perf-profile.children.cycles-pp.path_lookupat
3.16 ? 9% -1.5 1.62 ? 10% perf-profile.children.cycles-pp.security_cred_free
2.15 ? 12% -1.3 0.83 ? 10% perf-profile.children.cycles-pp.xlog_cil_insert_items
2.47 ? 8% -1.2 1.30 ? 10% perf-profile.children.cycles-pp.__do_sys_newfstatat
2.16 ? 9% -1.1 1.09 ? 11% perf-profile.children.cycles-pp.apparmor_cred_free
2.17 ? 10% -1.0 1.14 ? 9% perf-profile.children.cycles-pp.vfs_statx
1.62 ? 13% -1.0 0.60 ? 13% perf-profile.children.cycles-pp._raw_spin_lock
2.11 ? 10% -0.9 1.22 ? 10% perf-profile.children.cycles-pp.kmem_cache_alloc
1.87 ? 9% -0.9 1.01 ? 9% perf-profile.children.cycles-pp.link_path_walk
2.02 ? 10% -0.8 1.17 ? 8% perf-profile.children.cycles-pp.user_path_at_empty
1.99 ? 10% -0.8 1.15 ? 7% perf-profile.children.cycles-pp.getname_flags
1.67 ? 13% -0.8 0.84 ? 6% perf-profile.children.cycles-pp.xfs_trans_alloc
1.43 ? 13% -0.7 0.71 ? 7% perf-profile.children.cycles-pp.xfs_trans_reserve
1.40 ? 14% -0.7 0.69 ? 6% perf-profile.children.cycles-pp.xfs_log_reserve
1.54 ? 10% -0.7 0.84 ? 10% perf-profile.children.cycles-pp.__kmalloc
1.52 ? 10% -0.7 0.85 ? 7% perf-profile.children.cycles-pp.strncpy_from_user
1.06 ? 18% -0.6 0.44 ? 12% perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
1.21 ? 9% -0.5 0.69 ? 9% perf-profile.children.cycles-pp.kmem_cache_free
0.94 ? 9% -0.5 0.43 ? 8% perf-profile.children.cycles-pp.common_perm_cond
0.89 ? 10% -0.5 0.39 ? 8% perf-profile.children.cycles-pp.common_perm
0.97 ? 12% -0.4 0.54 ? 11% perf-profile.children.cycles-pp.kfree
0.80 ? 8% -0.4 0.40 ? 10% perf-profile.children.cycles-pp.inode_permission
0.86 ? 8% -0.4 0.48 ? 10% perf-profile.children.cycles-pp.walk_component
0.56 ? 22% -0.3 0.21 ? 10% perf-profile.children.cycles-pp.xlog_grant_add_space
0.61 ? 8% -0.3 0.29 ? 11% perf-profile.children.cycles-pp.generic_permission
0.52 ? 26% -0.3 0.19 ? 48% perf-profile.children.cycles-pp.xfs_trans_ijoin
0.67 ? 9% -0.3 0.37 ? 7% perf-profile.children.cycles-pp.__check_object_size
0.54 ? 10% -0.3 0.25 ? 11% perf-profile.children.cycles-pp.vfs_getattr
0.64 ? 11% -0.3 0.34 ? 12% perf-profile.children.cycles-pp.complete_walk
0.54 ? 9% -0.3 0.24 ? 10% perf-profile.children.cycles-pp.security_inode_getattr
0.61 ? 12% -0.3 0.33 ? 7% perf-profile.children.cycles-pp.down_read
0.59 ? 12% -0.3 0.33 ? 11% perf-profile.children.cycles-pp.unlazy_walk
0.48 ? 9% -0.3 0.22 ? 9% perf-profile.children.cycles-pp.security_path_chmod
0.56 ? 12% -0.2 0.31 ? 7% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.45 ? 14% -0.2 0.23 ? 9% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.45 ? 12% -0.2 0.23 ? 12% perf-profile.children.cycles-pp.obj_cgroup_charge
0.47 ? 7% -0.2 0.26 ? 8% perf-profile.children.cycles-pp.lookup_fast
0.46 ? 10% -0.2 0.26 ? 13% perf-profile.children.cycles-pp.__legitimize_path
0.44 ? 9% -0.2 0.24 ? 8% perf-profile.children.cycles-pp.xlog_grant_push_ail
0.43 ? 9% -0.2 0.24 ? 8% perf-profile.children.cycles-pp.xlog_grant_push_threshold
0.43 ? 9% -0.2 0.24 ? 8% perf-profile.children.cycles-pp.xlog_space_left
0.33 ? 23% -0.2 0.14 ? 13% perf-profile.children.cycles-pp.revert_creds
0.42 ? 7% -0.2 0.24 ? 11% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.29 ? 11% -0.2 0.11 ? 13% perf-profile.children.cycles-pp.capable_wrt_inode_uidgid
0.41 ? 11% -0.2 0.24 ? 8% perf-profile.children.cycles-pp.___might_sleep
0.28 ? 11% -0.2 0.11 ? 10% perf-profile.children.cycles-pp.ns_capable_common
0.28 ? 11% -0.2 0.10 ? 10% perf-profile.children.cycles-pp.security_capable
0.42 ? 13% -0.2 0.25 ? 16% perf-profile.children.cycles-pp.refill_obj_stock
0.26 ? 12% -0.2 0.09 ? 11% perf-profile.children.cycles-pp.apparmor_capable
0.36 ? 10% -0.2 0.20 ? 11% perf-profile.children.cycles-pp.path_put
0.36 ? 9% -0.2 0.20 ? 10% perf-profile.children.cycles-pp.dput
0.34 ? 12% -0.2 0.19 ? 10% perf-profile.children.cycles-pp.__entry_text_start
0.31 ? 11% -0.1 0.17 ? 7% perf-profile.children.cycles-pp.path_init
0.33 ? 7% -0.1 0.18 ? 10% perf-profile.children.cycles-pp.__check_heap_object
0.31 ? 7% -0.1 0.17 ? 7% perf-profile.children.cycles-pp.__d_lookup_rcu
0.28 ? 6% -0.1 0.16 ? 18% perf-profile.children.cycles-pp.xfs_inode_item_format
0.27 ? 10% -0.1 0.16 ? 11% perf-profile.children.cycles-pp.__might_sleep
0.24 ? 13% -0.1 0.13 ? 8% perf-profile.children.cycles-pp.__legitimize_mnt
0.23 ? 11% -0.1 0.11 ? 12% perf-profile.children.cycles-pp._cond_resched
0.27 ? 14% -0.1 0.16 ? 14% perf-profile.children.cycles-pp.cp_new_stat
0.17 ? 20% -0.1 0.06 ? 48% perf-profile.children.cycles-pp.up_read
0.23 ? 17% -0.1 0.13 ? 11% perf-profile.children.cycles-pp.memset_erms
0.20 ? 9% -0.1 0.10 ? 15% perf-profile.children.cycles-pp.__mod_memcg_state
0.27 ? 15% -0.1 0.17 ? 11% perf-profile.children.cycles-pp.xlog_ticket_alloc
0.20 ? 13% -0.1 0.11 ? 14% perf-profile.children.cycles-pp.__list_del_entry_valid
0.12 ? 12% -0.1 0.03 ?101% perf-profile.children.cycles-pp.__list_add_valid
0.19 ? 14% -0.1 0.11 ? 14% perf-profile.children.cycles-pp.step_into
0.17 ? 9% -0.1 0.09 ? 18% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.17 ? 14% -0.1 0.09 ? 18% perf-profile.children.cycles-pp.__might_fault
0.10 ? 10% -0.1 0.03 ?100% perf-profile.children.cycles-pp.security_inode_permission
0.18 ? 10% -0.1 0.10 ? 16% perf-profile.children.cycles-pp.lockref_get_not_dead
0.17 ? 8% -0.1 0.10 ? 6% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.15 ? 12% -0.1 0.09 ? 12% perf-profile.children.cycles-pp._copy_to_user
0.13 ? 6% -0.1 0.07 ? 15% perf-profile.children.cycles-pp.lockref_put_return
0.11 ? 13% -0.1 0.05 ? 47% perf-profile.children.cycles-pp.rcu_all_qs
0.13 ? 8% -0.1 0.07 ? 19% perf-profile.children.cycles-pp.down_write
0.09 ? 18% -0.1 0.04 ? 72% perf-profile.children.cycles-pp.mnt_want_write
0.11 ? 9% -0.0 0.06 ? 15% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.09 ? 14% -0.0 0.04 ? 72% perf-profile.children.cycles-pp.up_write
0.12 ? 11% -0.0 0.07 ? 11% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.09 ? 18% -0.0 0.05 ? 45% perf-profile.children.cycles-pp.__fget_light
0.10 ? 14% -0.0 0.06 ? 21% perf-profile.children.cycles-pp.__put_cred
0.08 ? 9% -0.0 0.04 ? 72% perf-profile.children.cycles-pp.terminate_walk
0.07 ? 16% -0.0 0.03 ? 70% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
0.10 ? 12% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.__virt_addr_valid
0.05 ? 45% +0.3 0.40 ? 12% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.16 ? 11% +1.4 1.52 ? 17% perf-profile.children.cycles-pp.override_creds
1.48 ? 38% +2.4 3.91 ? 57% perf-profile.children.cycles-pp.menu_select
0.00 +17.1 17.13 ? 12% perf-profile.children.cycles-pp.get_ucounts
0.00 +17.6 17.57 ? 12% perf-profile.children.cycles-pp.put_ucounts
0.76 ? 16% +30.2 30.99 ? 12% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.06 ? 13% +34.0 34.08 ? 12% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
13.33 ? 8% -9.2 4.18 ? 10% perf-profile.self.cycles-pp.prepare_creds
5.68 ? 6% -5.0 0.70 ? 10% perf-profile.self.cycles-pp.refcount_dec_not_one
5.70 ? 11% -4.1 1.55 ? 13% perf-profile.self.cycles-pp.key_put
4.91 ? 6% -2.1 2.77 ? 11% perf-profile.self.cycles-pp.put_cred_rcu
3.27 ? 16% -2.0 1.31 ? 10% perf-profile.self.cycles-pp.apparmor_cred_prepare
2.15 ? 9% -1.1 1.08 ? 11% perf-profile.self.cycles-pp.apparmor_cred_free
0.99 ? 17% -0.6 0.42 ? 12% perf-profile.self.cycles-pp.xfs_log_ticket_ungrant
0.88 ? 10% -0.5 0.39 ? 8% perf-profile.self.cycles-pp.common_perm
0.99 ? 10% -0.5 0.54 ? 9% perf-profile.self.cycles-pp.link_path_walk
1.07 ? 9% -0.4 0.63 ? 11% perf-profile.self.cycles-pp.kmem_cache_alloc
0.86 ? 13% -0.4 0.44 ? 7% perf-profile.self.cycles-pp._raw_spin_lock
0.86 ? 8% -0.4 0.48 ? 7% perf-profile.self.cycles-pp.kmem_cache_free
0.55 ? 22% -0.3 0.21 ? 10% perf-profile.self.cycles-pp.xlog_grant_add_space
0.76 ? 9% -0.3 0.43 ? 10% perf-profile.self.cycles-pp.__kmalloc
0.50 ? 16% -0.3 0.17 ? 22% perf-profile.self.cycles-pp.xfs_log_commit_cil
0.65 ? 15% -0.3 0.35 ? 10% perf-profile.self.cycles-pp.kfree
0.69 ? 12% -0.3 0.40 ? 8% perf-profile.self.cycles-pp.strncpy_from_user
0.58 ? 11% -0.3 0.31 ? 7% perf-profile.self.cycles-pp.down_read
0.56 ? 12% -0.2 0.31 ? 8% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.47 ? 9% -0.2 0.24 ? 7% perf-profile.self.cycles-pp.xlog_cil_insert_items
0.43 ? 11% -0.2 0.22 ? 12% perf-profile.self.cycles-pp.obj_cgroup_charge
0.43 ? 9% -0.2 0.23 ? 7% perf-profile.self.cycles-pp.xlog_space_left
0.33 ? 23% -0.2 0.14 ? 13% perf-profile.self.cycles-pp.revert_creds
0.38 ? 15% -0.2 0.19 ? 9% perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.40 ? 12% -0.2 0.23 ? 9% perf-profile.self.cycles-pp.___might_sleep
0.26 ? 11% -0.2 0.09 ? 11% perf-profile.self.cycles-pp.apparmor_capable
0.41 ? 13% -0.2 0.25 ? 15% perf-profile.self.cycles-pp.refill_obj_stock
0.34 ? 12% -0.2 0.19 ? 10% perf-profile.self.cycles-pp.__entry_text_start
0.40 ? 6% -0.1 0.25 ? 14% perf-profile.self.cycles-pp.do_faccessat
0.22 ? 31% -0.1 0.07 ? 60% perf-profile.self.cycles-pp.xfs_trans_ijoin
0.32 ? 7% -0.1 0.17 ? 10% perf-profile.self.cycles-pp.__check_heap_object
0.31 ? 6% -0.1 0.17 ? 9% perf-profile.self.cycles-pp.__d_lookup_rcu
0.28 ? 12% -0.1 0.15 ? 9% perf-profile.self.cycles-pp.path_init
0.28 ? 11% -0.1 0.16 ? 13% perf-profile.self.cycles-pp.generic_permission
0.24 ? 13% -0.1 0.13 ? 6% perf-profile.self.cycles-pp.__check_object_size
0.17 ? 20% -0.1 0.06 ? 49% perf-profile.self.cycles-pp.up_read
0.23 ? 13% -0.1 0.13 ? 8% perf-profile.self.cycles-pp.__legitimize_mnt
0.25 ? 10% -0.1 0.15 ? 12% perf-profile.self.cycles-pp.__might_sleep
0.22 ? 18% -0.1 0.12 ? 10% perf-profile.self.cycles-pp.memset_erms
0.22 ? 8% -0.1 0.12 ? 20% perf-profile.self.cycles-pp.xfs_inode_item_format
0.20 ? 9% -0.1 0.10 ? 15% perf-profile.self.cycles-pp.__mod_memcg_state
0.22 ? 9% -0.1 0.12 ? 14% perf-profile.self.cycles-pp.inode_permission
0.20 ? 8% -0.1 0.11 ? 19% perf-profile.self.cycles-pp.walk_component
0.21 ? 10% -0.1 0.11 ? 9% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.20 ? 14% -0.1 0.10 ? 14% perf-profile.self.cycles-pp.__list_del_entry_valid
0.18 ? 12% -0.1 0.10 ? 14% perf-profile.self.cycles-pp.xfs_trans_log_inode
0.11 ? 11% -0.1 0.03 ?101% perf-profile.self.cycles-pp.__list_add_valid
0.18 ? 16% -0.1 0.10 ? 15% perf-profile.self.cycles-pp.step_into
0.20 ? 7% -0.1 0.13 ? 13% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.17 ? 10% -0.1 0.10 ? 18% perf-profile.self.cycles-pp.lockref_get_not_dead
0.10 ? 7% -0.1 0.03 ?100% perf-profile.self.cycles-pp.security_inode_permission
0.15 ? 10% -0.1 0.08 ? 13% perf-profile.self.cycles-pp.lookup_fast
0.14 ? 11% -0.1 0.07 ? 10% perf-profile.self.cycles-pp.getname_flags
0.14 ? 17% -0.1 0.08 ? 12% perf-profile.self.cycles-pp.filename_lookup
0.12 ? 7% -0.1 0.07 ? 16% perf-profile.self.cycles-pp.lockref_put_return
0.08 ? 16% -0.1 0.03 ?102% perf-profile.self.cycles-pp.up_write
0.08 ? 23% -0.1 0.03 ? 99% perf-profile.self.cycles-pp.unlazy_walk
0.11 ? 9% -0.0 0.06 ? 15% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.11 ? 13% -0.0 0.06 ? 15% perf-profile.self.cycles-pp._cond_resched
0.12 ? 7% -0.0 0.07 ? 15% perf-profile.self.cycles-pp.rcu_read_unlock_strict
0.11 ? 9% -0.0 0.07 ? 13% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.10 ? 6% -0.0 0.06 ? 6% perf-profile.self.cycles-pp.path_lookupat
0.09 ? 17% -0.0 0.05 ? 45% perf-profile.self.cycles-pp.__fget_light
0.12 ? 12% -0.0 0.07 ? 16% perf-profile.self.cycles-pp.notify_change
0.07 ? 11% -0.0 0.03 ? 99% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.09 ? 14% -0.0 0.06 ? 8% perf-profile.self.cycles-pp.__virt_addr_valid
0.09 ? 13% -0.0 0.06 ? 16% perf-profile.self.cycles-pp.__put_cred
0.00 +0.2 0.18 ? 15% perf-profile.self.cycles-pp.get_ucounts
0.00 +0.2 0.18 ? 11% perf-profile.self.cycles-pp.put_ucounts
0.05 ? 45% +0.2 0.24 ? 17% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.67 ? 24% +1.3 1.94 ? 39% perf-profile.self.cycles-pp.cpuidle_enter_state
0.15 ? 10% +1.4 1.51 ? 17% perf-profile.self.cycles-pp.override_creds
0.88 ? 42% +2.4 3.25 ? 59% perf-profile.self.cycles-pp.menu_select
0.06 ? 15% +3.2 3.25 ? 9% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.75 ? 16% +30.2 30.98 ? 12% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



***************************************************************************************************
lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
filesystem/gcc-9/performance/1HDD/f2fs/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/access/stress-ng/60s/0x5003006

commit:
a88f997967 ("Increase size of ucounts to atomic_long_t")
e1e57d56fe ("Add a reference to ucounts for each cred")

a88f9979677c6f55 e1e57d56fef0dd06daf3743f094
---------------- ---------------------------
%stddev %change %stddev
\ | \
2266573 -44.5% 1258748 ? 4% stress-ng.access.ops
37776 -44.5% 20979 ? 4% stress-ng.access.ops_per_sec
10.37 ? 2% -45.1% 5.70 ? 5% stress-ng.time.user_time
0.30 -0.1 0.21 ? 3% mpstat.cpu.all.usr%
304.83 -1.6% 299.99 pmeter.Average_Active_Power
116.83 ? 7% +31.1% 153.17 ? 4% interrupts.CPU37.NMI:Non-maskable_interrupts
116.83 ? 7% +31.1% 153.17 ? 4% interrupts.CPU37.PMI:Performance_monitoring_interrupts
86.50 ? 38% +1387.1% 1286 ?123% interrupts.CPU60.NMI:Non-maskable_interrupts
86.50 ? 38% +1387.1% 1286 ?123% interrupts.CPU60.PMI:Performance_monitoring_interrupts
93.17 ? 38% +62.3% 151.17 ? 2% interrupts.CPU62.NMI:Non-maskable_interrupts
93.17 ? 38% +62.3% 151.17 ? 2% interrupts.CPU62.PMI:Performance_monitoring_interrupts
95.00 ? 32% +1047.9% 1090 ?189% interrupts.CPU77.NMI:Non-maskable_interrupts
95.00 ? 32% +1047.9% 1090 ?189% interrupts.CPU77.PMI:Performance_monitoring_interrupts
113.67 ? 9% +33.1% 151.33 ? 4% interrupts.CPU85.NMI:Non-maskable_interrupts
113.67 ? 9% +33.1% 151.33 ? 4% interrupts.CPU85.PMI:Performance_monitoring_interrupts
104.17 ? 31% +897.1% 1038 ?188% interrupts.CPU88.NMI:Non-maskable_interrupts
104.17 ? 31% +897.1% 1038 ?188% interrupts.CPU88.PMI:Performance_monitoring_interrupts
107.17 ? 33% +45.9% 156.33 ? 4% interrupts.CPU90.NMI:Non-maskable_interrupts
107.17 ? 33% +45.9% 156.33 ? 4% interrupts.CPU90.PMI:Performance_monitoring_interrupts
103.17 ? 35% +1114.1% 1252 ?195% interrupts.CPU91.NMI:Non-maskable_interrupts
103.17 ? 35% +1114.1% 1252 ?195% interrupts.CPU91.PMI:Performance_monitoring_interrupts
103.33 ? 33% +650.2% 775.17 ?133% interrupts.CPU92.NMI:Non-maskable_interrupts
103.33 ? 33% +650.2% 775.17 ?133% interrupts.CPU92.PMI:Performance_monitoring_interrupts
109.83 ? 23% +567.4% 733.00 ?176% interrupts.CPU93.NMI:Non-maskable_interrupts
109.83 ? 23% +567.4% 733.00 ?176% interrupts.CPU93.PMI:Performance_monitoring_interrupts
112.83 ? 21% +656.3% 853.33 ?182% interrupts.CPU94.NMI:Non-maskable_interrupts
112.83 ? 21% +656.3% 853.33 ?182% interrupts.CPU94.PMI:Performance_monitoring_interrupts
8.22 ? 5% +57.2% 12.91 perf-stat.i.MPKI
2.751e+09 -31.3% 1.89e+09 ? 3% perf-stat.i.branch-instructions
0.85 +0.1 0.95 perf-stat.i.branch-miss-rate%
22385388 -19.2% 18095507 ? 2% perf-stat.i.branch-misses
1.95 ? 2% +56.0% 3.04 ? 3% perf-stat.i.cpi
2.79e+10 +3.2% 2.88e+10 perf-stat.i.cpu-cycles
3.898e+09 ? 2% -34.1% 2.569e+09 ? 4% perf-stat.i.dTLB-loads
0.00 ? 12% +0.0 0.00 ? 18% perf-stat.i.dTLB-store-miss-rate%
2.176e+09 -38.4% 1.34e+09 ? 4% perf-stat.i.dTLB-stores
83.86 -7.8 76.11 perf-stat.i.iTLB-load-miss-rate%
12992520 ? 4% -43.6% 7326831 ? 3% perf-stat.i.iTLB-load-misses
1.427e+10 -33.3% 9.513e+09 ? 3% perf-stat.i.instructions
1195 ? 3% +14.8% 1372 perf-stat.i.instructions-per-iTLB-miss
0.53 -34.2% 0.35 ? 3% perf-stat.i.ipc
0.29 +3.2% 0.30 perf-stat.i.metric.GHz
93.35 -33.7% 61.88 ? 4% perf-stat.i.metric.M/sec
89.51 ? 2% +6.9 96.40 perf-stat.i.node-load-miss-rate%
2600729 ? 3% +135.6% 6126728 ? 7% perf-stat.i.node-load-misses
292982 ? 19% -38.5% 180109 ? 16% perf-stat.i.node-loads
13416978 ? 7% -27.9% 9679639 ? 9% perf-stat.i.node-store-misses
8.29 ? 5% +55.7% 12.90 perf-stat.overall.MPKI
0.81 +0.1 0.96 perf-stat.overall.branch-miss-rate%
1.96 +54.9% 3.03 ? 3% perf-stat.overall.cpi
0.00 ? 20% +0.0 0.00 ? 18% perf-stat.overall.dTLB-load-miss-rate%
0.00 ? 10% +0.0 0.00 ? 19% perf-stat.overall.dTLB-store-miss-rate%
84.89 -8.2 76.69 perf-stat.overall.iTLB-load-miss-rate%
1099 ? 3% +18.1% 1298 perf-stat.overall.instructions-per-iTLB-miss
0.51 -35.4% 0.33 ? 3% perf-stat.overall.ipc
89.87 ? 2% +7.3 97.15 perf-stat.overall.node-load-miss-rate%
2.707e+09 -31.3% 1.86e+09 ? 3% perf-stat.ps.branch-instructions
22015761 -19.1% 17801305 ? 2% perf-stat.ps.branch-misses
2.745e+10 +3.2% 2.833e+10 perf-stat.ps.cpu-cycles
3.835e+09 ? 2% -34.1% 2.528e+09 ? 4% perf-stat.ps.dTLB-loads
2.141e+09 -38.4% 1.318e+09 ? 4% perf-stat.ps.dTLB-stores
12784377 ? 4% -43.6% 7209165 ? 3% perf-stat.ps.iTLB-load-misses
1.404e+10 -33.3% 9.36e+09 ? 3% perf-stat.ps.instructions
2559028 ? 3% +135.6% 6028475 ? 7% perf-stat.ps.node-load-misses
288289 ? 19% -38.5% 177214 ? 16% perf-stat.ps.node-loads
13202172 ? 7% -27.9% 9524529 ? 9% perf-stat.ps.node-store-misses
8.819e+11 -33.1% 5.9e+11 ? 3% perf-stat.total.instructions
7.41 ? 9% -6.7 0.75 ? 12% perf-profile.calltrace.cycles-pp.free_uid.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.36 ? 9% -6.6 0.73 ? 11% perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat.do_syscall_64
7.33 ? 9% -6.6 0.71 ? 11% perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat
6.89 ? 12% -5.2 1.72 ? 12% perf-profile.calltrace.cycles-pp.key_put.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.28 ? 14% -3.0 2.29 ? 5% perf-profile.calltrace.cycles-pp.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.52 ? 17% -2.2 1.34 ? 5% perf-profile.calltrace.cycles-pp.apparmor_cred_prepare.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
3.50 ? 9% -1.8 1.68 ? 5% perf-profile.calltrace.cycles-pp.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.20 ? 7% -1.5 1.70 ? 6% perf-profile.calltrace.cycles-pp.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.73 ? 8% -1.3 1.38 ? 7% perf-profile.calltrace.cycles-pp.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.40 ? 11% -1.3 1.07 ? 6% perf-profile.calltrace.cycles-pp.apparmor_cred_free.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
2.79 ? 7% -1.3 1.51 ? 6% perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.44 ? 10% -1.2 1.19 ? 11% perf-profile.calltrace.cycles-pp.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.34 ? 10% -1.2 1.13 ? 10% perf-profile.calltrace.cycles-pp.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.39 ? 9% -1.2 1.22 ? 7% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.76 ? 7% -0.8 0.97 ? 7% perf-profile.calltrace.cycles-pp.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.64 ? 8% -0.8 0.86 ? 9% perf-profile.calltrace.cycles-pp.kmem_cache_alloc.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.72 ? 7% -0.8 0.95 ? 8% perf-profile.calltrace.cycles-pp.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.53 ? 10% -0.8 0.76 ? 12% perf-profile.calltrace.cycles-pp.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.64 ? 10% -0.8 0.88 ? 6% perf-profile.calltrace.cycles-pp.__kmalloc.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
1.51 ? 6% -0.7 0.81 ? 5% perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.do_faccessat.do_syscall_64
1.35 ? 11% -0.7 0.67 ? 13% perf-profile.calltrace.cycles-pp.f2fs_setattr.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64
1.00 ? 7% -0.6 0.37 ? 70% perf-profile.calltrace.cycles-pp.kmem_cache_free.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.33 ? 7% -0.6 0.73 ? 9% perf-profile.calltrace.cycles-pp.strncpy_from_user.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64
1.05 ? 7% -0.6 0.49 ? 45% perf-profile.calltrace.cycles-pp.kfree.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
0.97 ? 9% -0.5 0.45 ? 45% perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.75 ? 29% +1.3 3.05 ? 8% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.00 +1.6 1.57 ? 11% perf-profile.calltrace.cycles-pp.override_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +17.0 17.03 ? 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat
0.00 +17.5 17.51 ? 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat
0.00 +18.7 18.69 ? 7% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat.do_syscall_64
0.00 +19.0 19.04 ? 7% perf-profile.calltrace.cycles-pp.get_ucounts.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +19.2 19.19 ? 8% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64
0.00 +19.5 19.54 ? 8% perf-profile.calltrace.cycles-pp.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.41 ? 9% -6.7 0.75 ? 11% perf-profile.children.cycles-pp.free_uid
7.37 ? 9% -6.6 0.73 ? 12% perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
7.34 ? 9% -6.6 0.72 ? 11% perf-profile.children.cycles-pp.refcount_dec_not_one
6.92 ? 12% -5.2 1.73 ? 12% perf-profile.children.cycles-pp.key_put
5.29 ? 14% -3.0 2.29 ? 5% perf-profile.children.cycles-pp.security_prepare_creds
3.53 ? 17% -2.2 1.34 ? 5% perf-profile.children.cycles-pp.apparmor_cred_prepare
4.19 ? 7% -1.9 2.24 ? 6% perf-profile.children.cycles-pp.filename_lookup
3.50 ? 9% -1.8 1.68 ? 5% perf-profile.children.cycles-pp.security_cred_free
3.69 ? 7% -1.7 1.99 ? 6% perf-profile.children.cycles-pp.path_lookupat
2.73 ? 8% -1.3 1.38 ? 7% perf-profile.children.cycles-pp.__do_sys_newfstatat
2.41 ? 11% -1.3 1.08 ? 5% perf-profile.children.cycles-pp.apparmor_cred_free
2.44 ? 10% -1.2 1.19 ? 11% perf-profile.children.cycles-pp.__x64_sys_fchmod
2.35 ? 10% -1.2 1.14 ? 10% perf-profile.children.cycles-pp.chmod_common
2.40 ? 9% -1.2 1.23 ? 7% perf-profile.children.cycles-pp.vfs_statx
2.28 ? 8% -1.0 1.26 ? 7% perf-profile.children.cycles-pp.user_path_at_empty
2.24 ? 8% -1.0 1.24 ? 8% perf-profile.children.cycles-pp.getname_flags
2.03 ? 7% -0.9 1.09 ? 6% perf-profile.children.cycles-pp.link_path_walk
2.04 ? 8% -0.9 1.10 ? 8% perf-profile.children.cycles-pp.kmem_cache_alloc
1.69 ? 10% -0.8 0.92 ? 6% perf-profile.children.cycles-pp.__kmalloc
1.54 ? 10% -0.8 0.76 ? 11% perf-profile.children.cycles-pp.notify_change
1.71 ? 8% -0.8 0.94 ? 8% perf-profile.children.cycles-pp.strncpy_from_user
1.35 ? 11% -0.7 0.67 ? 13% perf-profile.children.cycles-pp.f2fs_setattr
1.31 ? 8% -0.6 0.68 ? 7% perf-profile.children.cycles-pp.kmem_cache_free
1.09 ? 11% -0.6 0.46 ? 8% perf-profile.children.cycles-pp.common_perm_cond
1.02 ? 11% -0.6 0.42 ? 9% perf-profile.children.cycles-pp.common_perm
0.87 ? 6% -0.5 0.41 ? 10% perf-profile.children.cycles-pp.inode_permission
1.06 ? 7% -0.5 0.59 ? 8% perf-profile.children.cycles-pp.kfree
0.94 ? 10% -0.4 0.52 ? 7% perf-profile.children.cycles-pp.walk_component
0.65 ? 8% -0.4 0.29 ? 7% perf-profile.children.cycles-pp.generic_permission
0.63 ? 9% -0.4 0.27 ? 8% perf-profile.children.cycles-pp.vfs_getattr
0.62 ? 9% -0.4 0.27 ? 8% perf-profile.children.cycles-pp.security_inode_getattr
0.75 ? 6% -0.4 0.40 ? 10% perf-profile.children.cycles-pp.__check_object_size
0.55 ? 12% -0.3 0.23 ? 11% perf-profile.children.cycles-pp.security_path_chmod
0.68 ? 10% -0.3 0.36 ? 6% perf-profile.children.cycles-pp.complete_walk
0.65 ? 10% -0.3 0.34 ? 6% perf-profile.children.cycles-pp.unlazy_walk
0.62 ? 9% -0.3 0.33 ? 8% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.46 ? 11% -0.2 0.21 ? 15% perf-profile.children.cycles-pp.f2fs_mark_inode_dirty_sync
0.49 ? 9% -0.2 0.25 ? 8% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.47 ? 15% -0.2 0.24 ? 9% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.50 ? 10% -0.2 0.27 ? 9% perf-profile.children.cycles-pp.__legitimize_path
0.52 ? 11% -0.2 0.29 ? 8% perf-profile.children.cycles-pp.lookup_fast
0.49 ? 9% -0.2 0.27 ? 7% perf-profile.children.cycles-pp.obj_cgroup_charge
0.40 ? 11% -0.2 0.18 ? 15% perf-profile.children.cycles-pp.f2fs_inode_dirtied
0.43 ? 9% -0.2 0.22 ? 10% perf-profile.children.cycles-pp.path_put
0.47 ? 10% -0.2 0.26 ? 11% perf-profile.children.cycles-pp.refill_obj_stock
0.42 ? 9% -0.2 0.21 ? 8% perf-profile.children.cycles-pp.___might_sleep
0.32 ? 13% -0.2 0.11 ? 11% perf-profile.children.cycles-pp.capable_wrt_inode_uidgid
0.31 ? 13% -0.2 0.11 ? 13% perf-profile.children.cycles-pp.ns_capable_common
0.31 ? 13% -0.2 0.11 ? 13% perf-profile.children.cycles-pp.security_capable
0.42 ? 9% -0.2 0.22 ? 11% perf-profile.children.cycles-pp.dput
0.42 ? 11% -0.2 0.21 ? 8% perf-profile.children.cycles-pp._raw_spin_lock
0.29 ? 15% -0.2 0.09 ? 13% perf-profile.children.cycles-pp.apparmor_capable
0.38 ? 7% -0.2 0.20 ? 8% perf-profile.children.cycles-pp.__entry_text_start
0.31 ? 28% -0.2 0.14 ? 18% perf-profile.children.cycles-pp.revert_creds
0.32 ? 10% -0.2 0.14 ? 9% perf-profile.children.cycles-pp.cp_new_stat
0.36 ? 7% -0.2 0.19 ? 8% perf-profile.children.cycles-pp.__check_heap_object
0.35 ? 11% -0.2 0.20 ? 7% perf-profile.children.cycles-pp.__d_lookup_rcu
0.34 ? 9% -0.1 0.20 ? 10% perf-profile.children.cycles-pp.path_init
0.26 ? 16% -0.1 0.12 ? 24% perf-profile.children.cycles-pp.f2fs_balance_fs
0.26 ? 12% -0.1 0.13 ? 7% perf-profile.children.cycles-pp.__legitimize_mnt
0.23 ? 9% -0.1 0.11 ? 10% perf-profile.children.cycles-pp.__mod_memcg_state
0.27 ? 11% -0.1 0.15 ? 8% perf-profile.children.cycles-pp.__might_sleep
0.20 ? 13% -0.1 0.09 ? 7% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.21 ? 11% -0.1 0.11 ? 16% perf-profile.children.cycles-pp.step_into
0.19 ? 11% -0.1 0.10 ? 14% perf-profile.children.cycles-pp._cond_resched
0.18 ? 6% -0.1 0.09 ? 14% perf-profile.children.cycles-pp._copy_to_user
0.20 ? 12% -0.1 0.11 ? 13% perf-profile.children.cycles-pp.__might_fault
0.19 ? 8% -0.1 0.11 ? 10% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.16 ? 9% -0.1 0.08 ? 13% perf-profile.children.cycles-pp.lockref_put_return
0.19 ? 13% -0.1 0.11 ? 9% perf-profile.children.cycles-pp.lockref_get_not_dead
0.16 ? 18% -0.1 0.08 ? 9% perf-profile.children.cycles-pp.memset_erms
0.14 ? 7% -0.1 0.07 ? 10% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.09 ? 10% -0.1 0.03 ?100% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
0.09 ? 14% -0.1 0.03 ?100% perf-profile.children.cycles-pp.rcu_all_qs
0.09 ? 10% -0.1 0.03 ?101% perf-profile.children.cycles-pp.terminate_walk
0.12 ? 7% -0.1 0.06 ? 17% perf-profile.children.cycles-pp.__put_cred
0.09 ? 6% -0.1 0.04 ? 71% perf-profile.children.cycles-pp.mnt_want_write
0.10 ? 10% -0.0 0.06 ? 8% perf-profile.children.cycles-pp.security_inode_permission
0.12 ? 13% -0.0 0.07 ? 9% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.09 ? 12% -0.0 0.05 ? 46% perf-profile.children.cycles-pp.__fget_light
0.06 ? 9% +0.3 0.40 ? 9% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
1.76 ? 29% +1.3 3.08 ? 8% perf-profile.children.cycles-pp.menu_select
0.18 ? 8% +1.4 1.57 ? 10% perf-profile.children.cycles-pp.override_creds
0.00 +19.0 19.04 ? 7% perf-profile.children.cycles-pp.get_ucounts
0.00 +19.5 19.54 ? 8% perf-profile.children.cycles-pp.put_ucounts
0.07 ? 11% +34.5 34.55 ? 8% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.06 ? 14% +37.9 37.96 ? 8% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
16.68 ? 10% -12.2 4.45 ? 11% perf-profile.self.cycles-pp.prepare_creds
7.30 ? 9% -6.6 0.71 ? 11% perf-profile.self.cycles-pp.refcount_dec_not_one
6.87 ? 12% -5.2 1.71 ? 12% perf-profile.self.cycles-pp.key_put
6.25 ? 8% -3.3 2.90 ? 8% perf-profile.self.cycles-pp.put_cred_rcu
3.50 ? 17% -2.2 1.33 ? 5% perf-profile.self.cycles-pp.apparmor_cred_prepare
2.39 ? 11% -1.3 1.08 ? 5% perf-profile.self.cycles-pp.apparmor_cred_free
1.01 ? 11% -0.6 0.41 ? 8% perf-profile.self.cycles-pp.common_perm
1.08 ? 8% -0.5 0.57 ? 6% perf-profile.self.cycles-pp.link_path_walk
1.03 ? 8% -0.5 0.58 ? 7% perf-profile.self.cycles-pp.kmem_cache_alloc
0.92 ? 8% -0.5 0.46 ? 7% perf-profile.self.cycles-pp.kmem_cache_free
0.84 ? 8% -0.4 0.46 ? 10% perf-profile.self.cycles-pp.__kmalloc
0.78 ? 9% -0.3 0.45 ? 6% perf-profile.self.cycles-pp.strncpy_from_user
0.69 ? 7% -0.3 0.39 ? 8% perf-profile.self.cycles-pp.kfree
0.62 ? 9% -0.3 0.33 ? 8% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.58 ? 11% -0.3 0.30 ? 12% perf-profile.self.cycles-pp.f2fs_setattr
0.47 ? 10% -0.2 0.26 ? 7% perf-profile.self.cycles-pp.obj_cgroup_charge
0.41 ? 9% -0.2 0.21 ? 7% perf-profile.self.cycles-pp.___might_sleep
0.46 ? 11% -0.2 0.25 ? 10% perf-profile.self.cycles-pp.refill_obj_stock
0.29 ? 15% -0.2 0.09 ? 13% perf-profile.self.cycles-pp.apparmor_capable
0.45 ? 7% -0.2 0.26 ? 5% perf-profile.self.cycles-pp.do_faccessat
0.38 ? 7% -0.2 0.20 ? 8% perf-profile.self.cycles-pp.__entry_text_start
0.38 ? 14% -0.2 0.21 ? 10% perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.31 ? 29% -0.2 0.14 ? 18% perf-profile.self.cycles-pp.revert_creds
0.35 ? 7% -0.2 0.18 ? 8% perf-profile.self.cycles-pp.__check_heap_object
0.35 ? 12% -0.2 0.20 ? 8% perf-profile.self.cycles-pp.__d_lookup_rcu
0.35 ? 12% -0.1 0.20 ? 9% perf-profile.self.cycles-pp._raw_spin_lock
0.31 ? 8% -0.1 0.17 ? 10% perf-profile.self.cycles-pp.path_init
0.29 ? 6% -0.1 0.16 ? 8% perf-profile.self.cycles-pp.generic_permission
0.27 ? 6% -0.1 0.14 ? 16% perf-profile.self.cycles-pp.__check_object_size
0.25 ? 16% -0.1 0.12 ? 25% perf-profile.self.cycles-pp.f2fs_balance_fs
0.25 ? 10% -0.1 0.13 ? 8% perf-profile.self.cycles-pp.__legitimize_mnt
0.23 ? 9% -0.1 0.11 ? 10% perf-profile.self.cycles-pp.__mod_memcg_state
0.25 ? 10% -0.1 0.13 ? 9% perf-profile.self.cycles-pp.__might_sleep
0.24 ? 5% -0.1 0.13 ? 16% perf-profile.self.cycles-pp.inode_permission
0.22 ? 12% -0.1 0.12 ? 4% perf-profile.self.cycles-pp.walk_component
0.20 ? 12% -0.1 0.10 ? 18% perf-profile.self.cycles-pp.step_into
0.22 ? 4% -0.1 0.12 ? 9% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.23 ? 13% -0.1 0.13 ? 8% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.16 ? 11% -0.1 0.08 ? 12% perf-profile.self.cycles-pp.getname_flags
0.19 ? 13% -0.1 0.11 ? 9% perf-profile.self.cycles-pp.lockref_get_not_dead
0.16 ? 11% -0.1 0.08 ? 12% perf-profile.self.cycles-pp.lockref_put_return
0.15 ? 19% -0.1 0.08 ? 10% perf-profile.self.cycles-pp.memset_erms
0.13 ? 8% -0.1 0.06 ? 11% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.16 ? 12% -0.1 0.09 ? 6% perf-profile.self.cycles-pp.filename_lookup
0.12 ? 11% -0.1 0.06 ? 8% perf-profile.self.cycles-pp.path_lookupat
0.15 ? 11% -0.1 0.09 ? 18% perf-profile.self.cycles-pp.lookup_fast
0.15 ? 10% -0.1 0.09 ? 8% perf-profile.self.cycles-pp.rcu_read_unlock_strict
0.11 ? 9% -0.1 0.06 ? 11% perf-profile.self.cycles-pp.__put_cred
0.10 ? 10% -0.0 0.06 ? 8% perf-profile.self.cycles-pp.security_inode_permission
0.12 ? 13% -0.0 0.07 ? 11% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.09 ? 11% -0.0 0.04 ? 45% perf-profile.self.cycles-pp.notify_change
0.10 ? 17% -0.0 0.06 ? 14% perf-profile.self.cycles-pp.__virt_addr_valid
0.09 ? 12% -0.0 0.05 ? 46% perf-profile.self.cycles-pp.__fget_light
0.00 +0.2 0.16 ? 11% perf-profile.self.cycles-pp.put_ucounts
0.00 +0.2 0.17 ? 10% perf-profile.self.cycles-pp.get_ucounts
0.06 ? 11% +0.2 0.23 ? 8% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.84 ? 23% +0.8 1.69 ? 7% perf-profile.self.cycles-pp.cpuidle_enter_state
1.20 ? 31% +1.2 2.44 ? 8% perf-profile.self.cycles-pp.menu_select
0.17 ? 9% +1.4 1.56 ? 10% perf-profile.self.cycles-pp.override_creds
0.06 ? 14% +3.3 3.41 ? 7% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.07 ? 11% +34.5 34.55 ? 8% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang

---

Attachments:

(No filename) (91.75 kB)
config-5.11.0-rc7-00013-ge1e57d56fef0 (175.11 kB)
job-script (8.49 kB)
job.yaml (5.79 kB)
reproduce (551.00 B)
Download all attachments

2021-03-05 17:58:32

by Eric W. Biederman

[permalink] [raw]

Subject: Re: d28296d248: stress-ng.sigsegv.ops_per_sec -82.7% regression

---

Alexey Gladkov <[email protected]> writes:

> On Wed, Feb 24, 2021 at 12:50:21PM -0600, Eric W. Biederman wrote:
>> Alexey Gladkov <[email protected]> writes:
>>
>> > On Wed, Feb 24, 2021 at 10:54:17AM -0600, Eric W. Biederman wrote:
>> >> kernel test robot <[email protected]> writes:
>> >>
>> >> > Greeting,
>> >> >
>> >> > FYI, we noticed a -82.7% regression of stress-ng.sigsegv.ops_per_sec due to commit:
>> >> >
>> >> >
>> >> > commit: d28296d2484fa11e94dff65e93eb25802a443d47 ("[PATCH v7 5/7] Reimplement RLIMIT_SIGPENDING on top of ucounts")
>> >> > url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
>> >> > base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next
>> >> >
>> >> > in testcase: stress-ng
>> >> > on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
>> >> > with following parameters:
>> >> >
>> >> > nr_threads: 100%
>> >> > disk: 1HDD
>> >> > testtime: 60s
>> >> > class: interrupt
>> >> > test: sigsegv
>> >> > cpufreq_governor: performance
>> >> > ucode: 0x42e
>> >> >
>> >> >
>> >> > In addition to that, the commit also has significant impact on the
>> >> > following tests:
>> >>
>> >> Thank you. Now we have a sense of where we need to test the performance
>> >> of these changes carefully.
>> >
>> > One of the reasons for this is that I rolled back the patch that changed
>> > the ucounts.count type to atomic_t. Now get_ucounts() is forced to use a
>> > spin_lock to increase the reference count.
>>
>> Which given the hickups with getting a working version seems justified.
>>
>> Now we can add incremental patches on top to improve the performance.
>
> I'm not sure that get_ucounts() should be used in __sigqueue_alloc() [1].
> I tried removing it and running KASAN tests that were failing before. So
> far, I have not found any problems.
>
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/legion/linux.git/tree/kernel/signal.c?h=patchset/per-userspace-rlimit/v7.1&id=2d4a2e2be7db42c95acb98abfc2a9b370ddd0604#n428

Hmm. The code you posted still seems to include the get_ucounts.

I like the idea of not needing to increment and decrement the ucount
reference count every time a signal is sent, unfortunately there is a
problem. The way we have implemented setresuid allows different threads
in a threaded application to have different cred->user values.

That is actually an extension of what posix supports and pthreads will
keep the creds of a process in sync. Still I recall looking into this a
few years ago and there were a few applications that take advantage of
the linux behavior.

In principle I think it is possible to hold a ucount reference in
somewhere such as task->signal. In practice there are enough
complicating factors I don't immediately see how to implement that.

If the creds were stored in signal_struct instead of in task_struct
we could simply move the sigpending counts in set_user, when the uid
of a process changed.

With the current state I don't know how to pick which is the real user.

Eric