Hello,
When we profile cgroup events with perf stat, it's very annoying to
specify events and cgroups on the command line as it requires the
mapping between events and cgroups. (Note that perf record can use
cgroup sampling but it's not usable for perf stat).
I guess most cases we just want to use a same set of events (N) for
all cgroups (M), but we need to specify NxM events and NxM cgroups.
This is not good especially when profiling large number of cgroups:
say M=200.
So I added --multiply-cgroup option to make it easy for that case. It
will create NxM events from N events and M cgroups. One more upside
is that it can handle metrics too.
For example, the following example measures IPC metric for 3 cgroups
$ cat perf-multi-cgrp.sh
#!/bin/sh
METRIC=${1:-IPC}
CGROUP_DIR=/sys/fs/cgroup/perf_event
sudo mkdir $CGROUP_DIR/A $CGROUP_DIR/B $CGROUP_DIR/C
# add backgroupd workload for each cgroup
echo $$ | sudo tee $CGROUP_DIR/A/cgroup.procs > /dev/null
yes > /dev/null &
echo $$ | sudo tee $CGROUP_DIR/B/cgroup.procs > /dev/null
yes > /dev/null &
echo $$ | sudo tee $CGROUP_DIR/C/cgroup.procs > /dev/null
yes > /dev/null &
# run 'perf stat' in the root cgroup
echo $$ | sudo tee $CGROUP_DIR/cgroup.procs > /dev/null
perf stat -a -M $METRIC --multiply-cgroup -G A,B,C sleep 1
kill %1 %2 %3
sudo rmdir $CGROUP_DIR/A $CGROUP_DIR/B $CGROUP_DIR/C
$ ./perf-multi-cgrp.sh IPC
Performance counter stats for 'system wide':
11,284,850,010 inst_retired.any A # 2.71 IPC
4,157,915,982 cpu_clk_unhalted.thread A
11,342,188,640 inst_retired.any B # 2.72 IPC
4,173,014,732 cpu_clk_unhalted.thread B
11,135,863,604 inst_retired.any C # 2.67 IPC
4,171,375,184 cpu_clk_unhalted.thread C
1.011948803 seconds time elapsed
The code is available at 'perf/cgroup-multiply-v1' branch on
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
Thanks
Namhyung
Namhyung Kim (4):
perf evsel: Add evsel__clone() function
perf stat: Add --multiply-cgroup option
perf tools: Copy metric events properly when multiply cgroups
perf test: Add multiply cgroup event test
tools/perf/builtin-stat.c | 21 ++-
tools/perf/tests/Build | 1 +
tools/perf/tests/builtin-test.c | 4 +
tools/perf/tests/multiply-cgroup.c | 203 +++++++++++++++++++++++++++++
tools/perf/tests/tests.h | 1 +
tools/perf/util/cgroup.c | 106 ++++++++++++++-
tools/perf/util/cgroup.h | 4 +
tools/perf/util/evlist.c | 11 ++
tools/perf/util/evlist.h | 1 +
tools/perf/util/evsel.c | 57 ++++++++
tools/perf/util/evsel.h | 1 +
tools/perf/util/metricgroup.c | 77 +++++++++++
tools/perf/util/metricgroup.h | 6 +
tools/perf/util/stat.h | 1 +
14 files changed, 488 insertions(+), 6 deletions(-)
create mode 100644 tools/perf/tests/multiply-cgroup.c
--
2.28.0.526.ge36021eeef-goog
The --multiply-cgroup option is a syntax sugar to monitor large number
of cgroups easily. Current command line requires to list all the
events and cgroups even if users want to monitor same events for each
cgroup. This patch addresses that usage by copying given events for
each cgroup on user's behalf.
For instance, if they want to monitor 6 events for 200 cgroups each
they should write 1200 event names (with -e) AND 1200 cgroup names
(with -G) on the command line. But with this change, they can just
specify 6 events and 200 cgroups plus one more option.
A simpler example below: It wants to measure 3 events for 2 cgroups
('a' and 'b'). The result is that total 6 events are counted like
below.
$ ./perf stat -a -e cpu-clock,cycles,instructions --multiply-cgroup -G a,b sleep 1
Performance counter stats for 'system wide':
988.18 msec cpu-clock a # 0.987 CPUs utilized
3,153,761,702 cycles a # 3.200 GHz (100.00%)
8,067,769,847 instructions a # 2.57 insn per cycle (100.00%)
982.71 msec cpu-clock b # 0.982 CPUs utilized
3,136,093,298 cycles b # 3.182 GHz (99.99%)
8,109,619,327 instructions b # 2.58 insn per cycle (99.99%)
1.001228054 seconds time elapsed
Signed-off-by: Namhyung Kim <[email protected]>
---
tools/perf/builtin-stat.c | 20 +++++++++-
tools/perf/util/cgroup.c | 78 +++++++++++++++++++++++++++++++++++++++
tools/perf/util/cgroup.h | 2 +
tools/perf/util/stat.h | 1 +
4 files changed, 100 insertions(+), 1 deletion(-)
diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 483a28ef4ec4..6724d23ce2e7 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -1068,6 +1068,16 @@ static int parse_control_option(const struct option *opt,
return 0;
}
+static int parse_stat_cgroups(const struct option *opt,
+ const char *str, int unset)
+{
+ stat_config.cgroups = strdup(str);
+ if (!stat_config.cgroups)
+ return -1;
+
+ return parse_cgroups(opt, str, unset);
+}
+
static struct option stat_options[] = {
OPT_BOOLEAN('T', "transaction", &transaction_run,
"hardware transaction statistics"),
@@ -1111,7 +1121,9 @@ static struct option stat_options[] = {
OPT_STRING('x', "field-separator", &stat_config.csv_sep, "separator",
"print counts with custom separator"),
OPT_CALLBACK('G', "cgroup", &evsel_list, "name",
- "monitor event in cgroup name only", parse_cgroups),
+ "monitor event in cgroup name only", parse_stat_cgroups),
+ OPT_BOOLEAN(0, "multiply-cgroup", &multiply_cgroup,
+ "multiply the event list by cgroups"),
OPT_STRING('o', "output", &output_name, "file", "output file name"),
OPT_BOOLEAN(0, "append", &append_file, "append to the output file"),
OPT_INTEGER(0, "log-fd", &output_fd,
@@ -2248,6 +2260,11 @@ int cmd_stat(int argc, const char **argv)
if (add_default_attributes())
goto out;
+ if (multiply_cgroup && stat_config.cgroups) {
+ if (evlist__multiply_cgroup(evsel_list, stat_config.cgroups) < 0)
+ goto out;
+ }
+
target__validate(&target);
if ((stat_config.aggr_mode == AGGR_THREAD) && (target.system_wide))
@@ -2412,6 +2429,7 @@ int cmd_stat(int argc, const char **argv)
evlist__delete(evsel_list);
+ free(stat_config.cgroups);
metricgroup__rblist_exit(&stat_config.metric_events);
runtime_stat_delete(&stat_config);
diff --git a/tools/perf/util/cgroup.c b/tools/perf/util/cgroup.c
index 050dea9f1e88..4892f9496bc2 100644
--- a/tools/perf/util/cgroup.c
+++ b/tools/perf/util/cgroup.c
@@ -12,6 +12,7 @@
#include <api/fs/fs.h>
int nr_cgroups;
+bool multiply_cgroup;
static int open_cgroup(const char *name)
{
@@ -156,6 +157,10 @@ int parse_cgroups(const struct option *opt, const char *str,
return -1;
}
+ /* delay processing cgroups after it sees all events */
+ if (multiply_cgroup)
+ return 0;
+
for (;;) {
p = strchr(str, ',');
e = p ? p : eos;
@@ -193,6 +198,79 @@ int parse_cgroups(const struct option *opt, const char *str,
return 0;
}
+int evlist__multiply_cgroup(struct evlist *evlist, const char *str)
+{
+ struct evlist *orig_list, *tmp_list;
+ struct evsel *pos, *evsel, *leader;
+ struct cgroup *cgrp = NULL;
+ const char *p, *e, *eos = str + strlen(str);
+ int ret = -1;
+
+ if (evlist->core.nr_entries == 0) {
+ fprintf(stderr, "must define events before cgroups\n");
+ return -EINVAL;
+ }
+
+ orig_list = evlist__new();
+ tmp_list = evlist__new();
+ if (orig_list == NULL || tmp_list == NULL) {
+ fprintf(stderr, "memory allocation failed\n");
+ return -ENOMEM;
+ }
+
+ /* save original events and init evlist */
+ perf_evlist__splice_list_tail(orig_list, &evlist->core.entries);
+ evlist->core.nr_entries = 0;
+
+ for (;;) {
+ p = strchr(str, ',');
+ e = p ? p : eos;
+
+ /* allow empty cgroups, i.e., skip */
+ if (e - str) {
+ /* termination added */
+ char *name = strndup(str, e - str);
+ if (!name)
+ break;
+
+ cgrp = cgroup__new(name);
+ free(name);
+ if (cgrp == NULL)
+ break;
+ } else {
+ cgrp = NULL;
+ }
+
+ leader = NULL;
+ evlist__for_each_entry(orig_list, pos) {
+ evsel = evsel__clone(pos);
+ evsel->cgrp = cgroup__get(cgrp);
+
+ if (evsel__is_group_leader(pos))
+ leader = evsel;
+ evsel->leader = leader;
+
+ evlist__add(tmp_list, evsel);
+ }
+ /* cgroup__new() has a refcount, release it here */
+ cgroup__put(cgrp);
+ nr_cgroups++;
+
+ perf_evlist__splice_list_tail(evlist, &tmp_list->core.entries);
+ tmp_list->core.nr_entries = 0;
+
+ if (!p) {
+ ret = 0;
+ break;
+ }
+ str = p+1;
+ }
+ evlist__delete(orig_list);
+ evlist__delete(tmp_list);
+
+ return ret;
+}
+
static struct cgroup *__cgroup__findnew(struct rb_root *root, uint64_t id,
bool create, const char *path)
{
diff --git a/tools/perf/util/cgroup.h b/tools/perf/util/cgroup.h
index e98d5975fe55..9a842f243dfb 100644
--- a/tools/perf/util/cgroup.h
+++ b/tools/perf/util/cgroup.h
@@ -17,6 +17,7 @@ struct cgroup {
};
extern int nr_cgroups; /* number of explicit cgroups defined */
+extern bool multiply_cgroup; /* create events for each cgroup */
struct cgroup *cgroup__get(struct cgroup *cgroup);
void cgroup__put(struct cgroup *cgroup);
@@ -24,6 +25,7 @@ void cgroup__put(struct cgroup *cgroup);
struct evlist;
struct cgroup *evlist__findnew_cgroup(struct evlist *evlist, const char *name);
+int evlist__multiply_cgroup(struct evlist *evlist, const char *cgroups);
void evlist__set_default_cgroup(struct evlist *evlist, struct cgroup *cgroup);
diff --git a/tools/perf/util/stat.h b/tools/perf/util/stat.h
index f8778cffd941..22f381fe93de 100644
--- a/tools/perf/util/stat.h
+++ b/tools/perf/util/stat.h
@@ -135,6 +135,7 @@ struct perf_stat_config {
struct rblist metric_events;
int ctl_fd;
int ctl_fd_ack;
+ char *cgroups;
};
void perf_stat__set_big_num(int set);
--
2.28.0.526.ge36021eeef-goog
The evsel__clone() is to create an exactly same evsel from same
attributes. Note that metric events will be handled by later patch.
It will be used by perf stat to generate separate events for each
cgroup.
Signed-off-by: Namhyung Kim <[email protected]>
---
tools/perf/util/evsel.c | 57 +++++++++++++++++++++++++++++++++++++++++
tools/perf/util/evsel.h | 1 +
2 files changed, 58 insertions(+)
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index fd865002cbbd..4f50f9499973 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -331,6 +331,63 @@ struct evsel *evsel__new_cycles(bool precise)
goto out;
}
+/**
+ * evsel__clone - create a new evsel copied from @orig
+ * @orig: original evsel
+ *
+ * The assumption is that @orig is not configured nor opened yet.
+ * So we only care about the attributes that can be set while it's parsed.
+ */
+struct evsel *evsel__clone(struct evsel *orig)
+{
+ struct evsel *evsel;
+ struct evsel_config_term *pos, *tmp;
+
+ BUG_ON(orig->core.fd);
+
+ evsel = evsel__new(&orig->core.attr);
+ if (evsel == NULL)
+ return NULL;
+
+ *evsel = *orig;
+ evsel->evlist = NULL;
+ INIT_LIST_HEAD(&evsel->core.node);
+
+ evsel->core.cpus = perf_cpu_map__get(orig->core.cpus);
+ evsel->core.own_cpus = perf_cpu_map__get(orig->core.own_cpus);
+ evsel->core.threads = perf_thread_map__get(orig->core.threads);
+ if (orig->name)
+ evsel->name = strdup(orig->name);
+ if (orig->group_name)
+ evsel->group_name = strdup(orig->group_name);
+ if (orig->pmu_name)
+ evsel->pmu_name = strdup(orig->pmu_name);
+
+ INIT_LIST_HEAD(&evsel->config_terms);
+ list_for_each_entry(pos, &orig->config_terms, list) {
+ tmp = malloc(sizeof(*tmp));
+ if (tmp == NULL) {
+ evsel__delete(evsel);
+ evsel = NULL;
+ break;
+ }
+
+ *tmp = *pos;
+ if (tmp->free_str) {
+ tmp->val.str = strdup(pos->val.str);
+ if (tmp->val.str == NULL) {
+ evsel__delete(evsel);
+ evsel = NULL;
+ free(tmp);
+ break;
+ }
+ }
+ list_add_tail(&tmp->list, &evsel->config_terms);
+ }
+
+ return evsel;
+}
+
/*
* Returns pointer with encoded error via <linux/err.h> interface.
*/
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 35e3f6d66085..507c31d6a389 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -169,6 +169,7 @@ static inline struct evsel *evsel__new(struct perf_event_attr *attr)
return evsel__new_idx(attr, 0);
}
+struct evsel *evsel__clone(struct evsel *orig);
struct evsel *evsel__newtp_idx(const char *sys, const char *name, int idx);
/*
--
2.28.0.526.ge36021eeef-goog
It'll multiply given events for cgroups A, B and C.
$ ./perf test -v 68
68: Event multiplication for cgroups :
--- start ---
test child forked, pid 983140
metric expr 1 / IPC for CPI
metric expr instructions / cycles for IPC
found event instructions
found event cycles
adding {instructions,cycles}:W
copying metric event for cgroup 'A': instructions (idx=0)
copying metric event for cgroup 'B': instructions (idx=0)
copying metric event for cgroup 'C': instructions (idx=0)
test child finished with 0
---- end ----
Event multiplication for cgroups: Ok
Cc: John Garry <[email protected]>
Signed-off-by: Namhyung Kim <[email protected]>
---
tools/perf/builtin-stat.c | 2 +-
tools/perf/tests/Build | 1 +
tools/perf/tests/builtin-test.c | 4 +
tools/perf/tests/multiply-cgroup.c | 203 +++++++++++++++++++++++++++++
tools/perf/tests/tests.h | 1 +
tools/perf/util/cgroup.c | 19 ++-
tools/perf/util/cgroup.h | 2 +-
7 files changed, 223 insertions(+), 9 deletions(-)
create mode 100644 tools/perf/tests/multiply-cgroup.c
diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 55a7dc175cdf..c231972f3581 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -2262,7 +2262,7 @@ int cmd_stat(int argc, const char **argv)
if (multiply_cgroup && stat_config.cgroups) {
if (evlist__multiply_cgroup(evsel_list, stat_config.cgroups,
- &stat_config.metric_events) < 0)
+ &stat_config.metric_events, true) < 0)
goto out;
}
diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index 84352fc49a20..e685a3441853 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -60,6 +60,7 @@ perf-y += api-io.o
perf-y += demangle-java-test.o
perf-y += pfm.o
perf-y += parse-metric.o
+perf-y += multiply-cgroup.o
$(OUTPUT)tests/llvm-src-base.c: tests/bpf-script-example.c tests/Build
$(call rule_mkdir)
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index d328caaba45d..de48b9d28230 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -341,6 +341,10 @@ static struct test generic_tests[] = {
.desc = "Parse and process metrics",
.func = test__parse_metric,
},
+ {
+ .desc = "Event multiplication for cgroups",
+ .func = test__multiply_cgroup_events,
+ },
{
.func = NULL,
},
diff --git a/tools/perf/tests/multiply-cgroup.c b/tools/perf/tests/multiply-cgroup.c
new file mode 100644
index 000000000000..89af1644cd6a
--- /dev/null
+++ b/tools/perf/tests/multiply-cgroup.c
@@ -0,0 +1,203 @@
+// SPDX-License-Identifier: GPL-2.0
+#include "tests.h"
+#include "debug.h"
+#include "evlist.h"
+#include "cgroup.h"
+#include "rblist.h"
+#include "metricgroup.h"
+#include "parse-events.h"
+#include "pmu-events/pmu-events.h"
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+static int test_multiply_events(struct evlist *evlist,
+ struct rblist *metric_events)
+{
+ int i, ret = TEST_FAIL;
+ int nr_events;
+ bool was_group_event;
+ int nr_members; /* for the first evsel only */
+ const char cgrp_str[] = "A,B,C";
+ const char *cgrp_name[] = { "A", "B", "C" };
+ int nr_cgrps = ARRAY_SIZE(cgrp_name);
+ char **ev_name;
+ struct evsel *evsel;
+
+ TEST_ASSERT_VAL("evlist is empty", !perf_evlist__empty(evlist));
+
+ nr_events = evlist->core.nr_entries;
+ ev_name = calloc(nr_events, sizeof(*ev_name));
+ if (ev_name == NULL) {
+ pr_debug("memory allocation failure\n");
+ return TEST_FAIL;
+ }
+ i = 0;
+ evlist__for_each_entry(evlist, evsel) {
+ ev_name[i] = strdup(evsel->name);
+ if (ev_name[i] == NULL) {
+ pr_debug("memory allocation failure\n");
+ goto out;
+ }
+ i++;
+ }
+ /* remember grouping info */
+ was_group_event = evsel__is_group_event(evlist__first(evlist));
+ nr_members = evlist__first(evlist)->core.nr_members;
+
+ ret = evlist__multiply_cgroup(evlist, cgrp_str, metric_events, false);
+ if (ret < 0) {
+ pr_debug("failed to multiply cgroup\n");
+ goto out;
+ }
+
+ ret = TEST_FAIL;
+ if (evlist->core.nr_entries != nr_events * nr_cgrps) {
+ pr_debug("event count doesn't match\n");
+ goto out;
+ }
+
+ i = 0;
+ evlist__for_each_entry(evlist, evsel) {
+ if (strcmp(evsel->name, ev_name[i % nr_events])) {
+ pr_debug("event name doesn't match:\n");
+ pr_debug(" evsel[%d]: %s\n expected: %s\n",
+ i, evsel->name, ev_name[i % nr_events]);
+ goto out;
+ }
+ if (strcmp(evsel->cgrp->name, cgrp_name[i / nr_events])) {
+ pr_debug("cgroup name doesn't match:\n");
+ pr_debug(" evsel[%d]: %s\n expected: %s\n",
+ i, evsel->cgrp->name, cgrp_name[i / nr_events]);
+ goto out;
+ }
+
+ if ((i % nr_events) == 0) {
+ if (evsel__is_group_event(evsel) != was_group_event) {
+ pr_debug("event group doesn't match: got %s, expect %s\n",
+ evsel__is_group_event(evsel) ? "true" : "false",
+ was_group_event ? "true" : "false");
+ goto out;
+ }
+ if (evsel->core.nr_members != nr_members) {
+ pr_debug("event group member doesn't match: %d vs %d\n",
+ evsel->core.nr_members, nr_members);
+ goto out;
+ }
+ }
+ i++;
+ }
+ ret = TEST_OK;
+
+out: for (i = 0; i < nr_events; i++)
+ free(ev_name[i]);
+ free(ev_name);
+ return ret;
+}
+
+static int multiply_default_events(void)
+{
+ int ret;
+ struct evlist *evlist;
+ struct rblist metric_events;
+
+ evlist = perf_evlist__new_default();
+ TEST_ASSERT_VAL("failed to get evlist", evlist);
+
+ rblist__init(&metric_events);
+ ret = test_multiply_events(evlist, &metric_events);
+ evlist__delete(evlist);
+ return ret;
+}
+
+static int multiply_group_events(void)
+{
+ int ret;
+ struct evlist *evlist;
+ struct rblist metric_events;
+ struct parse_events_error err;
+ const char event_str[] = "{cycles,instructions}";
+
+ symbol_conf.event_group = true;
+
+ evlist = evlist__new();
+ TEST_ASSERT_VAL("failed to get evlist", evlist);
+
+ ret = parse_events(evlist, event_str, &err);
+ if (ret < 0) {
+ pr_debug("failed to parse event '%s', err %d, str '%s'\n",
+ event_str, ret, err.str);
+ parse_events_print_error(&err, event_str);
+ evlist__delete(evlist);
+ return ret;
+ }
+
+ rblist__init(&metric_events);
+ ret = test_multiply_events(evlist, &metric_events);
+ evlist__delete(evlist);
+ return ret;
+}
+
+static int multiply_metric_events(void)
+{
+ int ret;
+ struct evlist *evlist;
+ struct rblist metric_events;
+ const char metric_str[] = "CPI";
+
+ struct pmu_event pme_test[] = {
+ {
+ .metric_expr = "instructions / cycles",
+ .metric_name = "IPC",
+ },
+ {
+ .metric_expr = "1 / IPC",
+ .metric_name = "CPI",
+ },
+ {
+ .metric_expr = NULL,
+ .metric_name = NULL,
+ },
+ };
+ struct pmu_events_map ev_map = {
+ .cpuid = "test",
+ .version = "1",
+ .type = "core",
+ .table = pme_test,
+ };
+
+ evlist = evlist__new();
+ TEST_ASSERT_VAL("failed to get evlist", evlist);
+
+ rblist__init(&metric_events);
+ ret = metricgroup__parse_groups_test(evlist, &ev_map, metric_str,
+ false, false, &metric_events);
+ if (ret < 0) {
+ pr_debug("failed to parse '%s' metric\n", metric_str);
+ goto out;
+ }
+
+ ret = test_multiply_events(evlist, &metric_events);
+
+out:
+ metricgroup__rblist_exit(&metric_events);
+ evlist__delete(evlist);
+ return ret;
+}
+
+int test__multiply_cgroup_events(struct test *test __maybe_unused,
+ int subtest __maybe_unused)
+{
+ int ret;
+
+ ret = multiply_default_events();
+ TEST_ASSERT_EQUAL("failed to multiply default events", ret, 0);
+
+ ret = multiply_group_events();
+ TEST_ASSERT_EQUAL("failed to multiply event group", ret, 0);
+
+ ret = multiply_metric_events();
+ TEST_ASSERT_EQUAL("failed to multiply metric events", ret, 0);
+
+ return ret;
+}
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 4447a516c689..4b7c9e3a4fc5 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -122,6 +122,7 @@ int test__pfm(struct test *test, int subtest);
const char *test__pfm_subtest_get_desc(int subtest);
int test__pfm_subtest_get_nr(void);
int test__parse_metric(struct test *test, int subtest);
+int test__multiply_cgroup_events(struct test *test, int subtest);
bool test__bp_signal_is_supported(void);
bool test__bp_account_is_supported(void);
diff --git a/tools/perf/util/cgroup.c b/tools/perf/util/cgroup.c
index 1ba61b5d66b4..0723632002fc 100644
--- a/tools/perf/util/cgroup.c
+++ b/tools/perf/util/cgroup.c
@@ -52,7 +52,7 @@ static struct cgroup *evlist__find_cgroup(struct evlist *evlist, const char *str
return NULL;
}
-static struct cgroup *cgroup__new(const char *name)
+static struct cgroup *cgroup__new(const char *name, bool do_open)
{
struct cgroup *cgroup = zalloc(sizeof(*cgroup));
@@ -62,9 +62,14 @@ static struct cgroup *cgroup__new(const char *name)
cgroup->name = strdup(name);
if (!cgroup->name)
goto out_err;
- cgroup->fd = open_cgroup(name);
- if (cgroup->fd == -1)
- goto out_free_name;
+
+ if (do_open) {
+ cgroup->fd = open_cgroup(name);
+ if (cgroup->fd == -1)
+ goto out_free_name;
+ } else {
+ cgroup->fd = -1;
+ }
}
return cgroup;
@@ -80,7 +85,7 @@ struct cgroup *evlist__findnew_cgroup(struct evlist *evlist, const char *name)
{
struct cgroup *cgroup = evlist__find_cgroup(evlist, name);
- return cgroup ?: cgroup__new(name);
+ return cgroup ?: cgroup__new(name, true);
}
static int add_cgroup(struct evlist *evlist, const char *str)
@@ -202,7 +207,7 @@ int parse_cgroups(const struct option *opt, const char *str,
}
int evlist__multiply_cgroup(struct evlist *evlist, const char *str,
- struct rblist *metric_events)
+ struct rblist *metric_events, bool open_cgroup)
{
struct evlist *orig_list, *tmp_list;
struct evsel *pos, *evsel, *leader;
@@ -240,7 +245,7 @@ int evlist__multiply_cgroup(struct evlist *evlist, const char *str,
if (!name)
break;
- cgrp = cgroup__new(name);
+ cgrp = cgroup__new(name, open_cgroup);
free(name);
if (cgrp == NULL)
break;
diff --git a/tools/perf/util/cgroup.h b/tools/perf/util/cgroup.h
index 87dde992a172..b3d75f18b3ec 100644
--- a/tools/perf/util/cgroup.h
+++ b/tools/perf/util/cgroup.h
@@ -27,7 +27,7 @@ struct rblist;
struct cgroup *evlist__findnew_cgroup(struct evlist *evlist, const char *name);
int evlist__multiply_cgroup(struct evlist *evlist, const char *cgroups,
- struct rblist *metric_events);
+ struct rblist *metric_events, bool open_cgroup);
void evlist__set_default_cgroup(struct evlist *evlist, struct cgroup *cgroup);
--
2.28.0.526.ge36021eeef-goog
The metricgroup__copy_metric_events() is to handle metrics events when
multiplying event for cgroups. As the metric events keep pointers to
evsel, it should be refreshed when events are cloned during the
operation.
The perf_stat__collect_metric_expr() is also called in case an event
has a metric directly.
During the copy, it references evsel by index as the evlist now has
cloned evsels for the given cgroup.
Cc: John Garry <[email protected]>
Cc: Kajol Jain <[email protected]>
Cc: Ian Rogers <[email protected]>
Signed-off-by: Namhyung Kim <[email protected]>
---
tools/perf/builtin-stat.c | 3 +-
tools/perf/util/cgroup.c | 15 ++++++-
tools/perf/util/cgroup.h | 4 +-
tools/perf/util/evlist.c | 11 +++++
tools/perf/util/evlist.h | 1 +
tools/perf/util/metricgroup.c | 77 +++++++++++++++++++++++++++++++++++
tools/perf/util/metricgroup.h | 6 +++
7 files changed, 114 insertions(+), 3 deletions(-)
diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 6724d23ce2e7..55a7dc175cdf 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -2261,7 +2261,8 @@ int cmd_stat(int argc, const char **argv)
goto out;
if (multiply_cgroup && stat_config.cgroups) {
- if (evlist__multiply_cgroup(evsel_list, stat_config.cgroups) < 0)
+ if (evlist__multiply_cgroup(evsel_list, stat_config.cgroups,
+ &stat_config.metric_events) < 0)
goto out;
}
diff --git a/tools/perf/util/cgroup.c b/tools/perf/util/cgroup.c
index 4892f9496bc2..1ba61b5d66b4 100644
--- a/tools/perf/util/cgroup.c
+++ b/tools/perf/util/cgroup.c
@@ -3,6 +3,9 @@
#include "evsel.h"
#include "cgroup.h"
#include "evlist.h"
+#include "rblist.h"
+#include "metricgroup.h"
+#include "stat.h"
#include <linux/zalloc.h>
#include <sys/types.h>
#include <sys/stat.h>
@@ -198,10 +201,12 @@ int parse_cgroups(const struct option *opt, const char *str,
return 0;
}
-int evlist__multiply_cgroup(struct evlist *evlist, const char *str)
+int evlist__multiply_cgroup(struct evlist *evlist, const char *str,
+ struct rblist *metric_events)
{
struct evlist *orig_list, *tmp_list;
struct evsel *pos, *evsel, *leader;
+ struct rblist orig_metric_events;
struct cgroup *cgrp = NULL;
const char *p, *e, *eos = str + strlen(str);
int ret = -1;
@@ -221,6 +226,8 @@ int evlist__multiply_cgroup(struct evlist *evlist, const char *str)
/* save original events and init evlist */
perf_evlist__splice_list_tail(orig_list, &evlist->core.entries);
evlist->core.nr_entries = 0;
+ orig_metric_events = *metric_events;
+ rblist__init(metric_events);
for (;;) {
p = strchr(str, ',');
@@ -256,6 +263,11 @@ int evlist__multiply_cgroup(struct evlist *evlist, const char *str)
cgroup__put(cgrp);
nr_cgroups++;
+ perf_stat__collect_metric_expr(tmp_list);
+ if (metricgroup__copy_metric_events(tmp_list, cgrp, metric_events,
+ &orig_metric_events) < 0)
+ break;
+
perf_evlist__splice_list_tail(evlist, &tmp_list->core.entries);
tmp_list->core.nr_entries = 0;
@@ -267,6 +279,7 @@ int evlist__multiply_cgroup(struct evlist *evlist, const char *str)
}
evlist__delete(orig_list);
evlist__delete(tmp_list);
+ rblist__exit(&orig_metric_events);
return ret;
}
diff --git a/tools/perf/util/cgroup.h b/tools/perf/util/cgroup.h
index 9a842f243dfb..87dde992a172 100644
--- a/tools/perf/util/cgroup.h
+++ b/tools/perf/util/cgroup.h
@@ -23,9 +23,11 @@ struct cgroup *cgroup__get(struct cgroup *cgroup);
void cgroup__put(struct cgroup *cgroup);
struct evlist;
+struct rblist;
struct cgroup *evlist__findnew_cgroup(struct evlist *evlist, const char *name);
-int evlist__multiply_cgroup(struct evlist *evlist, const char *cgroups);
+int evlist__multiply_cgroup(struct evlist *evlist, const char *cgroups,
+ struct rblist *metric_events);
void evlist__set_default_cgroup(struct evlist *evlist, struct cgroup *cgroup);
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index e3fa3bf7498a..457df8ce1fd9 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -1866,3 +1866,14 @@ int evlist__ctlfd_process(struct evlist *evlist, enum evlist_ctl_cmd *cmd)
return err;
}
+
+struct evsel *evlist__get_evsel(struct evlist *evlist, int idx)
+{
+ struct evsel *evsel;
+
+ evlist__for_each_entry(evlist, evsel) {
+ if (evsel->idx == idx)
+ return evsel;
+ }
+ return NULL;
+}
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index c73f7f7f120b..57f5fd5e6290 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -381,4 +381,5 @@ int evlist__ctlfd_process(struct evlist *evlist, enum evlist_ctl_cmd *cmd);
#define EVLIST_ENABLED_MSG "Events enabled\n"
#define EVLIST_DISABLED_MSG "Events disabled\n"
+struct evsel *evlist__get_evsel(struct evlist *evlist, int idx);
#endif /* __PERF_EVLIST_H */
diff --git a/tools/perf/util/metricgroup.c b/tools/perf/util/metricgroup.c
index 8831b964288f..f4a67bf85a7c 100644
--- a/tools/perf/util/metricgroup.c
+++ b/tools/perf/util/metricgroup.c
@@ -25,6 +25,7 @@
#include <api/fs/fs.h>
#include "util.h"
#include <asm/bug.h>
+#include "cgroup.h"
struct metric_event *metricgroup__lookup(struct rblist *metric_events,
struct evsel *evsel,
@@ -1104,3 +1105,79 @@ bool metricgroup__has_metric(const char *metric)
}
return false;
}
+
+int metricgroup__copy_metric_events(struct evlist *evlist, struct cgroup *cgrp,
+ struct rblist *new_metric_events,
+ struct rblist *old_metric_events)
+{
+ unsigned i;
+
+ for (i = 0; i < rblist__nr_entries(old_metric_events); i++) {
+ struct rb_node *nd;
+ struct metric_event *old_me, *new_me;
+ struct metric_expr *old_expr, *new_expr;
+ struct evsel *evsel;
+ size_t alloc_size;
+ int idx, nr;
+
+ nd = rblist__entry(old_metric_events, i);
+ old_me = container_of(nd, struct metric_event, nd);
+
+ evsel = evlist__get_evsel(evlist, old_me->evsel->idx);
+ new_me = metricgroup__lookup(new_metric_events, evsel, true);
+ if (!new_me)
+ return -ENOMEM;
+
+ pr_debug("copying metric event for cgroup '%s': %s (idx=%d)\n",
+ cgrp ? cgrp->name : "root", evsel->name, evsel->idx);
+
+ list_for_each_entry(old_expr, &old_me->head, nd) {
+ new_expr = malloc(sizeof(*new_expr));
+ if (!new_expr)
+ return -ENOMEM;
+
+ new_expr->metric_expr = old_expr->metric_expr;
+ new_expr->metric_name = old_expr->metric_name;
+ new_expr->metric_unit = old_expr->metric_unit;
+ new_expr->runtime = old_expr->runtime;
+
+ if (old_expr->metric_refs) {
+ /* calculate number of metric_events */
+ for (nr = 0; old_expr->metric_refs[nr].metric_name; nr++)
+ continue;
+ alloc_size = sizeof(*new_expr->metric_refs);
+ new_expr->metric_refs = calloc(nr + 1, alloc_size);
+ if (!new_expr->metric_refs) {
+ free(new_expr);
+ return -ENOMEM;
+ }
+
+ memcpy(new_expr->metric_refs, old_expr->metric_refs,
+ nr * alloc_size);
+ } else {
+ new_expr->metric_refs = NULL;
+ }
+
+ /* calculate number of metric_events */
+ for (nr = 0; old_expr->metric_events[nr]; nr++)
+ continue;
+ alloc_size = sizeof(*new_expr->metric_events);
+ new_expr->metric_events = calloc(nr + 1, alloc_size);
+ if (!new_expr->metric_events) {
+ free(new_expr->metric_refs);
+ free(new_expr);
+ return -ENOMEM;
+ }
+
+ /* copy evsel in the same position */
+ for (idx = 0; idx < nr; idx++) {
+ evsel = old_expr->metric_events[idx];
+ evsel = evlist__get_evsel(evlist, evsel->idx);
+ new_expr->metric_events[idx] = evsel;
+ }
+
+ list_add(&new_expr->nd, &new_me->head);
+ }
+ }
+ return 0;
+}
diff --git a/tools/perf/util/metricgroup.h b/tools/perf/util/metricgroup.h
index 62623a39cbec..bb9ea9e1ea9a 100644
--- a/tools/perf/util/metricgroup.h
+++ b/tools/perf/util/metricgroup.h
@@ -6,11 +6,13 @@
#include <linux/rbtree.h>
#include <stdbool.h>
+struct evlist;
struct evsel;
struct evlist;
struct option;
struct rblist;
struct pmu_events_map;
+struct cgroup;
struct metric_event {
struct rb_node nd;
@@ -54,4 +56,8 @@ void metricgroup__print(bool metrics, bool groups, char *filter,
bool metricgroup__has_metric(const char *metric);
int arch_get_runtimeparam(void);
void metricgroup__rblist_exit(struct rblist *metric_events);
+
+int metricgroup__copy_metric_events(struct evlist *evlist, struct cgroup *cgrp,
+ struct rblist *new_metric_events,
+ struct rblist *old_metric_events);
#endif
--
2.28.0.526.ge36021eeef-goog
On Tue, Sep 08, 2020 at 01:42:25PM +0900, Namhyung Kim wrote:
> The evsel__clone() is to create an exactly same evsel from same
> attributes. Note that metric events will be handled by later patch.
>
> It will be used by perf stat to generate separate events for each
> cgroup.
>
> Signed-off-by: Namhyung Kim <[email protected]>
> ---
> tools/perf/util/evsel.c | 57 +++++++++++++++++++++++++++++++++++++++++
> tools/perf/util/evsel.h | 1 +
> 2 files changed, 58 insertions(+)
>
> diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> index fd865002cbbd..4f50f9499973 100644
> --- a/tools/perf/util/evsel.c
> +++ b/tools/perf/util/evsel.c
> @@ -331,6 +331,63 @@ struct evsel *evsel__new_cycles(bool precise)
> goto out;
> }
>
> +/**
> + * evsel__clone - create a new evsel copied from @orig
> + * @orig: original evsel
> + *
> + * The assumption is that @orig is not configured nor opened yet.
> + * So we only care about the attributes that can be set while it's parsed.
> + */
> +struct evsel *evsel__clone(struct evsel *orig)
> +{
> + struct evsel *evsel;
> + struct evsel_config_term *pos, *tmp;
> +
> + BUG_ON(orig->core.fd);
> +
> + evsel = evsel__new(&orig->core.attr);
> + if (evsel == NULL)
> + return NULL;
> +
> + *evsel = *orig;
this seems wild ;-) I saw that assumption above,
but I wonder we could add some check or zero/init
the rest of the fields fields
jirka
On Tue, Sep 08, 2020 at 01:42:24PM +0900, Namhyung Kim wrote:
> Hello,
>
> When we profile cgroup events with perf stat, it's very annoying to
> specify events and cgroups on the command line as it requires the
> mapping between events and cgroups. (Note that perf record can use
> cgroup sampling but it's not usable for perf stat).
>
> I guess most cases we just want to use a same set of events (N) for
> all cgroups (M), but we need to specify NxM events and NxM cgroups.
> This is not good especially when profiling large number of cgroups:
> say M=200.
>
> So I added --multiply-cgroup option to make it easy for that case. It
> will create NxM events from N events and M cgroups. One more upside
> is that it can handle metrics too.
agreed that it's PITA to use -G option ;-)
>
> For example, the following example measures IPC metric for 3 cgroups
>
> $ cat perf-multi-cgrp.sh
> #!/bin/sh
>
> METRIC=${1:-IPC}
> CGROUP_DIR=/sys/fs/cgroup/perf_event
>
> sudo mkdir $CGROUP_DIR/A $CGROUP_DIR/B $CGROUP_DIR/C
>
> # add backgroupd workload for each cgroup
> echo $$ | sudo tee $CGROUP_DIR/A/cgroup.procs > /dev/null
> yes > /dev/null &
> echo $$ | sudo tee $CGROUP_DIR/B/cgroup.procs > /dev/null
> yes > /dev/null &
> echo $$ | sudo tee $CGROUP_DIR/C/cgroup.procs > /dev/null
> yes > /dev/null &
>
> # run 'perf stat' in the root cgroup
> echo $$ | sudo tee $CGROUP_DIR/cgroup.procs > /dev/null
> perf stat -a -M $METRIC --multiply-cgroup -G A,B,C sleep 1
would it be easier to have new option for this? like:
perf stat -a -M $METRIC --for-cgroup A,B,C
perf stat -a -M $METRIC --for-each-cgroup A,B,C
perf stat -a -M $METRIC --attach-cgroup A,B,C
perf stat -a -M $METRIC --attach-to-cgroup A,B,C
I'm still not sure how the --multiply-cgroup deals with empty
cgroup A,,C but looks like we don't need this behaviour now?
thanks,
jirka
Hi Jiri,
On Thu, Sep 10, 2020 at 5:59 PM Jiri Olsa <[email protected]> wrote:
>
> On Tue, Sep 08, 2020 at 01:42:25PM +0900, Namhyung Kim wrote:
> > The evsel__clone() is to create an exactly same evsel from same
> > attributes. Note that metric events will be handled by later patch.
> >
> > It will be used by perf stat to generate separate events for each
> > cgroup.
> >
> > Signed-off-by: Namhyung Kim <[email protected]>
> > ---
> > tools/perf/util/evsel.c | 57 +++++++++++++++++++++++++++++++++++++++++
> > tools/perf/util/evsel.h | 1 +
> > 2 files changed, 58 insertions(+)
> >
> > diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> > index fd865002cbbd..4f50f9499973 100644
> > --- a/tools/perf/util/evsel.c
> > +++ b/tools/perf/util/evsel.c
> > @@ -331,6 +331,63 @@ struct evsel *evsel__new_cycles(bool precise)
> > goto out;
> > }
> >
> > +/**
> > + * evsel__clone - create a new evsel copied from @orig
> > + * @orig: original evsel
> > + *
> > + * The assumption is that @orig is not configured nor opened yet.
> > + * So we only care about the attributes that can be set while it's parsed.
> > + */
> > +struct evsel *evsel__clone(struct evsel *orig)
> > +{
> > + struct evsel *evsel;
> > + struct evsel_config_term *pos, *tmp;
> > +
> > + BUG_ON(orig->core.fd);
> > +
> > + evsel = evsel__new(&orig->core.attr);
> > + if (evsel == NULL)
> > + return NULL;
> > +
> > + *evsel = *orig;
>
> this seems wild ;-) I saw that assumption above,
> but I wonder we could add some check or zero/init
> the rest of the fields fields
Alternatively, we could only copy relevant fields explicitly.
Other fields will remain zero by evsel__new() above.
But it might be easy to be missed by future changes..
Thanks
Namhyung
On Tue, Sep 08, 2020 at 01:42:24PM +0900, Namhyung Kim wrote:
> When we profile cgroup events with perf stat, it's very annoying to
> specify events and cgroups on the command line as it requires the
> mapping between events and cgroups. (Note that perf record can use
> cgroup sampling but it's not usable for perf stat).
The problem is real, but I don't really like your solution.
The option is ugly. Should rather be solved with some suitable
syntax in the expression parser to express: apply to all,
instead of adding adhoc options like this.
There are some additional problems that really need to be eventually
solved too:
- If you use the old syntax and some cgroups are not covered you don't
get any warning. At least that should be fixed too.
- And of course if everything works it is still very slow for the kernel
because there are so many perf events to handle. Long term we probably
need some more flexible way to just specify for given perf events which set of
cgroups they should apply, so that sharing and low overhead monitoring
of many cgroups is possible I hate to say it, but maybe some eBPF filter
is the solution here.
-Andi
On Thu, Sep 10, 2020 at 8:57 AM Andi Kleen <[email protected]> wrote:
>
> On Tue, Sep 08, 2020 at 01:42:24PM +0900, Namhyung Kim wrote:
> > When we profile cgroup events with perf stat, it's very annoying to
> > specify events and cgroups on the command line as it requires the
> > mapping between events and cgroups. (Note that perf record can use
> > cgroup sampling but it's not usable for perf stat).
>
> The problem is real, but I don't really like your solution.
> The option is ugly. Should rather be solved with some suitable
> syntax in the expression parser to express: apply to all,
> instead of adding adhoc options like this.
>
> There are some additional problems that really need to be eventually
> solved too:
>
> - If you use the old syntax and some cgroups are not covered you don't
> get any warning. At least that should be fixed too.
>
> - And of course if everything works it is still very slow for the kernel
> because there are so many perf events to handle. Long term we probably
> need some more flexible way to just specify for given perf events which set of
> cgroups they should apply, so that sharing and low overhead monitoring
> of many cgroups is possible I hate to say it, but maybe some eBPF filter
> is the solution here.
>
> -Andi
Just to throw in some 2c worth. A nice thing about Namhyung's approach
is that it can apply for metrics, events and pfm-events. It would be
nice if there were a single option for specifying these, perhaps we
can figure one out.
One thought that came to mind based on Namhyung's multiply name was
and assuming a cgroup could be a modifier:
perf record -e cycles:G1
there could be equivalent to a new syntax of lists and multiplies of:
perf record -e [cycles]*[:G1]
This would allow expressions like:
perf record -e [{instructions,cycles},branches]*[:u,:k]
which would be equivalent to (showing the effect on sibling groups):
perf record -e {instructions:u,cycles:u},branches:u,{instructions:k,cycles:k},branches:k
Adding in cgroups makes a longer list of events:
perf record -e [{instructions,cycles},branches]*[:u,:k]*[:G1,:G2]
becomes:
perf record -e {instructions:u:G1,cycles:u:G1},branches:u:G1,{instructions:k:G1,cycles:k:G1},branches:k:G1,{instructions:u:G2,cycles:u:G2},branches:u:G2,{instructions:k:G2,cycles:k:G2},branches:k:G2
This is somewhat similar to Arnaldo's proposal but trying to make
things a bit more generic, avoiding overloading the use of sibling
groups, .. Perhaps there is a syntax that others prefer or could be
borrowed from a familiar source like a programming language.
I think for Namhyung's sake it is important not to get too distracted
by desires for better syntax, as this change makes a benefit in a way
that works with the existing flags. If it is accepted, the man pages
need to be updated.
Thanks,
Ian
Hi Arnaldo,
On Thu, Sep 10, 2020 at 8:10 PM Arnaldo Carvalho de Melo
<[email protected]> wrote:
>
> Em Thu, Sep 10, 2020 at 11:15:42AM +0200, Jiri Olsa escreveu:
> > On Tue, Sep 08, 2020 at 01:42:24PM +0900, Namhyung Kim wrote:
> > > When we profile cgroup events with perf stat, it's very annoying to
> > > specify events and cgroups on the command line as it requires the
> > > mapping between events and cgroups. (Note that perf record can use
> > > cgroup sampling but it's not usable for perf stat).
>
> > > I guess most cases we just want to use a same set of events (N) for
> > > all cgroups (M), but we need to specify NxM events and NxM cgroups.
> > > This is not good especially when profiling large number of cgroups:
> > > say M=200.
>
> > > So I added --multiply-cgroup option to make it easy for that case. It
> > > will create NxM events from N events and M cgroups. One more upside
> > > is that it can handle metrics too.
>
> > agreed that it's PITA to use -G option ;-)
>
> yeah, its great that someone is looking at cgroups improvements, thanks
> Namyung, its great to have you working on this!
Thanks! :)
>
> More below.
>
> > > For example, the following example measures IPC metric for 3 cgroups
>
> > > $ cat perf-multi-cgrp.sh
> > > #!/bin/sh
>
> > > METRIC=${1:-IPC}
> > > CGROUP_DIR=/sys/fs/cgroup/perf_event
>
> > > sudo mkdir $CGROUP_DIR/A $CGROUP_DIR/B $CGROUP_DIR/C
>
> > > # add backgroupd workload for each cgroup
> > > echo $$ | sudo tee $CGROUP_DIR/A/cgroup.procs > /dev/null
> > > yes > /dev/null &
> > > echo $$ | sudo tee $CGROUP_DIR/B/cgroup.procs > /dev/null
> > > yes > /dev/null &
> > > echo $$ | sudo tee $CGROUP_DIR/C/cgroup.procs > /dev/null
> > > yes > /dev/null &
>
> > > # run 'perf stat' in the root cgroup
> > > echo $$ | sudo tee $CGROUP_DIR/cgroup.procs > /dev/null
> > > perf stat -a -M $METRIC --multiply-cgroup -G A,B,C sleep 1
> >
> > would it be easier to have new option for this? like:
> >
> > perf stat -a -M $METRIC --for-cgroup A,B,C
> > perf stat -a -M $METRIC --for-each-cgroup A,B,C
> > perf stat -a -M $METRIC --attach-cgroup A,B,C
> > perf stat -a -M $METRIC --attach-to-cgroup A,B,C
Looks good. I like the --for-each-cgroup.
Then we should make it and -G mutually exclusive IMHO.
> >
> > I'm still not sure how the --multiply-cgroup deals with empty
> > cgroup A,,C but looks like we don't need this behaviour now?
Yep, it can handle such case and bind the events to the root cgroup.
>
> Yeah, I also didn't like the --multiply-cgroup thing, perhaps we can use
> a per-event term? or per group, for example:
>
> perf stat -a -M $METRIC/cgroups=A,B,C/
> perf stat -a -e '{cycles,instructions,cache-misses}/cgroups=A,B,C/'
>
> Allowing wildcards or regexps would help with some use cases.
>
> We already have several terms that allows us to control per event knobs,
> this would be one more.
At some point, we can support this kind of flexibility, but it'd make the code
complex when multiple events or metrics have different cgroup membership.
So I'd like to do the simple case (all events are expanded to same cgroups)
first..
Thanks
Namhyung
Em Thu, Sep 10, 2020 at 11:15:42AM +0200, Jiri Olsa escreveu:
> On Tue, Sep 08, 2020 at 01:42:24PM +0900, Namhyung Kim wrote:
> > When we profile cgroup events with perf stat, it's very annoying to
> > specify events and cgroups on the command line as it requires the
> > mapping between events and cgroups. (Note that perf record can use
> > cgroup sampling but it's not usable for perf stat).
> > I guess most cases we just want to use a same set of events (N) for
> > all cgroups (M), but we need to specify NxM events and NxM cgroups.
> > This is not good especially when profiling large number of cgroups:
> > say M=200.
> > So I added --multiply-cgroup option to make it easy for that case. It
> > will create NxM events from N events and M cgroups. One more upside
> > is that it can handle metrics too.
> agreed that it's PITA to use -G option ;-)
yeah, its great that someone is looking at cgroups improvements, thanks
Namyung, its great to have you working on this!
More below.
> > For example, the following example measures IPC metric for 3 cgroups
> > $ cat perf-multi-cgrp.sh
> > #!/bin/sh
> > METRIC=${1:-IPC}
> > CGROUP_DIR=/sys/fs/cgroup/perf_event
> > sudo mkdir $CGROUP_DIR/A $CGROUP_DIR/B $CGROUP_DIR/C
> > # add backgroupd workload for each cgroup
> > echo $$ | sudo tee $CGROUP_DIR/A/cgroup.procs > /dev/null
> > yes > /dev/null &
> > echo $$ | sudo tee $CGROUP_DIR/B/cgroup.procs > /dev/null
> > yes > /dev/null &
> > echo $$ | sudo tee $CGROUP_DIR/C/cgroup.procs > /dev/null
> > yes > /dev/null &
> > # run 'perf stat' in the root cgroup
> > echo $$ | sudo tee $CGROUP_DIR/cgroup.procs > /dev/null
> > perf stat -a -M $METRIC --multiply-cgroup -G A,B,C sleep 1
>
> would it be easier to have new option for this? like:
>
> perf stat -a -M $METRIC --for-cgroup A,B,C
> perf stat -a -M $METRIC --for-each-cgroup A,B,C
> perf stat -a -M $METRIC --attach-cgroup A,B,C
> perf stat -a -M $METRIC --attach-to-cgroup A,B,C
>
> I'm still not sure how the --multiply-cgroup deals with empty
> cgroup A,,C but looks like we don't need this behaviour now?
Yeah, I also didn't like the --multiply-cgroup thing, perhaps we can use
a per-event term? or per group, for example:
perf stat -a -M $METRIC/cgroups=A,B,C/
perf stat -a -e '{cycles,instructions,cache-misses}/cgroups=A,B,C/'
Allowing wildcards or regexps would help with some use cases.
We already have several terms that allows us to control per event knobs,
this would be one more.
- Arnaldo
Hi Andi,
On Fri, Sep 11, 2020 at 1:05 AM Andi Kleen <[email protected]> wrote:
>
> On Tue, Sep 08, 2020 at 01:42:24PM +0900, Namhyung Kim wrote:
> > When we profile cgroup events with perf stat, it's very annoying to
> > specify events and cgroups on the command line as it requires the
> > mapping between events and cgroups. (Note that perf record can use
> > cgroup sampling but it's not usable for perf stat).
>
> The problem is real, but I don't really like your solution.
> The option is ugly. Should rather be solved with some suitable
> syntax in the expression parser to express: apply to all,
> instead of adding adhoc options like this.
Yeah, I'd admit that I'm terrible at naming. :)
I'm open to any suggestions as I'm also bad at making new syntax..
But as Ian said, my approach takes care of the common case
without dealing with controversial syntax changes.
>
> There are some additional problems that really need to be eventually
> solved too:
>
> - If you use the old syntax and some cgroups are not covered you don't
> get any warning. At least that should be fixed too.
I don't follow. Do you mean like when -G option and --multiply-cgroup option
is used at the same time? Or when -G option doesn't match to -e option?
>
> - And of course if everything works it is still very slow for the kernel
> because there are so many perf events to handle. Long term we probably
> need some more flexible way to just specify for given perf events which set of
> cgroups they should apply, so that sharing and low overhead monitoring
> of many cgroups is possible I hate to say it, but maybe some eBPF filter
> is the solution here.
Yes, I'm also interested in improving that too.
Thanks
Namhyung
Hi Ian,
On Fri, Sep 11, 2020 at 2:11 AM Ian Rogers <[email protected]> wrote:
> Just to throw in some 2c worth. A nice thing about Namhyung's approach
> is that it can apply for metrics, events and pfm-events. It would be
> nice if there were a single option for specifying these, perhaps we
> can figure one out.
Right! Thanks for sharing your opinion. :)
>
> One thought that came to mind based on Namhyung's multiply name was
> and assuming a cgroup could be a modifier:
>
> perf record -e cycles:G1
>
> there could be equivalent to a new syntax of lists and multiplies of:
>
> perf record -e [cycles]*[:G1]
>
> This would allow expressions like:
>
> perf record -e [{instructions,cycles},branches]*[:u,:k]
>
> which would be equivalent to (showing the effect on sibling groups):
>
> perf record -e {instructions:u,cycles:u},branches:u,{instructions:k,cycles:k},branches:k
>
> Adding in cgroups makes a longer list of events:
>
> perf record -e [{instructions,cycles},branches]*[:u,:k]*[:G1,:G2]
>
> becomes:
>
> perf record -e {instructions:u:G1,cycles:u:G1},branches:u:G1,{instructions:k:G1,cycles:k:G1},branches:k:G1,{instructions:u:G2,cycles:u:G2},branches:u:G2,{instructions:k:G2,cycles:k:G2},branches:k:G2
>
> This is somewhat similar to Arnaldo's proposal but trying to make
> things a bit more generic, avoiding overloading the use of sibling
> groups, .. Perhaps there is a syntax that others prefer or could be
> borrowed from a familiar source like a programming language.
Yes, but this is like a radical change. This would be nice to deal
with repetition in the event list. But I guess mostly we only use a
small number of events (beside the cgroups)...
>
> I think for Namhyung's sake it is important not to get too distracted
> by desires for better syntax, as this change makes a benefit in a way
> that works with the existing flags. If it is accepted, the man pages
> need to be updated.
Right. I wonder whether they're gonna take the new option
(maybe --for-each-cgroup ?) or not regardless of the syntax change.
Thanks
Namhyung
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 77b66fd55195289f5361af4f8a0978a3a90b9363 ("[PATCH 3/4] perf tools: Copy metric events properly when multiply cgroups")
url: https://github.com/0day-ci/linux/commits/Namhyung-Kim/perf-stat-Add-multiply-cgroup-option/20200908-124454
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 2cb5383b30d47c446ec7d884cd80f93ffcc31817
in testcase: perf-sanity-tests
version: perf-x86_64-34d4ddd359db-1_20200909
with following parameters:
perf_compiler: gcc
ucode: 0xdc
on test machine: 8 threads Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz with 16G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 1
1: vmlinux symtab matches kallsyms : Ok
2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 2
2: Detect openat syscall event : Ok
2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 3
3: Detect openat syscall event on all cpus : Ok
2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 4
4: Read samples using the mmap interface : Ok
2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 5
5: Test data source output : Ok
2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 6
6: Parse event definition strings : Ok
2020-09-23 04:43:46 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 7
7: Simple expression parser : Ok
2020-09-23 04:43:46 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 8
8: PERF_RECORD_* events & perf_sample fields : Ok
2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 9
9: Parse perf pmu format : Ok
2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 10
10: PMU events :
10.1: PMU event table sanity : Ok
10.2: PMU event map aliases : Ok
10.3: Parsing of PMU event table metrics : Skip (some metrics failed)
10.4: Parsing of PMU event table metrics with fake PMUs : Ok
2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 11
11: DSO data read : Ok
2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 12
12: DSO data cache : Ok
2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 13
13: DSO data reopen : Ok
2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 14
14: Roundtrip evsel->name : Ok
2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 15
15: Parse sched tracepoints fields : Ok
2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 16
16: syscalls:sys_enter_openat event fields : Ok
2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 17
17: Setup struct perf_event_attr : Ok
2020-09-23 04:43:54 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 18
18: Match and link multiple hists : Ok
2020-09-23 04:43:54 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 19
19: 'import perf' in python : FAILED!
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
Thanks,
Rong Chen
Hello,
On Wed, Sep 23, 2020 at 6:15 PM kernel test robot <[email protected]> wrote:
>
> Greeting,
>
> FYI, we noticed the following commit (built with gcc-9):
>
> commit: 77b66fd55195289f5361af4f8a0978a3a90b9363 ("[PATCH 3/4] perf tools: Copy metric events properly when multiply cgroups")
> url: https://github.com/0day-ci/linux/commits/Namhyung-Kim/perf-stat-Add-multiply-cgroup-option/20200908-124454
> base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 2cb5383b30d47c446ec7d884cd80f93ffcc31817
>
> in testcase: perf-sanity-tests
> version: perf-x86_64-34d4ddd359db-1_20200909
> with following parameters:
>
> perf_compiler: gcc
> ucode: 0xdc
>
>
>
> on test machine: 8 threads Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz with 16G memory
>
> caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
>
>
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <[email protected]>
Thanks for the report. I'll fix it and send it in the next version.
Thanks
Namhyung
>
>
> 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 1
> 1: vmlinux symtab matches kallsyms : Ok
> 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 2
> 2: Detect openat syscall event : Ok
> 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 3
> 3: Detect openat syscall event on all cpus : Ok
> 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 4
> 4: Read samples using the mmap interface : Ok
> 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 5
> 5: Test data source output : Ok
> 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 6
> 6: Parse event definition strings : Ok
> 2020-09-23 04:43:46 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 7
> 7: Simple expression parser : Ok
> 2020-09-23 04:43:46 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 8
> 8: PERF_RECORD_* events & perf_sample fields : Ok
> 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 9
> 9: Parse perf pmu format : Ok
> 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 10
> 10: PMU events :
> 10.1: PMU event table sanity : Ok
> 10.2: PMU event map aliases : Ok
> 10.3: Parsing of PMU event table metrics : Skip (some metrics failed)
> 10.4: Parsing of PMU event table metrics with fake PMUs : Ok
> 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 11
> 11: DSO data read : Ok
> 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 12
> 12: DSO data cache : Ok
> 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 13
> 13: DSO data reopen : Ok
> 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 14
> 14: Roundtrip evsel->name : Ok
> 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 15
> 15: Parse sched tracepoints fields : Ok
> 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 16
> 16: syscalls:sys_enter_openat event fields : Ok
> 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 17
> 17: Setup struct perf_event_attr : Ok
> 2020-09-23 04:43:54 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 18
> 18: Match and link multiple hists : Ok
> 2020-09-23 04:43:54 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 19
> 19: 'import perf' in python : FAILED!
>
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
>
>
> Thanks,
> Rong Chen
>
Em Thu, Sep 24, 2020 at 12:04:51PM +0900, Namhyung Kim escreveu:
> On Wed, Sep 23, 2020 at 6:15 PM kernel test robot <[email protected]> wrote:
> > FYI, we noticed the following commit (built with gcc-9):
> > commit: 77b66fd55195289f5361af4f8a0978a3a90b9363 ("[PATCH 3/4] perf tools: Copy metric events properly when multiply cgroups")
> > url: https://github.com/0day-ci/linux/commits/Namhyung-Kim/perf-stat-Add-multiply-cgroup-option/20200908-124454
> > base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 2cb5383b30d47c446ec7d884cd80f93ffcc31817
> > in testcase: perf-sanity-tests
> > version: perf-x86_64-34d4ddd359db-1_20200909
> > with following parameters:
> > perf_compiler: gcc
> > ucode: 0xdc
> > on test machine: 8 threads Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz with 16G memory
> > caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
> > If you fix the issue, kindly add following tag
> > Reported-by: kernel test robot <[email protected]>
>
> Thanks for the report. I'll fix it and send it in the next version.
I'm so glad this is in place, thanks to all involved!
- Arnaldo
> Thanks
> Namhyung
> >
> >
> > 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 1
> > 1: vmlinux symtab matches kallsyms : Ok
> > 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 2
> > 2: Detect openat syscall event : Ok
> > 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 3
> > 3: Detect openat syscall event on all cpus : Ok
> > 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 4
> > 4: Read samples using the mmap interface : Ok
> > 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 5
> > 5: Test data source output : Ok
> > 2020-09-23 04:43:45 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 6
> > 6: Parse event definition strings : Ok
> > 2020-09-23 04:43:46 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 7
> > 7: Simple expression parser : Ok
> > 2020-09-23 04:43:46 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 8
> > 8: PERF_RECORD_* events & perf_sample fields : Ok
> > 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 9
> > 9: Parse perf pmu format : Ok
> > 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 10
> > 10: PMU events :
> > 10.1: PMU event table sanity : Ok
> > 10.2: PMU event map aliases : Ok
> > 10.3: Parsing of PMU event table metrics : Skip (some metrics failed)
> > 10.4: Parsing of PMU event table metrics with fake PMUs : Ok
> > 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 11
> > 11: DSO data read : Ok
> > 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 12
> > 12: DSO data cache : Ok
> > 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 13
> > 13: DSO data reopen : Ok
> > 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 14
> > 14: Roundtrip evsel->name : Ok
> > 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 15
> > 15: Parse sched tracepoints fields : Ok
> > 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 16
> > 16: syscalls:sys_enter_openat event fields : Ok
> > 2020-09-23 04:43:48 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 17
> > 17: Setup struct perf_event_attr : Ok
> > 2020-09-23 04:43:54 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 18
> > 18: Match and link multiple hists : Ok
> > 2020-09-23 04:43:54 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-77b66fd55195289f5361af4f8a0978a3a90b9363/tools/perf/perf test 19
> > 19: 'import perf' in python : FAILED!
> >
> >
> >
> > To reproduce:
> >
> > git clone https://github.com/intel/lkp-tests.git
> > cd lkp-tests
> > bin/lkp install job.yaml # job file is attached in this email
> > bin/lkp run job.yaml
> >
> >
> >
> > Thanks,
> > Rong Chen
> >
--
- Arnaldo