changes v3 => v4:
1. Do not reshuffle intel_pmu_disable_all(). Use some inline to save LBR
entries. (Peter)
2. Move static_call(perf_snapshot_branch_stack) to the helper. (Alexei)
3. Add argument flags to bpf_get_branch_snapshot. (Andrii)
4. Make MAX_BRANCH_SNAPSHOT an enum (Andrii). And rename it as
PERF_MAX_BRANCH_SNAPSHOT
5. Make bpf_get_branch_snapshot similar to bpf_read_branch_records.
(Andrii)
6. Move the test target function to bpf_testmod. Updated kallsyms_find_next
to work properly with modules. (Andrii)
Changes v2 => v3:
1. Fix the use of static_call. (Peter)
2. Limit the use to perfmon version >= 2. (Peter)
3. Modify intel_pmu_snapshot_branch_stack() to use intel_pmu_disable_all
and intel_pmu_enable_all().
Changes v1 => v2:
1. Rename the helper as bpf_get_branch_snapshot;
2. Fix/simplify the use of static_call;
3. Instead of percpu variables, let intel_pmu_snapshot_branch_stack output
branch records to an output argument of type perf_branch_snapshot.
Branch stack can be very useful in understanding software events. For
example, when a long function, e.g. sys_perf_event_open, returns an errno,
it is not obvious why the function failed. Branch stack could provide very
helpful information in this type of scenarios.
This set adds support to read branch stack with a new BPF helper
bpf_get_branch_trace(). Currently, this is only supported in Intel systems.
It is also possible to support the same feaure for PowerPC.
The hardware that records the branch stace is not stopped automatically on
software events. Therefore, it is necessary to stop it in software soon.
Otherwise, the hardware buffers/registers will be flushed. One of the key
design consideration in this set is to minimize the number of branch record
entries between the event triggers and the hardware recorder is stopped.
Based on this goal, current design is different from the discussions in
original RFC [1]:
1) Static call is used when supported, to save function pointer
dereference;
2) intel_pmu_lbr_disable_all is used instead of perf_pmu_disable(),
because the latter uses about 10 entries before stopping LBR.
With current code, on Intel CPU, LBR is stopped after 10 branch entries
after fexit triggers:
ID: 0 from intel_pmu_lbr_disable_all+58 to intel_pmu_lbr_disable_all+93
ID: 1 from intel_pmu_lbr_disable_all+54 to intel_pmu_lbr_disable_all+58
ID: 2 from intel_pmu_snapshot_branch_stack+88 to intel_pmu_lbr_disable_all+0
ID: 3 from bpf_get_branch_snapshot+77 to intel_pmu_snapshot_branch_stack+0
ID: 4 from __brk_limit+478052814 to bpf_get_branch_snapshot+0
ID: 5 from __brk_limit+478036039 to __brk_limit+478052760
ID: 6 from __bpf_prog_enter+34 to __brk_limit+478036027
ID: 7 from migrate_disable+60 to __bpf_prog_enter+9
ID: 8 from __bpf_prog_enter+4 to migrate_disable+0
ID: 9 from __brk_limit+478036022 to __bpf_prog_enter+0
ID: 10 from bpf_testmod_loop_test+22 to __brk_limit+478036003
ID: 11 from bpf_testmod_loop_test+20 to bpf_testmod_loop_test+13
ID: 12 from bpf_testmod_loop_test+20 to bpf_testmod_loop_test+13
ID: 13 from bpf_testmod_loop_test+20 to bpf_testmod_loop_test+13
...
[1] https://lore.kernel.org/bpf/[email protected]/
Song Liu (3):
perf: enable branch record for software events
bpf: introduce helper bpf_get_branch_snapshot
selftests/bpf: add test for bpf_get_branch_snapshot
arch/x86/events/intel/core.c | 26 ++++-
arch/x86/events/intel/ds.c | 8 --
arch/x86/events/perf_event.h | 10 +-
include/linux/perf_event.h | 26 +++++
include/uapi/linux/bpf.h | 22 ++++
kernel/bpf/trampoline.c | 3 +-
kernel/events/core.c | 2 +
kernel/trace/bpf_trace.c | 40 +++++++
tools/include/uapi/linux/bpf.h | 22 ++++
.../selftests/bpf/bpf_testmod/bpf_testmod.c | 14 ++-
.../bpf/prog_tests/get_branch_snapshot.c | 101 ++++++++++++++++++
.../selftests/bpf/progs/get_branch_snapshot.c | 44 ++++++++
tools/testing/selftests/bpf/trace_helpers.c | 37 +++++++
tools/testing/selftests/bpf/trace_helpers.h | 5 +
14 files changed, 345 insertions(+), 15 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
create mode 100644 tools/testing/selftests/bpf/progs/get_branch_snapshot.c
--
2.30.2
Introduce bpf_get_branch_snapshot(), which allows tracing pogram to get
branch trace from hardware (e.g. Intel LBR). To use the feature, the
user need to create perf_event with proper branch_record filtering
on each cpu, and then calls bpf_get_branch_snapshot in the bpf function.
On Intel CPUs, VLBR event (raw event 0x1b00) can be use for this.
Signed-off-by: Song Liu <[email protected]>
---
include/uapi/linux/bpf.h | 22 +++++++++++++++++++
kernel/bpf/trampoline.c | 3 ++-
kernel/trace/bpf_trace.c | 40 ++++++++++++++++++++++++++++++++++
tools/include/uapi/linux/bpf.h | 22 +++++++++++++++++++
4 files changed, 86 insertions(+), 1 deletion(-)
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 791f31dd0abee..c986e6fad5bc0 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -4877,6 +4877,27 @@ union bpf_attr {
* Get the struct pt_regs associated with **task**.
* Return
* A pointer to struct pt_regs.
+ *
+ * long bpf_get_branch_snapshot(void *entries, u32 size, u64 flags)
+ * Description
+ * Get branch trace from hardware engines like Intel LBR. The
+ * branch trace is taken soon after the trigger point of the
+ * BPF program, so it may contain some entries after the
+ * trigger point. The user need to filter these entries
+ * accordingly.
+ *
+ * The data is stored as struct perf_branch_entry into output
+ * buffer *entries*. *size* is the size of *entries* in bytes.
+ * *flags* is reserved for now and must be zero.
+ *
+ * Return
+ * On success, number of bytes written to *buf*. On error, a
+ * negative value.
+ *
+ * **-EINVAL** if arguments invalid or **size** not a multiple
+ * of **sizeof**\ (**struct perf_branch_entry**\ ).
+ *
+ * **-ENOENT** if architecture does not support branch records.
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
@@ -5055,6 +5076,7 @@ union bpf_attr {
FN(get_func_ip), \
FN(get_attach_cookie), \
FN(task_pt_regs), \
+ FN(get_branch_snapshot), \
/* */
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index fe1e857324e66..39eaaff81953d 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -10,6 +10,7 @@
#include <linux/rcupdate_trace.h>
#include <linux/rcupdate_wait.h>
#include <linux/module.h>
+#include <linux/static_call.h>
/* dummy _ops. The verifier will operate on target program's ops. */
const struct bpf_verifier_ops bpf_extension_verifier_ops = {
@@ -526,7 +527,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
}
#define NO_START_TIME 1
-static u64 notrace bpf_prog_start_time(void)
+static __always_inline u64 notrace bpf_prog_start_time(void)
{
u64 start = NO_START_TIME;
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 8e2eb950aa829..a8ec3634a3329 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1017,6 +1017,44 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto_pe = {
.arg1_type = ARG_PTR_TO_CTX,
};
+static DEFINE_PER_CPU(struct perf_branch_snapshot, bpf_perf_branch_snapshot);
+
+BPF_CALL_3(bpf_get_branch_snapshot, void *, buf, u32, size, u64, flags)
+{
+#ifndef CONFIG_X86
+ return -ENOENT;
+#else
+ static const u32 br_entry_size = sizeof(struct perf_branch_entry);
+ u32 to_copy;
+
+ if (unlikely(flags))
+ return -EINVAL;
+
+ if (!buf || (size % br_entry_size != 0))
+ return -EINVAL;
+
+ static_call(perf_snapshot_branch_stack)(this_cpu_ptr(&bpf_perf_branch_snapshot));
+
+ if (this_cpu_ptr(&bpf_perf_branch_snapshot)->nr == 0)
+ return -ENOENT;
+
+ to_copy = this_cpu_ptr(&bpf_perf_branch_snapshot)->nr *
+ sizeof(struct perf_branch_entry);
+ to_copy = min_t(u32, size, to_copy);
+ memcpy(buf, this_cpu_ptr(&bpf_perf_branch_snapshot)->entries, to_copy);
+
+ return to_copy;
+#endif
+}
+
+static const struct bpf_func_proto bpf_get_branch_snapshot_proto = {
+ .func = bpf_get_branch_snapshot,
+ .gpl_only = true,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_UNINIT_MEM,
+ .arg2_type = ARG_CONST_SIZE_OR_ZERO,
+};
+
static const struct bpf_func_proto *
bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
@@ -1132,6 +1170,8 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
return &bpf_snprintf_proto;
case BPF_FUNC_get_func_ip:
return &bpf_get_func_ip_proto_tracing;
+ case BPF_FUNC_get_branch_snapshot:
+ return &bpf_get_branch_snapshot_proto;
default:
return bpf_base_func_proto(func_id);
}
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 791f31dd0abee..c986e6fad5bc0 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -4877,6 +4877,27 @@ union bpf_attr {
* Get the struct pt_regs associated with **task**.
* Return
* A pointer to struct pt_regs.
+ *
+ * long bpf_get_branch_snapshot(void *entries, u32 size, u64 flags)
+ * Description
+ * Get branch trace from hardware engines like Intel LBR. The
+ * branch trace is taken soon after the trigger point of the
+ * BPF program, so it may contain some entries after the
+ * trigger point. The user need to filter these entries
+ * accordingly.
+ *
+ * The data is stored as struct perf_branch_entry into output
+ * buffer *entries*. *size* is the size of *entries* in bytes.
+ * *flags* is reserved for now and must be zero.
+ *
+ * Return
+ * On success, number of bytes written to *buf*. On error, a
+ * negative value.
+ *
+ * **-EINVAL** if arguments invalid or **size** not a multiple
+ * of **sizeof**\ (**struct perf_branch_entry**\ ).
+ *
+ * **-ENOENT** if architecture does not support branch records.
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
@@ -5055,6 +5076,7 @@ union bpf_attr {
FN(get_func_ip), \
FN(get_attach_cookie), \
FN(task_pt_regs), \
+ FN(get_branch_snapshot), \
/* */
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
--
2.30.2
The typical way to access branch record (e.g. Intel LBR) is via hardware
perf_event. For CPUs with FREEZE_LBRS_ON_PMI support, PMI could capture
reliable LBR. On the other hand, LBR could also be useful in non-PMI
scenario. For example, in kretprobe or bpf fexit program, LBR could
provide a lot of information on what happened with the function. Add API
to use branch record for software use.
Note that, when the software event triggers, it is necessary to stop the
branch record hardware asap. Therefore, static_call is used to remove some
branch instructions in this process.
Signed-off-by: Song Liu <[email protected]>
---
arch/x86/events/intel/core.c | 26 +++++++++++++++++++++++---
arch/x86/events/intel/ds.c | 8 --------
arch/x86/events/perf_event.h | 10 ++++++++--
include/linux/perf_event.h | 26 ++++++++++++++++++++++++++
kernel/events/core.c | 2 ++
5 files changed, 59 insertions(+), 13 deletions(-)
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index ac6fd2dabf6a2..fe9bec93eb53b 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2143,7 +2143,7 @@ static __initconst const u64 knl_hw_cache_extra_regs
* However, there are some cases which may change PEBS status, e.g. PMI
* throttle. The PEBS_ENABLE should be updated where the status changes.
*/
-static void __intel_pmu_disable_all(void)
+static __always_inline void __intel_pmu_disable_all(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
@@ -2153,7 +2153,7 @@ static void __intel_pmu_disable_all(void)
intel_pmu_disable_bts();
}
-static void intel_pmu_disable_all(void)
+static __always_inline void intel_pmu_disable_all(void)
{
__intel_pmu_disable_all();
intel_pmu_pebs_disable_all();
@@ -2186,6 +2186,20 @@ static void intel_pmu_enable_all(int added)
__intel_pmu_enable_all(added, false);
}
+static int
+intel_pmu_snapshot_branch_stack(struct perf_branch_snapshot *br_snapshot)
+{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+ intel_pmu_disable_all();
+ intel_pmu_lbr_read();
+ memcpy(br_snapshot->entries, cpuc->lbr_entries,
+ sizeof(struct perf_branch_entry) * x86_pmu.lbr_nr);
+ br_snapshot->nr = x86_pmu.lbr_nr;
+ intel_pmu_enable_all(0);
+ return 0;
+}
+
/*
* Workaround for:
* Intel Errata AAK100 (model 26)
@@ -6283,9 +6297,15 @@ __init int intel_pmu_init(void)
x86_pmu.lbr_nr = 0;
}
- if (x86_pmu.lbr_nr)
+ if (x86_pmu.lbr_nr) {
pr_cont("%d-deep LBR, ", x86_pmu.lbr_nr);
+ /* only support branch_stack snapshot for perfmon >= v2 */
+ if (x86_pmu.disable_all == intel_pmu_disable_all)
+ static_call_update(perf_snapshot_branch_stack,
+ intel_pmu_snapshot_branch_stack);
+ }
+
intel_pmu_check_extra_regs(x86_pmu.extra_regs);
/* Support full width counters using alternative MSR range */
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 8647713276a73..8a832986578a9 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -1296,14 +1296,6 @@ void intel_pmu_pebs_enable_all(void)
wrmsrl(MSR_IA32_PEBS_ENABLE, cpuc->pebs_enabled);
}
-void intel_pmu_pebs_disable_all(void)
-{
- struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
-
- if (cpuc->pebs_enabled)
- wrmsrl(MSR_IA32_PEBS_ENABLE, 0);
-}
-
static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index e3ac05c97b5e5..171abbb359fe5 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1240,6 +1240,14 @@ static inline bool intel_pmu_has_bts(struct perf_event *event)
return intel_pmu_has_bts_period(event, hwc->sample_period);
}
+static __always_inline void intel_pmu_pebs_disable_all(void)
+{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+ if (cpuc->pebs_enabled)
+ wrmsrl(MSR_IA32_PEBS_ENABLE, 0);
+}
+
int intel_pmu_save_and_restart(struct perf_event *event);
struct event_constraint *
@@ -1314,8 +1322,6 @@ void intel_pmu_pebs_disable(struct perf_event *event);
void intel_pmu_pebs_enable_all(void);
-void intel_pmu_pebs_disable_all(void);
-
void intel_pmu_pebs_sched_task(struct perf_event_context *ctx, bool sched_in);
void intel_pmu_auto_reload_read(struct perf_event *event);
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index fe156a8170aa3..a368dfd754608 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -57,6 +57,7 @@ struct perf_guest_info_callbacks {
#include <linux/cgroup.h>
#include <linux/refcount.h>
#include <linux/security.h>
+#include <linux/static_call.h>
#include <asm/local.h>
struct perf_callchain_entry {
@@ -1612,4 +1613,29 @@ extern void __weak arch_perf_update_userpage(struct perf_event *event,
extern __weak u64 arch_perf_get_page_size(struct mm_struct *mm, unsigned long addr);
#endif
+/*
+ * Snapshot branch stack on software events.
+ *
+ * Branch stack can be very useful in understanding software events. For
+ * example, when a long function, e.g. sys_perf_event_open, returns an
+ * errno, it is not obvious why the function failed. Branch stack could
+ * provide very helpful information in this type of scenarios.
+ *
+ * On software event, it is necessary to stop the hardware branch recorder
+ * fast. Otherwise, the hardware register/buffer will be flushed with
+ * entries af the triggering event. Therefore, static call is used to
+ * stop the hardware recorder.
+ */
+enum {
+ PERF_MAX_BRANCH_SNAPSHOT = 32,
+};
+
+struct perf_branch_snapshot {
+ unsigned int nr;
+ struct perf_branch_entry entries[PERF_MAX_BRANCH_SNAPSHOT];
+};
+
+typedef int (perf_snapshot_branch_stack_t)(struct perf_branch_snapshot *);
+DECLARE_STATIC_CALL(perf_snapshot_branch_stack, perf_snapshot_branch_stack_t);
+
#endif /* _LINUX_PERF_EVENT_H */
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 011cc5069b7ba..d32a3cf37eb90 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -13437,3 +13437,5 @@ struct cgroup_subsys perf_event_cgrp_subsys = {
.threaded = true,
};
#endif /* CONFIG_CGROUP_PERF */
+
+DEFINE_STATIC_CALL_RET0(perf_snapshot_branch_stack, perf_snapshot_branch_stack_t);
--
2.30.2
This test uses bpf_get_branch_snapshot from a fexit program. The test uses
a target function (bpf_testmod_loop_test) and compares the record against
kallsyms. If there isn't enough record matching kallsyms, the test fails.
Signed-off-by: Song Liu <[email protected]>
---
.../selftests/bpf/bpf_testmod/bpf_testmod.c | 14 ++-
.../bpf/prog_tests/get_branch_snapshot.c | 101 ++++++++++++++++++
.../selftests/bpf/progs/get_branch_snapshot.c | 44 ++++++++
tools/testing/selftests/bpf/trace_helpers.c | 37 +++++++
tools/testing/selftests/bpf/trace_helpers.h | 5 +
5 files changed, 200 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
create mode 100644 tools/testing/selftests/bpf/progs/get_branch_snapshot.c
diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c
index 141d8da687d21..19635e57ff21a 100644
--- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c
@@ -13,6 +13,18 @@
DEFINE_PER_CPU(int, bpf_testmod_ksym_percpu) = 123;
+noinline int bpf_testmod_loop_test(int n)
+{
+ int i, sum = 0;
+
+ /* the primary goal of this test is to test LBR. Create a lot of
+ * branches in the function, so we can catch it easily.
+ */
+ for (i = 0; i < n; i++)
+ sum += i;
+ return sum;
+}
+
noinline ssize_t
bpf_testmod_test_read(struct file *file, struct kobject *kobj,
struct bin_attribute *bin_attr,
@@ -24,6 +36,7 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,
.len = len,
};
+ bpf_testmod_loop_test(101);
trace_bpf_testmod_test_read(current, &ctx);
return -EIO; /* always fail */
@@ -71,4 +84,3 @@ module_exit(bpf_testmod_exit);
MODULE_AUTHOR("Andrii Nakryiko");
MODULE_DESCRIPTION("BPF selftests module");
MODULE_LICENSE("Dual BSD/GPL");
-
diff --git a/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c b/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
new file mode 100644
index 0000000000000..03ffa5cdf9b09
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
@@ -0,0 +1,101 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2021 Facebook */
+#include <test_progs.h>
+#include "get_branch_snapshot.skel.h"
+
+static int *pfd_array;
+static int cpu_cnt;
+
+static int create_perf_events(void)
+{
+ struct perf_event_attr attr = {0};
+ int cpu;
+
+ /* create perf event */
+ attr.size = sizeof(attr);
+ attr.type = PERF_TYPE_RAW;
+ attr.config = 0x1b00;
+ attr.sample_type = PERF_SAMPLE_BRANCH_STACK;
+ attr.branch_sample_type = PERF_SAMPLE_BRANCH_KERNEL |
+ PERF_SAMPLE_BRANCH_USER | PERF_SAMPLE_BRANCH_ANY;
+
+ cpu_cnt = libbpf_num_possible_cpus();
+ pfd_array = malloc(sizeof(int) * cpu_cnt);
+ if (!pfd_array) {
+ cpu_cnt = 0;
+ return 1;
+ }
+
+ for (cpu = 0; cpu < cpu_cnt; cpu++) {
+ pfd_array[cpu] = syscall(__NR_perf_event_open, &attr,
+ -1, cpu, -1, PERF_FLAG_FD_CLOEXEC);
+ if (pfd_array[cpu] < 0)
+ break;
+ }
+
+ return cpu == 0;
+}
+
+static void close_perf_events(void)
+{
+ int cpu = 0;
+ int fd;
+
+ while (cpu++ < cpu_cnt) {
+ fd = pfd_array[cpu];
+ if (fd < 0)
+ break;
+ close(fd);
+ }
+ free(pfd_array);
+}
+
+void test_get_branch_snapshot(void)
+{
+ struct get_branch_snapshot *skel = NULL;
+ int err;
+
+ if (create_perf_events()) {
+ test__skip(); /* system doesn't support LBR */
+ goto cleanup;
+ }
+
+ skel = get_branch_snapshot__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "get_branch_snapshot__open_and_load"))
+ goto cleanup;
+
+ err = kallsyms_find("bpf_testmod_loop_test", &skel->bss->address_low);
+ if (!ASSERT_OK(err, "kallsyms_find"))
+ goto cleanup;
+
+ err = kallsyms_find_next("bpf_testmod_loop_test", &skel->bss->address_high);
+ if (!ASSERT_OK(err, "kallsyms_find_next"))
+ goto cleanup;
+
+ err = get_branch_snapshot__attach(skel);
+ if (!ASSERT_OK(err, "get_branch_snapshot__attach"))
+ goto cleanup;
+
+ /* trigger the program */
+ system("cat /sys/kernel/bpf_testmod > /dev/null 2>& 1");
+
+ if (skel->bss->total_entries < 16) {
+ /* too few entries for the hit/waste test */
+ test__skip();
+ goto cleanup;
+ }
+
+ ASSERT_GT(skel->bss->test1_hits, 1, "find_looptest_in_lbr");
+
+ /* Given we stop LBR in software, we will waste a few entries.
+ * But we should try to waste as few as possible entries. We are at
+ * about 11 on x86_64 systems.
+ * Add a check for < 15 so that we get heads-up when something
+ * changes and wastes too many entries.
+ */
+ ASSERT_LT(skel->bss->wasted_entries, 15, "check_wasted_entries");
+
+cleanup:
+ get_branch_snapshot__destroy(skel);
+ close_perf_events();
+}
diff --git a/tools/testing/selftests/bpf/progs/get_branch_snapshot.c b/tools/testing/selftests/bpf/progs/get_branch_snapshot.c
new file mode 100644
index 0000000000000..24a6e7a9c08ac
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/get_branch_snapshot.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2021 Facebook */
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+__u64 test1_hits = 0;
+__u64 address_low = 0;
+__u64 address_high = 0;
+int wasted_entries = 0;
+long total_entries = 0;
+
+struct perf_branch_entry entries[PERF_MAX_BRANCH_SNAPSHOT] = {};
+
+
+static inline bool in_range(__u64 val)
+{
+ return (val >= address_low) && (val < address_high);
+}
+
+SEC("fexit/bpf_testmod_loop_test")
+int BPF_PROG(test1, int n, int ret)
+{
+ long i;
+
+ total_entries = bpf_get_branch_snapshot(entries, sizeof(entries), 0);
+ total_entries /= sizeof(struct perf_branch_entry);
+
+ bpf_printk("total_entries %lu\n", total_entries);
+
+ for (i = 0; i < PERF_MAX_BRANCH_SNAPSHOT; i++) {
+ if (i >= total_entries)
+ break;
+ if (in_range(entries[i].from) && in_range(entries[i].to))
+ test1_hits++;
+ else if (!test1_hits)
+ wasted_entries++;
+ bpf_printk("i %d from %llx to %llx", i, entries[i].from,
+ entries[i].to);
+ }
+ return 0;
+}
diff --git a/tools/testing/selftests/bpf/trace_helpers.c b/tools/testing/selftests/bpf/trace_helpers.c
index e7a19b04d4eaf..5100a169b72b1 100644
--- a/tools/testing/selftests/bpf/trace_helpers.c
+++ b/tools/testing/selftests/bpf/trace_helpers.c
@@ -1,4 +1,5 @@
// SPDX-License-Identifier: GPL-2.0
+#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -117,6 +118,42 @@ int kallsyms_find(const char *sym, unsigned long long *addr)
return err;
}
+/* find the address of the next symbol of the same type, this can be used
+ * to determine the end of a function.
+ */
+int kallsyms_find_next(const char *sym, unsigned long long *addr)
+{
+ char type, found_type, name[500];
+ unsigned long long value;
+ bool found = false;
+ int err = 0;
+ FILE *f;
+
+ f = fopen("/proc/kallsyms", "r");
+ if (!f)
+ return -EINVAL;
+
+ while (fscanf(f, "%llx %c %499s%*[^\n]\n", &value, &type, name) > 0) {
+ /* Different types of symbols in kernel modules are mixed
+ * in /proc/kallsyms. Only return the next matching type.
+ * Use tolower() for type so that 'T' matches 't'.
+ */
+ if (found && found_type == tolower(type)) {
+ *addr = value;
+ goto out;
+ }
+ if (strcmp(name, sym) == 0) {
+ found = true;
+ found_type = tolower(type);
+ }
+ }
+ err = -ENOENT;
+
+out:
+ fclose(f);
+ return err;
+}
+
void read_trace_pipe(void)
{
int trace_fd;
diff --git a/tools/testing/selftests/bpf/trace_helpers.h b/tools/testing/selftests/bpf/trace_helpers.h
index d907b445524d5..bc8ed86105d94 100644
--- a/tools/testing/selftests/bpf/trace_helpers.h
+++ b/tools/testing/selftests/bpf/trace_helpers.h
@@ -16,6 +16,11 @@ long ksym_get_addr(const char *name);
/* open kallsyms and find addresses on the fly, faster than load + search. */
int kallsyms_find(const char *sym, unsigned long long *addr);
+/* find the address of the next symbol, this can be used to determine the
+ * end of a function
+ */
+int kallsyms_find_next(const char *sym, unsigned long long *addr);
+
void read_trace_pipe(void);
ssize_t get_uprobe_offset(const void *addr, ssize_t base);
--
2.30.2
On Tue, Aug 31, 2021 at 7:01 PM Song Liu <[email protected]> wrote:
>
> Introduce bpf_get_branch_snapshot(), which allows tracing pogram to get
> branch trace from hardware (e.g. Intel LBR). To use the feature, the
> user need to create perf_event with proper branch_record filtering
> on each cpu, and then calls bpf_get_branch_snapshot in the bpf function.
> On Intel CPUs, VLBR event (raw event 0x1b00) can be use for this.
>
> Signed-off-by: Song Liu <[email protected]>
> ---
> include/uapi/linux/bpf.h | 22 +++++++++++++++++++
> kernel/bpf/trampoline.c | 3 ++-
> kernel/trace/bpf_trace.c | 40 ++++++++++++++++++++++++++++++++++
> tools/include/uapi/linux/bpf.h | 22 +++++++++++++++++++
> 4 files changed, 86 insertions(+), 1 deletion(-)
>
> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> index 791f31dd0abee..c986e6fad5bc0 100644
> --- a/include/uapi/linux/bpf.h
> +++ b/include/uapi/linux/bpf.h
> @@ -4877,6 +4877,27 @@ union bpf_attr {
> * Get the struct pt_regs associated with **task**.
> * Return
> * A pointer to struct pt_regs.
> + *
> + * long bpf_get_branch_snapshot(void *entries, u32 size, u64 flags)
> + * Description
> + * Get branch trace from hardware engines like Intel LBR. The
> + * branch trace is taken soon after the trigger point of the
> + * BPF program, so it may contain some entries after the
> + * trigger point. The user need to filter these entries
> + * accordingly.
> + *
> + * The data is stored as struct perf_branch_entry into output
> + * buffer *entries*. *size* is the size of *entries* in bytes.
> + * *flags* is reserved for now and must be zero.
> + *
> + * Return
> + * On success, number of bytes written to *buf*. On error, a
> + * negative value.
> + *
> + * **-EINVAL** if arguments invalid or **size** not a multiple
> + * of **sizeof**\ (**struct perf_branch_entry**\ ).
> + *
> + * **-ENOENT** if architecture does not support branch records.
> */
> #define __BPF_FUNC_MAPPER(FN) \
> FN(unspec), \
> @@ -5055,6 +5076,7 @@ union bpf_attr {
> FN(get_func_ip), \
> FN(get_attach_cookie), \
> FN(task_pt_regs), \
> + FN(get_branch_snapshot), \
> /* */
>
> /* integer value in 'imm' field of BPF_CALL instruction selects which helper
> diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
> index fe1e857324e66..39eaaff81953d 100644
> --- a/kernel/bpf/trampoline.c
> +++ b/kernel/bpf/trampoline.c
> @@ -10,6 +10,7 @@
> #include <linux/rcupdate_trace.h>
> #include <linux/rcupdate_wait.h>
> #include <linux/module.h>
> +#include <linux/static_call.h>
>
> /* dummy _ops. The verifier will operate on target program's ops. */
> const struct bpf_verifier_ops bpf_extension_verifier_ops = {
> @@ -526,7 +527,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
> }
>
> #define NO_START_TIME 1
> -static u64 notrace bpf_prog_start_time(void)
> +static __always_inline u64 notrace bpf_prog_start_time(void)
> {
> u64 start = NO_START_TIME;
>
> diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> index 8e2eb950aa829..a8ec3634a3329 100644
> --- a/kernel/trace/bpf_trace.c
> +++ b/kernel/trace/bpf_trace.c
> @@ -1017,6 +1017,44 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto_pe = {
> .arg1_type = ARG_PTR_TO_CTX,
> };
>
> +static DEFINE_PER_CPU(struct perf_branch_snapshot, bpf_perf_branch_snapshot);
> +
> +BPF_CALL_3(bpf_get_branch_snapshot, void *, buf, u32, size, u64, flags)
> +{
> +#ifndef CONFIG_X86
> + return -ENOENT;
nit: -EOPNOTSUPP probably makes more sense for this?
> +#else
> + static const u32 br_entry_size = sizeof(struct perf_branch_entry);
> + u32 to_copy;
> +
> + if (unlikely(flags))
> + return -EINVAL;
> +
> + if (!buf || (size % br_entry_size != 0))
> + return -EINVAL;
> +
> + static_call(perf_snapshot_branch_stack)(this_cpu_ptr(&bpf_perf_branch_snapshot));
First, you have four this_cpu_ptr(&bpf_perf_branch_snapshot)
invocations in this function, probably cleaner to store the pointer in
local variable?
But second, this still has the reentrancy problem, right? And further,
we copy the same LBR data twice (to per-cpu buffer and into
user-provided destination).
What if we change perf_snapshot_branch_stack signature to this:
int perf_snapshot_branch_stack(struct perf_branch_entry *entries, int
max_nr_entries);
with the semantics that it will copy only min(max_nr_entreis,
PERF_MAX_BRANCH_RECORDS) * sizeof(struct perf_branch_entry) bytes.
That way we can copy directly into a user-provided buffer with no
per-cpu storage. Of course, perf_snapshot_branch_stack will return
number of entries copied, either as return result, or if static calls
don't support that, as another int *nr_entries output argument.
> +
> + if (this_cpu_ptr(&bpf_perf_branch_snapshot)->nr == 0)
> + return -ENOENT;
> +
> + to_copy = this_cpu_ptr(&bpf_perf_branch_snapshot)->nr *
> + sizeof(struct perf_branch_entry);
> + to_copy = min_t(u32, size, to_copy);
> + memcpy(buf, this_cpu_ptr(&bpf_perf_branch_snapshot)->entries, to_copy);
> +
> + return to_copy;
> +#endif
> +}
> +
[...]
On Tue, Aug 31, 2021 at 7:01 PM Song Liu <[email protected]> wrote:
>
> This test uses bpf_get_branch_snapshot from a fexit program. The test uses
> a target function (bpf_testmod_loop_test) and compares the record against
> kallsyms. If there isn't enough record matching kallsyms, the test fails.
>
> Signed-off-by: Song Liu <[email protected]>
> ---
LGTM, few minor nits below
Acked-by: Andrii Nakryiko <[email protected]>
> .../selftests/bpf/bpf_testmod/bpf_testmod.c | 14 ++-
> .../bpf/prog_tests/get_branch_snapshot.c | 101 ++++++++++++++++++
> .../selftests/bpf/progs/get_branch_snapshot.c | 44 ++++++++
> tools/testing/selftests/bpf/trace_helpers.c | 37 +++++++
> tools/testing/selftests/bpf/trace_helpers.h | 5 +
> 5 files changed, 200 insertions(+), 1 deletion(-)
> create mode 100644 tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
> create mode 100644 tools/testing/selftests/bpf/progs/get_branch_snapshot.c
>
[...]
> +
> +void test_get_branch_snapshot(void)
> +{
> + struct get_branch_snapshot *skel = NULL;
> + int err;
> +
> + if (create_perf_events()) {
> + test__skip(); /* system doesn't support LBR */
> + goto cleanup;
> + }
> +
> + skel = get_branch_snapshot__open_and_load();
> + if (!ASSERT_OK_PTR(skel, "get_branch_snapshot__open_and_load"))
> + goto cleanup;
> +
> + err = kallsyms_find("bpf_testmod_loop_test", &skel->bss->address_low);
> + if (!ASSERT_OK(err, "kallsyms_find"))
> + goto cleanup;
> +
> + err = kallsyms_find_next("bpf_testmod_loop_test", &skel->bss->address_high);
> + if (!ASSERT_OK(err, "kallsyms_find_next"))
> + goto cleanup;
> +
> + err = get_branch_snapshot__attach(skel);
> + if (!ASSERT_OK(err, "get_branch_snapshot__attach"))
> + goto cleanup;
> +
> + /* trigger the program */
> + system("cat /sys/kernel/bpf_testmod > /dev/null 2>& 1");
ugh :( see prog_tests/module_attach.c, we can extract and reuse
trigger_module_test_read() and trigger_module_test_write()
> +
> + if (skel->bss->total_entries < 16) {
> + /* too few entries for the hit/waste test */
> + test__skip();
> + goto cleanup;
> + }
> +
[...]
> +SEC("fexit/bpf_testmod_loop_test")
> +int BPF_PROG(test1, int n, int ret)
> +{
> + long i;
> +
> + total_entries = bpf_get_branch_snapshot(entries, sizeof(entries), 0);
> + total_entries /= sizeof(struct perf_branch_entry);
> +
> + bpf_printk("total_entries %lu\n", total_entries);
> +
> + for (i = 0; i < PERF_MAX_BRANCH_SNAPSHOT; i++) {
> + if (i >= total_entries)
> + break;
> + if (in_range(entries[i].from) && in_range(entries[i].to))
> + test1_hits++;
> + else if (!test1_hits)
> + wasted_entries++;
> + bpf_printk("i %d from %llx to %llx", i, entries[i].from,
> + entries[i].to);
debug leftovers? this will be polluting trace_pipe unnecessarily; same
for above total_entries bpf_printk()
> + }
> + return 0;
> +}
> diff --git a/tools/testing/selftests/bpf/trace_helpers.c b/tools/testing/selftests/bpf/trace_helpers.c
> index e7a19b04d4eaf..5100a169b72b1 100644
> --- a/tools/testing/selftests/bpf/trace_helpers.c
> +++ b/tools/testing/selftests/bpf/trace_helpers.c
> @@ -1,4 +1,5 @@
> // SPDX-License-Identifier: GPL-2.0
> +#include <ctype.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> @@ -117,6 +118,42 @@ int kallsyms_find(const char *sym, unsigned long long *addr)
> return err;
> }
>
[...]
> On Aug 31, 2021, at 9:08 PM, Andrii Nakryiko <[email protected]> wrote:
>
> On Tue, Aug 31, 2021 at 7:01 PM Song Liu <[email protected]> wrote:
>>
>> This test uses bpf_get_branch_snapshot from a fexit program. The test uses
>> a target function (bpf_testmod_loop_test) and compares the record against
>> kallsyms. If there isn't enough record matching kallsyms, the test fails.
>>
>> Signed-off-by: Song Liu <[email protected]>
>> ---
>
> LGTM, few minor nits below
>
> Acked-by: Andrii Nakryiko <[email protected]>
>
>> .../selftests/bpf/bpf_testmod/bpf_testmod.c | 14 ++-
>> .../bpf/prog_tests/get_branch_snapshot.c | 101 ++++++++++++++++++
>> .../selftests/bpf/progs/get_branch_snapshot.c | 44 ++++++++
>> tools/testing/selftests/bpf/trace_helpers.c | 37 +++++++
>> tools/testing/selftests/bpf/trace_helpers.h | 5 +
>> 5 files changed, 200 insertions(+), 1 deletion(-)
>> create mode 100644 tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
>> create mode 100644 tools/testing/selftests/bpf/progs/get_branch_snapshot.c
>>
>
> [...]
>
>> +
>> +void test_get_branch_snapshot(void)
>> +{
>> + struct get_branch_snapshot *skel = NULL;
>> + int err;
>> +
>> + if (create_perf_events()) {
>> + test__skip(); /* system doesn't support LBR */
>> + goto cleanup;
>> + }
>> +
>> + skel = get_branch_snapshot__open_and_load();
>> + if (!ASSERT_OK_PTR(skel, "get_branch_snapshot__open_and_load"))
>> + goto cleanup;
>> +
>> + err = kallsyms_find("bpf_testmod_loop_test", &skel->bss->address_low);
>> + if (!ASSERT_OK(err, "kallsyms_find"))
>> + goto cleanup;
>> +
>> + err = kallsyms_find_next("bpf_testmod_loop_test", &skel->bss->address_high);
>> + if (!ASSERT_OK(err, "kallsyms_find_next"))
>> + goto cleanup;
>> +
>> + err = get_branch_snapshot__attach(skel);
>> + if (!ASSERT_OK(err, "get_branch_snapshot__attach"))
>> + goto cleanup;
>> +
>> + /* trigger the program */
>> + system("cat /sys/kernel/bpf_testmod > /dev/null 2>& 1");
>
> ugh :( see prog_tests/module_attach.c, we can extract and reuse
> trigger_module_test_read() and trigger_module_test_write()
Will fix.
>
>> +
>> + if (skel->bss->total_entries < 16) {
>> + /* too few entries for the hit/waste test */
>> + test__skip();
>> + goto cleanup;
>> + }
>> +
>
> [...]
>
>> +SEC("fexit/bpf_testmod_loop_test")
>> +int BPF_PROG(test1, int n, int ret)
>> +{
>> + long i;
>> +
>> + total_entries = bpf_get_branch_snapshot(entries, sizeof(entries), 0);
>> + total_entries /= sizeof(struct perf_branch_entry);
>> +
>> + bpf_printk("total_entries %lu\n", total_entries);
>> +
>> + for (i = 0; i < PERF_MAX_BRANCH_SNAPSHOT; i++) {
>> + if (i >= total_entries)
>> + break;
>> + if (in_range(entries[i].from) && in_range(entries[i].to))
>> + test1_hits++;
>> + else if (!test1_hits)
>> + wasted_entries++;
>> + bpf_printk("i %d from %llx to %llx", i, entries[i].from,
>> + entries[i].to);
>
> debug leftovers? this will be polluting trace_pipe unnecessarily; same
> for above total_entries bpf_printk()
Oops.. I added/removed it for every version, but forgot this time. Will fix
in v5.
Thanks,
Song
> On Aug 31, 2021, at 9:02 PM, Andrii Nakryiko <[email protected]> wrote:
>
> On Tue, Aug 31, 2021 at 7:01 PM Song Liu <[email protected]> wrote:
>>
>> Introduce bpf_get_branch_snapshot(), which allows tracing pogram to get
>> branch trace from hardware (e.g. Intel LBR). To use the feature, the
>> user need to create perf_event with proper branch_record filtering
>> on each cpu, and then calls bpf_get_branch_snapshot in the bpf function.
>> On Intel CPUs, VLBR event (raw event 0x1b00) can be use for this.
>>
>> Signed-off-by: Song Liu <[email protected]>
>> ---
>> include/uapi/linux/bpf.h | 22 +++++++++++++++++++
>> kernel/bpf/trampoline.c | 3 ++-
>> kernel/trace/bpf_trace.c | 40 ++++++++++++++++++++++++++++++++++
>> tools/include/uapi/linux/bpf.h | 22 +++++++++++++++++++
>> 4 files changed, 86 insertions(+), 1 deletion(-)
>>
>> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
>> index 791f31dd0abee..c986e6fad5bc0 100644
>> --- a/include/uapi/linux/bpf.h
>> +++ b/include/uapi/linux/bpf.h
>> @@ -4877,6 +4877,27 @@ union bpf_attr {
>> * Get the struct pt_regs associated with **task**.
>> * Return
>> * A pointer to struct pt_regs.
>> + *
>> + * long bpf_get_branch_snapshot(void *entries, u32 size, u64 flags)
>> + * Description
>> + * Get branch trace from hardware engines like Intel LBR. The
>> + * branch trace is taken soon after the trigger point of the
>> + * BPF program, so it may contain some entries after the
>> + * trigger point. The user need to filter these entries
>> + * accordingly.
>> + *
>> + * The data is stored as struct perf_branch_entry into output
>> + * buffer *entries*. *size* is the size of *entries* in bytes.
>> + * *flags* is reserved for now and must be zero.
>> + *
>> + * Return
>> + * On success, number of bytes written to *buf*. On error, a
>> + * negative value.
>> + *
>> + * **-EINVAL** if arguments invalid or **size** not a multiple
>> + * of **sizeof**\ (**struct perf_branch_entry**\ ).
>> + *
>> + * **-ENOENT** if architecture does not support branch records.
>> */
>> #define __BPF_FUNC_MAPPER(FN) \
>> FN(unspec), \
>> @@ -5055,6 +5076,7 @@ union bpf_attr {
>> FN(get_func_ip), \
>> FN(get_attach_cookie), \
>> FN(task_pt_regs), \
>> + FN(get_branch_snapshot), \
>> /* */
>>
>> /* integer value in 'imm' field of BPF_CALL instruction selects which helper
>> diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
>> index fe1e857324e66..39eaaff81953d 100644
>> --- a/kernel/bpf/trampoline.c
>> +++ b/kernel/bpf/trampoline.c
>> @@ -10,6 +10,7 @@
>> #include <linux/rcupdate_trace.h>
>> #include <linux/rcupdate_wait.h>
>> #include <linux/module.h>
>> +#include <linux/static_call.h>
>>
>> /* dummy _ops. The verifier will operate on target program's ops. */
>> const struct bpf_verifier_ops bpf_extension_verifier_ops = {
>> @@ -526,7 +527,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
>> }
>>
>> #define NO_START_TIME 1
>> -static u64 notrace bpf_prog_start_time(void)
>> +static __always_inline u64 notrace bpf_prog_start_time(void)
>> {
>> u64 start = NO_START_TIME;
>>
>> diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
>> index 8e2eb950aa829..a8ec3634a3329 100644
>> --- a/kernel/trace/bpf_trace.c
>> +++ b/kernel/trace/bpf_trace.c
>> @@ -1017,6 +1017,44 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto_pe = {
>> .arg1_type = ARG_PTR_TO_CTX,
>> };
>>
>> +static DEFINE_PER_CPU(struct perf_branch_snapshot, bpf_perf_branch_snapshot);
>> +
>> +BPF_CALL_3(bpf_get_branch_snapshot, void *, buf, u32, size, u64, flags)
>> +{
>> +#ifndef CONFIG_X86
>> + return -ENOENT;
>
> nit: -EOPNOTSUPP probably makes more sense for this?
I had -EOPNOTSUPP in earlier version. But bpf_read_branch_records uses
-ENOENT, so I updated here in v4. I guess -ENOENT also makes sense? I
won't insist if you think -EOPNOTSUPP is better.
>
>> +#else
>> + static const u32 br_entry_size = sizeof(struct perf_branch_entry);
>> + u32 to_copy;
>> +
>> + if (unlikely(flags))
>> + return -EINVAL;
>> +
>> + if (!buf || (size % br_entry_size != 0))
>> + return -EINVAL;
>> +
>> + static_call(perf_snapshot_branch_stack)(this_cpu_ptr(&bpf_perf_branch_snapshot));
>
> First, you have four this_cpu_ptr(&bpf_perf_branch_snapshot)
> invocations in this function, probably cleaner to store the pointer in
> local variable?
>
> But second, this still has the reentrancy problem, right? And further,
> we copy the same LBR data twice (to per-cpu buffer and into
> user-provided destination).
>
> What if we change perf_snapshot_branch_stack signature to this:
>
> int perf_snapshot_branch_stack(struct perf_branch_entry *entries, int
> max_nr_entries);
>
> with the semantics that it will copy only min(max_nr_entreis,
> PERF_MAX_BRANCH_RECORDS) * sizeof(struct perf_branch_entry) bytes.
> That way we can copy directly into a user-provided buffer with no
> per-cpu storage. Of course, perf_snapshot_branch_stack will return
> number of entries copied, either as return result, or if static calls
> don't support that, as another int *nr_entries output argument.
I like this idea. Once we get feedback from Peter, I will change this
in v5.
Thanks,
Song
On Wed, Sep 1, 2021 at 8:41 AM Song Liu <[email protected]> wrote:
>
>
>
> > On Aug 31, 2021, at 9:02 PM, Andrii Nakryiko <[email protected]> wrote:
> >
> > On Tue, Aug 31, 2021 at 7:01 PM Song Liu <[email protected]> wrote:
> >>
> >> Introduce bpf_get_branch_snapshot(), which allows tracing pogram to get
> >> branch trace from hardware (e.g. Intel LBR). To use the feature, the
> >> user need to create perf_event with proper branch_record filtering
> >> on each cpu, and then calls bpf_get_branch_snapshot in the bpf function.
> >> On Intel CPUs, VLBR event (raw event 0x1b00) can be use for this.
> >>
> >> Signed-off-by: Song Liu <[email protected]>
> >> ---
> >> include/uapi/linux/bpf.h | 22 +++++++++++++++++++
> >> kernel/bpf/trampoline.c | 3 ++-
> >> kernel/trace/bpf_trace.c | 40 ++++++++++++++++++++++++++++++++++
> >> tools/include/uapi/linux/bpf.h | 22 +++++++++++++++++++
> >> 4 files changed, 86 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> >> index 791f31dd0abee..c986e6fad5bc0 100644
> >> --- a/include/uapi/linux/bpf.h
> >> +++ b/include/uapi/linux/bpf.h
> >> @@ -4877,6 +4877,27 @@ union bpf_attr {
> >> * Get the struct pt_regs associated with **task**.
> >> * Return
> >> * A pointer to struct pt_regs.
> >> + *
> >> + * long bpf_get_branch_snapshot(void *entries, u32 size, u64 flags)
> >> + * Description
> >> + * Get branch trace from hardware engines like Intel LBR. The
> >> + * branch trace is taken soon after the trigger point of the
> >> + * BPF program, so it may contain some entries after the
> >> + * trigger point. The user need to filter these entries
> >> + * accordingly.
> >> + *
> >> + * The data is stored as struct perf_branch_entry into output
> >> + * buffer *entries*. *size* is the size of *entries* in bytes.
> >> + * *flags* is reserved for now and must be zero.
> >> + *
> >> + * Return
> >> + * On success, number of bytes written to *buf*. On error, a
> >> + * negative value.
> >> + *
> >> + * **-EINVAL** if arguments invalid or **size** not a multiple
> >> + * of **sizeof**\ (**struct perf_branch_entry**\ ).
> >> + *
> >> + * **-ENOENT** if architecture does not support branch records.
> >> */
> >> #define __BPF_FUNC_MAPPER(FN) \
> >> FN(unspec), \
> >> @@ -5055,6 +5076,7 @@ union bpf_attr {
> >> FN(get_func_ip), \
> >> FN(get_attach_cookie), \
> >> FN(task_pt_regs), \
> >> + FN(get_branch_snapshot), \
> >> /* */
> >>
> >> /* integer value in 'imm' field of BPF_CALL instruction selects which helper
> >> diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
> >> index fe1e857324e66..39eaaff81953d 100644
> >> --- a/kernel/bpf/trampoline.c
> >> +++ b/kernel/bpf/trampoline.c
> >> @@ -10,6 +10,7 @@
> >> #include <linux/rcupdate_trace.h>
> >> #include <linux/rcupdate_wait.h>
> >> #include <linux/module.h>
> >> +#include <linux/static_call.h>
> >>
> >> /* dummy _ops. The verifier will operate on target program's ops. */
> >> const struct bpf_verifier_ops bpf_extension_verifier_ops = {
> >> @@ -526,7 +527,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
> >> }
> >>
> >> #define NO_START_TIME 1
> >> -static u64 notrace bpf_prog_start_time(void)
> >> +static __always_inline u64 notrace bpf_prog_start_time(void)
> >> {
> >> u64 start = NO_START_TIME;
> >>
> >> diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> >> index 8e2eb950aa829..a8ec3634a3329 100644
> >> --- a/kernel/trace/bpf_trace.c
> >> +++ b/kernel/trace/bpf_trace.c
> >> @@ -1017,6 +1017,44 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto_pe = {
> >> .arg1_type = ARG_PTR_TO_CTX,
> >> };
> >>
> >> +static DEFINE_PER_CPU(struct perf_branch_snapshot, bpf_perf_branch_snapshot);
> >> +
> >> +BPF_CALL_3(bpf_get_branch_snapshot, void *, buf, u32, size, u64, flags)
> >> +{
> >> +#ifndef CONFIG_X86
> >> + return -ENOENT;
> >
> > nit: -EOPNOTSUPP probably makes more sense for this?
>
> I had -EOPNOTSUPP in earlier version. But bpf_read_branch_records uses
> -ENOENT, so I updated here in v4. I guess -ENOENT also makes sense? I
> won't insist if you think -EOPNOTSUPP is better.
Hm... ok, I guess consistency takes priority, let's keep -ENOENT then.
>
> >
> >> +#else
> >> + static const u32 br_entry_size = sizeof(struct perf_branch_entry);
> >> + u32 to_copy;
> >> +
> >> + if (unlikely(flags))
> >> + return -EINVAL;
> >> +
> >> + if (!buf || (size % br_entry_size != 0))
> >> + return -EINVAL;
> >> +
> >> + static_call(perf_snapshot_branch_stack)(this_cpu_ptr(&bpf_perf_branch_snapshot));
> >
> > First, you have four this_cpu_ptr(&bpf_perf_branch_snapshot)
> > invocations in this function, probably cleaner to store the pointer in
> > local variable?
> >
> > But second, this still has the reentrancy problem, right? And further,
> > we copy the same LBR data twice (to per-cpu buffer and into
> > user-provided destination).
> >
> > What if we change perf_snapshot_branch_stack signature to this:
> >
> > int perf_snapshot_branch_stack(struct perf_branch_entry *entries, int
> > max_nr_entries);
> >
> > with the semantics that it will copy only min(max_nr_entreis,
> > PERF_MAX_BRANCH_RECORDS) * sizeof(struct perf_branch_entry) bytes.
> > That way we can copy directly into a user-provided buffer with no
> > per-cpu storage. Of course, perf_snapshot_branch_stack will return
> > number of entries copied, either as return result, or if static calls
> > don't support that, as another int *nr_entries output argument.
>
> I like this idea. Once we get feedback from Peter, I will change this
> in v5.
Sounds good, thanks!
>
> Thanks,
> Song
>