2015-06-12 19:27:24

by Arnaldo Carvalho de Melo

[permalink] [raw]
Subject: perf probe and structs

Hi Masami,

I tried somethig with perf probe today, namely to ask for a
variable that is a struct perf_event_attr to be collected after
the perf_event_open syscall copies it from userspace, and got this
message:

[root@zoo ~]# perf probe SYSC_perf_event_open:23 args
Failed to find 'args' in this function.
Error: Failed to add events.
[root@zoo ~]#

Ok, I guess it should be instead:

[root@zoo ~]# perf probe SYSC_perf_event_open:23 args
The 'args' variable is a struct, this is not supported yet.
Error: Failed to add events.
[root@zoo ~]#

:-)

That said, can't we just go ahead and collect sizeof(args) into the
buffer and let userspace cast the result to the right type, etc, i.e.
kinda like what is done now for a string.

Alexei, is this already possible with eBPF?

This is all in the context of:

[root@zoo ~]# perf trace -e perf_event_open &
[1] 20775
[root@zoo ~]# perf stat -e cycles usleep 1
7304.425 ( 0.519 ms): perf/20776 perf_event_open(attr_uptr: 0x222b220, pid: 20777, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3

Performance counter stats for 'usleep 1':

2,024,026 cycles

0.005277428 seconds time elapsed

[root@zoo ~]#

I want to decode that attr_uptr thing :-)

- Arnaldo


Subject: Re: perf probe and structs

On 2015/06/13 4:27, Arnaldo Carvalho de Melo wrote:
> Hi Masami,
>
> I tried somethig with perf probe today, namely to ask for a
> variable that is a struct perf_event_attr to be collected after
> the perf_event_open syscall copies it from userspace, and got this
> message:
>
> [root@zoo ~]# perf probe SYSC_perf_event_open:23 args
> Failed to find 'args' in this function.
> Error: Failed to add events.
> [root@zoo ~]#
>
> Ok, I guess it should be instead:
>
> [root@zoo ~]# perf probe SYSC_perf_event_open:23 args
> The 'args' variable is a struct, this is not supported yet.
> Error: Failed to add events.
> [root@zoo ~]#
>
> :-)

Hmm, I need to look into it. BTW, could you tried to run
`perf probe --vars SYSC_perf_event_open:23` ?


>
> That said, can't we just go ahead and collect sizeof(args) into the
> buffer and let userspace cast the result to the right type, etc, i.e.
> kinda like what is done now for a string.

I'd like to add dump data struct feature instead of dumping binary.
Or, at least suggesting which field you can specify :)

Thank you,


>
> Alexei, is this already possible with eBPF?
>
> This is all in the context of:
>
> [root@zoo ~]# perf trace -e perf_event_open &
> [1] 20775
> [root@zoo ~]# perf stat -e cycles usleep 1
> 7304.425 ( 0.519 ms): perf/20776 perf_event_open(attr_uptr: 0x222b220, pid: 20777, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3
>
> Performance counter stats for 'usleep 1':
>
> 2,024,026 cycles
>
> 0.005277428 seconds time elapsed
>
> [root@zoo ~]#
>
> I want to decode that attr_uptr thing :-)
>
> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>


--
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]

2015-06-13 02:58:09

by Alexei Starovoitov

[permalink] [raw]
Subject: Re: perf probe and structs

On 6/12/15 12:27 PM, Arnaldo Carvalho de Melo wrote:
> Alexei, is this already possible with eBPF?
> I want to decode that attr_uptr thing :-)

yes, it's already possible :)

Here is working example from our experimental c+python thingy:
#!/usr/bin/env python

from bpf import BPF
from subprocess import call

prog = """
#include <uapi/linux/ptrace.h>
#include <uapi/linux/perf_event.h>
int hello(struct pt_regs *ctx)
{
struct perf_event_attr attr = {};
bpf_probe_read(&attr, sizeof(attr), (void *) ctx->di);
char fmt[] = "type %x size %d config %d\\n";
bpf_trace_printk(fmt, sizeof(fmt), attr.type, attr.size, attr.config);
return 0;
}
"""
b = BPF(text=prog)
fn = b.load_func("hello", BPF.KPROBE)
BPF.attach_kprobe(fn, "SYSC_perf_event_open")
try:
call(["cat", "/sys/kernel/debug/tracing/trace_pipe"])
except KeyboardInterrupt:
pass

running above gives me output:
# ./example.py
perf_4.1.0-5544 [001] d.h3 3818.231428: : type 1 size 0 config 0
perf_4.1.0-5544 [001] d.h3 3818.231494: : type 0 size 112 config 0
perf_4.1.0-5544 [001] d.h3 3818.231530: : type 0 size 112 config 0
perf_4.1.0-5544 [001] d.h3 3818.231554: : type 0 size 112 config 0
perf_4.1.0-5544 [001] d.h3 3818.231564: : type 0 size 112 config 0

Subject: Re: perf probe and structs

On 2015/06/13 4:27, Arnaldo Carvalho de Melo wrote:
> Hi Masami,
>
> I tried somethig with perf probe today, namely to ask for a
> variable that is a struct perf_event_attr to be collected after
> the perf_event_open syscall copies it from userspace, and got this
> message:
>
> [root@zoo ~]# perf probe SYSC_perf_event_open:23 args
> Failed to find 'args' in this function.
> Error: Failed to add events.
> [root@zoo ~]#

BTW, would you sure that the args exists at the given place?
I couldn't find it in kernel/events/core.c.

And also, could you make sure to run below command to clarify
the existed variables and its valid ranges?
# perf probe -V --range SYSC_perf_event_open

For example, I got below outouts.

# perf probe -V SYSC_perf_event_open:23
Available variables at SYSC_perf_event_open:23
@<SYSC_perf_event_open+502>
int cpu
int group_fd
long unsigned int flags
pid_t pid
struct perf_event_attr attr
# perf probe SYSC_perf_event_open:23 attr
perf_event_attr exceeds max-bitwidth. Cut down to 64 bits.
Added new event:
probe:SYSC_perf_event_open (on SYSC_perf_event_open:23 with attr)

You can now use it in all perf tools, such as:

perf record -e probe:SYSC_perf_event_open -aR sleep 1

Of course, this is still not enough, I must fix this as you suggested.

>
> Ok, I guess it should be instead:
>
> [root@zoo ~]# perf probe SYSC_perf_event_open:23 args
> The 'args' variable is a struct, this is not supported yet.
> Error: Failed to add events.
> [root@zoo ~]#
>
> :-)
>
--
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]