Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 29 Dec 2020 12:15:04 -0300
From:   Arnaldo Carvalho de Melo <acme@kernel.org>
To:     Song Liu <songliubraving@fb.com>
Cc:     lkml <linux-kernel@vger.kernel.org>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "alexander.shishkin@linux.intel.com" 
        <alexander.shishkin@linux.intel.com>,
        "namhyung@kernel.org" <namhyung@kernel.org>,
        "mark.rutland@arm.com" <mark.rutland@arm.com>,
        "jolsa@redhat.com" <jolsa@redhat.com>,
        Kernel Team <Kernel-team@fb.com>
Subject: Re: [PATCH v6 3/4] perf-stat: enable counting events for BPF programs
Message-ID: <20201229151504.GI521329@kernel.org>
References: <20201228174054.907740-1-songliubraving@fb.com>
 <20201228174054.907740-4-songliubraving@fb.com>
 <20201228201159.GF521329@kernel.org>
 <6CB86649-9A1B-4585-8E1F-611F25935041@fb.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <6CB86649-9A1B-4585-8E1F-611F25935041@fb.com>
Precedence: bulk

Em Mon, Dec 28, 2020 at 11:43:25PM +0000, Song Liu escreveu:
> 
> 
> > On Dec 28, 2020, at 12:11 PM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > 
> > Em Mon, Dec 28, 2020 at 09:40:53AM -0800, Song Liu escreveu:
> >> Introduce perf-stat -b option, which counts events for BPF programs, like:
> >> 
> >> [root@localhost ~]# ~/perf stat -e ref-cycles,cycles -b 254 -I 1000
> >>     1.487903822            115,200      ref-cycles
> >>     1.487903822             86,012      cycles
> >>     2.489147029             80,560      ref-cycles
> >>     2.489147029             73,784      cycles
> >>     3.490341825             60,720      ref-cycles
> >>     3.490341825             37,797      cycles
> >>     4.491540887             37,120      ref-cycles
> >>     4.491540887             31,963      cycles
> >> 
> >> The example above counts cycles and ref-cycles of BPF program of id 254.
> >> This is similar to bpftool-prog-profile command, but more flexible.
> >> 
> >> perf-stat -b creates per-cpu perf_event and loads fentry/fexit BPF
> >> programs (monitor-progs) to the target BPF program (target-prog). The
> >> monitor-progs read perf_event before and after the target-prog, and
> >> aggregate the difference in a BPF map. Then the user space reads data
> >> from these maps.
> >> 
> >> A new struct bpf_counter is introduced to provide common interface that
> >> uses BPF programs/maps to count perf events.
> > 
> > Segfaulting here:
> > 
> > [root@five ~]# bpftool prog  | grep tracepoint
> > 110: tracepoint  name syscall_unaugme  tag 57cd311f2e27366b  gpl
> > 111: tracepoint  name sys_enter_conne  tag 3555418ac9476139  gpl
> > 112: tracepoint  name sys_enter_sendt  tag bc7fcadbaf7b8145  gpl
> > 113: tracepoint  name sys_enter_open  tag 0e59c3ac2bea5280  gpl
> > 114: tracepoint  name sys_enter_opena  tag 0baf443610f59837  gpl
> > 115: tracepoint  name sys_enter_renam  tag 24664e4aca62d7fa  gpl
> > 116: tracepoint  name sys_enter_renam  tag 20093e51a8634ebb  gpl
> > 117: tracepoint  name sys_enter  tag 0bc3fc9d11754ba1  gpl
> > 118: tracepoint  name sys_exit  tag 29c7ae234d79bd5c  gpl
> > [root@five ~]#
> > [root@five ~]# gdb perf
> > GNU gdb (GDB) Fedora 10.1-2.fc33
> > Reading symbols from perf...
> > (gdb) run stat -e instructions,cycles -b 113 -I 1000
> > Starting program: /root/bin/perf stat -e instructions,cycles -b 113 -I 1000
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > libbpf: elf: skipping unrecognized data section(9) .eh_frame
> > libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame
> > libbpf: elf: skipping unrecognized data section(9) .eh_frame
> > libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame
> > 
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x000000000058d55b in bpf_program_profiler__read (evsel=0xc612c0) at util/bpf_counter.c:217
> > 217			reading_map_fd = bpf_map__fd(skel->maps.accum_readings);
> > (gdb) bt
> > #0  0x000000000058d55b in bpf_program_profiler__read (evsel=0xc612c0) at util/bpf_counter.c:217
> > #1  0x0000000000000000 in ?? ()
> > (gdb)
> > 
> > [acme@five perf]$ clang -v |& head -2
> > clang version 11.0.0 (Fedora 11.0.0-2.fc33)
> > Target: x86_64-unknown-linux-gnu
> > [acme@five perf]$
> > 
> > Do you need any extra info?
> 
> Hmm... I am not able to reproduce this. I am trying to setup an environment similar
> to fc33 (clang 11, etc.). Does this segfault every time, and on all programs? 

I'll try it with a BPF proggie attached to a kprobes, but here is
something else I noticed:

[root@five perf]# export PYTHONPATH=/tmp/build/perf/python
[root@five perf]# tools/perf/python/twatch.py
Traceback (most recent call last):
  File "/home/acme/git/perf/tools/perf/python/twatch.py", line 9, in <module>
    import perf
ImportError: /tmp/build/perf/python/perf.cpython-39-x86_64-linux-gnu.so: undefined symbol: bpf_counter__destroy
[root@five perf]# perf test python
19: 'import perf' in python                                         : FAILED!
[root@five perf]# perf test -v python
19: 'import perf' in python                                         :
--- start ---
test child forked, pid 3198864
python usage test: "echo "import sys ; sys.path.append('/tmp/build/perf/python'); import perf" | '/usr/bin/python3' "
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /tmp/build/perf/python/perf.cpython-39-x86_64-linux-gnu.so: undefined symbol: bpf_counter__destroy
test child finished with -1
---- end ----
'import perf' in python: FAILED!
[root@five perf]#

This should be trivial, I hope, just add the new object file to
tools/perf/util/python-ext-sources, then do a 'perf test python', if it
fails, use 'perf test -v python' to see what is preventing the python
binding from loading.

- Arnaldo