Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220905114209.8389-1-adrian.hunter@intel.com>
In-Reply-To: <20220905114209.8389-1-adrian.hunter@intel.com>
From:   Namhyung Kim <namhyung@kernel.org>
Date:   Tue, 6 Sep 2022 10:50:31 -0700
Message-ID: <CAM9d7cibk1OU95v56J15RG-zZmJ9TccUVZGAp=VHu7F3HwFY8g@mail.gmail.com>
Subject: Re: [PATCH V2] libperf evlist: Fix per-thread mmaps for
 multi-threaded targets
To:     Adrian Hunter <adrian.hunter@intel.com>
Cc:     Arnaldo Carvalho de Melo <acme@kernel.org>,
        Jiri Olsa <jolsa@redhat.com>, Ian Rogers <irogers@google.com>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        linux-perf-users <linux-perf-users@vger.kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Mon, Sep 5, 2022 at 4:42 AM Adrian Hunter <adrian.hunter@intel.com> wrote:
>
> The offending commit removed mmap_per_thread(), which did not consider
> the different set-output rules for per-thread mmaps i.e. in the per-thread
> case set-output is used for file descriptors of the same thread not the
> same cpu.
>
> This was not immediately noticed because it only happens with
> multi-threaded targets and we do not have a test for that yet.

Yeah, this is unfortunate.  I feel like I need to spend some time on it.

>
> Reinstate mmap_per_thread() expanding it to cover also system-wide per-cpu
> events i.e. to continue to allow the mixing of per-thread and per-cpu
> mmaps.
>
> Debug messages (with -vv) show the file descriptors that are opened with
> sys_perf_event_open. New debug messages are added (needs -vvv) that show
> also which file descriptors are mmapped and which are redirected with
> set-output.
>
> In the per-cpu case (cpu != -1) file descriptors for the same CPU are
> set-output to the first file descriptor for that CPU.
>
> In the per-thread case (cpu == -1) file descriptors for the same thread are
> set-output to the first file descriptor for that thread.
>
> Example (process 17489 has 2 threads):
>
>  Before (but with new debug prints):
>
>    $ perf record --no-bpf-event -vvv --per-thread -p 17489
>    <SNIP>
>    sys_perf_event_open: pid 17489  cpu -1  group_fd -1  flags 0x8 = 5
>    sys_perf_event_open: pid 17490  cpu -1  group_fd -1  flags 0x8 = 6
>    <SNIP>
>    libperf: idx 0: mmapping fd 5
>    libperf: idx 0: set output fd 6 -> 5
>    failed to mmap with 22 (Invalid argument)
>
>  After:
>
>    $ perf record --no-bpf-event -vvv --per-thread -p 17489
>    <SNIP>
>    sys_perf_event_open: pid 17489  cpu -1  group_fd -1  flags 0x8 = 5
>    sys_perf_event_open: pid 17490  cpu -1  group_fd -1  flags 0x8 = 6
>    <SNIP>
>    libperf: mmap_per_thread: nr cpu values (may include -1) 1 nr threads 2
>    libperf: idx 0: mmapping fd 5
>    libperf: idx 1: mmapping fd 6
>    <SNIP>
>    [ perf record: Woken up 2 times to write data ]
>    [ perf record: Captured and wrote 0.018 MB perf.data (15 samples) ]

It'd be nice if the example had 2 events so that it could check the
set-output rule actually worked.

Thanks,
Namhyung

>
> Per-cpu example (process 20341 has 2 threads, same as above):
>
>    $ perf record --no-bpf-event -vvv -p 20341
>    <SNIP>
>    sys_perf_event_open: pid 20341  cpu 0  group_fd -1  flags 0x8 = 5
>    sys_perf_event_open: pid 20342  cpu 0  group_fd -1  flags 0x8 = 6
>    sys_perf_event_open: pid 20341  cpu 1  group_fd -1  flags 0x8 = 7
>    sys_perf_event_open: pid 20342  cpu 1  group_fd -1  flags 0x8 = 8
>    sys_perf_event_open: pid 20341  cpu 2  group_fd -1  flags 0x8 = 9
>    sys_perf_event_open: pid 20342  cpu 2  group_fd -1  flags 0x8 = 10
>    sys_perf_event_open: pid 20341  cpu 3  group_fd -1  flags 0x8 = 11
>    sys_perf_event_open: pid 20342  cpu 3  group_fd -1  flags 0x8 = 12
>    sys_perf_event_open: pid 20341  cpu 4  group_fd -1  flags 0x8 = 13
>    sys_perf_event_open: pid 20342  cpu 4  group_fd -1  flags 0x8 = 14
>    sys_perf_event_open: pid 20341  cpu 5  group_fd -1  flags 0x8 = 15
>    sys_perf_event_open: pid 20342  cpu 5  group_fd -1  flags 0x8 = 16
>    sys_perf_event_open: pid 20341  cpu 6  group_fd -1  flags 0x8 = 17
>    sys_perf_event_open: pid 20342  cpu 6  group_fd -1  flags 0x8 = 18
>    sys_perf_event_open: pid 20341  cpu 7  group_fd -1  flags 0x8 = 19
>    sys_perf_event_open: pid 20342  cpu 7  group_fd -1  flags 0x8 = 20
>    <SNIP>
>    libperf: mmap_per_cpu: nr cpu values 8 nr threads 2
>    libperf: idx 0: mmapping fd 5
>    libperf: idx 0: set output fd 6 -> 5
>    libperf: idx 1: mmapping fd 7
>    libperf: idx 1: set output fd 8 -> 7
>    libperf: idx 2: mmapping fd 9
>    libperf: idx 2: set output fd 10 -> 9
>    libperf: idx 3: mmapping fd 11
>    libperf: idx 3: set output fd 12 -> 11
>    libperf: idx 4: mmapping fd 13
>    libperf: idx 4: set output fd 14 -> 13
>    libperf: idx 5: mmapping fd 15
>    libperf: idx 5: set output fd 16 -> 15
>    libperf: idx 6: mmapping fd 17
>    libperf: idx 6: set output fd 18 -> 17
>    libperf: idx 7: mmapping fd 19
>    libperf: idx 7: set output fd 20 -> 19
>    <SNIP>
>    [ perf record: Woken up 7 times to write data ]
>    [ perf record: Captured and wrote 0.020 MB perf.data (17 samples) ]
>
> Fixes: ae4f8ae16a07 ("libperf evlist: Allow mixing per-thread and per-cpu mmaps")
> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>