Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CA+khW7gh=vO8m-_SVnwWwj7kv+EDeUPcuWFqebf2Zmi9T_oEAQ@mail.gmail.com>
In-Reply-To: <CA+khW7gh=vO8m-_SVnwWwj7kv+EDeUPcuWFqebf2Zmi9T_oEAQ@mail.gmail.com>
From:   Song Liu <song@kernel.org>
Date:   Mon, 24 Jan 2022 23:07:45 -0800
Message-ID: <CAPhsuW7F4KritXPXixoPSw4zbCsqpfZaYBuw5BgD+KKXaoeGxg@mail.gmail.com>
Subject: Re: [Question] How to reliably get BuildIDs from bpf prog
To:     Hao Luo <haoluo@google.com>
Cc:     Alexei Starovoitov <ast@kernel.org>,
        Andrii Nakryiko <andrii@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Song Liu <songliubraving@fb.com>, Yonghong Song <yhs@fb.com>,
        Martin KaFai Lau <kafai@fb.com>,
        KP Singh <kpsingh@kernel.org>, bpf <bpf@vger.kernel.org>,
        open list <linux-kernel@vger.kernel.org>,
        Jiri Olsa <jolsa@kernel.org>,
        Blake Jones <blakejones@google.com>,
        Alexey Alexandrov <aalexand@google.com>,
        Namhyung Kim <namhyung@google.com>,
        Ian Rogers <irogers@google.com>,
        "pasha.tatashin@soleen.com" <pasha.tatashin@soleen.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Mon, Jan 24, 2022 at 2:43 PM Hao Luo <haoluo@google.com> wrote:
>
> Dear BPF experts,
>
> I'm working on collecting some kernel performance data using BPF
> tracing prog. Our performance profiling team wants to associate the
> data with user stack information. One of the requirements is to
> reliably get BuildIDs from bpf_get_stackid() and other similar helpers
> [1].
>
> As part of an early investigation, we found that there are a couple
> issues that make bpf_get_stackid() much less reliable than we'd like
> for our use:
>
> 1. The first page of many binaries (which contains the ELF headers and
> thus the BuildID that we need) is often not in memory. The failure of
> find_get_page() (called from build_id_parse()) is higher than we would
> want.

Our top use case of bpf_get_stack() is called from NMI, so there isn't
much we can do. Maybe it is possible to improve it by changing the
layout of the binary and the libraries? Specifically, if the text is
also in the first page, it is likely to stay in memory?

> 2. When anonymous huge pages are used to hold some regions of process
> text, build_id_parse() also fails to get a BuildID because
> vma->vm_file is NULL.

How did the text get in anonymous memory? I guess it is NOT from JIT?
We had a hack to use transparent huge page for application text. The
hack looks like:

"At run time, the application creates an 8MB temporary buffer and the
hot section of the executable memory is copied to it. The 8MB region in
the executable memory is then converted to a huge page (by way of an
mmap() to anonymous pages and an madvise() to create a huge page), the
data is copied back to it, and it is made executable again using
mprotect()."

If your case is the same (or similar), it can probably be fixed with
CONFIG_READ_ONLY_THP_FOR_FS, and modified user space.

Thanks,
Song