Received-SPF: pass (google.com: domain of linux-kernel+bounces-170309-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Precedence: bulk
MIME-Version: 1.0
References: <20240504003006.3303334-1-andrii@kernel.org> <20240504003006.3303334-3-andrii@kernel.org>
 <2024050439-janitor-scoff-be04@gregkh> <CAEf4BzZ6CaMrqRR1Rah7=HnTpU5-zw5HUnSH9NWCzAZZ55ZXFQ@mail.gmail.com>
 <ZjjiFnNRbwsMJ3Gj@x1> <CAM9d7cgvCB8CBFGhMB_-4tCm6+jzoPBNg4CR7AEyMNo8pF9QKg@mail.gmail.com>
In-Reply-To: <CAM9d7cgvCB8CBFGhMB_-4tCm6+jzoPBNg4CR7AEyMNo8pF9QKg@mail.gmail.com>
From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Date: Mon, 6 May 2024 11:51:34 -0700
Message-ID: <CAEf4Bzb8E7wzwBn+cx-XAW0ofEqemeuZoawHTFoTc-jK1azasA@mail.gmail.com>
Subject: Re: [PATCH 2/5] fs/procfs: implement efficient VMA querying API for /proc/<pid>/maps
To: Namhyung Kim <namhyung@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>, Jiri Olsa <jolsa@kernel.org>, Ian Rogers <irogers@google.com>, 
	Greg KH <gregkh@linuxfoundation.org>, Andrii Nakryiko <andrii@kernel.org>, 
	linux-fsdevel@vger.kernel.org, brauner@kernel.org, viro@zeniv.linux.org.uk, 
	akpm@linux-foundation.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, 
	linux-mm@kvack.org, =?UTF-8?Q?Daniel_M=C3=BCller?= <deso@posteo.net>, 
	"linux-perf-use." <linux-perf-users@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, May 6, 2024 at 11:05=E2=80=AFAM Namhyung Kim <namhyung@kernel.org> =
wrote:
>
> Hello,
>
> On Mon, May 6, 2024 at 6:58=E2=80=AFAM Arnaldo Carvalho de Melo <acme@ker=
nel.org> wrote:
> >
> > On Sat, May 04, 2024 at 02:50:31PM -0700, Andrii Nakryiko wrote:
> > > On Sat, May 4, 2024 at 8:28=E2=80=AFAM Greg KH <gregkh@linuxfoundatio=
n.org> wrote:
> > > > On Fri, May 03, 2024 at 05:30:03PM -0700, Andrii Nakryiko wrote:
> > > > > Note also, that fetching VMA name (e.g., backing file path, or sp=
ecial
> > > > > hard-coded or user-provided names) is optional just like build ID=
 If
> > > > > user sets vma_name_size to zero, kernel code won't attempt to ret=
rieve
> > > > > it, saving resources.
> >
> > > > > Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
> >
> > > > Where is the userspace code that uses this new api you have created=
?
> >
> > > So I added a faithful comparison of existing /proc/<pid>/maps vs new
> > > ioctl() API to solve a common problem (as described above) in patch
> > > #5. The plan is to put it in mentioned blazesym library at the very
> > > least.
> > >
> > > I'm sure perf would benefit from this as well (cc'ed Arnaldo and
> > > linux-perf-user), as they need to do stack symbolization as well.
>
> I think the general use case in perf is different.  This ioctl API is gre=
at
> for live tracing of a single (or a small number of) process(es).  And
> yes, perf tools have those tracing use cases too.  But I think the
> major use case of perf tools is system-wide profiling.

The intended use case is also a system-wide profiling, but I haven't
heard that opening a file per process is a big bottleneck or a
limitation, tbh.

>
> For system-wide profiling, you need to process samples of many
> different processes at a high frequency.  Now perf record doesn't
> process them and just save it for offline processing (well, it does
> at the end to find out build-ID but it can be omitted).
>
> Doing it online is possible (like perf top) but it would add more
> overhead during the profiling.  And we cannot move processing
> or symbolization to the end of profiling because some (short-
> lived) tasks can go away.

We do have some setups where we install a BPF program that monitors
process exit and mmap() events and emits (proactively) VMA
information. It's not applicable everywhere, and in some setups (like
Oculus case) we just accept that short-lived processes will be missed
at the expense of less interruption, simpler and less privileged
"agents" doing profiling and address resolution logic.

So the problem space, as can be seen, is pretty vast and varied, and
there is no single API that would serve all the needs perfectly.

>
> Also it should support perf report (offline) on data from a
> different kernel or even a different machine.

We fetch build ID (and resolve file offset) and offload actual
symbolization to a dedicated fleet of servers, whenever possible. We
don't yet do it for kernel stack traces, but we are moving in this
direction (and there are their own problems with /proc/kallsyms being
text-based, listing everything, and pretty big all in itself; but
that's a separate topic).

>
> So it saves the memory map of processes and symbolizes
> the stack trace with it later.  Of course it needs to be updated
> as the memory map changes and that's why it tracks mmap
> or similar syscalls with PERF_RECORD_MMAP[2] records.
>
> A problem with this approach is to get the initial state of all
> (or a target for non-system-wide mode) existing processes.
> We call it synthesizing, and read /proc/PID/maps to generate
> the mmap records.
>
> I think the below comment from Arnaldo talked about how
> we can improve the synthesizing (which is sequential access
> to proc maps) using BPF.

Yep. We can also benchmark using this new ioctl() to fetch a full set
of VMAs, it might still be good enough.

>
> Thanks,
> Namhyung
>

[...]