Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
From:   Ivan Babrou <ivan@cloudflare.com>
Date:   Wed, 4 Dec 2019 19:46:10 -0800
Message-ID: <CABWYdi1ZKR=jmKnjoJTik08Q9uJBvyZ4W0C29iPiUJ5ef1obvw@mail.gmail.com>
Subject: perf not picking up symbols for namespaced processes
To:     linux-kernel <linux-kernel@vger.kernel.org>
Cc:     kernel-team <kernel-team@cloudflare.com>,
        Jiri Olsa <jolsa@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Namhyung Kim <namhyung@kernel.org>, sashal@kernel.org,
        Kenton Varda <kenton@cloudflare.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

We have a service that forks a child process in a namespace-based
sandbox where the mount namespace is intentionally designed to reflect
a totally empty filesystem. Our use case is very similar to Chrome's
sandbox, for example, but on a server. Within the sandbox, not even
the service's own binary is present in the mount namespace.

Process tree looks like this:

$ sudo pstree -psc 63989
edgeworker(63989)=E2=94=80=E2=94=AC=E2=94=80edgeworker/sbox(255716)=E2=94=
=80=E2=94=AC=E2=94=80edgeworker/zygt(255718)
                   =E2=94=82                         =E2=94=9C=E2=94=80{edg=
eworker/sbox}(255719)
                   =E2=94=82                         =E2=94=9C=E2=94=80{edg=
eworker/sbox}(255720)
                   =E2=94=82                         =E2=94=9C=E2=94=80{edg=
eworker/sbox}(255721)
                   =E2=94=9C=E2=94=80edgeworker/stry(5803)
                   =E2=94=9C=E2=94=80edgeworker/stry(63990)
                   =E2=94=9C=E2=94=80edgeworker/stry(106218)
                   =E2=94=9C=E2=94=80edgeworker/stry(191905)
                   =E2=94=9C=E2=94=80edgeworker/stry(255695)
                   =E2=94=9C=E2=94=80edgeworker/supr(255717)

Here sbox processes do actual work living in an empty mount namespaces
and stry is a helper process for error reporting. All tasks come from
the same binary that lives in the root mount namespace, launched by
systemd.

During "perf script" run on a trace obtained from the system there are
these possible outcomes:

1. The first pid to be processed is a non-namespaced helper and
symbols are present.
2. The first pid is not found and symbols are present.
3. The first pid is a sandboxed task and symbols are missing.

Symbols are missing, because "perf script" tries to jump into an empty
sandbox and find a binary there, when in fact it lives outside:

getcwd("/state/home/ivan", 4096)        =3D 17
open("/proc/self/ns/mnt", O_RDONLY)     =3D 5
open("/proc/255719/ns/mnt", O_RDONLY)   =3D 6
setns(6, CLONE_NEWNS)                   =3D 0
stat("/usr/local/bin/edgeworker", 0x7ffedb9b3ca0) =3D -1 ENOENT (No such
file or directory)

In the second outcome we don't have a PID to figure out the namespace
to jump into, so this doesn't happen. It's a good fallback, but it was
a bit confusing during debugging.

It's not entirely clear to me why sometimes a helper PID is picked,
even though it's not the first sample in the recorded trace (at least
not in the output). This happens deterministically, or at least
appears so. In my process tree it's 255695.

I think perf should try to fallback to the default namespace to look
up symbols if they are not found inside to cover our case. Relevant
piece of logic is here:

* https://elixir.free-electrons.com/linux/v5.4.1/source/tools/perf/util/dso=
.c#L520