Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com>
Date:   Thu, 5 Dec 2019 09:33:02 -0300
To:     Ivan Babrou <ivan@cloudflare.com>
Cc:     linux-kernel <linux-kernel@vger.kernel.org>,
        kernel-team <kernel-team@cloudflare.com>,
        Jiri Olsa <jolsa@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Namhyung Kim <namhyung@kernel.org>, sashal@kernel.org,
        Kenton Varda <kenton@cloudflare.com>
Subject: Re: perf not picking up symbols for namespaced processes
Message-ID: <20191205123302.GA25750@kernel.org>
References: <CABWYdi1ZKR=jmKnjoJTik08Q9uJBvyZ4W0C29iPiUJ5ef1obvw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CABWYdi1ZKR=jmKnjoJTik08Q9uJBvyZ4W0C29iPiUJ5ef1obvw@mail.gmail.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Em Wed, Dec 04, 2019 at 07:46:10PM -0800, Ivan Babrou escreveu:
> We have a service that forks a child process in a namespace-based
> sandbox where the mount namespace is intentionally designed to reflect
> a totally empty filesystem. Our use case is very similar to Chrome's
> sandbox, for example, but on a server. Within the sandbox, not even
> the service's own binary is present in the mount namespace.
> 
> Process tree looks like this:
> 
> $ sudo pstree -psc 63989
> edgeworker(63989)─┬─edgeworker/sbox(255716)─┬─edgeworker/zygt(255718)
>                    │                         ├─{edgeworker/sbox}(255719)
>                    │                         ├─{edgeworker/sbox}(255720)
>                    │                         ├─{edgeworker/sbox}(255721)
>                    ├─edgeworker/stry(5803)
>                    ├─edgeworker/stry(63990)
>                    ├─edgeworker/stry(106218)
>                    ├─edgeworker/stry(191905)
>                    ├─edgeworker/stry(255695)
>                    ├─edgeworker/supr(255717)
> 
> Here sbox processes do actual work living in an empty mount namespaces
> and stry is a helper process for error reporting. All tasks come from
> the same binary that lives in the root mount namespace, launched by
> systemd.
> 
> During "perf script" run on a trace obtained from the system there are
> these possible outcomes:
> 
> 1. The first pid to be processed is a non-namespaced helper and
> symbols are present.
> 2. The first pid is not found and symbols are present.
> 3. The first pid is a sandboxed task and symbols are missing.
> 
> Symbols are missing, because "perf script" tries to jump into an empty
> sandbox and find a binary there, when in fact it lives outside:
> 
> getcwd("/state/home/ivan", 4096)        = 17
> open("/proc/self/ns/mnt", O_RDONLY)     = 5
> open("/proc/255719/ns/mnt", O_RDONLY)   = 6
> setns(6, CLONE_NEWNS)                   = 0
> stat("/usr/local/bin/edgeworker", 0x7ffedb9b3ca0) = -1 ENOENT (No such
> file or directory)
> 
> In the second outcome we don't have a PID to figure out the namespace
> to jump into, so this doesn't happen. It's a good fallback, but it was
> a bit confusing during debugging.
> 
> It's not entirely clear to me why sometimes a helper PID is picked,
> even though it's not the first sample in the recorded trace (at least
> not in the output). This happens deterministically, or at least
> appears so. In my process tree it's 255695.
> 
> I think perf should try to fallback to the default namespace to look
> up symbols if they are not found inside to cover our case. Relevant
> piece of logic is here:

That should work for your use case, as you're sure that looking up by
pathname only will find, outside the namespace, the binary you want.

Even with pathname based looukups being fragile, it works for your
usecase, so please consider providing a patch for such fallback,
together with a pr_debug() or even pr_warning() if this don't get too
noisy, to warn the user.

- Arnaldo
 
> * https://elixir.free-electrons.com/linux/v5.4.1/source/tools/perf/util/dso.c#L520

-- 

- Arnaldo