Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Song Liu <songliubraving@fb.com>
To:     Arnaldo Carvalho de Melo <acme@kernel.org>
CC:     Peter Zijlstra <peterz@infradead.org>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Alexei Starovoitov <ast@kernel.org>,
        "Daniel Borkmann" <daniel@iogearbox.net>,
        Kernel Team <Kernel-team@fb.com>
Subject: Re: [PATCH perf,bpf 0/5] reveal invisible bpf programs
Thread-Topic: [PATCH perf,bpf 0/5] reveal invisible bpf programs
Thread-Index: AQHUgdQ11TRJMN8CIkSD/uQZUfyfEaVbiN6AgACRn4CABhB7AIAACnEAgAHTxwA=
Date:   Tue, 27 Nov 2018 19:21:41 +0000
Message-ID: <E268FE5F-E2CE-4266-8B1F-A134BFAC6462@fb.com>
References: <20181121195502.3259930-1-songliubraving@fb.com>
 <20181122093219.GK2131@hirez.programming.kicks-ass.net>
 <71189F83-A09F-4A03-95EC-694D37FD7675@fb.com>
 <20181126145004.GO2113@hirez.programming.kicks-ass.net>
 <20181126152726.GA5518@kernel.org>
In-Reply-To: <20181126152726.GA5518@kernel.org>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: fb.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-ID: <B969F263CF47A04DB154821CF196699F@namprd15.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 1ecf11a9-6dfc-43fb-a0a8-08d6549d8f97
X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Nov 2018 19:21:41.3931
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1517
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-11-27_16:,,
 signatures=0
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> On Nov 26, 2018, at 7:27 AM, Arnaldo Carvalho de Melo <acme@kernel.org> w=
rote:
>=20
> Em Mon, Nov 26, 2018 at 03:50:04PM +0100, Peter Zijlstra escreveu:
>> Now, I'm not saying this patch set is useless; but I'm saying most
>> people should not need this, and it is massive overkill for the needs of
>> most people.
>=20
> So, the comparision is sort of with kernel modules, that can come and go
> while you're profiling/tracing, if that happens, then samples, in post
> processing, are not resolvable, and that is the case for kernel modules
> right now. Sure, you're right, that doesn't happen so frequently, so
> nobody hollered (thankfully that is now verbotten ;-)) at us so far.
>=20
> You need to have the load-kernel-bin/unload-kernel-bin events recorded,
> and you need to somehow match those addresses to some symtab/src(for
> people that want to have src mixed up with assembly) and you need that
> jitted code, with timestamps of when it was loaded and it was unloaded.
>=20
> People doing post processing analysis of weird problems need all those
> details.
>=20
> Now I don't know how frequently those binary blobs gets loaded/unloaded
> in the brave new world of eBPF, but for completeness sake, we need those
> load/unload events and we need to grab a copy of the raw jitted binary,
> etc.

BPF programs get loaded/unloaded more often than kernel modules. This is=20
because BPF verifier makes sure BPF program will never crash the kernel,=20
thus BPF programs could often be debugged as user space apps.=20

Typical debug/tune process is like:

    perf record -- ./test_work
    perf report

    # make changes to bpf program and repeat

or even:

    perf record -o perf_1.data -- ./test_v1
    perf record -o perf_2.data -- ./test_v2
    perf record -o perf_3.data -- ./test_v3

    # perf report in 3 terminals, and compare the output

In such use cases, including all details in the perf.data is very helpful,=
=20
as /proc/kcore doesn't have details of the program when perf-report runs.=20

Thanks,
Song=