Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755282AbdL2I6e (ORCPT ); Fri, 29 Dec 2017 03:58:34 -0500 Received: from mail-it0-f42.google.com ([209.85.214.42]:38930 "EHLO mail-it0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752430AbdL2I6d (ORCPT ); Fri, 29 Dec 2017 03:58:33 -0500 X-Google-Smtp-Source: ACJfBou6LqIqrwGy0hPMmZhogi7xeMJvscO28s7u0OJFq1WK9Ly5p9VddwzBZ4hDuJt8mQN3IlM30lSh8H7866Aqg7c= MIME-Version: 1.0 From: Joel Fernandes Date: Fri, 29 Dec 2017 00:58:31 -0800 Message-ID: Subject: ANNOUNCE: bpfd - a remote proxy daemon for executing bpf code (with corres. bcc changes) To: LKML , iovisor-dev@lists.iovisor.org Cc: Alexei Starovoitov , bgregg@netflix.com, bblanco@plumgrid.com, Alison Chaiken , "Cc: Android Kernel" , Mathieu Desnoyers , Josef Bacik , Yonghong Song , Steven Rostedt Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4885 Lines: 106 Hi Guys, I've been working on an idea I discussed with other kernel developers (Alexei, Josef etc) last LPC about how to make it easier to run bcc tools on remote systems. Use case ======== Run bcc tools on a remotely connected system without having to load the entire LLVM infrastructure onto the remote target and have to sync the kernel sources with it. On architecture such as ARM64 especially, its a bit more work if you were to run the tools directly on the target itself (local to the target) because LLVM and Python have to be cross-compiled for it (along with syncing of kernel sources which takes up space and needs to be kept sync'ed for correct operation). I believe Facebook also has some usecases where they want to run bcc tools on remote instances. Lastly this is also the way arm64 development normally happens, you cross build for it and typically the ARM64 embedded systems may not have much space for kernel sources and clang so its better some times if the tools are remote. All our kernel development for android is cross developed with the cross-toolchain running remotely as well. I am looking forward to collaborating with interested developers on this idea and getting more feedback about the design etc. I am also planning to talk about it next year during SCALE and OSPM. Implementation ============== To facilitate this, I started working on a daemon called bpfd which is executed on the remote target and listening for commands: https://github.com/joelagnel/bpfd The daemon does more than proxy the bpf syscall, there's several things like registering a kprobe with perf, and perf callbacks that need to be replicated. All this infrastructure is pretty much code complete in bpfd. Sample commands sent to bpfd are as follows: https://github.com/joelagnel/bpfd/blob/master/tests/TESTS ------------------------ ; Program opensnoop BPF_CREATE_MAP 1 8 40 10240 0 BPF_CREATE_MAP 4 4 4 2 0 BPF_PROG_LOAD 2 248 GPL 264721 eRdwAAAAAAC3AQAAAAAAAHsa+P8AA[...] BPF_PROG_LOAD 2 664 GPL 264721 vxYAAAAAAACFAAAADgAAAHsK+P8AA[...] ------------------------ Binary streams are communicated using base64 making it possible to keep interaction with binary simple. Several patches is written on the bcc side to be able to send these commands using a "remotes infrastructure", available in the branch at: https://github.com/joelagnel/bcc/commits/bcc-bpfd My idea was to keep the remote infrastructure as generic/plug-and-play as possible - so in the future its easy to add other remotes like networking. Currently I've adb (android bridge) remote and a shell remote: https://github.com/joelagnel/bcc/tree/bcc-bpfd/src/python/bcc/remote The shell remote is more of a "test" remote that simply forks bpfd and communicates with it over stdio. This makes the development quite easy. Status ====== What's working: - executing several bcc tools across process boundary using "shell" remote (bcc tools and bpfd both running on local x86 machine). - communication with remote arm64 android target using the "adb remote". But there are several issues to do with arm64 and bcc tools that I'm ironing out. Since my arm64 bcc hackery is a bit recent, I created a separate WIP branch here: https://github.com/joelagnel/bcc/tree/bcc-bpfd-arm64. I don't suspect these to be a major issue since I noticed some folks have been using bcc tools on arm64 already. Issues: - Since bcc is building with clang on x86 - the eBPF backend code is generated for x86. Although it loads fine on arm64, there seem several issues such as kprobe handler doesn't see arguments or return code correctly in opensnoop. This is (probably)easy to fix by just user telling bcc we're build for a certain architecture - but that would mean we carry code for each arch when building the bcc libraries and dynamically select the code path to run - than building for the C++ compiler's target architecture. - Some operations are quite slow, such as stackcount when the number of stack traces are a lot. Each stack trace is a key and and every key iterated is at a cost, which adds up. Maybe we can batch these up so that they're faster instead of making each key iteration a separate remote command/response? - Some tools read the ps table on the local host. This needs to be remotely proxied. - Provide mechanism to make bcc/clang build eBPF for arm64 (using a command line switch) ? - Design a generic parser mechanism to be added to all bcc tools to be able to pass which remote method to use, what the remote architecture is and what the path to the kernel sources are (for kprobes to work) Thanks a lot to Alexei for discussing ideas in conference and for all the great advice and help. Regards, - Joel PS: We also have some usecases where our Android networking daemon has hardcoded eBPF asm and our teams want to write them C and load the binary stream. It seems bpfd can be a good fit here as well.