Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= <toke@redhat.com>
To:     Luigi Rizzo <lrizzo@google.com>
Cc:     linux-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Greg KH <gregkh@linuxfoundation.org>,
        naveen.n.rao@linux.ibm.com, ardb@kernel.org,
        Luigi Rizzo <rizzo@iet.unipi.it>,
        Paolo Abeni <pabeni@redhat.com>, giuseppe.lettieri@unipi.it,
        Jesper Dangaard Brouer <hawk@kernel.org>, mingo@redhat.com,
        acme@kernel.org, Steven Rostedt <rostedt@goodmis.org>,
        peterz@infradead.org
Subject: Re: [PATCH v3 0/2] kstats: kernel metric collector
In-Reply-To: <CAMOZA0Lzf2r7rFvgBEWpf-B=wXvyED2CxfzuO7qUA_qVsNtL7g@mail.gmail.com>
References: <20200226135027.34538-1-lrizzo@google.com> <87ftexz93y.fsf@toke.dk> <CAMOZA0Lzf2r7rFvgBEWpf-B=wXvyED2CxfzuO7qUA_qVsNtL7g@mail.gmail.com>
Date:   Thu, 27 Feb 2020 00:11:38 +0100
Message-ID: <878skpx7th.fsf@toke.dk>
MIME-Version: 1.0
Content-Type: text/plain
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Luigi Rizzo <lrizzo@google.com> writes:

> - the runtime cost and complexity of hooking bpf code is still a bit
> unclear to me. kretprobe or tracepoints are expensive, I suppose that
> some lean hook replace register_kretprobe() may exist and the
> difference from inline annotations would be marginal (we'd still need
> to put in the hooks around the code we want to time, though, so it
> wouldn't be a pure bpf solution). Any pointers to this are welcome;
> Alexei mentioned fentry/fexit and bpf trampolines, but I haven't found
> an example that lets me do something equivalent to kretprobe (take a
> timestamp before and one after a function without explicit
> instrumentation)

As Alexei said, with fentry/fexit the overhead should be on par with
your example. This functionality is pretty new, though, so I can
understand why it's not obvious how to do things with it yet :)

I think the best place to look is currently in selftests/bpf in the
kernel sources. Grep for 'fexit' and 'fentry' in the progs/ subdir.
test_overhead.c and kfree_skb.c seem to have some examples you may be
able to work from.

> - I still see some huge differences in usability, and this is in my
> opinion one very big difference between the two approaches. The
> systems where data collection may be of interest are not necessarily
> accessible to developers with the skills to write custom bpf code, or
> load bpf modules (security policies may prevent that). One thing is to
> tell a sysadmin to run "echo trace foo >
> /sys/kernel/debug/kstats/_config" or "watch grep CPUS
> /sys/kernel/debug/kstats/bar", another one is to tell them to load a
> bpf program (or write their own one).

With BPF the solution for this is to distribute a tool that does all the
setup for the user. Basically the userspace equivalent of what you're
proposing to include into the kernel here. You can make this arbitrarily
user-friendly, up to and including having a GUI list all the functions
available in the running kernel and letting the user just click on the
one to measure :)

-Toke