Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= <toke@redhat.com>
To:     Daniel Borkmann <daniel@iogearbox.net>,
        Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc:     Jiri Olsa <jolsa@kernel.org>, lkml <linux-kernel@vger.kernel.org>,
        netdev@vger.kernel.org, bpf@vger.kernel.org,
        Ingo Molnar <mingo@kernel.org>,
        Namhyung Kim <namhyung@kernel.org>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Peter Zijlstra <a.p.zijlstra@chello.nl>,
        Michael Petlan <mpetlan@redhat.com>,
        Jesper Dangaard Brouer <brouer@redhat.com>,
        Martin KaFai Lau <kafai@fb.com>,
        Song Liu <songliubraving@fb.com>, Yonghong Song <yhs@fb.com>,
        Andrii Nakryiko <andriin@fb.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>, kubakici@wp.pl
Subject: Re: [PATCH bpf v3] bpftool: Allow to link libbpf dynamically
In-Reply-To: <f6e8f6d2-6155-3b20-9975-81e29e460915@iogearbox.net>
References: <20191128160712.1048793-1-toke@redhat.com> <f6e8f6d2-6155-3b20-9975-81e29e460915@iogearbox.net>
Date:   Fri, 29 Nov 2019 15:00:40 +0100
Message-ID: <87a78evl2v.fsf@toke.dk>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Daniel Borkmann <daniel@iogearbox.net> writes:

> On 11/28/19 5:07 PM, Toke H=C3=B8iland-J=C3=B8rgensen wrote:
>> From: Jiri Olsa <jolsa@kernel.org>
>>=20
>> Currently we support only static linking with kernel's libbpf
>> (tools/lib/bpf). This patch adds LIBBPF_DYNAMIC compile variable
>> that triggers libbpf detection and bpf dynamic linking:
>>=20
>>    $ make -C tools/bpf/bpftool make LIBBPF_DYNAMIC=3D1
>>=20
>> If libbpf is not installed, build (with LIBBPF_DYNAMIC=3D1) stops with:
>>=20
>>    $ make -C tools/bpf/bpftool LIBBPF_DYNAMIC=3D1
>>      Auto-detecting system features:
>>      ...                        libbfd: [ on  ]
>>      ...        disassembler-four-args: [ on  ]
>>      ...                          zlib: [ on  ]
>>      ...                        libbpf: [ OFF ]
>>=20
>>    Makefile:102: *** Error: No libbpf devel library found, please instal=
l libbpf-devel or libbpf-dev.
>>=20
>> Adding LIBBPF_DIR compile variable to allow linking with
>> libbpf installed into specific directory:
>>=20
>>    $ make -C tools/lib/bpf/ prefix=3D/tmp/libbpf/ install_lib install_he=
aders
>>    $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=3D1 LIBBPF_DIR=3D/tmp/lib=
bpf/
>>=20
>> It might be needed to clean build tree first because features
>> framework does not detect the change properly:
>>=20
>>    $ make -C tools/build/feature clean
>>    $ make -C tools/bpf/bpftool/ clean
>>=20
>> Since bpftool uses bits of libbpf that are not exported as public API in
>> the .so version, we also pass in libbpf.a to the linker, which allows it=
 to
>> pick up the private functions from the static library without having to
>> expose them as ABI.
>>=20
>> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
>> Signed-off-by: Toke H=C3=B8iland-J=C3=B8rgensen <toke@redhat.com>
>> ---
>> v3:
>>    - Keep $(LIBBPF) in $LIBS, and just add -lbpf on top
>>    - Fix typo in error message
>> v2:
>>    - Pass .a file to linker when dynamically linking, so bpftool can use
>>      private functions from libbpf without exposing them as API.
>>=20
>>   tools/bpf/bpftool/Makefile | 34 ++++++++++++++++++++++++++++++++++
>>   1 file changed, 34 insertions(+)
>>=20
>> diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
>> index 39bc6f0f4f0b..15052dcaa39b 100644
>> --- a/tools/bpf/bpftool/Makefile
>> +++ b/tools/bpf/bpftool/Makefile
>> @@ -1,6 +1,15 @@
>>   # SPDX-License-Identifier: GPL-2.0-only
>> +# LIBBPF_DYNAMIC to enable libbpf dynamic linking.
>> +
>>   include ../../scripts/Makefile.include
>>   include ../../scripts/utilities.mak
>> +include ../../scripts/Makefile.arch
>> +
>> +ifeq ($(LP64), 1)
>> +  libdir_relative =3D lib64
>> +else
>> +  libdir_relative =3D lib
>> +endif
>>  =20
>>   ifeq ($(srctree),)
>>   srctree :=3D $(patsubst %/,%,$(dir $(CURDIR)))
>> @@ -63,6 +72,19 @@ RM ?=3D rm -f
>>   FEATURE_USER =3D .bpftool
>>   FEATURE_TESTS =3D libbfd disassembler-four-args reallocarray zlib
>>   FEATURE_DISPLAY =3D libbfd disassembler-four-args zlib
>> +ifdef LIBBPF_DYNAMIC
>> +  FEATURE_TESTS   +=3D libbpf
>> +  FEATURE_DISPLAY +=3D libbpf
>> +
>> +  # for linking with debug library run:
>> +  # make LIBBPF_DYNAMIC=3D1 LIBBPF_DIR=3D/opt/libbpf
>
> The Makefile already has BPF_DIR which points right now to
> '$(srctree)/tools/lib/bpf/' and LIBBPF_PATH for the final one and
> where $(LIBBPF_PATH)libbpf.a is expected to reside. Can't we improve
> the Makefile to reuse and work with these instead of adding yet
> another LIBBPF_DIR var which makes future changes in this area more
> confusing? The libbpf build spills out libbpf.{a,so*} by default
> anyway.

I see what you mean; however, LIBBPF_DIR is meant to be specifically an
override for the dynamic library, not just the path to libbpf.

Would it be less confusing to overload the LIBBPF_DYNAMIC variable
instead? I.e.,

make LIBBPF_DYNAMIC=3D1

would dynamically link against the libbpf installed in the system, but

make LIBBPF_DYNAMIC=3D/opt/libbpf

would override that and link against whatever is in /opt/libbpf instead?
WDYT?

> Was wondering whether we could drop LIBBPF_DYNAMIC altogether and have
> some sort of auto detection, but given for perf the `make
> LIBBPF_DYNAMIC=3D1` option was just applied to perf tree it's probably
> better to stay consistent plus static linking would stay as-is for
> preferred method for bpftool, so that part seems fine.

When adding LIBBPF_DYNAMIC in a packaging script, we *want* the build to
fail if it doesn't work, instead of just silently falling back to a
statically linked version. Also, for something in the kernel tree like
bpftool, I think it makes more sense to default to the in-tree version
and make dynamic linking explicitly opt-in.

>> +  ifdef LIBBPF_DIR
>> +    LIBBPF_CFLAGS  :=3D -I$(LIBBPF_DIR)/include
>> +    LIBBPF_LDFLAGS :=3D -L$(LIBBPF_DIR)/$(libdir_relative)
>> +    FEATURE_CHECK_CFLAGS-libbpf  :=3D $(LIBBPF_CFLAGS)
>> +    FEATURE_CHECK_LDFLAGS-libbpf :=3D $(LIBBPF_LDFLAGS)
>> +  endif
>> +endif
>>  =20
>>   check_feat :=3D 1
>>   NON_CHECK_FEAT_TARGETS :=3D clean uninstall doc doc-clean doc-install =
doc-uninstall
>> @@ -88,6 +110,18 @@ ifeq ($(feature-reallocarray), 0)
>>   CFLAGS +=3D -DCOMPAT_NEED_REALLOCARRAY
>>   endif
>>  =20
>> +ifdef LIBBPF_DYNAMIC
>> +  ifeq ($(feature-libbpf), 1)
>> +    # bpftool uses non-exported functions from libbpf, so just add the =
dynamic
>> +    # version of libbpf and let the linker figure it out
>> +    LIBS    :=3D -lbpf $(LIBS)
>
> Seems okay as a workaround for bpftool and avoids getting into the
> realm of declaring libbpf as another half-baked netlink library if
> we'd have exposed these. Ideally the netlink symbols shouldn't be
> needed at all from libbpf, but I presume the rationale back then was
> that given it's used internally in libbpf for some of the APIs and was
> needed in bpftool's net subcommand as well later on, it avoided
> duplicating the code given statically linked and both are in-tree
> anyway.

Yeah, I do think it's a little odd that bpftool is using "private" parts
of libbpf, but since we can solve it like this I think that is OK.

-Toke