Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Tue, 12 Apr 2022 17:44:00 +0200
From:   Jiri Olsa <olsajiri@gmail.com>
To:     Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc:     Jiri Olsa <jolsa@kernel.org>, Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Andrii Nakryiko <andrii@kernel.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Networking <netdev@vger.kernel.org>, bpf <bpf@vger.kernel.org>,
        lkml <linux-kernel@vger.kernel.org>,
        Martin KaFai Lau <kafai@fb.com>,
        Song Liu <songliubraving@fb.com>, Yonghong Song <yhs@fb.com>,
        John Fastabend <john.fastabend@gmail.com>,
        KP Singh <kpsingh@chromium.org>
Subject: Re: [RFC bpf-next 4/4] selftests/bpf: Add attach bench test
Message-ID: <YlWeQIUaqGnbg4K0@krava>
References: <20220407125224.310255-1-jolsa@kernel.org>
 <20220407125224.310255-5-jolsa@kernel.org>
 <CAEf4BzbE1n3Lie+tWTzN69RQUWgjxePorxRr9J8CuiQVUfy-kA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAEf4BzbE1n3Lie+tWTzN69RQUWgjxePorxRr9J8CuiQVUfy-kA@mail.gmail.com>
Precedence: bulk

On Mon, Apr 11, 2022 at 03:15:40PM -0700, Andrii Nakryiko wrote:

SNIP

> > +static int get_syms(char ***symsp, size_t *cntp)
> > +{
> > +       size_t cap = 0, cnt = 0, i;
> > +       char *name, **syms = NULL;
> > +       struct hashmap *map;
> > +       char buf[256];
> > +       FILE *f;
> > +       int err;
> > +
> > +       /*
> > +        * The available_filter_functions contains many duplicates,
> > +        * but other than that all symbols are usable in kprobe multi
> > +        * interface.
> > +        * Filtering out duplicates by using hashmap__add, which won't
> > +        * add existing entry.
> > +        */
> > +       f = fopen(DEBUGFS "available_filter_functions", "r");
> 
> I'm really curious how did you manage to attach to everything in
> available_filter_functions because when I'm trying to do that I fail.

the new code makes the differece ;-) so the main problem I could not
use available_filter_functions functions before were cases like:

  # cat available_filter_functions | grep sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall
  sys_ni_syscall

which when you try to resolve you'll find just one address:

  # cat /proc/kallsyms | egrep 'T sys_ni_syscall'
  ffffffff81170020 T sys_ni_syscall

this is caused by entries like:
    __SYSCALL(156, sys_ni_syscall)

when generating syscalls for given arch

this is handled by the new code by removing duplicates when
reading available_filter_functions


another case is the other way round, like with:

  # cat /proc/kallsyms | grep 't t_next'
  ffffffff8125c3f0 t t_next
  ffffffff8126a320 t t_next
  ffffffff81275de0 t t_next
  ffffffff8127efd0 t t_next
  ffffffff814d6660 t t_next

that has just one 'ftrace-able' instance:

  # cat available_filter_functions | grep '^t_next$'
  t_next

and this is handled by calling ftrace_location on address when
resolving symbols, to ensure each reasolved symbol lives in ftrace 

> available_filter_functions has a bunch of functions that should not be
> attachable (e.g., notrace functions). Look just at __bpf_tramp_exit:
> 
>   void notrace __bpf_tramp_exit(struct bpf_tramp_image *tr);
> 
> So first, curious what I am doing wrong or rather why it succeeds in
> your case ;)
> 
> But second, just wanted to plea to "fix" available_filter_functions to
> not list stuff that should not be attachable. Can you please take a
> look and checks what's going on there and why do we have notrace
> functions (and what else should *NOT* be there)?

yes, seems like a bug ;-) it's in available_filter_functions
but it does not have 'call __fentry__' at the entry..

I was going to check on that, because you brought that up before,
but did not get to it yet

> 
> 
> > +       if (!f)
> > +               return -EINVAL;
> > +
> > +       map = hashmap__new(symbol_hash, symbol_equal, NULL);
> > +       err = libbpf_get_error(map);
> > +       if (err)
> > +               goto error;
> > +
> 
> [...]
> 
> > +
> > +       attach_delta_ns = (attach_end_ns - attach_start_ns) / 1000000000.0;
> > +       detach_delta_ns = (detach_end_ns - detach_start_ns) / 1000000000.0;
> > +
> > +       fprintf(stderr, "%s: found %lu functions\n", __func__, cnt);
> > +       fprintf(stderr, "%s: attached in %7.3lfs\n", __func__, attach_delta_ns);
> > +       fprintf(stderr, "%s: detached in %7.3lfs\n", __func__, detach_delta_ns);
> > +
> > +       if (attach_delta_ns > 2.0)
> > +               PRINT_FAIL("attach time above 2 seconds\n");
> > +       if (detach_delta_ns > 2.0)
> > +               PRINT_FAIL("detach time above 2 seconds\n");
> 
> see my reply on the cover letter, any such "2 second" assumption are
> guaranteed to bite us. We've dealt with a lot of timing issues due to
> CI being slower and more unpredictable in terms of performance, I'd
> like to avoid dealing with one more case like that.

right, I'll remove the check

thanks,
jirka