Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Jiri Olsa <olsajiri@gmail.com>
Date:   Wed, 14 Jun 2023 16:14:19 +0200
To:     Steven Rostedt <rostedt@goodmis.org>
Cc:     Yonghong Song <yhs@meta.com>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Andrii Nakryiko <andrii@kernel.org>,
        lkml <linux-kernel@vger.kernel.org>,
        linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org,
        Andrii Nakryiko <andrii.nakryiko@gmail.com>,
        Jackie Liu <liu.yun@linux.dev>
Subject: Re: [PATCHv2] ftrace: Show all functions with addresses in
 available_filter_functions_addrs
Message-ID: <ZInLO4/xly/f+Zk3@krava>
References: <20230611130029.1202298-1-jolsa@kernel.org>
 <53a11f31-256d-e7bc-eca5-597571076dc5@meta.com>
 <20230611225407.3e9b8ad2@gandalf.local.home>
 <20230611225754.01350a50@gandalf.local.home>
 <d5ffd64c-65b7-e28c-b8ee-0d2ff9dcd78b@meta.com>
 <20230612110222.50c254f3@gandalf.local.home>
 <ZId/UL/iujOdgel+@krava>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZId/UL/iujOdgel+@krava>
Precedence: bulk

On Mon, Jun 12, 2023 at 10:25:55PM +0200, Jiri Olsa wrote:
> On Mon, Jun 12, 2023 at 11:02:22AM -0400, Steven Rostedt wrote:
> > On Mon, 12 Jun 2023 07:49:53 -0700
> > Yonghong Song <yhs@meta.com> wrote:
> > 
> > > I am actually interested in how available_filter_functions_addrs
> > > will be used. For example, bpf_program__attach_kprobe_multi_opts()
> > > can already take addresses from kallsyms. How to use
> > > available_filter_functions_addrs to facilitate kprobe_multi?
> 
> the problem is that we need to do 2 passes:
> 
>  - through available_filter_functions and find out if the function is traceable
>  - through /proc/kallsyms to get the address for traceable function
> 
> having available_filter_functions symbols together with addresses allow
> us to skip the kallsyms step
> 
> and we are ok with the address in available_filter_functions_addr not being the
> function entry, because kprobe_multi uses fprobe and that handles both entry and
> patch-site address properly
> 
> > > Do we need to change kernel APIs? It would be great at least we
> > > got a RFC patch to answer these questions.
> > 
> > I agree, having that information would also be useful to me.
> > 
> > Jiri? Andrii?
> 
> so we have 2 interfaces how to create kprobe_multi link:
> 
>   a) passing symbols to kernel
> 
>      1) user gathers symbols and need to ensure that they are
>         trace-able -> pass through available_filter_functions file
> 
>      2) kernel takes those symbols and translates them to addresses
>         through kallsyms api
> 
>      3) addresses are passed to fprobe/ftrace through:
> 
>          register_fprobe_ips
>          -> ftrace_set_filter_ips
> 
>   b) passing addresses to kernel
> 
>      1) user gathers symbols and needs to ensure that they are
>         trace-able -> pass through available_filter_functions file
> 
>      2) user takes those symbols and translates them to addresses
>        through /proc/kallsyms
> 
>      3) addresses are passed to the kernel and kernel calls:
> 
>          register_fprobe_ips
>          -> ftrace_set_filter_ips
> 
> 
> The new available_filter_functions_addrs file helps us with option b),
> because we can make 'b 1' and 'b 2' in one step - while filtering traceable
> functions, we get the address directly.
> 
> I tested the new available_filter_functions_addrs changes with some hacked
> selftest changes, you can check it in here [1].
> 
> I assume Jackie Liu will send new version of her patchset [2] based on this
> new available_filter_functions_addrs file.
> 
> I think we should have these changes coming together and add some perf
> measurements from before and after to make the benefit apparent.

FYI I did some perf meassurements and the speedup is not substantial :-\

looks like the symbols resolving to addresses we do in kernel for kprobe_multi
link is more faster/cheaper than I thought 

but still there is 'some' speedup and we will get rid of the extra
/proc/kallsyms parsing, so I think it's still worth it to have the
new file


base:

 Performance counter stats for './test_progs -n 103/1':

       103,423,103      cycles:u
    79,279,231,029      cycles:k
    79,382,694,663      cycles

      18.627593589 seconds time elapsed

       0.025999000 seconds user
      18.323855000 seconds sys


with fix:

 Performance counter stats for './test_progs -n 103/1':

       126,659,572      cycles:u
    77,951,768,179      cycles:k
    78,078,467,451      cycles

      18.651464273 seconds time elapsed

       0.025001000 seconds user
      18.243828000 seconds sys


jirka