Hi Masami:
I have been looking at perf-probe again and having a number of problems
with top of tree.
Here's the first one I have isolated:
$ perf probe -x /lib64/libc-2.18.so -a 'malloc size=%di'
Probe point 'malloc' not found.
Error: Failed to add events.
$ perf probe -x /lib64/libc-2.18.so -F | grep malloc
malloc
malloc@plt
malloc_atfork
malloc_check
malloc_consolidate
malloc_hook_ini
malloc_info
malloc_printerr
mallochook
ptmalloc_init
ptmalloc_init.part.8
ptmalloc_lock_all
ptmalloc_unlock_all
ptmalloc_unlock_all2
tr_mallochook
$ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di'
Probe point 'malloc' not found.
Error: Failed to add events.
A year ago (v3.12) this worked fine so I did a git bisect which points to:
commit fb7345bbf7fad9bf72ef63a19c707970b9685812
Author: Masami Hiramatsu <[email protected]>
Date: Thu Dec 26 05:41:53 2013 +0000
perf probe: Support basic dwarf-based operations on uprobe events
I have tried top of tree on Fedora 16, 18 and 20 with a variety of
kernels - and a variety of results. Reverting to
8a613d40e389b723fd5889ac8d4033ed4030be31 which is the commit before this
one and it works again.
David
Hi David,
(2014/12/30 13:21), David Ahern wrote:
> Hi Masami:
>
> I have been looking at perf-probe again and having a number of problems
> with top of tree.
>
> Here's the first one I have isolated:
>
> $ perf probe -x /lib64/libc-2.18.so -a 'malloc size=%di'
> Probe point 'malloc' not found.
> Error: Failed to add events.
It seems to fail when failed to open debuginfo.
I confirmed it on ubuntu 14.04 too.
I'll try to fix that.
Thank you for reporting!
>
> $ perf probe -x /lib64/libc-2.18.so -F | grep malloc
> malloc
> malloc@plt
> malloc_atfork
> malloc_check
> malloc_consolidate
> malloc_hook_ini
> malloc_info
> malloc_printerr
> mallochook
> ptmalloc_init
> ptmalloc_init.part.8
> ptmalloc_lock_all
> ptmalloc_unlock_all
> ptmalloc_unlock_all2
> tr_mallochook
>
> $ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di'
> Probe point 'malloc' not found.
> Error: Failed to add events.
>
> A year ago (v3.12) this worked fine so I did a git bisect which points to:
>
> commit fb7345bbf7fad9bf72ef63a19c707970b9685812
> Author: Masami Hiramatsu <[email protected]>
> Date: Thu Dec 26 05:41:53 2013 +0000
>
> perf probe: Support basic dwarf-based operations on uprobe events
>
> I have tried top of tree on Fedora 16, 18 and 20 with a variety of
> kernels - and a variety of results. Reverting to
> 8a613d40e389b723fd5889ac8d4033ed4030be31 which is the commit before this
> one and it works again.
>
> David
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
--
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
Fix to fall back to find a probe point in symbols if perf fails
to find it in debuginfo.
This can happen when the target function is an alias of another
function. Such alias doesn't have an entry in debuginfo but in
symbols.
David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6
name (strp) "malloc"
decl_file (data1) 25
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 81b5]
declaration (flag_present)
[ 8f58] formal_parameter
--
name (strp) "malloc"
decl_file (data1) 23
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 9f4a]
declaration (flag_present)
sibling (ref4) [ bb29]
...
-----
All these entires have no instances (all of them are declarations)
This is why the perf probe failed to find it in debuginfo.
However, there are some malloc instances in symbols.
-----
eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$
1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc
4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc
5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc
6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc
7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc
-----
As you an see, malloc and __libc_malloc have same address, and actually
__libc_malloc has an entry in debuginfo. So you can set up a probe on
__libc_malloc.
To fix this problem shortly, perf probe simply falls back to find probe
point(malloc) in symbols if it is not found in debuginfo.
Signed-off-by: Masami Hiramatsu <[email protected]>
Reported-by: David Ahern <[email protected]>
---
tools/perf/util/probe-event.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 28eb141..7f9b863 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
}
if (ntevs == 0) { /* No error but failed to find probe point. */
- pr_warning("Probe point '%s' not found.\n",
+ pr_warning("Probe point '%s' not found in debuginfo.\n",
synthesize_perf_probe_point(&pev->point));
- return -ENOENT;
+ if (need_dwarf)
+ return -ENOENT;
+ return 0;
}
/* Error path : ntevs < 0 */
pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);
Hi Masami,
On Wed, Dec 31, 2014 at 03:27:47PM +0900, Masami Hiramatsu wrote:
> Fix to fall back to find a probe point in symbols if perf fails
> to find it in debuginfo.
>
> This can happen when the target function is an alias of another
> function. Such alias doesn't have an entry in debuginfo but in
> symbols.
>
> David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
>
> I ensured the problem and deeper investigation discovers it.
> -----
> eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6
> name (strp) "malloc"
> decl_file (data1) 25
> decl_line (data2) 466
> prototyped (flag_present)
> type (ref4) [ 81b5]
> declaration (flag_present)
> [ 8f58] formal_parameter
> --
> name (strp) "malloc"
> decl_file (data1) 23
> decl_line (data2) 466
> prototyped (flag_present)
> type (ref4) [ 9f4a]
> declaration (flag_present)
> sibling (ref4) [ bb29]
> ...
> -----
> All these entires have no instances (all of them are declarations)
> This is why the perf probe failed to find it in debuginfo.
>
> However, there are some malloc instances in symbols.
> -----
> eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$
> 1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc
> 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc
> 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc
> 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc
> 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc
> -----
> As you an see, malloc and __libc_malloc have same address, and actually
> __libc_malloc has an entry in debuginfo. So you can set up a probe on
> __libc_malloc.
>
> To fix this problem shortly, perf probe simply falls back to find probe
> point(malloc) in symbols if it is not found in debuginfo.
Idealy, it'd be great if it can go to the definition. I think perf
only keeps a single symbol at a same address so if there're multiple
aliases at a specific address it might be missed even with this patch.
But that will be a rare case, so
Acked-by: Namhyung Kim <[email protected]>
Thanks,
Namhyung
>
> Signed-off-by: Masami Hiramatsu <[email protected]>
> Reported-by: David Ahern <[email protected]>
> ---
> tools/perf/util/probe-event.c | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 28eb141..7f9b863 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
> }
>
> if (ntevs == 0) { /* No error but failed to find probe point. */
> - pr_warning("Probe point '%s' not found.\n",
> + pr_warning("Probe point '%s' not found in debuginfo.\n",
> synthesize_perf_probe_point(&pev->point));
> - return -ENOENT;
> + if (need_dwarf)
> + return -ENOENT;
> + return 0;
> }
> /* Error path : ntevs < 0 */
> pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/