Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753195AbbD3PAL (ORCPT ); Thu, 30 Apr 2015 11:00:11 -0400 Received: from e23smtp09.au.ibm.com ([202.81.31.142]:52741 "EHLO e23smtp09.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750946AbbD3PAI (ORCPT ); Thu, 30 Apr 2015 11:00:08 -0400 Date: Thu, 30 Apr 2015 20:28:34 +0530 From: "Naveen N. Rao" To: Masami Hiramatsu Cc: acme@kernel.org, linux-kernel@vger.kernel.org, srikar@linux.vnet.ibm.com Subject: Re: [PATCH] perf probe: Ignore tail calls to probed functions Message-ID: <20150430145834.GA5694@naverao1-tp.in.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <554228E8.8070108@hitachi.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15043015-0033-0000-0000-0000016B1DEB Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 9661 Lines: 258 [CC'ing Srikar] On 2015/04/30 10:06PM, Masami Hiramatsu wrote: > On 2015/04/30 20:42, Naveen N. Rao wrote: > > perf probe currently errors out if there are any tail calls to probed > > functions: > > > > [root@rhel71be]# perf probe do_fork > > Failed to find probe point in any functions. > > Error: Failed to add events. > > > > Fix this by teaching perf to ignore tail calls. > > Hmm, why just ignore it, can't we use it? > Also, could you show us how the result will be changed with this? Without patch: [root@rhel71be perf]# ./perf probe -v do_fork probe-definition(0): do_fork symbol:do_fork file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (7 entries long) symsrc__init: build id mismatch for /boot/vmlinux. Using /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux for symbols Open Debuginfo file: /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux Try to find probe point from debuginfo. found inline addr: 0xc0000000000bb9b0 Probe point found: do_fork+0 found inline addr: 0xc0000000000bbe20 Probe point found: kernel_thread+48 found inline addr: 0xc0000000000bbe5c Probe point found: sys_fork+28 found inline addr: 0xc0000000000bbfac Probe point found: sys_vfork+44 found inline addr: 0xc0000000000bc27c Failed to find probe point in any functions. An error occurred in debuginfo analysis (-2). Error: Failed to add events. Reason: No such file or directory (Code: -2) With patch: [root@rhel71be perf]# ./perf probe -v do_fork probe-definition(0): do_fork symbol:do_fork file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (7 entries long) symsrc__init: build id mismatch for /boot/vmlinux. Using /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux for symbols Open Debuginfo file: /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux Try to find probe point from debuginfo. found inline addr: 0xc0000000000bb9b0 Probe point found: do_fork+0 found inline addr: 0xc0000000000bbe20 Probe point found: kernel_thread+48 found inline addr: 0xc0000000000bbe5c Probe point found: sys_fork+28 found inline addr: 0xc0000000000bbfac Probe point found: sys_vfork+44 found inline addr: 0xc0000000000bc27c Ignoring tail call from SyS_clone Found 4 probe_trace_events. Opening /sys/kernel/debug/tracing/kprobe_events write=1 No kprobe blacklist support, ignored Added new events: Writing event: p:probe/do_fork _text+768432 Failed to write event: Invalid argument Error: Failed to add events. Reason: Invalid argument (Code: -22) [Ignore the error about failure to write event - this kernel is missing a patch to resolve _text properly] The reason to ignore tail calls is that the address does not belong to any function frame. In the example above, the address in SyS_clone is 0xc0000000000bc27c, but looking at the debug-info: <1><830081>: Abbrev Number: 133 (DW_TAG_subprogram) <830083> DW_AT_external : 1 <830083> DW_AT_name : (indirect string, offset: 0x3cea3): SyS_clone <830087> DW_AT_decl_file : 7 <830088> DW_AT_decl_line : 1689 <83008a> DW_AT_prototyped : 1 <83008a> DW_AT_type : <0x8110eb> <83008e> DW_AT_low_pc : 0xc0000000000bc270 <830096> DW_AT_high_pc : 0xc <83009e> DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa) <8300a0> DW_AT_GNU_all_call_sites: 1 <8300a0> DW_AT_sibling : <0x830178> <3><830147>: Abbrev Number: 125 (DW_TAG_GNU_call_site) <830148> DW_AT_low_pc : 0xc0000000000bc27c <830150> DW_AT_GNU_tail_call: 1 <830150> DW_AT_abstract_origin: <0x82e7e1> The frame ends at 0xc0000000000bc27c. I suppose this is why this particular call is a "tail" call. FWIW, systemtap seems to ignore these as well and requires users to explicitly place probes at these call sites if necessary. I print out the caller so that users know. > > Thank you, Thanks for the review! - Naveen > > > > > Signed-off-by: Naveen N. Rao > > --- > > I'm seeing this with RHEL7 kernels on ppc64. The specific example in this case is do_fork: > > > > <1><82e7e1>: Abbrev Number: 118 (DW_TAG_subprogram) > > <82e7e2> DW_AT_external : 1 > > <82e7e2> DW_AT_name : (indirect string, offset: 0x3dfe0): do_fork > > <82e7e6> DW_AT_decl_file : 7 > > <82e7e7> DW_AT_decl_line : 1583 > > <82e7e9> DW_AT_prototyped : 1 > > <82e7e9> DW_AT_type : <0x8110eb> > > <82e7ed> DW_AT_inline : 1 (inlined) > > <82e7ee> DW_AT_sibling : <0x82e878> > > > > > > > > <1><830081>: Abbrev Number: 133 (DW_TAG_subprogram) > > <830083> DW_AT_external : 1 > > <830083> DW_AT_name : (indirect string, offset: 0x3cea3): SyS_clone > > <830087> DW_AT_decl_file : 7 > > <830088> DW_AT_decl_line : 1689 > > <83008a> DW_AT_prototyped : 1 > > <83008a> DW_AT_type : <0x8110eb> > > <83008e> DW_AT_low_pc : 0xc0000000000bc270 > > <830096> DW_AT_high_pc : 0xc > > <83009e> DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa) > > <8300a0> DW_AT_GNU_all_call_sites: 1 > > <8300a0> DW_AT_sibling : <0x830178> > > > > > > > > <3><830147>: Abbrev Number: 125 (DW_TAG_GNU_call_site) > > <830148> DW_AT_low_pc : 0xc0000000000bc27c > > <830150> DW_AT_GNU_tail_call: 1 > > <830150> DW_AT_abstract_origin: <0x82e7e1> > > > > > > - Naveen > > > > tools/perf/util/dwarf-aux.c | 37 +++++++++++++++++++++++++++++++++++++ > > tools/perf/util/dwarf-aux.h | 4 ++++ > > tools/perf/util/probe-finder.c | 12 +++++++++--- > > 3 files changed, 50 insertions(+), 3 deletions(-) > > > > diff --git a/tools/perf/util/dwarf-aux.c b/tools/perf/util/dwarf-aux.c > > index c34e024..851a76f 100644 > > --- a/tools/perf/util/dwarf-aux.c > > +++ b/tools/perf/util/dwarf-aux.c > > @@ -417,6 +417,43 @@ struct __addr_die_search_param { > > Dwarf_Die *die_mem; > > }; > > > > +static int __die_search_func_tail_cb(Dwarf_Die *fn_die, void *data) > > +{ > > + struct __addr_die_search_param *ad = data; > > + Dwarf_Addr addr = 0; > > + > > + if (dwarf_tag(fn_die) == DW_TAG_subprogram && > > + !dwarf_highpc(fn_die, &addr) && > > + addr == ad->addr) { > > + memcpy(ad->die_mem, fn_die, sizeof(Dwarf_Die)); > > + return DWARF_CB_ABORT; > > + } > > + return DWARF_CB_OK; > > +} > > + > > +/** > > + * die_find_tailfunc - Search for a non-inlined function with tail call at > > + * given address > > + * @cu_die: a CU DIE which including @addr > > + * @addr: target address > > + * @die_mem: a buffer for result DIE > > + * > > + * Search for a non-inlined function DIE with tail call at @addr. Stores the > > + * DIE to @die_mem and returns it if found. Returns NULL if failed. > > + */ > > +Dwarf_Die *die_find_tailfunc(Dwarf_Die *cu_die, Dwarf_Addr addr, > > + Dwarf_Die *die_mem) > > +{ > > + struct __addr_die_search_param ad; > > + ad.addr = addr; > > + ad.die_mem = die_mem; > > + /* dwarf_getscopes can't find subprogram. */ > > + if (!dwarf_getfuncs(cu_die, __die_search_func_tail_cb, &ad, 0)) > > + return NULL; > > + else > > + return die_mem; > > +} > > + > > /* die_find callback for non-inlined function search */ > > static int __die_search_func_cb(Dwarf_Die *fn_die, void *data) > > { > > diff --git a/tools/perf/util/dwarf-aux.h b/tools/perf/util/dwarf-aux.h > > index af7dbcd..c9278ed 100644 > > --- a/tools/perf/util/dwarf-aux.h > > +++ b/tools/perf/util/dwarf-aux.h > > @@ -82,6 +82,10 @@ extern Dwarf_Die *die_find_child(Dwarf_Die *rt_die, > > extern Dwarf_Die *die_find_realfunc(Dwarf_Die *cu_die, Dwarf_Addr addr, > > Dwarf_Die *die_mem); > > > > +/* Search a non-inlined function with tail call at given address */ > > +Dwarf_Die *die_find_tailfunc(Dwarf_Die *cu_die, Dwarf_Addr addr, > > + Dwarf_Die *die_mem); > > + > > /* Search the top inlined function including given address */ > > extern Dwarf_Die *die_find_top_inlinefunc(Dwarf_Die *sp_die, Dwarf_Addr addr, > > Dwarf_Die *die_mem); > > diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c > > index b5bf9d5..4cb461e 100644 > > --- a/tools/perf/util/probe-finder.c > > +++ b/tools/perf/util/probe-finder.c > > @@ -660,9 +660,15 @@ static int call_probe_finder(Dwarf_Die *sc_die, struct probe_finder *pf) > > /* If not a real subprogram, find a real one */ > > if (!die_is_func_def(sc_die)) { > > if (!die_find_realfunc(&pf->cu_die, pf->addr, &pf->sp_die)) { > > - pr_warning("Failed to find probe point in any " > > - "functions.\n"); > > - return -ENOENT; > > + if (die_find_tailfunc(&pf->cu_die, pf->addr, &pf->sp_die)) { > > + pr_warning("Ignoring tail call from %s\n", > > + dwarf_diename(&pf->sp_die)); > > + return 0; > > + } else { > > + pr_warning("Failed to find probe point in any " > > + "functions.\n"); > > + return -ENOENT; > > + } > > } > > } else > > memcpy(&pf->sp_die, sc_die, sizeof(Dwarf_Die)); > > > > > -- > Masami HIRAMATSU > Linux Technology Research Center, System Productivity Research Dept. > Center for Technology Innovation - Systems Engineering > Hitachi, Ltd., Research & Development Group > E-mail: masami.hiramatsu.pt@hitachi.com > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/