Perf probe misses to set retprobe flag back when falling back to
address-based alternative mode.
Can be reproduced as following:
$ perf probe -v -k vmlinux --add='sys_write%return'
...
Added new event:
Writing event: p:probe/sys_write _stext+1584952
probe:sys_write (on sys_write%return)
$ cat /sys/kernel/debug/tracing/kprobe_events
p:probe/sys_write _stext+1584952
After this patch:
$ perf probe -v -k vmlinux --add='sys_write%return'
Added new event:
Writing event: r:probe/sys_write SyS_write+0
probe:sys_write (on sys_write%return)
$ cat /sys/kernel/debug/tracing/kprobe_events
r:probe/sys_write SyS_write
Signed-off-by: He Kuang <[email protected]>
---
tools/perf/util/probe-event.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 30545ce..5483d98 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
else {
result->offset += pp->offset;
result->line += pp->line;
+ result->retprobe = pp->retprobe;
ret = 0;
}
--
2.3.3.220.g9ab698f
Use get_real_path() to enable --source option when probe with lazy_line
pattern.
Before this patch:
$ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
Failed to open fs/super.c: No such file or directory
Error: Failed to add events.
After this patch:
$ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
Added new events:
probe:_stext (on @fs/super.c)
probe:_stext_1 (on @fs/super.c)
...
Signed-off-by: He Kuang <[email protected]>
---
tools/perf/util/probe-event.c | 2 +-
tools/perf/util/probe-event.h | 2 ++
tools/perf/util/probe-finder.c | 18 +++++++++++++++---
3 files changed, 18 insertions(+), 4 deletions(-)
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 5483d98..35ee51a 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -661,7 +661,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
* a newly allocated path on success.
* Return 0 if file was found and readable, -errno otherwise.
*/
-static int get_real_path(const char *raw_path, const char *comp_dir,
+int get_real_path(const char *raw_path, const char *comp_dir,
char **new_path)
{
const char *prefix = symbol_conf.source_prefix;
diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
index d6b7834..21809ea 100644
--- a/tools/perf/util/probe-event.h
+++ b/tools/perf/util/probe-event.h
@@ -135,6 +135,8 @@ extern int show_available_vars(struct perf_probe_event *pevs, int npevs,
struct strfilter *filter, bool externs);
extern int show_available_funcs(const char *module, struct strfilter *filter,
bool user);
+extern int get_real_path(const char *raw_path, const char *comp_dir,
+ char **new_path);
/* Maximum index number of event-name postfix */
#define MAX_EVENT_INDEX 1024
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index 7831e2d..431c12d 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -791,11 +791,20 @@ static int find_lazy_match_lines(struct intlist *list,
ssize_t len;
int count = 0, linenum = 1;
char sbuf[STRERR_BUFSIZE];
+ char *realname = NULL;
+ int ret;
- fp = fopen(fname, "r");
+ ret = get_real_path(fname, NULL, &realname);
+ if (ret < 0) {
+ pr_warning("Failed to find source file %s.\n", fname);
+ return ret;
+ }
+
+ fp = fopen(realname, "r");
if (!fp) {
- pr_warning("Failed to open %s: %s\n", fname,
+ pr_warning("Failed to open %s: %s\n", realname,
strerror_r(errno, sbuf, sizeof(sbuf)));
+ free(realname);
return -errno;
}
@@ -817,7 +826,10 @@ static int find_lazy_match_lines(struct intlist *list,
fclose(fp);
if (count == 0)
- pr_debug("No matched lines found in %s.\n", fname);
+ pr_debug("No matched lines found in %s.\n", realname);
+
+ free(realname);
+
return count;
}
--
2.3.3.220.g9ab698f
The first argument passed to find_probe_point_lazy() should be CU die,
which will be passed to die_walk_lines() when lazy_line
matches. Currently, when we probe with lazy_line pattern to file without
function name, NULL pointer is passed and causes a segment fault.
Can be repoduced as following:
$ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
[ 1958.984658] perf[1020]: segfault at 10 ip 00007fc6e10d8c71 sp
00007ffcbfaaf900 error 4 in libdw-0.161.so[7fc6e10ce000+34000]
Segmentation fault
After this patch:
$ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
Added new event:
probe:_stext (on @fs/super.c)
You can now use it in all perf tools, such as:
perf record -e probe:_stext -aR sleep 1
Signed-off-by: He Kuang <[email protected]>
---
tools/perf/util/probe-finder.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index 431c12d..e91101b 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -1067,7 +1067,7 @@ static int debuginfo__find_probes(struct debuginfo *dbg,
if (pp->function)
ret = find_probe_point_by_func(pf);
else if (pp->lazy_line)
- ret = find_probe_point_lazy(NULL, pf);
+ ret = find_probe_point_lazy(&pf->cu_die, pf);
else {
pf->lno = pp->line;
ret = find_probe_point_by_line(pf);
--
2.3.3.220.g9ab698f
Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
> Perf probe misses to set retprobe flag back when falling back to
> address-based alternative mode.
And when is that happens, can you explain? Because I tried to follow your
instructions, but in my case it worked in the same way before and after your
patch.
Can you please provide more context for reviewing your patch?
Here are my results:
Before:
[root@ssdandy ~]# perf probe -v --add='sys_write%return'
probe-definition(0): sys_write%return
symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
0 arguments
Using /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda for symbols
Could not open debuginfo. Try to use symbols.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Added new event:
Writing event: r:probe/sys_write sys_write+0
probe:sys_write (on sys_write%return)
You can now use it in all perf tools, such as:
perf record -e probe:sys_write -aR sleep 1
[root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
r:probe/sys_write sys_write
Remove it:
[root@ssdandy ~]# perf probe --del *:*
Removed event: probe:sys_write
After:
[root@ssdandy ~]# perf probe -v --add='sys_write%return'
probe-definition(0): sys_write%return
symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
0 arguments
Using /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda for symbols
Could not open debuginfo. Try to use symbols.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Added new event:
Writing event: r:probe/sys_write sys_write+0
probe:sys_write (on sys_write%return)
You can now use it in all perf tools, such as:
perf record -e probe:sys_write -aR sleep 1
[root@ssdandy ~]#
[root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
r:probe/sys_write sys_write
Humm, noticed one other problem, but not with your patch, about this message:
"Could not open debuginfo. Try to use symbols."
That is really not clear, specially that "try to use symbols" :-)
[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
-rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)
Anyway, digression ended.
- Arnaldo
> Can be reproduced as following:
>
> $ perf probe -v -k vmlinux --add='sys_write%return'
> ...
> Added new event:
> Writing event: p:probe/sys_write _stext+1584952
> probe:sys_write (on sys_write%return)
>
> $ cat /sys/kernel/debug/tracing/kprobe_events
> p:probe/sys_write _stext+1584952
>
> After this patch:
>
> $ perf probe -v -k vmlinux --add='sys_write%return'
> Added new event:
> Writing event: r:probe/sys_write SyS_write+0
> probe:sys_write (on sys_write%return)
>
> $ cat /sys/kernel/debug/tracing/kprobe_events
> r:probe/sys_write SyS_write
>
> Signed-off-by: He Kuang <[email protected]>
> ---
> tools/perf/util/probe-event.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 30545ce..5483d98 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
> else {
> result->offset += pp->offset;
> result->line += pp->line;
> + result->retprobe = pp->retprobe;
> ret = 0;
> }
>
> --
> 2.3.3.220.g9ab698f
Em Mon, Apr 13, 2015 at 11:39:03AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
> > Perf probe misses to set retprobe flag back when falling back to
> > address-based alternative mode.
> Humm, noticed one other problem, but not with your patch, about this message:
> "Could not open debuginfo. Try to use symbols."
> That is really not clear, specially that "try to use symbols" :-)
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
> lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> -rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)
> Anyway, digression ended.
Interesting is that when testing your next patch I see:
[root@ssdandy linux]# perf probe -s ./kernel_src/ --add='fs/super.c;s->s_count=1;'
The /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda file has no debug information.
Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
Error: Failed to add events.
[root@ssdandy linux]#
Much, much clear message about that debuginfo file :-)
But then, to test your [2/3] patch I'll have to figure out how to setup the
environment so that I can match your results, shouldn't be hard, but would
save reviewing time if you stated it in the commit log message.
- Arnaldo
On 04/13/2015 10:39 PM, Arnaldo Carvalho de Melo wrote:
> Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
>> Perf probe misses to set retprobe flag back when falling back to
>> address-based alternative mode.
>
> And when is that happens, can you explain? Because I tried to follow your
> instructions, but in my case it worked in the same way before and after your
> patch.
>
> Can you please provide more context for reviewing your patch?
>
More details:
[root]# perf probe -v -k vmlinux --add='sys_write%return'
probe-definition(0): sys_write%return
symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
0 arguments
Use vmlinux: vmlinux
map_groups__set_modules_path_dir: cannot open /lib/modules/4.0.0-rc6+ dir
Problems setting modules path maps, continuing anyway...
Using vmlinux for symbols
Open Debuginfo file: vmlinux
>>>==========================
Try to find probe point from debuginfo.
Symbol sys_write address found : ffffffff811a8cf0
Probe point found: SyS_write+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Added new event:
Writing event: p:probe/sys_write _stext+1739560
probe:sys_write (on sys_write%return)
You can now use it in all perf tools, such as:
perf record -e probe:sys_write -aR sleep 1
[root@buildroot tmp]# cat /sys/kernel/debug/tracing/kprobe_events
p:probe/sys_write _stext+1739560
The line I marked is different from your result, which is:
"""Could not open debuginfo. Try to use symbols."""
When perf probe searched in debuginfo and failed, it tried with
alternative, in function get_alternative_probe_event():
memcpy(tmp, &pev->point, sizeof(*tmp));
memset(&pev->point, 0, sizeof(pev->point));
In this case, it drops the retprobe flag and forgets to set it back in
find_alternative_probe_point(), so the problem occurred.
> Here are my results:
>
> Before:
>
> [root@ssdandy ~]# perf probe -v --add='sys_write%return'
> probe-definition(0): sys_write%return
> symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
> 0 arguments
> Using /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda for symbols
> Could not open debuginfo. Try to use symbols.
> Opening /sys/kernel/debug/tracing/kprobe_events write=1
> Added new event:
> Writing event: r:probe/sys_write sys_write+0
> probe:sys_write (on sys_write%return)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe:sys_write -aR sleep 1
> [root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
> r:probe/sys_write sys_write
>
> Remove it:
>
> [root@ssdandy ~]# perf probe --del *:*
> Removed event: probe:sys_write
>
> After:
>
> [root@ssdandy ~]# perf probe -v --add='sys_write%return'
> probe-definition(0): sys_write%return
> symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
> 0 arguments
> Using /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda for symbols
> Could not open debuginfo. Try to use symbols.
> Opening /sys/kernel/debug/tracing/kprobe_events write=1
> Added new event:
> Writing event: r:probe/sys_write sys_write+0
> probe:sys_write (on sys_write%return)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe:sys_write -aR sleep 1
>
> [root@ssdandy ~]#
> [root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
> r:probe/sys_write sys_write
>
> Humm, noticed one other problem, but not with your patch, about this message:
>
> "Could not open debuginfo. Try to use symbols."
>
> That is really not clear, specially that "try to use symbols" :-)
>
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
> lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> -rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>
> I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)
>
> Anyway, digression ended.
>
> - Arnaldo
>
>> Can be reproduced as following:
>>
>> $ perf probe -v -k vmlinux --add='sys_write%return'
>> ...
>> Added new event:
>> Writing event: p:probe/sys_write _stext+1584952
>> probe:sys_write (on sys_write%return)
>>
>> $ cat /sys/kernel/debug/tracing/kprobe_events
>> p:probe/sys_write _stext+1584952
>>
>> After this patch:
>>
>> $ perf probe -v -k vmlinux --add='sys_write%return'
>> Added new event:
>> Writing event: r:probe/sys_write SyS_write+0
>> probe:sys_write (on sys_write%return)
>>
>> $ cat /sys/kernel/debug/tracing/kprobe_events
>> r:probe/sys_write SyS_write
>>
>> Signed-off-by: He Kuang <[email protected]>
>> ---
>> tools/perf/util/probe-event.c | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
>> index 30545ce..5483d98 100644
>> --- a/tools/perf/util/probe-event.c
>> +++ b/tools/perf/util/probe-event.c
>> @@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
>> else {
>> result->offset += pp->offset;
>> result->line += pp->line;
>> + result->retprobe = pp->retprobe;
>> ret = 0;
>> }
>>
>> --
>> 2.3.3.220.g9ab698f
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Hi, Arnaldo
On 04/13/2015 10:42 PM, Arnaldo Carvalho de Melo wrote:
> Em Mon, Apr 13, 2015 at 11:39:03AM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
>>> Perf probe misses to set retprobe flag back when falling back to
>>> address-based alternative mode.
>
>> Humm, noticed one other problem, but not with your patch, about this message:
>
>> "Could not open debuginfo. Try to use symbols."
>
>> That is really not clear, specially that "try to use symbols" :-)
>
>> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
>> lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>> -rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>
>> I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)
>
>> Anyway, digression ended.
>
> Interesting is that when testing your next patch I see:
>
> [root@ssdandy linux]# perf probe -s ./kernel_src/ --add='fs/super.c;s->s_count=1;'
> The /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda file has no debug information.
> Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
> Error: Failed to add events.
> [root@ssdandy linux]#
>
> Much, much clear message about that debuginfo file :-)
>
> But then, to test your [2/3] patch I'll have to figure out how to setup the
> environment so that I can match your results, shouldn't be hard, but would
> save reviewing time if you stated it in the commit log message.
Sorry for not providing enough information.
-s ./kernel_src/
The kernel_src dir is a kernel source tree path, it can be
anywhere, for test you can only put fs/super.c in it:
kernel_src/
-- fs
-- super.c
-k vmlinux
This is necessary and should be matched to your running kernel.
>
> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Hi,
This one should already be fixed with Naohiro's patch.
https://lkml.org/lkml/2015/3/13/16
Arnaldo, I've already given my ack for that.
https://lkml.org/lkml/2015/3/13/298
Should I resend that?
Thank you,
(2015/04/13 20:41), He Kuang wrote:
> Use get_real_path() to enable --source option when probe with lazy_line
> pattern.
>
> Before this patch:
>
> $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
> Failed to open fs/super.c: No such file or directory
> Error: Failed to add events.
>
> After this patch:
>
> $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
> Added new events:
> probe:_stext (on @fs/super.c)
> probe:_stext_1 (on @fs/super.c)
> ...
>
> Signed-off-by: He Kuang <[email protected]>
> ---
> tools/perf/util/probe-event.c | 2 +-
> tools/perf/util/probe-event.h | 2 ++
> tools/perf/util/probe-finder.c | 18 +++++++++++++++---
> 3 files changed, 18 insertions(+), 4 deletions(-)
>
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 5483d98..35ee51a 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -661,7 +661,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
> * a newly allocated path on success.
> * Return 0 if file was found and readable, -errno otherwise.
> */
> -static int get_real_path(const char *raw_path, const char *comp_dir,
> +int get_real_path(const char *raw_path, const char *comp_dir,
> char **new_path)
> {
> const char *prefix = symbol_conf.source_prefix;
> diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
> index d6b7834..21809ea 100644
> --- a/tools/perf/util/probe-event.h
> +++ b/tools/perf/util/probe-event.h
> @@ -135,6 +135,8 @@ extern int show_available_vars(struct perf_probe_event *pevs, int npevs,
> struct strfilter *filter, bool externs);
> extern int show_available_funcs(const char *module, struct strfilter *filter,
> bool user);
> +extern int get_real_path(const char *raw_path, const char *comp_dir,
> + char **new_path);
>
> /* Maximum index number of event-name postfix */
> #define MAX_EVENT_INDEX 1024
> diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> index 7831e2d..431c12d 100644
> --- a/tools/perf/util/probe-finder.c
> +++ b/tools/perf/util/probe-finder.c
> @@ -791,11 +791,20 @@ static int find_lazy_match_lines(struct intlist *list,
> ssize_t len;
> int count = 0, linenum = 1;
> char sbuf[STRERR_BUFSIZE];
> + char *realname = NULL;
> + int ret;
>
> - fp = fopen(fname, "r");
> + ret = get_real_path(fname, NULL, &realname);
> + if (ret < 0) {
> + pr_warning("Failed to find source file %s.\n", fname);
> + return ret;
> + }
> +
> + fp = fopen(realname, "r");
> if (!fp) {
> - pr_warning("Failed to open %s: %s\n", fname,
> + pr_warning("Failed to open %s: %s\n", realname,
> strerror_r(errno, sbuf, sizeof(sbuf)));
> + free(realname);
> return -errno;
> }
>
> @@ -817,7 +826,10 @@ static int find_lazy_match_lines(struct intlist *list,
> fclose(fp);
>
> if (count == 0)
> - pr_debug("No matched lines found in %s.\n", fname);
> + pr_debug("No matched lines found in %s.\n", realname);
> +
> + free(realname);
> +
> return count;
> }
>
>
--
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]
(2015/04/13 20:41), He Kuang wrote:
> Perf probe misses to set retprobe flag back when falling back to
> address-based alternative mode.
>
> Can be reproduced as following:
>
> $ perf probe -v -k vmlinux --add='sys_write%return'
> ...
> Added new event:
> Writing event: p:probe/sys_write _stext+1584952
> probe:sys_write (on sys_write%return)
>
> $ cat /sys/kernel/debug/tracing/kprobe_events
> p:probe/sys_write _stext+1584952
>
> After this patch:
>
> $ perf probe -v -k vmlinux --add='sys_write%return'
> Added new event:
> Writing event: r:probe/sys_write SyS_write+0
> probe:sys_write (on sys_write%return)
>
> $ cat /sys/kernel/debug/tracing/kprobe_events
> r:probe/sys_write SyS_write
>
> Signed-off-by: He Kuang <[email protected]>
Oops, I missed that!
Acked-by: Masami Hiramatsu <[email protected]>
Thank you!
> ---
> tools/perf/util/probe-event.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 30545ce..5483d98 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
> else {
> result->offset += pp->offset;
> result->line += pp->line;
> + result->retprobe = pp->retprobe;
> ret = 0;
> }
>
>
--
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]
(2015/04/13 20:41), He Kuang wrote:
> The first argument passed to find_probe_point_lazy() should be CU die,
> which will be passed to die_walk_lines() when lazy_line
> matches. Currently, when we probe with lazy_line pattern to file without
> function name, NULL pointer is passed and causes a segment fault.
>
> Can be repoduced as following:
>
> $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
> [ 1958.984658] perf[1020]: segfault at 10 ip 00007fc6e10d8c71 sp
> 00007ffcbfaaf900 error 4 in libdw-0.161.so[7fc6e10ce000+34000]
> Segmentation fault
>
> After this patch:
>
> $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
> Added new event:
> probe:_stext (on @fs/super.c)
>
> You can now use it in all perf tools, such as:
> perf record -e probe:_stext -aR sleep 1
>
Acked-by: Masami Hiramatsu <[email protected]>
Thanks!
> Signed-off-by: He Kuang <[email protected]>
> ---
> tools/perf/util/probe-finder.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> index 431c12d..e91101b 100644
> --- a/tools/perf/util/probe-finder.c
> +++ b/tools/perf/util/probe-finder.c
> @@ -1067,7 +1067,7 @@ static int debuginfo__find_probes(struct debuginfo *dbg,
> if (pp->function)
> ret = find_probe_point_by_func(pf);
> else if (pp->lazy_line)
> - ret = find_probe_point_lazy(NULL, pf);
> + ret = find_probe_point_lazy(&pf->cu_die, pf);
> else {
> pf->lno = pp->line;
> ret = find_probe_point_by_line(pf);
>
--
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]
Em Tue, Apr 14, 2015 at 12:41:03AM +0900, Masami Hiramatsu escreveu:
> Hi,
>
> This one should already be fixed with Naohiro's patch.
>
> https://lkml.org/lkml/2015/3/13/16
>
> Arnaldo, I've already given my ack for that.
>
> https://lkml.org/lkml/2015/3/13/298
>
> Should I resend that?
I will check that and apply, thanks!
- Arnaldo
>
> Thank you,
>
> (2015/04/13 20:41), He Kuang wrote:
> > Use get_real_path() to enable --source option when probe with lazy_line
> > pattern.
> >
> > Before this patch:
> >
> > $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
> > Failed to open fs/super.c: No such file or directory
> > Error: Failed to add events.
> >
> > After this patch:
> >
> > $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
> > Added new events:
> > probe:_stext (on @fs/super.c)
> > probe:_stext_1 (on @fs/super.c)
> > ...
> >
> > Signed-off-by: He Kuang <[email protected]>
> > ---
> > tools/perf/util/probe-event.c | 2 +-
> > tools/perf/util/probe-event.h | 2 ++
> > tools/perf/util/probe-finder.c | 18 +++++++++++++++---
> > 3 files changed, 18 insertions(+), 4 deletions(-)
> >
> > diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> > index 5483d98..35ee51a 100644
> > --- a/tools/perf/util/probe-event.c
> > +++ b/tools/perf/util/probe-event.c
> > @@ -661,7 +661,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
> > * a newly allocated path on success.
> > * Return 0 if file was found and readable, -errno otherwise.
> > */
> > -static int get_real_path(const char *raw_path, const char *comp_dir,
> > +int get_real_path(const char *raw_path, const char *comp_dir,
> > char **new_path)
> > {
> > const char *prefix = symbol_conf.source_prefix;
> > diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
> > index d6b7834..21809ea 100644
> > --- a/tools/perf/util/probe-event.h
> > +++ b/tools/perf/util/probe-event.h
> > @@ -135,6 +135,8 @@ extern int show_available_vars(struct perf_probe_event *pevs, int npevs,
> > struct strfilter *filter, bool externs);
> > extern int show_available_funcs(const char *module, struct strfilter *filter,
> > bool user);
> > +extern int get_real_path(const char *raw_path, const char *comp_dir,
> > + char **new_path);
> >
> > /* Maximum index number of event-name postfix */
> > #define MAX_EVENT_INDEX 1024
> > diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> > index 7831e2d..431c12d 100644
> > --- a/tools/perf/util/probe-finder.c
> > +++ b/tools/perf/util/probe-finder.c
> > @@ -791,11 +791,20 @@ static int find_lazy_match_lines(struct intlist *list,
> > ssize_t len;
> > int count = 0, linenum = 1;
> > char sbuf[STRERR_BUFSIZE];
> > + char *realname = NULL;
> > + int ret;
> >
> > - fp = fopen(fname, "r");
> > + ret = get_real_path(fname, NULL, &realname);
> > + if (ret < 0) {
> > + pr_warning("Failed to find source file %s.\n", fname);
> > + return ret;
> > + }
> > +
> > + fp = fopen(realname, "r");
> > if (!fp) {
> > - pr_warning("Failed to open %s: %s\n", fname,
> > + pr_warning("Failed to open %s: %s\n", realname,
> > strerror_r(errno, sbuf, sizeof(sbuf)));
> > + free(realname);
> > return -errno;
> > }
> >
> > @@ -817,7 +826,10 @@ static int find_lazy_match_lines(struct intlist *list,
> > fclose(fp);
> >
> > if (count == 0)
> > - pr_debug("No matched lines found in %s.\n", fname);
> > + pr_debug("No matched lines found in %s.\n", realname);
> > +
> > + free(realname);
> > +
> > return count;
> > }
> >
> >
>
>
> --
> Masami HIRAMATSU
> Linux Technology Research Center, System Productivity Research Dept.
> Center for Technology Innovation - Systems Engineering
> Hitachi, Ltd., Research & Development Group
> E-mail: [email protected]
>
Em Tue, Apr 14, 2015 at 12:42:52AM +0900, Masami Hiramatsu escreveu:
> (2015/04/13 20:41), He Kuang wrote:
> > Perf probe misses to set retprobe flag back when falling back to
> > address-based alternative mode.
> >
> > Can be reproduced as following:
> >
> > $ perf probe -v -k vmlinux --add='sys_write%return'
> > ...
> > Added new event:
> > Writing event: p:probe/sys_write _stext+1584952
> > probe:sys_write (on sys_write%return)
> >
> > $ cat /sys/kernel/debug/tracing/kprobe_events
> > p:probe/sys_write _stext+1584952
> >
> > After this patch:
> >
> > $ perf probe -v -k vmlinux --add='sys_write%return'
> > Added new event:
> > Writing event: r:probe/sys_write SyS_write+0
> > probe:sys_write (on sys_write%return)
> >
> > $ cat /sys/kernel/debug/tracing/kprobe_events
> > r:probe/sys_write SyS_write
> >
> > Signed-off-by: He Kuang <[email protected]>
>
> Oops, I missed that!
>
> Acked-by: Masami Hiramatsu <[email protected]>
>
> Thank you!
Thanks to both of you, will add the extra comments from He and apply
after testing with a freshly build kernel with CONFIG_DEBUG_INFO=y.
- Arnaldo
> > ---
> > tools/perf/util/probe-event.c | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> > index 30545ce..5483d98 100644
> > --- a/tools/perf/util/probe-event.c
> > +++ b/tools/perf/util/probe-event.c
> > @@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
> > else {
> > result->offset += pp->offset;
> > result->line += pp->line;
> > + result->retprobe = pp->retprobe;
> > ret = 0;
> > }
> >
> >
>
>
> --
> Masami HIRAMATSU
> Linux Technology Research Center, System Productivity Research Dept.
> Center for Technology Innovation - Systems Engineering
> Hitachi, Ltd., Research & Development Group
> E-mail: [email protected]
>
Em Mon, Apr 13, 2015 at 11:23:12PM +0800, He Kuang escreveu:
>
>
> On 04/13/2015 10:39 PM, Arnaldo Carvalho de Melo wrote:
> >Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
> >>Perf probe misses to set retprobe flag back when falling back to
> >>address-based alternative mode.
> >
> >And when is that happens, can you explain? Because I tried to follow your
> >instructions, but in my case it worked in the same way before and after your
> >patch.
> >
> >Can you please provide more context for reviewing your patch?
> >
>
> More details:
>
> [root]# perf probe -v -k vmlinux --add='sys_write%return'
> probe-definition(0): sys_write%return
> symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
> 0 arguments
> Use vmlinux: vmlinux
> map_groups__set_modules_path_dir: cannot open /lib/modules/4.0.0-rc6+ dir
> Problems setting modules path maps, continuing anyway...
> Using vmlinux for symbols
>
> Open Debuginfo file: vmlinux
> >>>==========================
>
> Try to find probe point from debuginfo.
> Symbol sys_write address found : ffffffff811a8cf0
> Probe point found: SyS_write+0
> Found 1 probe_trace_events.
> Opening /sys/kernel/debug/tracing/kprobe_events write=1
> Added new event:
> Writing event: p:probe/sys_write _stext+1739560
> probe:sys_write (on sys_write%return)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe:sys_write -aR sleep 1
>
> [root@buildroot tmp]# cat /sys/kernel/debug/tracing/kprobe_events
> p:probe/sys_write _stext+1739560
>
>
> The line I marked is different from your result, which is:
> """Could not open debuginfo. Try to use symbols."""
>
>
> When perf probe searched in debuginfo and failed, it tried with alternative,
> in function get_alternative_probe_event():
>
> memcpy(tmp, &pev->point, sizeof(*tmp));
> memset(&pev->point, 0, sizeof(pev->point));
>
> In this case, it drops the retprobe flag and forgets to set it back in
> find_alternative_probe_point(), so the problem occurred.
Thanks for the additional information! I will add it to the commit log.
I'm still not being able to reproduce your results, but this time was
because I tried using those separate .dwo debuginfo files, to check if
'perf' would support it already because elfutils would somehow hide that
from us, but that doesn't seem to be the case:
[root@ssdandy ~]# perf probe -v --add='sys_write%return'
probe-definition(0): sys_write%return
symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/4.0.0-rc6+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.0.0-rc6+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_write address found : ffffffff811e5f10
Failed to find scope of probe point.
An error occurred in debuginfo analysis (-2).
Error: Failed to add events. Reason: No such file or directory (Code: -2)
[root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
[root@ssdandy ~]#
[root@ssdandy linux]# grep CONFIG_DEBUG_INFO ../build/v4.0.0-rc6+/.config
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_INFO_SPLIT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
[root@ssdandy linux]#
Masami, this is about using CONFIG_DEBUG_INFO_SPLIT=y, that eventually we
should support, for now I will disable it and test again,
- Arnaldo
Em Mon, Apr 13, 2015 at 11:38:19PM +0800, He Kuang escreveu:
> Hi, Arnaldo
>
> On 04/13/2015 10:42 PM, Arnaldo Carvalho de Melo wrote:
> >Em Mon, Apr 13, 2015 at 11:39:03AM -0300, Arnaldo Carvalho de Melo escreveu:
> >>Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
> >>>Perf probe misses to set retprobe flag back when falling back to
> >>>address-based alternative mode.
> >
> >>Humm, noticed one other problem, but not with your patch, about this message:
> >
> >>"Could not open debuginfo. Try to use symbols."
> >
> >>That is really not clear, specially that "try to use symbols" :-)
> >
> >>[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
> >>lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> >>[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> >>-rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> >>[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> >
> >>I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)
> >
> >>Anyway, digression ended.
> >
> >Interesting is that when testing your next patch I see:
> >
> > [root@ssdandy linux]# perf probe -s ./kernel_src/ --add='fs/super.c;s->s_count=1;'
> > The /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda file has no debug information.
> > Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
> > Error: Failed to add events.
> > [root@ssdandy linux]#
> >
> >Much, much clear message about that debuginfo file :-)
> >
> >But then, to test your [2/3] patch I'll have to figure out how to setup the
> >environment so that I can match your results, shouldn't be hard, but would
> >save reviewing time if you stated it in the commit log message.
>
> Sorry for not providing enough information.
>
> -s ./kernel_src/
>
> The kernel_src dir is a kernel source tree path, it can be
> anywhere, for test you can only put fs/super.c in it:
>
> kernel_src/
> -- fs
> -- super.c
>
> -k vmlinux
>
> This is necessary and should be matched to your running kernel.
Normally I don't use this, as 'make install' will set it to a place that
the perf symbol code will look, check its build id, see that it matches
the what is running, i.e. this will hold true:
[root@ssdandy ~]# perf buildid-list --hell 2>&1 | grep -- --kernel
-k, --kernel Show current kernel build id
[root@ssdandy ~]# perf buildid-list --kernel
cd1d2cf9f473d0cac668e3afee32866da4540bd4
[root@ssdandy ~]#
[root@ssdandy ~]# perf buildid-list -i
/lib/modules/4.0.0-rc6+/build/vmlinux
cd1d2cf9f473d0cac668e3afee32866da4540bd4
I.e. no need to explicitely pass '-k vmlinux':
[root@ssdandy ~]# perf probe -v --add='sys_write%return'
probe-definition(0): sys_write%return
symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/4.0.0-rc6+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.0.0-rc6+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_write address found : ffffffff811e5f10
Probe point found: SyS_write+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Added new event:
Writing event: r:probe/sys_write SyS_write+0
probe:sys_write (on sys_write%return)
You can now use it in all perf tools, such as:
perf record -e probe:sys_write -aR sleep 1
[root@ssdandy ~]#
- Arnaldo
Em Mon, Apr 13, 2015 at 07:41:29PM +0800, He Kuang escreveu:
> Use get_real_path() to enable --source option when probe with lazy_line
> pattern.
>
> Before this patch:
>
> $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
> Failed to open fs/super.c: No such file or directory
> Error: Failed to add events.
>
> After this patch:
>
> $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
> Added new events:
> probe:_stext (on @fs/super.c)
> probe:_stext_1 (on @fs/super.c)
> ...
Tried just doing:
[root@ssdandy ~]# ulimit -c 100000000
[root@ssdandy ~]# perf probe --add='fs/super.c;s->s_count=1;'
Segmentation fault (core dumped)
I.e. not specifying vmlinux, as it finds the right one looking at the vmlinux path,
and as well not specifying -s, as it should find that as well from the vmlinux file,
but here I get a segfault:
[root@ssdandy ~]# gdb -c core.26128 ~/bin/perf
<SNIP>
Core was generated by `perf probe --add=fs/super.c;s->s_count=1;'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f2a021301c1 in dwarf_tag () from /lib64/libdw.so.1
Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libunwind-1.1-3.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) bt
#0 0x00007f2a021301c1 in dwarf_tag () from /lib64/libdw.so.1
#1 0x00000000004e3b78 in die_walk_lines (rt_die=rt_die@entry=0x0, callback=callback@entry=0x4e0b10 <probe_point_lazy_walker>, data=data@entry=0x7ffff2da1750) at util/dwarf-aux.c:690
#2 0x00000000004dff19 in find_probe_point_lazy (sp_die=sp_die@entry=0x0, pf=pf@entry=0x7ffff2da1750) at util/probe-finder.c:867
#3 0x00000000004e06e2 in debuginfo__find_probes (pf=pf@entry=0x7ffff2da1750, dbg=0x1767c50) at util/probe-finder.c:1058
#4 0x00000000004e13b2 in debuginfo__find_trace_events (dbg=dbg@entry=0x1767c50, pev=pev@entry=0x8fcd90 <params+16>, tevs=tevs@entry=0x1763978, max_tevs=max_tevs@entry=128) at util/probe-finder.c:1218
#5 0x00000000004dcbea in try_to_find_probe_trace_events (target=0x0, max_tevs=128, tevs=0x1763978, pev=0x8fcd90 <params+16>) at util/probe-event.c:611
#6 convert_to_probe_trace_events (target=0x0, max_tevs=128, tevs=0x1763978, pev=0x8fcd90 <params+16>) at util/probe-event.c:2645
#7 add_perf_probe_events (pevs=pevs@entry=0x8fcd90 <params+16>, npevs=<optimized out>, max_tevs=128, force_add=false) at util/probe-event.c:2680
#8 0x000000000044fa7b in __cmd_probe (argc=<optimized out>, argv=argv@entry=0x7ffff2da2600, prefix=<optimized out>) at builtin-probe.c:504
#9 0x000000000044fc34 in cmd_probe (argc=<optimized out>, argv=0x7ffff2da2600, prefix=<optimized out>) at builtin-probe.c:521
#10 0x0000000000470b35 in run_builtin (p=p@entry=0x838420 <commands+384>, argc=argc@entry=2, argv=argv@entry=0x7ffff2da2600) at perf.c:370
#11 0x000000000041d220 in handle_internal_command (argv=0x7ffff2da2600, argc=2) at perf.c:429
#12 run_argv (argv=0x7ffff2da2380, argcp=0x7ffff2da238c) at perf.c:473
#13 main (argc=2, argv=0x7ffff2da2600) at perf.c:588
(gdb)
Now with your patch, another coredump:
[root@ssdandy ~]# rm -f core.26128
[root@ssdandy ~]# perf probe --add='fs/super.c;s->s_count=1;'
Segmentation fault (core dumped)
[root@ssdandy ~]#
[root@ssdandy ~]# gdb -c core.26712 ~/bin/perf
<SNIP>
Core was generated by `perf probe --add=fs/super.c;s->s_count=1;'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f25cf4e61c1 in dwarf_tag () from /lib64/libdw.so.1
Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libunwind-1.1-3.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) bt
#0 0x00007f25cf4e61c1 in dwarf_tag () from /lib64/libdw.so.1
#1 0x00000000004e3c18 in die_walk_lines (rt_die=rt_die@entry=0x0, callback=callback@entry=0x4e0bb0 <probe_point_lazy_walker>, data=data@entry=0x7ffdf9244df0) at util/dwarf-aux.c:690
#2 0x00000000004dff9b in find_probe_point_lazy (sp_die=sp_die@entry=0x0, pf=pf@entry=0x7ffdf9244df0) at util/probe-finder.c:879
#3 0x00000000004e0782 in debuginfo__find_probes (pf=pf@entry=0x7ffdf9244df0, dbg=0xe3ec50) at util/probe-finder.c:1070
#4 0x00000000004e1452 in debuginfo__find_trace_events (dbg=dbg@entry=0xe3ec50, pev=pev@entry=0x8fcd90 <params+16>, tevs=tevs@entry=0xe3a978, max_tevs=max_tevs@entry=128) at util/probe-finder.c:1230
#5 0x00000000004dcc2a in try_to_find_probe_trace_events (target=0x0, max_tevs=128, tevs=0xe3a978, pev=0x8fcd90 <params+16>) at util/probe-event.c:611
#6 convert_to_probe_trace_events (target=0x0, max_tevs=128, tevs=0xe3a978, pev=0x8fcd90 <params+16>) at util/probe-event.c:2645
#7 add_perf_probe_events (pevs=pevs@entry=0x8fcd90 <params+16>, npevs=<optimized out>, max_tevs=128, force_add=false) at util/probe-event.c:2680
#8 0x000000000044faab in __cmd_probe (argc=<optimized out>, argv=argv@entry=0x7ffdf9245ca0, prefix=<optimized out>) at builtin-probe.c:504
#9 0x000000000044fc64 in cmd_probe (argc=<optimized out>, argv=0x7ffdf9245ca0, prefix=<optimized out>) at builtin-probe.c:521
#10 0x0000000000470b65 in run_builtin (p=p@entry=0x838420 <commands+384>, argc=argc@entry=2, argv=argv@entry=0x7ffdf9245ca0) at perf.c:370
#11 0x000000000041d250 in handle_internal_command (argv=0x7ffdf9245ca0, argc=2) at perf.c:429
#12 run_argv (argv=0x7ffdf9245a20, argcp=0x7ffdf9245a2c) at perf.c:473
#13 main (argc=2, argv=0x7ffdf9245ca0) at perf.c:588
(gdb)
Trying exactly as you did, first without the patch:
[root@ssdandy linux]# perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux --add='fs/super.c;s->s_count=1;'
Segmentation fault (core dumped)
[root@ssdandy linux]#
Now with it:
[root@ssdandy linux]# perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux --add='fs/super.c;s->s_count=1;'
Segmentation fault (core dumped)
[root@ssdandy linux]#
Core was generated by `perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux --add=fs/su'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f76795c01c1 in dwarf_tag () from /lib64/libdw.so.1
Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libunwind-1.1-3.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) bt
#0 0x00007f76795c01c1 in dwarf_tag () from /lib64/libdw.so.1
#1 0x00000000004e3c18 in die_walk_lines (rt_die=rt_die@entry=0x0, callback=callback@entry=0x4e0bb0 <probe_point_lazy_walker>, data=data@entry=0x7ffcf282f280) at util/dwarf-aux.c:690
#2 0x00000000004dff9b in find_probe_point_lazy (sp_die=sp_die@entry=0x0, pf=pf@entry=0x7ffcf282f280) at util/probe-finder.c:879
#3 0x00000000004e0782 in debuginfo__find_probes (pf=pf@entry=0x7ffcf282f280, dbg=0x19c1b30) at util/probe-finder.c:1070
#4 0x00000000004e1452 in debuginfo__find_trace_events (dbg=dbg@entry=0x19c1b30, pev=pev@entry=0x8fcd90 <params+16>, tevs=tevs@entry=0x19be778, max_tevs=max_tevs@entry=128) at util/probe-finder.c:1230
#5 0x00000000004dcc2a in try_to_find_probe_trace_events (target=0x0, max_tevs=128, tevs=0x19be778, pev=0x8fcd90 <params+16>) at util/probe-event.c:611
#6 convert_to_probe_trace_events (target=0x0, max_tevs=128, tevs=0x19be778, pev=0x8fcd90 <params+16>) at util/probe-event.c:2645
#7 add_perf_probe_events (pevs=pevs@entry=0x8fcd90 <params+16>, npevs=<optimized out>, max_tevs=128, force_add=false) at util/probe-event.c:2680
#8 0x000000000044faab in __cmd_probe (argc=<optimized out>, argv=argv@entry=0x7ffcf2830130, prefix=<optimized out>) at builtin-probe.c:504
#9 0x000000000044fc64 in cmd_probe (argc=<optimized out>, argv=0x7ffcf2830130, prefix=<optimized out>) at builtin-probe.c:521
#10 0x0000000000470b65 in run_builtin (p=p@entry=0x838420 <commands+384>, argc=argc@entry=6, argv=argv@entry=0x7ffcf2830130) at perf.c:370
#11 0x000000000041d250 in handle_internal_command (argv=0x7ffcf2830130, argc=6) at perf.c:429
#12 run_argv (argv=0x7ffcf282feb0, argcp=0x7ffcf282febc) at perf.c:473
#13 main (argc=6, argv=0x7ffcf2830130) at perf.c:588
(gdb)
And:
[acme@ssdandy linux]$ ls -la /home/acme/git/linux/fs/super.c
-rw-rw-r--. 1 acme acme 36192 Abr 7 17:59 /home/acme/git/linux/fs/super.c
Processed [1/3], looking if [3/3] can be applied independently of this one.
- Arnaldo
Em Mon, Apr 13, 2015 at 05:56:59PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Apr 13, 2015 at 07:41:29PM +0800, He Kuang escreveu:
>
> [root@ssdandy linux]# perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux --add='fs/super.c;s->s_count=1;'
> Segmentation fault (core dumped)
> [root@ssdandy linux]#
[3/3] fixes a segfault, ok, probably this one, checking :-)
- Arnaldo
Em Mon, Apr 13, 2015 at 05:57:51PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Apr 13, 2015 at 05:56:59PM -0300, Arnaldo Carvalho de Melo escreveu:
> > Em Mon, Apr 13, 2015 at 07:41:29PM +0800, He Kuang escreveu:
> >
> > [root@ssdandy linux]# perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux --add='fs/super.c;s->s_count=1;'
> > Segmentation fault (core dumped)
> > [root@ssdandy linux]#
>
> [3/3] fixes a segfault, ok, probably this one, checking :-)
Ok, all three applied, thanks!
- Arnaldo
Commit-ID: 9d7b45c5721dfd524666bc1f07cf7d0855edb4d5
Gitweb: http://git.kernel.org/tip/9d7b45c5721dfd524666bc1f07cf7d0855edb4d5
Author: He Kuang <[email protected]>
AuthorDate: Mon, 13 Apr 2015 19:41:28 +0800
Committer: Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Mon, 13 Apr 2015 17:25:22 -0300
perf probe: Set retprobe flag when probe in address-based alternative mode
When perf probe searched in a debuginfo file and failed, it tried with
an alternative, in function get_alternative_probe_event():
memcpy(tmp, &pev->point, sizeof(*tmp));
memset(&pev->point, 0, sizeof(pev->point));
In this case, it drops the retprobe flag and forgets to set it back in
find_alternative_probe_point(), so the problem occurs.
Can be reproduced as following:
$ perf probe -v -k vmlinux --add='sys_write%return'
...
Added new event:
Writing event: p:probe/sys_write _stext+1584952
probe:sys_write (on sys_write%return)
$ cat /sys/kernel/debug/tracing/kprobe_events
p:probe/sys_write _stext+1584952
After this patch:
$ perf probe -v -k vmlinux --add='sys_write%return'
Added new event:
Writing event: r:probe/sys_write SyS_write+0
probe:sys_write (on sys_write%return)
$ cat /sys/kernel/debug/tracing/kprobe_events
r:probe/sys_write SyS_write
Signed-off-by: He Kuang <[email protected]>
Tested-by: Arnaldo Carvalho de Melo <[email protected]>
Acked-by: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Wang Nan <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/util/probe-event.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 30545ce..5483d98 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
else {
result->offset += pp->offset;
result->line += pp->line;
+ result->retprobe = pp->retprobe;
ret = 0;
}
Commit-ID: f19e80c640d58ddfd70f2454ee597f81ba966690
Gitweb: http://git.kernel.org/tip/f19e80c640d58ddfd70f2454ee597f81ba966690
Author: He Kuang <[email protected]>
AuthorDate: Mon, 13 Apr 2015 19:41:30 +0800
Committer: Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Mon, 13 Apr 2015 20:12:21 -0300
perf probe: Fix segfault when probe with lazy_line to file
The first argument passed to find_probe_point_lazy() should be CU die,
which will be passed to die_walk_lines() when lazy_line matches.
Currently, when we probe with lazy_line pattern to file without function
name, NULL pointer is passed and causes a segment fault.
Can be reproduced as following:
$ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
[ 1958.984658] perf[1020]: segfault at 10 ip 00007fc6e10d8c71 sp
00007ffcbfaaf900 error 4 in libdw-0.161.so[7fc6e10ce000+34000]
Segmentation fault
After this patch:
$ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
Added new event:
probe:_stext (on @fs/super.c)
You can now use it in all perf tools, such as:
perf record -e probe:_stext -aR sleep 1
Signed-off-by: He Kuang <[email protected]>
Tested-by: Arnaldo Carvalho de Melo <[email protected]>
Acked-by: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Wang Nan <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/util/probe-finder.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index ff7865c..44554c3 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -1066,7 +1066,7 @@ static int debuginfo__find_probes(struct debuginfo *dbg,
if (pp->function)
ret = find_probe_point_by_func(pf);
else if (pp->lazy_line)
- ret = find_probe_point_lazy(NULL, pf);
+ ret = find_probe_point_lazy(&pf->cu_die, pf);
else {
pf->lno = pp->line;
ret = find_probe_point_by_line(pf);