If libelf unable to open debuginfo for an offline module but the ko has
symtab, something unexpected may happen.
# rm -rf ~/.debug/
# mv /usr/lib64/elfutils/libebl_x86_64.so{,.bak}
# ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
[mymodule] with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events
# ./perf buildid-cache -a ./mymodule.ko
# ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
Added new event:
probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
You can now use it in all perf tools, such as:
perf record -e probe:my_func -aR sleep 1
In the above example, probe fails if it isn't in buildid-cache. However,
user would expect it success in both case because perf is able to find
probe points actually.
The problem is because perf won't utilize module's full path if it
failed to open debuginfo. In
convert_to_probe_trace_events ->
find_probe_trace_events_from_map ->
get_target_map ->
kernel_get_module_map ->
machine__findnew_module_map ->
map_groups__find_by_name
map_groups__find_by_name() is able to find the map of that module, but
this information is found from /proc/modules before it knows the real
path of the offline module. Therefore, the map->dso->long_name is
set to something like '[mymodule]', which prevents dso__load() find
the real path of the module file.
In another aspect, if dso__load() can get the offline module through
buildid cache, it can read symble table from that ko. Even if debuginfo
is not available, 'perf probe' can success if the '.symtab' can be
found.
This patch fixes long_name so dso__load() is able to find module's path
and read symbol table in this case.
After this patch:
# rm -rf ~/.debug/
# mv /usr/lib64/elfutils/libebl_x86_64.so{,.bak}
# ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
Added new event:
probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
You can now use it in all perf tools, such as:
perf record -e probe:my_func -aR sleep 1
# mv /usr/lib64/elfutils/libebl_x86_64.so{.bak,}
Signed-off-by: Wang Nan <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
---
tools/perf/util/probe-event.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 93996ec..ea4f79f 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -2516,6 +2516,7 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
struct probe_trace_point *tp;
int num_matched_functions;
int ret, i, j, skipped = 0;
+ const char *dup_filename;
map = get_target_map(pev->target, pev->uprobes);
if (!map) {
@@ -2523,6 +2524,21 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
goto out;
}
+ /*
+ * If the map's dso is an offline module, give dso__load() a chance
+ * to find the file path of that module by fixing long_name.
+ */
+ if (map->dso && strchr(pev->target, '/')) {
+ if (!map->dso->long_name || map->dso->long_name[0] == '[') {
+ dup_filename = strdup(pev->target);
+ if (!dup_filename) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ dso__set_long_name(map->dso, dup_filename, true);
+ }
+ }
+
syms = malloc(sizeof(struct symbol *) * probe_conf.max_probes);
if (!syms) {
ret = -ENOMEM;
--
1.8.3.4
Em Wed, Nov 25, 2015 at 11:30:59AM +0000, Wang Nan escreveu:
> If libelf unable to open debuginfo for an offline module but the ko has
> symtab, something unexpected may happen.
>
> # rm -rf ~/.debug/
> # mv /usr/lib64/elfutils/libebl_x86_64.so{,.bak}
> # ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
> [mymodule] with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
> Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Understood, since we are giving the path to that module, we can, if its
build-id matches the one for the online module, use it, is that the
case, i.e. do we check, int his scenario, that the build-id matches?
- Arnaldo
> Error: Failed to add events
> # ./perf buildid-cache -a ./mymodule.ko
> # ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
> Added new event:
> probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe:my_func -aR sleep 1
>
> In the above example, probe fails if it isn't in buildid-cache. However,
> user would expect it success in both case because perf is able to find
> probe points actually.
>
> The problem is because perf won't utilize module's full path if it
> failed to open debuginfo. In
> convert_to_probe_trace_events ->
> find_probe_trace_events_from_map ->
> get_target_map ->
> kernel_get_module_map ->
> machine__findnew_module_map ->
> map_groups__find_by_name
>
> map_groups__find_by_name() is able to find the map of that module, but
> this information is found from /proc/modules before it knows the real
> path of the offline module. Therefore, the map->dso->long_name is
> set to something like '[mymodule]', which prevents dso__load() find
> the real path of the module file.
>
> In another aspect, if dso__load() can get the offline module through
> buildid cache, it can read symble table from that ko. Even if debuginfo
> is not available, 'perf probe' can success if the '.symtab' can be
> found.
>
> This patch fixes long_name so dso__load() is able to find module's path
> and read symbol table in this case.
>
> After this patch:
>
> # rm -rf ~/.debug/
> # mv /usr/lib64/elfutils/libebl_x86_64.so{,.bak}
> # ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
> Added new event:
> probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe:my_func -aR sleep 1
>
> # mv /usr/lib64/elfutils/libebl_x86_64.so{.bak,}
>
> Signed-off-by: Wang Nan <[email protected]>
> Cc: Arnaldo Carvalho de Melo <[email protected]>
> Cc: Masami Hiramatsu <[email protected]>
> Cc: Namhyung Kim <[email protected]>
> Cc: Zefan Li <[email protected]>
> Cc: [email protected]
> ---
> tools/perf/util/probe-event.c | 16 ++++++++++++++++
> 1 file changed, 16 insertions(+)
>
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 93996ec..ea4f79f 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -2516,6 +2516,7 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
> struct probe_trace_point *tp;
> int num_matched_functions;
> int ret, i, j, skipped = 0;
> + const char *dup_filename;
>
> map = get_target_map(pev->target, pev->uprobes);
> if (!map) {
> @@ -2523,6 +2524,21 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
> goto out;
> }
>
> + /*
> + * If the map's dso is an offline module, give dso__load() a chance
> + * to find the file path of that module by fixing long_name.
> + */
> + if (map->dso && strchr(pev->target, '/')) {
> + if (!map->dso->long_name || map->dso->long_name[0] == '[') {
> + dup_filename = strdup(pev->target);
> + if (!dup_filename) {
> + ret = -ENOMEM;
> + goto out;
> + }
> + dso__set_long_name(map->dso, dup_filename, true);
> + }
> + }
> +
> syms = malloc(sizeof(struct symbol *) * probe_conf.max_probes);
> if (!syms) {
> ret = -ENOMEM;
> --
> 1.8.3.4
From: Wang Nan [mailto:[email protected]]
>
>If libelf unable to open debuginfo for an offline module but the ko has
>symtab, something unexpected may happen.
>
> # rm -rf ~/.debug/
> # mv /usr/lib64/elfutils/libebl_x86_64.so{,.bak}
Please do give more possible usecase. removing libebl is crazy,
and broken environment.
If you'd like to use perf probe without debuginfo, make NO_DWARF=1 or
strip the target binary.
> # ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
> [mymodule] with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
> Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
> Error: Failed to add events
> # ./perf buildid-cache -a ./mymodule.ko
> # ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
> Added new event:
> probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe:my_func -aR sleep 1
>
>In the above example, probe fails if it isn't in buildid-cache. However,
>user would expect it success in both case because perf is able to find
>probe points actually.
>
>The problem is because perf won't utilize module's full path if it
>failed to open debuginfo. In
> convert_to_probe_trace_events ->
> find_probe_trace_events_from_map ->
> get_target_map ->
> kernel_get_module_map ->
> machine__findnew_module_map ->
> map_groups__find_by_name
>
>map_groups__find_by_name() is able to find the map of that module, but
>this information is found from /proc/modules before it knows the real
>path of the offline module. Therefore, the map->dso->long_name is
>set to something like '[mymodule]', which prevents dso__load() find
>the real path of the module file.
Hmm, if so, it should be fixed in map or machine, not in probe-event.c.
Thanks,
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?
On 2015/11/26 9:10, 平松雅巳 / HIRAMATU，MASAMI wrote:
> From: Wang Nan [mailto:[email protected]]
>> If libelf unable to open debuginfo for an offline module but the ko has
>> symtab, something unexpected may happen.
>>
>> # rm -rf ~/.debug/
>> # mv /usr/lib64/elfutils/libebl_x86_64.so{,.bak}
> Please do give more possible usecase. removing libebl is crazy,
> and broken environment.
It is a real problem we met, where perf itself is statically linked
and copied to target platform.
> If you'd like to use perf probe without debuginfo, make NO_DWARF=1 or
> strip the target binary.
Can't simply stripping the target binary because the symtab would
also be stipped out.
>> # ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
>> [mymodule] with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
>> Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
>> Error: Failed to add events
>> # ./perf buildid-cache -a ./mymodule.ko
>> # ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
>> Added new event:
>> probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
>>
>> You can now use it in all perf tools, such as:
>>
>> perf record -e probe:my_func -aR sleep 1
>>
>> In the above example, probe fails if it isn't in buildid-cache. However,
>> user would expect it success in both case because perf is able to find
>> probe points actually.
>>
>> The problem is because perf won't utilize module's full path if it
>> failed to open debuginfo. In
>> convert_to_probe_trace_events ->
>> find_probe_trace_events_from_map ->
>> get_target_map ->
>> kernel_get_module_map ->
>> machine__findnew_module_map ->
>> map_groups__find_by_name
>>
>> map_groups__find_by_name() is able to find the map of that module, but
>> this information is found from /proc/modules before it knows the real
>> path of the offline module. Therefore, the map->dso->long_name is
>> set to something like '[mymodule]', which prevents dso__load() find
>> the real path of the module file.
> Hmm, if so, it should be fixed in map or machine, not in probe-event.c.
Will do in next version.
Thank you.
Something unexpected may happen if copy statically linked perf to a
production environment:
# ./perf probe -m ./mymodule.ko my_func
[mymodule] with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events.
# ./perf buildid-cache -a ./mymodule.ko
# ./perf probe -m ./mymodule.ko my_func
Added new event:
probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
You can now use it in all perf tools, such as:
perf record -e probe:my_func -aR sleep 1
Where:
# ldd ./perf
not a dynamic executable
# strace -e open ./perf probe -m ./mymodule.ko my_func
...
open("/home/wangnan/kmodule/mymodule.ko", O_RDONLY) = 3
open("/home/wangnan/kmodule/../lib64/elfutils/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...
open("/lib64/tls/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("[mymodule]", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/wangnan/.debug/.build-id/32/6ab42550ef3d24944f53c817533728367effeb", O_RDONLY) = -1 ENOENT (No such file or directory)
open("[mymodule]", O_RDONLY) = -1 ENOENT (No such file or directory)
In the above example, probe fails before we put the module into
buildid-cache. However, user would expect it success in both case
because perf is able to find probe points actually.
The reason is because perf won't utilize module's full path if it
failed to open debuginfo. In
convert_to_probe_trace_events ->
find_probe_trace_events_from_map ->
get_target_map ->
kernel_get_module_map ->
machine__findnew_module_map ->
map_groups__find_by_name
map_groups__find_by_name() is able to find the map of that module, but
this information is found from /proc/module before it knows the real
path of the offline module. Therefore, the map->dso->long_name is
set to something like '[mymodule]', which prevent dso__load() find
the real path of the module file.
In another aspect, if dso__load() can get the offline module through
buildid cache, it can read symble table from that ko. Even if debuginfo
is not available, 'perf probe' can success if the '.symtab' can be
found.
This patch improves machine__findnew_module_map(): when dso->long_name
is leading with '[' (doesn't find path of module when parsing
/proc/modules), fixes it by dso__set_long_name(), so following
dso__load() is possible to find the symbol table.
This patch won't interfere with buildid matching. Here is the test
result:
# ./perf probe -m ./mymodule.ko my_func
Added new event:
probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
You can now use it in all perf tools, such as:
perf record -e probe:my_func -aR sleep 1
# ./perf probe -d '*'
Removed event: probe:my_func
# mv ./mymodule.{ko,.bak}
# mv ./moduleb.ko mymodule.ko
# ./perf probe -m ./mymodule.ko my_func
/home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events.
# ./perf probe -v -m ./mymodule.ko my_func
probe-definition(0): my_func
symbol:my_func file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Could not open debuginfo. Try to use symbols.
symsrc__init: build id mismatch for /home/wangnan/kmodule/mymodule.ko.
/home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events. Reason: No such file or directory (Code: -2)
Signed-off-by: Wang Nan <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
---
tools/perf/util/machine.c | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 7f5071a..cdfa97f 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -561,6 +561,24 @@ int machine__process_switch_event(struct machine *machine __maybe_unused,
return 0;
}
+static void adjust_dso_long_name(struct map *map, const char *filename)
+{
+ const char *dup_filename;
+
+ if (!filename || !map->dso || !map->dso->long_name)
+ return;
+ if (map->dso->long_name[0] != '[')
+ return;
+ if (!strchr(filename, '/'))
+ return;
+
+ dup_filename = strdup(filename);
+ if (!dup_filename)
+ return;
+
+ dso__set_long_name(map->dso, filename, true);
+}
+
struct map *machine__findnew_module_map(struct machine *machine, u64 start,
const char *filename)
{
@@ -573,8 +591,15 @@ struct map *machine__findnew_module_map(struct machine *machine, u64 start,
map = map_groups__find_by_name(&machine->kmaps, MAP__FUNCTION,
m.name);
- if (map)
+ if (map) {
+ /*
+ * If the map's dso is an offline module, give dso__load()
+ * a chance to find the file path of that module by fixing
+ * long_name.
+ */
+ adjust_dso_long_name(map, filename);
goto out;
+ }
dso = machine__findnew_module_dso(machine, &m, filename);
if (dso == NULL)
--
1.8.3.4
On 2015/11/26 3:35, Arnaldo Carvalho de Melo wrote:
> Em Wed, Nov 25, 2015 at 11:30:59AM +0000, Wang Nan escreveu:
>> If libelf unable to open debuginfo for an offline module but the ko has
>> symtab, something unexpected may happen.
>>
>> # rm -rf ~/.debug/
>> # mv /usr/lib64/elfutils/libebl_x86_64.so{,.bak}
>> # ./perf probe -m /home/wangnan/kmodule/mymodule.ko my_func
>> [mymodule] with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
>> Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
> Understood, since we are giving the path to that module, we can, if its
> build-id matches the one for the online module, use it, is that the
> case, i.e. do we check, int his scenario, that the build-id matches?
Please see my test result in v2 patch [1]. In symsrc__init() build-id
would be checked.
[1]
http://lkml.kernel.org/r/[email protected]
# ./perf probe -m ./mymodule.ko my_func
Added new event:
probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
You can now use it in all perf tools, such as:
perf record -e probe:my_func -aR sleep 1
# ./perf probe -d '*'
Removed event: probe:my_func
# mv ./mymodule.{ko,.bak}
# mv ./moduleb.ko mymodule.ko
# ./perf probe -m ./mymodule.ko my_func
/home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events.
# ./perf probe -v -m ./mymodule.ko my_func
probe-definition(0): my_func
symbol:my_funcfile:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Could not open debuginfo. Try to use symbols.
symsrc__init: build id mismatch for /home/wangnan/kmodule/mymodule.ko.
/home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events. Reason: No such file or directory (Code: -2)
Something unexpected may happen if copy statically linked perf to a
production environment:
# ./perf probe -m ./mymodule.ko my_func
[mymodule] with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events.
# ./perf buildid-cache -a ./mymodule.ko
# ./perf probe -m ./mymodule.ko my_func
Added new event:
probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
You can now use it in all perf tools, such as:
perf record -e probe:my_func -aR sleep 1
Where:
# ldd ./perf
not a dynamic executable
# strace -e open ./perf probe -m ./mymodule.ko my_func
...
open("/home/wangnan/kmodule/mymodule.ko", O_RDONLY) = 3
open("/home/wangnan/kmodule/../lib64/elfutils/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...
open("/lib64/tls/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("[mymodule]", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/wangnan/.debug/.build-id/32/6ab42550ef3d24944f53c817533728367effeb", O_RDONLY) = -1 ENOENT (No such file or directory)
open("[mymodule]", O_RDONLY) = -1 ENOENT (No such file or directory)
In the above example, probe fails before we put the module into
buildid-cache. However, user would expect it success in both case
because perf is able to find probe points actually.
The reason is because perf won't utilize module's full path if it
failed to open debuginfo. In
convert_to_probe_trace_events ->
find_probe_trace_events_from_map ->
get_target_map ->
kernel_get_module_map ->
machine__findnew_module_map ->
map_groups__find_by_name
map_groups__find_by_name() is able to find the map of that module, but
this information is found from /proc/module before it knows the real
path of the offline module. Therefore, the map->dso->long_name is
set to something like '[mymodule]', which prevent dso__load() find
the real path of the module file.
In another aspect, if dso__load() can get the offline module through
buildid cache, it can read symble table from that ko. Even if debuginfo
is not available, 'perf probe' can success if the '.symtab' can be
found.
This patch improves machine__findnew_module_map(): when dso->long_name
is leading with '[' (doesn't find path of module when parsing
/proc/modules), fixes it by dso__set_long_name(), so following
dso__load() is possible to find the symbol table.
This patch won't interfere with buildid matching. Here is the test
result:
# ./perf probe -m ./mymodule.ko my_func
Added new event:
probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
You can now use it in all perf tools, such as:
perf record -e probe:my_func -aR sleep 1
# ./perf probe -d '*'
Removed event: probe:my_func
# mv ./mymodule.{ko,.bak}
# mv ./moduleb.ko mymodule.ko
# ./perf probe -m ./mymodule.ko my_func
/home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events.
# ./perf probe -v -m ./mymodule.ko my_func
probe-definition(0): my_func
symbol:my_func file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Could not open debuginfo. Try to use symbols.
symsrc__init: build id mismatch for /home/wangnan/kmodule/mymodule.ko.
/home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events. Reason: No such file or directory (Code: -2)
Signed-off-by: Wang Nan <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
---
v2 -> v3: pass dso to adjust_dso_long_name() instead of map
because adjust_dso_long_name() doesn't use other part of map.
---
tools/perf/util/machine.c | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 7f5071a..5781992 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -561,6 +561,24 @@ int machine__process_switch_event(struct machine *machine __maybe_unused,
return 0;
}
+static void adjust_dso_long_name(struct dso *dso, const char *filename)
+{
+ const char *dup_filename;
+
+ if (!filename || !dso || !dso->long_name)
+ return;
+ if (dso->long_name[0] != '[')
+ return;
+ if (!strchr(filename, '/'))
+ return;
+
+ dup_filename = strdup(filename);
+ if (!dup_filename)
+ return;
+
+ dso__set_long_name(dso, filename, true);
+}
+
struct map *machine__findnew_module_map(struct machine *machine, u64 start,
const char *filename)
{
@@ -573,8 +591,15 @@ struct map *machine__findnew_module_map(struct machine *machine, u64 start,
map = map_groups__find_by_name(&machine->kmaps, MAP__FUNCTION,
m.name);
- if (map)
+ if (map) {
+ /*
+ * If the map's dso is an offline module, give dso__load()
+ * a chance to find the file path of that module by fixing
+ * long_name.
+ */
+ adjust_dso_long_name(map->dso, filename);
goto out;
+ }
dso = machine__findnew_module_dso(machine, &m, filename);
if (dso == NULL)
--
1.8.3.4
Em Thu, Nov 26, 2015 at 03:59:57AM +0000, Wang Nan escreveu:
> Something unexpected may happen if copy statically linked perf to a
> production environment:
>
> # ./perf probe -m ./mymodule.ko my_func
> [mymodule] with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
> Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
> Error: Failed to add events.
> # ./perf buildid-cache -a ./mymodule.ko
> # ./perf probe -m ./mymodule.ko my_func
> Added new event:
> probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe:my_func -aR sleep 1
>
> Where:
>
> # ldd ./perf
> not a dynamic executable
> # strace -e open ./perf probe -m ./mymodule.ko my_func
> ...
> open("/home/wangnan/kmodule/mymodule.ko", O_RDONLY) = 3
> open("/home/wangnan/kmodule/../lib64/elfutils/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> ...
> open("/lib64/tls/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> open("/lib64/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> open("/usr/lib64/tls/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> open("/usr/lib64/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> open("[mymodule]", O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/home/wangnan/.debug/.build-id/32/6ab42550ef3d24944f53c817533728367effeb", O_RDONLY) = -1 ENOENT (No such file or directory)
> open("[mymodule]", O_RDONLY) = -1 ENOENT (No such file or directory)
>
> In the above example, probe fails before we put the module into
> buildid-cache. However, user would expect it success in both case
> because perf is able to find probe points actually.
>
> The reason is because perf won't utilize module's full path if it
> failed to open debuginfo. In
> convert_to_probe_trace_events ->
> find_probe_trace_events_from_map ->
> get_target_map ->
> kernel_get_module_map ->
> machine__findnew_module_map ->
> map_groups__find_by_name
>
> map_groups__find_by_name() is able to find the map of that module, but
> this information is found from /proc/module before it knows the real
> path of the offline module. Therefore, the map->dso->long_name is
> set to something like '[mymodule]', which prevent dso__load() find
> the real path of the module file.
>
> In another aspect, if dso__load() can get the offline module through
> buildid cache, it can read symble table from that ko. Even if debuginfo
> is not available, 'perf probe' can success if the '.symtab' can be
> found.
>
> This patch improves machine__findnew_module_map(): when dso->long_name
> is leading with '[' (doesn't find path of module when parsing
> /proc/modules), fixes it by dso__set_long_name(), so following
> dso__load() is possible to find the symbol table.
>
> This patch won't interfere with buildid matching. Here is the test
> result:
>
> # ./perf probe -m ./mymodule.ko my_func
> Added new event:
> probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe:my_func -aR sleep 1
>
> # ./perf probe -d '*'
> Removed event: probe:my_func
> # mv ./mymodule.{ko,.bak}
> # mv ./moduleb.ko mymodule.ko
> # ./perf probe -m ./mymodule.ko my_func
> /home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
> Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
> Error: Failed to add events.
>
> # ./perf probe -v -m ./mymodule.ko my_func
> probe-definition(0): my_func
> symbol:my_func file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> Could not open debuginfo. Try to use symbols.
> symsrc__init: build id mismatch for /home/wangnan/kmodule/mymodule.ko.
> /home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
> Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
> Error: Failed to add events. Reason: No such file or directory (Code: -2)
>
> Signed-off-by: Wang Nan <[email protected]>
> Cc: Arnaldo Carvalho de Melo <[email protected]>
> Cc: Masami Hiramatsu <[email protected]>
> Cc: Namhyung Kim <[email protected]>
> Cc: Zefan Li <[email protected]>
> Cc: [email protected]
> ---
>
> v2 -> v3: pass dso to adjust_dso_long_name() instead of map
> because adjust_dso_long_name() doesn't use other part of map.
>
> ---
> tools/perf/util/machine.c | 27 ++++++++++++++++++++++++++-
> 1 file changed, 26 insertions(+), 1 deletion(-)
>
> diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
> index 7f5071a..5781992 100644
> --- a/tools/perf/util/machine.c
> +++ b/tools/perf/util/machine.c
> @@ -561,6 +561,24 @@ int machine__process_switch_event(struct machine *machine __maybe_unused,
> return 0;
> }
>
> +static void adjust_dso_long_name(struct dso *dso, const char *filename)
Here, to follow convention, and also to better reflect what it does,
since it deals _only_ with kernel modules, not with all dsos, I'd call
it:
static void dso__adjust_kmod_long_name(struct dso *dso, const char *filename)
It will validate if it is a long name that needs fixing, by looking at
the fist character in long_name.
I thought about dso__set_kmod_long_name() but that would imply that it
would always set it to the filename provided, but 'adjust' seems better
as it will only do it if needed, i.e. if it is still in the initial
'[name]' form.
Since this is just naming, I'll do these changes and apply as you
already addressed Masami's request that this be done in machine.c, and I
agree with that, just please take the above rationale into account for
future patches.
Thanks,
- Arnaldo
> +{
> + const char *dup_filename;
> +
> + if (!filename || !dso || !dso->long_name)
> + return;
> + if (dso->long_name[0] != '[')
> + return;
> + if (!strchr(filename, '/'))
> + return;
> +
> + dup_filename = strdup(filename);
> + if (!dup_filename)
> + return;
> +
> + dso__set_long_name(dso, filename, true);
> +}
> +
> struct map *machine__findnew_module_map(struct machine *machine, u64 start,
> const char *filename)
> {
> @@ -573,8 +591,15 @@ struct map *machine__findnew_module_map(struct machine *machine, u64 start,
>
> map = map_groups__find_by_name(&machine->kmaps, MAP__FUNCTION,
> m.name);
> - if (map)
> + if (map) {
> + /*
> + * If the map's dso is an offline module, give dso__load()
> + * a chance to find the file path of that module by fixing
> + * long_name.
> + */
> + adjust_dso_long_name(map->dso, filename);
> goto out;
> + }
>
> dso = machine__findnew_module_dso(machine, &m, filename);
> if (dso == NULL)
> --
> 1.8.3.4
Commit-ID: c03d5184f0e92fa696e4b57f54ffc3b19a92f704
Gitweb: http://git.kernel.org/tip/c03d5184f0e92fa696e4b57f54ffc3b19a92f704
Author: Wang Nan <[email protected]>
AuthorDate: Thu, 26 Nov 2015 03:59:57 +0000
Committer: Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Thu, 26 Nov 2015 13:47:43 -0300
perf machine: Adjust dso->long_name for offline module
Something unexpected may happen if copy statically linked perf to a
production environment:
# ./perf probe -m ./mymodule.ko my_func
[mymodule] with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events.
# ./perf buildid-cache -a ./mymodule.ko
# ./perf probe -m ./mymodule.ko my_func
Added new event:
probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
You can now use it in all perf tools, such as:
perf record -e probe:my_func -aR sleep 1
Where:
# ldd ./perf
not a dynamic executable
# strace -e open ./perf probe -m ./mymodule.ko my_func
...
open("/home/wangnan/kmodule/mymodule.ko", O_RDONLY) = 3
open("/home/wangnan/kmodule/../lib64/elfutils/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...
open("/lib64/tls/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libebl_x86_64.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("[mymodule]", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/wangnan/.debug/.build-id/32/6ab42550ef3d24944f53c817533728367effeb", O_RDONLY) = -1 ENOENT (No such file or directory)
open("[mymodule]", O_RDONLY) = -1 ENOENT (No such file or directory)
In the above example, probe fails before we put the module into
buildid-cache. However, user would expect it success in both case
because perf is able to find probe points actually.
The reason is because perf won't utilize module's full path if it failed
to open debuginfo. In:
convert_to_probe_trace_events ->
find_probe_trace_events_from_map ->
get_target_map ->
kernel_get_module_map ->
machine__findnew_module_map ->
map_groups__find_by_name
map_groups__find_by_name() is able to find the map of that module, but
this information is found from /proc/module before it knows the real
path of the offline module. Therefore, the map->dso->long_name is set to
something like '[mymodule]', which prevent dso__load() find the real
path of the module file.
In another aspect, if dso__load() can get the offline module through
buildid cache, it can read symble table from that ko. Even if debuginfo
is not available, 'perf probe' can success if the '.symtab' can be
found.
This patch improves machine__findnew_module_map(): when dso->long_name
is leading with '[' (doesn't find path of module when parsing
/proc/modules), fixes it by dso__set_long_name(), so following
dso__load() is possible to find the symbol table.
This patch won't interfere with buildid matching. Here is the test
result:
# ./perf probe -m ./mymodule.ko my_func
Added new event:
probe:my_func (on my_func in /home/wangnan/kmodule/mymodule.ko)
You can now use it in all perf tools, such as:
perf record -e probe:my_func -aR sleep 1
# ./perf probe -d '*'
Removed event: probe:my_func
# mv ./mymodule.{ko,.bak}
# mv ./moduleb.ko mymodule.ko
# ./perf probe -m ./mymodule.ko my_func
/home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events.
# ./perf probe -v -m ./mymodule.ko my_func
probe-definition(0): my_func
symbol:my_func file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Could not open debuginfo. Try to use symbols.
symsrc__init: build id mismatch for /home/wangnan/kmodule/mymodule.ko.
/home/wangnan/kmodule/mymodule.ko with build id 326ab42550ef3d24944f53c817533728367effeb not found, continuing without symbols
Failed to find symbol my_func in /home/wangnan/kmodule/mymodule.ko
Error: Failed to add events. Reason: No such file or directory (Code: -2)
Signed-off-by: Wang Nan <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
Link: http://lkml.kernel.org/r/[email protected]
[ Renamed adjust_dso_long_name() do dso__adjust_kmod_long_name() ]
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/util/machine.c | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index f0019b7..95a7f60 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -561,6 +561,24 @@ int machine__process_switch_event(struct machine *machine __maybe_unused,
return 0;
}
+static void dso__adjust_kmod_long_name(struct dso *dso, const char *filename)
+{
+ const char *dup_filename;
+
+ if (!filename || !dso || !dso->long_name)
+ return;
+ if (dso->long_name[0] != '[')
+ return;
+ if (!strchr(filename, '/'))
+ return;
+
+ dup_filename = strdup(filename);
+ if (!dup_filename)
+ return;
+
+ dso__set_long_name(dso, filename, true);
+}
+
struct map *machine__findnew_module_map(struct machine *machine, u64 start,
const char *filename)
{
@@ -573,8 +591,15 @@ struct map *machine__findnew_module_map(struct machine *machine, u64 start,
map = map_groups__find_by_name(&machine->kmaps, MAP__FUNCTION,
m.name);
- if (map)
+ if (map) {
+ /*
+ * If the map's dso is an offline module, give dso__load()
+ * a chance to find the file path of that module by fixing
+ * long_name.
+ */
+ dso__adjust_kmod_long_name(map->dso, filename);
goto out;
+ }
dso = machine__findnew_module_dso(machine, &m, filename);
if (dso == NULL)