2010-06-03 17:23:08

by Stephane Eranian

[permalink] [raw]
Subject: how to use perf annotate on the kernel

Arnaldo,

I am trying to understand how one is supposed
to use perf annotate to get sample correlations
with kernel symbols.

I do:

# perf record -a -e cycles:k dd if=/dev/zero of=/dev/null count=1000000
# perf report

# Events: 14K cycles
#
# Overhead Command Shared Object Symbol
# ........ ............... ................... ......
#
8.76% dd [kernel.kallsyms] [k] __lock_acquire
7.49% dd [kernel.kallsyms] [k] sched_clock
6.41% dd [kernel.kallsyms] [k] __clear_user
5.50% dd [kernel.kallsyms] [k] sched_clock_local
3.86% dd [kernel.kallsyms] [k] cpu_clock
3.82% dd [kernel.kallsyms] [k] trace_hardirqs_off_caller
3.50% dd [kernel.kallsyms] [k] lock_release
3.27% dd [kernel.kallsyms] [k] system_call_after_swapgs
3.15% dd [kernel.kallsyms] [k] system_call

# perf annotate (or annotate -d '[kernel.kallsyms]'?)
Can't annotate __lock_acquire: No vmlinux file was found in the path

# perf annotate -k vmlinux
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file
objdump: '[kernel.kallsyms]': No such file

Am I missing something here?
Thanks.


2010-06-03 17:55:10

by Arnaldo Carvalho de Melo

[permalink] [raw]
Subject: Re: how to use perf annotate on the kernel

Em Thu, Jun 03, 2010 at 07:23:02PM +0200, Stephane Eranian escreveu:
> Arnaldo,
>
> I am trying to understand how one is supposed
> to use perf annotate to get sample correlations
> with kernel symbols.
>
> I do:
>
> # perf record -a -e cycles:k dd if=/dev/zero of=/dev/null count=1000000
> # perf report
>
> # Events: 14K cycles
> #
> # Overhead Command Shared Object Symbol
> # ........ ............... ................... ......
> #
> 8.76% dd [kernel.kallsyms] [k] __lock_acquire
> 7.49% dd [kernel.kallsyms] [k] sched_clock
>
> # perf annotate (or annotate -d '[kernel.kallsyms]'?)
> Can't annotate __lock_acquire: No vmlinux file was found in the path
>
> # perf annotate -k vmlinux
> objdump: '[kernel.kallsyms]': No such file

This looks like a bug in how it reports this problem, probably vmlinux
is not in the current directory, or it is but has a build-id that
doesn't matches the one in the perf.data file.

But the message is wrong, should be like it is in the tui, for this same
situation:

ui_helpline__puts("No vmlinux file found, can't "
"annotate with just a kallsyms file");

> Am I missing something here?

If you don't specify it with --vmlinux/-k it will try to find it in one
of these places:

vmlinux_path[vmlinux_path__nr_entries] = strdup("vmlinux");
if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
goto out_fail;
++vmlinux_path__nr_entries;
vmlinux_path[vmlinux_path__nr_entries] = strdup("/boot/vmlinux");
if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
goto out_fail;
++vmlinux_path__nr_entries;
snprintf(bf, sizeof(bf), "/boot/vmlinux-%s", uts.release);
vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
goto out_fail;
++vmlinux_path__nr_entries;
snprintf(bf, sizeof(bf), "/lib/modules/%s/build/vmlinux", uts.release);
vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
if (vmlinux_path[vmlinux_path__nr_entries] == NULL)
goto out_fail;
++vmlinux_path__nr_entries;
snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s/vmlinux",
uts.release);
vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);

It also now looks at /sys/kernel/notes, gets the build-id, and looks up
in the build-id cache.

- Arnaldo

2010-06-03 18:06:38

by Arnaldo Carvalho de Melo

[permalink] [raw]
Subject: Re: how to use perf annotate on the kernel

Em Thu, Jun 03, 2010 at 02:54:45PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Thu, Jun 03, 2010 at 07:23:02PM +0200, Stephane Eranian escreveu:
> > # perf annotate (or annotate -d '[kernel.kallsyms]'?)
> > Can't annotate __lock_acquire: No vmlinux file was found in the path
> >
> > # perf annotate -k vmlinux
> > objdump: '[kernel.kallsyms]': No such file
>
> This looks like a bug in how it reports this problem, probably vmlinux
> is not in the current directory, or it is but has a build-id that
> doesn't matches the one in the perf.data file.
>
> But the message is wrong, should be like it is in the tui, for this same
> situation:
>
> ui_helpline__puts("No vmlinux file found, can't "
> "annotate with just a kallsyms file");
>
> > Am I missing something here?
>
> If you don't specify it with --vmlinux/-k it will try to find it in one
> of these places:
>
> vmlinux_path[vmlinux_path__nr_entries] = strdup("vmlinux");
> vmlinux_path[vmlinux_path__nr_entries] = strdup("/boot/vmlinux");
> snprintf(bf, sizeof(bf), "/boot/vmlinux-%s", uts.release);
> vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
> snprintf(bf, sizeof(bf), "/lib/modules/%s/build/vmlinux", uts.release);
> vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
> snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s/vmlinux",
> uts.release);
> vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
>
> It also now looks at /sys/kernel/notes, gets the build-id, and looks up
> in the build-id cache.

For instance, here using -vvv I get:

[root@emilia ~]# perf annotate -vvv memcpy > /dev/null
build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
Looking at the vmlinux_path (6 entries long)
No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
Using /lib/modules/2.6.35-rc1/build/vmlinux for symbols
hist_entry__inc_addr_samples: ip=0xffffffff811ea69e
0xffffffff811ea690 memcpy: period++ [ip: 0xffffffff811ea69e, 0xe] => 1
hist_entry__annotate: filename=/lib/modules/2.6.35-rc1/build/vmlinux, sym=memcpy, start=0xffffffff811ea690, end=0xffffffff811ea730
annotating [0x187e770] /lib/modules/2.6.35-rc1/build/vmlinux : [0x19b1f70] memcpy
Executing: objdump --start-address=0xffffffff811ea690 --stop-address=0xffffffff811ea730 -dS /lib/modules/2.6.35-rc1/build/vmlinux|grep -v /lib/modules/2.6.35-rc1/build/vmlinux|expand
[root@emilia ~]#

I'm removing all vmlinux from the path so that I can reproduce what you see.

- Arnaldo

2010-06-03 18:11:31

by Arnaldo Carvalho de Melo

[permalink] [raw]
Subject: Re: how to use perf annotate on the kernel

Em Thu, Jun 03, 2010 at 03:06:13PM -0300, Arnaldo Carvalho de Melo escreveu:
> For instance, here using -vvv I get:
>
> [root@emilia ~]# perf annotate -vvv memcpy > /dev/null
> build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
> build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
> Looking at the vmlinux_path (6 entries long)
> No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
> No build_id in vmlinux, ignoring it
> No build_id in /boot/vmlinux, ignoring it
> No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
> Using /lib/modules/2.6.35-rc1/build/vmlinux for symbols
> hist_entry__inc_addr_samples: ip=0xffffffff811ea69e
> 0xffffffff811ea690 memcpy: period++ [ip: 0xffffffff811ea69e, 0xe] => 1
> hist_entry__annotate: filename=/lib/modules/2.6.35-rc1/build/vmlinux, sym=memcpy, start=0xffffffff811ea690, end=0xffffffff811ea730
> annotating [0x187e770] /lib/modules/2.6.35-rc1/build/vmlinux : [0x19b1f70] memcpy
> Executing: objdump --start-address=0xffffffff811ea690 --stop-address=0xffffffff811ea730 -dS /lib/modules/2.6.35-rc1/build/vmlinux|grep -v /lib/modules/2.6.35-rc1/build/vmlinux|expand
> [root@emilia ~]#
>
> I'm removing all vmlinux from the path so that I can reproduce what you see.

[root@emilia ~]# mv /lib/modules/2.6.35-rc1/build/vmlinux /lib/modules/2.6.35-rc1/build/vmlinux.OFF
[root@emilia ~]# perf annotate -vvv memcpy > /dev/null
build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
Looking at the vmlinux_path (6 entries long)
No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
No build_id in /lib/modules/2.6.35-rc1/build/vmlinux, ignoring it
No build_id in /usr/lib/debug/lib/modules/2.6.35-rc1/vmlinux, ignoring it
Using /proc/kallsyms for symbols
[root@emilia ~]#
[root@emilia ~]# vi ~/.perfconfig
[root@emilia ~]# cat ~/.perfconfig
[tui]

report = on
annotate = off
[root@emilia ~]# perf annotate memcpy
[root@emilia ~]# perf annotate memcpy
[root@emilia ~]# perf annotate -k vmlinux memcpy
[kernel.kallsyms] with build id 310920b54cbbf242dbfa09052f652806d2c73821 not found, continuing without symbols

Yeah, these error messages suckz, will work on having proper messages
for these cases :-\

- Arnaldo

2010-06-03 18:12:36

by Stephane Eranian

[permalink] [raw]
Subject: Re: how to use perf annotate on the kernel

Hi,

My kernel is indeed not in the current subdir, I shortened
the cmdline in my post.

This is the live kernel but I don't have it in /lib/.

With verbose, I get output such as:

objdump --start-address=0xffffffff81246ad0
--stop-address=0xffffffff81246b44 -dS -C [kernel.kallsyms]|grep -v
[kernel.kallsyms]|expand


On Thu, Jun 3, 2010 at 8:06 PM, Arnaldo Carvalho de Melo
<[email protected]> wrote:
> Em Thu, Jun 03, 2010 at 02:54:45PM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Thu, Jun 03, 2010 at 07:23:02PM +0200, Stephane Eranian escreveu:
>> > # perf annotate (or annotate -d '[kernel.kallsyms]'?)
>> > Can't annotate __lock_acquire: No vmlinux file was found in the path
>> >
>> > # perf annotate -k vmlinux
>> > objdump: '[kernel.kallsyms]': No such file
>>
>> This looks like a bug in how it reports this problem, probably vmlinux
>> is not in the current directory, or it is but has a build-id that
>> doesn't matches the one in the perf.data file.
>>
>> But the message is wrong, should be like it is in the tui, for this same
>> situation:
>>
>>       ui_helpline__puts("No vmlinux file found, can't "
>>                         "annotate with just a kallsyms file");
>>
>> > Am I missing something here?
>>
>> If you don't specify it with --vmlinux/-k it will try to find it in one
>> of these places:
>>
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup("vmlinux");
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup("/boot/vmlinux");
>>         snprintf(bf, sizeof(bf), "/boot/vmlinux-%s", uts.release);
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
>>         snprintf(bf, sizeof(bf), "/lib/modules/%s/build/vmlinux", uts.release);
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
>>         snprintf(bf, sizeof(bf), "/usr/lib/debug/lib/modules/%s/vmlinux",
>>                  uts.release);
>>         vmlinux_path[vmlinux_path__nr_entries] = strdup(bf);
>>
>> It also now looks at /sys/kernel/notes, gets the build-id, and looks up
>> in the build-id cache.
>
> For instance, here using -vvv I get:
>
> [root@emilia ~]# perf annotate -vvv memcpy > /dev/null
> build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
> build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
> Looking at the vmlinux_path (6 entries long)
> No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
> No build_id in vmlinux, ignoring it
> No build_id in /boot/vmlinux, ignoring it
> No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
> Using /lib/modules/2.6.35-rc1/build/vmlinux for symbols
> hist_entry__inc_addr_samples: ip=0xffffffff811ea69e
> 0xffffffff811ea690 memcpy: period++ [ip: 0xffffffff811ea69e, 0xe] => 1
> hist_entry__annotate: filename=/lib/modules/2.6.35-rc1/build/vmlinux, sym=memcpy, start=0xffffffff811ea690, end=0xffffffff811ea730
> annotating [0x187e770] /lib/modules/2.6.35-rc1/build/vmlinux : [0x19b1f70]                         memcpy
> Executing: objdump --start-address=0xffffffff811ea690 --stop-address=0xffffffff811ea730 -dS /lib/modules/2.6.35-rc1/build/vmlinux|grep -v /lib/modules/2.6.35-rc1/build/vmlinux|expand
> [root@emilia ~]#
>
> I'm removing all vmlinux from the path so that I can reproduce what you see.
>
> - Arnaldo
>

2010-06-03 18:18:10

by Stephane Eranian

[permalink] [raw]
Subject: Re: how to use perf annotate on the kernel

perf annotate __lock_acquire -k /home/eranian/perfmon/official/tip/build/vmlinux
build id event received for [kernel.kallsyms]:
5c110a704293a259fbe11158dc1b25107ad7a23e
build id event received for
/lib/modules/2.6.35-rc1-tip/kernel/arch/x86/kernel/cpu/cpufreq/mperf.ko:
cb5
build id event received for
/lib/modules/2.6.35-rc1-tip/kernel/drivers/cpufreq/cpufreq_ondemand.ko:
cdb5
Using /home/eranian/perfmon/official/tip/build/vmlinux for symbols
hist_entry__inc_addr_samples: ip=0xffffffff81073b36
0xffffffff81073b30 __lock_acquire: period++ [ip: 0xffffffff81073b36, 0x6] => 1
hist_entry__inc_addr_samples: ip=0xffffffff81073c94

But in the end I get:
objdump: '[kernel.kallsyms]': No such file


------------------------------------------------
Percent | Source code & Disassembly of [kernel.kallsyms]
------------------------------------------------


On Thu, Jun 3, 2010 at 8:11 PM, Arnaldo Carvalho de Melo
<[email protected]> wrote:
> Em Thu, Jun 03, 2010 at 03:06:13PM -0300, Arnaldo Carvalho de Melo escreveu:
>> For instance, here using -vvv I get:
>>
>> [root@emilia ~]# perf annotate -vvv memcpy > /dev/null
>> build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
>> build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
>> Looking at the vmlinux_path (6 entries long)
>> No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
>> No build_id in vmlinux, ignoring it
>> No build_id in /boot/vmlinux, ignoring it
>> No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
>> Using /lib/modules/2.6.35-rc1/build/vmlinux for symbols
>> hist_entry__inc_addr_samples: ip=0xffffffff811ea69e
>> 0xffffffff811ea690 memcpy: period++ [ip: 0xffffffff811ea69e, 0xe] => 1
>> hist_entry__annotate: filename=/lib/modules/2.6.35-rc1/build/vmlinux, sym=memcpy, start=0xffffffff811ea690, end=0xffffffff811ea730
>> annotating [0x187e770] /lib/modules/2.6.35-rc1/build/vmlinux : [0x19b1f70]                         memcpy
>> Executing: objdump --start-address=0xffffffff811ea690 --stop-address=0xffffffff811ea730 -dS /lib/modules/2.6.35-rc1/build/vmlinux|grep -v /lib/modules/2.6.35-rc1/build/vmlinux|expand
>> [root@emilia ~]#
>>
>> I'm removing all vmlinux from the path so that I can reproduce what you see.
>
> [root@emilia ~]# mv /lib/modules/2.6.35-rc1/build/vmlinux /lib/modules/2.6.35-rc1/build/vmlinux.OFF
> [root@emilia ~]# perf annotate -vvv memcpy > /dev/null
> build id event received for [kernel.kallsyms]: 310920b54cbbf242dbfa09052f652806d2c73821
> build id event received for /lib64/libtinfo.so.5.7: cd9da7b5b1fcb25ee6b3762bb9fe62f782590fd5
> Looking at the vmlinux_path (6 entries long)
> No build_id in /root/.debug/.build-id/31/0920b54cbbf242dbfa09052f652806d2c73821, ignoring it
> No build_id in vmlinux, ignoring it
> No build_id in /boot/vmlinux, ignoring it
> No build_id in /boot/vmlinux-2.6.35-rc1, ignoring it
> No build_id in /lib/modules/2.6.35-rc1/build/vmlinux, ignoring it
> No build_id in /usr/lib/debug/lib/modules/2.6.35-rc1/vmlinux, ignoring it
> Using /proc/kallsyms for symbols
> [root@emilia ~]#
> [root@emilia ~]# vi ~/.perfconfig
> [root@emilia ~]# cat ~/.perfconfig
> [tui]
>
>        report = on
>        annotate = off
> [root@emilia ~]# perf annotate memcpy
> [root@emilia ~]# perf annotate memcpy
> [root@emilia ~]# perf annotate -k vmlinux memcpy
> [kernel.kallsyms] with build id 310920b54cbbf242dbfa09052f652806d2c73821 not found, continuing without symbols
>
>        Yeah, these error messages suckz, will work on having proper messages
> for these cases :-\
>
> - Arnaldo
>

2010-06-04 00:21:08

by Arnaldo Carvalho de Melo

[permalink] [raw]
Subject: Re: how to use perf annotate on the kernel

Em Thu, Jun 03, 2010 at 08:18:04PM +0200, Stephane Eranian escreveu:
> perf annotate __lock_acquire -k /home/eranian/perfmon/official/tip/build/vmlinux
> build id event received for [kernel.kallsyms]:
> 5c110a704293a259fbe11158dc1b25107ad7a23e
> build id event received for
> /lib/modules/2.6.35-rc1-tip/kernel/arch/x86/kernel/cpu/cpufreq/mperf.ko:
> cb5
> build id event received for
> /lib/modules/2.6.35-rc1-tip/kernel/drivers/cpufreq/cpufreq_ondemand.ko:
> cdb5
> Using /home/eranian/perfmon/official/tip/build/vmlinux for symbols
> hist_entry__inc_addr_samples: ip=0xffffffff81073b36
> 0xffffffff81073b30 __lock_acquire: period++ [ip: 0xffffffff81073b36, 0x6] => 1
> hist_entry__inc_addr_samples: ip=0xffffffff81073c94
>
> But in the end I get:
> objdump: '[kernel.kallsyms]': No such file

Reproduced, working on it now. I've been testing only with vmlinux in
the path it searches for, not by directly specifying it, will get that
fixed, thanks,

- Arnaldo

2010-06-04 00:39:23

by Arnaldo Carvalho de Melo

[permalink] [raw]
Subject: Re: how to use perf annotate on the kernel

Em Thu, Jun 03, 2010 at 09:20:39PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Thu, Jun 03, 2010 at 08:18:04PM +0200, Stephane Eranian escreveu:
> > perf annotate __lock_acquire -k /home/eranian/perfmon/official/tip/build/vmlinux
> > build id event received for [kernel.kallsyms]:
> > 5c110a704293a259fbe11158dc1b25107ad7a23e
> > build id event received for
> > /lib/modules/2.6.35-rc1-tip/kernel/arch/x86/kernel/cpu/cpufreq/mperf.ko:
> > cb5
> > build id event received for
> > /lib/modules/2.6.35-rc1-tip/kernel/drivers/cpufreq/cpufreq_ondemand.ko:
> > cdb5
> > Using /home/eranian/perfmon/official/tip/build/vmlinux for symbols
> > hist_entry__inc_addr_samples: ip=0xffffffff81073b36
> > 0xffffffff81073b30 __lock_acquire: period++ [ip: 0xffffffff81073b36, 0x6] => 1
> > hist_entry__inc_addr_samples: ip=0xffffffff81073c94
> >
> > But in the end I get:
> > objdump: '[kernel.kallsyms]': No such file
>
> Reproduced, working on it now. I've been testing only with vmlinux in
> the path it searches for, not by directly specifying it, will get that
> fixed, thanks,

Should be fixed by this patch:

[root@emilia ~]# perf annotate -k /lib/modules/2.6.35-rc1/build/vmlinux.OFF n_tty_write | head -50


------------------------------------------------
Percent | Source code & Disassembly of vmlinux.OFF
------------------------------------------------
:
:
:
: Disassembly of section .text:
:
: ffffffff81288820 <n_tty_write>:
: * lock themselves)
: */
:
: static ssize_t n_tty_write(struct tty_struct *tty, struct file *file,
: const unsigned char *buf, size_t nr)
: {
0.00 : ffffffff81288820: 41 57 push %r15
:
: DECLARE_PER_CPU(struct task_struct *, current_task);
:
: static __always_inline struct task_struct *get_current(void)
: {
: return percpu_read_stable(current_task);
0.00 : ffffffff81288822: 65 48 8b 04 25 40 cc mov %gs:0xcc40,%rax
0.00 : ffffffff81288829: 00 00
0.00 : ffffffff8128882b: 41 56 push %r14
0.00 : ffffffff8128882d: 41 55 push %r13
0.00 : ffffffff8128882f: 49 89 cd mov %rcx,%r13
0.00 : ffffffff81288832: 41 54 push %r12
0.00 : ffffffff81288834: 49 89 d4 mov %rdx,%r12
0.00 : ffffffff81288837: 55 push %rbp
0.00 : ffffffff81288838: 48 89 f5 mov %rsi,%rbp
0.00 : ffffffff8128883b: 53 push %rbx
0.00 : ffffffff8128883c: 48 89 fb mov %rdi,%rbx
0.00 : ffffffff8128883f: 48 83 ec 58 sub $0x58,%rsp
: const unsigned char *b = buf;
: DECLARE_WAITQUEUE(wait, current);
0.00 : ffffffff81288843: 48 c7 44 24 20 00 00 movq $0x0,0x20(%rsp)
0.00 : ffffffff8128884a: 00 00
7.14 : ffffffff8128884c: 48 c7 44 24 38 00 00 movq $0x0,0x38(%rsp)
0.00 : ffffffff81288853: 00 00
0.00 : ffffffff81288855: 48 c7 44 24 40 00 00 movq $0x0,0x40(%rsp)
0.00 : ffffffff8128885c: 00 00
7.14 : ffffffff8128885e: 48 89 44 24 28 mov %rax,0x28(%rsp)
0.00 : ffffffff81288863: 48 c7 44 24 30 30 49 movq $0xffffffff81044930,0x30(%rsp)
0.00 : ffffffff8128886a: 04 81
: int c;
: ssize_t retval = 0;
:
[root@emilia ~]#

>From 842b66de6807e72bddfd24a63742bacd0f853872 Mon Sep 17 00:00:00 2001
From: Arnaldo Carvalho de Melo <[email protected]>
Date: Thu, 3 Jun 2010 18:35:55 -0300
Subject: [PATCH 1/1] perf symbols: Set the DSO long name when using symbol_conf.vmlinux_name
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

We need to set the long name to the name specified via, for instance,
'perf annotate --vmlinux /path/to/vmlinux', if not it will remain as
'[kernel.kallsyms]' and that will make annotate fail when passing this
as the vmlinux name in the call to objdump.

The way this is setup grew unwieldly and dso__load_vmlinux is the
function that should allocate space for the long name, with callers not
assuming that filenames should be allocated somehow by then (strdup,
dso__build_id_filename, etc).

For now this is the minimalistic patch, a proper fix for .36 will be
made.

Reported-by: Stephane Eranian <[email protected]>
Cc: David S. Miller <[email protected]>
Cc: Fr?d?ric Weisbecker <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Mike Galbraith <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Stephane Eranian <[email protected]>
Cc: Tom Zanussi <[email protected]>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
---
tools/perf/util/symbol.c | 8 ++++++--
1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 7fd6b15..b63e571 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -1745,7 +1745,12 @@ static int dso__load_kernel_sym(struct dso *self, struct map *map,
if (symbol_conf.vmlinux_name != NULL) {
err = dso__load_vmlinux(self, map,
symbol_conf.vmlinux_name, filter);
- goto out_try_fixup;
+ if (err > 0) {
+ dso__set_long_name(self,
+ strdup(symbol_conf.vmlinux_name));
+ goto out_fixup;
+ }
+ return err;
}

if (vmlinux_path != NULL) {
@@ -1806,7 +1811,6 @@ do_kallsyms:
pr_debug("Using %s for symbols\n", kallsyms_filename);
free(kallsyms_allocated_filename);

-out_try_fixup:
if (err > 0) {
out_fixup:
if (kallsyms_filename != NULL)
--
1.6.5.2

2010-06-04 08:55:27

by Stephane Eranian

[permalink] [raw]
Subject: Re: how to use perf annotate on the kernel

Hi,

This patch seems to work now.
Thanks.

On Fri, Jun 4, 2010 at 2:39 AM, Arnaldo Carvalho de Melo
<[email protected]> wrote:
> Em Thu, Jun 03, 2010 at 09:20:39PM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Thu, Jun 03, 2010 at 08:18:04PM +0200, Stephane Eranian escreveu:
>> > perf annotate __lock_acquire -k /home/eranian/perfmon/official/tip/build/vmlinux
>> > build id event received for [kernel.kallsyms]:
>> > 5c110a704293a259fbe11158dc1b25107ad7a23e
>> > build id event received for
>> > /lib/modules/2.6.35-rc1-tip/kernel/arch/x86/kernel/cpu/cpufreq/mperf.ko:
>> > cb5
>> > build id event received for
>> > /lib/modules/2.6.35-rc1-tip/kernel/drivers/cpufreq/cpufreq_ondemand.ko:
>> > cdb5
>> > Using /home/eranian/perfmon/official/tip/build/vmlinux for symbols
>> > hist_entry__inc_addr_samples: ip=0xffffffff81073b36
>> > 0xffffffff81073b30 __lock_acquire: period++ [ip: 0xffffffff81073b36, 0x6] => 1
>> > hist_entry__inc_addr_samples: ip=0xffffffff81073c94
>> >
>> > But in the end I get:
>> > objdump: '[kernel.kallsyms]': No such file
>>
>> Reproduced, working on it now. I've been testing only with vmlinux in
>> the path it searches for, not by directly specifying it, will get that
>> fixed, thanks,
>
> Should be fixed by this patch:
>
> [root@emilia ~]# perf annotate -k /lib/modules/2.6.35-rc1/build/vmlinux.OFF n_tty_write | head -50
>
>
> ------------------------------------------------
>  Percent |      Source code & Disassembly of vmlinux.OFF
> ------------------------------------------------
>         :
>         :
>         :
>         :      Disassembly of section .text:
>         :
>         :      ffffffff81288820 <n_tty_write>:
>         :       *                lock themselves)
>         :       */
>         :
>         :      static ssize_t n_tty_write(struct tty_struct *tty, struct file *file,
>         :                                 const unsigned char *buf, size_t nr)
>         :      {
>    0.00 :      ffffffff81288820:       41 57                   push   %r15
>         :
>         :      DECLARE_PER_CPU(struct task_struct *, current_task);
>         :
>         :      static __always_inline struct task_struct *get_current(void)
>         :      {
>         :              return percpu_read_stable(current_task);
>    0.00 :      ffffffff81288822:       65 48 8b 04 25 40 cc    mov    %gs:0xcc40,%rax
>    0.00 :      ffffffff81288829:       00 00
>    0.00 :      ffffffff8128882b:       41 56                   push   %r14
>    0.00 :      ffffffff8128882d:       41 55                   push   %r13
>    0.00 :      ffffffff8128882f:       49 89 cd                mov    %rcx,%r13
>    0.00 :      ffffffff81288832:       41 54                   push   %r12
>    0.00 :      ffffffff81288834:       49 89 d4                mov    %rdx,%r12
>    0.00 :      ffffffff81288837:       55                      push   %rbp
>    0.00 :      ffffffff81288838:       48 89 f5                mov    %rsi,%rbp
>    0.00 :      ffffffff8128883b:       53                      push   %rbx
>    0.00 :      ffffffff8128883c:       48 89 fb                mov    %rdi,%rbx
>    0.00 :      ffffffff8128883f:       48 83 ec 58             sub    $0x58,%rsp
>         :              const unsigned char *b = buf;
>         :              DECLARE_WAITQUEUE(wait, current);
>    0.00 :      ffffffff81288843:       48 c7 44 24 20 00 00    movq   $0x0,0x20(%rsp)
>    0.00 :      ffffffff8128884a:       00 00
>    7.14 :      ffffffff8128884c:       48 c7 44 24 38 00 00    movq   $0x0,0x38(%rsp)
>    0.00 :      ffffffff81288853:       00 00
>    0.00 :      ffffffff81288855:       48 c7 44 24 40 00 00    movq   $0x0,0x40(%rsp)
>    0.00 :      ffffffff8128885c:       00 00
>    7.14 :      ffffffff8128885e:       48 89 44 24 28          mov    %rax,0x28(%rsp)
>    0.00 :      ffffffff81288863:       48 c7 44 24 30 30 49    movq   $0xffffffff81044930,0x30(%rsp)
>    0.00 :      ffffffff8128886a:       04 81
>         :              int c;
>         :              ssize_t retval = 0;
>         :
> [root@emilia ~]#
>
> From 842b66de6807e72bddfd24a63742bacd0f853872 Mon Sep 17 00:00:00 2001
> From: Arnaldo Carvalho de Melo <[email protected]>
> Date: Thu, 3 Jun 2010 18:35:55 -0300
> Subject: [PATCH 1/1] perf symbols: Set the DSO long name when using symbol_conf.vmlinux_name
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
>
> We need to set the long name to the name specified via, for instance,
> 'perf annotate --vmlinux /path/to/vmlinux', if not it will remain as
> '[kernel.kallsyms]' and that will make annotate fail when passing this
> as the vmlinux name in the call to objdump.
>
> The way this is setup grew unwieldly and dso__load_vmlinux is the
> function that should allocate space for the long name, with callers not
> assuming that filenames should be allocated somehow by then (strdup,
> dso__build_id_filename, etc).
>
> For now this is the minimalistic patch, a proper fix for .36 will be
> made.
>
> Reported-by: Stephane Eranian <[email protected]>
> Cc: David S. Miller <[email protected]>
> Cc: Frédéric Weisbecker <[email protected]>
> Cc: Ingo Molnar <[email protected]>
> Cc: Mike Galbraith <[email protected]>
> Cc: Paul Mackerras <[email protected]>
> Cc: Peter Zijlstra <[email protected]>
> Cc: Stephane Eranian <[email protected]>
> Cc: Tom Zanussi <[email protected]>
> LKML-Reference: <new-submission>
> Signed-off-by: Arnaldo Carvalho de Melo <[email protected]>
> ---
>  tools/perf/util/symbol.c |    8 ++++++--
>  1 files changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
> index 7fd6b15..b63e571 100644
> --- a/tools/perf/util/symbol.c
> +++ b/tools/perf/util/symbol.c
> @@ -1745,7 +1745,12 @@ static int dso__load_kernel_sym(struct dso *self, struct map *map,
>        if (symbol_conf.vmlinux_name != NULL) {
>                err = dso__load_vmlinux(self, map,
>                                        symbol_conf.vmlinux_name, filter);
> -               goto out_try_fixup;
> +               if (err > 0) {
> +                       dso__set_long_name(self,
> +                                          strdup(symbol_conf.vmlinux_name));
> +                       goto out_fixup;
> +               }
> +               return err;
>        }
>
>        if (vmlinux_path != NULL) {
> @@ -1806,7 +1811,6 @@ do_kallsyms:
>                pr_debug("Using %s for symbols\n", kallsyms_filename);
>        free(kallsyms_allocated_filename);
>
> -out_try_fixup:
>        if (err > 0) {
>  out_fixup:
>                if (kallsyms_filename != NULL)
> --
> 1.6.5.2
>
>