CCing linux-perf-users, this may be interesting when investigating
similar problems for other arches.
Thanks James, got the vmlinux file and and already found and fixed two
bugs, patches sent CCed to you.
Ok, now how does it look like?
[acme@doppio linux-2.6-tip]$ perf report -i ~/iles/perf.data --vmlinux ~/iles/vmlinux
Failed to open /lib/ld-2.8.so, continuing without symbols
Failed to open /root/perf, continuing without symbols
Failed to open /lib/libc-2.8.so, continuing without symbols
# Samples: 40653353521
#
# Overhead Command Shared Object Symbol
# ........ ............... ................. ......
#
99.94% swapper c0020ac4 [k] 0x000000c0020ac4
0.04% perf c0068f48 [k] 0x000000c0068f48
0.01% perf libc-2.8.so [.] 0x00000000067198
0.01% sleep c00acb44 [k] 0x000000c00acb44
0.00% perf perf [.] 0x0000000000265c
0.00% perf ld-2.8.so [.] 0x00000000015fa8
0.00% sleep ld-2.8.so [.] 0x00000000008ff0
0.00% perf [kernel.kallsyms] [k] smp_call_function_single
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
[acme@doppio linux-2.6-tip]$
So we manage to decode at least one kernel entry, yay!
To make sure we really did it right, we can use -v to show the address
too:
[acme@doppio linux-2.6-tip]$ perf report -v -i ~/iles/perf.data --vmlinux ~/iles/vmlinux
Failed to open /lib/ld-2.8.so, continuing without symbols
Failed to open /root/perf, continuing without symbols
Failed to open /lib/libc-2.8.so, continuing without symbols
# Samples: 40653353521
#
# Overhead Command Shared Object Symbol
# ........ ............... ................. ......
#
99.94% swapper c0020ac4 0x00000000c0020ac4 ! [k] 0x000000c0020ac4
0.04% perf c0068f48 0x00000000c0068f48 ! [k] 0x000000c0068f48
0.01% perf /lib/libc-2.8.so 0x0000000000067198 K [.] 0x00000000067198
0.01% sleep c00acb44 0x00000000c00acb44 ! [k] 0x000000c00acb44
0.00% perf /root/perf 0x000000000000265c K [.] 0x0000000000265c
0.00% perf /lib/ld-2.8.so 0x0000000000015fa8 K [.] 0x00000000015fa8
0.00% sleep /lib/ld-2.8.so 0x0000000000008ff0 K [.] 0x00000000008ff0
0.00% perf [kernel.kallsyms] 0x00000000c0068f48 ! [k] smp_call_function_single
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
[acme@doppio linux-2.6-tip]$
Ok, so smp_call_function_single is 0xc0068f48, lets see if that matches
what is in the vmlinux symtab you provided me:
[acme@doppio linux-2.6-tip]$ file ~/iles/vmlinux
/home/acme/iles/vmlinux: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, not stripped
[acme@doppio linux-2.6-tip]$ readelf -s ~/iles/vmlinux | grep smp_call_function_single
57421: c0068f18 60 FUNC GLOBAL DEFAULT 3 smp_call_function_single
[acme@doppio linux-2.6-tip]$
Ok, it matches, as 0xc0068f18 <= 0xc0068f48 < (0xc0068f18 + 0x60).
But what about the other addresses for [k]ernel space perf hits such as
0xc0020ac4, 0xc0068f48, etc?
0xc0020ac4 is around here:
385: 00000000 0 FILE LOCAL DEFAULT ABS process.c
<SNIP>
411: c0020aa0 40 FUNC LOCAL DEFAULT 3 default_idle
412: c0020ac8 0 NOTYPE LOCAL DEFAULT 3 $a
so its well possible that this is really a very idle machine, right?
0xc0068f48 is around here:
7997: 00000000 0 FILE LOCAL DEFAULT ABS up.c
7998: c0068f18 0 NOTYPE LOCAL DEFAULT 3 $a
7999: c0068f50 0 NOTYPE LOCAL DEFAULT 3 $d
0xc00acb44 around here:
13645: 00000000 0 FILE LOCAL DEFAULT ABS mmap.c
<SNIP>
13679: c00aca98 216 FUNC LOCAL DEFAULT 3 __vma_link_file
13680: c00acb70 0 NOTYPE LOCAL DEFAULT 3 $a
13681: c00acb70 428 FUNC LOCAL DEFAULT 3 unmap_region
We now need to try to investigate why these things are being
misannotated in the symtab.
- Arnaldo
Hi Arnaldo,
On Tue, Jan 19, 2010 at 11:09:58AM -0200, Arnaldo Carvalho de Melo wrote:
> CCing linux-perf-users, this may be interesting when investigating
> similar problems for other arches.
>
> Thanks James, got the vmlinux file and and already found and fixed two
> bugs, patches sent CCed to you.
>
> Ok, now how does it look like?
Well, with both patches applied, I still don't see any kernel symbols in perf
top (even the raw addresses) or resolved names in report if I specify a
vmlinux or if one gets autodetected.
For example, when keeping the CPU busy with 'dd if=/dev/zero of=/dev/null', If
I record with 'perf record -af -- sleep 1' then:
root@picopc7302:~# ./perf record -af -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.071 MB perf.data (~3092 samples) ]
root@picopc7302:~# ./perf report --vmlinux=/tmp/vmlinux -v | head -10
no symbols found in /bin/busybox, maybe install a debug package?
# Samples: 435596970
#
# Overhead Command Shared Object Symbol
# ........ ............... ...................... ......
#
81.14% dd c0020ac8 0x00000000c0020ac8 ! [k] 0x000000c0020ac8
9.26% perf c0071bfc 0x00000000c0071bfc ! [k] 0x000000c0071bfc
3.53% dd /bin/busybox 0x000000000001d6f4 K [.] 0x0000000001d6f4
1.55% sleep c006c3a0 0x00000000c006c3a0 ! [k] 0x000000c006c3a0
1.38% dd /lib/libc-2.8.so 0x00000000000b4a2c d [.] __GI___libc_read
root@picopc7302:~# mv /boot/vmlinux /boot/vmlinux
root@picopc7302:~# mv /boot/vmlinux /boot/v
root@picopc7302:~# ./perf report -v | head -10
Looking at the vmlinux_path (5 entries long)
no symbols found in /bin/busybox, maybe install a debug package?
# Samples: 435596970
#
# Overhead Command Shared Object Symbol
# ........ ............... ...................... ......
#
32.57% dd [kernel.kallsyms] 0x00000000c02aaef0 k [k] _raw_spin_unlock_irqrestore
12.30% dd [kernel.kallsyms] 0x00000000c006b5c4 k [k] lock_acquire
6.23% dd [kernel.kallsyms] 0x00000000c006c220 k [k] lock_release
5.38% dd [kernel.kallsyms] 0x00000000c006b9f0 k [k] lock_acquired
5.04% dd [kernel.kallsyms] 0x00000000c0020ac8 k [k] vector_swi
>
[snip]
>
> But what about the other addresses for [k]ernel space perf hits such as
> 0xc0020ac4, 0xc0068f48, etc?
>
> 0xc0020ac4 is around here:
>
> 385: 00000000 0 FILE LOCAL DEFAULT ABS process.c
> <SNIP>
> 411: c0020aa0 40 FUNC LOCAL DEFAULT 3 default_idle
> 412: c0020ac8 0 NOTYPE LOCAL DEFAULT 3 $a
>
> so its well possible that this is really a very idle machine, right?
Yes, the run I sent you was pretty much just idling away.
Jamie
Em Tue, Jan 19, 2010 at 01:51:45PM +0000, Jamie Iles escreveu:
> Hi Arnaldo,
>
> On Tue, Jan 19, 2010 at 11:09:58AM -0200, Arnaldo Carvalho de Melo wrote:
> > CCing linux-perf-users, this may be interesting when investigating
> > similar problems for other arches.
> >
> > Thanks James, got the vmlinux file and and already found and fixed two
> > bugs, patches sent CCed to you.
> >
> > Ok, now how does it look like?
> Well, with both patches applied, I still don't see any kernel symbols in perf
> top (even the raw addresses) or resolved names in report if I specify a
> vmlinux or if one gets autodetected.
What if you try to do what I did, i.e. to try to decode the perf.data
file on your x86 workstation? Does specifying --vmlinux
/path/to/arm-vmlinux gets the symbols decoded?
- Arnaldo