Hi,
'perf top' can't display any functions when modules are loaded on ARM.
But if all modules are unloaded, everything is OK and kernel functions can be
displayed.
So how can I make 'perf top' to profile kernel functions when modules are loaded
on ARM?
thanks,
--
Lei Ming
2010/11/11 Ming Lei <[email protected]>:
> Hi,
>
> 'perf top' can't display any functions when modules are loaded on ARM.
Sorry, should be can't display any kernel functions when modules are loaded
on ARM.
Also, when modules are loaded, 'perf report' can't display functions names,
follows result of 'perf report':
root@beagleboard:~# perf report
# Events: 4K cycles
#
# Overhead Command Shared Object Symbol
# ........ ........... ................. ......................
#
97.36% swapper [musb_hdrc] [k] 10466c4
2.19% perf [musb_hdrc] [k] 10da070
0.09% ksoftirqd/0 [musb_hdrc] [k] 10db998
0.06% perf libc-2.9.so [.] memchr
0.04% perf libc-2.9.so [.] 663c0
0.02% perf perf [.] 2e80
0.02% perf libc-2.9.so [.] closedir
0.02% perf libc-2.9.so [.] __read
0.02% flush-179:0 [musb_hdrc] [k] 10ff124
0.02% perf libc-2.9.so [.] memset
0.02% perf libc-2.9.so [.] __memcpy_chk
0.02% perf perf [.] event__synthesize_comm
0.02% perf ffff0fc4 [.] ffff0fc4
0.02% perf perf [.] hex2u64
0.02% perf libc-2.9.so [.] _IO_vfprintf
0.02% perf libc-2.9.so [.] fgets
0.02% perf libpthread-2.9.so [.] flockfile
0.02% perf perf [.] kallsyms__parse
0.02% perf libc-2.9.so [.] _IO_getline_info
Any suggestions or comments?
thanks,
--
Lei Ming
* Ming Lei <[email protected]> wrote:
> 2010/11/11 Ming Lei <[email protected]>:
> > Hi,
> >
> > 'perf top' can't display any functions when modules are loaded on ARM.
>
> Sorry, should be can't display any kernel functions when modules are loaded
> on ARM.
>
> Also, when modules are loaded, 'perf report' can't display functions names,
> follows result of 'perf report':
>
> root@beagleboard:~# perf report
> # Events: 4K cycles
> #
> # Overhead Command Shared Object Symbol
> # ........ ........... ................. ......................
> #
> 97.36% swapper [musb_hdrc] [k] 10466c4
> 2.19% perf [musb_hdrc] [k] 10da070
> 0.09% ksoftirqd/0 [musb_hdrc] [k] 10db998
> 0.06% perf libc-2.9.so [.] memchr
> 0.04% perf libc-2.9.so [.] 663c0
> 0.02% perf perf [.] 2e80
> 0.02% perf libc-2.9.so [.] closedir
> 0.02% perf libc-2.9.so [.] __read
> 0.02% flush-179:0 [musb_hdrc] [k] 10ff124
> 0.02% perf libc-2.9.so [.] memset
> 0.02% perf libc-2.9.so [.] __memcpy_chk
> 0.02% perf perf [.] event__synthesize_comm
> 0.02% perf ffff0fc4 [.] ffff0fc4
> 0.02% perf perf [.] hex2u64
> 0.02% perf libc-2.9.so [.] _IO_vfprintf
> 0.02% perf libc-2.9.so [.] fgets
> 0.02% perf libpthread-2.9.so [.] flockfile
> 0.02% perf perf [.] kallsyms__parse
> 0.02% perf libc-2.9.so [.] _IO_getline_info
>
> Any suggestions or comments?
Must be some bug in the ELF symbol handling code - we've had many cases of quirks
and special cases there. Please run 'perf archive' and send the resulting
perf.data.tar.bz2 to Arnaldo in private mail. (the file will be big)
Thanks,
Ingo
On Thu, Nov 11, 2010 at 4:52 PM, Ming Lei <[email protected]> wrote:
> 2010/11/11 Ming Lei <[email protected]>:
>> Hi,
>>
>> 'perf top' can't display any functions when modules are loaded on ARM.
>
> Sorry, should be can't display any kernel functions when modules are loaded
> on ARM.
>
> Also, when modules are loaded, 'perf report' can't display functions names,
> follows result of 'perf report':
>
Hi,
I tried to build the perf as static and build success on ARM. But
there's no display and no top results also.
How do you build the perf on ARM? any hints or clues?
Thank you,
Kyungmin Park
> root@beagleboard:~# perf report
> # Events: 4K cycles
> #
> # Overhead ? ? ?Command ? ? ?Shared Object ? ? ? ? ? ? ? ? ?Symbol
> # ........ ?........... ?................. ?......................
> #
> ? ?97.36% ? ? ?swapper ?[musb_hdrc] ? ? ? ?[k] ?10466c4
> ? ? 2.19% ? ? ? ? perf ?[musb_hdrc] ? ? ? ?[k] ?10da070
> ? ? 0.09% ?ksoftirqd/0 ?[musb_hdrc] ? ? ? ?[k] ?10db998
> ? ? 0.06% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] memchr
> ? ? 0.04% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] ? ?663c0
> ? ? 0.02% ? ? ? ? perf ?perf ? ? ? ? ? ? ? [.] ? ? 2e80
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] closedir
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] __read
> ? ? 0.02% ?flush-179:0 ?[musb_hdrc] ? ? ? ?[k] ?10ff124
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] memset
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] __memcpy_chk
> ? ? 0.02% ? ? ? ? perf ?perf ? ? ? ? ? ? ? [.] event__synthesize_comm
> ? ? 0.02% ? ? ? ? perf ? ? ? ? ? ffff0fc4 ?[.] ffff0fc4
> ? ? 0.02% ? ? ? ? perf ?perf ? ? ? ? ? ? ? [.] hex2u64
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] _IO_vfprintf
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] fgets
> ? ? 0.02% ? ? ? ? perf ?libpthread-2.9.so ?[.] flockfile
> ? ? 0.02% ? ? ? ? perf ?perf ? ? ? ? ? ? ? [.] kallsyms__parse
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] _IO_getline_info
>
> Any suggestions or comments?
>
> thanks,
> --
> Lei Ming
>
> _______________________________________________
> linux-arm-kernel mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
> 2010/11/11 Ming Lei <[email protected]>:
> > Hi,
> >
> > 'perf top' can't display any functions when modules are loaded on ARM.
>
> Sorry, should be can't display any kernel functions when modules are loaded
> on ARM.
I've noticed this myself and I spent a bit of time looking into it last week
but didn't get too far. I did wonder if it was something to do with the map
fixups after loading the modules. If you look at the raw events after a perf
record with 'perf report -D' then the last module has a size that takes it to
0xFFFFFFFF which overlaps with the rest of the kernel symbols.
On x86 it looks like the modules are loaded after the kernel image in VM
unlike ARM.
Jamie
>2010/11/11 Ingo Molnar <[email protected]>:
> Must be some bug in the ELF symbol handling code - we've had many cases of quirks
> and special cases there. Please run 'perf archive' and send the resulting
> perf.data.tar.bz2 to Arnaldo in private mail. (the file will be big)
Sent to Arnaldo already.
2010/11/11 Kyungmin Park <[email protected]>:
>
> I tried to build the perf as static and build success on ARM. But
> there's no display and no top results also.
> How do you build the perf on ARM? any hints or clues?
Cross-compiling perf tool is very painful, so I choose to compile
perf in Angstom on beagle-xm(ARM) directly, isn't it fun? :-)
thanks,
--
Lei Ming
On Thu, 2010-11-11 at 09:17 +0000, Jamie Iles wrote:
> On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
> > 2010/11/11 Ming Lei <[email protected]>:
> > > Hi,
> > >
> > > 'perf top' can't display any functions when modules are loaded on ARM.
> >
> > Sorry, should be can't display any kernel functions when modules are loaded
> > on ARM.
> I've noticed this myself and I spent a bit of time looking into it last week
> but didn't get too far. I did wonder if it was something to do with the map
> fixups after loading the modules. If you look at the raw events after a perf
> record with 'perf report -D' then the last module has a size that takes it to
> 0xFFFFFFFF which overlaps with the rest of the kernel symbols.
>
> On x86 it looks like the modules are loaded after the kernel image in VM
> unlike ARM.
Order shouldn't matter, but that overlap is very likely what kills it.
That module size is buggy.
On Thu, Nov 11, 2010 at 01:06:16PM +0100, Peter Zijlstra wrote:
> On Thu, 2010-11-11 at 09:17 +0000, Jamie Iles wrote:
> > On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
> > > 2010/11/11 Ming Lei <[email protected]>:
> > > > Hi,
> > > >
> > > > 'perf top' can't display any functions when modules are loaded on ARM.
> > >
> > > Sorry, should be can't display any kernel functions when modules are loaded
> > > on ARM.
> > I've noticed this myself and I spent a bit of time looking into it last week
> > but didn't get too far. I did wonder if it was something to do with the map
> > fixups after loading the modules. If you look at the raw events after a perf
> > record with 'perf report -D' then the last module has a size that takes it to
> > 0xFFFFFFFF which overlaps with the rest of the kernel symbols.
> >
> > On x86 it looks like the modules are loaded after the kernel image in VM
> > unlike ARM.
>
> Order shouldn't matter, but that overlap is very likely what kills it.
> That module size is buggy.
I'm guessing that its in util/symbol.c:__map_groups__fixup_end():
for (nd = rb_next(prevnd); nd; nd = rb_next(nd)) {
prev = curr;
curr = rb_entry(nd, struct map, rb_node);
prev->end = curr->start - 1;
}
/*
* We still haven't the actual symbols, so guess the
* last map final address.
*/
curr->end = ~0UL;
I've noticed that if I change the symbol_conf.{use_modules,try_vmlinux_path}
then I can get perf events for kernel symbols from /proc/kallsyms but nothing
for the modules.
Jamie
On Thu, 2010-11-11 at 12:21 +0000, Jamie Iles wrote:
> On Thu, Nov 11, 2010 at 01:06:16PM +0100, Peter Zijlstra wrote:
> > On Thu, 2010-11-11 at 09:17 +0000, Jamie Iles wrote:
> > > On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
> > > > 2010/11/11 Ming Lei <[email protected]>:
> > > > > Hi,
> > > > >
> > > > > 'perf top' can't display any functions when modules are loaded on ARM.
> > > >
> > > > Sorry, should be can't display any kernel functions when modules are loaded
> > > > on ARM.
> > > I've noticed this myself and I spent a bit of time looking into it last week
> > > but didn't get too far. I did wonder if it was something to do with the map
> > > fixups after loading the modules. If you look at the raw events after a perf
> > > record with 'perf report -D' then the last module has a size that takes it to
> > > 0xFFFFFFFF which overlaps with the rest of the kernel symbols.
> > >
> > > On x86 it looks like the modules are loaded after the kernel image in VM
> > > unlike ARM.
> >
> > Order shouldn't matter, but that overlap is very likely what kills it.
> > That module size is buggy.
> I'm guessing that its in util/symbol.c:__map_groups__fixup_end():
>
> for (nd = rb_next(prevnd); nd; nd = rb_next(nd)) {
> prev = curr;
> curr = rb_entry(nd, struct map, rb_node);
> prev->end = curr->start - 1;
> }
>
> /*
> * We still haven't the actual symbols, so guess the
> * last map final address.
> */
> curr->end = ~0UL;
>
> I've noticed that if I change the symbol_conf.{use_modules,try_vmlinux_path}
> then I can get perf events for kernel symbols from /proc/kallsyms but nothing
> for the modules.
Hrm,.. depending on how we load things this might be easy or hard to
fix. If we load the module symbols after having loaded the kernel
symbols we can replace ~0ULL with the address before the next symbol.
If however we load modules first we're in a bind and should look at
changing the load order to make the first suggestion work.
2010/11/11 Peter Zijlstra <[email protected]>:
> On Thu, 2010-11-11 at 12:21 +0000, Jamie Iles wrote:
>> On Thu, Nov 11, 2010 at 01:06:16PM +0100, Peter Zijlstra wrote:
>> > On Thu, 2010-11-11 at 09:17 +0000, Jamie Iles wrote:
>> > > On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
>> > > > 2010/11/11 Ming Lei <[email protected]>:
>> > > > > Hi,
>> > > > >
>> > > > > 'perf top' can't display any functions when modules are loaded on ARM.
>> > > >
>> > > > Sorry, should be can't display any kernel functions when modules are loaded
>> > > > on ARM.
>> > > I've noticed this myself and I spent a bit of time looking into it last week
>> > > but didn't get too far. I did wonder if it was something to do with the map
>> > > fixups after loading the modules. If you look at the raw events after a perf
>> > > record with 'perf report -D' then the last module has a size that takes it to
>> > > 0xFFFFFFFF which overlaps with the rest of the kernel symbols.
>> > >
>> > > On x86 it looks like the modules are loaded after the kernel image in VM
>> > > unlike ARM.
>> >
>> > Order shouldn't matter, but that overlap is very likely what kills it.
>> > That module size is buggy.
>> I'm guessing that its in util/symbol.c:__map_groups__fixup_end():
>>
>> ? ? ? for (nd = rb_next(prevnd); nd; nd = rb_next(nd)) {
>> ? ? ? ? ? ? ? ? prev = curr;
>> ? ? ? ? ? ? ? ? curr = rb_entry(nd, struct map, rb_node);
>> ? ? ? ? ? ? ? ? prev->end = curr->start - 1;
>> ? ? ? ? }
>>
>> ? ? ? ? /*
>> ? ? ? ? ?* We still haven't the actual symbols, so guess the
>> ? ? ? ? ?* last map final address.
>> ? ? ? ? ?*/
>> ? ? ? ? curr->end = ~0UL;
>>
>> I've noticed that if I change the symbol_conf.{use_modules,try_vmlinux_path}
>> then I can get perf events for kernel symbols from /proc/kallsyms but nothing
>> for the modules.
>
> Hrm,.. depending on how we load things this might be easy or hard to
> fix. If we load the module symbols after having loaded the kernel
> symbols we can replace ~0ULL with the address before the next symbol.
If possible, could you post a patch so that I can verify if the issue is caused
by ~0ULL?
> If however we load modules first we're in a bind and should look at
> changing the load order to make the first suggestion work.
>
thanks,
--
Lei Ming