I was trying to write some simple docs on how to do profiling for people
to use for really basic stuff. I got it all wrong, but John's kindly
corrected it ;-) Andrew asked me to do this as a patch for the
documentation directory ... feedback would be much appreciated
(yes, it's oversimplified - it's meant to be).
diff -urpN -X /home/fletch/.diff.exclude virgin/Documentation/basic_profiling.txt oprofile_doc/Documentation/basic_profiling.txt
--- virgin/Documentation/basic_profiling.txt Wed Dec 31 16:00:00 1969
+++ oprofile_doc/Documentation/basic_profiling.txt Fri Feb 28 00:05:59 2003
@@ -0,0 +1,44 @@
+These instructions are deliberately very basic. If you want something clever,
+go read the real docs ;-) Please don't add more stuff, but feel free to
+correct my mistakes ;-) ([email protected])
+Thanks to John Levon and Dave Hansen for help writing this.
+
+<test> is the thing you're trying to measure.
+Make sure you have the correct System.map / vmlinux referenced!
+IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
+to copy config files, system.map, vmlinux to /boot.
+
+Readprofile
+-----------
+get readprofile binary fixed for 2.5 / akpm's 2.5 patch from
+ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
+add "profile=2" to the kernel command line.
+
+clear echo 2 > /proc/profile
+ <test>
+dump output readprofile -m /boot/System.map > catured_profile
+
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3 opcontrol --setup --vmlinux=/boot/vmlinux \
+ --ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athalon opcontrol --setup --vmlinux=/boot/vmlinux \
+ --ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4 opcontrol --setup --vmlinux=/boot/vmlinux \
+ --ctr0-event=GLOBAL_POWER_EVENTS \
+ --ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon opcontrol --start-daemon
+clear opcontrol --reset
+start opcontrol --start
+ <test>
+stop opcontrol --stop
+dump output oprofpp -dl -i /boot/vmlinux > output_file
+
"Martin J. Bligh" <[email protected]> wrote:
>
> ...
> +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from
> +ftp://ftp.kernel.org/pub/linux/ people/mbligh/tools/readprofile/
kernel/
> +add "profile=2" to the kernel command line.
> +
> +clear echo 2 > /proc/profile
> + <test>
> +dump output readprofile -m /boot/System.map > catured_profile
util-linux-2.11z (at least) has the fixed readprofile.
Of course, installing standard util-linux turns your boot process
into a complete mess because vendors have added incompatible features
to their versions. But it seems to struggle through.
> +Oprofile
> +--------
> +get source (I use 0.5) from http://oprofile.sourceforge.net/
> +add "poll=idle" to the kernel command line
> +Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
> +./configure --with-kernel-support
> +make install
> +
> +One time setup (pick appropriate one for your CPU):
> +P3 opcontrol --setup --vmlinux=/boot/vmlinux \
> + --ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
> +Athalon opcontrol --setup --vmlinux=/boot/vmlinux \
Athlon
> + --ctr0-event=RETIRED_INSNS --ctr0-count=100000
> +P4 opcontrol --setup --vmlinux=/boot/vmlinux \
> + --ctr0-event=GLOBAL_POWER_EVENTS \
> + --ctr0-unit-mask=1 --ctr0-count=100000
> +
> +start daemon opcontrol --start-daemon
> +clear opcontrol --reset
> +start opcontrol --start
> + <test>
> +stop opcontrol --stop
> +dump output oprofpp -dl -i /boot/vmlinux > output_file
> +
OK. Might be worth adding a pointer to this in REPORTING-BUGS, but
nobody reads that anyway.
On Fri, Feb 28, 2003 at 12:29:35AM -0800, Andrew Morton wrote:
> > +Athalon opcontrol --setup --vmlinux=/boot/vmlinux \
> Athlon
Also x86-64 as of 0.5.1
Dave
On Fri, Feb 28, 2003 at 11:22:38AM +0000, Dave Jones wrote:
> On Fri, Feb 28, 2003 at 12:29:35AM -0800, Andrew Morton wrote:
>
> > > +Athalon opcontrol --setup --vmlinux=/boot/vmlinux \
> > Athlon
>
> Also x86-64 as of 0.5.1
Alpha is supported too... (as least I saw the kernel part go in)
On Fri, 28 Feb 2003 00:12:42 -0800
"Martin J. Bligh" <[email protected]> wrote:
| I was trying to write some simple docs on how to do profiling for people
| to use for really basic stuff. I got it all wrong, but John's kindly
| corrected it ;-) Andrew asked me to do this as a patch for the
| documentation directory ... feedback would be much appreciated
| (yes, it's oversimplified - it's meant to be).
|
| diff -urpN -X /home/fletch/.diff.exclude virgin/Documentation/basic_profiling.txt oprofile_doc/Documentation/basic_profiling.txt
| --- virgin/Documentation/basic_profiling.txt Wed Dec 31 16:00:00 1969
| +++ oprofile_doc/Documentation/basic_profiling.txt Fri Feb 28 00:05:59 2003
| @@ -0,0 +1,44 @@
| +These instructions are deliberately very basic. If you want something clever,
| +go read the real docs ;-) Please don't add more stuff, but feel free to
| +correct my mistakes ;-) ([email protected])
| +Thanks to John Levon and Dave Hansen for help writing this.
| +
| +<test> is the thing you're trying to measure.
| +Make sure you have the correct System.map / vmlinux referenced!
| +IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
| +to copy config files, system.map, vmlinux to /boot.
| +
| +Readprofile
| +-----------
| +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from
| +ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
| +add "profile=2" to the kernel command line.
These: ^------------v should be the same value (as you have it).
v
| +clear echo 2 > /proc/profile
man page says to use "readprofile -r". Doesn't that still work?
| + <test>
| +dump output readprofile -m /boot/System.map > catured_profile
> captured_profile
--
~Randy
> These: ^------------v should be the same value (as you have it).
> v
>| +clear echo 2 > /proc/profile
> man page says to use "readprofile -r". Doesn't that still work?
Dunno. I always have done the above ... have you been using -r with
success? If so, I'll change it.
M.
Martin J. Bligh wrote:
>>These: ^------------v should be the same value (as you have it).
>> v
>>| +clear echo 2 > /proc/profile
>>man page says to use "readprofile -r". Doesn't that still work?
>
> Dunno. I always have done the above ... have you been using -r with
> success? If so, I'll change it.
It's what I've always used.
--
Dave Hansen
[email protected]
On Fri, 28 Feb 2003 10:13:51 -0800
Dave Hansen <[email protected]> wrote:
| Martin J. Bligh wrote:
| >>These: ^------------v should be the same value (as you have it).
| >> v
| >>| +clear echo 2 > /proc/profile
| >>man page says to use "readprofile -r". Doesn't that still work?
| >
| > Dunno. I always have done the above ... have you been using -r with
| > success? If so, I'll change it.
|
| It's what I've always used.
Same here (-r).
--
~Randy
OK, fixed a couple of things ... thanks for the feedback everyone.
diff -purN -X /home/mbligh/.diff.exclude virgin/Documentation/basic_profiling.txt prof_docs/Documentation/basic_profiling.txt
--- virgin/Documentation/basic_profiling.txt Wed Dec 31 16:00:00 1969
+++ prof_docs/Documentation/basic_profiling.txt Fri Feb 28 13:44:11 2003
@@ -0,0 +1,44 @@
+These instructions are deliberately very basic. If you want something clever,
+go read the real docs ;-) Please don't add more stuff, but feel free to
+correct my mistakes ;-) ([email protected])
+Thanks to John Levon and Dave Hansen for help writing this.
+
+<test> is the thing you're trying to measure.
+Make sure you have the correct System.map / vmlinux referenced!
+IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
+to copy config files, system.map, vmlinux to /boot.
+
+Readprofile
+-----------
+get readprofile binary fixed for 2.5 / akpm's 2.5 patch from
+ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
+add "profile=2" to the kernel command line.
+
+clear readprofile -r
+ <test>
+dump output readprofile -m /boot/System.map > catured_profile
+
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3 opcontrol --setup --vmlinux=/boot/vmlinux \
+ --ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athlon/x86-64 opcontrol --setup --vmlinux=/boot/vmlinux \
+ --ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4 opcontrol --setup --vmlinux=/boot/vmlinux \
+ --ctr0-event=GLOBAL_POWER_EVENTS \
+ --ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon opcontrol --start-daemon
+clear opcontrol --reset
+start opcontrol --start
+ <test>
+stop opcontrol --stop
+dump output oprofpp -dl -i /boot/vmlinux > output_file
+
On Fri, Feb 28, 2003 at 12:12:42AM -0800, Martin J. Bligh wrote:
> +Readprofile
> +-----------
> +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from
> +ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
> +add "profile=2" to the kernel command line.
> +
> +clear echo 2 > /proc/profile
As far as I can see, the 2 is meaningless here. This should just be
echo > /proc/profile
(On SMP when writing sizeof(int) bytes, the value written
is significant. But 1 or 2 is not sizeof(int).)
Andries
-----
Fragment of some notes:
<sect1>Profiling the kernel<p>
There are several facilities to see where the kernel spends
its resources. A simple one is the profiling function, that
stores the current EIP (instruction pointer) at each clock tick.
<p>
Boot the kernel with command line option <tt>profile=2</tt>
(or some other number instead of 2). This will cause
a file <tt>/proc/profile</tt> to be created.
The number given after <tt>profile=</tt> is the number of positions
EIP is shifted right when profiling. So a large number gives a
coarse profile.
The counters are reset by writing to <tt>/proc/profile</tt>.
The utility <tt>readprofile</tt> will output statistics for you.
It does not sort - you have to invoke <tt>sort</tt> explicitly.
But given a memory map it will translate addresses to kernel symbols.
<p>
See <tt>kernel/profile.c</tt> and <tt>fs/proc/proc_misc.c</tt>
and <tt>readprofile(1)</tt>.
<p>
For example:
<verb>
# echo > /proc/profile
...
# readprofile -m System.map-2.5.59 | sort -nr | head -2
510502 total 0.1534
508548 default_idle 10594.7500
</verb>
> Alpha is supported too... (as least I saw the kernel part go in)
sparc64 and ppc64 too, but only timer tick profiling so far.
Anton
>> Alpha is supported too... (as least I saw the kernel part go in)
>
> sparc64 and ppc64 too, but only timer tick profiling so far.
I need the magic incantation for oprofile for these then ...
Actually, better still, we need a wrapper script that works out this
from /proc/cpuinfo and auto-sets it up for you, if someone who knows
enough about different Pentium types knows how ... I'm happy to go
write it if it's easy to detect ...
My laptop says:
model name : Pentium III (Coppermine)
but a list of pattern matches from someone (or list of people) who
know how this works would be very helpful (ie matching up cpuinfo
to magic incantation to oprofile).
M.
Martin J. Bligh wrote:
>>>Alpha is supported too... (as least I saw the kernel part go in)
>>
>>sparc64 and ppc64 too, but only timer tick profiling so far.
>
> I need the magic incantation for oprofile for these then ...
>
> Actually, better still, we need a wrapper script that works out this
> from /proc/cpuinfo and auto-sets it up for you, if someone who knows
> enough about different Pentium types knows how ... I'm happy to go
> write it if it's easy to detect ...
>
> My laptop says:
>
> model name : Pentium III (Coppermine)
>
> but a list of pattern matches from someone (or list of people) who
> know how this works would be very helpful (ie matching up cpuinfo
> to magic incantation to oprofile).
Actually, oprofile already does this somehow. opcontrol --list-events
will tell you the perf counters for your particular CPU. When my
scripts are deciding whether the machine is P3/4, they grep through this
list.
I think the proper way to do this is in oprofile itself. John mentioned
earlier in this thread that they have considered doing default events,
but there may be a better way to do it.
--
Dave Hansen
[email protected]
OK, fixed a couple more things (spelling, and pointer to new tools).
I don't really want to add a list of 200 arches, but I'll do a script
with any info people email me the cpuinfo output and correct oprofile
setting for ... I'll do that for the existing ones instead of what's
there now as well.
diff -purN -X /home/mbligh/.diff.exclude virgin/Documentation/basic_profiling.txt prof_docs/Documentation/basic_profiling.txt
--- virgin/Documentation/basic_profiling.txt Wed Dec 31 16:00:00 1969
+++ prof_docs/Documentation/basic_profiling.txt Fri Feb 28 15:04:17 2003
@@ -0,0 +1,48 @@
+These instructions are deliberately very basic. If you want something clever,
+go read the real docs ;-) Please don't add more stuff, but feel free to
+correct my mistakes ;-) ([email protected])
+Thanks to John Levon and Dave Hansen for help writing this.
+
+<test> is the thing you're trying to measure.
+Make sure you have the correct System.map / vmlinux referenced!
+IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
+to copy config files, system.map, vmlinux to /boot.
+
+Readprofile
+-----------
+You need a fixed readprofile command for 2.5 ... either get hold of
+a current version from:
+http://www.kernel.org/pub/linux/utils/util-linux/
+or get readprofile binary fixed for 2.5 / akpm's 2.5 patch from
+ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/tools/readprofile/
+
+Add "profile=2" to the kernel command line.
+
+clear readprofile -r
+ <test>
+dump output readprofile -m /boot/System.map > captured_profile
+
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3 opcontrol --setup --vmlinux=/boot/vmlinux \
+ --ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athlon/x86-64 opcontrol --setup --vmlinux=/boot/vmlinux \
+ --ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4 opcontrol --setup --vmlinux=/boot/vmlinux \
+ --ctr0-event=GLOBAL_POWER_EVENTS \
+ --ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon opcontrol --start-daemon
+clear opcontrol --reset
+start opcontrol --start
+ <test>
+stop opcontrol --stop
+dump output oprofpp -dl -i /boot/vmlinux > output_file
+
On Fri, Feb 28, 2003 at 03:11:21PM -0800, Martin J. Bligh wrote:
> +start daemon opcontrol --start-daemon
--start implies starting the daemon if it isn't started
already.
> +stop opcontrol --stop
--stop unsupported. use "--shutdown"
> +dump output oprofpp -dl -i /boot/vmlinux > output_file
opcontrol --dump
op_time -l is also worth adding to this doc imo.
Dave
On Sat, Mar 01, 2003 at 05:51:14PM +0000, Dave Jones wrote:
> > +start daemon opcontrol --start-daemon
>
> --start implies starting the daemon if it isn't started
> already.
>
> > +stop opcontrol --stop
>
> --stop unsupported. use "--shutdown"
Stop running 2.4 ! :)
--stop works on 2.5 only ...
regards
john
> --start implies starting the daemon if it isn't started
> already.
I think John suggested to do those seperately, to minimise overhead
or something.
> > +dump output oprofpp -dl -i /boot/vmlinux > output_file
>
> opcontrol --dump
>
> op_time -l is also worth adding to this doc imo.
OK, will add.
M.
Martin J. Bligh wrote:
>>--start implies starting the daemon if it isn't started
>>already.
>
> I think John suggested to do those seperately, to minimise overhead
> or something.
On a heavily loaded system, --start can take quite a while if the daemon
isn't already running. Before starting my machine-killing benchmark
(which I want to profile), I use --start-daemon which gets most of the
work out of the way. The subsequent --start, which occurs while the
benchmark is already steaming along, will happen quickly and with much
less impact to the benchmark results.
--
Dave Hansen
[email protected]
On Sat, Mar 01, 2003 at 08:48:57PM +0000, John Levon wrote:
> > > +stop opcontrol --stop
> > --stop unsupported. use "--shutdown"
>
> Stop running 2.4 ! :)
>
> --stop works on 2.5 only ...
That seems quite confusing. Why not make --stop an
alias for --shutdown if running on 2.4 ?
Dave
Hi!
> +Oprofile
> +--------
> +get source (I use 0.5) from http://oprofile.sourceforge.net/
> +add "poll=idle" to the kernel command line
This should be idle=poll, AFAICS.
Pavel
--
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]