LinuxLists.cc - [patch] Performance Counters for Linux, v4

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Ingo Molnar writes:

> We are pleased to announce the v4 release of our performance counters
> subsystem implementation.

Looking at the code, I am wondering what you are planning to do to
support machines that have constraints on what sets of events can be
counted simultaneously. Currently you have the core code calling
counter->hw_ops->hw_perf_counter_enable which can't return an error.
The core expects it to be able to add any counter regardless of what
event it's counting, subject only to a maximum number of counters.
I assume you're going to change that.

I think the core should put together a list of counters and counter
groups that it would like to have on the PMU simultaneously and then
make one call to the arch layer to ask if that is possible. That
could either return success or failure. If it returns failure then
the core needs to ask for something less, or something different. I'm
not sure how the core should choose what to ask for instead, though.

Paul.

2008-12-15 12:12:24

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Ingo Molnar writes:

> For example, a full kernel build's statistics on a 16-way x86 box are:
>
> $ timec -e -5,-4,-3,1,2,3,5 make -j32 bzImage
>
> Performance counter stats for 'make':
>
> 142420.882 task clock ticks (millisecs)
>
> 9951033 pagefaults (events)
> 302628 context switches (events)
> 57810 CPU migrations (events)
> 208439082509 instructions (events)
> 657918810 cache references (events)
> 120243697 cache misses (events)
> 3134162468 branch misses (events)

Does this machine have sufficient hardware counters to count those
four hardware events at the same time? Or were those counters
timeshared onto 1 or 2 hardware counters? If it's the latter, are
those counts from half or a quarter of the total execution?

Paul.

2008-12-15 17:39:53

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Hello

I see a large (2300 instruction) fixed overhead when measuring
retired instruction count using the "timec" command
compared to the "pfmon" tool that comes with perfmon3
(the pfmon tool has essentially no overhead when
doing aggragate counts).

Is this an inherent weakness with the new proposed performance
counter infrastructure?

I wanted to compare perfmon3 against Ingo's proposed
performance counter infrastructure. This is on
a Core2 Q6600 (the only machine I have that supports
Ingo's codebase).

For perfmon3 comparison, it's the same machine running
2.6.27.4 patched with the appropriate full (not stripped-down)
perfmon3 patchset available from perfmon2.sf.net.

All code for these tests can be had from:
http://www.csl.cornell.edu/~vince/projects/perf_counter/

#
# 100 instruction test
#

Testing with a 100 instruction assembly program:

# perfmon3

tasse:~/assembly_tests% pfmon -e INSTRUCTIONS_RETIRED ./100_insns
100 INSTRUCTIONS_RETIRED

# Ingo

tasse:~/assembly_tests% ./timec -e 1 ./100_insns

Performance counter stats for './100_insns':

0.762 task clock ticks (millisecs)

2446 instructions (events)

As we can see, timec overcounts by a lot! Is it 24x, or
a fixed value?

#
# 8 billion instruction comparison
#

# perfmon3

tasse:~/assembly_tests% time pfmon -e INSTRUCTIONS_RETIRED ./8B_insns
8000000440 INSTRUCTIONS_RETIRED
1.77s user 0.00s system 100% cpu 1.771 total

Note that on almost all x86 chips that any hardware interrupt that
occurs adds an extra retired instruction to the total count
(some AMD engineers told me this is probably due to some artifact
due to long pipelines and how the microcode changes user/kernel
flag).

So you see that in 1.77s we acccumulate 1.77s*250Hz timer interrupts
which is 442.5 which is roughly the extra instructions we see.

(for more info on sources of non-determinism in instruction counting
with performance counters see the paper here:
http://www.csl.cornell.edu/~vince/papers/iiswc08 )

# ingo

tasse:~/assembly_tests% ./timec -e 1 ./8B_insns

Performance counter stats for './8B_insns':

1743.446 task clock ticks (millisecs)

8000002799 instructions (events)

So it turns out the overhead isn't 24x, but is actually
a fixed 2300 or so.

Still, that's overhead perfmon does not have.

Will this be fixed, or is it an inherent limitation of
the new proposal?

Vince

2008-12-15 21:02:42

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Hello

I'm trying a more complicated benchmark and getting even stranger
results.

This is still on the Q6600 machine

The benchmark does a loop, reading some memory. It should have
roughly:
12295 instructions
4096 memory loads
4096 branches

perfmon3 is close on all of these stats, and this is consistent
across runs with a small variation (+/- 3 or so).

The timec program returns 0 (!) for all of the stats except
retired instruction count! And with certain combinations
of counters I get 0 for all counts. No error messages
are printed.

Is this expected behavior?

The test program can be had from:
http://www.csl.cornell.edu/~vince/projects/perf_counter/

Details below:

#
# Perfmon results
#

# First, trying to read all 5 events at once fails, only 4 counters
# avail

tasse:~/assembly_tests% pfmon -e INSTRUCTIONS_RETIRED,BRANCH_INSTRUCTIONS_RETIRED,L1D_ALL_CACHE_REF,MEM_LOAD_RETIRED:L1D_MISS ./read_test
cannot configure events: set0 events incompatible or too many events

# Cache results are close to expected, L1D looks a little high

tasse:~/assembly_tests% pfmon -e INSTRUCTIONS_RETIRED,L1D_ALL_CACHE_REF,MEM_LOAD_RETIRED:L1D_MISS ./read_test
12299 INSTRUCTIONS_RETIRED
4164 L1D_ALL_CACHE_REF
4 MEM_LOAD_RETIRED:L1D_MISS

# Branch results. Close to what they should be, though a bit higher
# than expected.

tasse:~/assembly_tests% pfmon -e INSTRUCTIONS_RETIRED,BRANCH_INSTRUCTIONS_RETIRED,MISPREDICTED_BRANCH_RETIRED ./read_test
12299 INSTRUCTIONS_RETIRED
4102 BRANCH_INSTRUCTIONS_RETIRED
1 MISPREDICTED_BRANCH_RETIRED

#
# performance counter v4
#

# Including all stats gives no errors, but gives no results
# either

tasse:~/assembly_tests% ./timec -e 0 -e 1 -e 2 -e 3 -e 4 -e 5 ./read_test

Performance counter stats for './read_test':

0.716 task clock ticks (millisecs)

85049 cycles (events)
0 instructions (events)
0 cache references (events)
0 cache misses (events)
0 branches (events)
0 branch misses (events)

#
# If I include the cycles count, I consistently get 0
# for all counts???

tasse:~/assembly_tests% ./timec -e 0 -e 1 -e 2 -e 3 ./read_test

Performance counter stats for './read_test':

0.520 task clock ticks (millisecs)

73833 cycles (events)
0 instructions (events)
0 cache references (events)
0 cache misses (events)

#
# If I drop the cycles count, I get an instruction count
# with a value 2300 too high (see previous e-mail)
# And really low cache values.

tasse:~/assembly_tests% ./timec -e 1 -e 2 -e 3 ./read_test

Performance counter stats for './read_test':

0.723 task clock ticks (millisecs)

14644 instructions (events)
8 cache references (events)
0 cache misses (events)

#
# And the branch stats don't work either
#

tasse:~/assembly_tests% ./timec -e 1 -e 4 -e 5 ./read_test

Performance counter stats for './read_test':

0.711 task clock ticks (millisecs)

14643 instructions (events)
0 branches (events)
0 branch misses (events)

2008-12-15 21:42:58

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Vince Weaver writes:

> I see a large (2300 instruction) fixed overhead when measuring
> retired instruction count using the "timec" command
> compared to the "pfmon" tool that comes with perfmon3
> (the pfmon tool has essentially no overhead when
> doing aggragate counts).

Looks like timec will be counting the fork() and execvp() system calls
that are used to run your executable, as well as the executable
itself. The fork() overhead could be removed fairly easily I think,
the execvp would be hard to get rid of without using ptrace() - and
the use of ptrace was one of the things that Ingo et al. objected to
in perfmon3.

Paul.

2008-12-15 22:09:20

by Stephane Eranian

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

On Mon, Dec 15, 2008 at 10:42 PM, Paul Mackerras <[email protected]> wrote:
> Vince Weaver writes:
>
>> I see a large (2300 instruction) fixed overhead when measuring
>> retired instruction count using the "timec" command
>> compared to the "pfmon" tool that comes with perfmon3
>> (the pfmon tool has essentially no overhead when
>> doing aggragate counts).
>
> Looks like timec will be counting the fork() and execvp() system calls
> that are used to run your executable, as well as the executable
> itself. The fork() overhead could be removed fairly easily I think,
> the execvp would be hard to get rid of without using ptrace() - and
> the use of ptrace was one of the things that Ingo et al. objected to
> in perfmon3.
>
Paul, I think your analysis is correct. This is likely what is happening.

Not that timec could not use ptrace() to block the task from executing
its first instruction, but you'd still have a problem because of prctl(ENABLE)
which applies to the current task, not another task, unless I am mistaken.

Prctl() looks odd to me because you have all those supposedly independent
file descriptors to identify events you want to measure, but they are not
used to start/stop. If you are attaching to multiple tasks at the same time
which you can do with the current API, you may not necessarily want to
start/stop all counters at the same time.

Looks like prctl() is not what we want after all...

2008-12-15 22:53:58

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Vince Weaver writes:

> I'm trying a more complicated benchmark and getting even stranger
> results.
>
> This is still on the Q6600 machine
>
> The benchmark does a loop, reading some memory. It should have
> roughly:
> 12295 instructions
> 4096 memory loads
> 4096 branches
>
> perfmon3 is close on all of these stats, and this is consistent
> across runs with a small variation (+/- 3 or so).
>
> The timec program returns 0 (!) for all of the stats except
> retired instruction count! And with certain combinations
> of counters I get 0 for all counts. No error messages
> are printed.
>
> Is this expected behavior?

When you have more software counters than hardware counters, the
kernel will be time-slicing the counters, but because your program
doesn't run for very long, it's possible that only the first set will
ever get to count anything.

That doesn't seem to explain all the 0 values (since the kernel should
be able to use at least 2 hardware counters on your machine, I expect)
but it might be part of the explanation.

Paul.

2008-12-16 12:22:56

by Pavel Machek

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

On Sun 2008-12-14 22:28:29, Ingo Molnar wrote:
> We are pleased to announce the v4 release of our performance counters
> subsystem implementation. The kernel changes can be picked up from:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git perfcounters/core
>
> (also in the master branch. There's also a kernel patch attached
> below.)
>
> The biggest new feature in this release is the implementation of
> "performance counter inheritance" for the per task counters: the ability
> to extend performance counters to cover the execution of child tasks
> too, transparently and automatically - following them to other CPUs.
>
> This can be used to monitor a hierarchy of tasks without stopping them
> (or impacting them in any observable way), and extending that monitoring
> to all child tasks as well.
>
> We've written a new utility: 'timec', which takes advantage of this new
> kernel capability:
>
> http://redhat.com/~mingo/perfcounters/timec.c
>
> 'timec' works like /usr/bin/time, but it extends the dimension of "time"
> with all the metrics that hardware and software performance counters are
> able to capture.

Hmm, if I timec some setuid program, what happens?

Performance counters seem like great tool to pull secret keys out of
other processes :-).
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

2008-12-16 12:50:43

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

* Pavel Machek <[email protected]> wrote:

> Hmm, if I timec some setuid program, what happens?

yes, i already had a quick look at that a few days ago when i implemented
counter inheritance (for different reasons) and couldnt find the cleanest
place to put the exec() flushing into so i procrastinated that a bit :)

> Performance counters seem like great tool to pull secret keys out of
> other processes :-).

if you worry about _that_ angle you also have to:

- turn off the cycle counter

- turn off precise utimes

- plus you have to forbid SMT CPUs as well. On HT a task could
co-schedule with your setuid task and observe its timing
characteristics via its _own_ behavior. (which is impacted by whatever
is running on another SMT/HT thread.)

the real exec() worry are: active, IRQ driven samples/events. Not possible
yet via the current iteration of counter inheritance (hence my
procrastination) - but it makes sense and that's why i was looking at the
exec() angle.

and that will flush simple counters too, removing your theoretical attack
angle as well.

So how about the patch below?

Ingo

--------------->
Subject: perfcounters: flush on setuid exec
From: Ingo Molnar <[email protected]>
Date: Tue Dec 16 13:40:44 CET 2008

Pavel Machek pointed out that performance counters should be flushed
when crossing protection domains on setuid execution.

Reported-by: Pavel Machek <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
---
fs/exec.c | 8 ++++++++
1 file changed, 8 insertions(+)

Index: linux/fs/exec.c
===================================================================
--- linux.orig/fs/exec.c
+++ linux/fs/exec.c
@@ -33,6 +33,7 @@
#include <linux/string.h>
#include <linux/init.h>
#include <linux/pagemap.h>
+#include <linux/perf_counter.h>
#include <linux/highmem.h>
#include <linux/spinlock.h>
#include <linux/key.h>
@@ -1015,6 +1016,13 @@ int flush_old_exec(struct linux_binprm *
set_dumpable(current->mm, suid_dumpable);
}

+ /*
+ * Flush performance counters when crossing a
+ * security domain:
+ */
+ if (!get_dumpable(current->mm))
+ perf_counter_exit_task(current);
+
/* An exec changes our domain. We are no longer part of the thread
group */

2008-12-16 12:57:42

by Pavel Machek

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

On Tue 2008-12-16 13:50:00, Ingo Molnar wrote:
>
> * Pavel Machek <[email protected]> wrote:
>
> > Hmm, if I timec some setuid program, what happens?
>
> yes, i already had a quick look at that a few days ago when i implemented
> counter inheritance (for different reasons) and couldnt find the cleanest
> place to put the exec() flushing into so i procrastinated that a bit :)
>
> > Performance counters seem like great tool to pull secret keys out of
> > other processes :-).
>
> if you worry about _that_ angle you also have to:
>
> - turn off the cycle counter
>
> - turn off precise utimes

Probably good idea, yes.

> - plus you have to forbid SMT CPUs as well. On HT a task could
> co-schedule with your setuid task and observe its timing
> characteristics via its _own_ behavior. (which is impacted by whatever
> is running on another SMT/HT thread.)

Yes, SMT is evil.

> the real exec() worry are: active, IRQ driven samples/events. Not possible
> yet via the current iteration of counter inheritance (hence my
> procrastination) - but it makes sense and that's why i was looking at the
> exec() angle.
>
> and that will flush simple counters too, removing your theoretical attack
> angle as well.
>
> So how about the patch below?

Thanks!

> Subject: perfcounters: flush on setuid exec
> From: Ingo Molnar <[email protected]>
> Date: Tue Dec 16 13:40:44 CET 2008
>
> Pavel Machek pointed out that performance counters should be flushed
> when crossing protection domains on setuid execution.
>
> Reported-by: Pavel Machek <[email protected]>
> Signed-off-by: Ingo Molnar <[email protected]>

Acked-by: Pavel Machek <[email protected]>

> @@ -1015,6 +1016,13 @@ int flush_old_exec(struct linux_binprm *
> set_dumpable(current->mm, suid_dumpable);
> }
>
> + /*
> + * Flush performance counters when crossing a
> + * security domain:
> + */
> + if (!get_dumpable(current->mm))
> + perf_counter_exit_task(current);
> +
> /* An exec changes our domain. We are no longer part of the thread
> group */
>

--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

2008-12-16 13:03:33

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

* Pavel Machek <[email protected]> wrote:

> On Tue 2008-12-16 13:50:00, Ingo Molnar wrote:
> >
> > * Pavel Machek <[email protected]> wrote:
> >
> > > Hmm, if I timec some setuid program, what happens?
> >
> > yes, i already had a quick look at that a few days ago when i implemented
> > counter inheritance (for different reasons) and couldnt find the cleanest
> > place to put the exec() flushing into so i procrastinated that a bit :)
> >
> > > Performance counters seem like great tool to pull secret keys out of
> > > other processes :-).
> >
> > if you worry about _that_ angle you also have to:
> >
> > - turn off the cycle counter
> >
> > - turn off precise utimes
>
> Probably good idea, yes.
>
> > - plus you have to forbid SMT CPUs as well. On HT a task could
> > co-schedule with your setuid task and observe its timing
> > characteristics via its _own_ behavior. (which is impacted by whatever
> > is running on another SMT/HT thread.)
>
> Yes, SMT is evil.

HT got added back to Nehalem, so SMT is coming to you in every future x86
CPU. It brings a serious performance win, so nobody will turn off SMT
threading in practice. If SMT worries you, it needs explicit partitioning
of security-relevant processing to different physical CPUs, via
cgroups/cpusets/etc.

> > the real exec() worry are: active, IRQ driven samples/events. Not possible
> > yet via the current iteration of counter inheritance (hence my
> > procrastination) - but it makes sense and that's why i was looking at the
> > exec() angle.
> >
> > and that will flush simple counters too, removing your theoretical attack
> > angle as well.
> >
> > So how about the patch below?
>
> Thanks!
>
> > Subject: perfcounters: flush on setuid exec
> > From: Ingo Molnar <[email protected]>
> > Date: Tue Dec 16 13:40:44 CET 2008
> >
> > Pavel Machek pointed out that performance counters should be flushed
> > when crossing protection domains on setuid execution.
> >
> > Reported-by: Pavel Machek <[email protected]>
> > Signed-off-by: Ingo Molnar <[email protected]>
>
> Acked-by: Pavel Machek <[email protected]>

find below the final commit, thanks Pavel.

Ingo

------------>
>From f65cb45cba63f249458b669aa67069eabc37b2f5 Mon Sep 17 00:00:00 2001
From: Ingo Molnar <[email protected]>
Date: Tue, 16 Dec 2008 13:40:44 +0100
Subject: [PATCH] perfcounters: flush on setuid exec

Pavel Machek pointed out that performance counters should be flushed
when crossing protection domains on setuid execution.

Reported-by: Pavel Machek <[email protected]>
Acked-by: Pavel Machek <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
---
fs/exec.c | 8 ++++++++
1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/fs/exec.c b/fs/exec.c
index ec5df9a..d5165d8 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -33,6 +33,7 @@
#include <linux/string.h>
#include <linux/init.h>
#include <linux/pagemap.h>
+#include <linux/perf_counter.h>
#include <linux/highmem.h>
#include <linux/spinlock.h>
#include <linux/key.h>
@@ -1017,6 +1018,13 @@ int flush_old_exec(struct linux_binprm * bprm)
set_dumpable(current->mm, suid_dumpable);
}

+ /*
+ * Flush performance counters when crossing a
+ * security domain:
+ */
+ if (!get_dumpable(current->mm))
+ perf_counter_exit_task(current);
+
/* An exec changes our domain. We are no longer part of the thread
group */

2008-12-16 13:13:48

by Arjan van de Ven

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

On Tue, 16 Dec 2008 14:03:02 +0100
Ingo Molnar <[email protected]> wrote:

> > > - plus you have to forbid SMT CPUs as well. On HT a task could
> > > co-schedule with your setuid task and observe its timing
> > > characteristics via its _own_ behavior. (which is impacted by
> > > whatever is running on another SMT/HT thread.)
> >
> > Yes, SMT is evil.
>
> HT got added back to Nehalem, so SMT is coming to you in every future
> x86 CPU. It brings a serious performance win, so nobody will turn off
> SMT threading in practice. If SMT worries you, it needs explicit
> partitioning of security-relevant processing to different physical
> CPUs, via cgroups/cpusets/etc.

and/or you use properly implemented crypto code (see Bruce Schneider's
books). The timing "problem" isn't really SMT specific. If you have
improperly implemented crypto (eg crypto code where the code paths and
not just the data payload are key dependent) then on any system with
more than one (logical) processor there is interference that an
attacker can use.

The only possible answer is to use proper implementation; turning off HT
may make you feel good but you go from shoddy crypto for which there is
some internet papers on how to crack it, to shoddy crypto for which the
same papers apply ;)

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2008-12-16 14:23:30

by Peter Zijlstra

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

On Mon, 2008-12-15 at 23:11 +1100, Paul Mackerras wrote:
> Ingo Molnar writes:
>
> > We are pleased to announce the v4 release of our performance counters
> > subsystem implementation.
>
> Looking at the code, I am wondering what you are planning to do to
> support machines that have constraints on what sets of events can be
> counted simultaneously. Currently you have the core code calling
> counter->hw_ops->hw_perf_counter_enable which can't return an error.
> The core expects it to be able to add any counter regardless of what
> event it's counting, subject only to a maximum number of counters.
> I assume you're going to change that.
>
> I think the core should put together a list of counters and counter
> groups that it would like to have on the PMU simultaneously and then
> make one call to the arch layer to ask if that is possible. That
> could either return success or failure. If it returns failure then
> the core needs to ask for something less, or something different. I'm
> not sure how the core should choose what to ask for instead, though.

I think the constraint set should be applied when we add to a group, if
when we add a counter to the group, the result isn't schedulable
anymore, we should fail the group addition - and thereby the counter
creation.

This would leave us with groups that are always schedulable in an atomic
fashion.

>From what I understand the code RRs groups (co-scheduling groups where
possible) (ungrouped counter is a group of one), this means that with
the above addition you'd have the needed control over things.

If you need things to be atomic, create a single group, if you're fine
with RR time-sharing, create multiple.

This seems to leave a hole where multiple monitors collide and create
multiple groups unaware of each-other - could we plug this hole with a
group attribute?

2008-12-16 14:43:52

by Peter Zijlstra

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

On Tue, 2008-12-16 at 08:42 +1100, Paul Mackerras wrote:
> the execvp would be hard to get rid of without using ptrace() - and
> the use of ptrace was one of the things that Ingo et al. objected to
> in perfmon3.

I don't think using ptrace in this case is a big issue - aside from the
fact that ptrace is crap in that you'd not be able to timec from a
debugger context :-(

The biggest objection to using ptrace was that ptrace was needed
_during_ the execution of the monitored load, thereby distorting the
load.

This case is different in that it would be used to start off the load.

Still it would be good if we could find another (elegant) way to fix
this.

Also, I'm pretty sure the regular 'time' suffers the very same issue and
counts the exec syscall as well - I saw that when I tinkered with the
execve argument code.

Furthermore, I think output of tools such as time and now timec are most
relevant when compared between runs - that is, the change in values
between runs, not the absolute values as such. At least, that's what I
usually do:

time ./foo

tinker with foo.c

time ./foo

if time2 < time1 :-)
else :-(

2008-12-16 14:46:00

by Peter Zijlstra

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Could people fix this perfctl-devel list thingy to not generate silly
bounces - otherwise I'm going to drop it from the CC on further emails.

---
The reason it is being held:

Too many recipients to the message

2008-12-16 15:55:48

by Martin Cracauer

[permalink] [raw]

Subject: Re: [Perfctr-devel] [patch] Performance Counters for Linux, v4

Ingo Molnar wrote on Sun, Dec 14, 2008 at 10:28:29PM +0100:
> We are pleased to announce the v4 release of our performance counters
> subsystem implementation. The kernel changes can be picked up from:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git perfcounters/core
>
> (also in the master branch. There's also a kernel patch attached
> below.)
>
> The biggest new feature in this release is the implementation of
> "performance counter inheritance" for the per task counters: the ability
> to extend performance counters to cover the execution of child tasks
> too, transparently and automatically - following them to other CPUs.

Does this come with a PAPI frontend or is one planned?

I picked the old perfctr mainly because I need PAPI. Outside-process
measurement doesn't do for my application, I need subdivisions in my
code.

Thanks
Martin
--
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <[email protected]> http://www.cons.org/cracauer/

2008-12-16 16:50:52

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

> On Tue, 2008-12-16 at 08:42 +1100, Paul Mackerras wrote:
> Furthermore, I think output of tools such as time and now timec are most
> relevant when compared between runs - that is, the change in values
> between runs, not the absolute values as such. At least, that's what I
> usually do:

That's doesn't do you any good when comparing results across different
machines, or even different kernels on the same machine.

perfmon shows that good results can be had, even if it's not the cleanest
way in the world. It would be a shame to lose that.

Small micro-benchmarks like this are important. You can't always trust
the performance counters to work, so being able to sanity check them with
exact test-cases is critical. Otherwise you might just be measuring
nonsense.

And while it might be able to subtract the exec() overhead for something
like retired instructions, it gets a lot more complicated when you have
something like cache bus snoops or branch mispredicts where it's hard to
tell what comes from the program and what is overhead from the monitoring
infrastructure.

Vince

2008-12-16 17:34:24

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

I'm trying to evaluate this new proposal for the kind of workloads I use
performance counters for, and even the simplest tests don't work.

I'm trying to do a simple aggragate count for some benchmarks here using
timec and I'm getting poor results.

Are any of the problems I'm reporting going to be fixed?

In any case, I was testing aggregate counts on a longer running benchmark,
this time equake from the spec2k benchmark suite, still on the q6600.

If I only count retired instructions, I get consistent results:

timec -e 1

119175255369 instructions (events)
119175255561 instructions (events)
119175255383 instructions (events)

however the minute I add another count, say cycles so I can calculate
CPI/IPC the results for instructions are suddenly off by 33%.

Needless to say, perfmon can handle reading both cycles and instructions
at the same time.

timec -e 0, -e 1
91758816320 cycles (events)
79428247907 instructions (events)

91849140396 cycles (events)
79449560742 instructions (events)

It gets worse when trying to look at cache statistics:

timec -e 1 -e 2 -e 3

59611457943 instructions (events)
1872499771 cache references (events)
97471971 cache misses (events)

59601907232 instructions (events)
1871766376 cache references (events)
97435199 cache misses (events)

and so on

timec -e1 -e2 -e3 -e4

47671703285 instructions (events)
1498246999 cache references (events)
77838085 cache misses (events)
3394839360 branches (events)

47666131604 instructions (events)
1497069685 cache references (events)
78065325 cache misses (events)
3393244879 branches (events)

So apparently this performance counter infrastructure will always be
useless for trying to get plain aggregate counts? It's the simplest case
to get right, so it makes me wonder about the design of the rest of the
infrastructure.

Vince

2008-12-16 19:48:04

by Corey Ashford

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Vince Weaver wrote:
>
> I'm trying to evaluate this new proposal for the kind of workloads I use
> performance counters for, and even the simplest tests don't work.
>
> I'm trying to do a simple aggragate count for some benchmarks here using
> timec and I'm getting poor results.
>
> Are any of the problems I'm reporting going to be fixed?
>
> In any case, I was testing aggregate counts on a longer running
> benchmark, this time equake from the spec2k benchmark suite, still on
> the q6600.
>
> If I only count retired instructions, I get consistent results:
>
> timec -e 1
>
> 119175255369 instructions (events)
> 119175255561 instructions (events)
> 119175255383 instructions (events)
>
>
> however the minute I add another count, say cycles so I can calculate
> CPI/IPC the results for instructions are suddenly off by 33%.
>
> Needless to say, perfmon can handle reading both cycles and instructions
> at the same time.
>
>
> timec -e 0, -e 1
> 91758816320 cycles (events)
> 79428247907 instructions (events)
>
> 91849140396 cycles (events)
> 79449560742 instructions (events)
>
>
> It gets worse when trying to look at cache statistics:
>
> timec -e 1 -e 2 -e 3
>
> 59611457943 instructions (events)
> 1872499771 cache references (events)
> 97471971 cache misses (events)
>
> 59601907232 instructions (events)
> 1871766376 cache references (events)
> 97435199 cache misses (events)
>
> and so on
>
> timec -e1 -e2 -e3 -e4
>
>
> 47671703285 instructions (events)
> 1498246999 cache references (events)
> 77838085 cache misses (events)
> 3394839360 branches (events)
>
> 47666131604 instructions (events)
> 1497069685 cache references (events)
> 78065325 cache misses (events)
> 3393244879 branches (events)
>
>
>
> So apparently this performance counter infrastructure will always be
> useless for trying to get plain aggregate counts? It's the simplest
> case to get right, so it makes me wonder about the design of the rest of
> the infrastructure.
>
> Vince

Your test case demonstrates that scaling is missing from the current
version of Performance Counters for Linux.

When each set of events is scheduled onto a set of hardware event
counters, in order to scale the results properly, a cycles counter needs
to be included in each set as well.

When the counts are read up, the counts from each set need to be scaled
by a factor of
(total cycles)/(cycles in that set)

This is something that can be handled by perfmon3 (full) because set
multiplexing is explicitly programmed, not transparent as it is in
Ingo's current code. In perfmon3, the set switching can be determined
by events counter overflow, as well as time.

In common with both perfmon3 and Ingo's solution is that as more and
more events are scheduled onto the same set of hardware registers, the
accuracy drops and has to be compensated with longer run times.

Another source of error is that if the sets are rotated across the
hardware at a fixed periodic rate, if there's any correlation between
that rate and what's going on in the program being analyzed, the results
will be dubious. Ideally, you'd want to have some sort of pseudo-random
set switching rate to mitigate this sort of problem.

If Ingo could make some sort of provision for including a cycles count
in every set, and then transparently performing the scaling, that would
make it easier to use. As it stands now, I don't think there's any way
to recover the needed scaling information, because you cannot tell what
events are in what sets and how many cycles are associated with each set.

- Corey

Corey Ashford
Software Engineer
IBM Linux Technology Center, Linux Toolchain
Beaverton, OR
503-578-3507
[email protected]

2008-12-16 19:57:16

by William Cohen

[permalink] [raw]

Subject: Re: [Perfctr-devel] [patch] Performance Counters for Linux, v4

Ingo Molnar wrote:
> We are pleased to announce the v4 release of our performance counters
> subsystem implementation. The kernel changes can be picked up from:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git perfcounters/core
>
> (also in the master branch. There's also a kernel patch attached
> below.)

PAPI also has the concept of event presets to hide some of the event
selection details. The following URL lists the presets and which
processors they are supported on:

http://icl.cs.utk.edu/projects/papi/presets.html

Looking through the PAPI preset events can see lots of variation due
to the difference in what precisely the performance monitoring
hardware support within the processor. The following events are
defined in include/linux/perf_counter.h. It would be helpful to state
what is meant by of the following events:

PERF_COUNT_CYCLES
PERF_COUNT_INSTRUCTIONS
PERF_COUNT_CACHE_REFERENCES
PERF_COUNT_CACHE_MISSES
PERF_COUNT_BRANCH_INSTRUCTIONS
PERF_COUNT_BRANCH_MISSES

PERF_COUNT_CYCLES and PERF_COUNT_INSTRUCTIONS

Is this the cpu clock rate to compute clocks per instruction (CPI)? On
some processors there are several possible sources of "cycles":

Reference clock frequency (fixed frequency, e.g. always 2.2GHz)
Cycles of processor subject to frequency changes and halts

Is PERF_COUNT_INSTRUCTIONS the instructions actually retired by the
processor and would be used with PERF_COUNT_CYCLES to estimate CPI? In
the case of a SMT (symetric multi-threaded) processor these are
going to be kept on a per-virtual-CPU basis?

PERF_COUNT_CACHE_REFERENCES and PERF_COUNT_CACHE_MISSES

PERF_COUNT_CACHE_REFERENCES and PERF_COUNT_CACHE_MISSES are not single
monolitic events on many processors. There are multiple cache
levels. The L1 cache most processors have separate instruction and
data caches and require multiple counters to implement. Would these
refer to the last level of cache before memory and just be used to
compute the hit/miss rate for that last level? Some processors in the
same family have L2 and some processors have L3 cache. The setup code
would need to distinguish between these processor variants.

What memory references and misses are included and excluded in cache operation
counts? TLB accesses? Cache eviction/snooping operations?

PERF_COUNT_BRANCH_INSTRUCTIONS and PERF_COUNT_BRANCH_MISSES

Assume the PERF_COUNT_BRANCH_INSTRUCTIONS and PERF_COUNT_BRANCH_MISSES
refer only to retired instructions are are used to compute branch
misprediction ratio. Speculative instructions don't count for these numbers.

Any thought to including events that are not in the Intel architected events
such as ITLB/DTLB accesses and misses?

-Will

2008-12-16 20:04:51

by Pavel Machek

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

On Tue 2008-12-16 14:13:30, Arjan van de Ven wrote:
> On Tue, 16 Dec 2008 14:03:02 +0100
> Ingo Molnar <[email protected]> wrote:
>
> > > > - plus you have to forbid SMT CPUs as well. On HT a task could
> > > > co-schedule with your setuid task and observe its timing
> > > > characteristics via its _own_ behavior. (which is impacted by
> > > > whatever is running on another SMT/HT thread.)
> > >
> > > Yes, SMT is evil.
> >
> > HT got added back to Nehalem, so SMT is coming to you in every future
> > x86 CPU. It brings a serious performance win, so nobody will turn
> > off

Fortunately, Intel is not the only x86 vendor :-).

> > SMT threading in practice. If SMT worries you, it needs explicit
> > partitioning of security-relevant processing to different physical
> > CPUs, via cgroups/cpusets/etc.

I guess we should refuse to run threads from different uids on one
physical core...

> and/or you use properly implemented crypto code (see Bruce Schneider's
> books). The timing "problem" isn't really SMT specific. If you have
> improperly implemented crypto (eg crypto code where the code paths and
> not just the data payload are key dependent) then on any system with
> more than one (logical) processor there is interference that an
> attacker can use.
>
> The only possible answer is to use proper implementation; turning off HT
> may make you feel good but you go from shoddy crypto for which there is
> some internet papers on how to crack it, to shoddy crypto for which the
> same papers apply ;)

It is not only timing. If attacker has access to detailed cache miss
statistics, things are easier for him... I probably should review the
books, but even if code paths are key-independend (hard), you'll get
timing differences due to [data] cache misses...?

Ok, this is getting off-topic.

Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

2008-12-16 20:51:26

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

On Tue, 16 Dec 2008, Corey Ashford wrote:
> In common with both perfmon3 and Ingo's solution is that as more and more
> events are scheduled onto the same set of hardware registers, the accuracy
> drops and has to be compensated with longer run times.

There seems to be some confusion.

I want aggregate instruction count. I do not want any sort of scaling or
sampling.

When I cound retired instructions and cycles, I want the full counts for
those. The q6600 definitely has more than 2 counters available, so it
should be able to give me exact aggregate counts for those counters. No
sampling or scaling should be involved.

Is it not possible to get raw, aggregate count with Ingo's infrastructure?
The documentation is vague on this.

Vince

2008-12-16 21:52:44

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Vince Weaver writes:
>
> > On Tue, 2008-12-16 at 08:42 +1100, Paul Mackerras wrote:
> > Furthermore, I think output of tools such as time and now timec are most
> > relevant when compared between runs - that is, the change in values
> > between runs, not the absolute values as such. At least, that's what I
> > usually do:

Just for the record, I didn't write the quoted paragraph, Peter
Zijlstra did.

Paul.

2008-12-16 23:07:30

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Peter Zijlstra writes:

> On Mon, 2008-12-15 at 23:11 +1100, Paul Mackerras wrote:
> > I think the core should put together a list of counters and counter
> > groups that it would like to have on the PMU simultaneously and then
> > make one call to the arch layer to ask if that is possible. That
> > could either return success or failure. If it returns failure then
> > the core needs to ask for something less, or something different. I'm
> > not sure how the core should choose what to ask for instead, though.
>
> I think the constraint set should be applied when we add to a group, if
> when we add a counter to the group, the result isn't schedulable
> anymore, we should fail the group addition - and thereby the counter
> creation.
>
> This would leave us with groups that are always schedulable in an atomic
> fashion.

I agree that if adding a counter to a group results in that group not
being schedulable any more, we should fail the addition.

> >From what I understand the code RRs groups (co-scheduling groups where
> possible) (ungrouped counter is a group of one), this means that with
> the above addition you'd have the needed control over things.
>
> If you need things to be atomic, create a single group, if you're fine
> with RR time-sharing, create multiple.

I think we need a "full-time" attribute for counters and groups that
says "I need to be on the whole time", where "whole time" means
whenever the task is running, for a per-task counter or group, or
continuously for per-cpu counters/groups.

With that, the core can check at creation or enable time and return an
error if it can't fit all the full-time counters and groups on, or if
there is any part-time counter/group that would never get to go on
when all the full-time counters/groups are on. (And I guess creation
or enabling of a part-time counter or group should fail if it would
never be able to go on.)

> This seems to leave a hole where multiple monitors collide and create
> multiple groups unaware of each-other - could we plug this hole with a
> group attribute?

There could be a "whole-PMU" group attribute, that says "I need raw
access to the PMU with no other counters scheduled", which would allow
monitoring programs to use arcane PMU features that the kernel doesn't
necessarily know about. But I think that's a bit different from what
you're talking about.

The perf counter subsystem will, in Ingo's design, naturally try to
schedule as many counters and groups on as it can. Given a list of
counters/groups, it could start with the first and keep on trying to
add counters or groups while it can, essentially trying all possible
combinations until it either fills up all the hardware counters or
exhausts the possible combinations. If it moves all the
counters/groups that do fit on up to the head of the list, and then
rotates them to the back of the list when the timeslice expires, that
would probably be OK. In fact the computation about what set of
counters/groups to put on should be done when adding/removing a
counter/group and when the timeslice expires, rather than at context
switch time. (I'm talking about the list of part-time counters/groups
here, of course.)

Paul.

2008-12-16 23:51:53

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

* Paul Mackerras <[email protected]> wrote:

> Peter Zijlstra writes:
>
> > On Mon, 2008-12-15 at 23:11 +1100, Paul Mackerras wrote:

> > > I think the core should put together a list of counters and counter
> > > groups that it would like to have on the PMU simultaneously and then
> > > make one call to the arch layer to ask if that is possible. That
> > > could either return success or failure. If it returns failure then
> > > the core needs to ask for something less, or something different.
> > > I'm not sure how the core should choose what to ask for instead,
> > > though.
> >
> > I think the constraint set should be applied when we add to a group,
> > if when we add a counter to the group, the result isn't schedulable
> > anymore, we should fail the group addition - and thereby the counter
> > creation.
> >
> > This would leave us with groups that are always schedulable in an
> > atomic fashion.
>
> I agree that if adding a counter to a group results in that group not
> being schedulable any more, we should fail the addition.
>
> > > From what I understand the code RRs groups (co-scheduling groups
> > > where possible) (ungrouped counter is a group of one), this means
> > > that with the above addition you'd have the needed control over
> > > things.
> >
> > If you need things to be atomic, create a single group, if you're fine
> > with RR time-sharing, create multiple.
>
> I think we need a "full-time" attribute for counters and groups that
> says "I need to be on the whole time", where "whole time" means whenever
> the task is running, for a per-task counter or group, or continuously
> for per-cpu counters/groups.
>
> With that, the core can check at creation or enable time and return an
> error if it can't fit all the full-time counters and groups on, or if
> there is any part-time counter/group that would never get to go on when
> all the full-time counters/groups are on. (And I guess creation or
> enabling of a part-time counter or group should fail if it would never
> be able to go on.)
>
> > This seems to leave a hole where multiple monitors collide and create
> > multiple groups unaware of each-other - could we plug this hole with a
> > group attribute?
>
> There could be a "whole-PMU" group attribute, that says "I need raw
> access to the PMU with no other counters scheduled", which would allow
> monitoring programs to use arcane PMU features that the kernel doesn't
> necessarily know about. But I think that's a bit different from what
> you're talking about.
>
> The perf counter subsystem will, in Ingo's design, naturally try to
> schedule as many counters and groups on as it can. Given a list of
> counters/groups, it could start with the first and keep on trying to add
> counters or groups while it can, essentially trying all possible
> combinations until it either fills up all the hardware counters or
> exhausts the possible combinations. If it moves all the counters/groups
> that do fit on up to the head of the list, and then rotates them to the
> back of the list when the timeslice expires, that would probably be OK.
> In fact the computation about what set of counters/groups to put on
> should be done when adding/removing a counter/group and when the
> timeslice expires, rather than at context switch time. (I'm talking
> about the list of part-time counters/groups here, of course.)

yeah, something like that sounds sensible.

The principle is this: as long as it's an add-on attribute with a default
value of zero, so that normal usage does not need to care about it, it's a
reasonable extension. What we want to avoid is burdening the normal case
with the weird cases. (and none of the things you mentioned seem to be in
that category so we are fine.)

Ingo

2008-12-17 01:51:29

by Andi Kleen

[permalink] [raw]

Subject: Re: [Perfctr-devel] [patch] Performance Counters for Linux, v4

William Cohen <[email protected]> writes:
>
> PERF_COUNT_CACHE_REFERENCES and PERF_COUNT_CACHE_MISSES are not single
> monolitic events on many processors. There are multiple cache
> levels. The L1 cache most processors have separate instruction and
> data caches and require multiple counters to implement. Would these
> refer to the last level of cache before memory and just be used to
> compute the hit/miss rate for that last level? Some processors in the
> same family have L2 and some processors have L3 cache. The setup code
> would need to distinguish between these processor variants.

The difference between L1 and L3 caches can be huge (in some cases
two orders of magnitude). With that I'm not sure a single cache
miss/hit event even makes any sense.

On a modern CPU with L2 and L3 caches as soon as you fall out
of the L2 you're going to perform more poorly on parallel workloads.

With that you really have to distingush some levels.

-Andi

--
[email protected]

2008-12-17 01:54:43

by Andi Kleen

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Paul Mackerras <[email protected]> writes:
>
> The perf counter subsystem will, in Ingo's design, naturally try to
> schedule as many counters and groups on as it can. Given a list of
> counters/groups, it could start with the first and keep on trying to
> add counters or groups while it can, essentially trying all possible
> combinations until it either fills up all the hardware counters or
> exhausts the possible combinations. If it moves all the
> counters/groups that do fit on up to the head of the list, and then
> rotates them to the back of the list when the timeslice expires, that
> would probably be OK. In fact the computation about what set of
> counters/groups to put on should be done when adding/removing a
> counter/group and when the timeslice expires, rather than at context
> switch time. (I'm talking about the list of part-time counters/groups
> here, of course.)

One issue is that PMU counts can cover more than one CPU. One example
for this are the Uncore events on Nehalem (which cover a whole socket)
or when you are in AnyThreads monitoring mode (then you get events
from both SMT siblings in a core)

With that you would need to examine other CPU's state at context switch
time. Probably not a good idea for scalability.

-Andi

--
[email protected]

2008-12-17 02:22:16

by Samuel Thibault

[permalink] [raw]

Subject: Re: [Perfctr-devel] [patch] Performance Counters for Linux, v4

Andi Kleen, le Wed 17 Dec 2008 02:51:54 +0100, a ?crit :
> William Cohen <[email protected]> writes:
> > PERF_COUNT_CACHE_REFERENCES and PERF_COUNT_CACHE_MISSES are not single
> > monolitic events on many processors. There are multiple cache
> > levels. The L1 cache most processors have separate instruction and
> > data caches and require multiple counters to implement. Would these
> > refer to the last level of cache before memory and just be used to
> > compute the hit/miss rate for that last level? Some processors in the
> > same family have L2 and some processors have L3 cache. The setup code
> > would need to distinguish between these processor variants.
>
> The difference between L1 and L3 caches can be huge (in some cases
> two orders of magnitude). With that I'm not sure a single cache
> miss/hit event even makes any sense.

Confirmed. I have a code for which I'd like to know whether it fits
into at least L2 or even L1.

Samuel

2008-12-17 02:28:49

by Dan Terpstra

[permalink] [raw]

Subject: RE: [Perfctr-devel] [patch] Performance Counters for Linux, v4

> Peter Zijlstra writes:
>
> > On Mon, 2008-12-15 at 23:11 +1100, Paul Mackerras wrote:
> > > I think the core should put together a list of counters and counter
> > > groups that it would like to have on the PMU simultaneously and then
> > > make one call to the arch layer to ask if that is possible. That
> > > could either return success or failure. If it returns failure then
> > > the core needs to ask for something less, or something different. I'm
> > > not sure how the core should choose what to ask for instead, though.
> >
> > I think the constraint set should be applied when we add to a group, if
> > when we add a counter to the group, the result isn't schedulable
> > anymore, we should fail the group addition - and thereby the counter
> > creation.
> >
> > This would leave us with groups that are always schedulable in an atomic
> > fashion.
>
> I agree that if adding a counter to a group results in that group not
> being schedulable any more, we should fail the addition.
>
That's what PAPI does.
In userspace.
Using libpfm.
Before counting anything.
On linux, AIX, Windows, Cray...
Talking to perfmon, perfctr, our own drivers, and maybe someday even the
linux performance counter subsystem.
- d

2008-12-17 07:34:21

by Stephane Eranian

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Paul,

On Wed, Dec 17, 2008 at 12:06 AM, Paul Mackerras <[email protected]> wrote:
>
> I think we need a "full-time" attribute for counters and groups that
> says "I need to be on the whole time", where "whole time" means
> whenever the task is running, for a per-task counter or group, or
> continuously for per-cpu counters/groups.
>
Who would want not to be the only owner of the PMU?

Especially when you know that if you don't, you may be sharing it
with another user who may perturb your measurement, for instance
because of heavy sampling while you are just counting. It is a tradeoff
between flexibility and accuracy. Which one is more important?

2008-12-17 16:01:22

by William Cohen

[permalink] [raw]

Subject: Re: [Perfctr-devel] [patch] Performance Counters for Linux, v4

2008-12-17 20:54:37

by Corey Ashford

[permalink] [raw]

Subject: Re: [Perfctr-devel] [patch] Performance Counters for Linux, v4

William Cohen wrote:
> Ingo Molnar wrote:
>> We are pleased to announce the v4 release of our performance counters
>> subsystem implementation. The kernel changes can be picked up from:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git
>> perfcounters/core
>>
>> (also in the master branch. There's also a kernel patch attached
>> below.)
>
> Machines that support virtualization are becoming very common. How is
> this performance monitoring support going to work with virtualization
> (e.g. KVM)? Having the performance counters only work on physical
> machines would be pretty limiting.

On Power machines, the PMU counter registers are virtualized by the
hypervisor, at the request of the OS, on a per-cpu/per-partition basis.
So this issue is handled transparently on Power anyway.

- Corey

Corey Ashford
Software Engineer
IBM Linux Technology Center, Linux Toolchain
Beaverton, OR
503-578-3507
[email protected]

2009-01-16 18:01:52

by Corey Ashford

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Andi Kleen wrote:
> Paul Mackerras <[email protected]> writes:
>> The perf counter subsystem will, in Ingo's design, naturally try to
>> schedule as many counters and groups on as it can. Given a list of
>> counters/groups, it could start with the first and keep on trying to
>> add counters or groups while it can, essentially trying all possible
>> combinations until it either fills up all the hardware counters or
>> exhausts the possible combinations. If it moves all the
>> counters/groups that do fit on up to the head of the list, and then
>> rotates them to the back of the list when the timeslice expires, that
>> would probably be OK. In fact the computation about what set of
>> counters/groups to put on should be done when adding/removing a
>> counter/group and when the timeslice expires, rather than at context
>> switch time. (I'm talking about the list of part-time counters/groups
>> here, of course.)
>
> One issue is that PMU counts can cover more than one CPU. One example
> for this are the Uncore events on Nehalem (which cover a whole socket)
> or when you are in AnyThreads monitoring mode (then you get events
> from both SMT siblings in a core)
>
> With that you would need to examine other CPU's state at context switch
> time. Probably not a good idea for scalability.
>
> -Andi
>

Over time, it seems clear that we will see multi-core processor designs
with increasingly large uncore/nest facilities, so this could become
more and more of an issue.

- Corey

Corey Ashford
Software Engineer
IBM Linux Technology Center, Linux Toolchain
Beaverton, OR
503-578-3507
[email protected]

2009-01-16 22:14:58

by Maynard Johnson

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

Corey Ashford wrote:
> Andi Kleen wrote:
>> Paul Mackerras <[email protected]> writes:
>>> The perf counter subsystem will, in Ingo's design, naturally try to
>>> schedule as many counters and groups on as it can. Given a list of
>>> counters/groups, it could start with the first and keep on trying to
>>> add counters or groups while it can, essentially trying all possible
>>> combinations until it either fills up all the hardware counters or
>>> exhausts the possible combinations. If it moves all the
>>> counters/groups that do fit on up to the head of the list, and then
>>> rotates them to the back of the list when the timeslice expires, that
>>> would probably be OK. In fact the computation about what set of
>>> counters/groups to put on should be done when adding/removing a
>>> counter/group and when the timeslice expires, rather than at context
>>> switch time. (I'm talking about the list of part-time counters/groups
>>> here, of course.)
>> One issue is that PMU counts can cover more than one CPU. One example
>> for this are the Uncore events on Nehalem (which cover a whole socket)
>> or when you are in AnyThreads monitoring mode (then you get events
>> from both SMT siblings in a core)
>>
>> With that you would need to examine other CPU's state at context switch
>> time. Probably not a good idea for scalability.
>>
>> -Andi
>>
>
> Over time, it seems clear that we will see multi-core processor designs
> with increasingly large uncore/nest facilities, so this could become
> more and more of an issue.
Ingo, I'll add my voice to the chorus here. To reiterate the point, some PMUs count events that are external to the processor cores, and these events cannot be attributed to any one particular CPU -- and certainly not to a particular pid. The current interface has a restriction that the user cannot pass -1 for both pid and cpu. But it seems to me that's exactly what would be needed for such off-core events. Can this feature fit in with the current interface or is some sort of extension needed?

Thanks.
-Maynard
>
> - Corey
>
> Corey Ashford
> Software Engineer
> IBM Linux Technology Center, Linux Toolchain
> Beaverton, OR
> 503-578-3507
> [email protected]
>

2009-01-16 23:12:45

[permalink] [raw]

Subject: Re: [patch] Performance Counters for Linux, v4

* Maynard Johnson <[email protected]> wrote:

> > Over time, it seems clear that we will see multi-core processor
> > designs with increasingly large uncore/nest facilities, so this could
> > become more and more of an issue.

> Ingo, I'll add my voice to the chorus here. To reiterate the point,
> some PMUs count events that are external to the processor cores, and
> these events cannot be attributed to any one particular CPU -- and
> certainly not to a particular pid. The current interface has a
> restriction that the user cannot pass -1 for both pid and cpu. But it
> seems to me that's exactly what would be needed for such off-core
> events. Can this feature fit in with the current interface or is some
> sort of extension needed?

They fit in just fine, they just will have constraints that dont allow
their scheduling in a conflicting way. I.e. you'll only be able to occupy
it from a single counter per physical package - but otherwise it still
behaves like a normal counter if you define a single such counter per
physical package, as a percpu counter. (they dont make much sense as task
counters)

Btw., those kind of constraints make them quite noisy and hard to
interpret as well - because they summarize per physical package
characteristics (of up to 8 logical CPUs on Nehalem for example) and
cannot be tied to tasks easily.

I'm not dismissing them entirely: they do give an overview of "all stuff
that happens" at that level, and they do show a few things that is
obviously tied to the 'uncore' of a CPU (the cache, the memory interlink,
etc.), which cannot be provided by the normal PMCs - but they are too
highlevel to really be useful for finegrained analysis and for eventing.

In any case, despite their limitations they can still be provided just
fine, and any limitations they have is an inherent limitation of those
hardware counters, not of the perfcounters framework.

Ingo

2009-01-17 01:28:26