Would you want Stephane to guard the extended
functionalities with tunables or something to
Disable their regular use and herd enterprise
Tools into a standard mold... yet allow R&D to
Move on by enabling the extentions?
Just crippling flexibility/cutting functionality
is like removing words out of a dictionary to
prevent people from thinking different.
It would restrict the R&D mindset, and new ideas.
The field hasn't grown yet to a stable mature form.
It is just beginning: profiling, monitoring, tuning,
compilers, JIT...
Flexibility is/was needed because:
- Tools need to port to Perfmon with min cost.
- Ability to support novel R&D ideas.
- Ability to support growth beyond just PMU data
- Allows early data aggregation
- Allow OS data correlated to PMU
What standardization adds:
- Coordinated access to PMU rssources from all tools
- All tools/formats etc all plug into the same OS framework.
- The interface gets ported across multiple platforms.
- The functionality is rich for all (fast data transfers,
multiplexing, system vs thead, etc.)
Dan-
> -----Original Message-----
> From: [email protected] [mailto:perfmon-
> [email protected]] On Behalf Of Andrew Morton
> Sent: Thursday, December 22, 2005 5:47 AM
> To: Truong, Dan
> Cc: Eranian, Stephane; [email protected]; linux-
> [email protected]; [email protected]; perfctr-
> [email protected]
> Subject: Re: [perfmon] Re: quick overview of the perfmon2 interface
>
> "Truong, Dan" <[email protected]> wrote:
> >
> > The PMU is becoming a standard commodity. Once Perfmon is
> > "the" Linux interface, all the tools can align on it and
> > coexist, push their R&D forward, and more importantly become
> > fully productized for businesses usage.
> >
>
> The apparently-extreme flexibility of the perfmon interfaces would
tend to
> militate against that, actually. It'd become better productised if it
had
> one interface and stuck to it.
>
> (I haven't processed Stephane's reply yet - will get there)
>
> _______________________________________________
> perfmon mailing list
> [email protected]
> http://www.hpl.hp.com/hosted/linux/mail-archives/perfmon/
"Truong, Dan" <[email protected]> wrote:
>
> Would you want Stephane to guard the extended
> functionalities with tunables or something to
> Disable their regular use and herd enterprise
> Tools into a standard mold... yet allow R&D to
> Move on by enabling the extentions?
argh. I'd prefer to avoid one-month gaps in the conversation, so we don't
all forget what we were talking about.
Look, we just need to get these patches on the wire so we can all look at
them, see what they do, understand what decisions were taken and why.
The conciseness and completeness of those patches' covering descriptions
will be key to helping this process along.
On Fri, 2006-01-20 at 10:37 -0800, Truong, Dan wrote:
> Would you want Stephane to guard the extended
> functionalities with tunables or something to
> Disable their regular use and herd enterprise
> Tools into a standard mold... yet allow R&D to
> Move on by enabling the extentions?
I'd prefer to see all of the extended stuff left out entirely for now.
The mainline kernel has no PMU support for any popular architecture,
even though external patches have existed in stable form for years.
Filling that gap ought to be the priority; the interface can be extended
when actual users of new features show up and ask for them.
> It would restrict the R&D mindset, and new ideas.
> The field hasn't grown yet to a stable mature form.
The place for flailing around with uncooked ideas is arguably not the
mainline kernel.
> Flexibility is/was needed because:
> - Tools need to port to Perfmon with min cost.
> - Ability to support novel R&D ideas.
> - Ability to support growth beyond just PMU data
> - Allows early data aggregation
> - Allow OS data correlated to PMU
Speculatively adding complicated and unused interfaces to the kernel in
the hope that some wild-eyed visionary might eventually up and use them
helps nobody.
<b
Bryan,
On Wed, Jan 25, 2006 at 12:33:32PM -0800, Bryan O'Sullivan wrote:
> On Fri, 2006-01-20 at 10:37 -0800, Truong, Dan wrote:
> > Would you want Stephane to guard the extended
> > functionalities with tunables or something to
> > Disable their regular use and herd enterprise
> > Tools into a standard mold... yet allow R&D to
> > Move on by enabling the extentions?
>
> I'd prefer to see all of the extended stuff left out entirely for now.
I usually don't add things to the interface just because they are cool
ideas but rather because there is a need expressed by some tool
developer or system person. So it would help if you could
name the extended features you referring to.
The problem with an incremental approach is to maintained backward compatibility
for existing applications. I have had to deal with this on IA-64. For instance
moving from a single syscall to multiple syscall. Similarly, when passing
data structures, you have to provision some reserved fields for potential
extensions. You don't really want to add more system call if you need to
to add a feature.
> The mainline kernel has no PMU support for any popular architecture,
> even though external patches have existed in stable form for years.
You do not count Oprofile. I think this is a fine tool. And perfmon
does allow it to continue working using almost all of its kernel code.
This is leveraging the custom sampling buffer format support in perfmon.
So you can say this is an extended feature that adds complexity.
But OTOH, this is one elegant way of supporting an existing interface
without breaking all the tools.
Take another example, suppose some tool comes along and say: "I would
like to add in each recorded sample the kernel call stack at the point
of the counter overflow". How would you do this without having to hack
kernel code? With the buffer format, you simply insert of module that
does what you want. There are hundreds of things you can include in your
samples. I don't think that we can come up with a very generic sampling
buffer format.
Sometimes, it is not so much what is recorded but how it is recorded.
Some tool may prefer to have samples aggreagated in the kernel, other
would like to use a double-buffer approach to minimize blind spots.
All are valid requests. Our infrastructure allows this without modification
to the core interface nor core kernel code. I believe this is a very strong
value-add.
Without this infrastructure, it would have been pretty difficult to add
support for the P4 Precise Event Based Sampling (PEBS) which by the way,
nobody was able to offer so far. We were able to proide this support
with a few hundred lines of code without hacking the regular sampling
format. Instead we simply created a dedicated PEBS format as a kernel module.
> Filling that gap ought to be the priority; the interface can be extended
> when actual users of new features show up and ask for them.
>
Again that is fine as long as you can keep backward complexity and a clean
interface.
> > It would restrict the R&D mindset, and new ideas.
> > The field hasn't grown yet to a stable mature form.
>
I would agree with you, that people have not yet realized the potential
of those performance counters. But this maybe in part a chicken and egg
problem. People cannot take full advantage because they don't have
a generic interface on any platform.
Designing a generic perfmon interface is hard because:
- the hardware is extremely diverse
- there are so many things you can measure
--
-Stephane
On Wed, 2006-01-25 at 14:28 -0800, Stephane Eranian wrote:
> So it would help if you could
> name the extended features you referring to.
I'm dubious about the hands-off buffer format in general. Does this
mean that userspace needs to modprobe a specific set of modules in order
to do normal sampling? If so, how do you work around the need for users
to be root in order to use these interfaces?
> And perfmon
> does allow it to continue working using almost all of its kernel code.
> This is leveraging the custom sampling buffer format support in perfmon.
> So you can say this is an extended feature that adds complexity.
> But OTOH, this is one elegant way of supporting an existing interface
> without breaking all the tools.
So are you saying that part of the existing oprofile code can be deleted
if perfmon is merged, and that userspace won't notice?
> We were able to proide this support
> with a few hundred lines of code without hacking the regular sampling
> format. Instead we simply created a dedicated PEBS format as a kernel module.
Does this mean I can't sample the PMCs on a P4 if I don't have the
special PEBS module loaded? Do I need to be root to do that?
<b
Bryan,
On Wed, Jan 25, 2006 at 02:46:43PM -0800, Bryan O'Sullivan wrote:
> On Wed, 2006-01-25 at 14:28 -0800, Stephane Eranian wrote:
>
> > So it would help if you could
> > name the extended features you referring to.
>
> I'm dubious about the hands-off buffer format in general. Does this
> mean that userspace needs to modprobe a specific set of modules in order
> to do normal sampling? If so, how do you work around the need for users
> to be root in order to use these interfaces?
As I said, there is a builtin default format that is fairly generic. It does
work for HP Caliper, pfmon, q-tools. I suspect it is good enough for VTUNE.
You need to be root to insert the module. But I believe that for many user
environments, this is more practical than having to recompile a custom kernel.
You can imagine the format being shipped with the tool, when the sysadmin
installs the tool it also installs the module.
>
> > And perfmon
> > does allow it to continue working using almost all of its kernel code.
> > This is leveraging the custom sampling buffer format support in perfmon.
> > So you can say this is an extended feature that adds complexity.
> > But OTOH, this is one elegant way of supporting an existing interface
> > without breaking all the tools.
>
> So are you saying that part of the existing oprofile code can be deleted
> if perfmon is merged, and that userspace won't notice?
>
The part of Oprofile that does actual programming of the PMU can be removed.
The part that stays is the one that deals with recording samples, exporting
samples, and collecting OS events such as exit, mmap, exec. As the user
level, they need to migrated from the Oprofile way of programming counters
to the perfmon way. This has been done many years ago on Itanium and did
not cause any major problems.
> > We were able to proide this support
> > with a few hundred lines of code without hacking the regular sampling
> > format. Instead we simply created a dedicated PEBS format as a kernel module.
>
> Does this mean I can't sample the PMCs on a P4 if I don't have the
> special PEBS module loaded? Do I need to be root to do that?
PEBS is a P4 feature that has two advantages:
- record the exact IP of where a counter overflows (no skid)
- the CPU directly record the samples into a memory area designated
by the kernel. As such, you only get a PMU when that area fills up.
There are some limitations:
- you cannot sample on any event
- the format of a sample is fixed, it does not contain extra PMDs, just
IP and some general registers. The process id is not recorded
so it is not well suited for system-wide monitoring.
- it appears to broken for HyperThreading setups.
So, it all depends on what you are after. Some people do care about avoiding
the skid of regular sampling and they want they like PEBS just for that. Others
would like to record a set of extra PMDs (PERFCTR) and they are willing to
compromise a bit on the skid of IP, so they can live with the default format.
--
-Stephane
On Wed, 2006-01-25 at 23:48 -0800, Stephane Eranian wrote:
> You need to be root to insert the module. But I believe that for many user
> environments, this is more practical than having to recompile a custom kernel.
Clearly.
> You can imagine the format being shipped with the tool, when the sysadmin
> installs the tool it also installs the module.
In that case, you need some kind of per-distro cruft to make sure the
module gets loaded at every boot, or a setuid program that can install
the module, right?. Neither of these approaches works well in a cluster
environment where you're running your tools from a shared directory.
I'd really like the default mode of operation for users to not require
root privileges to get at normal functionality. This is something
perfctr makes possible, for example.
<b