LinuxLists.cc - [GIT PULL] Performance Counters for Linux

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 06:03:29PM +0200, Ingo Molnar wrote:
> Linus,
>
> Please consider pulling the performance counters Git tree from:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git perfcounters-for-linus

Err, no. This adds tons of userspace code into tools/ which
should not be in the kernel tree but a proper package.

2009-06-11 16:27:41

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Christoph Hellwig wrote:
>
> Err, no. This adds tons of userspace code into tools/ which
> should not be in the kernel tree but a proper package.

I disagree.

We've had tons of cases where we tried to "separate" the user-land code
and the kernel code, in the name of "beauty" of whatever.

It's almost invariably a disaster.

Look at oprofile. F*ck me, what a horrid piece of crap. It took literally
months for the user mode tools to catch up and get the patches to support
new functionality into CVS (or is it SVN?), and after that it took even
longer for them to become part of a release and be picked up by
distributions. In fact, I'm not sure it is part of a release even now - I
had to make a bug report to Fedora to get atom and Nehalem support in my
tools: I think they took the unofficial patch.

Or look at the crazy things we used to do for X. It's going away (slowly),
because some of the most incestuous things are actually just being
integrated into the kernel, and so there's less of the "two broken pieces"
approach, and more of a "one working piece" kind of thing.

So I'd much rather have kernel tools with the kernel, than have to depend
on some external entity that doesn't really care.

Linus

2009-06-11 16:35:12

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Hi Linus,

> > Err, no. This adds tons of userspace code into tools/ which
> > should not be in the kernel tree but a proper package.
>
> I disagree.
>
> We've had tons of cases where we tried to "separate" the user-land code
> and the kernel code, in the name of "beauty" of whatever.
>
> It's almost invariably a disaster.
>
> Look at oprofile. F*ck me, what a horrid piece of crap. It took literally
> months for the user mode tools to catch up and get the patches to support
> new functionality into CVS (or is it SVN?), and after that it took even
> longer for them to become part of a release and be picked up by
> distributions. In fact, I'm not sure it is part of a release even now - I
> had to make a bug report to Fedora to get atom and Nehalem support in my
> tools: I think they took the unofficial patch.
>
> Or look at the crazy things we used to do for X. It's going away (slowly),
> because some of the most incestuous things are actually just being
> integrated into the kernel, and so there's less of the "two broken pieces"
> approach, and more of a "one working piece" kind of thing.
>
> So I'd much rather have kernel tools with the kernel, than have to depend
> on some external entity that doesn't really care.

so do you expect us to merge stuff like ip, iw, rfkill, crda, the WiMAX
tools, the Bluetooth ones and whatever we have that are all have the
same issues to be merged into the kernel source code as well.

I see no reason this can't be maintained properly outside the kernel
source. You will always have bad sheeps and screw-ups, but just putting
everything into one single location is not a good idea either. Other
subsystems do this well and so could Ingo.

Also please consider the distro point of view. All these distros have
already a hard time to keep up with the kernel patches etc. It is a lot
easier to update a userspace package then having to provide a patches
kernel source.

Regards

Marcel

2009-06-11 16:39:49

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Marcel Holtmann wrote:
>
> so do you expect us to merge stuff like ip, iw, rfkill, crda, the WiMAX
> tools, the Bluetooth ones and whatever we have that are all have the
> same issues to be merged into the kernel source code as well.

No. Only stuff that I expect to be really close to hardware, and used for
kernel purposes.

> Also please consider the distro point of view. All these distros have
> already a hard time to keep up with the kernel patches etc. It is a lot
> easier to update a userspace package then having to provide a patches
> kernel source.

Feel free to split it all up if it turns out to be stable later.

But I refuse to go through another oprofile.

Linus

2009-06-11 16:47:18

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Hi Linus,

> > so do you expect us to merge stuff like ip, iw, rfkill, crda, the WiMAX
> > tools, the Bluetooth ones and whatever we have that are all have the
> > same issues to be merged into the kernel source code as well.
>
> No. Only stuff that I expect to be really close to hardware, and used for
> kernel purposes.

and where exactly do we draw the line? It is just no clear to me.

> > Also please consider the distro point of view. All these distros have
> > already a hard time to keep up with the kernel patches etc. It is a lot
> > easier to update a userspace package then having to provide a patches
> > kernel source.
>
> Feel free to split it all up if it turns out to be stable later.
>
> But I refuse to go through another oprofile.

Point taken on why you wanna do it. No questions asked here. However I
still think it is a bad idea to begin with. The perf tool could very
well has its own repository on git.kernel.org and be maintained side by
side with the kernel.

Regards

Marcel

2009-06-11 16:47:29

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 09:38:32AM -0700, Linus Torvalds wrote:
> > so do you expect us to merge stuff like ip, iw, rfkill, crda, the WiMAX
> > tools, the Bluetooth ones and whatever we have that are all have the
> > same issues to be merged into the kernel source code as well.
>
> No. Only stuff that I expect to be really close to hardware, and used for
> kernel purposes.

Did you take a look a tools/perf/? There is nothing close to hardware
at all. It's all pretty highly abstracted away from anything resembling
the hardware through the perfcounters interface.

2009-06-11 16:52:42

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

2009-06-11 16:55:27

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Christoph Hellwig wrote:
>
> Did you take a look a tools/perf/? There is nothing close to hardware
> at all. It's all pretty highly abstracted away from anything resembling
> the hardware through the perfcounters interface.

The thing is, the raw perfcounters interface isn't going to be useful as
is. And I have seen where things go when you split them up. So when I get
the choice, I'll go down the road of unproven failure, in the hope that it
will be successful, rather than doing the same mistake once more.

"Insanity: doing the same thing over and over, expecting to get
different results."

And I'm not insane.

Anyway, feel free to disagree. I just don't care.

Linus

2009-06-11 16:56:34

by Peter Zijlstra

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 2009-06-11 at 17:52 +0100, Al Viro wrote:

> Do you consider "put into tools/" as permission to change interface at will?
> More to the point, do the authors of that stuff consider it as such?

No, once a kernel with this syscall gets released we most certainly
intend to maintain its ABI.

2009-06-11 16:59:34

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Al Viro wrote:
>
> Yes. So's sysfs, so's udev, so's hal, so's any number of revolting
> strings of intertwined copulating tapeworms hanging off the kernel's arse.

Those are about a different thing, though - they are largely about policy.
Very different from something like profiling (or graphics acceleration).

I do like your visuals, though.

Linus

2009-06-11 17:01:07

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 06:56:18PM +0200, Peter Zijlstra wrote:
> No, once a kernel with this syscall gets released we most certainly
> intend to maintain its ABI.

So what point is there in keeping it in-tree except making life hell for
packagers?

2009-06-11 17:05:29

by Ray Lee

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 10:00 AM, Christoph Hellwig<[email protected]> wrote:
> On Thu, Jun 11, 2009 at 06:56:18PM +0200, Peter Zijlstra wrote:
>> No, once a kernel with this syscall gets released we most certainly
>> intend to maintain its ABI.
>
> So what point is there in keeping it in-tree except making life hell for
> packagers?

Packagers are quite used to taking a single source tree and building
multiple packages out of it. This isn't rocket science. It's the
multiple separate trees that need to be released in lock-step that are
headaches.

2009-06-11 17:08:00

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Christoph Hellwig wrote:
>
> So what point is there in keeping it in-tree except making life hell for
> packagers?

Give it up. Packagers can trivially generate their own sub-packages. They
do it all the time. They already do it for the user-mode header files,
extracted from the kernel - something you've worked on yourself.

So your point is clearly bogus, and dishonest.

You haven't actually looked the real problem in the eye, and acknowledged
the disaster that is oprofile. Let's give a _new_ approach a chance, and
see if we can avoid the mistakes of yesteryear this time.

Linus

2009-06-11 17:08:22

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Hi Ray,

> >> No, once a kernel with this syscall gets released we most certainly
> >> intend to maintain its ABI.
> >
> > So what point is there in keeping it in-tree except making life hell for
> > packagers?
>
> Packagers are quite used to taking a single source tree and building
> multiple packages out of it. This isn't rocket science. It's the
> multiple separate trees that need to be released in lock-step that are
> headaches.

with the kernel as source package it is a headache and really painful.
All distros struggle already enough with kernel updates.

Regards

Marcel

2009-06-11 17:13:27

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 10:05:02AM -0700, Ray Lee wrote:
> On Thu, Jun 11, 2009 at 10:00 AM, Christoph Hellwig<[email protected]> wrote:
> > On Thu, Jun 11, 2009 at 06:56:18PM +0200, Peter Zijlstra wrote:
> >> No, once a kernel with this syscall gets released we most certainly
> >> intend to maintain its ABI.
> >
> > So what point is there in keeping it in-tree except making life hell for
> > packagers?
>
> Packagers are quite used to taking a single source tree and building
> multiple packages out of it. This isn't rocket science. It's the
> multiple separate trees that need to be released in lock-step that are
> headaches.

Wrong. Remember the fun bisecting around sysfs/udev incompatible change?
Oops, went back past the cutoff line, got to downgrade udev for the next
boot. Oh, it oopses? Too fucking bad, can't just boot the previous kernel,
should've kept _two_ working ones so that with any userland state we could
come back to working system.

This isn't a rocket science, this is a goddamn load of horse manure.
Packages that need to be updated in the lock-step *are* headaches from
hell when you are trying to do development. Even if you have all of
them already built.

2009-06-11 17:22:50

by Ray Lee

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 10:12 AM, Al Viro<[email protected]> wrote:
> On Thu, Jun 11, 2009 at 10:05:02AM -0700, Ray Lee wrote:
>> Packagers are quite used to taking a single source tree and building
>> multiple packages out of it. This isn't rocket science. It's the
>> multiple separate trees that need to be released in lock-step that are
>> headaches.
>
> Wrong. Remember the fun bisecting around sysfs/udev incompatible change?
> Oops, went back past the cutoff line, got to downgrade udev for the next
> boot. Oh, it oopses? Too fucking bad, can't just boot the previous kernel,
> should've kept _two_ working ones so that with any userland state we could
> come back to working system.
>
> This isn't a rocket science, this is a goddamn load of horse manure.
> Packages that need to be updated in the lock-step *are* headaches from
> hell when you are trying to do development. Even if you have all of
> them already built.

Well, welcome to our new world order of Xorg and udev and hal. I have
had to deal with bisecting the problem just as you have, and dealt
with the fallout.

The choices are:

- Don't bisect, throw up your hands and hope someone else deals with it

- keep the old versions around for installs, as you point out (I do
this regularly)

- build all the packages every time

The last one is the most reasonable and I'd argue it's the right thing
to do. But it's tricky with multiple source trees -- which version of
udev works with this kernel again? A single source tree for packages
that are kept in lock-step, as so many seem to be, makes that a hell
of a lot easier on me.

But perhaps I'm an odd-ball.

I think your complaint is "Why the hell can't they have a stable ABI?"
Probably for the same reason anything so close to the hardware hasn't
had a stable ABI. I'm sure udev/hal/Xorg will have a stable
kernel-userland interface any day now. Once they do, I'm sure
everything else that touches the hardware so intimately will have a
stable ABI too.

Sheesh.

2009-06-11 17:59:45

by Pekka Enberg

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Christoph Hellwig wrote:
>> So what point is there in keeping it in-tree except making life hell for
>> packagers?

On Thu, Jun 11, 2009 at 8:06 PM, Linus
Torvalds<[email protected]> wrote:
> Give it up. Packagers can trivially generate their own sub-packages. They
> do it all the time. They already do it for the user-mode header files,
> extracted from the kernel - something you've worked on yourself.
>
> So your point is clearly bogus, and dishonest.
>
> You haven't actually looked the real problem in the eye, and acknowledged
> the disaster that is oprofile. Let's give a _new_ approach a chance, and
> see if we can avoid the mistakes of yesteryear this time.

Yup, I wonder what all the fuzz is about. We already have userspace
tools in the kernel but people keep putting them under Documentation
(to avoid this discussion, probably).

For those who think an external repository is a good idea, I invite
you to compare the success of kmemtrace (kernel memory profiler) and
perf. The former has its userspace part out-of-tree and has gained
zero new developers. Sure, there are probably fewer people interested
in memory profiling and I or Eduard surely don't have the sex appeal
of Ingo Molnar (yet anyway). But even if you take these factors into
account, I'd argue that big part of the success has been the fact that
it's easily accessible and hackable. And that pretty much means that
the code needs to sit in the kernel tree, following kernel development
process.

And really, what do we gain by moving perf out of tree and making it
follow its own release cycle (and getting out of sync eventually)?

Pekka

2009-06-11 18:04:57

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Linus Torvalds wrote:
> On Thu, 11 Jun 2009, Marcel Holtmann wrote:
>
>> so do you expect us to merge stuff like ip, iw, rfkill, crda, the WiMAX
>> tools, the Bluetooth ones and whatever we have that are all have the
>> same issues to be merged into the kernel source code as well.
>>
>
> No. Only stuff that I expect to be really close to hardware, and used for
> kernel purposes.

Whilst having no opinion on the matter, I can't help noticing that Ingo
said its "main focus is on measuring/profilig user-space apps" (sic), so
I think its use is not for kernel purposes. Coupled with Christoph's
observation that the tool is nowhere close to the hardware, it appears
neither of your two criteria apply to this.

2009-06-11 18:11:10

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Pekka Enberg wrote:
> And really, what do we gain by moving perf out of tree and making it
> follow its own release cycle (and getting out of sync eventually)?

I'm sure perf will change, for example as faults are discovered in it.
Perhaps, too, the kernel side counters will change, but will the ABI?
Peter Zijlstra comment ("we most certainly intend to maintain its ABI")
implies it won't, or won't in such a way as to break user space tools.

What I'm saying is that this doesn't sound like something that needs
user-space in lock-step with kernel.

2009-06-11 18:22:48

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Fri, 12 Jun 2009, David Newall wrote:
>
> What I'm saying is that this doesn't sound like something that needs
> user-space in lock-step with kernel.

Give it a rest.

If that's true, then in a year or two we can just split it up already.

But I note (once more) how _nobody_ has actually been able to accept the
fact that oprofile was an abject failure as it was split up. Instead, you
all dance around totally irrelevant issues.

Linus

2009-06-11 18:24:32

by Martin Bligh

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

>> So what point is there in keeping it in-tree except making life hell for
>> packagers?
>
> Give it up. Packagers can trivially generate their own sub-packages. They
> do it all the time. They already do it for the user-mode header files,
> extracted from the kernel - something you've worked on yourself.
>
> So your point is clearly bogus, and dishonest.
>
> You haven't actually looked the real problem in the eye, and acknowledged
> the disaster that is oprofile. Let's give a _new_ approach a chance, and
> see if we can avoid the mistakes of yesteryear this time.

We actually ended up coming to the same conclusion as you for some of the
internal tools we use that are tightly tied to the kernel. There is one hitch,
which is that if you boot between different kernel versions, you need multiple
userspace versions of the tools, so you may need to put them in
/lib/modules/<kernel-version> or something equivalent, not one fixed place.

M.

2009-06-11 18:35:34

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Martin Bligh wrote:
>
> We actually ended up coming to the same conclusion as you for some of the
> internal tools we use that are tightly tied to the kernel. There is one hitch,
> which is that if you boot between different kernel versions, you need multiple
> userspace versions of the tools, so you may need to put them in
> /lib/modules/<kernel-version> or something equivalent, not one fixed place.

So I actually think this is broken.

No tool should ever be _that_ tightly tied to a kernel. If they are, they
are broken, plain and simple.

A stable user-space ABI is still a requirement.

What the "keep it in the kernel sources" approach hopefully allows is

- taking advantage of new features in a timely manner.

NOT with some ABI breakage, but simply things like supporting a new CPU
architecture or new counters. The thing that oprofile failed at so
badly in my experience.

- Make it easier for developers, and _avoiding_ the horrible situation
where you have two different groups that don't talk well to each other
because one is a group of user-space weenies, and the other is a group
of manly kernel people, and there is no common ground.

And no, I'm not going to "guarantee" that this works well. Again, I just
know that the separation didn't work. Let's just _try_ to do it this way,
and see how it works.

But at no point will it be acceptable to have kernel version dependencies.
Install the newest version of the binaries, and it should support older
kernels too (within reason).

The "within reason" is because (a) it's new, so early on you might see
breakage, and (b) because we do tend to allow system tools to break
occasionally. Not nearly often enough to make it valid to design around
it, though.

Linus

2009-06-11 18:39:03

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Linus Torvalds wrote:
> I note (once more) how _nobody_ has actually been able to accept the
> fact that oprofile was an abject failure as it was split up. Instead, you
> all dance around totally irrelevant issues.

Not I. I'm totally comfortable with your decision, even though it seems
counter- to your own stated policy.

While I'm still unable to help noticing things, nobody seems to have
presented an argument that user-space and kernel-side will need to be
developed together. There's been an obvious assumption, with oprofile
given as that assumption's poster-boy, but why that should be the case
for tools/perf remains unclear. Probably the reasons are so obvious that
they go without saying, but as a disinterested observer, it seems to me
that in this case the two sides really are quite separate and
independent in a very real sense. Perhaps in the same sense that acct
and quota are.

2009-06-11 18:46:35

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Fri, 12 Jun 2009, David Newall wrote:
>
> While I'm still unable to help noticing things, nobody seems to have
> presented an argument that user-space and kernel-side will need to be
> developed together.

Well, I tried to give two reasons in my reply to Martin (it's there
somewhere in cyberspace, a couple of minutes ago). Basically timeliness
(kernel features vs taking advantage of them) and co-development (there
always seems to be a huge impedance mis-match between user-level
developers and kernel developers).

Linus

2009-06-11 18:48:42

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 09:26:55AM -0700, Linus Torvalds wrote:
>
>
> On Thu, 11 Jun 2009, Christoph Hellwig wrote:
> >
> > Err, no. This adds tons of userspace code into tools/ which
> > should not be in the kernel tree but a proper package.
>
> I disagree.
>
> We've had tons of cases where we tried to "separate" the user-land code
> and the kernel code, in the name of "beauty" of whatever.
>
> It's almost invariably a disaster.

This is cheating. I had this as a topic for the kernel summit and
was looking forward to read an interesting article about people
dancing on the table and fighting in the corners about it.
[I do not attend myself]

People say that this would be a nightmare for the packagers.
I frankly do not see what the issue is here.

We should be able to add the necessary stuff to create the few popular
package formats.
And tools like kernels may update 4 times/year with ease - so the kernel
release frequency should be a non-issue too.

Others just say "no userspace in the kernel" - and I honestly have not understood why.

Where to draw the line?
We can ask a few simple questions:
- Are the tool part of a kernel hackers toolbox?
- Are the tool maintained by kernel people?
- Are the tool updated with new features in the kernel (*)?

If the answer is yes it is a good candidate.

(*) No excuse for ABI changes..

Simple example. I needed vmstat on my embedded platfrom the other day.
Got lots of hits on google but could not find the source - and gave up as I was busy.
[Today I found it in second try - sigh.]

Sam

2009-06-11 18:51:35

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 11:21:46AM -0700, Linus Torvalds wrote:
> But I note (once more) how _nobody_ has actually been able to accept the
> fact that oprofile was an abject failure as it was split up. Instead, you
> all dance around totally irrelevant issues.

I don't think oprofile has been a desaster because of any kind of split,
but because the design has been a failure from day 1.

2009-06-11 19:06:20

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Hi Linus,

> > What I'm saying is that this doesn't sound like something that needs
> > user-space in lock-step with kernel.
>
> Give it a rest.
>
> If that's true, then in a year or two we can just split it up already.
>
> But I note (once more) how _nobody_ has actually been able to accept the
> fact that oprofile was an abject failure as it was split up. Instead, you
> all dance around totally irrelevant issues.

so your whole reasoning is based on the fact that oprofile was a
horrible failure. All the other projects/subsystems that manage this
perfectly successful without breaking API/ABI abruptly and emerging
slowly when things change don't count. Stop dancing around oprofile so
much. It makes you dizzy ;)

Regards

Marcel

2009-06-11 19:07:49

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Linus Torvalds wrote:
> timeliness
> (kernel features vs taking advantage of them) and co-development (there
> always seems to be a huge impedance mis-match between user-level
> developers and kernel developers).

You seem to be saying that putting the code in kernel tree will make
user-level developers more responsive. FWIW (very little) I would have
quietly guessed the opposite result.

2009-06-11 19:24:59

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Fri, 12 Jun 2009, David Newall wrote:
>
> You seem to be saying that putting the code in kernel tree will make
> user-level developers more responsive. FWIW (very little) I would have
> quietly guessed the opposite result.

No. I'm saying that if there's a big overlap with _kernel_ developers
(which there is), then they can maintain the tree.

Linus

2009-06-11 19:30:40

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Linus Torvalds wrote:
>
> No. I'm saying that if there's a big overlap with _kernel_ developers
> (which there is), then they can maintain the tree.

To take the oprofile example that decided it for me: the code to actually
support new processors was all done by basically kernel developers. And it
didn't hit user land for almost a year, because the user-land tools didn't
take the patch and propagate it up.

Linus

2009-06-11 19:36:18

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Linus Torvalds wrote:
> To take the oprofile example that decided it for me: the code to actually
> support new processors was all done by basically kernel developers. And it
> didn't hit user land for almost a year, because the user-land tools didn't
> take the patch and propagate it up.
>

Bad developer, Spot, you only did half the job. Not sure there's much
more one can say.

2009-06-11 19:37:40

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Sorry, Linus, for the duplicate post.

Linus Torvalds wrote:

> > I'm saying that if there's a big overlap with _kernel_ developers
> > (which there is), then they can maintain the tree.
>

Are you suggesting that maintaining it as a separate application is
harder for them? As previously observed, the user-space stuff is quite
divorced from the hardware; and it's intended audience is user-space
developers and administrators. Did I miss something? (Obviously I didn't
miss that your decision has already been made.)

2009-06-11 19:50:25

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Fri, 12 Jun 2009, David Newall wrote:

> Linus Torvalds wrote:
> > To take the oprofile example that decided it for me: the code to actually
> > support new processors was all done by basically kernel developers. And it
> > didn't hit user land for almost a year, because the user-land tools didn't
> > take the patch and propagate it up.
>
> Bad developer, Spot, you only did half the job. Not sure there's much
> more one can say.

Umm. The kernel developer _did_ do the job. The patch to the user land
side was available for that whole year. It just didn't get merged, and
then didn't get merged some more, and then got merged but only in a SVN
tree, not a release, and then finally when I did a bugzilla request to
fedora, they took the patch and put it in their distro.

Anyway, it's clearly not worth discussing this with you. I've tried. I
give up. Happily, I don't _need_ to convince you.

Linus

2009-06-11 19:59:25

by Andrew Morton

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009 10:06:55 -0700 (PDT)
Linus Torvalds <[email protected]> wrote:

> You haven't actually looked the real problem in the eye, and acknowledged
> the disaster that is oprofile. Let's give a _new_ approach a chance, and
> see if we can avoid the mistakes of yesteryear this time.

+1, metoo.

We've had numerous problems in the past where kernel developers have
shied away from altering or distributing userspace code. One effect of
this which we see again and again is that people shove presentation and
parsing code into the kernel which should have been in userspace.

It could be that shipping userspace code in the kernel bundle will
improve that situation. So let's give it a try. If it turns out to be
good, let's do it again. If it turns out to be bad, let's move perf
out of the kernel tree and not do it again.

2009-06-11 20:10:36

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Andrew Morton wrote:
>
> It could be that shipping userspace code in the kernel bundle will
> improve that situation. So let's give it a try. If it turns out to be
> good, let's do it again. If it turns out to be bad, let's move perf
> out of the kernel tree and not do it again.

Exactly. Right now, I use the oprofile experience as a reason for why we
should try to do this. But hey, who knows, in one year, maybe people will
use _this_ experience as a reason why we should never do it again.

We just don't know yet. But that's no reason not to try. Either way, we'll
hopefully learn something.

Or to quote Edison:
"I have not failed 700 times. I have not failed once. I have succeeded in
proving that those 700 ways will not work. When I have eliminated the
ways that will not work, I will find the way that will work."

Linus

2009-06-11 20:24:17

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

* Linus Torvalds <[email protected]> wrote:

[...]
> What the "keep it in the kernel sources" approach hopefully allows is
>
> - taking advantage of new features in a timely manner.
>
> NOT with some ABI breakage, but simply things like supporting a
> new CPU architecture or new counters. The thing that oprofile
> failed at so badly in my experience.
>
> - Make it easier for developers, and _avoiding_ the horrible
> situation where you have two different groups that don't talk
> well to each other because one is a group of user-space
> weenies, and the other is a group of manly kernel people, and
> there is no common ground.

Yes, very much agreed.

Btw., here are a couple of other arguments why i find it useful to
have the tools/perf/ in the kernel repo:

1) Super-fast and synchronized release cycles

The kernel is one of the fastest moving packages in Linux - most
user-space packages have (much!) longer release cycles than 3
months.

A tight release schedule forces a certain amount of release
discipline on tooling as well - so i'm glad that the two will be
coupled. It's so easy for a promising tool to degrade into
tinkerware with odd release cycles with time - if it's part of the
kernel then at least the release cycles wont be odd but at precise 3
months.

2) Performance _matters_

This is an argument pretty specific to perfcounters: Performance
analysis tools under Linux suck pretty summarily. Yet, one of the
major strengths of Linux is (or at least used to be) performance. So
i find it very fitting that the kernel community takes performance
analysis tooling into their own hand.

3) Strict quality control under a proven mode

In the kernel repo i can be sure that:

- No one will even think of adding autofools to tools/perf/.

- No one will send us code with Hungarian notation and two spaces
tabulation.

- No one will put getopt.h into the code

- No one will rewrite it in some weird language

[ Or at least, even though such incidents might happen
occasionally, i can just sit back in my chair and watch the
resulting showdown on lkml, without having to worry about the
outcome ;-) ]

I can point contributors to well-established kernel coding
principles, without having to argue no end about them.

All in one - the Linux kernel is a fire breathing monster engine
when it comes to producing good software. Who says it that that this
infrastructure and experience can only be used to produce kernel
space code?

4) Code reuse

We actually use code from the kernel: list.h primitives and
rbtrees.c. We privatized them for now under
tools/perf/util/rbtree.[ch] and tools/perf/util/list.h because
there's some header and type pollution in them, but it would be nice
to include them directly and share the facilities.

5) Reality check for kernel developers

I think kernel hackers need a reality check too. It's easy to say
that user-space sucks - but now there's a way and channel that
frustration via direct action and make a real difference. I do hope
that the extra superfluous mental energies visible in this thread
can be used for good purposes too ;-)

6) It's a lot of fun

I never thought i'd say that - but hacking properly structured
user-space code in the kernel repo is serious fun. It's even
relaxing at times: i can be reasonably sure that i wont crash the
kernel.

All in one, we did this because we found that it produces better
code in practice and does it faster - and i dont think we should
rigidly limit the kernel repo to kernel-space projects alone.

Ingo

2009-06-11 20:49:28

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Hi Ingo,

> > What the "keep it in the kernel sources" approach hopefully allows is
> >
> > - taking advantage of new features in a timely manner.
> >
> > NOT with some ABI breakage, but simply things like supporting a
> > new CPU architecture or new counters. The thing that oprofile
> > failed at so badly in my experience.
> >
> > - Make it easier for developers, and _avoiding_ the horrible
> > situation where you have two different groups that don't talk
> > well to each other because one is a group of user-space
> > weenies, and the other is a group of manly kernel people, and
> > there is no common ground.
>
> Yes, very much agreed.
>
> Btw., here are a couple of other arguments why i find it useful to
> have the tools/perf/ in the kernel repo:
>
> 1) Super-fast and synchronized release cycles
>
> The kernel is one of the fastest moving packages in Linux - most
> user-space packages have (much!) longer release cycles than 3
> months.

that might be true for some projects, but for others this is wrong. You
are just making an assumption out of thin air.

> A tight release schedule forces a certain amount of release
> discipline on tooling as well - so i'm glad that the two will be
> coupled. It's so easy for a promising tool to degrade into
> tinkerware with odd release cycles with time - if it's part of the
> kernel then at least the release cycles wont be odd but at precise 3
> months.

And you can't do that within a perf.git tree on kernel.org because?

> 2) Performance _matters_
>
> This is an argument pretty specific to perfcounters: Performance
> analysis tools under Linux suck pretty summarily. Yet, one of the
> major strengths of Linux is (or at least used to be) performance. So
> i find it very fitting that the kernel community takes performance
> analysis tooling into their own hand.
>
> 3) Strict quality control under a proven mode
>
> In the kernel repo i can be sure that:
>
> - No one will even think of adding autofools to tools/perf/.

That argument is non-sense. While autoconf/automake is maybe not to your
liking, nobody forces you to use it. Projects like git, iw etc. do
perfectly fine without it. I don't mind having autoconf/automake around.

> - No one will send us code with Hungarian notation and two spaces
> tabulation.

What kind of shitty argument it is that. I enforce kernel coding style
in my userspace project all the time. No problem with that.

> - No one will put getopt.h into the code

And that is so bad because?

> - No one will rewrite it in some weird language

And they can do as they please. You don't have to accept the re-write.
These are all non-sense arguments. If you maintain a userspace project
properly then you will not see any of these problems.

> I can point contributors to well-established kernel coding
> principles, without having to argue no end about them.

Come on. A lot of projects use kernel coding style nowadays. That is not
a problem here.

> All in one - the Linux kernel is a fire breathing monster engine
> when it comes to producing good software. Who says it that that this
> infrastructure and experience can only be used to produce kernel
> space code?

And who says that all userspace people have no idea what they are doing.
We have a lot of successful project that follow almost the same rules as
the kernel.

> 4) Code reuse
>
> We actually use code from the kernel: list.h primitives and
> rbtrees.c. We privatized them for now under
> tools/perf/util/rbtree.[ch] and tools/perf/util/list.h because
> there's some header and type pollution in them, but it would be nice
> to include them directly and share the facilities.

Lets see if you are making up an argument or if you are really trying to
work this out and solve it.

> 5) Reality check for kernel developers
>
> I think kernel hackers need a reality check too. It's easy to say
> that user-space sucks - but now there's a way and channel that
> frustration via direct action and make a real difference. I do hope
> that the extra superfluous mental energies visible in this thread
> can be used for good purposes too ;-)
>
> 6) It's a lot of fun
>
> I never thought i'd say that - but hacking properly structured
> user-space code in the kernel repo is serious fun. It's even
> relaxing at times: i can be reasonably sure that i wont crash the
> kernel.
>
> All in one, we did this because we found that it produces better
> code in practice and does it faster - and i dont think we should
> rigidly limit the kernel repo to kernel-space projects alone.

Linus has a bad expierience with oprofile and wants to try something new
and I can follow that argument to a certain degree. I don't agree with
it, but that is fine.

So you are saying that only good code comes from including it into
linux-2.6.git and otherwise you will never get there. Have you actually
tried to maintain this in a separate repository on kernel.org?

Regards

Marcel

2009-06-11 21:06:19

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

>
> So you are saying that only good code comes from including it into
> linux-2.6.git and otherwise you will never get there. Have you actually
> tried to maintain this in a separate repository on kernel.org?

Could you please remind us what the arguments agains including a few
seleted tools within the kernel source tree was.

I ask because I really cannot see why so much nosie is generated?
As a naive user that like easy access to the stuff I work with
this looks like an optimal place to find the kernel-hacking
tools I need. Why should I hunt somewhere else to find it?

Sam

2009-06-11 21:14:35

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

* Marcel Holtmann <[email protected]> wrote:

> Hi Ingo,
>
> > > What the "keep it in the kernel sources" approach hopefully allows is
> > >
> > > - taking advantage of new features in a timely manner.
> > >
> > > NOT with some ABI breakage, but simply things like supporting a
> > > new CPU architecture or new counters. The thing that oprofile
> > > failed at so badly in my experience.
> > >
> > > - Make it easier for developers, and _avoiding_ the horrible
> > > situation where you have two different groups that don't talk
> > > well to each other because one is a group of user-space
> > > weenies, and the other is a group of manly kernel people, and
> > > there is no common ground.
> >
> > Yes, very much agreed.
> >
> > Btw., here are a couple of other arguments why i find it useful to
> > have the tools/perf/ in the kernel repo:
> >
> > 1) Super-fast and synchronized release cycles
> >
> > The kernel is one of the fastest moving packages in Linux - most
> > user-space packages have (much!) longer release cycles than 3
> > months.
>
> that might be true for some projects, but for others this is
> wrong. You are just making an assumption out of thin air.
>
> > A tight release schedule forces a certain amount of release
> > discipline on tooling as well - so i'm glad that the two will be
> > coupled. It's so easy for a promising tool to degrade into
> > tinkerware with odd release cycles with time - if it's part of
> > the kernel then at least the release cycles wont be odd but at
> > precise 3 months.
>
> And you can't do that within a perf.git tree on kernel.org
> because?

We actually tried the tools as separate code, and for the first
three months of the project we only got three contributions - while
the kernel code was essentially finished. (Pekka reported a similar
experience in this thread, with another tool that has close kernel
ties.)

Once we moved it into the same repo as the kernel code (three months
ago), the patches started flowing in - at an amazing rate. We now
have a dozen contributors, most of them kernel developers, and we
have over a hundred good changes to the tools - in just another 3
months.

The key difference was the location of the tools. It is very
convenient and productive to have a shared repository for a project
that frequently involves both kernel and tool changes.

So my point is: this model clearly works in practice and all the
current tools/perf/ contributors like this kind of coding
environment.

Most of your arguments seem to center around the notion that it
could all be done in a separate repo too and that such a repo could
be run as well as the Linux kernel. If you think you could do it
even better in a separate repo you are certainly free to try it.

Ingo

2009-06-11 21:17:27

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Hi Sam,

> > So you are saying that only good code comes from including it into
> > linux-2.6.git and otherwise you will never get there. Have you actually
> > tried to maintain this in a separate repository on kernel.org?
>
> Could you please remind us what the arguments agains including a few
> seleted tools within the kernel source tree was.
>
> I ask because I really cannot see why so much nosie is generated?
> As a naive user that like easy access to the stuff I work with
> this looks like an optimal place to find the kernel-hacking
> tools I need. Why should I hunt somewhere else to find it?

I personally would expect a perf.git on kernel.org for the userspace
tools for it. Like we have udev.git there, iproute2.git and others.

Seems to be working perfectly fine (except of course oprofile) and makes
packaging and security updates a lot easier. The distros have always a
really hard problem with releasing new kernel packages. And as long as
the source changes the whole set of binary packages needs to be rebuilt
and in theory if you install a new kernel, you should reboot. So if
there is an issue in perf userspace, then the current processes in most
distros will propose the user a reboot for no good reason.

There is nothing wrong with trying something new, but to be honest I
don't buy into the arguments why we do it. It seems like it is all based
on bad experience with some userspace maintainers and not really
technical grounds why it is a must to have this inside the kernel source
code. Of course you can make the argument the other way around and say
why not. And I give Linus that he wants to try. However all the
arguments from Ingo are a joke and basically tells that all userspace
developers have no clue and can't get right anyway.

Maybe it is just a sneaky attempt to get a higher hit in Greg's
statistics by just writing some userspace code which otherwise would not
be counted ;)

Regards

Marcel

2009-06-11 21:24:26

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

2009-06-11 22:00:01

by Steven Rostedt

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 11:17:16PM +0200, Marcel Holtmann wrote:
> Hi Sam,
>
> > > So you are saying that only good code comes from including it into
> > > linux-2.6.git and otherwise you will never get there. Have you actually
> > > tried to maintain this in a separate repository on kernel.org?
> >
> > Could you please remind us what the arguments agains including a few
> > seleted tools within the kernel source tree was.
> >
> > I ask because I really cannot see why so much nosie is generated?
> > As a naive user that like easy access to the stuff I work with
> > this looks like an optimal place to find the kernel-hacking
> > tools I need. Why should I hunt somewhere else to find it?
>
> I personally would expect a perf.git on kernel.org for the userspace
> tools for it. Like we have udev.git there, iproute2.git and others.
>
> Seems to be working perfectly fine (except of course oprofile) and makes
> packaging and security updates a lot easier. The distros have always a
> really hard problem with releasing new kernel packages. And as long as
> the source changes the whole set of binary packages needs to be rebuilt
> and in theory if you install a new kernel, you should reboot. So if
> there is an issue in perf userspace, then the current processes in most
> distros will propose the user a reboot for no good reason.
>
> There is nothing wrong with trying something new, but to be honest I
> don't buy into the arguments why we do it. It seems like it is all based
> on bad experience with some userspace maintainers and not really
> technical grounds why it is a must to have this inside the kernel source
> code. Of course you can make the argument the other way around and say
> why not. And I give Linus that he wants to try. However all the
> arguments from Ingo are a joke and basically tells that all userspace
> developers have no clue and can't get right anyway.

Here's another point that I have not really seen anyone make. The tools that
would be packaged with the kernel are the ones that I would expect the average
kernel developer to use. Things to help us in developing better code.

The tools you mentioned

"ip, iw, rfkill, crda, the WiMAX"

I have no idea what they do. I don't think I would use them as I don't
work on bluetooth, and I don't see how they would help me with what I do
work on.

I use 'udev' only to boot my machine, and I only notice it when it doesn't
work.

As for something like perf, that is something I can see myself using to
analyze my own code. And I can see other developers (even you) using it for
the same purpose. This is a tool that I would like to have the latest version
for the latest version of the kernel I am developing on. That is, if the
latest kernel had a new feature that perf can take advantage of, it would be
nice to have it with the new kernel I just pulled.

This could also work with a perf.git, but I would probably not bother with
it if I had to keep checking the perf.git repo to see if it uses the
new features that are in the kernel. I constantly do 'git pull' for the
kernel and I would get the latest perf with the latest kernel and I
would not need to bother checking someplace else.

Actually, I can also see that if a new feature in the kernel was added that
perf uses, I would probably notice it first with compiling perf and doing
a perf --help.

>
> Maybe it is just a sneaky attempt to get a higher hit in Greg's
> statistics by just writing some userspace code which otherwise would not
> be counted ;)

No, that would be something that I do ;-)

/me plans on sending patches for perf.

-- Steve

2009-06-11 22:18:59

by Jiri Slaby

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Hi.

On 06/11/2009 11:26 PM, Sam Ravnborg wrote:
> On Thu, Jun 11, 2009 at 11:17:16PM +0200, Marcel Holtmann wrote:
>> I personally would expect a perf.git on kernel.org for the userspace
>> tools for it. Like we have udev.git there, iproute2.git and others.
>>
>> Seems to be working perfectly fine (except of course oprofile) and makes
>> packaging and security updates a lot easier.
> There is nothing preventing us from adding support for rpm and source rpms.
> So you just grab the relevant tre and issue a few cammnds and you have your
> packages.

Bah, having 40M .src.rpm for a 5k binary package?

Maybe I'm missing something, how exactly do you conceive the packaging?
Or do you expect packagers to download a kernel package, untar it, get
tools/ dir, tar it and package? I hope not :).

And how would we cope with a different release cycle of the userspace
tool? If one rewrites a part totally independent on the kernel, do they
need to wait for the next kernel release? Or just merge it at any time
and packagers pick it up?

> And for security fixes we have the stable kernels.

So packagers will stick with the latest stable, right? With backporting
of (only stable) new fancy features from current git until next kernel
release.

2009-06-11 22:29:19

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Fri, 12 Jun 2009, Jiri Slaby wrote:
>
> Bah, having 40M .src.rpm for a 5k binary package?

Why do people who don't even know how packaging works bother to even
participate in the discussion?

Look at how many git binary packages there are some day. For CVS users,
for SVN people, graphical tools etc. Do you think that each of them has a
source package?

No.

You can generate multiple binary packages from the same source package
(trivial example: debug builds etc). But you want to make a point, and
then YOU USE SOME DAMN IDIOTIC AND IGNORANT argument to do so.

Not smart.

Linus

2009-06-11 22:39:51

by Alan

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

> Look at how many git binary packages there are some day. For CVS users,
> for SVN people, graphical tools etc. Do you think that each of them has a
> source package?

That misses the point, at least for the systems as the work now. Having a
single source package to multiple binaries is easy. Managing setups where
you push only some of those binaries into the system gets really ugly.

A pile of git-foo packages works because you almost never push a fix to
one tool alone, and if you do its small enough not to be a big deal.

2009-06-11 22:50:39

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 Jun 2009, Alan Cox wrote:
>
> That misses the point, at least for the systems as the work now. Having a
> single source package to multiple binaries is easy. Managing setups where
> you push only some of those binaries into the system gets really ugly.

Umm. But what's the problem?

Sure, you'd always update the 'kernel-perftool' package (or whatever you'd
call it) when you update the kernel. But so what? It's going to be tiny.
And appropriate.

IOW, where's the downside?

Linus

2009-06-11 23:20:18

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 03:27:36PM -0700, Linus Torvalds wrote:
>
>
> On Fri, 12 Jun 2009, Jiri Slaby wrote:
> >
> > Bah, having 40M .src.rpm for a 5k binary package?
>
> Why do people who don't even know how packaging works bother to even
> participate in the discussion?
>
> Look at how many git binary packages there are some day. For CVS users,
> for SVN people, graphical tools etc. Do you think that each of them has a
> source package?
>
> No.
>
> You can generate multiple binary packages from the same source package
> (trivial example: debug builds etc). But you want to make a point, and
> then YOU USE SOME DAMN IDIOTIC AND IGNORANT argument to do so.

Linus, the real question that needs to be answered is this:

What shall be done to ABI-breaking changes when users of that ABI are
in tools/*?

_That_ is the real issue. Because I can guarantee that there will be attempts
to use that as an excuse for ABI breakage. We have one specimen in this
thread already, complete with "oh, bisect problems don't matter, just rebuild
all packages" (and install them where, exactly? if it, say, break-the-boot
kind of incompatibility, how does one recover from running into a b0rken
kernel during bisect?)

If you are willing to ban that kind of crap - great; there are real remaining
issues (mostly with choosing the dependencies between binary packages), but
that's more or less survivable. If not... we'll have one hell of a PITA
to deal with when that kind of excuse gets actually used.

2009-06-11 23:26:32

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Fri, 12 Jun 2009, Al Viro wrote:
>
> Linus, the real question that needs to be answered is this:

No it's not.

People have already told you that the intent isn't to change the ABI. So
your whole "hard-hitting journalism" is just bogus posturing.

What does this have to do with anything?

Linus

2009-06-12 00:26:56

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 04:25:19PM -0700, Linus Torvalds wrote:
>
>
> On Fri, 12 Jun 2009, Al Viro wrote:
> >
> > Linus, the real question that needs to be answered is this:
>
> No it's not.
>
> People have already told you that the intent isn't to change the ABI. So
> your whole "hard-hitting journalism" is just bogus posturing.
>
> What does this have to do with anything?

Oh, for... I can bloody well read, I've seen the reply from Peter and
I've no reasons to doubt his words (and if I had, I would've said so).
Not the issue. I don't know who you are confusing me with, but for the
record - I have no problem with this particular code being in tree.

I do have a problem with another thing: suggestions I've heard quite a few
times before; basically, "let's allow special breakable ABIs for use by
userland code living in kernel tree and tied to specific version". No,
I'm not saying that this is what's happening with that merge. But your
support for userland code in the tree (and BTW, I agree that it's a good
idea - hell, mount(8) makes a good candidate as far as I'm concerned) will
be parsed as green light for that. Has been already, in this thread.

So could you please clarify the situation? If the ABI compatibility
requirements remain the same as they used to be, whether the userland code
is in-tree or not, I'm fine with the entire thing. If they do not (and *ONLY*
in that case), I think we have a real problem.

For the record, I don't give a damn about packaging-related arguments and
theories about keeping userland source separate as a matter of some principle.
As far as I'm concerned, it's not a problem - as long as we take care of
later version's $TOOL working on older kernel as well as $TOOL from that
older kernel used to work, I'm fine with it.

I realize that multi-side flamefests are messy, but let's keep track of
who's saying what, OK?

2009-06-12 02:01:39

by tip-bot for Robert Richter

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On 11.06.09 12:49:25, Linus Torvalds wrote:
>
>
> On Fri, 12 Jun 2009, David Newall wrote:
>
> > Linus Torvalds wrote:
> > > To take the oprofile example that decided it for me: the code to actually
> > > support new processors was all done by basically kernel developers. And it
> > > didn't hit user land for almost a year, because the user-land tools didn't
> > > take the patch and propagate it up.
> >
> > Bad developer, Spot, you only did half the job. Not sure there's much
> > more one can say.
>
> Umm. The kernel developer _did_ do the job. The patch to the user land
> side was available for that whole year. It just didn't get merged, and
> then didn't get merged some more, and then got merged but only in a SVN
> tree, not a release, and then finally when I did a bugzilla request to
> fedora, they took the patch and put it in their distro.

Having the oprofile user land in the kernel would not solve the
problem. Then you would have code in the kernel tree you actually
don't wont there: XML encoder, autoconf scripts, graphical tools, c++
code, man page docs, etc., and maybe different coding style.

The problem is another one. First, as Christoph mentioned, it is a
design problem of oprofile. Changes in the kernel require user land
changes. This could be done better, but everybody knows it is hard to
change the user/kernel i/f and maybe, keep backward compatibility
too. So this is not easy to fix.

Second, there are different users with different expectations. (Linus,
I suggest oprofile has one user less, hmm...) Some run the latest
kernel on the latest systems, others use it in their clusters using
stable, not often changing well tested releases and hardware. If a
user land release aims more the seconds, it must conflict with the
others. Also, being in sync with the kernel would require release
cycles as for the kernel, which was the problem here with oprofile.

But, user land patches exist, even at the day of the kernel
release. Otherwise the code would have been badly or not tested. And
the patches are also in a repository, _somewhere_. This, was true for
oprofile too, the patches were in cvs at least on the day the kernel
was released. (I think a git repository would be nicer, but that's a
different question.) And this is the next problem, the patches are
somewhere, sometimes not under control of the kernel developer. And
this could be best solved if the kernel developer who brings the
kernel code upstream maintains a user land repository at
git.kernel.org. (Marcel already suggested this too.) There could be
all patches in required to run the latest kernel, based on the latest
user land release. (You can blame then the kernel maintainer, if
something does not work.) And of course the user is required then to
compile the user land himself, as he does for the kernel. And maybe,
distros pick up the patches too when adding a new kernel to it.

So, I think this would be much nicer than having a user land in the
kernel tree. And this would also solve the problems with the oprofile
user land.

-Robert

--
Advanced Micro Devices, Inc.
Operating System Research Center
email: [email protected]

2009-06-12 03:00:00

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Fri, 12 Jun 2009, Al Viro wrote:
>
> So could you please clarify the situation? If the ABI compatibility
> requirements remain the same as they used to be, whether the userland code
> is in-tree or not, I'm fine with the entire thing. If they do not (and *ONLY*
> in that case), I think we have a real problem.

I think the ABI requirements are the same.

That said, I also suspect that as with oprofile itself, we'll end up
having expansions of the ABI that may well be CPU-specific. I also suspect
that there will probably be breakage early on just because things will
inevitably settle.

And I think that for something like a profiling tool, such breakage is
much more acceptable than for the actual binaries you'd profile. It's not
like we're talking about breaking the boot or functionality of a machine,
as happens when we break the X server (which has happened).

Linus

2009-06-12 03:22:17

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Linus Torvalds wrote:
> On Fri, 12 Jun 2009, David Newall wrote:
>> Linus Torvalds wrote:
>>
>>> To take the oprofile example that decided it for me: the code to actually
>>> support new processors was all done by basically kernel developers. And it
>>> didn't hit user land for almost a year, because the user-land tools didn't
>>> take the patch and propagate it up.
>>>
>> Bad developer, Spot, you only did half the job. Not sure there's much
>> more one can say.
>>
>
> Umm. The kernel developer _did_ do the job. The patch to the user land
> side was available for that whole year.

I don't know this oprofile problem you had, only what you've said, which
is that somebody* did the kernel bit and somebody else did the userspace
bit; and the person doing the userspace bit was unresponsive so good
stuff got ignored for a year. That situation did not occur because the
userspace was out-of-tree, it occurred because you let it. You could
have given the userspace (back) to the kernel developer. That's what
you'd eventually do if a kernel sub-system maintainer became
unresponsive, isn't it?

*the singular is intended to include the plural and the male to include
female.

> Anyway, it's clearly not worth discussing this with you. I've tried. I
> give up. Happily, I don't _need_ to convince you.

Indeed, no, you don't need to convince me, particularly as I've made it
abundantly clear that I'm entirely happy with your decision. Notice I've
not argued with you, merely pointed out inconsistencies in what you've
said. I realise that can be annoying, and acknowledge your absolute
right to be as consistent or inconsistent as you choose. I wasn't (and
still aren't) trying to be annoying, but to confirm there was no
confusion. If you hadn't seen these inconsistencies before, you surely
do now. That actually should be worth your while. I personally welcome
being corrected; andconsider that a trait of an open mind.

2009-06-12 04:06:23

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, Jun 11, 2009 at 07:58:37PM -0700, Linus Torvalds wrote:
>
>
> On Fri, 12 Jun 2009, Al Viro wrote:
> >
> > So could you please clarify the situation? If the ABI compatibility
> > requirements remain the same as they used to be, whether the userland code
> > is in-tree or not, I'm fine with the entire thing. If they do not (and *ONLY*
> > in that case), I think we have a real problem.
>
> I think the ABI requirements are the same.

OK, then.

> That said, I also suspect that as with oprofile itself, we'll end up
> having expansions of the ABI that may well be CPU-specific. I also suspect
> that there will probably be breakage early on just because things will
> inevitably settle.
>
> And I think that for something like a profiling tool, such breakage is
> much more acceptable than for the actual binaries you'd profile. It's not
> like we're talking about breaking the boot or functionality of a machine,
> as happens when we break the X server (which has happened).

Sure.

2009-06-12 04:08:13

by Kyle McMartin

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

[With my Fedora on.]

On Thu, Jun 11, 2009 at 10:06:55AM -0700, Linus Torvalds wrote:
>
>
> On Thu, 11 Jun 2009, Christoph Hellwig wrote:
> >
> > So what point is there in keeping it in-tree except making life hell for
> > packagers?
>
> Give it up. Packagers can trivially generate their own sub-packages. They
> do it all the time. They already do it for the user-mode header files,
> extracted from the kernel - something you've worked on yourself.
>
> So your point is clearly bogus, and dishonest.
>
> You haven't actually looked the real problem in the eye, and acknowledged
> the disaster that is oprofile. Let's give a _new_ approach a chance, and
> see if we can avoid the mistakes of yesteryear this time.
>

This is actually somewhat complicated for (at least, I can only speak
from experience for...) Fedora and Debian/Ubuntu. Having this in-kernel
means any bugfixes needed for the 'perf' tool, require patching the
kernel source, which will result in a whole new kernel rpm being built.
So in order to update their 'perf' tool, users will get a new kernel,
debuginfo, etc., with it.

So either we need to split it out into its own source tarball, or ship
the kernel source again in a seperate source package. I know which I'm
going to tend to favour...

Obviously, I understand the reasons for doing this, but I don't really
see it as a sensible long term option for a mature tool. But,
whatever, it's not my call. We'll just work around whatever happens.

regards, Kyle

2009-06-12 07:35:49

by Alan

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

> Sure, you'd always update the 'kernel-perftool' package (or whatever you'd
> call it) when you update the kernel. But so what? It's going to be tiny.
> And appropriate.
>
> IOW, where's the downside?

Why you need to update the perftool not the kernel, which is very likely
to be the case early on. There are ways for vendors to cope anyway - such
as by deleting the tools directory from their kernel and keeping a
separate perftool package that gets updated now and then from the kernel
tree but is otherwise a fork

Alan

2009-06-12 09:57:04

by Stephane Eranian

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Hi,

On Thu, Jun 11, 2009 at 6:03 PM, Ingo Molnar <[email protected]> wrote:

> The counter concept got objected to in past discussions on lkml, by
> DaveM and by Stephane Eranian (i've Cc:-ed them) - so this code was
> not eligible for linux-next testing - nevertheless we gave it good
> testing on PowerPC and x86 and i've done a wide cross-build test as
> well to try to make sure it breaks no other architecture.

I don't think you can quote me saying "I object to this code". I posted
a detailed review of the API and implementation on X86 outlining lots
of issues. Some got fixed, but many others are left unresolved at this
point. And I will post some more shortly.

I don't think that because this code is coming from you, it should be
allowed to short-circuit the established release process. You have
to respond to questions, fix issues like everybody else and if that
slows down the integration you cannot blame the reviewers for it.

2009-06-12 10:20:21

by Jörn Engel

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

On Thu, 11 June 2009 17:59:41 -0400, Steven Rostedt wrote:
>
> The tools you mentioned
>
> "ip, iw, rfkill, crda, the WiMAX"
>
> I have no idea what they do.

ip I can explain to you. Ten years back when I was a netadmin I faced
the problem of implementing traffic shaping of some sorts. Details
don't matter much. After a very short while I learned that ip was the
solution to my problem. One week later I started digging into the
kernel code because I simply couldn't work out how to use this thing.
Another week later I was playing with the idea of writing my own traffic
shaper in the kernel instead. It was that bad.

Then I found something called tinybsd, a bsd distro on one floppy disk.
We allotted an old 486 with two network cards, I spent some
uncomfortable time configuring the beast with the crappy editor you can
expect on 1.44MB and the thing just worked henceforth.

Oh, the bsd had their equivalent of ip tightly coupled with their
kernel. Not sure if that caused the marked difference, but I'll gladly
add this shred of anecdotal support.

[ And in case someone takes offence or considers me an idiot for not
being able to use ip or tc, I would _love_ to see a howto explaining how
one can limit the amount of traffic on one interface to - say - 1GB per
month. ]

Jörn

--
Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface.
-- Doug MacIlroy

2009-06-12 10:28:27

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

* stephane eranian <[email protected]> wrote:

> Hi,
>
> On Thu, Jun 11, 2009 at 6:03 PM, Ingo Molnar <[email protected]> wrote:
>
> > The counter concept got objected to in past discussions on lkml,
> > by DaveM and by Stephane Eranian (i've Cc:-ed them) - so this
> > code was not eligible for linux-next testing - nevertheless we
> > gave it good testing on PowerPC and x86 and i've done a wide
> > cross-build test as well to try to make sure it breaks no other
> > architecture.
>
> I don't think you can quote me saying "I object to this code".
> [...]

I never saw you retract/change this negative opinion of yours about
the whole separate-counters concept:

" In summary, although the idea of simplifying tools by moving the
complexity elsewhere is legitimate, pushing it down to the
kernel is the wrong approach in my opinion, perfmon has avoided
that as much as possible for good reasons. "

http://lkml.org/lkml/2008/12/5/359

Do you like the concept now? That would be great news - you have a
lot of experience with various PMU details and we could certainly
welcome help with the perf tool and with the kernel side of
perfcounters!

> [...] I posted a detailed review of the API and implementation on
> X86 outlining lots of issues. Some got fixed, but many others are
> left unresolved at this point. And I will post some more shortly.

Hm, Peter replied to you mail a week ago, in detail. We addressed a
good number of issues pointed out by you, and we credited you for
them:

earth4:~/tip> git log v2.6.30..linus | grep 'Reported-by: Stephane Eranian'
Reported-by: Stephane Eranian <[email protected]>
Reported-by: Stephane Eranian <[email protected]>
Reported-by: Stephane Eranian <[email protected]>
Reported-by: Stephane Eranian <[email protected]>
Reported-by: Stephane Eranian <[email protected]>
Reported-by: Stephane Eranian <[email protected]>
Reported-by: Stephane Eranian <[email protected]>

You were on the Cc: of the commit notifications. If you see issues
left unaddressed after reply+commit please repeat it - it probably
just got lost in noise.

> I don't think that because this code is coming from you, it should
> be allowed to short-circuit the established release process. You
> have to respond to questions, fix issues like everybody else and
> if that slows down the integration you cannot blame the reviewers
> for it.

There's three maintainers of perfcounters: Peter Zijlstra, Paul
Mackerras and me - and if some real problem missed the attention of
all of us then please repeat it - it probably was just missed in a
bigger mail or so. I certainly dont remember anything major. We
generally try to reply to any and all feedback.

Thanks,

Ingo

2009-06-15 13:56:57

by Giacomo A. Catenazzi

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Sam Ravnborg wrote:
> On Thu, Jun 11, 2009 at 09:26:55AM -0700, Linus Torvalds wrote:
>>
>> On Thu, 11 Jun 2009, Christoph Hellwig wrote:
>>> Err, no. This adds tons of userspace code into tools/ which
>>> should not be in the kernel tree but a proper package.
>> I disagree.
>>
>> We've had tons of cases where we tried to "separate" the user-land code
>> and the kernel code, in the name of "beauty" of whatever.
>>
>> It's almost invariably a disaster.
>
> This is cheating. I had this as a topic for the kernel summit and
> was looking forward to read an interesting article about people
> dancing on the table and fighting in the corners about it.
> [I do not attend myself]
>
> People say that this would be a nightmare for the packagers.
> I frankly do not see what the issue is here.

Kernels don't fit well in distribution models.
We have distribution since 15 (and more) years, but still with
hackish support for kernels.

Kernel:
- people are used to install multiple "parallel" kernels
- from different sources (distribution, kernel.org)
- and a lot of people configure own kernel

This is a lot different of usual packages:
- packages have dependencies (done at pre-installation time)
- packages normally support only upgrades (and not downgrades)
- support for multiple version exist only on libraries (SONAME)

Thus a program could depends on specific version of the libc,
but it cannot depends on a specific kernel (system doesn't know
the kernel of next boot), which requires a lot of hack in init.d
scripts.

BTW one of the most frequent question on distribution was
about configuring the kernel and the error about missing
lib[n]curse[X]-dev[el].

So the 15 year without finding a good solution could explains the
nightmare, (but it could be finally the opportunity to really
solve the problem).

To conclude: a user space program should not only have a stable
ABI, but also have nice messages about unsupported features
(and wrong kernel) and not changing runtime dependencies like
socks.

ciao
cate

2009-06-15 15:15:59

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

>
> Kernels don't fit well in distribution models.
> We have distribution since 15 (and more) years, but still with
> hackish support for kernels.

Because the source are in the same git tree as the kernel does
not require the source to be considered a kernel.

You are talking from the wrong assumption that because the
source live in the git tree of the kernel it has hard
dependencies on that particular kernel.
This is wrong. There is the same requirements about being
backward/forward compatible as if the tool
lived in a git tree outside the kernel.

What we achieve by letting the userspace tools live inside
the kernel are:
- Kernel hacker tools are avaialble for the kernel hackers
- The tools are easy to upgrade when we add new stuff to the kernel
- We know that the updates are done upstream
- We have the kernel base of developers at easy reach,
no need to build up a new development community

Sam

2009-06-18 21:58:50

by Stephane Eranian

[permalink] [raw]

Subject: Re: [GIT PULL] Performance Counters for Linux

Hi,

On Fri, Jun 12, 2009 at 12:28 PM, Ingo Molnar<[email protected]> wrote:
>
>>
>> On Thu, Jun 11, 2009 at 6:03 PM, Ingo Molnar <[email protected]> wrote:
>>
>> > The counter concept got objected to in past discussions on lkml,
>> > by DaveM and by Stephane Eranian (i've Cc:-ed them) - so this
>> > code was not eligible for linux-next testing - nevertheless we
>> > gave it good testing on PowerPC and x86 and i've done a wide
>> > cross-build test as well to try to make sure it breaks no other
>> > architecture.
>>
>> I don't think you can quote me saying "I object to this code".
>> [...]
>
> I never saw you retract/change this negative opinion of yours about
> the whole separate-counters concept:
>
> " In summary, although the idea of simplifying tools by moving the
> complexity elsewhere is legitimate, pushing it down to the
> kernel is the wrong approach in my opinion, perfmon has avoided
> that as much as possible for good reasons. "
>
> http://lkml.org/lkml/2008/12/5/359
>

I, indeed, did not retract because I still have reservations about the approach
even after 6 months of intense development.

> Do you like the concept now? That would be great news - you have a
> lot of experience with various PMU details and we could certainly
> welcome help with the perf tool and with the kernel side of
> perfcounters!
>

I still have reservations. I could be convinced, though. But for that to happen,
there are a couple of milestones that need to be reached:
- Full Intel Nehalem support: core PMU, uncore PMU, LBR, PEBS
(incl. load latency),
offcore_response.
- Full Intel Itanium 2 dual-core (Montecito) support: D-EAR,
I-EAR, opcode matching, range
restrictions, user level control

Those represent very advanced and very useful PMUs. Having implemented
user and kernel
support for both of them, I can attest that they challenge any
interfaces. But perfmon is the proof
that those can be exposed with their full strength thru a generic
kernel API. Therefore, I am
relatively hopeful, there should be a way to expose them through your API.

Another important consequence of your design is that the event
assignment logic is in the kernel.
As discussed early on, this can be quite complicated. Today, you only
have very partial support
for architected Intel X86 and AMD64 processors (I know about Power). I
am sure you will update
this shortly. But I think getting complete support for Intel Nehalem
and Itanium 2 in that area is
another important milestone.

Concerning help, I am sure you realize I am already helping you out by posting
detailed reviews. I have yet to see anybody else posting this kind of
information
concerning your API. I will keep posting as I find new issues. My goal is not to
torpedo this API, it's already upstream anyway, but instead I am
trying to ensure
it does what I want based on my experience developing tools, talking with PMU
architects and feedback from tool developers.

I think we could have a much more constructive working relationship if
people showed
some more respect for the work I and many others have done. Perfmon
certainly has
issues and could be implemented better. You certainly have better
skills than me in that
area. I have no problem admitting that. But I do not think perfmon
deserves the kind of
comments I have seen, repeated over and over, from you and Zijlstra
since December.
Regardless of your personal opinion, perfmon deserves some credit for
what it has offered
to many people around the world. If it had been as bad as you
described it, it could not
possibly have supported all the PMUs and their advanced features.
Nobody would have
used it. But this is not what happened.

>> [...] I posted a detailed review of the API and implementation on
>> X86 outlining lots of issues. Some got fixed, but many others are
>> left unresolved at this point. And I will post some more shortly.
>
> Hm, Peter replied to you mail a week ago, in detail. We addressed a
> good number of issues pointed out by you, and we credited you for
> them:
>
> earth4:~/tip> git log v2.6.30..linus | grep 'Reported-by: Stephane Eranian'
> Reported-by: Stephane Eranian <[email protected]>
> Reported-by: Stephane Eranian <[email protected]>
> Reported-by: Stephane Eranian <[email protected]>
> Reported-by: Stephane Eranian <[email protected]>
> Reported-by: Stephane Eranian <[email protected]>
> Reported-by: Stephane Eranian <[email protected]>
> Reported-by: Stephane Eranian <[email protected]>
>
I know. I appreciate that. I wish you had also acknowledged the fact
that I suggested that you split the config field into type and config fields
in my initial posting. I had to discover this change by looking at the GIT
log.

2009-06-22 13:11:17