First, thank-you, Hubertus, for comparing me to a puppy, rather
than a kitten. I am definitely a dog person, not a cat person,
and I appreciate your considerate choice of analog.
I gather from the tone of your post yesterday that there is
a disconnect between us - you speak with the frustration of
someone who has been shouting into the wind and not being
heard.
I suspect that the disconnect, if such be, is not where you
think it is:
Hubertus wrote:
>
> The disconnect is that you do not want to recognize that CKRM does NOT
> have to be systemwide. Once you open your mind to the fact that CKRM can
> be deployed with in a subset of disconnected resources (cpu domains)
> and manages shares independently within that domain, I truely don't see
> what the problem is.
I have recognized for months that eventually we'd want to allow
for cpuset-relative CKRM domains, and I'm pretty sure I've
dropped comments to that affect one time or another here on lkml.
I suspect instead that "CKRM" is one layer more abstract than
I am normally comfortable with.
As best as I can tell, CKRM has evolved from its origins as a
fair share scheduler, into a framework (*) for things called by
such names as classes and controllers. As you may recall from
an inconclusive thread between us on the ckrm-tech email list two
months ago, I find those terms uncomfortably vague and abstract.
In general, frameworks are high risk business. What they
gain in generality, covering a wider range of situations in
a uniform pattern, they lose in down to earth concreteness,
leaving their users less confident of what works, and less able
to rely on their intuitions. The risk of serious design flaws,
shrouded for a long time in the fog of abstraction, is higher.
The more successful frameworks, such as vfs for example,
typically have deep roots in prior art, and a sizable population
of journeyman and master practitioners.
CKRM is young, its roots more shallow, and the population of
its practitioners small.
(*) P.S. - It's more like CKRM is now the combination of
a virtual resource manager framework and a particular
instance of such (the fair shair controllers that have
their conceptual origins in IBM's WLM, I suspect). If
numa placement controllers (aka cpusets) are going to
exist as well, then CKRM needs to split into (1) a
virtual resource manager framework (vrm), and (2) the
fair share stuff. The vrm framework should be neutral
of either fair share or numa placement bias.
===
So here I am with this new cpuset design (Simon Derr, primary
architect, both Simon and I feel a strong sense of ownership)
for numa placement, perhaps the 4th or 5th in SGI's history,
and the 2nd in mine. I am finding that it deliciously and
elegantly reflects the needs of its anticipated users (Sylvain
might demur, noting a couple of things I removed).
I am now being asked to morph it into a CKRM controller.
Further I deduce from the efforts over the last few days to talk
me down from meeting all the requirements satisfied by my current
cpuset patch that something of cpusets will be lost in the translation.
But I haven't figured out exactly what will be lost. And I lack the
mastery of CKRM that would enable me to engage in a constructive dialog
on the various tradeoffs that come into play here.
I look at the CKRM patch, and see something that looks an order
of magnitude larger than my cpuset patch. With its increased
number of hooks in the kernel, and its more abstract style
(it is a framework afterall), I also see something with a
higher risk of performance impact, especially on the large NUMA
configurations that I care about.
And I am looking at trading what I thought had hope of being a
Sept or Oct date for acceptance into Linus's kernel, into some
unknown schedule that is definitely further out.
I've got the bacon sizzling on the skillet, I can smell it, my
mouth is watering, and just as I go to lift it off the burner,
Andrew asks me to consider trading it for a pig in a poke.
Thanks a bunch, Andrew - you da man ;).
Putting aside for a moment my personal frustrations (which
are after all my problem - and my dogs) I am simply unable to
make sense yet of how deep would be the hit on the capabilities
of cpusets, if so morphed, and I am painfully aware of the
undetermined schedule delays and increased risks to product
performance and even ultimate success that attend such a change.
>From what my field engineers tell me, whom I've been polling
furiously on this matter the last few days, at least in the
markets that SGI frequents, there is very little overlap between
system configurations which benefit from fair share resource
management and those which benefit from numa placement resource
management. So, if that experience is generally applicable, we
are at risk of marrying a helicopter and a boat, just because
both have a motor and a hull, to the detriment of both.
Merging projects always has risks. The payoff for synergies
gained is not always greater than the cost of the inefficiencies
and compromises introduced, and the less immediate involvement
of the participants in the end result.
I cannot in good conscience recommend such a change.
Keep talking.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373
Paul Jackson wrote:
> First, thank-you, Hubertus, for comparing me to a puppy, rather
> than a kitten. I am definitely a dog person, not a cat person,
> and I appreciate your considerate choice of analog.
Heeeh .. where did I compare you to a puppy? I was talking about *MY*
puppy. And the moral of the story was that you can teach them new
tricks. That's all. So in case you took this in any other way my sincere
apologies.
>
> I gather from the tone of your post yesterday that there is
> a disconnect between us - you speak with the frustration of
> someone who has been shouting into the wind and not being
> heard.
Frustration ... only in the sense that what seems to me to
be a pretty clear path to melt your functionality into CKRM.
Andrews, intial request was the challenge to see whether CKRM suffice as
an API and with an additional controller suffices to provide the
functionality.
>
> I suspect that the disconnect, if such be, is not where you
> think it is:
>
> Hubertus wrote:
>
>>The disconnect is that you do not want to recognize that CKRM does NOT
>>have to be systemwide. Once you open your mind to the fact that CKRM can
>>be deployed with in a subset of disconnected resources (cpu domains)
>>and manages shares independently within that domain, I truely don't see
>>what the problem is.
>
>
> I have recognized for months that eventually we'd want to allow
> for cpuset-relative CKRM domains, and I'm pretty sure I've
> dropped comments to that affect one time or another here on lkml.
>
> I suspect instead that "CKRM" is one layer more abstract than
> I am normally comfortable with.
>
> As best as I can tell, CKRM has evolved from its origins as a
> fair share scheduler, into a framework (*) for things called by
> such names as classes and controllers. As you may recall from
> an inconclusive thread between us on the ckrm-tech email list two
> months ago, I find those terms uncomfortably vague and abstract.
>
> In general, frameworks are high risk business. What they
> gain in generality, covering a wider range of situations in
> a uniform pattern, they lose in down to earth concreteness,
> leaving their users less confident of what works, and less able
> to rely on their intuitions. The risk of serious design flaws,
> shrouded for a long time in the fog of abstraction, is higher.
>
> The more successful frameworks, such as vfs for example,
> typically have deep roots in prior art, and a sizable population
> of journeyman and master practitioners.
>
> CKRM is young, its roots more shallow, and the population of
> its practitioners small.
>
> (*) P.S. - It's more like CKRM is now the combination of
> a virtual resource manager framework and a particular
> instance of such (the fair shair controllers that have
> their conceptual origins in IBM's WLM, I suspect). If
> numa placement controllers (aka cpusets) are going to
> exist as well, then CKRM needs to split into (1) a
> virtual resource manager framework (vrm), and (2) the
> fair share stuff. The vrm framework should be neutral
> of either fair share or numa placement bias.
As indicated in many notes so are the usage of cpusets.
Very few people have the #cpus to even worry about this.
As Andrew said, its quite possible that the installations can maintain
their own kernel, although
>
> ===
>
>
> Putting aside for a moment my personal frustrations (which
> are after all my problem - and my dogs) I am simply unable to
> make sense yet of how deep would be the hit on the capabilities
> of cpusets, if so morphed, and I am painfully aware of the
> undetermined schedule delays and increased risks to product
> performance and even ultimate success that attend such a change.
>
> From what my field engineers tell me, whom I've been polling
> furiously on this matter the last few days, at least in the
> markets that SGI frequents, there is very little overlap between
> system configurations which benefit from fair share resource
> management and those which benefit from numa placement resource
> management. So, if that experience is generally applicable, we
> are at risk of marrying a helicopter and a boat, just because
> both have a motor and a hull, to the detriment of both.
I learned my lesson, no more analogies with you....
Bottom line I believe the cpusets should be first morphed into
sched_domains. The problem with large systems is the load balancing
which is highly unscalable. You can twist an turn but at the end of
the day you set cpu_affinity masks. The load balancing of the system
needs to be aware of the structure of the system.
sched_domains to me are the right approach for that not setting some
affinity masks underneath.
Assuming that will be resolved at some point and given Andrew's
hypothetical assumption that CKRM makes it into his kernel, then
I don't see the obstacle of adopting an existing API to serve at the
API for cpusets/sched_domains.
>
> Merging projects always has risks. The payoff for synergies
> gained is not always greater than the cost of the inefficiencies
> and compromises introduced, and the less immediate involvement
> of the participants in the end result.
>
> I cannot in good conscience recommend such a change.
But by self admission, you are driven by timing constraints as
your bacon is sizzling.
>
> Keep talking.
To whom ? :-)
-- Hubertus
Hubertus, responding to Paul:
> > (*) P.S. - It's more like CKRM is now the combination of
> > a virtual resource manager framework and a particular
> > instance of such (the fair shair controllers that have
> > their conceptual origins in IBM's WLM, I suspect). If
> > numa placement controllers (aka cpusets) are going to
> > exist as well, then CKRM needs to split into (1) a
> > virtual resource manager framework (vrm), and (2) the
> > fair share stuff. The vrm framework should be neutral
> > of either fair share or numa placement bias.
>
> As indicated in many notes so are the usage of cpusets.
Cpusets pretends to be nothing more than what it is now. I am not
recommending to Andrew that cpusets incorporate your fair shair
scheduling.
CKRM aspires to be both a general purpose resource management framework
and the embodiment of fair share scheduling.
Let me put the question again, and this time don't try to dodge it by
saying "but cpusets does it too ..."
> > If numa placement controllers (aka cpusets) are going to
> > exist as well, then CKRM needs to split into (1) a
> > virtual resource manager framework (vrm), and (2) the
> > fair share stuff. The vrm framework should be neutral
> > of either fair share or numa placement bias.
Hubertus' second response to the above:
>
> Very few people have the #cpus to even worry about this.
If for whatever reason, you don't think it is worth the effort to morph
the virtual resouce manager that is currently embedded within CKRM into
an independent, neutral framework, then don't expect the rest of us to
embrace it. Do you think Reiser would have gladly used vfs to plug in
his file system if it had been called "ext"? In my personal opinion, it
would be foolhardy for SGI, NEC, Bull, Platform (LSF) or Altair (PBS) to
rely on critical technology so clearly biased toward and dominated by a
natural competitor.
> But by self admission, you are driven by timing constraints as
> your bacon is sizzling.
You broke your promise - you said no more analogies ;)
Of _course_ I have scheduling pressures. You don't?
> > Keep talking.
>
> To whom ? :-)
A duh ... to us, here.
Just in case there was a communication failure here, let me be explicit.
When I said "Here's where I stand today - keep talking" it meant that my
current position was thus, but that I was still open to further
discussion and analysis.
When someone offers you an open door ("Keep talking"), don't slam it in
their face. It might not open again.
... keep talking ...
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373
On Fri, 2004-10-08 at 06:14, Paul Jackson wrote:
> So here I am with this new cpuset design (Simon Derr, primary
> architect, both Simon and I feel a strong sense of ownership)
> for numa placement, perhaps the 4th or 5th in SGI's history,
> and the 2nd in mine. I am finding that it deliciously and
> elegantly reflects the needs of its anticipated users (Sylvain
> might demur, noting a couple of things I removed).
>
> I am now being asked to morph it into a CKRM controller.
>
> Further I deduce from the efforts over the last few days to talk
> me down from meeting all the requirements satisfied by my current
> cpuset patch that something of cpusets will be lost in the translation.
>
> But I haven't figured out exactly what will be lost. And I lack the
> mastery of CKRM that would enable me to engage in a constructive dialog
> on the various tradeoffs that come into play here.
I hope that *nothing* will be lost. We (I) aim to still offer
users/admins named groupings of CPUs and memory. They may not be called
cpusets, in favor of names like classes or domains, but they will
*still* be named groupings of CPUs and memory. I further aim to not
change your API significantly. I really like the filesystem API you
came up with to interact with cpusets from userspace. I'd like to
incorporate this into CKRM's filesystem API (called rcfs) with minimal
changes. I really like the exclusive cpusets you describe. You tried
to implement them with some kernel exclusivity (your vitamin
precursors), while I'd like to see the kernel offer real exclusivity.
This shouldn't affect your customers because real exclusivity will
*still* be offered. I also aim to support what seems like a large
portion of your non-exclusive cpusets through nested, hierarchical
sched_domains. And I hope to do all of this with less overhead.
Now, I'm certainly not saying my patch provides all these things now. I
am saying that I believe the approach/model I'm using could do all these
things with some further work.
> And I am looking at trading what I thought had hope of being a
> Sept or Oct date for acceptance into Linus's kernel, into some
> unknown schedule that is definitely further out.
>
> I've got the bacon sizzling on the skillet, I can smell it, my
> mouth is watering, and just as I go to lift it off the burner,
> Andrew asks me to consider trading it for a pig in a poke.
> Thanks a bunch, Andrew - you da man ;).
Andrew is da man. Sometimes da man works with you, sometimes da man
works against you. As the French say, c'est la vie.
I think we can figure out how to merge cpusets into CKRM's framework,
with minimal changes to both the cpusets API & functionality without
slipping that *too* much. End of the year isn't unreasonable....
> Keep talking.
You asked for it!!! ;)
-Matt
On Fri, 2004-10-08 at 11:23, Paul Jackson wrote:
> CKRM aspires to be both a general purpose resource management framework
> and the embodiment of fair share scheduling.
I think your missing something here. CKRM, as I understand it, aspires
to be a general purpose resource management framework. To that point I
will accede. But the second part, about CKRM being the embodiment of
fair share scheduling, is secondary. That is simply one of it's
potential functions as a general purpose resource management framework.
It could also be the embodiment of unfair scheduling, if you choose to
implement such a resource controller. It wouldn't be very useful, but
it could be a fun exercise! ;)
> If for whatever reason, you don't think it is worth the effort to morph
> the virtual resouce manager that is currently embedded within CKRM into
> an independent, neutral framework, then don't expect the rest of us to
> embrace it. Do you think Reiser would have gladly used vfs to plug in
> his file system if it had been called "ext"? In my personal opinion, it
> would be foolhardy for SGI, NEC, Bull, Platform (LSF) or Altair (PBS) to
> rely on critical technology so clearly biased toward and dominated by a
> natural competitor.
I don't think that is terribly fair. I can honestly say that I'm not
opposing your implementation because of who you work for. I could care
less. I'm opposing it because I think I've got an alternative that can
offer the same functionality with less impact. I don't work on CKRM,
and when I wrote my code CKRM was not even on my radar. If
sched_domains will play nicer with CKRM than cpusets, thats just a
bonus. I certainly didn't design it that way!
> When someone offers you an open door ("Keep talking"), don't slam it in
> their face. It might not open again.
*More* analogies!?! ;)
> ... keep talking ...
I warned you!
-Matt
Matthrew, responding to Paul:
> > If for whatever reason, you don't think it is worth the effort to morph
> > the virtual resouce manager that is currently embedded within CKRM into
> > an independent, neutral framework, then don't expect the rest of us to
> > embrace it. Do you think Reiser would have gladly used vfs to plug in
> > his file system if it had been called "ext"? In my personal opinion, it
> > would be foolhardy for SGI, NEC, Bull, Platform (LSF) or Altair (PBS) to
> > rely on critical technology so clearly biased toward and dominated by a
> > natural competitor.
>
> I don't think that is terribly fair. I can honestly say that I'm not
> opposing your implementation because of who you work for.
Good point. I was painting with too wide a brush (hmmm ... someday I
should see if I can get through an entire post without an analogy ...)
When people show a good ability to work with others on lkml who have a
shared interest and sufficient competence in an area, then it doesn't
much matter what company they work for. I see such a discussion
happening on another portion of this thread, with yourself, Nick, Peter
and Erich, involving the domain scheduler. That works well.
So far I have been unable to achieve confidence in my ability to
interact well with the key CKRM folks. In various ways, I have found
their project, and their demeanor on this list, to be difficult for me
to approach.
Normally this wouldn't matter. However Andrew and others have proposed
that cpusets have a critical dependency on CKRM. Now it matters.
If I am to have a critical project dependency on an external group with
whom I lack confidence that we share sufficient common interest and a
healthy ability to communicate, then I prefer a more adversarial style
of relations, with explicit contracts, minimum clearly defined and
verifiable deliverables, and suitable fallback contingencies in place.
I keep a sharp eye out for potential conflicts of interest.
My suggestion to separate the virtual resource management framework
(which I named 'vrm') from CKRM's other elements, such as fair share
scheduling, was an attempt to establish such a minimum verifiable
deliverable. That suggestion was clearly dead on arrival.
My apologies for implicating everyone whose email ends in "ibm.com" in
my earlier comment. IBM is a big place, and all manner and variety
of people work there. It's a pleasure working with yourself, Matthew,
and many others from IBM.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373
Matthew writes:
> > CKRM aspires to be both a general purpose resource management framework
> > and the embodiment of fair share scheduling.
>
> I think your missing something here. CKRM, as I understand it, aspires
> to be a general purpose resource management framework. To that point I
> will accede. But the second part, about CKRM being the embodiment of
> fair share scheduling, is secondary.
Ok - you may well be right that CKRM does not aspire to be the embodiment
of fair share scheduling. But doesn't it embody a fair share sheduler
(and no other such policy) as a matter of current implementation fact?
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373
Matthew, responding to Paul:
> > But I haven't figured out exactly what will be lost. And I lack the
> > mastery of CKRM that would enable me to engage in a constructive dialog
> > on the various tradeoffs that come into play here.
>
> I hope that *nothing* will be lost. We (I) aim to still offer
> users/admins named groupings of CPUs and memory. They may not be called
> cpusets, in favor of names like classes or domains, but they will
> *still* be named groupings of CPUs and memory. I further aim to not
> change your API significantly.
This might work.
I've no earthly idea yet how it might work. But I take you at your word
that there's potential here worth pursuing.
I've gotten behind on my email the last three days - sleeping off a
cold. Do you have any suggestions for readings, or further explanations
you can provide, that might help me better understand how you intend to
accomplish this minor miracle? Perhaps there is something in one of the
messages that I haven't digested yet. I see your work-in-progress patch
of Wed, 06 Oct 2004 17:51:07 is one of the messages still in my input
queue.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373
Matthew wrote:
> while I'd like to see the kernel offer real exclusivity.
I agree - once we've identified some reason the kernel needs real
exclusivity, and I think we did, it is best to discard my vitamin
precursors and go with the real thing.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373
On Sat, 2004-10-09 at 13:08, Paul Jackson wrote:
> Matthew, responding to Paul:
> > > If for whatever reason, you don't think it is worth the effort to morph
> > > the virtual resouce manager that is currently embedded within CKRM into
> > > an independent, neutral framework, then don't expect the rest of us to
> > > embrace it. Do you think Reiser would have gladly used vfs to plug in
> > > his file system if it had been called "ext"? In my personal opinion, it
> > > would be foolhardy for SGI, NEC, Bull, Platform (LSF) or Altair (PBS) to
> > > rely on critical technology so clearly biased toward and dominated by a
> > > natural competitor.
> >
> > I don't think that is terribly fair. I can honestly say that I'm not
> > opposing your implementation because of who you work for.
>
> Good point. I was painting with too wide a brush (hmmm ... someday I
> should see if I can get through an entire post without an analogy ...)
Doubtful. I've read too many of your posts to think that it's very
likely! ;)
> My suggestion to separate the virtual resource management framework
> (which I named 'vrm') from CKRM's other elements, such as fair share
> scheduling, was an attempt to establish such a minimum verifiable
> deliverable. That suggestion was clearly dead on arrival.
My (completely uninformed) guess is that the CKRM folks thought it would
be extremely unlikely to be able to get the 'vrm' into the kernel
without something to use it. Linus, and the rest of the community, has
been understandably reluctant to pick up large chunks of code on the
assurance that "someone, someday will use these hooks". The fair share
scheduler is thus both a proof of concept that the 'vrm' works and a
user of the 'vrm'. The 'vrm' and the fair share scheduler, should be
logically separate pieces of code, though. I should *really* read
through the CKRM code before I continue any further as I am purely
speculating now...
> My apologies for implicating everyone whose email ends in "ibm.com" in
> my earlier comment. IBM is a big place, and all manner and variety
> of people work there. It's a pleasure working with yourself, Matthew,
> and many others from IBM.
Apology accepted, Paul. IBM is a large company, and this thread in
particular has had many @ibm.com posters. It can seem there is some
large IBM conspiracy to block your efforts, but I can assure you that
isn't the case. Unless the small, painless chips on the backs of our
necks are working far better than I think they do.... :)
-Matt
On Sat, 2004-10-09 at 17:05, Paul Jackson wrote:
> Matthew writes:
> > > CKRM aspires to be both a general purpose resource management framework
> > > and the embodiment of fair share scheduling.
> >
> > I think your missing something here. CKRM, as I understand it, aspires
> > to be a general purpose resource management framework. To that point I
> > will accede. But the second part, about CKRM being the embodiment of
> > fair share scheduling, is secondary.
>
> Ok - you may well be right that CKRM does not aspire to be the embodiment
> of fair share scheduling. But doesn't it embody a fair share sheduler
> (and no other such policy) as a matter of current implementation fact?
Yes. That is true, but it is by no means meant to be the end-all,
be-all of CKRM. It is my understanding that the fair share scheduler is
a proof-of-concept and an example of how to write a 'controller' for
others, but not the full extent of CKRM's power.
-Matt
> Yes. That is true, but it is by no means meant to be the end-all,
> be-all of CKRM.
All well and good. Except that it has taken me an inordinate amount of
effort to grok what CKRM is, and this mingling of a framework for
resource management with one particular instance of such, a fair share
scheduler, has contributed to my confusions. My teen age son can no
doubt offer additional explanations for my confusions.
And indeed, while a new kernel framework should come with at least one
good example of something worth so framing, still it's better to keep
the two clearly distinguished. If these two are well distinguished now,
then I am unaware of that.
Perhaps this effort to add a placement manager to the existing fair
share manager in CKRM's repetoire will result in a clearer separation of
the CKRM framework from that which it frames.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373
Matthrew wrote:
> My (completely uninformed) guess is that the CKRM folks thought it would
> be extremely unlikely to be able to get the 'vrm' into the kernel
> without something to use it.
I'd guess the same thing.
> The 'vrm' and the fair share scheduler, should be
> logically separate pieces of code, though.
I agree - should be.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373