On Fri, 29 May 2009, George Dunlap wrote:
> David Miller wrote:
> > I don't see Ingo's comments, whether I agree with them or not, as
> > an implication of Xen being niche. Rather I see his comments as
> > an opposition to how Xen is implemented.
> >
> It's in his definition of "improving Linux". Jeremy is saying that allowing
> Linux to run as dom0 *is* improving Linux. The lack of dom0 support is at
> this moment making life more difficult for a huge number of Linux users who
Exactly that's the point. Adding dom0 makes life easier for a group of
users who decided to use Xen some time ago, but what Ingo wants is
technical improvement of the kernel.
There are many features which have been wildly used in the distro
world where developers tried to push support into the kernel with the
same line of arguments.
The kernel policy always was and still is to accept only those
features which have a technical benefit to the code base.
I'm just picking a few examples:
Aside of the paravirt, which seems to expand through arch/x86 like a
hydra, the new patches sprinkle "if (xen_...)" all over the
place. These extra xen dependencies are no improvement, they are a
royal pain in the ... They are sticky once they got merged simply
because the hypervisor relies on them and we need to provide
compatibility for a long time.
Aside of that it grows interfaces like pat_disable() just because the
CPU model of Xen is obviously not able to kill the PAT flags in the
CPUid emulation. Why for heavens sake do we have a cpuid paravirt op
when we need to disable stuff seperately which can be disabled by
paravirt functionality already? I don't see this as an improvement
either, it's simple sloppy hackery.
The changelogs of the patches are partially confusing as hell:
commit 7d2b03ff4ae27b7c9e99a421a5b965f20e4bfaab
x86: fix up flush_tlb_all
- initialize the locks before the first use
- make sure preemption is disabled
[ Impact: Bug fixes: boot time warning, and crash ]
This patch is in the Xen queue and I assume it's XEN related as we
have not seen anywhere a boot time warning and crash with the current
code AFAICT, but the changelog reads like this is some generic BUG in
the SMP boot code. There is neither a hint to Xen nor to another patch
which caused that problem. While the patch itself is harmless I do not
see what is improved and why the change was necessary in the first
place.
That's what maintainers have to look at and not who is using the code
already and wants to see it merged.
> use Xen, including Mozilla, Debian, and Amazon. Adding dom0 support would
> make Linux even more useful to a wide variety of people not using Xen at the
> moment.
I really have a hard time to see why dom0 support makes Linux more
useful to people who do not use it. It does not improve the Linux
experience of Joe User at all.
In fact it could be harmful to the average user, if it's merged in a
crappy way that increases overhead, has a performance cost and draws
away development and maintenance resources from other areas of the
kernel.
Aside of that it can also hinder the development of a properly
designed hypervisor in Linux: 'why bother with that new stuff, it
might be cleaner and nicer, but we have this Xen dom0 stuff
already?'.
Thanks,
tglx
Thomas Gleixner wrote:
> Exactly that's the point. Adding dom0 makes life easier for a group of
> users who decided to use Xen some time ago, but what Ingo wants is
> technical improvement of the kernel.
>
> There are many features which have been wildly used in the distro
> world where developers tried to push support into the kernel with the
> same line of arguments.
>
> The kernel policy always was and still is to accept only those
> features which have a technical benefit to the code base.
>
I can appreciate the idea of resisting the pushing of random features.
Still, your definition of "improving Linux" is still lacking. Obviously
a new scheduler is taking something that's existing and improving it.
But adding a new filesystem, a new driver, or adding a new feature, such
as notifications, AIO, a new hardware architecture, or even KVM: How do
those classify as "technical improvement to the kernel" or "features
which have technical benefit to the code base" in a way that Xen does not?
If you mean "increases Linux's technical capability", and define Xen as
outside of Linux, then I think the definition is too small. After all,
allowing Linux to run on an ARM processor isn't increasing Linux'
technical capability, it's just allowing a new group of people (people
with ARM chips) to use Linux. It's the same with Xen.
No one disputes the idea that changes shouldn't be ugly; no one disputes
the idea that changes shouldn't introduce performance regressions. But
there are patchqueues that are ready, signed-off by other maintainers,
and which Ingo admits that he has no technical objections to, but
refuses to merge.
(His most recent "objection" is that he claims the currently existing
pv_ops infrastructure (which KVM and others benefit from as well as Xen)
introduces almost a 1% overhead on native in an mm-heavy
microbenchmark. So he refuses to merge feature Y (dom0 support) until
the Xen community helps technically unrelated existing feature X
(pv_ops) meets some criteria. So it has nothing to do with the quality
of the patches themselves.)
[Not qualified to speak to the specific technical objections.]
> I really have a hard time to see why dom0 support makes Linux more
> useful to people who do not use it. It does not improve the Linux
> experience of Joe User at all.
>
If Joe User uses Amazon, he benefits. If Joe User downloads an Ubuntu
or Debian distro, and the hosting providers were more secure and had to
do less work because dom0 was inlined, then he benefits because of the
lower cost / resources freed to do other things.
But what I was actually talking about is the number of people who don't
use it now but would use it if it were merged in. There hundreds of
thousands of instances running now, and more people are chosing to use
it at the moment, even though those who use it have the devil's choice
between doing patching or using a 3-year old kernel. How many more
would use it if it were in mainline?
> In fact it could be harmful to the average user, if it's merged in a
> crappy way that increases overhead, has a performance cost and draws
> away development and maintenance resources from other areas of the
> kernel.
>
No one is asking for something to be merged in a crappy way, or with
unacceptable performance cost. There are a number of patchqueues that
Ingo has no technical objections to, but which he still refuses to merge.
"Drawing away development and maintenance resources" is a cost/benefits
question, and Jeremy's main point was that there is a *high* benefit for
dom0 being merged into mainline. The same could be said of almost
anything: are you suggesting not accepting any more KVM code because it
might "draw away development and maintenance resources from other areas
of the kernel"?
> Aside of that it can also hinder the development of a properly
> designed hypervisor in Linux: 'why bother with that new stuff, it
> might be cleaner and nicer, but we have this Xen dom0 stuff
> already?'.
>
This argument doesn't make any sense. Would you advocate only having
one filesystem for fear that people would somehow be discouraged from
working on a new filesystem?
Even if that were a valid argument, it wouldn't apply in this situation.
KVM has plenty of mind-share, and the support of RedHat. Also, I'd
wager that it's a lot easier for a Linux kernel developer to get
involved in KVM than in Xen, because they're already familiar with
Linux. I don't think anyone working on KVM will be tempted to give up
just because Xen is also available, unless it becomes clear that
linux-as-hypervisor isn't the best technical solution; in which case,
moving to Xen would be the right thing to do anyway. Merging dom0 Xen
will in no way interfere with the development of KVM or other
linux-as-hypervisor projects.
The main point of Jeremy's e-mail was NOT to say, "Lots of people use
this so you should merge it." He's was responding to Xen being treated
like it had no benefit. It does have a benefit; it is a feature.
-George
George Dunlap wrote:
> Thomas Gleixner wrote:
> No one disputes the idea that changes shouldn't be ugly; no one disputes
> the idea that changes shouldn't introduce performance regressions. But
> there are patchqueues that are ready, signed-off by other maintainers,
> and which Ingo admits that he has no technical objections to, but
> refuses to merge.
I can't comment on this part, but if so that seems unfortunate.
> The main point of Jeremy's e-mail was NOT to say, "Lots of people use
> this so you should merge it." He's was responding to Xen being treated
> like it had no benefit. It does have a benefit; it is a feature.
I don't know about others, but I certainly interpreted a number of posts
saying exactly that--that it's useful so it should be included.
I don't think anyone is arguing that Xen is not useful or that it should
not ever be included, rather the question is whether the current set of
patches is suitable for addition or whether they are too messy and
should be cleaned up first.
Chris
On Tue, 2 Jun 2009, George Dunlap wrote:
>
> idea that changes shouldn't introduce performance regressions. But there are
> patchqueues that are ready, signed-off by other maintainers, and which Ingo
> admits that he has no technical objections to, but refuses to merge.
I've seen technical objects in this thread. The whole thing _started_ with
one, and Thomas brought up others.
As a top-level maintainer, I can also very much sympathise with the "don't
merge new stuff if there are known problems and no known solutions to
those issues". Is Ingo supposed to just continue to merge crap, when it's
admitted that it has problems and pollutes code that he has to maintain?
The fact is (and this is a _fact_): Xen is a total mess from a development
standpoint. I talked about this in private with Jeremy. Xen pollutes the
architecture code in ways that NO OTHER subsystem does. And I have never
EVER seen the Xen developers really acknowledge that and try to fix it.
Thomas pointed to patches that add _explicitly_ Xen-related special cases
that aren't even trying to make sense. See the local apic thing.
So quite frankly, I wish some of the Xen people looked themselves in the
mirror, and then asked themselves "would _I_ merge something ugly like
that, if it was filling my subsystem with totally unrelated hacks for some
other crap"?
Seriously.
If it was just the local APIC, fine. But it may be just the local APIC
code this time around, next time it will be something else. It's been TLB,
it's been entry_*.S, it's been all over. Some of them are performance
issues.
I dunno. I just do know that I pointed out the statistics for how
mindlessly incestuous the Xen patches have historically been to Jeremy. He
admitted it. I've not seen _anybody_ say that things will improve.
Xen has been painful. If you give maintainers pain, don't expect them to
love you or respect you.
So I would really suggest that Xen people should look at _why_ they are
giving maintainers so much pain.
Linus
On Tue, 2 Jun 2009, Linus Torvalds wrote:
>
> I dunno. I just do know that I pointed out the statistics for how
> mindlessly incestuous the Xen patches have historically been to Jeremy. He
> admitted it. I've not seen _anybody_ say that things will improve.
In case people want to look at this on their own, get a git tree, and run
the examples I asked Jeremy to run:
git log --pretty=oneline --full-diff --stat arch/x86/kvm/ |
grep -v '/kvm' |
less -S
and then go ahead and do the same except with "xen" instead of "kvm".
Now, once you've done that, ask yourself which one is going to be merged
easily and without any pushback.
Btw, this is NOT meant to be a "xen vs kvm" thing. Before you react to the
"kvm" part, replace "arch/x86/kvm" above with "drivers/scsi" or something.
The point? Xen really is horribly badly separated out. It gets way more
incestuous with other systems than it should. It's entirely possible that
this is very fundamental to both paravirtualization and to hypervisor
behavior, but it doesn't matter - it just measn that I can well see that
Xen is a f*cking pain to merge.
So please, Xen people, look at your track record, and look at the issues
from the standpoint of somebody merging your code, rather than just from
the standpoint of somebody who whines "I want my code to be merged".
IOW, if you have trouble getting your code merged, ask yourself what _you_
are doing wrong.
Linus
Linus Torvalds wrote:
> The point? Xen really is horribly badly separated out. It gets way more
> incestuous with other systems than it should. It's entirely possible that
> this is very fundamental to both paravirtualization and to hypervisor
> behavior, but it doesn't matter - it just measn that I can well see that
> Xen is a f*cking pain to merge.
>
> So please, Xen people, look at your track record, and look at the issues
> from the standpoint of somebody merging your code, rather than just from
> the standpoint of somebody who whines "I want my code to be merged".
>
> IOW, if you have trouble getting your code merged, ask yourself what _you_
> are doing wrong.
>
There is in fact a way to get dom0 support with nearly no changes to
Linux, but it involves massive changes to Xen itself and requires
hardware support: run dom0 as a fully virtualized guest, and assign it
all the resources dom0 can access. It's probably a massive effort though.
I've considered it for kvm when faced with the "I want a thin
hypervisor" question: compile the hypervisor kernel with PCI support but
nothing else (no CONFIG_BLOCK or CONFIG_NET, no device drivers), load
userspace from initramfs, and assign host devices to one or more
privileged guests. You could probably run the host with a heavily
stripped configuration, and enjoy the slimness while every interrupt
invokes the scheduler, a context switch, and maybe an IPI for good measure.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
On Tue, 2 Jun 2009, George Dunlap wrote:
> Thomas Gleixner wrote:
> > Exactly that's the point. Adding dom0 makes life easier for a group of
> > users who decided to use Xen some time ago, but what Ingo wants is
> > technical improvement of the kernel.
> >
> > There are many features which have been wildly used in the distro
> > world where developers tried to push support into the kernel with the
> > same line of arguments.
> >
> > The kernel policy always was and still is to accept only those
> > features which have a technical benefit to the code base.
> >
> I can appreciate the idea of resisting the pushing of random features. Still,
> your definition of "improving Linux" is still lacking. Obviously a new
> scheduler is taking something that's existing and improving it. But adding a
> new filesystem, a new driver, or adding a new feature, such as notifications,
> AIO, a new hardware architecture, or even KVM: How do those classify as
> "technical improvement to the kernel" or "features which have technical
> benefit to the code base" in a way that Xen does not?
There is a huge difference between new filesystems, drivers,
architectures and Xen.
A new filesystem is not intrusive to the filesystem layers, it's not
adding its special cases all over the place. There is no single "if
(fs_whatever)" hackery in the code base. Neither does a driver nor a
new architecture.
If the new functionality needs some extension to the generic code base
then this is carefully added with the maintainers of that code and the
extension is usually useful to other (filesystems, drivers,
architectures) as well. If it's necessary to add some special case for
one architecture then this is done by proper abstraction to keep the
burden and the maintainence cost down.
There is no #ifdef ARCH_ARM in mm/ fs/ kernel/ block/ .....
Talking about KVM, there is not a single "if (kvm)" line in the
arch/x86 code base. There is _ONE_ lonely #ifdef CONFIG_KVM_CLOCK
(which could be eliminated) in the whole x86 codebase, but at least 10
CONFIG_XEN* ones all over the place. The KVM developers went great
length to avoid adding restrictions to the existing code base.
I'm not saying that the Xen folks did not listen to us, they improved
lots of their code base and Jeremy was particularly helpful to unify
the 32/64bit code.
But right now I see a big code dump with subtle details where some of
them are just not acceptable to me.
> If you mean "increases Linux's technical capability", and define Xen as
> outside of Linux, then I think the definition is too small. After all,
> allowing Linux to run on an ARM processor isn't increasing Linux' technical
> capability, it's just allowing a new group of people (people with ARM chips)
> to use Linux. It's the same with Xen.
No, it's not. ARM does not interfere with anything and it keeps its
architecture specific limitations confined in arch/arm.
Xen injects its design limitation workarounds into the arch/x86
codebase and burdens developers and maintainers with it.
> No one disputes the idea that changes shouldn't be ugly; no one disputes the
> idea that changes shouldn't introduce performance regressions. But there are
> patchqueues that are ready, signed-off by other maintainers, and which Ingo
> admits that he has no technical objections to, but refuses to merge.
> (His most recent "objection" is that he claims the currently existing pv_ops
> infrastructure (which KVM and others benefit from as well as Xen) introduces
> almost a 1% overhead on native in an mm-heavy microbenchmark. So he refuses
> to merge feature Y (dom0 support) until the Xen community helps technically
> unrelated existing feature X (pv_ops) meets some criteria. So it has nothing
> to do with the quality of the patches themselves.)
Oh well. It has a lot to do with the quality of the patches. The
design is part of the quality and right now the short comings of the
design are papered over by adding Xen restrictions into the x86 code
base.
> [Not qualified to speak to the specific technical objections.]
> > I really have a hard time to see why dom0 support makes Linux more
> > useful to people who do not use it. It does not improve the Linux
> > experience of Joe User at all.
> >
> If Joe User uses Amazon, he benefits. If Joe User downloads an Ubuntu or
> Debian distro, and the hosting providers were more secure and had to do less
> work because dom0 was inlined, then he benefits because of the lower cost /
> resources freed to do other things.
Right, then they can concentrate on adding another bunch out of tree
patches to their kernels. Next time you stand up and tell me the same
argument for apparmour, ndiswrapper or whatever people like to use.
> But what I was actually talking about is the number of people who don't use it
> now but would use it if it were merged in. There hundreds of thousands of
> instances running now, and more people are chosing to use it at the moment,
> even though those who use it have the devil's choice between doing patching or
> using a 3-year old kernel. How many more would use it if it were in mainline?
How many more would use ndiswrapper if it were in mainline ?
> > In fact it could be harmful to the average user, if it's merged in a
> > crappy way that increases overhead, has a performance cost and draws
> > away development and maintenance resources from other areas of the
> > kernel.
> >
> No one is asking for something to be merged in a crappy way, or with
> unacceptable performance cost. There are a number of patchqueues that Ingo
> has no technical objections to, but which he still refuses to merge.
Right, because the lineup of patches is not completely untangled and
we still have objections against the overall outcome and design of the
Dom0 integration into the kernel proper.
It's not our fault that the Dom0 design decisions were made in total
disconnect to the kernel community and now a "swallow them as is"
policy is imposed on us with the argument that the newer kernels need
to run on ancient hypervisors as well.
You whine about users having to use 3 year old kernels, but 3 years
old hypervisors are fine, right ?
I'm not against merging dom0 in general, I'm opposing that we need to
buy inferior technical solutions which we can not change for a long
time. Once we merged them the "you can not break existent hypervisors"
argument will be used to prevent any design change and cleanup.
> The main point of Jeremy's e-mail was NOT to say, "Lots of people use this so
> you should merge it." He's was responding to Xen being treated like it had no
> benefit. It does have a benefit; it is a feature.
Right, a feature which comes with cost. The cost is the de facto
injection of an dom0 ABI into the arch/x86 code base. A new driver is
a feature as well, but it just adds the feature w/o impact to the
general system.
Thanks,
tglx
Thomas Gleixner wrote:
> On Fri, 29 May 2009, George Dunlap wrote:
>> David Miller wrote:
>>> I don't see Ingo's comments, whether I agree with them or not, as
>>> an implication of Xen being niche. Rather I see his comments as
>>> an opposition to how Xen is implemented.
>>>
>> It's in his definition of "improving Linux". Jeremy is saying that allowing
>> Linux to run as dom0 *is* improving Linux. The lack of dom0 support is at
>> this moment making life more difficult for a huge number of Linux users who
>
> Exactly that's the point. Adding dom0 makes life easier for a group of
> users who decided to use Xen some time ago, but what Ingo wants is
> technical improvement of the kernel.
>
> There are many features which have been wildly used in the distro
> world where developers tried to push support into the kernel with the
> same line of arguments.
>
> The kernel policy always was and still is to accept only those
> features which have a technical benefit to the code base.
>
> I'm just picking a few examples:
>
> Aside of the paravirt, which seems to expand through arch/x86 like a
> hydra, the new patches sprinkle "if (xen_...)" all over the
> place. These extra xen dependencies are no improvement, they are a
> royal pain in the ... They are sticky once they got merged simply
> because the hypervisor relies on them and we need to provide
> compatibility for a long time.
>
Wait, let's not classify something as "no improvement" when you mean "I don't
need it." The fact that processors without hardware VM can run virtual machines
is a non-trivial benefit for many users, and in future embedded applications,
where both hvm and 64 bit capability may not justify their power requirements.
And the improved PV performance over full virtualization is an improvement, even
though it certainly isn't night and day.
Having replace some systems with new hardware just so I could use KVM does not
make me forget that I used xen for some time, and that PV is still a savings,
even with the latest hardware.
Let's stick to technical issues, and not deny that there are a number of users
who really will have expanded capability. The technical points are valid, but as
a former and probable future xen (CentOS) user, so are the benefits.
--
Bill Davidsen <[email protected]>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
On Wed, 3 Jun 2009, Bill Davidsen wrote:
> Thomas Gleixner wrote:
> > Aside of the paravirt, which seems to expand through arch/x86 like a
> > hydra, the new patches sprinkle "if (xen_...)" all over the
> > place. These extra xen dependencies are no improvement, they are a
> > royal pain in the ... They are sticky once they got merged simply
> > because the hypervisor relies on them and we need to provide
> > compatibility for a long time.
> >
> Wait, let's not classify something as "no improvement" when you mean "I don't
> need it."
It's not about "I don't need it.". It's about having Xen dependencies
in the code all over the place which make mainatainence harder. I have
to balance the users benefit (xen dom0 support) vs. the impact on
maintainability and the restrictions which are going to be set almost
in stone by merging it.
> Let's stick to technical issues, and not deny that there are a number of users
> who really will have expanded capability. The technical points are valid, but
> as a former and probable future xen (CentOS) user, so are the benefits.
Refusing random "if (xen...)" dependencies is a purely technical
decision. I have said more than once that I'm not against merging dom0
in general, I'm just frightened by the technical impact of a defacto
ABI which we swallow with it.
We have enough problems with real silicon and BIOS/ACPI already, why
should we add artifical and _avoidable_ virtual silicon horror ?
Thanks,
tglx
Thomas Gleixner wrote:
> On Wed, 3 Jun 2009, Bill Davidsen wrote:
>
>> Thomas Gleixner wrote:
>>
>>> Aside of the paravirt, which seems to expand through arch/x86 like a
>>> hydra, the new patches sprinkle "if (xen_...)" all over the
>>> place. These extra xen dependencies are no improvement, they are a
>>> royal pain in the ... They are sticky once they got merged simply
>>> because the hypervisor relies on them and we need to provide
>>> compatibility for a long time.
>>>
>>>
>> Wait, let's not classify something as "no improvement" when you mean "I don't
>> need it."
>>
>
> It's not about "I don't need it.". It's about having Xen dependencies
> in the code all over the place which make mainatainence harder. I have
> to balance the users benefit (xen dom0 support) vs. the impact on
> maintainability and the restrictions which are going to be set almost
> in stone by merging it.
>
>
>> Let's stick to technical issues, and not deny that there are a number of users
>> who really will have expanded capability. The technical points are valid, but
>> as a former and probable future xen (CentOS) user, so are the benefits.
>>
>
> Refusing random "if (xen...)" dependencies is a purely technical
> decision. I have said more than once that I'm not against merging dom0
> in general, I'm just frightened by the technical impact of a defacto
> ABI which we swallow with it.
>
>
I was referring to your "no benefit" comment, I don't dispute the
technical issues. I think the idea of moving the hypervisor into the
kernel and letting xen folks do the external parts as they please.
> We have enough problems with real silicon and BIOS/ACPI already, why
> should we add artifical and _avoidable_ virtual silicon horror ?
>
I guess my point wasn't clear, sorry, it's just that I felt as though
the features lacking KVM (old/small/BIOS-limited CPUs) might be hidden
in the smoke due to the technical issues.
--
Bill Davidsen <[email protected]>
Even purely technical things can appear to be magic, if the documentation is
obscure enough. For example, PulseAudio is configured by dancing naked around a
fire at midnight, shaking a rattle with one hand and a LISP manual with the
other, while reciting the GNU manifesto in hexadecimal. The documentation fails
to note that you must circle the fire counter-clockwise in the southern
hemisphere.
Bill Davidsen wrote:
> I was referring to your "no benefit" comment, I don't dispute the
> technical issues. I think the idea of moving the hypervisor into the
> kernel and letting xen folks do the external parts as they please.
Where does that come from? AFAICT Thomas never made a "no benefit" comment
other than limited to the context of the technical implementation.
I've always understood his meaning in this thread to be: "the proposed
patch set does not improve the technical standard of the linux kernel,
but would instead lower it considerably".
Thomas has been extremely correct in this thread and IMO does not deserve
this attack.
Let's look at his exact comments (emphasis mine).
! The kernel policy always was and still is to accept only those
! features which have a technical benefit **to the code base**.
and
! Aside of the paravirt, which seems to expand through arch/x86 like a
! hydra, the new patches sprinkle "if (xen_...)" all over the
! place. These extra xen dependencies are no improvement, they are a
! royal pain in the ...
Also clearly limited to technical implementation.
! I really have a hard time to see why dom0 support makes Linux more
! useful **to people who do not use it**. It does not improve the Linux
! experience **of Joe User** at all.
Or has Thomas made some "no benefit" comment I've missed?
Cheers,
FJP
Frans Pop wrote:
> ! The kernel policy always was and still is to accept only those
> ! features which have a technical benefit **to the code base**.
>
Yes, I think I understood him better after I responded to his e-mail
(unfortunately). When people say things like "dom0 adds all these hooks
but doesn't add anything to Linux", they mean something like this
(please correct me anyone, if I'm wrong).
Kernel developers want Linux, as a project, to have cool things in it.
They want it to be cool. Adding new features, new capabilities, new
technical code, makes it cooler. Sometimes adding new features to make
it cooler has some cost in terms of adding things to other parts of the
code, possibly making it a little less clean or a little more
convoluted. But if the coolness is cool enough, it's worth the cost.
The feeling is that adding a bunch of these dom0 hooks (especially of
the type, "if(xen) { foo; }"), are a cost to Linux. They make the code
ugly. They do allow a new kind of coolness, a (linux-dom0 + Xen)
coolness. But none of the coolness actually happens in Linux; it all
happens in Xen. So coolness may happen, and world happiness might
increase marginally, but Linux itself doesn't seem any cooler, it just
has the cost of all these ugly hooks. Thus the "Linux is Xen's sex
slave" analogy. :-)
If (hypothetically) we merged Xen into Linux, then (people are
suggesting) the coolness of Xen would actually contribute to the
coolness of Linux ("add technical benefit to the code base"). People
would feel like working on the interface between linux-xen and the rest
of linux would be making their own piece of software, Linux, work
better, rather than feeling like they have to work with some foreign
project that doesn't make their code any cooler.
Is that a pretty accurate representation of the "adding features which
have a technical benefit to the code base" argument?
-George
Linus Torvalds wrote:
> Seriously.
>
> If it was just the local APIC, fine. But it may be just the local APIC
> code this time around, next time it will be something else. It's been TLB,
> it's been entry_*.S, it's been all over. Some of them are performance
> issues.
>
> I dunno. I just do know that I pointed out the statistics for how
> mindlessly incestuous the Xen patches have historically been to Jeremy. He
> admitted it. I've not seen _anybody_ say that things will improve.
>
> Xen has been painful. If you give maintainers pain, don't expect them to
> love you or respect you.
>
> So I would really suggest that Xen people should look at _why_ they are
> giving maintainers so much pain.
>
> Linus
Seriously, reading this is discouraging. I had to stop myself
criticizing too much this opinion here, but it's kind of hard to read
"mindless", "painful" and such considering the consequences of the
current state.
As time passes, it's becoming more and more unmaintainable to manage the
dom0 patch on one side, and the mainline kernel on the other, even for a
user/admin point of view. THIS is years of mindless and painful
administration/patching tasks. We've all bee waiting too long already.
We need the Xen dom0 "feature" NOW! Not tomorrow, not in one week, not
in 10 years...
As a developer myself (not on the kernel though), I can perfectly
understand the standpoint about ugliness of the code. However, refusing
to merge gives bad headaches to hundreds of people trying to deal and
maintain productions with the issues it creates.
I stand on Steven Rostedt's side (and many others too). Merging WILL
make it possible to have Xen going the way you wish. Otherwise, it's
again a cathedral type of development. Keir Fraser and others seems to
be willing to do the changes in the API if needed. It's just not right
to tell they don't want to. And if there is such need for ABI/API
compatibility, why not just add a config option "compatibility to old
style Xen (dirty hugly slow feature)" if there are some issues?
Now, about merging the Xen hypervisor, that's another discussion that
can happen later on, IMHO. What's URGENT (I insist here) is dom0 support
(including with 64 bits).
Thomas
On Thu, Jun 04, 2009 at 02:21:08PM +0100, George Dunlap wrote:
> If (hypothetically) we merged Xen into Linux, then (people are
> suggesting) the coolness of Xen would actually contribute to the
> coolness of Linux ("add technical benefit to the code base"). People
> would feel like working on the interface between linux-xen and the rest
> of linux would be making their own piece of software, Linux, work
> better, rather than feeling like they have to work with some foreign
> project that doesn't make their code any cooler.
>
> Is that a pretty accurate representation of the "adding features which
> have a technical benefit to the code base" argument?
The other argument is that by merging Xen into Linux, it becomes
easier for kernel developers to understand *why* "if (xen) ..." shows
up in random places in core kernel code, and it becomes easier to
clean that up.
If Xen isn't merged, it becomes much harder to believe that those
cleanups will occur, since the Xen developers might stonewall such
cleanups for reasons that Linux developers might not consider valid.
So the threshold for accepting patches might be much higher, since the
subsystem maintainers involved might decide to NAK patches as
uglifying the Linux kernel codebase with no real benefit to the Linux
codebase --- and not much hope that said ugly hacks will get cleaned
up later. Historically, once code with warts gets merged, we lose all
leverage towards fixing those warts afterwards; this is true in
general, and not a statement of a lack of trust of Xen developers
specifically.
This doesn't make merging Xen *impossible*, but probably makes it
harder, since each of those objections will have to be cleared,
possibly by refactoring the code so that it adds benefits not just for
Xen, but some other in-kernel user of that abstraction (i.e., like
KVM, lguest, etc.) or by cleaning up the code in general, in order to
clear NAK's by the relevant developers.
If Xen is merged, then ultimately Linus gets to make the call about
whether something gets fixed, even at the cost of making a change to
the hypervisor/dom0 interface. So this would likely decrease the
threshold of what has to be fixed before people are willing to ACK a
Xen merge, since there's better confidence that these warts will be
cleaned up. An example of that might be XFS, which had all sorts of
Irix warts which has been gradually cleaned up over the years. Of
course, there might still be some hideous abstraction violations that
would have to be cleaned up first; but that's up to the relevant
subsystem maintainers.
- Ted
George Dunlap wrote:
> Frans Pop wrote:
>
>>! The kernel policy always was and still is to accept only those
>>! features which have a technical benefit **to the code base**.
> If (hypothetically) we merged Xen into Linux, then (people are
> suggesting) the coolness of Xen would actually contribute to the
> coolness of Linux ("add technical benefit to the code base"). People
> would feel like working on the interface between linux-xen and the rest
> of linux would be making their own piece of software, Linux, work
> better, rather than feeling like they have to work with some foreign
> project that doesn't make their code any cooler.
I suspect that there is an element of this.
There is also the factor that if Xen was merged into linux, we would
then be able to work towards a sane(r) virtualization layer that would
be useful for KVM, Xen, and possibly others. This provides a technical
benefit to the code base by introducing a more logical organization
rather than having ad-hoc changes sprinkled all over.
Chris
Frans Pop wrote:
> Bill Davidsen wrote:
>
>> I was referring to your "no benefit" comment, I don't dispute the
>> technical issues. I think the idea of moving the hypervisor into the
>> kernel and letting xen folks do the external parts as they please.
>>
>
> Where does that come from? AFAICT Thomas never made a "no benefit" comment
> other than limited to the context of the technical implementation.
>
>
Where it comes from is his very recent statement, which contains those
very words. You may interpret what he said in any way you choose, but
denying that he said it shows that you didn't follow the link back. I
never denied the ugliness of the code, nor does the author, but it adds
a great deal of value for many people, and that's the point I was making.
--
Bill Davidsen <[email protected]>
Even purely technical things can appear to be magic, if the documentation is
obscure enough. For example, PulseAudio is configured by dancing naked around a
fire at midnight, shaking a rattle with one hand and a LISP manual with the
other, while reciting the GNU manifesto in hexadecimal. The documentation fails
to note that you must circle the fire counter-clockwise in the southern
hemisphere.
Bill Davidsen wrote:
> Where it comes from is his very recent statement, which contains those
> very words. You may interpret what he said in any way you choose, but
> denying that he said it shows that you didn't follow the link back. I
> never denied the ugliness of the code, nor does the author, but it adds
> a great deal of value for many people, and that's the point I was making.
Lots of code could be said to add a great deal of value for many people
(semi-closed video card drivers, ndiswrapper, etc.), but it's never
going to be accepted into the kernel.
The maintainers get to decide whether the perceived benefit outweighs
the perceived cost. So far, they've decided that Xen isn't worth it.
The most likely way to get Xen merged is to lower the cost (reduce the
churn and ugliness), increase the benefit (improve the virtualization
layer, thus cleaning up other code as well), or both.
Chris
* Avi Kivity <[email protected]> wrote:
> Linus Torvalds wrote:
>> The point? Xen really is horribly badly separated out. It gets way more
>> incestuous with other systems than it should. It's entirely possible
>> that this is very fundamental to both paravirtualization and to
>> hypervisor behavior, but it doesn't matter - it just measn that I can
>> well see that Xen is a f*cking pain to merge.
>>
>> So please, Xen people, look at your track record, and look at the
>> issues from the standpoint of somebody merging your code, rather
>> than just from the standpoint of somebody who whines "I want my
>> code to be merged".
>>
>> IOW, if you have trouble getting your code merged, ask yourself
>> what _you_ are doing wrong.
>
> There is in fact a way to get dom0 support with nearly no changes
> to Linux, but it involves massive changes to Xen itself and
> requires hardware support: run dom0 as a fully virtualized guest,
> and assign it all the resources dom0 can access. It's probably a
> massive effort though.
>
> I've considered it for kvm when faced with the "I want a thin
> hypervisor" question: compile the hypervisor kernel with PCI
> support but nothing else (no CONFIG_BLOCK or CONFIG_NET, no device
> drivers), load userspace from initramfs, and assign host devices
> to one or more privileged guests. You could probably run the host
> with a heavily stripped configuration, and enjoy the slimness
> while every interrupt invokes the scheduler, a context switch, and
> maybe an IPI for good measure.
This would be an acceptable model i suspect, if someone wants a
'slim hypervisor'.
We can context switch way faster than we handle IRQs. Plus in a
slimmed-down config we could intentionally slim down aspects of the
scheduler as well, if it ever became a measurable performance issue.
The hypervisor would run a minimal user-space and most of the
context-switching overhead relates to having a full-fledged
user-space with rich requirements. So there's no real conceptual
friction between a 'lean and mean' hypervisor and a full-featured
native kernel.
This would certainly be an utterly clean design, and it would be
interesting to see a Linux/Xen + Linux/Dom0 combo engineered in such
a way - if people really find this layered kernel approach
interesting. So the door is not closed to dom0 at all - but it has
to be designed cleanly without messing up the native kernel.
Ingo
Ingo Molnar wrote:
>> There is in fact a way to get dom0 support with nearly no changes
>> to Linux, but it involves massive changes to Xen itself and
>> requires hardware support: run dom0 as a fully virtualized guest,
>> and assign it all the resources dom0 can access. It's probably a
>> massive effort though.
>>
>> I've considered it for kvm when faced with the "I want a thin
>> hypervisor" question: compile the hypervisor kernel with PCI
>> support but nothing else (no CONFIG_BLOCK or CONFIG_NET, no device
>> drivers), load userspace from initramfs, and assign host devices
>> to one or more privileged guests. You could probably run the host
>> with a heavily stripped configuration, and enjoy the slimness
>> while every interrupt invokes the scheduler, a context switch, and
>> maybe an IPI for good measure.
>>
>
> This would be an acceptable model i suspect, if someone wants a
> 'slim hypervisor'.
>
> We can context switch way faster than we handle IRQs. Plus in a
> slimmed-down config we could intentionally slim down aspects of the
> scheduler as well, if it ever became a measurable performance issue.
> The hypervisor would run a minimal user-space and most of the
> context-switching overhead relates to having a full-fledged
> user-space with rich requirements. So there's no real conceptual
> friction between a 'lean and mean' hypervisor and a full-featured
> native kernel.
>
The context switch would be taken by the Xen scheduler, not the Linux
scheduler. It's how interrupts work under Xen: an interrupt is taken,
Xen schedules the domain that owns the interrupts (dom0 usually), which
then handles the interrupt. The Linux scheduler would only be involved
if you thread your interrupt handlers.
This context switch is necessary regardless of how dom0 is integrated
into Linux; it's simply a side effect of implementing device drivers
outside the kernel (in this context, the kernel is Xen, and dom0 is just
another userspace, albeit with elevated privileges. The Linux
equivalent to dom0 is a process that uses uio.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
* Avi Kivity <[email protected]> wrote:
> Ingo Molnar wrote:
>>> There is in fact a way to get dom0 support with nearly no changes to
>>> Linux, but it involves massive changes to Xen itself and requires
>>> hardware support: run dom0 as a fully virtualized guest, and assign
>>> it all the resources dom0 can access. It's probably a massive effort
>>> though.
>>>
>>> I've considered it for kvm when faced with the "I want a thin
>>> hypervisor" question: compile the hypervisor kernel with PCI support
>>> but nothing else (no CONFIG_BLOCK or CONFIG_NET, no device drivers),
>>> load userspace from initramfs, and assign host devices to one or more
>>> privileged guests. You could probably run the host with a heavily
>>> stripped configuration, and enjoy the slimness while every interrupt
>>> invokes the scheduler, a context switch, and maybe an IPI for good
>>> measure.
>>>
>>
>> This would be an acceptable model i suspect, if someone wants a 'slim
>> hypervisor'.
>>
>> We can context switch way faster than we handle IRQs. Plus in a
>> slimmed-down config we could intentionally slim down aspects of the
>> scheduler as well, if it ever became a measurable performance issue.
>> The hypervisor would run a minimal user-space and most of the
>> context-switching overhead relates to having a full-fledged user-space
>> with rich requirements. So there's no real conceptual friction between
>> a 'lean and mean' hypervisor and a full-featured native kernel.
>>
>
> The context switch would be taken by the Xen scheduler, not the Linux
> scheduler. [...]
The 'slim hypervisor' model i was suggesting was a slimmed down
_Linux_ kernel.
Ingo
Ingo Molnar wrote:
> * Avi Kivity <[email protected]> wrote:
>
>
>> Ingo Molnar wrote:
>>
>>>> There is in fact a way to get dom0 support with nearly no changes to
>>>> Linux, but it involves massive changes to Xen itself and requires
>>>> hardware support: run dom0 as a fully virtualized guest, and assign
>>>> it all the resources dom0 can access. It's probably a massive effort
>>>> though.
>>>>
>>>> I've considered it for kvm when faced with the "I want a thin
>>>> hypervisor" question: compile the hypervisor kernel with PCI support
>>>> but nothing else (no CONFIG_BLOCK or CONFIG_NET, no device drivers),
>>>> load userspace from initramfs, and assign host devices to one or more
>>>> privileged guests. You could probably run the host with a heavily
>>>> stripped configuration, and enjoy the slimness while every interrupt
>>>> invokes the scheduler, a context switch, and maybe an IPI for good
>>>> measure.
>>>>
>>>>
>>> This would be an acceptable model i suspect, if someone wants a 'slim
>>> hypervisor'.
>>>
>>> We can context switch way faster than we handle IRQs. Plus in a
>>> slimmed-down config we could intentionally slim down aspects of the
>>> scheduler as well, if it ever became a measurable performance issue.
>>> The hypervisor would run a minimal user-space and most of the
>>> context-switching overhead relates to having a full-fledged user-space
>>> with rich requirements. So there's no real conceptual friction between
>>> a 'lean and mean' hypervisor and a full-featured native kernel.
>>>
>>>
>> The context switch would be taken by the Xen scheduler, not the Linux
>> scheduler. [...]
>>
>
> The 'slim hypervisor' model i was suggesting was a slimmed down
> _Linux_ kernel.
>
Yeah, I lost the context. I should reduce my own context switching.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
On Sun, 2009-06-07 at 15:46 +0300, Avi Kivity wrote:
> Ingo Molnar wrote:
> >
> > The 'slim hypervisor' model i was suggesting was a slimmed down
> > _Linux_ kernel.
> >
>
> Yeah, I lost the context. I should reduce my own context switching.
>
It would be better if we monitor the switching, entry/exit and other
useful parameters in count and frequency using debugfs to increase the
performance.
Thanks,
--
JSR