In OLS 2005, we described the work that we have been doing in VMware
with respect a common interface for paravirtualization of Linux. We
shared the general vision in Rik's virtualization BoF.
This note is an update on our further work on the Virtual Machine
Interface, VMI. The patches provided have been tested on 2.6.16-rc6.
We are currently recollecting performance information for the new -rc6
kernel, but expect our numbers to match previous results, which showed
no impact whatsoever on macro benchmarks, and nearly neglible impact
on microbenchmarks.
Unlike the full-virtualization techniques used in the traditional VMware
products, paravirtualization is a technique where the operating system
is modified to enlighten the hypervisor with timely knowledge about the
operating system's activities. Since the hypervisor now depends on the
kernel to tell it about common idioms etc, it does not need to write
protect OS objects such as page and descriptor tables as a solution
based on full-virtualization needs. This has two important effects (a)
it shortens the critical path, since faulting is expensive on modern
processors (b) by eliminating complex heuristics the hypervisor is
simplified. While the former delivers performance, the latter is quite
important too.
Not surprisingly, paravirtualization's strength, ie that it encourages
tighter communication between the kernel and the hypervisor, is also its
weakness. Unless the changes to the operating system are moderated, you
can very quickly find yourself with a kernel that (a) looks and feels
like a brand new kernel or (b) cannot run on native machines or on newer
versions of the hypervisor without a full recompile. The former can
impede innovation in the Linux kernel, and the latter can be a problem
for software vendors.
VMware proposes VMI as a paravirtualization interface for Linux that
solves these problems.
- A VMI'fied Linux kernel runs unmodified on native hardware, and on
many hypervisors, while simultaneously delivering on the performance
promise of paravirtualization.
- VMI has a rich and low level interface, which allows the kernel to
cope with future hardware evolution by querying for hardware
capability. It is our expectation that a single kernel will run
unmodified on both today's processors with limited hardware
virtualization support and also keep up with any evolution on the
processor front
- VMI Linux is a fairly clean interface, with distinct name spaces
for objects from the kernel and the hypervisor. Nowhere do we mingle
names from the hypervisor with that of the kernel. This separation
allows innovation in the kernel to proceed at the same speed as
always. For most kernel developers, a VMI kernel looks and feels like
a regular Linux kernel.
- VMI Linux still supports "native" hypervisor device drivers, for
example a hypervisor vendor's own private network or block device
drivers which are free to use any interface desired to communicate
with the hypervisor.
At present, we are sharing a working implementation of the VMI for
2.6.16-rc6 version of Linux. We have verified that VMI Linux does indeed
run well on native machines (both P4 and Opterons), and on VMware style
hypervisors. VMI Linux has negligible overheads on native machines, so
much so, that we are confident that VMI Linux can, in the long run, be
the default Linux for i386. We believe that this interface is both
cleaner and more powerful than other proposals that have been made
towards virtualization of Linux, and can easily be adapted to work with
other hypervisors.
This is by no means finished work. A few of the areas that need more
attention and exploration are (a) 64bit support is still lacking, but we
feel a port of VMI to the 64 bit Linux can be done without too much
trouble (b) the Xen compatibility layer needs some work to bring it
up to the Xen 3.0 interfaces. Work is underway on this already, and
no major issues are expected at this time.
Two final notes. This is not an attempt to force a proprietary interface
into the Linux kernel. This is an attempt to find a common interface
that can be used by many hypervisors by isolating hypervisor specific
idioms into a neutral layer. This new layer is just what is claims to
be - a virtual machine interface, which allows hypervisor dependent code
to be abstracted in a way that benefits both Linux and hypervisor
development.
This is also not an attempt to define an exact and final specification
of how virtualization should be done in Linux. This is very much a work
in progress, and it is understood that the interfaces proposed here will
change in time to accommodate the needs of all interested parties. We
hope to find a common solution that can eventually become part of the
Linux kernel and serve as a model for other operating systems as well.
We appreciate your feedback on this design and the patches to Linux, and
welcome working with anyone who is interested in making virtualization
in Linux a friendly environment to innovate in. If you find the ideas
here interesting, please volunteer to help improve them.
> Two final notes. This is not an attempt to force a proprietary interface
> into the Linux kernel. This is an attempt to find a common interface
> that can be used by many hypervisors by isolating hypervisor specific
> idioms into a neutral layer. This new layer is just what is claims to
> be - a virtual machine interface, which allows hypervisor dependent code
> to be abstracted in a way that benefits both Linux and hypervisor
> development.
such an interface should be defined with source visibility of both sides
though. At least of one user. Can XEN or any of the other open
hypervisors use this? What does it look like? And if not, why not,
wouldn't that make VMA a VMwareInterface instead ? ;)
Why can't vmware use the Xen interface instead?
Arjan van de Ven wrote:
>> Two final notes. This is not an attempt to force a proprietary interface
>> into the Linux kernel. This is an attempt to find a common interface
>> that can be used by many hypervisors by isolating hypervisor specific
>> idioms into a neutral layer. This new layer is just what is claims to
>> be - a virtual machine interface, which allows hypervisor dependent code
>> to be abstracted in a way that benefits both Linux and hypervisor
>> development.
>>
>
>
> such an interface should be defined with source visibility of both sides
> though. At least of one user. Can XEN or any of the other open
> hypervisors use this? What does it look like? And if not, why not,
> wouldn't that make VMA a VMwareInterface instead ? ;)
>
Yes, Xen can use this interface, even without modification to Xen. The
interface was used successfully to run a VMI kernel on Xen 2.0. As it
stands now, the interface does need to change a bit to accomodate Xen
3.0 - but it is possible to do. Rather than wait until we have a
working prototype of that, we thought the interface itself warrants
discussion now.
> Why can't vmware use the Xen interface instead?
>
We could. But it is our opinion that the Xen interface is unnecessarily
complicated, without a clean separation between the layer of interaction
with the hypervisor and the kernel proper. The interface we propose we
believe is more powerful, and more conducive to performance
optimizations while providing significant advantages - most
specifically, a single binary image that is properly virtualizable on
multiple hypervisors and capable of running on native hardware.
Zach
> The interface we propose we
> believe is more powerful, and more conducive to performance
> optimizations while providing significant advantages - most
> specifically, a single binary image that is properly virtualizable on
> multiple hypervisors and capable of running on native hardware.
that is mostly an advantage in the binary would though.. less so in the
open source world.
Arjan van de Ven wrote:
>> The interface we propose we
>> believe is more powerful, and more conducive to performance
>> optimizations while providing significant advantages - most
>> specifically, a single binary image that is properly virtualizable on
>> multiple hypervisors and capable of running on native hardware.
>>
>
> that is mostly an advantage in the binary would though.. less so in the
> open source world.
>
It is an advantage for everyone. It cuts support and certification
costs for Linux distributors, software vendors, makes debugging and
development easier, and gives hypervisors room to grow while maintaining
binary compatibility with already released kernels.
Zach
On Mon, 2006-03-13 at 10:30 -0800, Zachary Amsden wrote:
> Arjan van de Ven wrote:
> >> The interface we propose we
> >> believe is more powerful, and more conducive to performance
> >> optimizations while providing significant advantages - most
> >> specifically, a single binary image that is properly virtualizable on
> >> multiple hypervisors and capable of running on native hardware.
> >>
> >
> > that is mostly an advantage in the binary would though.. less so in the
> > open source world.
> >
>
> It is an advantage for everyone. It cuts support and certification
> costs for Linux distributors,
that I'll buy
> software vendors,
that I'll buy a lot less except those with kernel modules (which is
evil ;)
> makes debugging and
> development easier,
that I don't buy; a fixed interface tends to make debugging harder not
easier since you can't change it to add more information
> and gives hypervisors room to grow while maintaining
> binary compatibility with already released kernels.
that I buy for binary only hypervisors. But in an open source world I'll
buy this a LOT less as being relevant.
Arjan van de Ven wrote:
>
>> makes debugging and
>> development easier,
>>
>
> that I don't buy; a fixed interface tends to make debugging harder not
> easier since you can't change it to add more information
This we find to be quite true. Now, you can use a VMI kernel, make
changes to it, run it on native hardware, and be confident that it will
run properly in a VM as well. And you can develop in a VM, with
confidence that you can run on native hardware. You can even replace
the entire "ROM" image with your own custom debugging image to add any
type of debugging or performance monitoring facility you want - and you
have some very, very interesting hook points into the kernel that make
that task much more achievable.
> that I buy for binary only hypervisors. But in an open source world I'll
> buy this a LOT less as being relevant.
>
This is not about the open source versus the closed source world. It is
about the real world, where customers want to make as few changes as
possible to a working and already deployed system. If they have to
recompile a kernel just to get their system to run again, that is a pain
point that is easily avoided.
Zach
>
> On Mar 13, 2006, at 19:42, Arjan van de Ven wrote:
>
> On Mon, 2006-03-13 at 10:30 -0800, Zachary Amsden wrote:
>> and gives hypervisors room to grow while maintaining
>> binary compatibility with already released kernels.
>
> that I buy for binary only hypervisors. But in an open source world
> I'll
> buy this a LOT less as being relevant.
>
Binary compatibility to Linux is pretty important for applications.
Even though Apache is open source, I don't want to recompile it for
every new Linux kernel. Fortunately I don't have to, because glibc
abstracts the Linux kernel interface. Consider VMI in the same role
as glibc -- when the hypervisor changes, VMI maintains compatibility
with your pre-existing infrastructure, while letting you have some of
the benefits of the new hypervisor. The upgrade and recompile game
can quickly end in a stalemate when you have packages with
conflicting dependencies (one package requires the old version, and
the other package requires the new version).
Josh
On Monday 13 March 2006 12:30, Zachary Amsden wrote:
> It is an advantage for everyone. ?It cuts support and certification
> costs for Linux distributors, software vendors, makes debugging and
> development easier, and gives hypervisors room to grow while maintaining
> binary compatibility with already released kernels.
It certainly is good for kernel developers and end-users.
However, it would be a foolish distributor or ISV who tests with one
hypervisor and decides that covers all hypervisors which implement the same
interface. So I'm not sure there's any advantage w.r.t. support and
certification costs.
--
Hollis Blanchard
IBM Linux Technology Center
* Zachary Amsden ([email protected]) wrote:
> This we find to be quite true. Now, you can use a VMI kernel, make
> changes to it, run it on native hardware, and be confident that it will
> run properly in a VM as well. And you can develop in a VM, with
> confidence that you can run on native hardware. You can even replace
> the entire "ROM" image with your own custom debugging image to add any
> type of debugging or performance monitoring facility you want - and you
> have some very, very interesting hook points into the kernel that make
> that task much more achievable.
Replacing a ROM image is easier in terms package management, but still
requires full validation and certification process. Swapping out the
underlying core is a major change and needs to be re-validated.
thanks,
-chris
Hollis Blanchard wrote:
> On Monday 13 March 2006 12:30, Zachary Amsden wrote:
>
>> It is an advantage for everyone. It cuts support and certification
>> costs for Linux distributors, software vendors, makes debugging and
>> development easier, and gives hypervisors room to grow while maintaining
>> binary compatibility with already released kernels.
>>
>
> It certainly is good for kernel developers and end-users.
>
> However, it would be a foolish distributor or ISV who tests with one
> hypervisor and decides that covers all hypervisors which implement the same
> interface. So I'm not sure there's any advantage w.r.t. support and
> certification costs.
>
Your point is well noted. I'm not arguing that it would be smart to
test with just one hypervisor (or worse, yet, test only on native
hardware), and proudly declare your kernel virtualization compatible.
There are some things you can do (instrument a torture test verification
module in a native VMI ROM) to help with that test load.
But in the end, having a single binary reduces the complexity and work
that goes into a certification, which does simplify the process - even
if you still have to validate against the list of all supported vendors
/ hardware.
Zach
Zachary Amsden wrote:
>In OLS 2005, we described the work that we have been doing in VMware
>with respect a common interface for paravirtualization of Linux. We
>shared the general vision in Rik's virtualization BoF.
>[...]
>Unlike the full-virtualization techniques used in the traditional VMware
>products, paravirtualization is a technique where the operating system
>is modified to enlighten the hypervisor with timely knowledge about the
>operating system's activities. Since the hypervisor now depends on the
>kernel to tell it about common idioms etc, it does not need to write
>protect OS objects such as page and descriptor tables as a solution
>based on full-virtualization needs. This has two important effects (a)
>it shortens the critical path, since faulting is expensive on modern
>processors (b) by eliminating complex heuristics the hypervisor is
>simplified. While the former delivers performance, the latter is quite
>important too.
>
>
An interesting vision, especially if it merges the current VMWare / Xen
techno-social rift.
I think there will still be a place for the "complete" (eg, QEMU, or of
course your own product) and the "ultimately lightweight" (eg,
vserver/openvz/jails/containers) approaches to virtualisation, though.
While the same kernel may be able to run in these different situations,
having a "real" hardware emulator like QEMU/VMWare will allow you to
test all of those alternate code paths. As time goes on this will no
doubt seem a more and more superfluous requirement, especially if the
actual code in those places is minimal as you suggest.
Looking the other way, there is no way that a system like this will ever
approach the performance of fork(), as vserver and related technology
does. No doubt for certain common applications of virtualisation, such
as providing "complete" virtual servers, this will be seen as less and
less important as time goes on. However, for other applications - such
as jailing services, or systems that make use of advantages of single
kernel virtualisation (such as shared VFS/network, visibility into other
systems' processes, etc) - the extra kernel that a virtualisation
context implies is simply unwanted. No doubt still other users will
simply prefer the simplicity of system call level virtualisation, such
as only having one set of routing tables/iptables rules/VGs to manage, etc.
There are currently two debates on virtualisation happening here, on and
off. We have Xen/VMI, and vserver/openvz/jails/containers. Let's just
try not to get them confused :-). From the perspective of the vserver
project, I consider your work orthogonal and complementary and wish you
well in the success of your gracious offering to the Linux community.
Sam.
Zachary Amsden wrote:
> This is by no means finished work. A few of the areas that need more
> attention and exploration are (a) 64bit support is still lacking, but we
> feel a port of VMI to the 64 bit Linux can be done without too much
> trouble (b) the Xen compatibility layer needs some work to bring it
> up to the Xen 3.0 interfaces. Work is underway on this already, and
> no major issues are expected at this time.
>
Hi Zach,
Can you please post the Xen compatibility layer (even if it is for
2.0.x). I think it's important to see that code to understand the
advantages/disadvantages compared to the existing Xen paravirtualization
interface. Likewise, any Xen performance data would be useful as there
has been some discussion about whether VMI would negatively impact Xen
performance[1].
Thanks,
Anthony Liguori
> Two final notes. This is not an attempt to force a proprietary interface
> into the Linux kernel. This is an attempt to find a common interface
> that can be used by many hypervisors by isolating hypervisor specific
> idioms into a neutral layer. This new layer is just what is claims to
> be - a virtual machine interface, which allows hypervisor dependent code
> to be abstracted in a way that benefits both Linux and hypervisor
> development.
>
> This is also not an attempt to define an exact and final specification
> of how virtualization should be done in Linux. This is very much a work
> in progress, and it is understood that the interfaces proposed here will
> change in time to accommodate the needs of all interested parties. We
> hope to find a common solution that can eventually become part of the
> Linux kernel and serve as a model for other operating systems as well.
>
> We appreciate your feedback on this design and the patches to Linux, and
> welcome working with anyone who is interested in making virtualization
> in Linux a friendly environment to innovate in. If you find the ideas
> here interesting, please volunteer to help improve them.
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Virtualization mailing list
> [email protected]
> https://lists.osdl.org/mailman/listinfo/virtualization
>
Anthony Liguori wrote:
> Zachary Amsden wrote:
>> This is by no means finished work. A few of the areas that need more
>> attention and exploration are (a) 64bit support is still lacking, but we
>> feel a port of VMI to the 64 bit Linux can be done without too much
>> trouble (b) the Xen compatibility layer needs some work to bring it
>> up to the Xen 3.0 interfaces. Work is underway on this already, and
>> no major issues are expected at this time.
> Hi Zach,
>
> Can you please post the Xen compatibility layer (even if it is for
> 2.0.x). I think it's important to see that code to understand the
> advantages/disadvantages compared to the existing Xen
> paravirtualization interface. Likewise, any Xen performance data
> would be useful as there has been some discussion about whether VMI
> would negatively impact Xen performance[1].
About performance - I actually believe that it is possible to implement
VMI Linux in such a way that it actually has _better_ performance on Xen
than the current XenoLinux kernels.
I'm working on getting together the older interface pieces now.
Zach
On Mon, 13 Mar 2006, Zachary Amsden wrote:
> About performance - I actually believe that it is possible to implement
> VMI Linux in such a way that it actually has _better_ performance on Xen
> than the current XenoLinux kernels.
How would VMI allow page table batching at fault time?
(one of the future optimizations that are probably worth
making for Xen)
--
All Rights Reversed
Hi Zach,
A number of the files you posted (including the vmi_spec.txt) have the
phrase 'All rights reserved'. That seems incompatible with the GPL. In
particular, it makes it unclear about how one can use the actual vmi spec.
In your next round of patches, could you clarify the actual licensing of
the files?
Regards,
Anthony Liguori
Zachary Amsden wrote:
> In OLS 2005, we described the work that we have been doing in VMware
> with respect a common interface for paravirtualization of Linux. We
> shared the general vision in Rik's virtualization BoF.
>
> This note is an update on our further work on the Virtual Machine
> Interface, VMI. The patches provided have been tested on 2.6.16-rc6.
> We are currently recollecting performance information for the new -rc6
> kernel, but expect our numbers to match previous results, which showed
> no impact whatsoever on macro benchmarks, and nearly neglible impact
> on microbenchmarks.
>
> Unlike the full-virtualization techniques used in the traditional VMware
> products, paravirtualization is a technique where the operating system
> is modified to enlighten the hypervisor with timely knowledge about the
> operating system's activities. Since the hypervisor now depends on the
> kernel to tell it about common idioms etc, it does not need to write
> protect OS objects such as page and descriptor tables as a solution
> based on full-virtualization needs. This has two important effects (a)
> it shortens the critical path, since faulting is expensive on modern
> processors (b) by eliminating complex heuristics the hypervisor is
> simplified. While the former delivers performance, the latter is quite
> important too.
>
> Not surprisingly, paravirtualization's strength, ie that it encourages
> tighter communication between the kernel and the hypervisor, is also its
> weakness. Unless the changes to the operating system are moderated, you
> can very quickly find yourself with a kernel that (a) looks and feels
> like a brand new kernel or (b) cannot run on native machines or on newer
> versions of the hypervisor without a full recompile. The former can
> impede innovation in the Linux kernel, and the latter can be a problem
> for software vendors.
>
> VMware proposes VMI as a paravirtualization interface for Linux that
> solves these problems.
> - A VMI'fied Linux kernel runs unmodified on native hardware, and on
> many hypervisors, while simultaneously delivering on the performance
> promise of paravirtualization.
> - VMI has a rich and low level interface, which allows the kernel to
> cope with future hardware evolution by querying for hardware
> capability. It is our expectation that a single kernel will run
> unmodified on both today's processors with limited hardware
> virtualization support and also keep up with any evolution on the
> processor front
> - VMI Linux is a fairly clean interface, with distinct name spaces
> for objects from the kernel and the hypervisor. Nowhere do we mingle
> names from the hypervisor with that of the kernel. This separation
> allows innovation in the kernel to proceed at the same speed as
> always. For most kernel developers, a VMI kernel looks and feels like
> a regular Linux kernel.
> - VMI Linux still supports "native" hypervisor device drivers, for
> example a hypervisor vendor's own private network or block device
> drivers which are free to use any interface desired to communicate
> with the hypervisor.
>
> At present, we are sharing a working implementation of the VMI for
> 2.6.16-rc6 version of Linux. We have verified that VMI Linux does indeed
> run well on native machines (both P4 and Opterons), and on VMware style
> hypervisors. VMI Linux has negligible overheads on native machines, so
> much so, that we are confident that VMI Linux can, in the long run, be
> the default Linux for i386. We believe that this interface is both
> cleaner and more powerful than other proposals that have been made
> towards virtualization of Linux, and can easily be adapted to work with
> other hypervisors.
>
> This is by no means finished work. A few of the areas that need more
> attention and exploration are (a) 64bit support is still lacking, but we
> feel a port of VMI to the 64 bit Linux can be done without too much
> trouble (b) the Xen compatibility layer needs some work to bring it
> up to the Xen 3.0 interfaces. Work is underway on this already, and
> no major issues are expected at this time.
>
> Two final notes. This is not an attempt to force a proprietary interface
> into the Linux kernel. This is an attempt to find a common interface
> that can be used by many hypervisors by isolating hypervisor specific
> idioms into a neutral layer. This new layer is just what is claims to
> be - a virtual machine interface, which allows hypervisor dependent code
> to be abstracted in a way that benefits both Linux and hypervisor
> development.
>
> This is also not an attempt to define an exact and final specification
> of how virtualization should be done in Linux. This is very much a work
> in progress, and it is understood that the interfaces proposed here will
> change in time to accommodate the needs of all interested parties. We
> hope to find a common solution that can eventually become part of the
> Linux kernel and serve as a model for other operating systems as well.
>
> We appreciate your feedback on this design and the patches to Linux, and
> welcome working with anyone who is interested in making virtualization
> in Linux a friendly environment to innovate in. If you find the ideas
> here interesting, please volunteer to help improve them.
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Virtualization mailing list
> [email protected]
> https://lists.osdl.org/mailman/listinfo/virtualization
>
Anthony Liguori wrote:
> Hi Zach,
>
> A number of the files you posted (including the vmi_spec.txt) have the
> phrase 'All rights reserved'. That seems incompatible with the GPL.
> In particular, it makes it unclear about how one can use the actual
> vmi spec.
>
> In your next round of patches, could you clarify the actual licensing
> of the files?
I'm sorry about the legalese. The patches are patches to the Linux
kernel, and therefore under GPL v2 by default. I thought that would be
implicit.
Zach
On Mon, 13 Mar 2006, Zachary Amsden wrote:
> Anthony Liguori wrote:
> > In your next round of patches, could you clarify the actual licensing of the
> > files?
>
> I'm sorry about the legalese. The patches are patches to the Linux kernel,
> and therefore under GPL v2 by default. I thought that would be implicit.
It would be very bad if Linus started applying code with
a dubious license to the kernel, if we want to keep the
kernel GPL v2.
Having an explicit license and a Signed-off-by: line are
things to remember with big patch sets. At the very least
a Signed-off-by: line..
--
All Rights Reversed
Rik van Riel wrote:
> On Mon, 13 Mar 2006, Zachary Amsden wrote:
>
>
>> About performance - I actually believe that it is possible to implement
>> VMI Linux in such a way that it actually has _better_ performance on Xen
>> than the current XenoLinux kernels.
>>
>
> How would VMI allow page table batching at fault time?
> (one of the future optimizations that are probably worth
> making for Xen)
>
This is exactly what we do. All page table transitions from P->NP or
P->P already require a flushing call (FlushTLB or InvalPage). The
remaining transitions, NP->P require explicit flushing, and we have
added the appropriate call sites to do so. It turns out, the external
MMU cache on Sparc provided exactly the required hook point in this case
- update_mmu_cache().
Zach
Rik van Riel wrote:
> It would be very bad if Linus started applying code with
> a dubious license to the kernel, if we want to keep the
> kernel GPL v2.
>
I believe it says explicitly in our patches that they are licensed under
GPL v2.
> Having an explicit license and a Signed-off-by: line are
> things to remember with big patch sets. At the very least
> a Signed-off-by: line.
>
There is a Signed-off-by line on every patch I send out, with full
knowledge that this constitutes the work of the author of the said line,
and full knowledge that this commits the patch into the domain of the
GPL license. Sorry for sounding like a lawyer here. IANAL, but I
thought that was completely implicit in all patches made to GPL'd
software. The signed off by provides accountability and open licensing
simultaneously.
But most importantly, I really don't understand how it is possible to
make a patch to the Linux kernel and not release it under GPL.
Zach
On Mon, 13 Mar 2006, Zachary Amsden wrote:
> There is a Signed-off-by line on every patch I send out,
You're right. It was just the first 1/24 that was missing it,
it was there in the second copy.
> But most importantly, I really don't understand how it is possible to
> make a patch to the Linux kernel and not release it under GPL.
This can really only be done if the person posting the patch
does not have the right to release the code. This is what the
Signed-off-by lines are for, IIRC.
--
All Rights Reversed
Rik van Riel wrote:
> On Mon, 13 Mar 2006, Zachary Amsden wrote:
>
>
>> There is a Signed-off-by line on every patch I send out,
>>
>
> You're right. It was just the first 1/24 that was missing it,
> it was there in the second copy.
>
BTW, I have no idea why the first 1/24 was missing it. I checked right
before sending, and it was there - perhaps I forgot to save my changes.
The second copy turned out fine, but didn't make it to LKML. Everyone
cc'd directly got it, but the LKML filter has a ban on the word
propasition, and being blackholed by it, I merely assumed the patch was
too large - so I split it up, and actually ended up binary searching
down to the problematic section before finding the taboo list.
_Every_ problem eventually turns into a binary search.
Zach
On Mon, Mar 13, 2006 at 10:22:15AM -0800, Zachary Amsden wrote:
> >Why can't vmware use the Xen interface instead?
> >
>
> We could. But it is our opinion that the Xen interface is unnecessarily
> complicated, without a clean separation between the layer of interaction
> with the hypervisor and the kernel proper. The interface we propose we
> believe is more powerful, and more conducive to performance
> optimizations while providing significant advantages - most
> specifically, a single binary image that is properly virtualizable on
> multiple hypervisors and capable of running on native hardware.
I agree with Zach here, the Xen hypervisor <-> kernel interface is
not very nice. This proposal seems like a step forward althogh it'll
probably need to go through a few iterations. Without and actually
useable opensource hypevisor reference implementation it's totally
unacceptable, though.
Christoph Hellwig wrote:
> I agree with Zach here, the Xen hypervisor <-> kernel interface is
> not very nice. This proposal seems like a step forward althogh it'll
> probably need to go through a few iterations. Without and actually
> useable opensource hypevisor reference implementation it's totally
> unacceptable, though.
>
Which is why our top priority is getting VMI Linux to run on Xen. The
churn rate on both ends has been very high, and we really wanted to
release our patches with Xen support, but we also didn't want to wait
some unknown number of weeks more to release them - and we're actually
looking for volunteers to help with the port if anyone is interested.
What we are hoping for, in the end, is a Linux kernel with a clean
virtualization interface, that is maintainable, does not slow hypervisor
exploitation of new technologies, still offers all of the same
performance advantages, works for the open source community, and allows
hypervisor vendors of many creeds to benefit from cross-kernel binary
compatibility.
Zach
On Mar 15, 2006, at 11:25 , Christoph Hellwig wrote:
> On Mon, Mar 13, 2006 at 10:22:15AM -0800, Zachary Amsden wrote:
>>> Why can't vmware use the Xen interface instead?
>>>
>>
>> We could. But it is our opinion that the Xen interface is
>> unnecessarily
>> complicated, without a clean separation between the layer of
>> interaction
>> with the hypervisor and the kernel proper. The interface we
>> propose we
>> believe is more powerful, and more conducive to performance
>> optimizations while providing significant advantages - most
>> specifically, a single binary image that is properly virtualizable on
>> multiple hypervisors and capable of running on native hardware.
>
> I agree with Zach here, the Xen hypervisor <-> kernel interface is
> not very nice. This proposal seems like a step forward althogh it'll
> probably need to go through a few iterations. Without and actually
> useable opensource hypevisor reference implementation it's totally
> unacceptable, though.
>
As part of our pre-virtualization work, we developed a virtualization
solution similar to VMI. We support Xen v2 and v3 with high
performance. We added support for the first generation of VMI to our
project, and are currently adding support for the latest VMI patch.
Our work is open source. We'll announce when we finish the VMI updates.
We also experimented with other architectures and found the approach
highly suitable, such as for Itanium.
Joshua
Joshua LeVasseur <[email protected]> wrote:
>
> As part of our pre-virtualization work, we developed a virtualization
> solution similar to VMI. We support Xen v2 and v3 with high
> performance. We added support for the first generation of VMI to our
> project, and are currently adding support for the latest VMI patch.
> Our work is open source. We'll announce when we finish the VMI updates.
Who is "we" and what product are you referring to?
(I think an important part of this discussion is getting an understanding
of which virtualisation products (current or planned) could use a VMI).
Thanks.
On Mar 15, 2006, at 21:02 , Andrew Morton wrote:
> Joshua LeVasseur <[email protected]> wrote:
>>
>> As part of our pre-virtualization work, we developed a virtualization
>> solution similar to VMI. We support Xen v2 and v3 with high
>> performance. We added support for the first generation of VMI to our
>> project, and are currently adding support for the latest VMI patch.
>> Our work is open source. We'll announce when we finish the VMI
>> updates.
>
> Who is "we" and what product are you referring to?
>
> (I think an important part of this discussion is getting an
> understanding
> of which virtualisation products (current or planned) could use a
> VMI).
>
> Thanks.
This is a project at the University of Karlsruhe. The project web
page is:
http://l4ka.org/projects/virtualization/afterburn/
and there you'll find documentation and source code. The source code
supports our pre-virtualization project on L4 and Xen (and we have
some nascent implementations for Linux-as-hypervisor and Windows-as-
hypervisor). We automate the transformations to the Linux code, thus
minimizing the number of manual modifications. Due to the
similarities of our approach and VMI, our virtualization runtime
supports VMI with some minor additions.
Joshua
>> > and gives hypervisors room to grow while maintaining
>> > binary compatibility with already released kernels.
>>
>> that I buy for binary only hypervisors. But in an open source world I'll
>> buy this a LOT less as being relevant.
>
> Binary compatibility to Linux is pretty important for applications. Even
> though Apache is open source, I don't want to recompile it for every new Linux
> kernel. Fortunately I don't have to, because glibc abstracts the Linux kernel
> interface. Consider VMI in the same role as glibc -- when the hypervisor
> changes, VMI maintains compatibility with your pre-existing infrastructure,
VMI = kernel code (AFAIU)
I would rather like a user-space-based compat layer.
Jan Engelhardt
--
>
> But most importantly, I really don't understand how it is possible to make a
> patch to the Linux kernel and not release it under GPL.
>
If the patch is so ultimatively trivial that there is only a few solutions (one
or two), then there is no use in gpl'ing that flock of patchcode, in which case
I think, it is (or at best should be) public domain. In conjunction with the
patched function, they will/should become GPL.
Jan Engelhardt
--
In-Reply-To: <[email protected]>
On Wed, 15 Mar 2006 10:25:22 +0000, Christoph Hellwig wrote:
> I agree with Zach here, the Xen hypervisor <-> kernel interface is
> not very nice. This proposal seems like a step forward althogh it'll
> probably need to go through a few iterations. Without and actually
> useable opensource hypevisor reference implementation it's totally
> unacceptable, though.
I'd like to see a test harness implementation that has no actual
hypervisor functionality and just implements the VMI calls natively.
This could be used to test the interface and would provide a nice
starting point for those who want to write a VMI hypervisor.
--
Chuck
"Penguins don't come from next door, they come from the Antarctic!"
Chuck Ebbert wrote:
> In-Reply-To: <[email protected]>
>
> On Wed, 15 Mar 2006 10:25:22 +0000, Christoph Hellwig wrote:
>
> I'd like to see a test harness implementation that has no actual
> hypervisor functionality and just implements the VMI calls natively.
> This could be used to test the interface and would provide a nice
> starting point for those who want to write a VMI hypervisor.
>
I was going to make one yesterday. But Fry's electronics stopped
carrying flashable blank PCI cards. :) Anyone know of a vendor?
It is possible to do in a software layer, although it really is a lot
easier to have the BIOS take care of all the fuss of finding a place in
low memory for you to live, setting up the various memory maps and
everything else for you.
There is enormous benefit to having such a layer - you have a very power
test harness, not just to make sure VMI works, but even more
importantly, to inspect and verify the native kernel operation as well.
You have a plethora of imporant hooks into the system, which feed you
knowledge you can not otherwise gain about which page tables have been
made active, when you take IRQs, where the kernel stack lives.
All of this is ripe for a debug harness that can verify the kernel
doesn't overflow the kernel stack, doesn't write to active page table
entries without proper accessors and subsequent invalidations, and obeys
the rules that are required for correctness when running under a
hypervisor. You probably even want to do hypervisor like things - such
as write protecting the kernel page tables so that you can be confident
there are no stray raw PTE accesses.
We actually found one (harmless on native) in i386, which was enabling
NX bit.
Zach
Zachary Amsden wrote:
> Chuck Ebbert wrote:
>> In-Reply-To: <[email protected]>
>>
>> On Wed, 15 Mar 2006 10:25:22 +0000, Christoph Hellwig wrote:
>> I'd like to see a test harness implementation that has no actual
>> hypervisor functionality and just implements the VMI calls natively.
>> This could be used to test the interface and would provide a nice
>> starting point for those who want to write a VMI hypervisor.
>>
>
> I was going to make one yesterday. But Fry's electronics stopped
> carrying flashable blank PCI cards. :) Anyone know of a vendor?
It's very practical to just patch Qemu to load a VMI rom as an option
ROM. That makes such an example VMI ROM very practical without having
to build a special PCI device.
Regards,
Anthony Liguori
> It is possible to do in a software layer, although it really is a lot
> easier to have the BIOS take care of all the fuss of finding a place
> in low memory for you to live, setting up the various memory maps and
> everything else for you.
>
> There is enormous benefit to having such a layer - you have a very
> power test harness, not just to make sure VMI works, but even more
> importantly, to inspect and verify the native kernel operation as
> well. You have a plethora of imporant hooks into the system, which
> feed you knowledge you can not otherwise gain about which page tables
> have been made active, when you take IRQs, where the kernel stack lives.
>
> All of this is ripe for a debug harness that can verify the kernel
> doesn't overflow the kernel stack, doesn't write to active page table
> entries without proper accessors and subsequent invalidations, and
> obeys the rules that are required for correctness when running under a
> hypervisor. You probably even want to do hypervisor like things -
> such as write protecting the kernel page tables so that you can be
> confident there are no stray raw PTE accesses.
>
> We actually found one (harmless on native) in i386, which was enabling
> NX bit.
>
> Zach
>
> _______________________________________________
> Xen-devel mailing list
> [email protected]
> http://lists.xensource.com/xen-devel
[Apologies for resend: earlier email with html attachments was
rejected. Resending with txt attachments.]
>From: Zachary Amsden [mailto:[email protected]]
>Sent: Monday, March 13, 2006 9:58 AM
>In OLS 2005, we described the work that we have been doing in VMware
>with respect a common interface for paravirtualization of Linux. We
>shared the general vision in Rik's virtualization BoF.
>This note is an update on our further work on the Virtual Machine
>Interface, VMI. The patches provided have been tested on 2.6.16-rc6.
>We are currently recollecting performance information for the new -rc6
>kernel, but expect our numbers to match previous results, which showed
>no impact whatsoever on macro benchmarks, and nearly neglible impact
>on microbenchmarks.
Folks,
I'm a member of the performance team at VMware & I recently did a
round of testing measuring the performance of a set of benchmarks
on the following 2 linux variants, both running natively:
1) 2.6.16-rc6 including VMI + 64MB hole
2) 2.6.16-rc6 not including VMI + no 64MB hole
The intent was to measure the overhead of VMI calls on native runs.
Data was collected on both p4 & opteron boxes. The workloads used
were dbench/1client, netperf/receive+send, UP+SMP kernel compile,
lmbench, & some VMware in-house kernel microbenchmarks. The CPU(s)
were pegged for all workloads except netperf, for which I include
CPU utilization measurements.
Attached please find a text file presenting the benchmark results
collected in terms of ratio of 1) to 2), along with the raw scores
given in brackets. System configurations & benchmark descriptions
are given at the end of the page; more details are available on
request. Also attached for reference is a text file giving the
width of the 95% confidence interval around the mean of the scores
reported for each benchmark, expressed as a percentage of the mean.
The VMI-Native & Native scores for almost all workloads match
within the 95% confidence interval. On the P4, only 4 workloads,
all lmbench microbenchmarks (forkproc,shproc,mmap,pagefault) were
outside the interval & the overheads (2%,1%,2%,1%, respectively)
were low. The opteron microbenchmark data was a little more
ragged than the P4 in terms of variance, but it appears that only
a few lmbench microbenchmarks (forkproc,execproc,shproc) were
outside their confidence intervals and they show low overheads
(4%,3%,2%, respectively); our in-house segv & divzero seemed to
show measureable overheads as well (8%,9%).
-Regards, Anne Holler ([email protected])