2008-06-16 15:06:29

by Martin Michlmayr

[permalink] [raw]
Subject: Linux with kvm-intel locks up VMplayer guest is started

When I start a guest in vmplayer, my laptop locks up hard. I found
out that this only happens when the kvm-intel module has been loaded.
When I unload this module, I can start vmware guests just fine.

I see this problem on a HP Compaq 2510p notebook with Intel Core2 Duo
U7600 @ 1.20GHz, but http://laptop006.livejournal.com/41314.html
describes exactly the same problem on a Lenovo T61 (custom build),
Core 2 T7500 on a Santa Rosa. I've seen this with at least 2.6.24 and
2.6.25.

I don't have a serial console, but I started VMplayer and
immediately switched to the console but nothing is displayed
(i.e. no kernel panic; it simply locks up hard).

Any idea if this is a kernel bug, BIOS issue or a bug in VMplayer?

--
Martin Michlmayr
http://www.cyrius.com/


2008-06-16 15:22:12

by Anthony Liguori

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

Martin Michlmayr wrote:
> When I start a guest in vmplayer, my laptop locks up hard. I found
> out that this only happens when the kvm-intel module has been loaded.
> When I unload this module, I can start vmware guests just fine.
>
> I see this problem on a HP Compaq 2510p notebook with Intel Core2 Duo
> U7600 @ 1.20GHz, but http://laptop006.livejournal.com/41314.html
> describes exactly the same problem on a Lenovo T61 (custom build),
> Core 2 T7500 on a Santa Rosa. I've seen this with at least 2.6.24 and
> 2.6.25.
>
> I don't have a serial console, but I started VMplayer and
> immediately switched to the console but nothing is displayed
> (i.e. no kernel panic; it simply locks up hard).
>
> Any idea if this is a kernel bug, BIOS issue or a bug in VMplayer?
>

This has been fixed in the latest KVM code base AFAIK.

Regards,

Anthony Liguori

2008-06-17 12:41:22

by Martin Michlmayr

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

* Anthony Liguori <[email protected]> [2008-06-16 10:21]:
>> Any idea if this is a kernel bug, BIOS issue or a bug in VMplayer?
>
> This has been fixed in the latest KVM code base AFAIK.

Can you give some more information? Which commit fixed the problem?
Was the patch submitted to the 2.6.25-stable tree?
--
Martin Michlmayr
http://www.cyrius.com/

2008-06-17 13:17:23

by Anthony Liguori

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

Martin Michlmayr wrote:
> * Anthony Liguori <[email protected]> [2008-06-16 10:21]:
>
>>> Any idea if this is a kernel bug, BIOS issue or a bug in VMplayer?
>>>
>> This has been fixed in the latest KVM code base AFAIK.
>>
>
> Can you give some more information? Which commit fixed the problem?
>

commit 1f4e1d64122369d65d8a9c616112780019fb6188
Author: Eli Collins <[email protected]>
Date: Sun Jun 1 20:24:40 2008 -0700

> Was the patch submitted to the 2.6.25-stable tree?
>

No, it's not a regression. I misread your first email btw, you have to
unload the kvm-intel kernel module to use VMware.

VMware is a binary kernel module that's out of kernel. KVM is not
misbehaving and the fact that VMware breaks when the KVM module is
loaded isn't our problem. If they submitted their code for inclusion in
mainline, we could possibly come up with solution for arbitrating who is
using VT.

Regards,

Anthony Liguori

2008-06-17 13:40:29

by Martin Michlmayr

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

* Anthony Liguori <[email protected]> [2008-06-17 08:16]:
> VMware is a binary kernel module that's out of kernel. KVM is not
> misbehaving and the fact that VMware breaks when the KVM module is
> loaded isn't our problem. If they submitted their code for
> inclusion in mainline, we could possibly come up with solution for
> arbitrating who is using VT.

I feared I'd get a response like this. But unless this is a known
issue in VMware (which I don't think it is), you don't know whether
it's not a bug in kvm-intel.

--
Martin Michlmayr
http://www.cyrius.com/

2008-06-17 13:52:50

by Martin Michlmayr

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

* Martin Michlmayr <[email protected]> [2008-06-17 15:32]:
> * Anthony Liguori <[email protected]> [2008-06-17 08:16]:
> > VMware is a binary kernel module that's out of kernel. KVM is not
> > misbehaving and the fact that VMware breaks when the KVM module is
> > loaded isn't our problem. If they submitted their code for
> > inclusion in mainline, we could possibly come up with solution for
> > arbitrating who is using VT.
>
> I feared I'd get a response like this. But unless this is a known
> issue in VMware (which I don't think it is), you don't know whether
> it's not a bug in kvm-intel.

Sorry, Anthony, I just realized I misparsed your response. So you're
saying that it's a known issue and that VMware is the problem. Thanks
a lot! I'll take it up with VMware.

--
Martin Michlmayr
http://www.cyrius.com/

2008-06-17 14:03:05

by Javier Guerra

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

On Tue, Jun 17, 2008 at 8:44 AM, Martin Michlmayr <[email protected]> wrote:
> Sorry, Anthony, I just realized I misparsed your response. So you're
> saying that it's a known issue and that VMware is the problem. Thanks
> a lot! I'll take it up with VMware.

i think what he's saying is that VMWare is a closed binary blob
executing in the kernel; so there's no way to certify anything with
this.

as soon as you put some unknown (and unknowable, unverifiable,
untrustable) code in the kernel, you can't know what will work and
what won't.

--
Javier

2008-06-17 15:47:03

by Alan

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

On Tue, 17 Jun 2008 15:32:34 +0200
Martin Michlmayr <[email protected]> wrote:

> * Anthony Liguori <[email protected]> [2008-06-17 08:16]:
> > VMware is a binary kernel module that's out of kernel. KVM is not
> > misbehaving and the fact that VMware breaks when the KVM module is
> > loaded isn't our problem. If they submitted their code for
> > inclusion in mainline, we could possibly come up with solution for
> > arbitrating who is using VT.
>
> I feared I'd get a response like this. But unless this is a known
> issue in VMware (which I don't think it is), you don't know whether
> it's not a bug in kvm-intel.

So ask vmware. They have source to both parts we don't.

And its a bug in vmware by definition. If the emulator differs from the
real thing then the emulator is at fault. It might well be that the limit
is a genuine technical limit - but its still an emulator fault.

Alan

2008-06-17 18:10:15

by Anthony Liguori

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

Martin Michlmayr wrote:
> * Anthony Liguori <[email protected]> [2008-06-17 08:16]:
>
>> VMware is a binary kernel module that's out of kernel. KVM is not
>> misbehaving and the fact that VMware breaks when the KVM module is
>> loaded isn't our problem. If they submitted their code for
>> inclusion in mainline, we could possibly come up with solution for
>> arbitrating who is using VT.
>>
>
> I feared I'd get a response like this. But unless this is a known
> issue in VMware (which I don't think it is), you don't know whether
> it's not a bug in kvm-intel.
>

We know exactly what the problem is. KVM activates VT unconditionally.
There's no hardware mechanism to arbitrate access to VT. KVM is the
only thing in the Linux kernel that uses VT so we don't have a software
mechanism to arbitrate access to VT.

If the VMware code was upstream, then we could work together to make a
software arbitration mechanism. It's not, and worse yet, it's closed
source so there's no chance it will be. Even if someone wrote an
arbitration mechanism and got VMware to use it, it still shouldn't be
merged because KVM would be the only thing using that mechanism
upstream. I'm not interested in adding kernel infrastructure to support
external binary kernel modules.

Regards,

Anthony Liguori

2008-06-18 13:56:50

by Avi Kivity

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

Anthony Liguori wrote:
>
> We know exactly what the problem is. KVM activates VT
> unconditionally. There's no hardware mechanism to arbitrate access to
> VT. KVM is the only thing in the Linux kernel that uses VT so we
> don't have a software mechanism to arbitrate access to VT.
>
> If the VMware code was upstream, then we could work together to make a
> software arbitration mechanism. It's not, and worse yet, it's closed
> source so there's no chance it will be. Even if someone wrote an
> arbitration mechanism and got VMware to use it, it still shouldn't be
> merged because KVM would be the only thing using that mechanism
> upstream. I'm not interested in adding kernel infrastructure to
> support external binary kernel modules.

Well a recent patch from Eli Collins mentioned VMware are standardizing
CR4.VMXE as an indicator of whether someone is using the VT hardware or
not, and now kvm clears that bit when unloading. We could check the bit
and fail if is set, thus have working mutual exclusion. Of course, it
will only work with newer versions of kvm and vmware.

I don't see any reason to poke sticks into the wheels here.

--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

2008-06-18 19:05:55

by Roland

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

i`d like to give a comment about those claims in this thread (see below), vmware kernel part would be closed source or binary module.

i think, they aren?t. they just are not available via public/opensource code repository.

just go and download workstation 6.5 beta from:

http://download3.vmware.com/software/wkst/VMware-workstation-e.x.p-91182.i386.tar.gz
(you may need register for that)

unpack and take a look at vmware-distrib/lib/modules/source - there is source for 6 independent kernel
modules in those .tar files:

vmblock, vmci, vmmon, vmnet, vmppuser and vsock. (~3MB alltogether)

regards
roland


------------------------------------------------------------------------------------

>i think what he's saying is that VMWare is a closed binary blob
>executing in the kernel; so there's no way to certify anything with
>this.

>as soon as you put some unknown (and unknowable, unverifiable,
>untrustable) code in the kernel, you can't know what will work and
>what won't.

------------------------------------------------------------------------------------

>So ask vmware. They have source to both parts we don't."

------------------------------------------------------------------------------------

>If the VMware code was upstream, then we could work together to make a
>software arbitration mechanism. It's not, and worse yet, it's closed
>source so there's no chance it will be. Even if someone wrote an
>arbitration mechanism and got VMware to use it, it still shouldn't be
>merged because KVM would be the only thing using that mechanism
>upstream. I'm not interested in adding kernel infrastructure to support
>external binary kernel modules.

------------------------------------------------------------------------------------

>VMware is a binary kernel module that's out of kernel. KVM is not
>misbehaving and the fact that VMware breaks when the KVM module is
>loaded isn't our problem. If they submitted their code for inclusion in
>mainline, we could possibly come up with solution for arbitrating who is
>using VT.

------------------------------------------------------------------------------------


_______________________________________________________________________
Jetzt neu! Sch?tzen Sie Ihren PC mit McAfee und WEB.DE. 30 Tage
kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220

2008-06-18 19:53:30

by Anthony Liguori

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

[email protected] wrote:
> i`d like to give a comment about those claims in this thread (see below), vmware kernel part would be closed source or binary module.
>
> i think, they aren?t. they just are not available via public/opensource code repository.
>
> just go and download workstation 6.5 beta from:
>
> http://download3.vmware.com/software/wkst/VMware-workstation-e.x.p-91182.i386.tar.gz
> (you may need register for that)
>
> unpack and take a look at vmware-distrib/lib/modules/source - there is source for 6 independent kernel
> modules in those .tar files:
>
> vmblock, vmci, vmmon, vmnet, vmppuser and vsock. (~3MB alltogether)
>

vmmon is sort of a stub that loads the actual monitor which is a binary
blob.

Regards,

Anthony Liguori

> regards
> roland
>
>
> ------------------------------------------------------------------------------------
>
>
>> i think what he's saying is that VMWare is a closed binary blob
>> executing in the kernel; so there's no way to certify anything with
>> this.
>>
>
>
>> as soon as you put some unknown (and unknowable, unverifiable,
>> untrustable) code in the kernel, you can't know what will work and
>> what won't.
>>
>
> ------------------------------------------------------------------------------------
>
>
>> So ask vmware. They have source to both parts we don't."
>>
>
> ------------------------------------------------------------------------------------
>
>
>> If the VMware code was upstream, then we could work together to make a
>> software arbitration mechanism. It's not, and worse yet, it's closed
>> source so there's no chance it will be. Even if someone wrote an
>> arbitration mechanism and got VMware to use it, it still shouldn't be
>> merged because KVM would be the only thing using that mechanism
>> upstream. I'm not interested in adding kernel infrastructure to support
>> external binary kernel modules.
>>
>
> ------------------------------------------------------------------------------------
>
>
>> VMware is a binary kernel module that's out of kernel. KVM is not
>> misbehaving and the fact that VMware breaks when the KVM module is
>> loaded isn't our problem. If they submitted their code for inclusion in
>> mainline, we could possibly come up with solution for arbitrating who is
>> using VT.
>>
>
> ------------------------------------------------------------------------------------
>
>
> _______________________________________________________________________
> Jetzt neu! Sch?tzen Sie Ihren PC mit McAfee und WEB.DE. 30 Tage
> kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220
>
>

2008-06-18 21:00:10

by Anthony Liguori

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started

Avi Kivity wrote:
> Anthony Liguori wrote:
>>
>> We know exactly what the problem is. KVM activates VT
>> unconditionally. There's no hardware mechanism to arbitrate access
>> to VT. KVM is the only thing in the Linux kernel that uses VT so we
>> don't have a software mechanism to arbitrate access to VT.
>>
>> If the VMware code was upstream, then we could work together to make
>> a software arbitration mechanism. It's not, and worse yet, it's
>> closed source so there's no chance it will be. Even if someone wrote
>> an arbitration mechanism and got VMware to use it, it still shouldn't
>> be merged because KVM would be the only thing using that mechanism
>> upstream. I'm not interested in adding kernel infrastructure to
>> support external binary kernel modules.
>
> Well a recent patch from Eli Collins mentioned VMware are
> standardizing CR4.VMXE as an indicator of whether someone is using the
> VT hardware or not, and now kvm clears that bit when unloading. We
> could check the bit and fail if is set, thus have working mutual
> exclusion. Of course, it will only work with newer versions of kvm
> and vmware.

Yup, but the original bug reporter would not be helped by this.

> I don't see any reason to poke sticks into the wheels here.

An occasional poke at binary kernel modules is always a good thing IMHO :-)

Regards,

Anthony Liguori

2008-06-19 01:09:17

by Eli Collins

[permalink] [raw]
Subject: Re: Linux with kvm-intel locks up VMplayer guest is started


Hi Martin,

This lockup is fixed in the latest dot releases of Workstation, Player,
and Server.

The vmmon module now checks if VT is already enabled (actually it just
tests if CR4.VMXE is set since checking if the cpu is in VMX mode is a
pain). The vmmon module enables VMX mode when world switching to our
monitor and disables it when world switching back to the host. The
kvm-intel module puts the cpu(s) in VMX mode and does not exit VMX mode
until the module is unloaded, so if you want to run player and kvm you
need to first unload the kvm-intel module. However, this only works with
very recent kvm modules. If you're using a kvm-intel module that doesn't
contain the patch I submitted a couple weeks ago then even unloading the
kvm-intel module is not enough since it leaves CR4.VMXE set. So for "old"
modules you need to not load the kvm-intel module in the first place.

Note that the above is only relevant to kvm-intel. The vmmon and kvm-amd
modules do not need exclusive access to SVM mode. I've run VMs on kvm-69
and Workstation 6.5 beta simultaneously on an AMD-V host without any
issues.

Also note that this issue is relevant for any software that wants to use
VT: QEMU, VirtualBox, Parallels, etc. won't be able to if the host (via
the kvm-intel module) has already claimed it.

Thanks,
Eli