Kenneth reported that a VMWare guest crashes in the VMWare BIOS due to a
Split-Lock induced #AC which is injected by the VMWare hypervisor into the
guest.
While this is a good thing in principle, it's not really practical.
That means that Split-Lock-Detection has to be disabled when any
unprepared VMX hypervisor is loaded.
As hypervisor modules are not really identifiable, the only safe solution
we came up with is to scan the module text at load time for a VMLAUNCH
instruction. If VMLAUNCH is found then Split-Lock-Detection is disabled on
the host to prevent the above. If the hypervisor has at least minimal
handling code, the module can tell the kernel by adding MOD_INFO(sld_safe,
"Y") which disables the text scan.
For KVM it's simple enough to handle it at least at the basic level by
checking guest CR0.AM and EFLAGS.AC state and a trivial host side
handler which depending on the SLD mode handles it gracefully or tells
the VMX handler to deliver the #AC to user space which then can crash
and burn itself.
As Peter and myself don't have access to a SLD enabled machine, the
KVM/VMX part is untested. The module scan part works.
Alternatively we can obviously revert SLD, but that does not make the
problem vs. out of tree hypervisors go away magically. So we can just
get over it now.
Thanks,
tglx
On Thu, 2 Apr 2020, Thomas Gleixner wrote:
> As Peter and myself don't have access to a SLD enabled machine, the
> KVM/VMX part is untested. The module scan part works.
I just applied both of these patches to my (Linus' tip) tree, and unfortunately
VMWare still hangs if split_lock_detect= is set to anything but "off".
Was there anything else I'd needed to do?
-Kenny
--
Kenneth R. Crudup Sr. SW Engineer, Scott County Consulting, Silicon Valley
On Thu, Apr 02, 2020 at 06:43:19AM -0700, Kenneth R. Crudup wrote:
>
> On Thu, 2 Apr 2020, Thomas Gleixner wrote:
>
> > As Peter and myself don't have access to a SLD enabled machine, the
> > KVM/VMX part is untested. The module scan part works.
>
> I just applied both of these patches to my (Linus' tip) tree, and unfortunately
> VMWare still hangs if split_lock_detect= is set to anything but "off".
>
> Was there anything else I'd needed to do?
Hmm, you're not seeing this:
+ pr_warn("disabled due to VMLAUNCH in module: %s\n", me->name);
fire when you load the vmware kernel module? Could you slip me a copy of
that thing by private mail?
> On Apr 2, 2020, at 7:37 AM, Thomas Gleixner <[email protected]> wrote:
>
> "Kenneth R. Crudup" <[email protected]> writes:
>
>> On Thu, 2 Apr 2020, Thomas Gleixner wrote:
>>
>>> As Peter and myself don't have access to a SLD enabled machine, the
>>> KVM/VMX part is untested. The module scan part works.
>>
>> I just applied both of these patches to my (Linus' tip) tree, and unfortunately
>> VMWare still hangs if split_lock_detect= is set to anything but "off".
>>
>> Was there anything else I'd needed to do?
>
> Hmm. Not really. Does dmesg show the warning when the VMWare module loads?
> Something like:
>
> x86/split lock detection: disabled due to VMLAUNCH in module: ….
I ran an objdump on VMware workstation and indeed I do not see a
VMLAUNCH/VMRESUME. I do see however VMXON which should also be good for
detecting hypervisors. I will try to understand why VMLAUNCH is not there.
"Kenneth R. Crudup" <[email protected]> writes:
> On Thu, 2 Apr 2020, Thomas Gleixner wrote:
>
>> As Peter and myself don't have access to a SLD enabled machine, the
>> KVM/VMX part is untested. The module scan part works.
>
> I just applied both of these patches to my (Linus' tip) tree, and unfortunately
> VMWare still hangs if split_lock_detect= is set to anything but "off".
>
> Was there anything else I'd needed to do?
Hmm. Not really. Does dmesg show the warning when the VMWare module loads?
Something like:
x86/split lock detection: disabled due to VMLAUNCH in module: ....
Thanks,
tglx
On Thu, 2 Apr 2020, Peter Zijlstra wrote:
> Hmm, you're not seeing this:
> + pr_warn("disabled due to VMLAUNCH in module: %s\n", me->name);
> fire when you load the vmware kernel module?
I just looked back at the syslog's copy of the kernel messages at the time
I'd tried it, and no, I don't see that message.
> Could you slip me a copy of that thing by private mail?
OK, gimme a couple of days though, I've gotta get a little work done.
(Also, what "thing" exactly did you want?)
-Kenny
--
Kenneth R. Crudup Sr. SW Engineer, Scott County Consulting, Silicon Valley
On Thu, Apr 02, 2020 at 07:41:41AM -0700, Kenneth R. Crudup wrote:
>
> On Thu, 2 Apr 2020, Peter Zijlstra wrote:
>
> > Hmm, you're not seeing this:
> > + pr_warn("disabled due to VMLAUNCH in module: %s\n", me->name);
> > fire when you load the vmware kernel module?
>
> I just looked back at the syslog's copy of the kernel messages at the time
> I'd tried it, and no, I don't see that message.
Dang!
> > Could you slip me a copy of that thing by private mail?
>
> OK, gimme a couple of days though, I've gotta get a little work done.
> (Also, what "thing" exactly did you want?)
All the .ko files that come with vmware. I want to dig through them to
see why the VMLAUNCH detection thing isn't working.
> > (Also, what "thing" exactly did you want?)
On Thu, 2 Apr 2020, Peter Zijlstra wrote:
> All the .ko files that come with vmware.
Ah, gotcha. One thing you/VMWare may want to look at is it turns out that
"vmw_vmci", part of the kernel tree (CONFIG_VMWARE_VMCI) has to be compiled
into the kernel as a module so udev can properly create the Misc device node
(I'd tried making it a built-in and messing with udev rules and the other,
compiled-in-later VMWare module loading system didn't seem to like that).
Maybe some sort of mitigation for this can be done there, putting it back in-tree?
But anyway, I'll send you a .tar.bz2 in a little bit.
-Kenny
--
Kenneth R. Crudup Sr. SW Engineer, Scott County Consulting, Silicon Valley
On Thu, Apr 02, 2020 at 02:47:33PM +0000, Nadav Amit wrote:
> > On Apr 2, 2020, at 7:37 AM, Thomas Gleixner <[email protected]> wrote:
> >
> > "Kenneth R. Crudup" <[email protected]> writes:
> >
> >> On Thu, 2 Apr 2020, Thomas Gleixner wrote:
> >>
> >>> As Peter and myself don't have access to a SLD enabled machine, the
> >>> KVM/VMX part is untested. The module scan part works.
> >>
> >> I just applied both of these patches to my (Linus' tip) tree, and unfortunately
> >> VMWare still hangs if split_lock_detect= is set to anything but "off".
> >>
> >> Was there anything else I'd needed to do?
> >
> > Hmm. Not really. Does dmesg show the warning when the VMWare module loads?
> > Something like:
> >
> > x86/split lock detection: disabled due to VMLAUNCH in module: ….
>
> I ran an objdump on VMware workstation and indeed I do not see a
> VMLAUNCH/VMRESUME. I do see however VMXON which should also be good for
> detecting hypervisors. I will try to understand why VMLAUNCH is not there.
Let me send a version with VMXON detection added in as well.