DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 546ED80F9B
Date: Wed, 15 Mar 2017 21:05:15 +0200
From: "Michael S. Tsirkin" <mst@redhat.com>
To: "Gabriel L. Somlo" <gsomlo@gmail.com>
Cc: Radim =?utf-8?B?S3LEjW3DocWZ?= <rkrcmar@redhat.com>,
        linux-kernel@vger.kernel.org, Paolo Bonzini <pbonzini@redhat.com>,
        Jonathan Corbet <corbet@lwn.net>, Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, "H. Peter Anvin" <hpa@zytor.com>,
        x86@kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org
Subject: Re: [PATCH] kvm: better MWAIT emulation for guests
Message-ID: <20170315210442-mutt-send-email-mst@kernel.org>
References: <1489098555-23856-1-git-send-email-mst@redhat.com>
 <20170313154618.GA4547@potion>
 <20170313180046-mutt-send-email-mst@kernel.org>
 <20170313193910.GB4547@potion>
 <20170313215016-mutt-send-email-mst@kernel.org>
 <20170313214355.GC4547@potion>
 <20170315181425.GA2239@HEDWIG.INI.CMU.EDU>
 <20170315202049-mutt-send-email-mst@kernel.org>
 <20170315190112.GB2239@HEDWIG.INI.CMU.EDU>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20170315190112.GB2239@HEDWIG.INI.CMU.EDU>
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3761
Lines: 83

On Wed, Mar 15, 2017 at 03:01:12PM -0400, Gabriel L. Somlo wrote:
> On Wed, Mar 15, 2017 at 08:29:23PM +0200, Michael S. Tsirkin wrote:
> > On Wed, Mar 15, 2017 at 02:14:26PM -0400, Gabriel L. Somlo wrote:
> > > Michael,
> > > 
> > > I tested this on OS X 10.7 (Lion), the last version that doesn't check
> > > CPUID for MWAIT support.
> > > 
> > > I used the latest kvm from git://git.kernel.org/pub/scm/virt/kvm/kvm.git
> > > first as-is, then with your v2 MWAIT patch applied.
> > > 
> > > Single-(V)CPU guest works as expected (but then again, single-vcpu
> > > guests worked even back when I tried emulating MWAIT the same as HLT).
> > > 
> > > When I try starting a SMP guest (with "-smp 4,cores=2"), the guest OS
> > > hangs after generating some output in text/verbose boot mode -- I gave
> > > up waiting for it after about 5 minutes. Works fine before your patch,
> > > which leads me to suspect that, as I feared, MWAIT doesn't wake
> > > immediately upon another VCPU writing to the MONITOR-ed memory location.
> > > 
> > > Tangentially, I remember back in the days of OS X 10.7, the
> > > alternative to exiting guest mode and emulating MWAIT and MONITOR as
> > > NOPs was to allow them both to run in guest mode.
> > > 
> > > While poorly documented by Intel at the time, MWAIT at L>0 effectively
> > > behaves as a NOP (i.e., doesn't actually put the physical core into
> > > low-power mode, because doing that would allow a guest to effectively
> > > DOS the host hardware).
> > 
> > Thanks for the testing, interesting.
> > Testing with Linux guest seems to show it works.
> > This could be an interrupt thing not a monitor thing.
> > Question: does your host CPU have this in its MWAIT leaf?
> > 	Bit 01: Supports treating interrupts as break-event for MWAIT, even when interrupts disabled
> 
> How would I check for this (I'm sorry, haven't hacked on any KVM
> related thing in a while, so I don't have it "cached") :)
> 
> > 
> > We really should check that before enabling,
> > I'll add that.
> > 
> > > 
> > > Given how unusual it is for a guest to use MONITOR/MWAIT in the first
> > > place, what's wrong with leaving it all as is (i.e., emulated as NOP)?
> > > 
> > 
> > I'm really looking into ways to use mwait within Linux guests,
> > this is just a building block that should help Mac OSX
> > as a side effect (and we do not want it broken if at all possible).
> 
> A few years ago I tried really emulating MONITOR and MWAIT for a
> project -- while not a total abject failure, the resulting patch
> worked only intermittently (on OS X 10.7, which was the hot new thing
> at the time, and hadn't started checking CPUID yet).
> 
> My collected wisdom on the topic from back then is here:
> 
>    http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/mwait.html
> 
> The problem is that MWAIT is required to wake synchronously with
> any other "thing" (either another (v)CPU, or DMA, or whatever) writing
> to the memory location "marked" by the last preceding MONITOR. While
> interrupts of any kind may also wake an MWAIT, it is strictly not allowed
> to "miss" a write to the MONITOR-ed memory location. So unless we implement
> some sort of condition queue that guarantees re-enabling the "parked" vcpu
> on an intercepted write to a specific memory location by another vcpu,
> we can't guarantee architecturally correct behavior.
> 
> If linux uses it in a very specific way that can be "faked" even
> without ISA compliance, that's OK with me -- but other guest OSs might
> take the x86 ISA more literally :)
> 
> Let me know if there's anything else you'd like me to test, now that I
> have set up a 4.11.0-rc2+ (a.k.a. kvm git master) testing rig...
> 
> Regards,
> --Gabe

I'm going to post a patch in a couple of minutes.

-- 
MST