Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Mon, 29 Jan 2018 12:31:13 -0500
From:   Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To:     David Woodhouse <dwmw2@infradead.org>, daniel.kiper@oracle.com,
        Mihai Carabas <mihai.carabas@oracle.com>
Cc:     Liran Alon <liran.alon@oracle.com>, luto@kernel.org,
        tglx@linutronix.de, torvalds@linux-foundation.org,
        gregkh@linuxfoundation.org, asit.k.mallick@intel.com,
        dave.hansen@intel.com, karahmed@amazon.de, jun.nakajima@intel.com,
        dan.j.williams@intel.com, ashok.raj@intel.com,
        daniel.kiper@oracle.com, arjan.van.de.ven@intel.com,
        tim.c.chen@linux.intel.com, pbonzini@redhat.com,
        linux-kernel@vger.kernel.org, ak@linux.intel.com,
        kvm@vger.kernel.org, aarcange@redhat.com
Subject: Re: [PATCH] x86: vmx: Allow direct access to MSR_IA32_SPEC_CTRL
Message-ID: <20180129173113.GO22045@char.us.oracle.com>
References: <6b9a1ec2-5ebd-4624-a825-3f31db5cefb5@default>
 <1517215563.6624.118.camel@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1517215563.6624.118.camel@infradead.org>
User-Agent: Mutt/1.8.3 (2017-05-23)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Jan 29, 2018 at 08:46:03AM +0000, David Woodhouse wrote:
> On Sun, 2018-01-28 at 16:39 -0800, Liran Alon wrote:
> > 
> > Windows use IBRS and Microsoft don't have any plans to switch to retpoline.
> > Running a Windows guest should be a pretty common use-case no?
> > 
> > In addition, your handle of the first WRMSR intercept could be different.
> > It could signal you to start doing the following:
> > 1. Disable intercept on SPEC_CTRL MSR.
> > 2. On VMEntry, Write vCPU SPEC_CTRL value into physical MSR.
> > 3. On VMExit, read physical MSR into vCPU SPEC_CTRL value.
> > (And if IBRS is used at host, also set physical SPEC_CTRL MSR here to 1)
> > 
> > That way, you will both have fastest option as long as guest don't use IBRS
> > and also won't have the 3% performance hit compared to Konrad's proposal.
> > 
> > Am I missing something?
> 
> Reads from the SPEC_CTRL MSR are strangely slow. I suspect a large part
> of the 3% speedup you observe is because in the above, the vmentry path
> doesn't need to *read* the host's value and store it; the host is
> expected to restore it for itself anyway?

Yes for at least the purpose of correctness. That is based on what I have
heard is that you when you transition to a higher ring you have to write 1, then write zero
when you transition back to lower rings. That is it is like a knob.

But then I heard that on some CPUs it is more like reset button and
just writting 1 to IBRS is fine. But again, correctness here.

> 
> I'd actually quite like to repeat the benchmark on the new fixed
> microcode, if anyone has it yet, to see if that read/swap slowness is
> still quite as excessive. I'm certainly not ruling this out, but I'm
> just a little wary of premature optimisation, and I'd like to make sure
> we have everything *else* in the KVM patches right first.
> 
> The fact that the save-and-restrict macros I have in the tip of my
> working tree at the moment are horrid and causing 0-day nastygrams,
> probably doesn't help persuade me to favour the approach ;)
> 
> ... hm, the CPU actually has separate MSR save/restore lists for
> entry/exit, doesn't it? Is there any way to sanely make use of that and
> do the restoration manually on vmentry but let it be automatic on
> vmexit, by having it *only* in the guest's MSR-store area to be saved
> on exit and restored on exit, but *not* in the host's MSR-store area?

Oh. That sounds sounds interesting
> 
> Reading the code and comparing with the SDM, I can't see where we're
> ever setting VM_EXIT_MSR_STORE_{ADDR,COUNT} except in the nested
> case...

Right. We (well Daniel Kiper, CC-ed) implemented it for this and
that is where we got the numbers.

Daniel, you OK posting it here? Granted with the caveats thta it won't even
compile against upstream as it was based on a distro kernel.