Subject: Re: [tip:x86/pti] x86/cpu/AMD: Use LFENCE_RDTSC instead of
 MFENCE_RDTSC
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>, bp@alien8.de,
        dwmw@amazon.co.uk, gregkh@linux-foundation.org, pjt@google.com,
        mingo@kernel.org, linux-kernel@vger.kernel.org, hpa@zytor.com,
        tim.c.chen@linux.intel.com, torvalds@linux-foundation.org,
        peterz@infradead.org, dave.hansen@intel.com,
        linux-tip-commits@vger.kernel.org
References: <20180105160756.23786.4220.stgit@tlendack-t1.amdoffice.net>
 <tip-0bf17c102177d5da9363bf8b1e4704b9996d5079@git.kernel.org>
 <alpine.DEB.2.20.1801081105120.1735@nanos>
 <1b179b8b-6cc8-d736-81dc-2445be4baf02@citrix.com>
 <alpine.DEB.2.20.1801081208500.1735@nanos>
 <4d9a1f0d-a401-3f0f-9ee2-dd42f4b4716a@amd.com>
 <alpine.DEB.2.20.1801081611180.1735@nanos>
From: Tom Lendacky <thomas.lendacky@amd.com>
Message-ID: <b950e80b-e3a7-3395-8f22-74adbe91965d@amd.com>
Date: Mon, 8 Jan 2018 11:31:34 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.5.2
MIME-Version: 1.0
In-Reply-To: <alpine.DEB.2.20.1801081611180.1735@nanos>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jan 2018 17:31:39.7324 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 95714ba0-14cf-4038-9fa9-08d556bdae41
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1148
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Return-Path: <linux-kernel-owner@vger.kernel.org>

On 1/8/2018 9:15 AM, Thomas Gleixner wrote:
> On Mon, 8 Jan 2018, Tom Lendacky wrote:
>> On 1/8/2018 5:10 AM, Thomas Gleixner wrote:
>>>>> Second thoughts on that. As pointed out by someone in one of the insane
>>>>> long threads:
>>>>>
>>>>> What happens if the kernel runs as a guest and
>>>>>
>>>>>   - the hypervisor did not set the LFENCE to serializing on the host
>>>>>
>>>>>   - the hypervisor does not allow writing MSR_AMD64_DE_CFG
>>>>>
>>>>> That would bring the guest into a pretty bad state or am I missing
>>>>> something essential here?
>>>>
>>>> What I did in Xen was to attempt to set it, then read it back and see. 
>>>> If LFENCE still isn't serialising, using repoline is the only available
>>>> mitigation.
>>>>
>>>> My understanding from the folk at AMD is that retpoline is safe to use,
>>>> but has higher overhead than the LFENCE approach.
>>
>> Correct, the retpoline will work, it just takes more cycles.
>>
>>>
>>> That still does not help vs. rdtsc_ordered() and LFENCE_RDTSC ...
>>
>> Ok, I can add the read-back check before setting the feature flag(s).
>>
>> But... what about the case where the guest is a different family than
>> hypervisor? If we're on, say, a Fam15h hypervisor but the guest is started
>> as a Fam0fh guest where the MSR doesn't exist and LFENCE is supposed to be
>> serialized?  I'll have to do a rdmsr_safe() and only set the flag(s) if I
>> can successfully read the MSR back and validate the bit.
> 
> But that still does not make this patch correct and neither the next one.
> 
> If you cannot set the flag and you cannot prove that you run on bare metal,
> then you still need the MFENCE_RDTSC variant as you have no idea what the
> underlying hypervisor is and how it has the LFENCE configured.

So now I'm also concerned about setting the retpoline method and using
LFENCE as the speculation barrier.  If we go back to the original
statement:

  - the hypervisor did not set the LFENCE to serializing on the host
  - the hypervisor does not allow writing MSR_AMD64_DE_CFG

It looks like I'll need to attempt to write MSR_AMD64_DE_CFG and then read
it back checking for whether MSR_F10H_DECFG_LFENCE_SERIALIZE has been set.
If the MSR can be read and the bit is set, then I can set RETPOLINE_AMD
(once those patches go in) and LFENCE_RDTSC.  Otherwise, set (only)
MFENCE_RDTSC.  This also means we need to use MFENCE as the speculation
barrier instead of LFENCE in this situation (another alternative added to
the __nospec_barrier() implementation).

Does that sound right?  Or does the "cannot prove that you run on bare
metal" mess this all up?

Thanks,
Tom

> 
> Thanks,
> 
> 	tglx
>