Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754226AbeAHRbv (ORCPT + 1 other); Mon, 8 Jan 2018 12:31:51 -0500 Received: from mail-by2nam03on0074.outbound.protection.outlook.com ([104.47.42.74]:51277 "EHLO NAM03-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754006AbeAHRbo (ORCPT ); Mon, 8 Jan 2018 12:31:44 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; Subject: Re: [tip:x86/pti] x86/cpu/AMD: Use LFENCE_RDTSC instead of MFENCE_RDTSC To: Thomas Gleixner Cc: Andrew Cooper , bp@alien8.de, dwmw@amazon.co.uk, gregkh@linux-foundation.org, pjt@google.com, mingo@kernel.org, linux-kernel@vger.kernel.org, hpa@zytor.com, tim.c.chen@linux.intel.com, torvalds@linux-foundation.org, peterz@infradead.org, dave.hansen@intel.com, linux-tip-commits@vger.kernel.org References: <20180105160756.23786.4220.stgit@tlendack-t1.amdoffice.net> <1b179b8b-6cc8-d736-81dc-2445be4baf02@citrix.com> <4d9a1f0d-a401-3f0f-9ee2-dd42f4b4716a@amd.com> From: Tom Lendacky Message-ID: Date: Mon, 8 Jan 2018 11:31:34 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Originating-IP: [165.204.78.1] X-ClientProxiedBy: MWHPR18CA0048.namprd18.prod.outlook.com (10.175.9.162) To DM5PR12MB1148.namprd12.prod.outlook.com (10.168.236.143) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 95714ba0-14cf-4038-9fa9-08d556bdae41 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(48565401081)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(5600026)(4604075)(2017052603307)(7153060)(7193020);SRVR:DM5PR12MB1148; X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1148;3:otoZY2UD3xQ6DWpCDbOzqbhrM2g/DH1NcLRQQNOHhUqigZ+qyUfvtN92/lUWvKKfZl01yv+g6kflvmY4S9XIoqAFVR1R8Jflp1uTbAmbBHUEei3GEorpkU+CpXIAtehQqYFTFxzVuodV+NHcP3WH269rqXBiSQTI/r2cQkgoN/iOT0PDEK5uO9eHsGIbVP8tp+2sZcI1FtzCZ22tvy7iGvaE/bxQqhbcmE9WNqvUlx6hYKm2B6PpG1iCQFnn+sOk;25:4WxHhT06RRdL1V4dHZVdJ66T85vWeJel1vPkVDyLl447b8POpKKEUvQD5mbNTV+71nyG0tbUsMzdWYEtNRvkm6SsPBgpXkWJYNYbyMS62j0kKIwzK0/FbAth+drRO0hQcnHEMySJ6OLTwxc7S1qpOquJtGA5uaJPvdTet58B6SiM+A7OdzD5nP/w1rVIna8JRdQRuoadvjZKFuyYzvSmaXT1PptMG4lIyNc/wMV+e4ZQR0qh6F2yuuG2/U5QkkvcTo2rSicAZgnlkJyvtYPMHuyt1pUDdE1xO4gL11W287/0ggwDnzIsig0NIODWERV/jZ9nmpWaW0S31JbhsoHEqA==;31:YAn/nr0iiLiAC4ybL2jDMGjJi7j4x1d4TItHpViO0ml+OE/2Y/PDRVvwbsAgrVySXeLGjsD/Vf1tiu9Wm6NJNnoGFicIdablV3xjBMeBjG+rhF2U6ISngUu9ozkbz7I9sUr0sJcFd78Yh5RNhHpm2w2K52Jc7pFlAz3b4z0F02dx7D+hWFwjnZ4155iPyFlV187yIk5ADV2IJEAx4Oy/ErE1jqtuH+Fs+15HgF1n+3s= X-MS-TrafficTypeDiagnostic: DM5PR12MB1148: X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1148;20:5CIGIOGVmHwC3vXkifxWJ9Ti44ZIpeEL5eZ+xCm2Tf4P9bJgnACFKClobeMWKxjWMRyjb/DruS86BLK7IMt5cdgC6pPswAOCCMu8wQoRV88Afs0T8LGjaUVAPhe5y4d81/phTt3FCMfIwpt4w7+6ZPrdIjyx5lyDRL47BNjxn+V2W5QZv6IT7ez0Xxnl+iqenuVUPZdXNkZMI/trcFlgQqPWFC4fSAk+Dzc6anc7UwLRQolTPzLSgSBHH23Tv+xtDMBLIpFg5ZwMRNDCe/8MLUD0kISEQAVCgBwTgWnzIjOrYCaYUixXTQl9X0FiN1dvj0Jrl+ju2Q6ZIZDgoKzIz8PrsZbfKcdvyoaNX1IcaMEJFp+j680PgZsd+g389WaFg8Z7H03t6V3PnXkMTVbYUC0HiyT080SXSVDMgWnXY+YDDIgpLlYA2S/wE+ADmRUsavG/Eo6vVDf3AHLSylNhNh+7UFdFRibaJWt/d7LMbtZGQ428qFlx0X2sWGeYmI4s;4:DQY15F20UQbQhkVpMcENaJehG9jyWlPK/w/mXI7f/XgMjMDLuM1AyvHjrVoIPxoshg9JBqnBYt7vncY87R3wIvI2J+bimh58yDYyPxTHej2a7os4cSJY5tfI3BDAmYb7TXICXSylah/s41brRit/wTysBjaNGZHKNwlDwoWWdwfrQCbzwGeqKiU9M1FqtNdqlM9UoaD9UWUo0JdyYSEgwAv+T/EI55QzOlz82EzwbuepFouiIiPmsSYgg2WVrzqC5e36OC3+RrIOBp8aPnc2vQ== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040470)(2401047)(8121501046)(5005006)(93006095)(93001095)(3231023)(944501075)(3002001)(10201501046)(6055026)(6041268)(20161123562045)(20161123558120)(20161123564045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011);SRVR:DM5PR12MB1148;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:DM5PR12MB1148; X-Forefront-PRVS: 054642504A X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6049001)(376002)(366004)(39860400002)(346002)(39380400002)(396003)(189003)(199004)(24454002)(86362001)(68736007)(76176011)(52146003)(65826007)(2950100002)(6916009)(6666003)(575784001)(31686004)(8936002)(8666007)(31696002)(105586002)(478600001)(316002)(50466002)(4326008)(2870700001)(52116002)(53546011)(23676004)(386003)(2906002)(59450400001)(2486003)(53936002)(5660300001)(64126003)(7416002)(7736002)(25786009)(3260700006)(229853002)(16526018)(16576012)(90366009)(305945005)(58126008)(3846002)(72206003)(6116002)(6246003)(83506002)(81156014)(47776003)(36756003)(6486002)(77096006)(65956001)(66066001)(97736004)(65806001)(93886005)(8676002)(81166006)(106356001);DIR:OUT;SFP:1101;SCL:1;SRVR:DM5PR12MB1148;H:[10.236.65.116];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTVQUjEyTUIxMTQ4OzIzOkJ2S2lXTzZWQ2tHOHdvMUdrWGJwb2NzTFVK?= =?utf-8?B?T2VqaFJvYlFHMGtrQzRHdjFHZkxlUTFoUnhwM3dDZ0ZDMzZ2U2NBeklyVk9B?= =?utf-8?B?N0JtbURpOWNJWjRTL0xKUy9BMmU2MG0rTW1CbjdxOFJlVGdmUGFBZGNBblcx?= =?utf-8?B?UkpXaVZOdjlpYmgrODRUUTZ1SHVyUnNMYkJNZEtGaFJ2SXpCRlhWSkxtUjYz?= =?utf-8?B?a05wUWlDN1YrWVJsSUpKb29FMDRkODRyYXE1eG5KNnAwK1BLZnlpUTJXWklM?= =?utf-8?B?ODRwazFXL1ZIQXVQdHZCc0xZbW1pcnBnRGQ1L3JJK09DVFBqekNkYk5uZTBw?= =?utf-8?B?Z05lSktnMW11cFdlZkJocGtMOWVFTW44dXF1M2Fhc1QzRDRWbUo0TzRzNEZX?= =?utf-8?B?a0dVSmVHdm1mZ1QwSklDa2dCU1dvQ0E0Tnk0ZVpMcnlaUkZXNFZUTHFaOVVo?= =?utf-8?B?MWc0UXNINFBkQS9HQ2J4R0FhNGkzZ1NvUzZVNHZyUk5OM1JHRmt0eU1tRGZs?= =?utf-8?B?QUxuUS9tK0d6UFRrS3YzdkRUbXNVaWwwTHhuT1FvTUFxUjkyYThmMktXTUw2?= =?utf-8?B?Rnc3S2pLZHNlS2YvUHdaWU1QVkRoVyt3a0V5SDE3NVdaa1RuN1ZpYU0rWFkz?= =?utf-8?B?RmRwcWFwblV0WlUzeHlOTmx4d0t6SGxHdXNTN1JYeWNLRUJncE4vbVNLb1R3?= =?utf-8?B?N1ZwOW9BWFBaWHF4cFBMQ1lZdjd0bUJXcU5iN0RlRUFaMTgwdCs2Q25DbUtt?= =?utf-8?B?bTc5UjZZOHlUSHhSajVMUDJVN1dDVXRGVTNDTnNFaWVuU3RlUUc2ZTJpamN0?= =?utf-8?B?c3V1N0d4b1E4ZU1jSGZReXVMWmZLODN5eWorNzBvcUZodjRRenpjSTg0Zklx?= =?utf-8?B?dnZTSnFiSUVBMDNkaFJXZFJpbnJVbFhHWlFDQU10Y3pTZ1dRM01YalZWZ25E?= =?utf-8?B?SDZXU2RFamt0bjBtUU9ZazAxRmR6eFF6VkUzMWtEMHJoc2xWVkIrcnZtNWxJ?= =?utf-8?B?RGFQRnFSSU1uV0dFRUZ4YkVxTUpuMDlrbXQvZkxhOVdhTU5ac2xBQ0dJQ2hy?= =?utf-8?B?SzJHTUVrMVZoMlNZM0RkdWpsUUwrRGUzVE5nd2x4L2hUQmliOHBZRkNLV3B4?= =?utf-8?B?cjZrSm45RzQwTWNqTFIwMStzSG1vQjZWeE81aGFyN1hvOEtzWVBVN1JBaDV2?= =?utf-8?B?TjE5UWo3SXF6QVNlZzFrMElHU0ZXbU5ZTDNROFZ0WFBNcFIrbzdWVzNzeDNQ?= =?utf-8?B?WFN0UUhUbHovcVorUFhTWm50WXBXb0VYdDZxc29ZSVlyRzRUWTBIbFhYN2pY?= =?utf-8?B?NTBTbGxlYjVLL2VjQzN1MHZUQ1JtQmpsMDE4U2JPNmx6cFNFZklSTEc5TVRZ?= =?utf-8?B?eERTYTE0eWU0Smh5MTUvL2JFVmpCUk5uYmxWNXQzTlNVV0tpMGduUUFyNU95?= =?utf-8?B?bUNFQVJQQlg0bTd1OE9YcXY1akhHN055QWVtZU40NHk2V2hEN0hjUFh6Z2t0?= =?utf-8?B?TzY0aWFmdGs5OUo5VDV2bnRSMlBncGxEalBmTWNLSGwvTEZZK1cwOU5GZ1py?= =?utf-8?B?NTJHT0xEUlFQeVY4citjVDhtUlhuYXhVMWx2Rlg5aEFOV0hGMUVkcWxHN0hX?= =?utf-8?B?WTJVZEF0cEZIenpOeVBqR3ozVXBZRnJrclJPb1h2MC9jeUQwV3U0VzhBUDU1?= =?utf-8?B?UW0rSmUxRTVJclZhVm1lbUJFZXJxVmsrRFg4RjlRTU40M0pBR1ZhNFp3bmdH?= =?utf-8?B?VnVQU3hBUW1yUEI0cU9ZcTZURHFYY0tYVDEvMzR0L1FxTEhaL2JWYllqQTMy?= =?utf-8?B?OEZPVzgwWWxOUy9kcUhYb3VRYmVVa1JKWHpqa3NWV29Zc2loK3FHS0FuTUU2?= =?utf-8?B?M0dQM01TUW5rSDFxNnRISENQYVhNdTZ3ckhCU2NuWFZKR2dqYURkMzROdlo0?= =?utf-8?B?REZzNlBjc29iblVndGRMREtqL1VnTTlMaXlmWmlyMVNjNDNNY2EzcmdxTGFt?= =?utf-8?B?cGlKZTRuaitsQ0hqRVNMb1N4anpkaHd5MzZ2T3Q4SURLZHRQSzhzWVltZ2E0?= =?utf-8?Q?6c4bQCDH/OepCb2PCJOKItNZB?= X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1148;6:07x4bltnRd7SlQGNpvnMh7g4j7wCIw61eugewnPVJNl/G5vdAyoUNyT6V+zUkdEyzYtgiQG6yspDJ+w3SA9vvafJt7XeZNIjkksWxc6WlI+dEvG09xN4pbwKkLovLqm8TTfTl0gGS4MDib5rPn+fKi07ZYgl5hoiK2TYviiyTtqhEuuU990xUlIAnZT/AW6UWT+3a5b6EKsmOxYncseMZfHQ0x1exm7PhIOl6Ww0fsw3xFkMuH/SsEdOic9ripxKnga5LvNOxQMj4xAKH1khfayUTTom8q26UHuqxiA4nJEYMJScx+C4H2z5hyzQGxTp5WWkD3mxMqvEYrVbVesnE94TBDDKALq7d5QeECHXf5U=;5:H5PRChE1flAImiAxYpjZt/hVvliLK8tgFaidn78/YgJCvUNcrNC0MF/Y0UxC4QBhk41UFVNQfsanYVKAZ6YuPtgbPlQr39nsnQP5pxxa9cPNZ+iLQOQtxNjHdFyTSnM6H8lTWegYTJAqQZttQr96l36l9E0JeshppLH8AkDmzKE=;24:mrrOd+0yse7th9Wnq0VHI0fttV4xHq9N3N3obQhhAdDfd32LAo6FouMWBK5M6c9RJwF9HjrjKnSbis/gvb0JmeESwr/8y5bY/pKZBVa8bNQ=;7:+bqGc4Dk/lMJnclAy+v3ww3gqCgF4qYa7Ljdooh76LTXEKwRWtmHN99Mhv9+eBEaTHqaDlFxP1tjNTMphkLmvk3OMK3TRqrSKH9OzKUQss6/9KUCUEWtdCAvl1r0UM6VALUTcnaMB0KDx3LhQZcd3m5j81jAgBGE2GD9Mz8qMjMesEW97iNhm6tefPN76Y/iQhy+ihg99Lem1lvT+o+NhUueiHMR8EmdrMXrLgIbk7QhrYVmGguIA/I4rw9MIe1g SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1148;20:ykbt21EKSM/gkCYO0NsOn+NWGQhdvE1JYR41sp2j7QT02ysvgS6C/Ky483cZs6e2fyEYD8HYTUe5jv2M/jSPViZe3wE22N1GKJFYxV5ab7iOuIJvhabloYK7whfqGqDPbYsu5zAW1jRrOghQmA4qfkFkKS9YKoYrrM31Rcz0tJBCWnrbZNJNTkH1wsTC5m7hxkPJjekQ+WzNbcZ5u3VO3hRigEjQs7CkyvFUnLDtaH47u65z5MB7FooidYiAWIWz X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jan 2018 17:31:39.7324 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 95714ba0-14cf-4038-9fa9-08d556bdae41 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1148 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Return-Path: On 1/8/2018 9:15 AM, Thomas Gleixner wrote: > On Mon, 8 Jan 2018, Tom Lendacky wrote: >> On 1/8/2018 5:10 AM, Thomas Gleixner wrote: >>>>> Second thoughts on that. As pointed out by someone in one of the insane >>>>> long threads: >>>>> >>>>> What happens if the kernel runs as a guest and >>>>> >>>>> - the hypervisor did not set the LFENCE to serializing on the host >>>>> >>>>> - the hypervisor does not allow writing MSR_AMD64_DE_CFG >>>>> >>>>> That would bring the guest into a pretty bad state or am I missing >>>>> something essential here? >>>> >>>> What I did in Xen was to attempt to set it, then read it back and see.  >>>> If LFENCE still isn't serialising, using repoline is the only available >>>> mitigation. >>>> >>>> My understanding from the folk at AMD is that retpoline is safe to use, >>>> but has higher overhead than the LFENCE approach. >> >> Correct, the retpoline will work, it just takes more cycles. >> >>> >>> That still does not help vs. rdtsc_ordered() and LFENCE_RDTSC ... >> >> Ok, I can add the read-back check before setting the feature flag(s). >> >> But... what about the case where the guest is a different family than >> hypervisor? If we're on, say, a Fam15h hypervisor but the guest is started >> as a Fam0fh guest where the MSR doesn't exist and LFENCE is supposed to be >> serialized? I'll have to do a rdmsr_safe() and only set the flag(s) if I >> can successfully read the MSR back and validate the bit. > > But that still does not make this patch correct and neither the next one. > > If you cannot set the flag and you cannot prove that you run on bare metal, > then you still need the MFENCE_RDTSC variant as you have no idea what the > underlying hypervisor is and how it has the LFENCE configured. So now I'm also concerned about setting the retpoline method and using LFENCE as the speculation barrier. If we go back to the original statement: - the hypervisor did not set the LFENCE to serializing on the host - the hypervisor does not allow writing MSR_AMD64_DE_CFG It looks like I'll need to attempt to write MSR_AMD64_DE_CFG and then read it back checking for whether MSR_F10H_DECFG_LFENCE_SERIALIZE has been set. If the MSR can be read and the bit is set, then I can set RETPOLINE_AMD (once those patches go in) and LFENCE_RDTSC. Otherwise, set (only) MFENCE_RDTSC. This also means we need to use MFENCE as the speculation barrier instead of LFENCE in this situation (another alternative added to the __nospec_barrier() implementation). Does that sound right? Or does the "cannot prove that you run on bare metal" mess this all up? Thanks, Tom > > Thanks, > > tglx >