Received: by 2002:a25:1104:0:0:0:0:0 with SMTP id 4csp121960ybr; Fri, 22 May 2020 02:34:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyVF7gVQ0mWolaWMFHYPkLViM6/OVm9ViyamGvmN1KZO/anv5817LbkC3U2A9hbLx48DeMM X-Received: by 2002:a50:8e56:: with SMTP id 22mr2135508edx.268.1590140095781; Fri, 22 May 2020 02:34:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1590140095; cv=none; d=google.com; s=arc-20160816; b=oQn7mVP2U5imwopKOXMnS1I+4waw4/vieRpI3ftbXaV5M0s2s8vzUu4+kstc6HCSfT NlhYe+C5z2pWBc0QveWOts1pc/4yyaECJTXXAZylFsQ1P3R0uEYHxeheSb5dovaa/2KZ CPMOR76cg20VSRMUkd9AgKJKfY6y8f0nE8fyfnaQ7kIe/4UbQnaedYZw4cfM7ibtvtPc tsxdh1oLJajR25AtYQy4ySE5tPHJ8PxXYxbFC7vNewnSBarij9WRltQP8WgMSo7PKSpu qTrO1KP3Rxn84VY5efQFWj22v8baopKKBTjTXbZ9QARjnGMmOi9+k3sZ2Y952NgqVLd+ xhVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :robot-unsubscribe:robot-id:message-id:mime-version:references :in-reply-to:cc:subject:to:reply-to:from:date; bh=rGz8DPrDIsWy/j8zRTJgBBZnANVkoqnF76Dr4ZbFTIs=; b=ZWkREcvgtEvtJMKs/Kxx+J8VlGyJG1mKKplfDuw9TFR4/GYv111F0Ps9L35miytkXY Co9YZdPuFJ01HByxlTxXwd/w9MTqvdDHVCFGGo7TaALERgp9bPl+8IGd63ZCSI1bxmHr sekM9eXQNPGguPWete6X5D2U88NzamX33j3f6e9VknVICsDOpyjm0UsLCTlaZhHIu2pA Z0UDIDhVAJkojnv3Od5aXp1h7gsAseNPrNkmhKiXjv7KE2BF/sYotAQGp5nPe263mbBS gRVUQqpc3qTHeQL3jHZ7FJpLi8i5sNDHjMipKBY1N0Zo5Ul7ZWwUqtWPhur3EhtsP/n9 uZBg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id n15si4490185eda.19.2020.05.22.02.34.31; Fri, 22 May 2020 02:34:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729751AbgEVJdE (ORCPT + 99 others); Fri, 22 May 2020 05:33:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729619AbgEVJdC (ORCPT ); Fri, 22 May 2020 05:33:02 -0400 Received: from Galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34D90C05BD43; Fri, 22 May 2020 02:33:01 -0700 (PDT) Received: from [5.158.153.53] (helo=tip-bot2.lab.linutronix.de) by Galois.linutronix.de with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1jc42z-0001Pq-BR; Fri, 22 May 2020 11:32:57 +0200 Received: from [127.0.1.1] (localhost [IPv6:::1]) by tip-bot2.lab.linutronix.de (Postfix) with ESMTP id C33151C0475; Fri, 22 May 2020 11:32:56 +0200 (CEST) Date: Fri, 22 May 2020 09:32:56 -0000 From: "tip-bot2 for Balbir Singh" Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/mm] x86/kvm: Refactor L1D flush operations Cc: Balbir Singh , Thomas Gleixner , Kees Cook , x86 , LKML In-Reply-To: <20200510014803.12190-3-sblbir@amazon.com> References: <20200510014803.12190-3-sblbir@amazon.com> MIME-Version: 1.0 Message-ID: <159013997669.17951.10091686973701417401.tip-bot2@tip-bot2> X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Linutronix-Spam-Score: -1.0 X-Linutronix-Spam-Level: - X-Linutronix-Spam-Status: No , -1.0 points, 5.0 required, ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The following commit has been merged into the x86/mm branch of tip: Commit-ID: e3efae20ec69e9a8c9db1ad81b37de629219bbc4 Gitweb: https://git.kernel.org/tip/e3efae20ec69e9a8c9db1ad81b37de629219bbc4 Author: Balbir Singh AuthorDate: Sun, 10 May 2020 11:47:59 +10:00 Committer: Thomas Gleixner CommitterDate: Wed, 13 May 2020 18:12:19 +02:00 x86/kvm: Refactor L1D flush operations Move the L1D flush functions into builtin code so they can be reused for L1D flush on context switch. Split them up into: - Hardware L1D flush - TLB pre-populating of L1D pages for software based flushing - Software based L1D flush Adjust the KVM code accordingly. [ tglx: Massaged changelog ] Signed-off-by: Balbir Singh Signed-off-by: Thomas Gleixner Reviewed-by: Kees Cook Link: https://lkml.kernel.org/r/20200510014803.12190-3-sblbir@amazon.com --- arch/x86/include/asm/cacheflush.h | 3 ++- arch/x86/kernel/l1d_flush.c | 49 ++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/vmx.c | 29 +----------------- 3 files changed, 55 insertions(+), 26 deletions(-) diff --git a/arch/x86/include/asm/cacheflush.h b/arch/x86/include/asm/cacheflush.h index bac56fc..21cc3b2 100644 --- a/arch/x86/include/asm/cacheflush.h +++ b/arch/x86/include/asm/cacheflush.h @@ -8,7 +8,10 @@ #define L1D_CACHE_ORDER 4 void clflush_cache_range(void *addr, unsigned int size); +void l1d_flush_populate_tlb(void *l1d_flush_pages); void *l1d_flush_alloc_pages(void); void l1d_flush_cleanup_pages(void *l1d_flush_pages); +void l1d_flush_sw(void *l1d_flush_pages); +int l1d_flush_hw(void); #endif /* _ASM_X86_CACHEFLUSH_H */ diff --git a/arch/x86/kernel/l1d_flush.c b/arch/x86/kernel/l1d_flush.c index 4f298b7..32119ee 100644 --- a/arch/x86/kernel/l1d_flush.c +++ b/arch/x86/kernel/l1d_flush.c @@ -37,3 +37,52 @@ void l1d_flush_cleanup_pages(void *l1d_flush_pages) free_pages((unsigned long)l1d_flush_pages, L1D_CACHE_ORDER); } EXPORT_SYMBOL_GPL(l1d_flush_cleanup_pages); + +void l1d_flush_populate_tlb(void *l1d_flush_pages) +{ + int size = PAGE_SIZE << L1D_CACHE_ORDER; + + asm volatile( + /* First ensure the pages are in the TLB */ + "xorl %%eax, %%eax\n" + ".Lpopulate_tlb:\n\t" + "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" + "addl $4096, %%eax\n\t" + "cmpl %%eax, %[size]\n\t" + "jne .Lpopulate_tlb\n\t" + "xorl %%eax, %%eax\n\t" + "cpuid\n\t" + :: [flush_pages] "r" (l1d_flush_pages), + [size] "r" (size) + : "eax", "ebx", "ecx", "edx"); +} +EXPORT_SYMBOL_GPL(l1d_flush_populate_tlb); + +int l1d_flush_hw(void) +{ + if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { + wrmsrl(MSR_IA32_FLUSH_CMD, L1D_FLUSH); + return 0; + } + return -ENOTSUPP; +} +EXPORT_SYMBOL_GPL(l1d_flush_hw); + +void l1d_flush_sw(void *l1d_flush_pages) +{ + int size = PAGE_SIZE << L1D_CACHE_ORDER; + + asm volatile( + /* Fill the cache */ + "xorl %%eax, %%eax\n" + ".Lfill_cache:\n" + "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" + "addl $64, %%eax\n\t" + "cmpl %%eax, %[size]\n\t" + "jne .Lfill_cache\n\t" + "lfence\n" + :: [flush_pages] "r" (l1d_flush_pages), + [size] "r" (size) + : "eax", "ecx"); +} +EXPORT_SYMBOL_GPL(l1d_flush_sw); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 225aa82..786d161 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5983,8 +5983,6 @@ unexpected_vmexit: */ static void vmx_l1d_flush(struct kvm_vcpu *vcpu) { - int size = PAGE_SIZE << L1D_CACHE_ORDER; - /* * This code is only executed when the the flush mode is 'cond' or * 'always' @@ -6013,32 +6011,11 @@ static void vmx_l1d_flush(struct kvm_vcpu *vcpu) vcpu->stat.l1d_flush++; - if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { - wrmsrl(MSR_IA32_FLUSH_CMD, L1D_FLUSH); + if (!l1d_flush_hw()) return; - } - asm volatile( - /* First ensure the pages are in the TLB */ - "xorl %%eax, %%eax\n" - ".Lpopulate_tlb:\n\t" - "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" - "addl $4096, %%eax\n\t" - "cmpl %%eax, %[size]\n\t" - "jne .Lpopulate_tlb\n\t" - "xorl %%eax, %%eax\n\t" - "cpuid\n\t" - /* Now fill the cache */ - "xorl %%eax, %%eax\n" - ".Lfill_cache:\n" - "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" - "addl $64, %%eax\n\t" - "cmpl %%eax, %[size]\n\t" - "jne .Lfill_cache\n\t" - "lfence\n" - :: [flush_pages] "r" (vmx_l1d_flush_pages), - [size] "r" (size) - : "eax", "ebx", "ecx", "edx"); + l1d_flush_populate_tlb(vmx_l1d_flush_pages); + l1d_flush_sw(vmx_l1d_flush_pages); } static void update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr)