Received: by 10.223.185.116 with SMTP id b49csp3213895wrg; Mon, 5 Mar 2018 16:42:31 -0800 (PST) X-Google-Smtp-Source: AG47ELtmWhv7W2IveHb6zHdsZuULD/o5GXivWfxUJnbcLMtyVU2MrP1zTCyuhKFszzGJqQHEAc/J X-Received: by 2002:a17:902:9307:: with SMTP id bc7-v6mr4815330plb.29.1520296951492; Mon, 05 Mar 2018 16:42:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520296951; cv=none; d=google.com; s=arc-20160816; b=OpI4Mq+RXW9EaUgC5IHFBIKqiktDOGCevdCjd42/UkGDZUjgo8Yy3JMwulq+RMxgZu smfxBN2bw4vo+ncuJ4VvrVYtNTcpY6cOAqrU8UoViqBfQwC3Ya/0yOaiYp02lJXHplk/ u+tKw6VQ+5fzSs0QRUU5Yh2VdR1wKMv/avIiZ0bOfCT3zou7TBJmrGAo0LmVpEM375LK fJsDIHZ3bJYTNws6Q0da6xp/kpCP5j/gItsUxpt1KvjJuqHthxZpE5pf2AWu4wQmUn/a dObXNfxJEaHxUklhua529GXyXQZo/EJ7lV41O0eYCUY2vI4zYwlO6v4zk9raduMCfEk6 KRkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:to:from:dkim-signature:arc-authentication-results; bh=bbbZNlQR1zRCJ8uxrvwncBKDWqm4wopjwMGTfoLTDHY=; b=a/C+LbhdYlz2Yg7mZ/TnRdB/juqOvO22Bw31JF95R9gfEoTCkS6Z/m/GBo8M5sUrdd 4TkHOQDQGPMyAf+U4jga68Qcq4ftBjRJuPPX712qya9Omj9rY8c5TIAee6oWQJeyMkb/ dtnE3cs6kAtL/7d1LEZVsx8PkWpcCgHE7mnwAotYN/3EmcWhRNtYSTU0hyhiBGUdoPD0 +raGE8cFQhGVHNhEsCYZJZhtzn6HPgLyBZMytJCV4wE4LGk5h3UGzYFPAsAWIlw5Wbt9 FvIwnWG4dp49QdV+I3+11z7owzSF409N/9XDgEVVCh9cRYgabT2C3Rt/jYTNXI+GD/oR NXwA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2017-10-26 header.b=FFtw3RGo; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x7si9050957pgo.122.2018.03.05.16.42.17; Mon, 05 Mar 2018 16:42:31 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2017-10-26 header.b=FFtw3RGo; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932906AbeCFA0I (ORCPT + 99 others); Mon, 5 Mar 2018 19:26:08 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:42564 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932830AbeCFA0F (ORCPT ); Mon, 5 Mar 2018 19:26:05 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w260LT6T073663; Tue, 6 Mar 2018 00:26:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2017-10-26; bh=bbbZNlQR1zRCJ8uxrvwncBKDWqm4wopjwMGTfoLTDHY=; b=FFtw3RGo2irUBgWn2othDLAuuTdUJVy6TPNjac38ykuXb/TvLQyLJCx5uTrD+03g+dA3 klir/Qukn5swEUWpeJ6T6oXkLWNFywwrAogcOAPkGd9aupKOjfS4FlbHBfORCFkmLh3/ TTml7pD2FRnwNO+i/TsgyYqqYO/Tmv/0vvbiehy7Gze7i9058hIJenk50KXg4oeJBl6b tODzuAfS80jOLNmqDMfKFaLoBx8ZLKkrtxbu/6on01dx0dcq+8D+4GsldO7xEt0ibZRr BQVeDXFsikkr4tF/jKjuvRf6/VEMNfirfV7foMKp4Xpe2n5SaWGhlvLyTXOI5JO5MVrR rg== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2120.oracle.com with ESMTP id 2ghe5xgeyh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 06 Mar 2018 00:26:00 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id w260Q0BV009960 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 6 Mar 2018 00:26:00 GMT Received: from abhmp0008.oracle.com (abhmp0008.oracle.com [141.146.116.14]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id w260PxmO032626; Tue, 6 Mar 2018 00:26:00 GMT Received: from localhost.localdomain (/98.216.35.41) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 05 Mar 2018 16:25:59 -0800 From: Pavel Tatashin To: steven.sistare@oracle.com, daniel.m.jordan@oracle.com, linux-kernel@vger.kernel.org, Alexander.Levin@microsoft.com, dan.j.williams@intel.com, sathyanarayanan.kuppuswamy@intel.com, pankaj.laxminarayan.bharadiya@intel.com, akuster@mvista.com, cminyard@mvista.com, pasha.tatashin@oracle.com, gregkh@linuxfoundation.org, stable@vger.kernel.org Subject: [PATCH 4.1 04/65] x86/mm: If INVPCID is available, use it to flush global mappings Date: Mon, 5 Mar 2018 19:24:37 -0500 Message-Id: <20180306002538.1761-5-pasha.tatashin@oracle.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180306002538.1761-1-pasha.tatashin@oracle.com> References: <20180306002538.1761-1-pasha.tatashin@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8823 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=934 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1803060003 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Andy Lutomirski commit d8bced79af1db6734f66b42064cc773cada2ce99 upstream. On my Skylake laptop, INVPCID function 2 (flush absolutely everything) takes about 376ns, whereas saving flags, twiddling CR4.PGE to flush global mappings, and restoring flags takes about 539ns. Signed-off-by: Andy Lutomirski Reviewed-by: Borislav Petkov Cc: Andrew Morton Cc: Andrey Ryabinin Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Dave Hansen Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Luis R. Rodriguez Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Toshi Kani Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/ed0ef62581c0ea9c99b9bf6df726015e96d44743.1454096309.git.luto@kernel.org Signed-off-by: Ingo Molnar Signed-off-by: Greg Kroah-Hartman (cherry picked from commit 85d3700c744a11ee2989252acf50ccbbd814167a) Signed-off-by: Pavel Tatashin --- arch/x86/include/asm/tlbflush.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index ed2317f19ec7..433eeaafe498 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -152,6 +152,15 @@ static inline void __native_flush_tlb_global(void) { unsigned long flags; + if (static_cpu_has(X86_FEATURE_INVPCID)) { + /* + * Using INVPCID is considerably faster than a pair of writes + * to CR4 sandwiched inside an IRQ flag save/restore. + */ + invpcid_flush_all(); + return; + } + /* * Read-modify-write to CR4 - protect it from preemption and * from interrupts. (Use the raw variant because this code can -- 2.16.2