Received: by 10.213.65.68 with SMTP id h4csp2874761imn; Mon, 9 Apr 2018 10:22:06 -0700 (PDT) X-Google-Smtp-Source: AIpwx496p0iBfse2pcwHSMKA4+TVc/ZlgYWn9FQxPDsafka7kaD8uEJLrT+Jx6RjX8mYCIB8uwu0 X-Received: by 2002:a17:902:b786:: with SMTP id e6-v6mr4999823pls.246.1523294526330; Mon, 09 Apr 2018 10:22:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523294526; cv=none; d=google.com; s=arc-20160816; b=MrmGabGQ4i2BWtSt+RcclPMbF7VvSU90O8ZnW5WGVsYiM91Yk2tEjzzNPoB4vsxw1g vBWXbgbRuLHjTHXl6as1injJpmjle90xaJiIhPFJD6AOOQ4z1u5PhiY/eZ+DNO8VR/km PgMx2D8b1kp7yb3zgJ2EyN3ROSx4ruOWOuTmG6vcOnBc68bPe/+t7ifVPD3jnsL2vaUB /5L2gLnyc4jRbgcf4w7QSpoWFy/MQOEjAi1t8KrEt73l7w3v1lt3OM+5Dvh3oT2AH9/l QESJznWTnT9quMI/oTYr5THKlidUYb4GEeXiEJDHinB2l0iQd0zI4aPlkPYDRTUpuJiC Pp3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-disposition :content-transfer-encoding:mime-version:robot-unsubscribe:robot-id :git-commit-id:subject:to:references:in-reply-to:reply-to:cc :message-id:from:date:arc-authentication-results; bh=Rgs5hDK9D763uC4Xo6A5elj0FKu6JIZ0B9Rb2XQ0PSk=; b=ePkKcenRmiJDeM5qEZD54DiaeBmZNCh/4jTIwHdQdtbwKPDL3W/TFOJAL/BFzvjsjJ hMnzDvKylmfXqcOEtyWBZYGNYJaMyusKmpSSRg2r+WMuwowEa893Qqs8mx7bd8jpuNrP Apb0w4J0wKCUlTU7lT6i2VWUPMG3ATwYPE80uBhiO6flYH/oI7+Blz+R2BdU5gt4itYX wNmq+ELz9RRV7X53iOqUv+SNExea+T1rA7a8SnHzAS8nCKVSQU3OQ6GppCQWId3w9yGD ygfm85KRcIiwtvEphntNERbhhZ1ROXBrX8uXNEcP0JkHfOJl4FXoOAjgg74sd50VHEAy EQkQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i7si465749pgq.583.2018.04.09.10.21.29; Mon, 09 Apr 2018 10:22:06 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754040AbeDIRRI (ORCPT + 99 others); Mon, 9 Apr 2018 13:17:08 -0400 Received: from terminus.zytor.com ([198.137.202.136]:60759 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751291AbeDIRRG (ORCPT ); Mon, 9 Apr 2018 13:17:06 -0400 Received: from terminus.zytor.com (localhost [127.0.0.1]) by terminus.zytor.com (8.15.2/8.15.2) with ESMTP id w39HG5D71894757; Mon, 9 Apr 2018 10:16:05 -0700 Received: (from tipbot@localhost) by terminus.zytor.com (8.15.2/8.15.2/Submit) id w39HG5k31894754; Mon, 9 Apr 2018 10:16:05 -0700 Date: Mon, 9 Apr 2018 10:16:05 -0700 X-Authentication-Warning: terminus.zytor.com: tipbot set sender to tipbot@zytor.com using -f From: tip-bot for Dave Hansen Message-ID: Cc: torvalds@linux-foundation.org, luto@kernel.org, linux-kernel@vger.kernel.org, namit@vmware.com, peterz@infradead.org, dave.hansen@linux.intel.com, tglx@linutronix.de, arjan@linux.intel.com, dwmw2@infradead.org, keescook@google.com, dan.j.williams@intel.com, bp@alien8.de, jpoimboe@redhat.com, hpa@zytor.com, gregkh@linuxfoundation.org, mingo@kernel.org, jgross@suse.com, hughd@google.com, aarcange@redhat.com Reply-To: namit@vmware.com, dave.hansen@linux.intel.com, tglx@linutronix.de, peterz@infradead.org, linux-kernel@vger.kernel.org, luto@kernel.org, torvalds@linux-foundation.org, aarcange@redhat.com, mingo@kernel.org, gregkh@linuxfoundation.org, hughd@google.com, jgross@suse.com, dan.j.williams@intel.com, hpa@zytor.com, bp@alien8.de, jpoimboe@redhat.com, dwmw2@infradead.org, arjan@linux.intel.com, keescook@google.com In-Reply-To: <20180406205517.C80FBE05@viggo.jf.intel.com> References: <20180406205517.C80FBE05@viggo.jf.intel.com> To: linux-tip-commits@vger.kernel.org Subject: [tip:x86/pti] x86/pti: Never implicitly clear _PAGE_GLOBAL for kernel image Git-Commit-ID: a5df4f1f0d7872f6030dd12b166e570e60ae9e1d X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on terminus.zytor.com Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: a5df4f1f0d7872f6030dd12b166e570e60ae9e1d Gitweb: https://git.kernel.org/tip/a5df4f1f0d7872f6030dd12b166e570e60ae9e1d Author: Dave Hansen AuthorDate: Fri, 6 Apr 2018 13:55:17 -0700 Committer: Ingo Molnar CommitDate: Mon, 9 Apr 2018 18:27:34 +0200 x86/pti: Never implicitly clear _PAGE_GLOBAL for kernel image Summary: In current kernels, with PTI enabled, no pages are marked Global. This potentially increases TLB misses. But, the mechanism by which the Global bit is set and cleared is rather haphazard. This patch makes the process more explicit. In the end, it leaves us with Global entries in the page tables for the areas truly shared by userspace and kernel and increases TLB hit rates. The place this patch really shines in on systems without PCIDs. In this case, we are using an lseek microbenchmark[1] to see how a reasonably non-trivial syscall behaves. Higher is better: No Global pages (baseline): 6077741 lseeks/sec 88 Global Pages (this set): 7528609 lseeks/sec (+23.9%) On a modern Skylake desktop with PCIDs, the benefits are tangible, but not huge for a kernel compile (lower is better): No Global pages (baseline): 186.951 seconds time elapsed ( +- 0.35% ) 28 Global pages (this set): 185.756 seconds time elapsed ( +- 0.09% ) -1.195 seconds (-0.64%) I also re-checked everything using the lseek1 test[1]: No Global pages (baseline): 15783951 lseeks/sec 28 Global pages (this set): 16054688 lseeks/sec +270737 lseeks/sec (+1.71%) The effect is more visible, but still modest. Details: The kernel page tables are inherited from head_64.S which rudely marks them as _PAGE_GLOBAL. For PTI, we have been relying on the grace of $DEITY and some insane behavior in pageattr.c to clear _PAGE_GLOBAL. This patch tries to do better. First, stop filtering out "unsupported" bits from being cleared in the pageattr code. It's fine to filter out *setting* these bits but it is insane to keep us from clearing them. Then, *explicitly* go clear _PAGE_GLOBAL from the kernel identity map. Do not rely on pageattr to do it magically. After this patch, we can see that "GLB" shows up in each copy of the page tables, that we have the same number of global entries in each and that they are the *same* entries. /sys/kernel/debug/page_tables/current_kernel:11 /sys/kernel/debug/page_tables/current_user:11 /sys/kernel/debug/page_tables/kernel:11 9caae8ad6a1fb53aca2407ec037f612d current_kernel.GLB 9caae8ad6a1fb53aca2407ec037f612d current_user.GLB 9caae8ad6a1fb53aca2407ec037f612d kernel.GLB A quick visual audit also shows that all the entries make sense. 0xfffffe0000000000 is the cpu_entry_area and 0xffffffff81c00000 is the entry/exit text: 0xfffffe0000000000-0xfffffe0000002000 8K ro GLB NX pte 0xfffffe0000002000-0xfffffe0000003000 4K RW GLB NX pte 0xfffffe0000003000-0xfffffe0000006000 12K ro GLB NX pte 0xfffffe0000006000-0xfffffe0000007000 4K ro GLB x pte 0xfffffe0000007000-0xfffffe000000d000 24K RW GLB NX pte 0xfffffe000002d000-0xfffffe000002e000 4K ro GLB NX pte 0xfffffe000002e000-0xfffffe000002f000 4K RW GLB NX pte 0xfffffe000002f000-0xfffffe0000032000 12K ro GLB NX pte 0xfffffe0000032000-0xfffffe0000033000 4K ro GLB x pte 0xfffffe0000033000-0xfffffe0000039000 24K RW GLB NX pte 0xffffffff81c00000-0xffffffff81e00000 2M ro PSE GLB x pmd [1.] https://github.com/antonblanchard/will-it-scale/blob/master/tests/lseek1.c Signed-off-by: Dave Hansen Cc: Andrea Arcangeli Cc: Andy Lutomirski Cc: Arjan van de Ven Cc: Borislav Petkov Cc: Dan Williams Cc: David Woodhouse Cc: Greg Kroah-Hartman Cc: Hugh Dickins Cc: Josh Poimboeuf Cc: Juergen Gross Cc: Kees Cook Cc: Linus Torvalds Cc: Nadav Amit Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/20180406205517.C80FBE05@viggo.jf.intel.com Signed-off-by: Ingo Molnar --- arch/x86/mm/init.c | 8 +------- arch/x86/mm/pageattr.c | 12 +++++++++--- arch/x86/mm/pti.c | 25 +++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 583a88c8a6ee..fec82b577c18 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -161,12 +161,6 @@ struct map_range { static int page_size_mask; -static void enable_global_pages(void) -{ - if (!static_cpu_has(X86_FEATURE_PTI)) - __supported_pte_mask |= _PAGE_GLOBAL; -} - static void __init probe_page_size_mask(void) { /* @@ -187,7 +181,7 @@ static void __init probe_page_size_mask(void) __supported_pte_mask &= ~_PAGE_GLOBAL; if (boot_cpu_has(X86_FEATURE_PGE)) { cr4_set_bits_and_update_boot(X86_CR4_PGE); - enable_global_pages(); + __supported_pte_mask |= _PAGE_GLOBAL; } /* By the default is everything supported: */ diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index a7324045d87d..0f3d50f4c48c 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -1411,11 +1411,11 @@ static int change_page_attr_set_clr(unsigned long *addr, int numpages, memset(&cpa, 0, sizeof(cpa)); /* - * Check, if we are requested to change a not supported - * feature: + * Check, if we are requested to set a not supported + * feature. Clearing non-supported features is OK. */ mask_set = canon_pgprot(mask_set); - mask_clr = canon_pgprot(mask_clr); + if (!pgprot_val(mask_set) && !pgprot_val(mask_clr) && !force_split) return 0; @@ -1758,6 +1758,12 @@ int set_memory_4k(unsigned long addr, int numpages) __pgprot(0), 1, 0, NULL); } +int set_memory_nonglobal(unsigned long addr, int numpages) +{ + return change_page_attr_clear(&addr, numpages, + __pgprot(_PAGE_GLOBAL), 0); +} + static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc) { struct cpa_data cpa; diff --git a/arch/x86/mm/pti.c b/arch/x86/mm/pti.c index 8082f8b0c10e..1470b173963f 100644 --- a/arch/x86/mm/pti.c +++ b/arch/x86/mm/pti.c @@ -372,6 +372,27 @@ static void __init pti_clone_entry_text(void) _PAGE_RW); } +/* + * This is the only user for it and it is not arch-generic like + * the other set_memory.h functions. Just extern it. + */ +extern int set_memory_nonglobal(unsigned long addr, int numpages); +void pti_set_kernel_image_nonglobal(void) +{ + /* + * The identity map is created with PMDs, regardless of the + * actual length of the kernel. We need to clear + * _PAGE_GLOBAL up to a PMD boundary, not just to the end + * of the image. + */ + unsigned long start = PFN_ALIGN(_text); + unsigned long end = ALIGN((unsigned long)_end, PMD_PAGE_SIZE); + + pr_debug("set kernel image non-global\n"); + + set_memory_nonglobal(start, (end - start) >> PAGE_SHIFT); +} + /* * Initialize kernel page table isolation */ @@ -383,6 +404,10 @@ void __init pti_init(void) pr_info("enabled\n"); pti_clone_user_shared(); + + /* Undo all global bits from the init pagetables in head_64.S: */ + pti_set_kernel_image_nonglobal(); + /* Replace some of the global bits just for shared entry text: */ pti_clone_entry_text(); pti_setup_espfix64(); pti_setup_vsyscall();