Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753749AbbFMJua (ORCPT ); Sat, 13 Jun 2015 05:50:30 -0400 Received: from mail-wi0-f182.google.com ([209.85.212.182]:35392 "EHLO mail-wi0-f182.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752012AbbFMJtm (ORCPT ); Sat, 13 Jun 2015 05:49:42 -0400 From: Ingo Molnar To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andy Lutomirski , Andrew Morton , Denys Vlasenko , Brian Gerst , Peter Zijlstra , Borislav Petkov , "H. Peter Anvin" , Linus Torvalds , Oleg Nesterov , Thomas Gleixner , Waiman Long Subject: [PATCH 05/12] mm: Introduce arch_pgd_init_late() Date: Sat, 13 Jun 2015 11:49:08 +0200 Message-Id: <1434188955-31397-6-git-send-email-mingo@kernel.org> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1434188955-31397-1-git-send-email-mingo@kernel.org> References: <1434188955-31397-1-git-send-email-mingo@kernel.org> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3829 Lines: 119 Add a late PGD init callback to places that allocate a new MM with a new PGD: copy_process() and exec(). The purpose of this callback is to allow architectures to implement lockless initialization of task PGDs, to remove the scalability limit of pgd_list/pgd_lock. Architectures can opt in to this callback via the ARCH_HAS_PGD_INIT_LATE Kconfig flag. There's zero overhead on architectures that are not using it. Cc: Andrew Morton Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Oleg Nesterov Cc: Peter Zijlstra Cc: Rik van Riel Cc: Thomas Gleixner Cc: Waiman Long Cc: linux-arch@vger.kernel.org Cc: linux-mm@kvack.org Signed-off-by: Ingo Molnar --- arch/Kconfig | 9 +++++++++ fs/exec.c | 3 +++ include/linux/mm.h | 6 ++++++ kernel/fork.c | 16 ++++++++++++++++ 4 files changed, 34 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index a65eafb24997..a8e866cd4247 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -491,6 +491,15 @@ config PGTABLE_LEVELS int default 2 +config ARCH_HAS_PGD_INIT_LATE + bool + help + Architectures that want a late PGD initialization can define + the arch_pgd_init_late() callback and it will be called + by the generic new task (fork()) code after a new task has + been made visible on the task list, but before it has been + first scheduled. + config ARCH_HAS_ELF_RANDOMIZE bool help diff --git a/fs/exec.c b/fs/exec.c index 1977c2a553ac..4ce1383d5bba 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -860,7 +860,10 @@ static int exec_mmap(struct mm_struct *mm) } task_lock(tsk); active_mm = tsk->active_mm; + tsk->mm = mm; + arch_pgd_init_late(mm); + tsk->active_mm = mm; activate_mm(active_mm, mm); tsk->mm->vmacache_seqnum = 0; diff --git a/include/linux/mm.h b/include/linux/mm.h index 0755b9fd03a7..a3edc839e431 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1134,6 +1134,12 @@ int follow_phys(struct vm_area_struct *vma, unsigned long address, int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); +#ifdef CONFIG_ARCH_HAS_PGD_INIT_LATE +void arch_pgd_init_late(struct mm_struct *mm); +#else +static inline void arch_pgd_init_late(struct mm_struct *mm) { } +#endif + static inline void unmap_shared_mapping_range(struct address_space *mapping, loff_t const holebegin, loff_t const holelen) { diff --git a/kernel/fork.c b/kernel/fork.c index 03c1eaaa6ef5..cfa84971fb52 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1592,6 +1592,22 @@ static struct task_struct *copy_process(unsigned long clone_flags, syscall_tracepoint_update(p); write_unlock_irq(&tasklist_lock); + /* + * If we have a new PGD then initialize it: + * + * This method is called after a task has been made visible + * on the task list already. + * + * Architectures that manage per task kernel pagetables + * might use this callback to initialize them after they + * are already visible to new updates. + * + * NOTE: any user-space parts of the PGD are already initialized + * and must not be clobbered. + */ + if (!(clone_flags & CLONE_VM)) + arch_pgd_init_late(p->mm); + proc_fork_connector(p); cgroup_post_fork(p); if (clone_flags & CLONE_THREAD) -- 2.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/