Received: by 2002:a05:6a10:22f:0:0:0:0 with SMTP id 15csp682766pxk; Thu, 1 Oct 2020 11:12:16 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyie8TBeNm1hfg4To9fE5y5lf7DCOU1WZZEcnMAoGnrtGbUpRDsp7wpKzd2OoKFesN0WbTh X-Received: by 2002:a17:906:441:: with SMTP id e1mr9111079eja.396.1601575936768; Thu, 01 Oct 2020 11:12:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601575936; cv=none; d=google.com; s=arc-20160816; b=B0b4BaU4GvzlC05kZlFzsghKdNP/YvPTffv66C+ohZKU7sGE954MDgLzK1MHMc1Z5I O6U6uTzl7ZFD3en10dChyAcmmI8XcJJZfSPCMR40F6+mP32bkOvCD1GTpFNSsJ2Doged +hqTYJ7koP59K/DhJV4i0QLMRVDRFFe2b2VD9XwLKhpX4MQed/jLMpIv5QC29kqc9wcg V0RiA1F9bmBZnvTqbiPw6o9SRCQyaS+C+OaHs7+/S1+rb1PY5dVEHzS6IEonzlmSrK/3 Y7BvYp9tQ/s+HC1hAw217r2yXooLghSC58eXnuoMjd5zZjH+v4OupbyiljOct7WH88WI 16jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:cc:to:subject :message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=5k48lXNCpXrvc//iysJwDBgdFvSGnGwhyGrPtUgyDYg=; b=CJw23TTs91u7AUV/8QXMEWMNsFDmN11311mtMRDmegwXEb17OAcFSQpHRLAvwnZ8dY +zZAIUnQJfRLmdZT3LtVlKirjE9DYtnbWiK5QvutX8Tt4YumJVtGqntSyh3zTlFK7GjV KRoORRYBCEHG3atfKADiiMePeNAaDb9Ozo388EZog3v0iae0SLDdLrNfUyhYyQsZ5BJf dxWKdHpvMUluvuqoVUhBhxoA7Oo7mRUZ9GPIB3CqE6XmS7J14EEUmcoCDqK0T9u9ehHd /hzemVFh0YnynuiUBuOmPXAtUUp7+TL1eKYe9Pb7I0wRW8kKYeM9vQwcnmlWPqBtbh/0 vdqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=D9G7o9e5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id s21si1446675edi.88.2020.10.01.11.11.52; Thu, 01 Oct 2020 11:12:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=D9G7o9e5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732944AbgJASKn (ORCPT + 99 others); Thu, 1 Oct 2020 14:10:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730116AbgJASKn (ORCPT ); Thu, 1 Oct 2020 14:10:43 -0400 Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E44A0C0613E2 for ; Thu, 1 Oct 2020 11:10:42 -0700 (PDT) Received: by mail-pg1-x541.google.com with SMTP id m34so4625036pgl.9 for ; Thu, 01 Oct 2020 11:10:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=5k48lXNCpXrvc//iysJwDBgdFvSGnGwhyGrPtUgyDYg=; b=D9G7o9e5VA684K9qsYwR6hPwEgX/cIjeS8ETcdfwUHesCS4dSmieZiYf1Vv2ZCwHzE ravrrUwiTqvexHirj+tvedRXPtVaEqvWMs+1gAAWNX3I9v5dXCNvRwm8lwFgF114RnnN dTSb3wTofiOBupGiyTk2DRA3P9RFUtIABSc7+Uve7qzCvIn5W2ylnC7ma+425GndZx1u dmFNmw6aVR4G4+NPy5WXcVN43ffnqKKL4db6GwzBSxFOD9lx1Iv+hYMhVBte4TFv908Q 2m3pgVsxG2qY9m8PFbD58T6xuXHpePEdvnQhi0H7FagcKuiBDV/FrW9/Gg3nJuxoZDq+ wL7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=5k48lXNCpXrvc//iysJwDBgdFvSGnGwhyGrPtUgyDYg=; b=UcYZhH3eggf9FUUoT0gkUc01nVI3BMhHQ7kYItB0U+xDNIUMz1gBMJdX2HBkunWlwu IJfD1N2z9NwECORPLWxJlb3jnXv3rHlGhbT9+c+LKWpB0pXc+CT1iPFRqkk/7xLirjPp JYaKr1CueY2Iz5+CHv/v+qRMekRCrNhRy6swIyM8+oOcfW4/dcOQhcPq9ylic77bBeH9 6ZCLoZmA8mI9iwa7L64bisCDkvMmb57uS4FlXukDdgzqq5NZPPeFHe61x1vpOUJDNzpR IDuLwA/Kq+gbK63XiUkQ6rz4LlzXT2UmSTu+mzhzBDU9PZJo8XUm7ITBSGDFuY3VZcPv krEA== X-Gm-Message-State: AOAM531J9C6GmNs8QildtEpxb2Nk+H2nSRvgvXq19HrTD9Y+MVgLxqnz sApa85Q1a5HFOXPPQKwsFVKIXQgCN/faTlN2gaxl+g== X-Received: by 2002:a65:4481:: with SMTP id l1mr6978077pgq.19.1601575842139; Thu, 01 Oct 2020 11:10:42 -0700 (PDT) MIME-Version: 1.0 References: <20200930222130.4175584-1-kaleshsingh@google.com> <20200930222130.4175584-4-kaleshsingh@google.com> <20201001123651.arcr5gqtia2myt22@black.fi.intel.com> In-Reply-To: From: Kalesh Singh Date: Thu, 1 Oct 2020 14:10:31 -0400 Message-ID: Subject: Re: [PATCH 3/5] mm: Speedup mremap on 1GB or larger regions To: "Kirill A. Shutemov" Cc: Suren Baghdasaryan , Minchan Kim , Joel Fernandes , Lokesh Gidra , "Cc: Android Kernel" , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "the arch/x86 maintainers" , "H. Peter Anvin" , Andrew Morton , Shuah Khan , Kees Cook , "Aneesh Kumar K.V" , Peter Zijlstra , Arnd Bergmann , Sami Tolvanen , Masahiro Yamada , Frederic Weisbecker , Krzysztof Kozlowski , Hassan Naveed , Christian Brauner , Stephen Boyd , Mark Rutland , Mark Brown , Mike Rapoport , Gavin Shan , Chris von Recklinghausen , Jia He , Zhenyu Ye , John Hubbard , Sandipan Das , Dave Hansen , Ralph Campbell , Ram Pai , William Kucharski , Brian Geffon , Mina Almasry , Masami Hiramatsu , SeongJae Park , LKML , "moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)" , "open list:MEMORY MANAGEMENT" , "open list:KERNEL SELFTEST FRAMEWORK" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Oct 1, 2020 at 12:40 PM Kalesh Singh wrote= : > > On Thu, Oct 1, 2020 at 8:37 AM Kirill A. Shutemov > wrote: > > > > On Wed, Sep 30, 2020 at 10:21:20PM +0000, Kalesh Singh wrote: > > > Android needs to move large memory regions for garbage collection. > > > Optimize mremap for >=3D 1GB-sized regions by moving at the PUD/PGD > > > level if the source and destination addresses are PUD-aligned. > > > For CONFIG_PGTABLE_LEVELS =3D=3D 3, moving at the PUD level in effect= moves > > > PGD entries, since the PUD entry is =E2=80=9Cfolded back=E2=80=9D ont= o the PGD entry. > > > Add HAVE_MOVE_PUD so that architectures where moving at the PUD level > > > isn't supported/tested can turn this off by not selecting the config. > > > > > > Signed-off-by: Kalesh Singh > > > --- > > > arch/Kconfig | 7 + > > > arch/arm64/include/asm/pgtable.h | 1 + > > > mm/mremap.c | 211 ++++++++++++++++++++++++++---= -- > > > 3 files changed, 189 insertions(+), 30 deletions(-) > > > > > > diff --git a/arch/Kconfig b/arch/Kconfig > > > index af14a567b493..5eabaa00bf9b 100644 > > > --- a/arch/Kconfig > > > +++ b/arch/Kconfig > > > @@ -602,6 +602,13 @@ config HAVE_IRQ_TIME_ACCOUNTING > > > Archs need to ensure they use a high enough resolution clock = to > > > support irq time accounting and then call enable_sched_clock_= irqtime(). > > > > > > +config HAVE_MOVE_PUD > > > + bool > > > + help > > > + Architectures that select this are able to move page tables a= t the > > > + PUD level. If there are only 3 page table levels, the move ef= fectively > > > + happens at the PGD level. > > > + > > > config HAVE_MOVE_PMD > > > bool > > > help > > > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/as= m/pgtable.h > > > index d5d3fbe73953..8848125e3024 100644 > > > --- a/arch/arm64/include/asm/pgtable.h > > > +++ b/arch/arm64/include/asm/pgtable.h > > > @@ -415,6 +415,7 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) > > > #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(p= fn) << PAGE_SHIFT) | pgprot_val(prot)) > > > > > > #define set_pmd_at(mm, addr, pmdp, pmd) set_pte_at(mm, addr, (p= te_t *)pmdp, pmd_pte(pmd)) > > > +#define set_pud_at(mm, addr, pudp, pud) set_pte_at(mm, addr, (p= te_t *)pudp, pud_pte(pud)) > > > > > > #define __p4d_to_phys(p4d) __pte_to_phys(p4d_pte(p4d)) > > > #define __phys_to_p4d_val(phys) __phys_to_pte_val(phys) > > > > This doesn't belong to the patch. > Good catch. I'll move this into a separate patch. > > > > > diff --git a/mm/mremap.c b/mm/mremap.c > > > index 138abbae4f75..a5a1440bd366 100644 > > > --- a/mm/mremap.c > > > +++ b/mm/mremap.c > > > @@ -249,14 +249,167 @@ static bool move_normal_pmd(struct vm_area_str= uct *vma, unsigned long old_addr, > > > > > > return true; > > > } > > > +#else > > > +static inline bool move_normal_pmd(struct vm_area_struct *vma, unsig= ned long old_addr, > > > + unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pm= d) > > > +{ > > > + return false; > > > +} > > > #endif > > > > > > +#ifdef CONFIG_HAVE_MOVE_PUD > > > +static pud_t *get_old_pud(struct mm_struct *mm, unsigned long addr) > > > +{ > > > + pgd_t *pgd; > > > + p4d_t *p4d; > > > + pud_t *pud; > > > + > > > + pgd =3D pgd_offset(mm, addr); > > > + if (pgd_none_or_clear_bad(pgd)) > > > + return NULL; > > > + > > > + p4d =3D p4d_offset(pgd, addr); > > > + if (p4d_none_or_clear_bad(p4d)) > > > + return NULL; > > > + > > > + pud =3D pud_offset(p4d, addr); > > > + if (pud_none_or_clear_bad(pud)) > > > + return NULL; > > > + > > > + return pud; > > > +} > > > + > > > +static pud_t *alloc_new_pud(struct mm_struct *mm, struct vm_area_str= uct *vma, > > > + unsigned long addr) > > > +{ > > > + pgd_t *pgd; > > > + p4d_t *p4d; > > > + pud_t *pud; > > > + > > > + pgd =3D pgd_offset(mm, addr); > > > + p4d =3D p4d_alloc(mm, pgd, addr); > > > + if (!p4d) > > > + return NULL; > > > + pud =3D pud_alloc(mm, p4d, addr); > > > + if (!pud) > > > + return NULL; > > > + > > > + return pud; > > > +} > > > + > > > +static bool move_normal_pud(struct vm_area_struct *vma, unsigned lon= g old_addr, > > > + unsigned long new_addr, pud_t *old_pud, pud_t *new_pu= d) > > > +{ > > > + spinlock_t *old_ptl, *new_ptl; > > > + struct mm_struct *mm =3D vma->vm_mm; > > > + pud_t pud; > > > + > > > + /* > > > + * The destination pud shouldn't be established, free_pgtables(= ) > > > + * should have released it. > > > + */ > > > + if (WARN_ON_ONCE(!pud_none(*new_pud))) > > > + return false; > > > + > > > + /* > > > + * We don't have to worry about the ordering of src and dst > > > + * ptlocks because exclusive mmap_lock prevents deadlock. > > > + */ > > > + old_ptl =3D pud_lock(vma->vm_mm, old_pud); > > > + new_ptl =3D pud_lockptr(mm, new_pud); > > > + if (new_ptl !=3D old_ptl) > > > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); > > > + > > > + /* Clear the pud */ > > > + pud =3D *old_pud; > > > + pud_clear(old_pud); > > > + > > > + VM_BUG_ON(!pud_none(*new_pud)); > > > + > > > + /* Set the new pud */ > > > + set_pud_at(mm, new_addr, new_pud, pud); > > > + flush_tlb_range(vma, old_addr, old_addr + PUD_SIZE); > > > + if (new_ptl !=3D old_ptl) > > > + spin_unlock(new_ptl); > > > + spin_unlock(old_ptl); > > > + > > > + return true; > > > +} > > > +#else > > > +static inline bool move_normal_pud(struct vm_area_struct *vma, unsig= ned long old_addr, > > > + unsigned long new_addr, pud_t *old_pud, pud_t *new_pu= d) > > > +{ > > > + return false; > > > +} > > > +#endif > > > + > > > +enum pgt_entry { > > > + NORMAL_PMD, > > > + HPAGE_PMD, > > > + NORMAL_PUD, > > > +}; > > > + > > > +/* > > > + * Returns an extent of the corresponding size for the pgt_entry spe= cified if valid. > > > + * Else returns a smaller extent bounded by the end of the source an= d destination > > > + * pgt_entry. Returns 0 if an invalid pgt_entry is specified. > > > + */ > > > +static unsigned long get_extent(enum pgt_entry entry, unsigned long = old_addr, > > > + unsigned long old_end, unsigned long new_addr) > > > +{ > > > + unsigned long next, extent, mask, size; > > > + > > > + if (entry =3D=3D NORMAL_PMD || entry =3D=3D HPAGE_PMD) { > > > + mask =3D PMD_MASK; > > > + size =3D PMD_SIZE; > > > + } else if (entry =3D=3D NORMAL_PUD) { > > > + mask =3D PUD_MASK; > > > + size =3D PUD_SIZE; > > > + } else > > > + return 0; > > > + > > > + next =3D (old_addr + size) & mask; > > > + /* even if next overflowed, extent below will be ok */ > > > + extent =3D (next > old_end) ? old_end - old_addr : next - old_a= ddr; > > > + next =3D (new_addr + size) & mask; > > > + if (extent > next - new_addr) > > > + extent =3D next - new_addr; > > > + return extent; > > > +} > > > + > > > +/* > > > + * Attempts to speedup the move by moving entry at the level corresp= onding to > > > + * pgt_entry. Returns true if the move was successful, else false. > > > + */ > > > +static bool move_pgt_entry(enum pgt_entry entry, struct vm_area_stru= ct *vma, > > > + unsigned long old_addr, unsigned long new_addr,= void *old_entry, > > > + void *new_entry, bool need_rmap_locks) > > > +{ > > > + bool moved =3D false; > > > + > > > + /* See comment in move_ptes() */ > > > + if (need_rmap_locks) > > > + take_rmap_locks(vma); > > > + if (entry =3D=3D NORMAL_PMD) > > > + moved =3D move_normal_pmd(vma, old_addr, new_addr, old= _entry, new_entry); > > > + else if (entry =3D=3D NORMAL_PUD) > > > + moved =3D move_normal_pud(vma, old_addr, new_addr, old= _entry, new_entry); > > > + else if (entry =3D=3D HPAGE_PMD) > > > + moved =3D move_huge_pmd(vma, old_addr, new_addr, old_e= ntry, new_entry); > > > + else > > > + WARN_ON_ONCE(1); > > > > BUILD_BUG() should work. Please ignore the previous comment. You are right, BUILD_BUG() would work. > This doesn't get caught at compile time since entry isn't a constant. > > > > And why not use switch() instead of ifs. > I'll move to switch() in the next version. > Thanks, Kalesh > > > > > + if (need_rmap_locks) > > > + drop_rmap_locks(vma); > > > + > > > + return moved; > > > +} > > > + > > > unsigned long move_page_tables(struct vm_area_struct *vma, > > > unsigned long old_addr, struct vm_area_struct *new_vma, > > > unsigned long new_addr, unsigned long len, > > > bool need_rmap_locks) > > > { > > > - unsigned long extent, next, old_end; > > > + unsigned long extent, old_end; > > > struct mmu_notifier_range range; > > > pmd_t *old_pmd, *new_pmd; > > > > > > @@ -269,14 +422,27 @@ unsigned long move_page_tables(struct vm_area_s= truct *vma, > > > > > > for (; old_addr < old_end; old_addr +=3D extent, new_addr +=3D = extent) { > > > cond_resched(); > > > - next =3D (old_addr + PMD_SIZE) & PMD_MASK; > > > - /* even if next overflowed, extent below will be ok */ > > > - extent =3D next - old_addr; > > > - if (extent > old_end - old_addr) > > > - extent =3D old_end - old_addr; > > > - next =3D (new_addr + PMD_SIZE) & PMD_MASK; > > > - if (extent > next - new_addr) > > > - extent =3D next - new_addr; > > > +#ifdef CONFIG_HAVE_MOVE_PUD > > > + /* > > > + * If extent is PUD-sized try to speed up the move by m= oving at the > > > + * PUD level if possible. > > > + */ > > > + extent =3D get_extent(NORMAL_PUD, old_addr, old_end, ne= w_addr); > > > + if (extent =3D=3D PUD_SIZE) { > > > + pud_t *old_pud, *new_pud; > > > + > > > + old_pud =3D get_old_pud(vma->vm_mm, old_addr); > > > + if (!old_pud) > > > + continue; > > > + new_pud =3D alloc_new_pud(vma->vm_mm, vma, new_= addr); > > > + if (!new_pud) > > > + break; > > > + if (move_pgt_entry(NORMAL_PUD, vma, old_addr, n= ew_addr, > > > + old_pud, new_pud, need_rmap_= locks)) > > > + continue; > > > + } > > > +#endif > > > + extent =3D get_extent(NORMAL_PMD, old_addr, old_end, ne= w_addr); > > > old_pmd =3D get_old_pmd(vma->vm_mm, old_addr); > > > if (!old_pmd) > > > continue; > > > @@ -284,18 +450,10 @@ unsigned long move_page_tables(struct vm_area_s= truct *vma, > > > if (!new_pmd) > > > break; > > > if (is_swap_pmd(*old_pmd) || pmd_trans_huge(*old_pmd) |= | pmd_devmap(*old_pmd)) { > > > - if (extent =3D=3D HPAGE_PMD_SIZE) { > > > - bool moved; > > > - /* See comment in move_ptes() */ > > > - if (need_rmap_locks) > > > - take_rmap_locks(vma); > > > - moved =3D move_huge_pmd(vma, old_addr, = new_addr, > > > - old_pmd, new_pmd)= ; > > > - if (need_rmap_locks) > > > - drop_rmap_locks(vma); > > > - if (moved) > > > - continue; > > > - } > > > + if (extent =3D=3D HPAGE_PMD_SIZE && > > > + move_pgt_entry(HPAGE_PMD, vma, old_addr, ne= w_addr, old_pmd, > > > + new_pmd, need_rmap_locks)) > > > + continue; > > > split_huge_pmd(vma, old_pmd, old_addr); > > > if (pmd_trans_unstable(old_pmd)) > > > continue; > > > @@ -305,15 +463,8 @@ unsigned long move_page_tables(struct vm_area_st= ruct *vma, > > > * If the extent is PMD-sized, try to speed the= move by > > > * moving at the PMD level if possible. > > > */ > > > - bool moved; > > > - > > > - if (need_rmap_locks) > > > - take_rmap_locks(vma); > > > - moved =3D move_normal_pmd(vma, old_addr, new_ad= dr, > > > - old_pmd, new_pmd); > > > - if (need_rmap_locks) > > > - drop_rmap_locks(vma); > > > - if (moved) > > > + if (move_pgt_entry(NORMAL_PMD, vma, old_addr, n= ew_addr, old_pmd, > > > + new_pmd, need_rmap_locks)) > > > continue; > > > #endif > > > } > > > -- > > > 2.28.0.709.gb0816b6eb0-goog > > > > > > > -- > > Kirill A. Shutemov > > > > -- > > To unsubscribe from this group and stop receiving emails from it, send = an email to kernel-team+unsubscribe@android.com. > >