LinuxLists.cc - [PATCH 0/4] mm: mremap: fix move page tables

2023-07-31 08:20:54

Subject: [PATCH 0/4] mm: mremap: fix move page tables

The first three patches use correct flush tlb functions when move page
tables, and patch 4 is a small optimization for hugepage on arm64.

Kefeng Wang (4):
mm: hugetlb: use flush_hugetlb_tlb_range() in
move_hugetlb_page_tables()
mm: mremap: use flush_pmd_tlb_range() in move_normal_pmd()
mm: mremap: use flush_pud_tlb_range in move_normal_pud()
arm64: tlb: set huge page size to stride for hugepage

arch/arm64/include/asm/tlbflush.h | 21 +++++++++++----------
mm/hugetlb.c | 4 ++--
mm/mremap.c | 4 ++--
3 files changed, 15 insertions(+), 14 deletions(-)

--
2.41.0

2023-07-31 08:54:25

by Kefeng Wang

[permalink] [raw]

Subject: [PATCH 3/4] mm: mremap: use flush_pud_tlb_range in move_normal_pud()

Archs may need to do special things when flushing thp tlb,
so use the more applicable flush_pud_tlb_range() instead of
flush_tlb_range().

Fixes: c49dd3401802 ("mm: speedup mremap on 1GB or larger regions")
Signed-off-by: Kefeng Wang <[email protected]>
---
mm/mremap.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/mremap.c b/mm/mremap.c
index 1883205fa22b..25114e56901f 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -333,7 +333,7 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr,
VM_BUG_ON(!pud_none(*new_pud));

pud_populate(mm, new_pud, pud_pgtable(pud));
- flush_tlb_range(vma, old_addr, old_addr + PUD_SIZE);
+ flush_pud_tlb_range(vma, old_addr, old_addr + PUD_SIZE);
if (new_ptl != old_ptl)
spin_unlock(new_ptl);
spin_unlock(old_ptl);
--
2.41.0

2023-07-31 09:00:41

by Kefeng Wang

[permalink] [raw]

Subject: [PATCH 2/4] mm: mremap: use flush_pmd_tlb_range() in move_normal_pmd()

Archs may need to do special things when flushing thp tlb,
so use the more applicable flush_pud_tlb_range() instead of
flush_tlb_range().

Fixes: 2c91bd4a4e2e ("mm: speed up mremap by 20x on large regions")
Signed-off-by: Kefeng Wang <[email protected]>
---
mm/mremap.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/mremap.c b/mm/mremap.c
index 11e06e4ab33b..1883205fa22b 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -284,7 +284,7 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
VM_BUG_ON(!pmd_none(*new_pmd));

pmd_populate(mm, new_pmd, pmd_pgtable(pmd));
- flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
+ flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
if (new_ptl != old_ptl)
spin_unlock(new_ptl);
spin_unlock(old_ptl);
--
2.41.0

2023-07-31 11:18:42

by Catalin Marinas

[permalink] [raw]

Subject: Re: [PATCH 2/4] mm: mremap: use flush_pmd_tlb_range() in move_normal_pmd()

On Mon, Jul 31, 2023 at 03:48:27PM +0800, Kefeng Wang wrote:
> Archs may need to do special things when flushing thp tlb,
> so use the more applicable flush_pud_tlb_range() instead of
> flush_tlb_range().
>
> Fixes: 2c91bd4a4e2e ("mm: speed up mremap by 20x on large regions")
> Signed-off-by: Kefeng Wang <[email protected]>
> ---
> mm/mremap.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/mremap.c b/mm/mremap.c
> index 11e06e4ab33b..1883205fa22b 100644
> --- a/mm/mremap.c
> +++ b/mm/mremap.c
> @@ -284,7 +284,7 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
> VM_BUG_ON(!pmd_none(*new_pmd));
>
> pmd_populate(mm, new_pmd, pmd_pgtable(pmd));
> - flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
> + flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);

I don't think that's correct for arm64. The assumption in the
flush_p*d_tlb_range() was that they are called only for block mappings
at that p*d level (and we use FEAT_TTL on arm64 indicating that the leaf
level is level 2 for pmd, 1 for pud). IIUC move_normal_pmd() is only
called for table pmds which would have a leaf level of 3 (the pte).

Same for the next patch doing the equivalent for the pud.

--
Catalin

2023-07-31 12:01:19

by Kefeng Wang

[permalink] [raw]

Subject: Re: [PATCH 2/4] mm: mremap: use flush_pmd_tlb_range() in move_normal_pmd()

On 2023/7/31 19:05, Catalin Marinas wrote:
> On Mon, Jul 31, 2023 at 03:48:27PM +0800, Kefeng Wang wrote:
>> Archs may need to do special things when flushing thp tlb,
>> so use the more applicable flush_pud_tlb_range() instead of
>> flush_tlb_range().
>>
>> Fixes: 2c91bd4a4e2e ("mm: speed up mremap by 20x on large regions")
>> Signed-off-by: Kefeng Wang <[email protected]>
>> ---
>> mm/mremap.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/mm/mremap.c b/mm/mremap.c
>> index 11e06e4ab33b..1883205fa22b 100644
>> --- a/mm/mremap.c
>> +++ b/mm/mremap.c
>> @@ -284,7 +284,7 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
>> VM_BUG_ON(!pmd_none(*new_pmd));
>>
>> pmd_populate(mm, new_pmd, pmd_pgtable(pmd));
>> - flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
>> + flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
>
> I don't think that's correct for arm64. The assumption in the
> flush_p*d_tlb_range() was that they are called only for block mappings
> at that p*d level (and we use FEAT_TTL on arm64 indicating that the leaf
> level is level 2 for pmd, 1 for pud). IIUC move_normal_pmd() is only
> called for table pmds which would have a leaf level of 3 (the pte).

oops, yes, this is for NORMAL_PMD case, not HPAGE_PMD, please ignore
patch 2/3.

>
> Same for the next patch doing the equivalent for the pud.
>

2023-07-31 14:32:31

by kernel test robot

[permalink] [raw]

Subject: Re: [PATCH 2/4] mm: mremap: use flush_pmd_tlb_range() in move_normal_pmd()

Hi Kefeng,

kernel test robot noticed the following build errors:

[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on arm-perf/for-next/perf linus/master v6.5-rc4 next-20230731]
[cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url: https://github.com/intel-lab-lkp/linux/commits/Kefeng-Wang/mm-hugetlb-use-flush_hugetlb_tlb_range-in-move_hugetlb_page_tables/20230731-154016
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
patch link: https://lore.kernel.org/r/20230731074829.79309-3-wangkefeng.wang%40huawei.com
patch subject: [PATCH 2/4] mm: mremap: use flush_pmd_tlb_range() in move_normal_pmd()
config: x86_64-defconfig (https://download.01.org/0day-ci/archive/20230731/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce: (https://download.01.org/0day-ci/archive/20230731/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from <command-line>:
In function 'move_normal_pmd',
inlined from 'move_pgt_entry' at mm/mremap.c:463:11,
inlined from 'move_page_tables' at mm/mremap.c:565:8:
>> include/linux/compiler_types.h:397:45: error: call to '__compiletime_assert_338' declared with attribute error: BUILD_BUG failed
397 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^
include/linux/compiler_types.h:378:25: note: in definition of macro '__compiletime_assert'
378 | prefix ## suffix(); \
| ^~~~~~
include/linux/compiler_types.h:397:9: note: in expansion of macro '_compiletime_assert'
397 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
| ^~~~~~~~~~~~~~~~~~
include/linux/build_bug.h:59:21: note: in expansion of macro 'BUILD_BUG_ON_MSG'
59 | #define BUILD_BUG() BUILD_BUG_ON_MSG(1, "BUILD_BUG failed")
| ^~~~~~~~~~~~~~~~
include/linux/pgtable.h:1415:49: note: in expansion of macro 'BUILD_BUG'
1415 | #define flush_pmd_tlb_range(vma, addr, end) BUILD_BUG()
| ^~~~~~~~~
mm/mremap.c:287:9: note: in expansion of macro 'flush_pmd_tlb_range'
287 | flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
| ^~~~~~~~~~~~~~~~~~~

vim +/__compiletime_assert_338 +397 include/linux/compiler_types.h

eb5c2d4b45e3d2 Will Deacon 2020-07-21 383
eb5c2d4b45e3d2 Will Deacon 2020-07-21 384 #define _compiletime_assert(condition, msg, prefix, suffix) \
eb5c2d4b45e3d2 Will Deacon 2020-07-21 385 __compiletime_assert(condition, msg, prefix, suffix)
eb5c2d4b45e3d2 Will Deacon 2020-07-21 386
eb5c2d4b45e3d2 Will Deacon 2020-07-21 387 /**
eb5c2d4b45e3d2 Will Deacon 2020-07-21 388 * compiletime_assert - break build and emit msg if condition is false
eb5c2d4b45e3d2 Will Deacon 2020-07-21 389 * @condition: a compile-time constant condition to check
eb5c2d4b45e3d2 Will Deacon 2020-07-21 390 * @msg: a message to emit if condition is false
eb5c2d4b45e3d2 Will Deacon 2020-07-21 391 *
eb5c2d4b45e3d2 Will Deacon 2020-07-21 392 * In tradition of POSIX assert, this macro will break the build if the
eb5c2d4b45e3d2 Will Deacon 2020-07-21 393 * supplied condition is *false*, emitting the supplied error message if the
eb5c2d4b45e3d2 Will Deacon 2020-07-21 394 * compiler has support to do so.
eb5c2d4b45e3d2 Will Deacon 2020-07-21 395 */
eb5c2d4b45e3d2 Will Deacon 2020-07-21 396 #define compiletime_assert(condition, msg) \
eb5c2d4b45e3d2 Will Deacon 2020-07-21 @397 _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
eb5c2d4b45e3d2 Will Deacon 2020-07-21 398

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

2023-07-31 17:11:57

by kernel test robot

[permalink] [raw]

Subject: Re: [PATCH 3/4] mm: mremap: use flush_pud_tlb_range in move_normal_pud()

Hi Kefeng,

kernel test robot noticed the following build errors:

[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on arm-perf/for-next/perf linus/master v6.5-rc4 next-20230731]
[cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url: https://github.com/intel-lab-lkp/linux/commits/Kefeng-Wang/mm-hugetlb-use-flush_hugetlb_tlb_range-in-move_hugetlb_page_tables/20230731-154016
base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
patch link: https://lore.kernel.org/r/20230731074829.79309-4-wangkefeng.wang%40huawei.com
patch subject: [PATCH 3/4] mm: mremap: use flush_pud_tlb_range in move_normal_pud()
config: riscv-allmodconfig (https://download.01.org/0day-ci/archive/20230801/[email protected]/config)
compiler: riscv64-linux-gcc (GCC) 12.3.0
reproduce: (https://download.01.org/0day-ci/archive/20230801/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

mm/mremap.c: In function 'move_normal_pud':
>> mm/mremap.c:336:9: error: implicit declaration of function 'flush_pud_tlb_range'; did you mean 'flush_pmd_tlb_range'? [-Werror=implicit-function-declaration]
336 | flush_pud_tlb_range(vma, old_addr, old_addr + PUD_SIZE);
| ^~~~~~~~~~~~~~~~~~~
| flush_pmd_tlb_range
cc1: some warnings being treated as errors

vim +336 mm/mremap.c

302
303 #if CONFIG_PGTABLE_LEVELS > 2 && defined(CONFIG_HAVE_MOVE_PUD)
304 static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr,
305 unsigned long new_addr, pud_t *old_pud, pud_t *new_pud)
306 {
307 spinlock_t *old_ptl, *new_ptl;
308 struct mm_struct *mm = vma->vm_mm;
309 pud_t pud;
310
311 if (!arch_supports_page_table_move())
312 return false;
313 /*
314 * The destination pud shouldn't be established, free_pgtables()
315 * should have released it.
316 */
317 if (WARN_ON_ONCE(!pud_none(*new_pud)))
318 return false;
319
320 /*
321 * We don't have to worry about the ordering of src and dst
322 * ptlocks because exclusive mmap_lock prevents deadlock.
323 */
324 old_ptl = pud_lock(vma->vm_mm, old_pud);
325 new_ptl = pud_lockptr(mm, new_pud);
326 if (new_ptl != old_ptl)
327 spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
328
329 /* Clear the pud */
330 pud = *old_pud;
331 pud_clear(old_pud);
332
333 VM_BUG_ON(!pud_none(*new_pud));
334
335 pud_populate(mm, new_pud, pud_pgtable(pud));
> 336 flush_pud_tlb_range(vma, old_addr, old_addr + PUD_SIZE);
337 if (new_ptl != old_ptl)
338 spin_unlock(new_ptl);
339 spin_unlock(old_ptl);
340
341 return true;
342 }
343 #else
344 static inline bool move_normal_pud(struct vm_area_struct *vma,
345 unsigned long old_addr, unsigned long new_addr, pud_t *old_pud,
346 pud_t *new_pud)
347 {
348 return false;
349 }
350 #endif
351

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

2023-07-31 22:30:01

by kernel test robot

[permalink] [raw]