mremap(2) does not allow source and destination regions to overlap, but
shift_arg_pages() calls move_page_tables() directly and in this case the
source and destination overlap often. It confuses move_normal_pmd():
WARNING: CPU: 3 PID: 27091 at mm/mremap.c:211 move_page_tables+0x6ef/0x720
move_normal_pmd() expects the destination PMD to be empty, but when
ranges overlap nobody removes PTE page tables on source side.
move_ptes() only removes PTE entries, leaving tables behind.
When the source PMD becomes destination and alignment/size is right we
step onto the warning.
The warning is harmless: kernel correctly fallbacks to handle entries on
per-entry basis.
The fix is to avoid move_normal_pmd() if we see that source and
destination ranges overlap.
Signed-off-by: Kirill A. Shutemov <[email protected]>
Reported-and-tested-by: Naresh Kamboju <[email protected]>
Fixes: 2c91bd4a4e2e ("mm: speed up mremap by 20x on large regions")
Cc: Joel Fernandes <[email protected]>
Cc: William Kucharski <[email protected]>
---
mm/mremap.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/mm/mremap.c b/mm/mremap.c
index 5dd572d57ca9..e33fcee541fe 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -245,6 +245,18 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
unsigned long extent, next, old_end;
struct mmu_notifier_range range;
pmd_t *old_pmd, *new_pmd;
+ bool overlaps;
+
+ /*
+ * shift_arg_pages() can call move_page_tables() on overlapping ranges.
+ * In this case we cannot use move_normal_pmd() because destination pmd
+ * might be established page table: move_ptes() doesn't free page
+ * table.
+ */
+ if (old_addr > new_addr)
+ overlaps = old_addr - new_addr < len;
+ else
+ overlaps = new_addr - old_addr < len;
old_end = old_addr + len;
flush_cache_range(vma, old_addr, old_end);
@@ -282,7 +294,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
split_huge_pmd(vma, old_pmd, old_addr);
if (pmd_trans_unstable(old_pmd))
continue;
- } else if (extent == PMD_SIZE) {
+ } else if (!overlaps && extent == PMD_SIZE) {
#ifdef CONFIG_HAVE_MOVE_PMD
/*
* If the extent is PMD-sized, try to speed the move by
--
2.26.2
On Wed, Jul 15, 2020 at 03:35:13PM +0300, Kirill A. Shutemov wrote:
> mremap(2) does not allow source and destination regions to overlap, but
> shift_arg_pages() calls move_page_tables() directly and in this case the
> source and destination overlap often. It confuses move_normal_pmd():
>
> WARNING: CPU: 3 PID: 27091 at mm/mremap.c:211 move_page_tables+0x6ef/0x720
>
> move_normal_pmd() expects the destination PMD to be empty, but when
> ranges overlap nobody removes PTE page tables on source side.
> move_ptes() only removes PTE entries, leaving tables behind.
> When the source PMD becomes destination and alignment/size is right we
> step onto the warning.
>
> The warning is harmless: kernel correctly fallbacks to handle entries on
> per-entry basis.
A link to the debugging effort could be added to the change log:
https://lore.kernel.org/lkml/[email protected]/
> The fix is to avoid move_normal_pmd() if we see that source and
> destination ranges overlap.
Reviewed-by: Joel Fernandes (Google) <[email protected]>
And one thing that bothers me:
> mm/mremap.c | 14 +++++++++++++-
> 1 file changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/mm/mremap.c b/mm/mremap.c
> index 5dd572d57ca9..e33fcee541fe 100644
> --- a/mm/mremap.c
> +++ b/mm/mremap.c
> @@ -245,6 +245,18 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
> unsigned long extent, next, old_end;
> struct mmu_notifier_range range;
> pmd_t *old_pmd, *new_pmd;
> + bool overlaps;
> +
> + /*
> + * shift_arg_pages() can call move_page_tables() on overlapping ranges.
> + * In this case we cannot use move_normal_pmd() because destination pmd
> + * might be established page table: move_ptes() doesn't free page
> + * table.
> + */
> + if (old_addr > new_addr)
> + overlaps = old_addr - new_addr < len;
> + else
> + overlaps = new_addr - old_addr < len;
Does the code really work properly if old_addr < new_addr and overlaps ==
true ? If not, then we should add a warning here in the else IMHO:
if (old_addr >= new_addr) {
overlaps = old_addr - new_addr < len;
} else {
overlaps = new_addr - old_addr < len;
WARN_ON(overlaps);
}
(More so, since you have added code that detects overlaps for such a case).
thanks,
- Joel
>
> old_end = old_addr + len;
> flush_cache_range(vma, old_addr, old_end);
> @@ -282,7 +294,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
> split_huge_pmd(vma, old_pmd, old_addr);
> if (pmd_trans_unstable(old_pmd))
> continue;
> - } else if (extent == PMD_SIZE) {
> + } else if (!overlaps && extent == PMD_SIZE) {
> #ifdef CONFIG_HAVE_MOVE_PMD
> /*
> * If the extent is PMD-sized, try to speed the move by
> --
> 2.26.2
>
>
On Wed, Jul 15, 2020 at 09:14:51AM -0400, Joel Fernandes wrote:
> On Wed, Jul 15, 2020 at 03:35:13PM +0300, Kirill A. Shutemov wrote:
> > mremap(2) does not allow source and destination regions to overlap, but
> > shift_arg_pages() calls move_page_tables() directly and in this case the
> > source and destination overlap often. It confuses move_normal_pmd():
> >
> > WARNING: CPU: 3 PID: 27091 at mm/mremap.c:211 move_page_tables+0x6ef/0x720
> >
> > move_normal_pmd() expects the destination PMD to be empty, but when
> > ranges overlap nobody removes PTE page tables on source side.
> > move_ptes() only removes PTE entries, leaving tables behind.
> > When the source PMD becomes destination and alignment/size is right we
> > step onto the warning.
> >
> > The warning is harmless: kernel correctly fallbacks to handle entries on
> > per-entry basis.
>
> A link to the debugging effort could be added to the change log:
> https://lore.kernel.org/lkml/[email protected]/
>
> > The fix is to avoid move_normal_pmd() if we see that source and
> > destination ranges overlap.
>
> Reviewed-by: Joel Fernandes (Google) <[email protected]>
Thanks.
> And one thing that bothers me:
>
> > mm/mremap.c | 14 +++++++++++++-
> > 1 file changed, 13 insertions(+), 1 deletion(-)
> >
> > diff --git a/mm/mremap.c b/mm/mremap.c
> > index 5dd572d57ca9..e33fcee541fe 100644
> > --- a/mm/mremap.c
> > +++ b/mm/mremap.c
> > @@ -245,6 +245,18 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
> > unsigned long extent, next, old_end;
> > struct mmu_notifier_range range;
> > pmd_t *old_pmd, *new_pmd;
> > + bool overlaps;
> > +
> > + /*
> > + * shift_arg_pages() can call move_page_tables() on overlapping ranges.
> > + * In this case we cannot use move_normal_pmd() because destination pmd
> > + * might be established page table: move_ptes() doesn't free page
> > + * table.
> > + */
> > + if (old_addr > new_addr)
> > + overlaps = old_addr - new_addr < len;
> > + else
> > + overlaps = new_addr - old_addr < len;
>
> Does the code really work properly if old_addr < new_addr and overlaps ==
> true ? If not, then we should add a warning here in the else IMHO:
>
> if (old_addr >= new_addr) {
> overlaps = old_addr - new_addr < len;
> } else {
> overlaps = new_addr - old_addr < len;
> WARN_ON(overlaps);
> }
>
> (More so, since you have added code that detects overlaps for such a case).
Right, I'll update. But we should warn for old_addr == new_addr too.
Nobody should do this.
>
> thanks,
>
> - Joel
>
> >
> > old_end = old_addr + len;
> > flush_cache_range(vma, old_addr, old_end);
> > @@ -282,7 +294,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
> > split_huge_pmd(vma, old_pmd, old_addr);
> > if (pmd_trans_unstable(old_pmd))
> > continue;
> > - } else if (extent == PMD_SIZE) {
> > + } else if (!overlaps && extent == PMD_SIZE) {
> > #ifdef CONFIG_HAVE_MOVE_PMD
> > /*
> > * If the extent is PMD-sized, try to speed the move by
> > --
> > 2.26.2
> >
> >
>
--
Kirill A. Shutemov