Hi,
Here's an update, mainly just collecting reviewed-by tags:
Changes since v4:
* Patch 1: reflowed the commit description to 72 columns.
* Added reviewed-by's from Christoph and Jan for remaining patches.
* Rebased onto 5.17-rc3.
Changes since v3:
* Patch 1: Commit log: removed a reference to an out of tree module,
and updated tags and added a short note to that effect.
* Patch 2: Fixed up do_move_pages(), to squash -EEXIST returns from
follow_page() into something that move_pages() allows: -EFAULT.
This is from reviews by Jan Kara, Matthew Wilcox, Jason Gunthorpe.
Removed all Reviewed-by tags for this patch, due to that update.
* Patch 4: Split into two patches, as recommended by Christoph
Hellwig. Left the Reviewed-by tags intact because no code changes
were required.
* Added Reviewed-by tags from Claudio Imbrenda.
Changes since v2:
* Patch 2: Removed an unnecessary line that was causing a
clang-analyzer complaint, as reported by Lukas Bulwahn
[1] (thanks!), and retested locally.
Jason: I've boldly left your Reviewed-by tag on this patch,
because I am predicting you'll agree with it...
* Added Reviewed-by tags from Jan Kara, Christoph Hellwig, and
Jason Gunthorpe that have collected since v2.
Changes since v1:
* Patch 4: changed from get_user_pages(), to get_user_pages_fast().
* Patch 4: Rewrote the commit description--thanks to Jan Kara for
that feedback.
* Patch 1: Removed Jerome's Cc from patch 1, due to a stale email
address.
* Added Reviewed-by's from David Hildenbrand and Jason Gunthorpe.
Original cover letter, updated as necessary:
I'm including Peter's patch as the first one in this tiny series. (The
commit description has my r-b tag in place of my Cc, and removes
Jerome's Cc because he is no longer at redhat.com) The second patch is
what I had in mind for a follow-up to that, when we were discussing that
fix [2].
Plus, a couple more small removals that I had queued up:
The third patch removes a completely unused routine:
pin_user_pages_locked().
The forth patch removes a similar routine, get_user_pages_locked(), that
only has one caller. It now calls get_user_pages_fast(), instead.
v1 of this patchset is here:
https://lore.kernel.org/all/[email protected]/
v2:
https://lore.kernel.org/r/[email protected]
v3:
https://lore.kernel.org/r/[email protected]
v4:
https://lore.kernel.org/r/[email protected]
[1] https://lore.kernel.org/r/CAKXUXMxFK9bo8jDoRZbQ0r2j-JwAGg3Xc5cpAcLaHfwHddJ7ew@mail.gmail.com
[2] https://lore.kernel.org/all/[email protected]/
John Hubbard (4):
mm/gup: follow_pfn_pte(): -EEXIST cleanup
mm/gup: remove unused pin_user_pages_locked()
mm: change lookup_node() to use get_user_pages_fast()
mm/gup: remove unused get_user_pages_locked()
Peter Xu (1):
mm: Fix invalid page pointer returned with FOLL_PIN gups
include/linux/mm.h | 4 --
mm/gup.c | 101 ++++-----------------------------------------
mm/mempolicy.c | 21 ++++------
mm/migrate.c | 7 ++++
4 files changed, 24 insertions(+), 109 deletions(-)
base-commit: dfd42facf1e4ada021b939b4e19c935dcdd55566
--
2.35.1
The purpose of calling get_user_pages_locked() from lookup_node() was to
allow for unlocking the mmap_lock when reading a page from the disk
during a page fault (hidden behind VM_FAULT_RETRY). The idea was to
reduce contention on the heavily-used mmap_lock. (Thanks to Jan Kara for
clearly pointing that out, and in fact I've used some of his wording
here.)
However, it is unlikely for lookup_node() to take a page fault. With
that in mind, change over to calling get_user_pages_fast(). This
simplifies the code, runs a little faster in the expected case, and
allows removing get_user_pages_locked() entirely, in a subsequent patch.
Reviewed-by: Christoph Hellwig <[email protected]>
Reviewed-by: Jan Kara <[email protected]>
Reviewed-by: Jason Gunthorpe <[email protected]>
Reviewed-by: Claudio Imbrenda <[email protected]>
Signed-off-by: John Hubbard <[email protected]>
---
mm/mempolicy.c | 21 +++++++++------------
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 028e8dd82b44..3f8dc58da3e8 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -907,17 +907,14 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)
static int lookup_node(struct mm_struct *mm, unsigned long addr)
{
struct page *p = NULL;
- int err;
+ int ret;
- int locked = 1;
- err = get_user_pages_locked(addr & PAGE_MASK, 1, 0, &p, &locked);
- if (err > 0) {
- err = page_to_nid(p);
+ ret = get_user_pages_fast(addr & PAGE_MASK, 1, 0, &p);
+ if (ret > 0) {
+ ret = page_to_nid(p);
put_page(p);
}
- if (locked)
- mmap_read_unlock(mm);
- return err;
+ return ret;
}
/* Retrieve NUMA policy */
@@ -968,14 +965,14 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,
if (flags & MPOL_F_NODE) {
if (flags & MPOL_F_ADDR) {
/*
- * Take a refcount on the mpol, lookup_node()
- * will drop the mmap_lock, so after calling
- * lookup_node() only "pol" remains valid, "vma"
- * is stale.
+ * Take a refcount on the mpol, because we are about to
+ * drop the mmap_lock, after which only "pol" remains
+ * valid, "vma" is stale.
*/
pol_refcount = pol;
vma = NULL;
mpol_get(pol);
+ mmap_read_unlock(mm);
err = lookup_node(mm, addr);
if (err < 0)
goto out;
--
2.35.1
This routine was used for a short while, but then the calling code was
refactored and the only caller was removed.
Reviewed-by: David Hildenbrand <[email protected]>
Reviewed-by: Jason Gunthorpe <[email protected]>
Reviewed-by: Jan Kara <[email protected]>
Reviewed-by: Christoph Hellwig <[email protected]>
Reviewed-by: Claudio Imbrenda <[email protected]>
Signed-off-by: John Hubbard <[email protected]>
---
include/linux/mm.h | 2 --
mm/gup.c | 29 -----------------------------
2 files changed, 31 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 213cc569b192..80c540c17d83 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1918,8 +1918,6 @@ long pin_user_pages(unsigned long start, unsigned long nr_pages,
struct vm_area_struct **vmas);
long get_user_pages_locked(unsigned long start, unsigned long nr_pages,
unsigned int gup_flags, struct page **pages, int *locked);
-long pin_user_pages_locked(unsigned long start, unsigned long nr_pages,
- unsigned int gup_flags, struct page **pages, int *locked);
long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
struct page **pages, unsigned int gup_flags);
long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
diff --git a/mm/gup.c b/mm/gup.c
index 2df0d0103c43..6664dfa7a38f 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -3146,32 +3146,3 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
return get_user_pages_unlocked(start, nr_pages, pages, gup_flags);
}
EXPORT_SYMBOL(pin_user_pages_unlocked);
-
-/*
- * pin_user_pages_locked() is the FOLL_PIN variant of get_user_pages_locked().
- * Behavior is the same, except that this one sets FOLL_PIN and rejects
- * FOLL_GET.
- */
-long pin_user_pages_locked(unsigned long start, unsigned long nr_pages,
- unsigned int gup_flags, struct page **pages,
- int *locked)
-{
- /*
- * FIXME: Current FOLL_LONGTERM behavior is incompatible with
- * FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on
- * vmas. As there are no users of this flag in this call we simply
- * disallow this option for now.
- */
- if (WARN_ON_ONCE(gup_flags & FOLL_LONGTERM))
- return -EINVAL;
-
- /* FOLL_GET and FOLL_PIN are mutually exclusive. */
- if (WARN_ON_ONCE(gup_flags & FOLL_GET))
- return -EINVAL;
-
- gup_flags |= FOLL_PIN;
- return __get_user_pages_locked(current->mm, start, nr_pages,
- pages, NULL, locked,
- gup_flags | FOLL_TOUCH);
-}
-EXPORT_SYMBOL(pin_user_pages_locked);
--
2.35.1
Now that the last caller of get_user_pages_locked() is gone, remove it.
Reviewed-by: Christoph Hellwig <[email protected]>
Reviewed-by: Jan Kara <[email protected]>
Reviewed-by: Jason Gunthorpe <[email protected]>
Reviewed-by: Claudio Imbrenda <[email protected]>
Signed-off-by: John Hubbard <[email protected]>
---
include/linux/mm.h | 2 --
mm/gup.c | 59 ----------------------------------------------
2 files changed, 61 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 80c540c17d83..528ef1cb4f3a 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1916,8 +1916,6 @@ long get_user_pages(unsigned long start, unsigned long nr_pages,
long pin_user_pages(unsigned long start, unsigned long nr_pages,
unsigned int gup_flags, struct page **pages,
struct vm_area_struct **vmas);
-long get_user_pages_locked(unsigned long start, unsigned long nr_pages,
- unsigned int gup_flags, struct page **pages, int *locked);
long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
struct page **pages, unsigned int gup_flags);
long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages,
diff --git a/mm/gup.c b/mm/gup.c
index 6664dfa7a38f..7337520afa49 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2145,65 +2145,6 @@ long get_user_pages(unsigned long start, unsigned long nr_pages,
}
EXPORT_SYMBOL(get_user_pages);
-/**
- * get_user_pages_locked() - variant of get_user_pages()
- *
- * @start: starting user address
- * @nr_pages: number of pages from start to pin
- * @gup_flags: flags modifying lookup behaviour
- * @pages: array that receives pointers to the pages pinned.
- * Should be at least nr_pages long. Or NULL, if caller
- * only intends to ensure the pages are faulted in.
- * @locked: pointer to lock flag indicating whether lock is held and
- * subsequently whether VM_FAULT_RETRY functionality can be
- * utilised. Lock must initially be held.
- *
- * It is suitable to replace the form:
- *
- * mmap_read_lock(mm);
- * do_something()
- * get_user_pages(mm, ..., pages, NULL);
- * mmap_read_unlock(mm);
- *
- * to:
- *
- * int locked = 1;
- * mmap_read_lock(mm);
- * do_something()
- * get_user_pages_locked(mm, ..., pages, &locked);
- * if (locked)
- * mmap_read_unlock(mm);
- *
- * We can leverage the VM_FAULT_RETRY functionality in the page fault
- * paths better by using either get_user_pages_locked() or
- * get_user_pages_unlocked().
- *
- */
-long get_user_pages_locked(unsigned long start, unsigned long nr_pages,
- unsigned int gup_flags, struct page **pages,
- int *locked)
-{
- /*
- * FIXME: Current FOLL_LONGTERM behavior is incompatible with
- * FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on
- * vmas. As there are no users of this flag in this call we simply
- * disallow this option for now.
- */
- if (WARN_ON_ONCE(gup_flags & FOLL_LONGTERM))
- return -EINVAL;
- /*
- * FOLL_PIN must only be set internally by the pin_user_pages*() APIs,
- * never directly by the caller, so enforce that:
- */
- if (WARN_ON_ONCE(gup_flags & FOLL_PIN))
- return -EINVAL;
-
- return __get_user_pages_locked(current->mm, start, nr_pages,
- pages, NULL, locked,
- gup_flags | FOLL_TOUCH);
-}
-EXPORT_SYMBOL(get_user_pages_locked);
-
/*
* get_user_pages_unlocked() is suitable to replace the form:
*
--
2.35.1
Remove a quirky special case from follow_pfn_pte(), and adjust its
callers to match. Caller changes include:
__get_user_pages(): Regardless of any FOLL_* flags, get_user_pages() and
its variants should handle PFN-only entries by stopping early, if the
caller expected **pages to be filled in. This makes for a more reliable
API, as compared to the previous approach of skipping over such entries
(and thus leaving them silently unwritten).
move_pages(): squash the -EEXIST error return from follow_page() into
-EFAULT, because -EFAULT is listed in the man page, whereas -EEXIST is
not.
Cc: Peter Xu <[email protected]>
Cc: Lukas Bulwahn <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: Claudio Imbrenda <[email protected]>
Reviewed-by: Christoph Hellwig <[email protected]>
Reviewed-by: Jan Kara <[email protected]>
Suggested-by: Jason Gunthorpe <[email protected]>
Signed-off-by: John Hubbard <[email protected]>
---
mm/gup.c | 13 ++++++++-----
mm/migrate.c | 7 +++++++
2 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 80229ecf0114..2df0d0103c43 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -464,10 +464,6 @@ static struct page *no_page_table(struct vm_area_struct *vma,
static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address,
pte_t *pte, unsigned int flags)
{
- /* No page to get reference */
- if (flags & (FOLL_GET | FOLL_PIN))
- return -EFAULT;
-
if (flags & FOLL_TOUCH) {
pte_t entry = *pte;
@@ -1205,8 +1201,15 @@ static long __get_user_pages(struct mm_struct *mm,
} else if (PTR_ERR(page) == -EEXIST) {
/*
* Proper page table entry exists, but no corresponding
- * struct page.
+ * struct page. If the caller expects **pages to be
+ * filled in, bail out now, because that can't be done
+ * for this page.
*/
+ if (pages) {
+ ret = PTR_ERR(page);
+ goto out;
+ }
+
goto next_page;
} else if (IS_ERR(page)) {
ret = PTR_ERR(page);
diff --git a/mm/migrate.c b/mm/migrate.c
index c7da064b4781..be0d5ae36dc1 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1761,6 +1761,13 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,
continue;
}
+ /*
+ * The move_pages() man page does not have an -EEXIST choice, so
+ * use -EFAULT instead.
+ */
+ if (err == -EEXIST)
+ err = -EFAULT;
+
/*
* If the page is already on the target node (!err), store the
* node, otherwise, store the err.
--
2.35.1
From: Peter Xu <[email protected]>
Alex reported invalid page pointer returned with pin_user_pages_remote()
from vfio after upstream commit 4b6c33b32296 ("vfio/type1: Prepare for
batched pinning with struct vfio_batch").
It turns out that it's not the fault of the vfio commit; however after
vfio switches to a full page buffer to store the page pointers it starts
to expose the problem easier.
The problem is for VM_PFNMAP vmas we should normally fail with an
-EFAULT then vfio will carry on to handle the MMIO regions. However
when the bug triggered, follow_page_mask() returned -EEXIST for such a
page, which will jump over the current page, leaving that entry in
**pages untouched. However the caller is not aware of it, hence the
caller will reference the page as usual even if the pointer data can be
anything.
We had that -EEXIST logic since commit 1027e4436b6a ("mm: make GUP
handle pfn mapping unless FOLL_GET is requested") which seems very
reasonable. It could be that when we reworked GUP with FOLL_PIN we
could have overlooked that special path in commit 3faa52c03f44 ("mm/gup:
track FOLL_PIN pages"), even if that commit rightfully touched up
follow_devmap_pud() on checking FOLL_PIN when it needs to return an
-EEXIST.
Attaching the Fixes to the FOLL_PIN rework commit, as it happened later
than 1027e4436b6a.
Cc: Andrea Arcangeli <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Kirill A. Shutemov <[email protected]>
Fixes: 3faa52c03f44 ("mm/gup: track FOLL_PIN pages")
Reviewed-by: Claudio Imbrenda <[email protected]>
Reviewed-by: Christoph Hellwig <[email protected]>
Reviewed-by: Jan Kara <[email protected]>
Reported-by: Alex Williamson <[email protected]>
Debugged-by: Alex Williamson <[email protected]>
Tested-by: Alex Williamson <[email protected]>
Signed-off-by: Peter Xu <[email protected]>
[jhubbard: added some tags, removed a reference to an out of tree
module, and reflowed to 72 columns.]
Signed-off-by: John Hubbard <[email protected]>
---
mm/gup.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/gup.c b/mm/gup.c
index a9d4d724aef7..80229ecf0114 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -465,7 +465,7 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address,
pte_t *pte, unsigned int flags)
{
/* No page to get reference */
- if (flags & FOLL_GET)
+ if (flags & (FOLL_GET | FOLL_PIN))
return -EFAULT;
if (flags & FOLL_TOUCH) {
--
2.35.1
On 2/11/22 16:27, Andrew Morton wrote:
...
>> Hi Andrew,
>>
>> Is this OK to put in mmotm yet? All the patches here have been reviewed.
>> And this small series also does not conflict with Willy's "[PATCH 00/75]
>> MM folio patches for 5.18" series.
>
> The v4 series is is -mm. Shall update to v5 soon...
Oh, sorry for the noise, then. Somehow I did not see the usual notification
emails this time around, I'm not sure what happened there.
thanks,
--
John Hubbard
NVIDIA
On Fri, 11 Feb 2022 16:24:15 -0800 John Hubbard <[email protected]> wrote:
> On 2/6/22 22:22, John Hubbard wrote:
> > Hi,
> >
> > Here's an update, mainly just collecting reviewed-by tags:
> >
> > Changes since v4:
> >
> > * Patch 1: reflowed the commit description to 72 columns.
> >
> > * Added reviewed-by's from Christoph and Jan for remaining patches.
> >
> > * Rebased onto 5.17-rc3.
> >
>
> Hi Andrew,
>
> Is this OK to put in mmotm yet? All the patches here have been reviewed.
> And this small series also does not conflict with Willy's "[PATCH 00/75]
> MM folio patches for 5.18" series.
The v4 series is is -mm. Shall update to v5 soon...
On 2/6/22 22:22, John Hubbard wrote:
> Hi,
>
> Here's an update, mainly just collecting reviewed-by tags:
>
> Changes since v4:
>
> * Patch 1: reflowed the commit description to 72 columns.
>
> * Added reviewed-by's from Christoph and Jan for remaining patches.
>
> * Rebased onto 5.17-rc3.
>
Hi Andrew,
Is this OK to put in mmotm yet? All the patches here have been reviewed.
And this small series also does not conflict with Willy's "[PATCH 00/75]
MM folio patches for 5.18" series.
thanks,
John Hubbard
>
> Changes since v3:
>
> * Patch 1: Commit log: removed a reference to an out of tree module,
> and updated tags and added a short note to that effect.
>
> * Patch 2: Fixed up do_move_pages(), to squash -EEXIST returns from
> follow_page() into something that move_pages() allows: -EFAULT.
> This is from reviews by Jan Kara, Matthew Wilcox, Jason Gunthorpe.
>
> Removed all Reviewed-by tags for this patch, due to that update.
>
> * Patch 4: Split into two patches, as recommended by Christoph
> Hellwig. Left the Reviewed-by tags intact because no code changes
> were required.
>
> * Added Reviewed-by tags from Claudio Imbrenda.
>
> Changes since v2:
>
> * Patch 2: Removed an unnecessary line that was causing a
> clang-analyzer complaint, as reported by Lukas Bulwahn
> [1] (thanks!), and retested locally.
>
> Jason: I've boldly left your Reviewed-by tag on this patch,
> because I am predicting you'll agree with it...
>
> * Added Reviewed-by tags from Jan Kara, Christoph Hellwig, and
> Jason Gunthorpe that have collected since v2.
>
>
> Changes since v1:
> * Patch 4: changed from get_user_pages(), to get_user_pages_fast().
>
> * Patch 4: Rewrote the commit description--thanks to Jan Kara for
> that feedback.
>
> * Patch 1: Removed Jerome's Cc from patch 1, due to a stale email
> address.
>
> * Added Reviewed-by's from David Hildenbrand and Jason Gunthorpe.
>
> Original cover letter, updated as necessary:
>
> I'm including Peter's patch as the first one in this tiny series. (The
> commit description has my r-b tag in place of my Cc, and removes
> Jerome's Cc because he is no longer at redhat.com) The second patch is
> what I had in mind for a follow-up to that, when we were discussing that
> fix [2].
>
> Plus, a couple more small removals that I had queued up:
>
> The third patch removes a completely unused routine:
> pin_user_pages_locked().
>
> The forth patch removes a similar routine, get_user_pages_locked(), that
> only has one caller. It now calls get_user_pages_fast(), instead.
>
> v1 of this patchset is here:
> https://lore.kernel.org/all/[email protected]/
>
> v2:
> https://lore.kernel.org/r/[email protected]
>
> v3:
> https://lore.kernel.org/r/[email protected]
>
> v4:
> https://lore.kernel.org/r/[email protected]
>
>
> [1] https://lore.kernel.org/r/CAKXUXMxFK9bo8jDoRZbQ0r2j-JwAGg3Xc5cpAcLaHfwHddJ7ew@mail.gmail.com
>
> [2] https://lore.kernel.org/all/[email protected]/
>
>
> John Hubbard (4):
> mm/gup: follow_pfn_pte(): -EEXIST cleanup
> mm/gup: remove unused pin_user_pages_locked()
> mm: change lookup_node() to use get_user_pages_fast()
> mm/gup: remove unused get_user_pages_locked()
>
> Peter Xu (1):
> mm: Fix invalid page pointer returned with FOLL_PIN gups
>
> include/linux/mm.h | 4 --
> mm/gup.c | 101 ++++-----------------------------------------
> mm/mempolicy.c | 21 ++++------
> mm/migrate.c | 7 ++++
> 4 files changed, 24 insertions(+), 109 deletions(-)
>
>
> base-commit: dfd42facf1e4ada021b939b4e19c935dcdd55566