From: Peter Xu <[email protected]>
PageAnonExclusive() used to forbid tail pages for hugetlbfs, as that used
to be called mostly in hugetlb specific paths and the head page was
guaranteed.
As we move forward towards merging hugetlb paths into generic mm, we may
start to pass in tail hugetlb pages (when with cont-pte/cont-pmd huge
pages) for such check. Allow it to properly fetch the head, in which case
the anon-exclusiveness of the head will always represents the tail page.
There's already a sign of it when we look at the fast-gup which already
contain the hugetlb processing altogether: we used to have a specific
commit 5805192c7b72 ("mm/gup: handle cont-PTE hugetlb pages correctly in
gup_must_unshare() via GUP-fast") covering that area. Now with this more
generic change, that can also go away.
Signed-off-by: Peter Xu <[email protected]>
---
include/linux/page-flags.h | 8 +++++++-
mm/internal.h | 10 ----------
2 files changed, 7 insertions(+), 11 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 888353c209c0..225357f48a79 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
static __always_inline int PageAnonExclusive(const struct page *page)
{
VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
- VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
+ /*
+ * Allow the anon-exclusive check to work on hugetlb tail pages.
+ * Here hugetlb pages will always guarantee the anon-exclusiveness
+ * of the head page represents the tail pages.
+ */
+ if (PageHuge(page) && !PageHead(page))
+ page = compound_head(page);
return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
}
diff --git a/mm/internal.h b/mm/internal.h
index 9512de7398d5..87f6e4fd56a5 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1259,16 +1259,6 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))
smp_rmb();
- /*
- * During GUP-fast we might not get called on the head page for a
- * hugetlb page that is mapped using cont-PTE, because GUP-fast does
- * not work with the abstracted hugetlb PTEs that always point at the
- * head page. For hugetlb, PageAnonExclusive only applies on the head
- * page (as it cannot be partially COW-shared), so lookup the head page.
- */
- if (unlikely(!PageHead(page) && PageHuge(page)))
- page = compound_head(page);
-
/*
* Note that PageKsm() pages cannot be exclusive, and consequently,
* cannot get pinned.
--
2.44.0
On 03.04.24 03:32, [email protected] wrote:
> From: Peter Xu <[email protected]>
>
> PageAnonExclusive() used to forbid tail pages for hugetlbfs, as that used
> to be called mostly in hugetlb specific paths and the head page was
> guaranteed.
>
> As we move forward towards merging hugetlb paths into generic mm, we may
> start to pass in tail hugetlb pages (when with cont-pte/cont-pmd huge
> pages) for such check. Allow it to properly fetch the head, in which case
> the anon-exclusiveness of the head will always represents the tail page.
>
> There's already a sign of it when we look at the fast-gup which already
"GUP-fast" ;)
> contain the hugetlb processing altogether: we used to have a specific
> commit 5805192c7b72 ("mm/gup: handle cont-PTE hugetlb pages correctly in
> gup_must_unshare() via GUP-fast") covering that area. Now with this more
> generic change, that can also go away.
>
> Signed-off-by: Peter Xu <[email protected]>
> ---
> include/linux/page-flags.h | 8 +++++++-
> mm/internal.h | 10 ----------
> 2 files changed, 7 insertions(+), 11 deletions(-)
>
> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> index 888353c209c0..225357f48a79 100644
> --- a/include/linux/page-flags.h
> +++ b/include/linux/page-flags.h
> @@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
> static __always_inline int PageAnonExclusive(const struct page *page)
> {
> VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
> - VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
> + /*
> + * Allow the anon-exclusive check to work on hugetlb tail pages.
> + * Here hugetlb pages will always guarantee the anon-exclusiveness
> + * of the head page represents the tail pages.
> + */
> + if (PageHuge(page) && !PageHead(page))
> + page = compound_head(page);
> return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
> }
>
> diff --git a/mm/internal.h b/mm/internal.h
> index 9512de7398d5..87f6e4fd56a5 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -1259,16 +1259,6 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
> if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))
> smp_rmb();
>
> - /*
> - * During GUP-fast we might not get called on the head page for a
> - * hugetlb page that is mapped using cont-PTE, because GUP-fast does
> - * not work with the abstracted hugetlb PTEs that always point at the
> - * head page. For hugetlb, PageAnonExclusive only applies on the head
> - * page (as it cannot be partially COW-shared), so lookup the head page.
> - */
> - if (unlikely(!PageHead(page) && PageHuge(page)))
> - page = compound_head(page);
> -
> /*
> * Note that PageKsm() pages cannot be exclusive, and consequently,
> * cannot get pinned.
Acked-by: David Hildenbrand <[email protected]>
--
Cheers,
David / dhildenb
On Tue, Apr 02, 2024 at 09:32:47PM -0400, [email protected] wrote:
> +++ b/include/linux/page-flags.h
> @@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
> static __always_inline int PageAnonExclusive(const struct page *page)
> {
> VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
> - VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
> + /*
> + * Allow the anon-exclusive check to work on hugetlb tail pages.
> + * Here hugetlb pages will always guarantee the anon-exclusiveness
> + * of the head page represents the tail pages.
> + */
> + if (PageHuge(page) && !PageHead(page))
> + page = compound_head(page);
I think this should be written as:
/*
* HugeTLB stores this information on the head page; THP keeps
* it per page
*/
if (PageHuge(page))
page = compound_head(page);
> return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
> }
On Thu, Apr 04, 2024 at 01:05:57AM +0100, Matthew Wilcox wrote:
> On Tue, Apr 02, 2024 at 09:32:47PM -0400, [email protected] wrote:
> > +++ b/include/linux/page-flags.h
> > @@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
> > static __always_inline int PageAnonExclusive(const struct page *page)
> > {
> > VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
> > - VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
> > + /*
> > + * Allow the anon-exclusive check to work on hugetlb tail pages.
> > + * Here hugetlb pages will always guarantee the anon-exclusiveness
> > + * of the head page represents the tail pages.
> > + */
> > + if (PageHuge(page) && !PageHead(page))
> > + page = compound_head(page);
>
> I think this should be written as:
>
> /*
> * HugeTLB stores this information on the head page; THP keeps
> * it per page
> */
This comment does look clean and cleaner indeed. And yes, mention THP can
be helpful too.
> if (PageHuge(page))
> page = compound_head(page);
I would think PageHead() check would help us to avoid compound_head() on
heads, which should still be the majority cases iiuc (assuming page->flags
is already around in the cache anyway). I've no strong opinion though, as
I can hardly tell a difference in reality.
>
> > return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
> > }
>
Thanks,
--
Peter Xu
On Thu, Apr 04, 2024 at 03:06:25PM +0100, Matthew Wilcox wrote:
> On Thu, Apr 04, 2024 at 09:45:58AM -0400, Peter Xu wrote:
> > On Thu, Apr 04, 2024 at 01:05:57AM +0100, Matthew Wilcox wrote:
> > > if (PageHuge(page))
> > > page = compound_head(page);
> >
> > I would think PageHead() check would help us to avoid compound_head() on
> > heads, which should still be the majority cases iiuc (assuming page->flags
> > is already around in the cache anyway). I've no strong opinion though, as
> > I can hardly tell a difference in reality.
>
> compound_head() includes a check for PageHead(). Adding the check just
> makes things slower.
They check different fields (compound_head, offset 0x8 for the former).
Again I'm ok with either way to go and I don't expect measurable
difference.. so if you prefer this I'm happy with it too.
Andrew, could you help update with Matthew's fixup? The comment is
definitely better than what I wrote in all cases.
Thanks,
--
Peter Xu
On Thu, Apr 04, 2024 at 09:45:58AM -0400, Peter Xu wrote:
> On Thu, Apr 04, 2024 at 01:05:57AM +0100, Matthew Wilcox wrote:
> > if (PageHuge(page))
> > page = compound_head(page);
>
> I would think PageHead() check would help us to avoid compound_head() on
> heads, which should still be the majority cases iiuc (assuming page->flags
> is already around in the cache anyway). I've no strong opinion though, as
> I can hardly tell a difference in reality.
compound_head() includes a check for PageHead(). Adding the check just
makes things slower.
On Thu, 4 Apr 2024 10:21:21 -0400 Peter Xu <[email protected]> wrote:
> Andrew, could you help update with Matthew's fixup? The comment is
> definitely better than what I wrote in all cases.
From: Andrew Morton <[email protected]>
Subject: mm-allow-anon-exclusive-check-over-hugetlb-tail-pages-fix
Date: Thu Apr 4 01:27:47 PM PDT 2024
simplify PageAnonExclusive(), per Matthew
Link: https://lkml.kernel.org/r/[email protected]
Cc: David Hildenbrand <[email protected]>
Cc: Huacai Chen <[email protected]>
Cc: Jason Gunthorpe <[email protected]>
Cc: Matthew Wilcox (Oracle) <[email protected]>
Cc: Nathan Chancellor <[email protected]>
Cc: Peter Xu <[email protected]>
Cc: Ryan Roberts <[email protected]>
Cc: WANG Xuerui <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
---
include/linux/page-flags.h | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
--- a/include/linux/page-flags.h~mm-allow-anon-exclusive-check-over-hugetlb-tail-pages-fix
+++ a/include/linux/page-flags.h
@@ -1096,11 +1096,10 @@ static __always_inline int PageAnonExclu
{
VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
/*
- * Allow the anon-exclusive check to work on hugetlb tail pages.
- * Here hugetlb pages will always guarantee the anon-exclusiveness
- * of the head page represents the tail pages.
+ * HugeTLB stores this information on the head page; THP keeps it per
+ * page
*/
- if (PageHuge(page) && !PageHead(page))
+ if (PageHuge(page))
page = compound_head(page);
return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
}
_
On Thu, Apr 04, 2024 at 01:31:37PM -0700, Andrew Morton wrote:
> On Thu, 4 Apr 2024 10:21:21 -0400 Peter Xu <[email protected]> wrote:
>
> > Andrew, could you help update with Matthew's fixup? The comment is
> > definitely better than what I wrote in all cases.
>
> From: Andrew Morton <[email protected]>
> Subject: mm-allow-anon-exclusive-check-over-hugetlb-tail-pages-fix
> Date: Thu Apr 4 01:27:47 PM PDT 2024
>
> simplify PageAnonExclusive(), per Matthew
Thank you!
--
Peter Xu