============== OVERVIEW ===========================
Now that many hugetlb helper functions that deal with hugetlb specific
flags[1] and hugetlb cgroups[2] are converted to folios, higher level
allocation, prep, and freeing functions within hugetlb can also be
converted to operate in folios.
Patch 1 of this series implements the wrapper functions around setting
the compound destructor and compound order for a folio. Besides the user
added in patch 1, patch 2 and patch 9 also use these helper functions.
Patches 2-10 convert the higher level hugetlb functions to folios.
============== TESTING ===========================
LTP:
Ran 10 back to back rounds of the LTP hugetlb test suite.
Gigantic Huge Pages:
Test allocation and freeing via hugeadm commands:
hugeadm --pool-pages-min 1GB:10
hugeadm --pool-pages-min 1GB:0
Demote:
Demote 1 1GB hugepages to 512 2MB hugepages
echo 1 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
echo 1 > /sys/kernel/mm/hugepages/hugepages-1048576kB/demote
cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
# 512
cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
# 0
Rebased on 10/29/2022 mm-unstable
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
v1 -> v2:
- fix conflict with "mm,thp,rmap: simplify compound page mapcount handling"
v2 -> v3:
- v2 contained wrong version of patch 1
v3 -> v4:
- change instances of folio_{clear, set}_head() to __folio_{clear, set}_head()
- rebase on top of hugetlb: Fix __prep_compound_gigantic_page page flag setting
v4 -> v5:
- change comment with HPageVmemmapOptimized to hugetlb_vmemmap_optimized
per Tarun Sahu
- fix Smatch warning in patch 10 by reorganizing alloc_pool_huge_page
- fix NULL pointer dereference issue in patch 10 per John Hubbard
- use NULL rather than 0 in patch 6 per David Hildenbrand
Sidhartha Kumar (10):
mm: add folio dtor and order setter functions
mm/hugetlb: convert destroy_compound_gigantic_page() to folios
mm/hugetlb: convert dissolve_free_huge_page() to folios
mm/hugetlb: convert remove_hugetlb_page() to folios
mm/hugetlb: convert update_and_free_page() to folios
mm/hugetlb: convert add_hugetlb_page() to folios and add
hugetlb_cma_folio()
mm/hugetlb: convert enqueue_huge_page() to folios
mm/hugetlb: convert free_gigantic_page() to folios
mm/hugetlb: convert hugetlb prep functions to folios
mm/hugetlb: change hugetlb allocation functions to return a folio
include/linux/mm.h | 16 ++
mm/hugetlb.c | 409 ++++++++++++++++++++++-----------------------
2 files changed, 219 insertions(+), 206 deletions(-)
--
2.38.1
Make more progress on converting the free_huge_page() destructor to
operate on folios by converting update_and_free_page() to folios.
Signed-off-by: Sidhartha Kumar <[email protected]>
---
mm/hugetlb.c | 30 ++++++++++++++++--------------
1 file changed, 16 insertions(+), 14 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 90ba01a76f87..83777d1ccbf3 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1478,7 +1478,7 @@ static void __remove_hugetlb_folio(struct hstate *h, struct folio *folio,
* apply.
*
* This handles the case where more than one ref is held when and
- * after update_and_free_page is called.
+ * after update_and_free_hugetlb_folio is called.
*
* In the case of demote we do not ref count the page as it will soon
* be turned into a page of smaller size.
@@ -1609,7 +1609,7 @@ static void __update_and_free_page(struct hstate *h, struct page *page)
}
/*
- * As update_and_free_page() can be called under any context, so we cannot
+ * As update_and_free_hugetlb_folio() can be called under any context, so we cannot
* use GFP_KERNEL to allocate vmemmap pages. However, we can defer the
* actual freeing in a workqueue to prevent from using GFP_ATOMIC to allocate
* the vmemmap pages.
@@ -1657,11 +1657,11 @@ static inline void flush_free_hpage_work(struct hstate *h)
flush_work(&free_hpage_work);
}
-static void update_and_free_page(struct hstate *h, struct page *page,
+static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio,
bool atomic)
{
- if (!HPageVmemmapOptimized(page) || !atomic) {
- __update_and_free_page(h, page);
+ if (!folio_test_hugetlb_vmemmap_optimized(folio) || !atomic) {
+ __update_and_free_page(h, &folio->page);
return;
}
@@ -1672,16 +1672,18 @@ static void update_and_free_page(struct hstate *h, struct page *page,
* empty. Otherwise, schedule_work() had been called but the workfn
* hasn't retrieved the list yet.
*/
- if (llist_add((struct llist_node *)&page->mapping, &hpage_freelist))
+ if (llist_add((struct llist_node *)&folio->mapping, &hpage_freelist))
schedule_work(&free_hpage_work);
}
static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list)
{
struct page *page, *t_page;
+ struct folio *folio;
list_for_each_entry_safe(page, t_page, list, lru) {
- update_and_free_page(h, page, false);
+ folio = page_folio(page);
+ update_and_free_hugetlb_folio(h, folio, false);
cond_resched();
}
}
@@ -1751,12 +1753,12 @@ void free_huge_page(struct page *page)
if (folio_test_hugetlb_temporary(folio)) {
remove_hugetlb_folio(h, folio, false);
spin_unlock_irqrestore(&hugetlb_lock, flags);
- update_and_free_page(h, page, true);
+ update_and_free_hugetlb_folio(h, folio, true);
} else if (h->surplus_huge_pages_node[nid]) {
/* remove the page from active list */
remove_hugetlb_folio(h, folio, true);
spin_unlock_irqrestore(&hugetlb_lock, flags);
- update_and_free_page(h, page, true);
+ update_and_free_hugetlb_folio(h, folio, true);
} else {
arch_clear_hugepage_flags(page);
enqueue_huge_page(h, page);
@@ -2172,8 +2174,8 @@ int dissolve_free_huge_page(struct page *page)
spin_unlock_irq(&hugetlb_lock);
/*
- * Normally update_and_free_page will allocate required vmemmmap
- * before freeing the page. update_and_free_page will fail to
+ * Normally update_and_free_hugtlb_folio will allocate required vmemmmap
+ * before freeing the page. update_and_free_hugtlb_folio will fail to
* free the page if it can not allocate required vmemmap. We
* need to adjust max_huge_pages if the page is not freed.
* Attempt to allocate vmemmmap here so that we can take
@@ -2181,7 +2183,7 @@ int dissolve_free_huge_page(struct page *page)
*/
rc = hugetlb_vmemmap_restore(h, &folio->page);
if (!rc) {
- update_and_free_page(h, &folio->page, false);
+ update_and_free_hugetlb_folio(h, folio, false);
} else {
spin_lock_irq(&hugetlb_lock);
add_hugetlb_page(h, &folio->page, false);
@@ -2818,7 +2820,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page,
* Pages have been replaced, we can safely free the old one.
*/
spin_unlock_irq(&hugetlb_lock);
- update_and_free_page(h, old_page, false);
+ update_and_free_hugetlb_folio(h, old_folio, false);
}
return ret;
@@ -2827,7 +2829,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page,
spin_unlock_irq(&hugetlb_lock);
/* Page has a zero ref count, but needs a ref to be freed */
folio_ref_unfreeze(new_folio, 1);
- update_and_free_page(h, new_page, false);
+ update_and_free_hugetlb_folio(h, new_folio, false);
return ret;
}
--
2.38.1
On 11/29/22 14:50, Sidhartha Kumar wrote:
> Make more progress on converting the free_huge_page() destructor to
> operate on folios by converting update_and_free_page() to folios.
>
> Signed-off-by: Sidhartha Kumar <[email protected]>
> ---
> mm/hugetlb.c | 30 ++++++++++++++++--------------
> 1 file changed, 16 insertions(+), 14 deletions(-)
>
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 90ba01a76f87..83777d1ccbf3 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
<snip>
Minor nit, not worth fixing now. But, we should cleanup sometime. Comments
below still refer to pages when we are operating on folios.
Reviewed-by: Mike Kravetz <[email protected]>
--
Mike Kravetz
> @@ -2818,7 +2820,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page,
> * Pages have been replaced, we can safely free the old one.
> */
> spin_unlock_irq(&hugetlb_lock);
> - update_and_free_page(h, old_page, false);
> + update_and_free_hugetlb_folio(h, old_folio, false);
> }
>
> return ret;
> @@ -2827,7 +2829,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page,
> spin_unlock_irq(&hugetlb_lock);
> /* Page has a zero ref count, but needs a ref to be freed */
> folio_ref_unfreeze(new_folio, 1);
> - update_and_free_page(h, new_page, false);
> + update_and_free_hugetlb_folio(h, new_folio, false);
>
> return ret;
> }
> --
> 2.38.1
>