Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc:     Muchun Song <songmuchun@bytedance.com>,
        Joao Martins <joao.m.martins@oracle.com>,
        Oscar Salvador <osalvador@suse.de>,
        David Hildenbrand <david@redhat.com>,
        Miaohe Lin <linmiaohe@huawei.com>,
        David Rientjes <rientjes@google.com>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        Barry Song <song.bao.hua@hisilicon.com>,
        Michal Hocko <mhocko@suse.com>,
        Matthew Wilcox <willy@infradead.org>,
        Xiongchun Duan <duanxiongchun@bytedance.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>
Subject: [PATCH v3 09/12] hugetlb: batch freeing of vmemmap pages
Date:   Fri, 15 Sep 2023 15:15:42 -0700
Message-ID: <20230915221548.552084-10-mike.kravetz@oracle.com>
In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com>
References: <20230915221548.552084-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?iFrVz4s6v4uIN5jTg6lJouiLySvuAWKJ+Rlt7377C+D469XIA0hFR4LJ6Gg9?=
 =?us-ascii?Q?WLtnEnXU854ZFbQKWNOwLcaLCmPXr2zi0I5PeMU2NqXt+S9BY8DSOuWGv1we?=
 =?us-ascii?Q?NFU/fH61ERr4glCFPZBvJfwO6XbBt+TQc6Wj35ARC0atva2A1PU9D8v/tTDm?=
 =?us-ascii?Q?5/QJGGP0EZVM+odr4NG6kiSIk/nlQe5zVGCV0HHcA/PaEzWg+txLznct2iET?=
 =?us-ascii?Q?7l3fS3xKC2Wxe2Pdba/Nsmov4slkUXs3b3BSqMJ2lnZ7+c2YW+8etaB9dJGc?=
 =?us-ascii?Q?V8PnbOA/QZFR3ZaOjdf0l1c0ODjaz1H6+51OBMQfQeMLvIBA3gPbY2hGUhiV?=
 =?us-ascii?Q?UfFraMe4hKbqduttUBnA0UnNK6Bv1GSVRZcRb0j2AQcDEie792kuacY4E7Tk?=
 =?us-ascii?Q?2B/MghaN7PX4q/3WgR8LXKZUU4nQkzkpzBdgZnMfNIN/yopIVSWBltSNVyEj?=
 =?us-ascii?Q?2PaJZIQHC52lko4eUsFQs0UUicvhSfr0229zzXtgOzrINptg11iPVB1Y5Rps?=
 =?us-ascii?Q?1NBchThJomiDK5Oqk7q3D2qlwbTITBLP/8cN8YtAGngx+DgvRsXeOeq5s8wf?=
 =?us-ascii?Q?uIr3Kuo1Dt+5Wc2uR7+dqYqEttXfFi+9KgKnkgm4/vrSQ+HnFUY6EALIb4ar?=
 =?us-ascii?Q?MU8Bzd8qy9sb/cGEB23IQzo70aoQUmf5N/+5UaAGbDGh5APSZsPEM543+WU3?=
 =?us-ascii?Q?l2njXYYFoG5IuypY/w7wzqTszJJVTVLp0J5ahkSvcplx+yzCjqi+6af5i5NY?=
 =?us-ascii?Q?VqXHVGaHqa00W1cSEa62pw6oEZMpCInV9itNxfxjAjoJCrbp+BLD56g/THqC?=
 =?us-ascii?Q?FBA8UTbuu0C18yDFEpu4Z6JD/T8Myet6Fl33x0nTsHMWlGdZX/ABvwG6spnX?=
 =?us-ascii?Q?Wc1ZPnWvxrEVyaRv5UfDTKhxcfY1Y8M4AB1TwgZrEpbgohBffTgALh119UV4?=
 =?us-ascii?Q?4cm36f23GyzX1qYzOoWaS48i4r3MpdGQtJdTk5XZpKt+LbUuEsmTMRZvzJvE?=
 =?us-ascii?Q?a0Ch/2QaJFe4V3r6K69zH2SYQxl1rxiG4fiiZsZqFax9xhpJRLX2AdmWgz4l?=
 =?us-ascii?Q?hwI2kPVSo+R9WrOfU96cPnPSf3b63uZ/1+S74Jcl893ll1YCEZAZDvXy04V0?=
 =?us-ascii?Q?yLayZZyUAsZFVo/SpgVonkT2CCbOokgDWlnLvuplr5lKWRD2xL61DEjmXtlS?=
 =?us-ascii?Q?D9OTKmiVm+HJ10j5+cstbiJF1zjHV33NdYPxI/i9e3/YDwMHTDofff+0FqsI?=
 =?us-ascii?Q?fctfc5pN384gsEXkEAk3vZ7xDyvfSc9Np49hIBlk5eM0ldNx15caoGdkCFQH?=
 =?us-ascii?Q?0OFVXW3a1dgbpIsUCnFKRHAh2Izqococ4j3miZO6gPJT3Moo91IXutx5npDy?=
 =?us-ascii?Q?pDjm7OthcLVoASIMIZRvj2IRn2CarVMJCn7aoZriH0czPrJkjRWNiq/B2iyk?=
 =?us-ascii?Q?ApxZpKG8yR/i5XYsCe+zbft/wukekWXWFmsOfhJMSubu8bjGL16N6PziPOPo?=
 =?us-ascii?Q?UNSP2i/kDIfZzgFCIbpYlQezQbHPhYAeS3X7dRRLh5aMPZaRwQ3m0K4tbgHW?=
 =?us-ascii?Q?7ryzLbo7LQxkOW6ZfSFbrt2W2+gLTUhdpQwiuN9LTwI4gNZi+ZT0Qfikb8xP?=
 =?us-ascii?Q?NA=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?ps0r0JVQXB4tlo465QFf+bb24uD88888b/BhSFqwyj/RPBKEiF3KM5F6yKAt?=
 =?us-ascii?Q?vZdNPPEPmR82LCVPsFyFwW1ZLBW/JXPOpw4Oi/yjUcHWaPdDf2ZyEfhlUaVK?=
 =?us-ascii?Q?2ZUycvdijipt8TBqyuTVWC3iFXhUFgO/bja9xnvX5bXBZVr6NoLmeUD/yZXj?=
 =?us-ascii?Q?xvBT1GNNIiWkIxE6YCgZjhi2yehnM8FdelsCOWvw0OWwEoOsOixx+Bk7fDou?=
 =?us-ascii?Q?+xW+rhdPVrtwQ2bTLl6ZsSZzJB70IfPfJmf0m1TnGN8TRUcA8oU+hBw/52ow?=
 =?us-ascii?Q?1TA5M22iomzJJKi2CORtEj2/6iWUBCokcnDzL1c+k9l7SbjtheARVTmrXU7C?=
 =?us-ascii?Q?RVQFYmo0mMLSkSb5lldTsSXNdK+da6hMylDjVCoekIkB8p/TB7k7GkjO62Ii?=
 =?us-ascii?Q?GicVyz2EtG0nY1CjkhFTV1nr9esTVGl5qbpN5PEHxAanx7ON2NtjCMJ9yjWN?=
 =?us-ascii?Q?BjR0uhP/PZiD2BMbqVgncjCMJgHpE8ZMn38lJTGVsEpqmrOXFwCfv5fOgrQJ?=
 =?us-ascii?Q?rdvgKIj8Sy1SXzRyTQk4Tf5U7LdNWH3i8w13jU5UgnvNJAQ1GuWeLYt5lphz?=
 =?us-ascii?Q?YIJCIK4IsBXzrQ6wKXEc7oAeeoB686aMWliXjx7RmjnCinIgpfpB8BdFsUFk?=
 =?us-ascii?Q?02rp3TYQ2sfc3CmQsjF9xaoflaxoUwOlatgrP+IvumVYsh38qAV1L+GDNsyF?=
 =?us-ascii?Q?Fpz/XLnJR/zb9Hlw9Dnu/De5j+57YxGXoBy+peCH5KhgR/2Kw2OtmxPGSsuh?=
 =?us-ascii?Q?WpZl3gdvK+OLl6tGxnGJQYCUJ1Ht/E2Q8Q/tDKvOvBtwZzgvv/GMY4lMK2AS?=
 =?us-ascii?Q?rLZwPfuCJnLSNYT8n/ipEUTuWwHZbdESL7uhbExEgLI3jTSuIZPP5NdJECH5?=
 =?us-ascii?Q?ZQWSVjK0zxzd72Ls/GYzlJ2rfpry9x7qMjhGSJM9AkSpKzEZGHa7Iq4ViWxU?=
 =?us-ascii?Q?snq3m3ul+AdZu47rUQapOOLoe8S3gijXo82+G2gyjNeDIM/kbFUmKIz+IlEF?=
 =?us-ascii?Q?ZehUgVnfvwt5v7s5Mf+8H82r2gyCYU+OYwZ2xnXe+CFdvSKOxnEwF+d2Cwsy?=
 =?us-ascii?Q?Sd4VE88nk6TAB/+dGhUrQLEl2qRstzZKuwfSSfAtUUP3NtEjkKgfx6HeGFEP?=
 =?us-ascii?Q?QM2JZE5fM4pn?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1e2c3b76-2789-44b8-2034-08dbb639618e
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:17.0488
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: f+qqFJ6mvOfgJ/Vf6y2CkI3FG7KFafnmZNedrdQMI/XYzIq9JeJzWIWp0vi1ocVzWfSpvwj7YKfqz3DAgkksOw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26
 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 phishscore=0
 spamscore=0 mlxlogscore=999 bulkscore=0 suspectscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000
 definitions=main-2309150200
X-Proofpoint-ORIG-GUID: GBV07K0Cu-JuI4Xj5UkNQK0it1kmM2rJ
X-Proofpoint-GUID: GBV07K0Cu-JuI4Xj5UkNQK0it1kmM2rJ
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 15 Sep 2023 15:17:52 -0700 (PDT)

Now that batching of hugetlb vmemmap optimization processing is possible,
batch the freeing of vmemmap pages.  When freeing vmemmap pages for a
hugetlb page, we add them to a list that is freed after the entire batch
has been processed.

This enhances the ability to return contiguous ranges of memory to the
low level allocators.

Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
 mm/hugetlb_vmemmap.c | 85 ++++++++++++++++++++++++++++++--------------
 1 file changed, 59 insertions(+), 26 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 0eeb503d8a4c..8f8a559ff6ac 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -222,6 +222,9 @@ static void free_vmemmap_page_list(struct list_head *list)
 {
 	struct page *page, *next;
 
+	if (list_empty(list))
+		return;
+
 	list_for_each_entry_safe(page, next, list, lru)
 		free_vmemmap_page(page);
 }
@@ -251,7 +254,7 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr,
 	}
 
 	entry = mk_pte(walk->reuse_page, pgprot);
-	list_add_tail(&page->lru, walk->vmemmap_pages);
+	list_add(&page->lru, walk->vmemmap_pages);
 	set_pte_at(&init_mm, addr, pte, entry);
 }
 
@@ -306,18 +309,20 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr,
  * @end:	end address of the vmemmap virtual address range that we want to
  *		remap.
  * @reuse:	reuse address.
+ * @vmemmap_pages: list to deposit vmemmap pages to be freed.  It is callers
+ *		responsibility to free pages.
  *
  * Return: %0 on success, negative error code otherwise.
  */
 static int vmemmap_remap_free(unsigned long start, unsigned long end,
-			      unsigned long reuse)
+			      unsigned long reuse,
+			      struct list_head *vmemmap_pages)
 {
 	int ret;
-	LIST_HEAD(vmemmap_pages);
 	struct vmemmap_remap_walk walk = {
 		.remap_pte	= vmemmap_remap_pte,
 		.reuse_addr	= reuse,
-		.vmemmap_pages	= &vmemmap_pages,
+		.vmemmap_pages	= vmemmap_pages,
 	};
 	int nid = page_to_nid((struct page *)start);
 	gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN;
@@ -334,7 +339,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end,
 	if (walk.reuse_page) {
 		copy_page(page_to_virt(walk.reuse_page),
 			  (void *)walk.reuse_addr);
-		list_add(&walk.reuse_page->lru, &vmemmap_pages);
+		list_add(&walk.reuse_page->lru, vmemmap_pages);
 	}
 
 	/*
@@ -365,15 +370,13 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end,
 		walk = (struct vmemmap_remap_walk) {
 			.remap_pte	= vmemmap_restore_pte,
 			.reuse_addr	= reuse,
-			.vmemmap_pages	= &vmemmap_pages,
+			.vmemmap_pages	= vmemmap_pages,
 		};
 
 		vmemmap_remap_range(reuse, end, &walk);
 	}
 	mmap_read_unlock(&init_mm);
 
-	free_vmemmap_page_list(&vmemmap_pages);
-
 	return ret;
 }
 
@@ -389,7 +392,7 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end,
 		page = alloc_pages_node(nid, gfp_mask, 0);
 		if (!page)
 			goto out;
-		list_add_tail(&page->lru, list);
+		list_add(&page->lru, list);
 	}
 
 	return 0;
@@ -576,24 +579,17 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h
 	return true;
 }
 
-/**
- * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages.
- * @h:		struct hstate.
- * @head:	the head page whose vmemmap pages will be optimized.
- *
- * This function only tries to optimize @head's vmemmap pages and does not
- * guarantee that the optimization will succeed after it returns. The caller
- * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages
- * have been optimized.
- */
-void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head)
+static int __hugetlb_vmemmap_optimize(const struct hstate *h,
+					struct page *head,
+					struct list_head *vmemmap_pages)
 {
+	int ret = 0;
 	unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;
 	unsigned long vmemmap_reuse;
 
 	VM_WARN_ON_ONCE(!PageHuge(head));
 	if (!vmemmap_should_optimize(h, head))
-		return;
+		return ret;
 
 	static_branch_inc(&hugetlb_optimize_vmemmap_key);
 
@@ -603,21 +599,58 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head)
 
 	/*
 	 * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end)
-	 * to the page which @vmemmap_reuse is mapped to, then free the pages
-	 * which the range [@vmemmap_start, @vmemmap_end] is mapped to.
+	 * to the page which @vmemmap_reuse is mapped to.  Add pages previously
+	 * mapping the range to vmemmap_pages list so that they can be freed by
+	 * the caller.
 	 */
-	if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse))
+	ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages);
+	if (ret)
 		static_branch_dec(&hugetlb_optimize_vmemmap_key);
 	else
 		SetHPageVmemmapOptimized(head);
+
+	return ret;
+}
+
+/**
+ * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages.
+ * @h:		struct hstate.
+ * @head:	the head page whose vmemmap pages will be optimized.
+ *
+ * This function only tries to optimize @head's vmemmap pages and does not
+ * guarantee that the optimization will succeed after it returns. The caller
+ * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages
+ * have been optimized.
+ */
+void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head)
+{
+	LIST_HEAD(vmemmap_pages);
+
+	__hugetlb_vmemmap_optimize(h, head, &vmemmap_pages);
+	free_vmemmap_page_list(&vmemmap_pages);
 }
 
 void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
 {
 	struct folio *folio;
+	LIST_HEAD(vmemmap_pages);
 
-	list_for_each_entry(folio, folio_list, lru)
-		hugetlb_vmemmap_optimize(h, &folio->page);
+	list_for_each_entry(folio, folio_list, lru) {
+		int ret = __hugetlb_vmemmap_optimize(h, &folio->page,
+								&vmemmap_pages);
+
+		/*
+		 * Pages may have been accumulated, thus free what we have
+		 * and try again.
+		 */
+		if (ret == -ENOMEM) {
+			free_vmemmap_page_list(&vmemmap_pages);
+			INIT_LIST_HEAD(&vmemmap_pages);
+			__hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages);
+		}
+	}
+
+	free_vmemmap_page_list(&vmemmap_pages);
 }
 
 static struct ctl_table hugetlb_vmemmap_sysctls[] = {
-- 
2.41.0