Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc:     Muchun Song <songmuchun@bytedance.com>,
        Joao Martins <joao.m.martins@oracle.com>,
        Konrad Dybcio <konradybcio@kernel.org>,
        Oscar Salvador <osalvador@suse.de>,
        David Hildenbrand <david@redhat.com>,
        Miaohe Lin <linmiaohe@huawei.com>,
        David Rientjes <rientjes@google.com>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        Barry Song <21cnbao@gmail.com>, Michal Hocko <mhocko@suse.com>,
        Matthew Wilcox <willy@infradead.org>,
        Xiongchun Duan <duanxiongchun@bytedance.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>
Subject: [PATCH v7 3/8] hugetlb: perform vmemmap optimization on a list of pages
Date:   Thu,  5 Oct 2023 20:20:05 -0700
Message-ID: <20231006032012.296473-4-mike.kravetz@oracle.com>
In-Reply-To: <20231006032012.296473-1-mike.kravetz@oracle.com>
References: <20231006032012.296473-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?FRTkTxz3gm90NtwPK7Nsq7Unn5wzfugGjK4veGra0BSBZ7zxXiATvg8YbT06?=
 =?us-ascii?Q?5YDGShCOL9J+86qRuSS62FdMQP3qKQGyCxUbUL9sfRxrkgYQ1qPp3uTnE9sl?=
 =?us-ascii?Q?xLEVRmtLtmgdhN8qpX5SvmTNqlotXiBmpLlkU9hQf/BxSi2ubHoDdYOszQ4R?=
 =?us-ascii?Q?zM+hWbL4qUg/uPRW7W9b0hYu4SG+hb+QOV2cE5Cu65DH8ZaRIIjoLw4tN/Kz?=
 =?us-ascii?Q?G8juY5yqedUjG1HXKKR48wuc0/88ANzKOyjFPgf2ItrxKixjCe7HKyYyFm1g?=
 =?us-ascii?Q?m8IzJxmalOHqE0+lSWmsJRc1qsApuX2jfP33JUerfKegUp0QSXIAhdD+n69w?=
 =?us-ascii?Q?E6PtopNXdRuJUocBCHAp92G9xkR1uKWach3H6Cu0bqjvFI4oYrhE03TCmdCS?=
 =?us-ascii?Q?ZKJFewe7VDRhXY0pmNh5EzNhaXY5RRX32P05kwROwza4r+nLYjyNXoXaxZl2?=
 =?us-ascii?Q?H7Wrp3LE9FsL8/WDMcXfx4GhlbcujaNDF/r4ODhvD6KDlZkD1BRkHeDfPbxW?=
 =?us-ascii?Q?4LZf5QCT2JQMvXDWl8ZMUyqr4OnZ21g22ygV8b1Zqm+sGfOgoyadPYYSmwwk?=
 =?us-ascii?Q?JIUXwekjednMWXHj+uG2Tx+dwRTG1rCyQ1mNLWqkdqAJiGCrapzRMHXiQInq?=
 =?us-ascii?Q?9um3UNrV0UrktJVUZhMvf2R1JFlFPrPz7ymWZ312dGNmNYqpYRW8uwQKZxwh?=
 =?us-ascii?Q?W/Z+kBf7WA5ts8B5UwXFyy03ypMpozd1iMZTU2M1wKwwrKb/FbSL36e6Hxoz?=
 =?us-ascii?Q?BCcMuzVCNt+3BC8UEocRvK0hNymTdrwalTDfN/ToOjRMkLsIc+kJCX8oCKdu?=
 =?us-ascii?Q?07EzJD925860m72BcVPmqMITrvnv3bP4m2Dggqjb5zWSvK9fzh/Zt9/1/OU6?=
 =?us-ascii?Q?4+fBvrQ7Kem1oZlLijOplhPbV0FL73HWoSgAFEQyOfKQfYzeyPWZzO8DrBmx?=
 =?us-ascii?Q?Zf4f4MTTz8tNJJWGFuj6bgupfQ+WgE7nLqTYyrhA4/c813A6tzLDGokBoqfF?=
 =?us-ascii?Q?08nW9xTOjjsBf8nKlc3CtYNSa6c+mbxySjf5Cf6fJSwzdXQbbMNI/6wpbcBw?=
 =?us-ascii?Q?nu0gtzYfpJBn9vC27KcItyyAEhWZUNXYnH4dlHCu0wvOmFY2+XFZ08y00QiN?=
 =?us-ascii?Q?c+32WG0gWcSNhIMAQiaz2dd+h2Tbw6eKVuKVVanDlOB3uPnwkCU3o/f11GGV?=
 =?us-ascii?Q?Lppih/xzqSsv/+XbJO/J/CoJNPneTzqRferChbTptpMzcv4YOf9AK4XrNL9Q?=
 =?us-ascii?Q?6xySZVgv95WhBZhUd+8hkAF1/Ulv+SgH2FbpcT/8BCwxFqLZAoPQ/zdpxzxN?=
 =?us-ascii?Q?adCAXvp/TAahNqS64kW2/PMs/w8OpsZ26ixw09VM1N/ewYEB16maP3+xK4EX?=
 =?us-ascii?Q?/SHxxpemRRT5jT1+dZqIhIPZMQAEt9HAo7fuzyP8EsfI6mZcJZEP3a9eoQcM?=
 =?us-ascii?Q?myGovmlyB2e5ACDb6uoxb/DqYl58os9wzA/wJaUI43o/FwGJScWG2CVSI/bb?=
 =?us-ascii?Q?YuPrh/VorxVzfblcwQLQeR0dRQBhk9pYwdfikcnKIK+YnmJaVgTZRsxsU1Ga?=
 =?us-ascii?Q?w3wcCGwdbrT+jc/T8n2ptNK2t3ddIVLG7z5j3s19?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?QHXCsKytGmI/APSq/c++lAA2f2V7cnUUEM1GzUgEFlKjvD0rxjT17crxRXZi?=
 =?us-ascii?Q?fMc15hFiV6t4msRckYMvkRvLkzlb03IBXXSyjp2sKjBlkLi62uAQdKldfBvj?=
 =?us-ascii?Q?IypI4AlhMPred/jFbugJ6ih2qUAyOHMh1+Vyd5tjQZp9ngUTax2S+uqcPL33?=
 =?us-ascii?Q?1Mwlvwr/kt631zuqGdxE5NPatfV5YDlV0oyIJan8e+jjhF5J3Jx+wp+f1bPo?=
 =?us-ascii?Q?2yMpoNfUZnIDD7QeagH5pKcrTe6bDEbuTPCkBARLcweJlEngGXIkqpKSb0ez?=
 =?us-ascii?Q?hjl7JKN+B2xsDo5MijwHqfH5ICIFqmqKFUvM/oRiLcFiF74VWInGCkaBsQXJ?=
 =?us-ascii?Q?U6foBCGtEdLNDH8CnDhYg5y4zo6lQdMMVKahkyTHOty0FZI071ZH3USramFs?=
 =?us-ascii?Q?HPB44kjSA85hnA2ROy2Vn834A//hUm3dY2xSWQDEMd/tEfPmCCVeVhxAmnF2?=
 =?us-ascii?Q?l6ecAMj78ShEfmP+6W5R7arIXVNy6q+xLs008ygdNh9842d5rPoTO+E1TA5E?=
 =?us-ascii?Q?2/WwAJS2JZlluL36124RrRjlCH1M94hSrmcQ6D+ABm0xMxtiwr3SXU6/sjeH?=
 =?us-ascii?Q?9nWuXU3u1V9KvoF6dbRxNYaAbHq88ldC7i9w6mbGRaHQnzmxmvNcfgKlo+UZ?=
 =?us-ascii?Q?WfTyFCMMm1fmgubGqfRNwBoIK5a5YGNSok5nkQiWE9yzmM48Qawd0evtwgd1?=
 =?us-ascii?Q?02dUjF3VbQMp4nVGnwCOwglzsEVKCoqts+cCw1pprKih6yZlcaF9pvbTBpcR?=
 =?us-ascii?Q?k4Leokh0ACrO8jS8dhxwxLznO5NQKrouZLlQhi5oKxss1iOi1r2vYAikcQ98?=
 =?us-ascii?Q?b36w9P7Ul5acVomjEpKmuI7uvK6r1F0/uOZr7bbaN3Icy888ax7g4mU+329W?=
 =?us-ascii?Q?9m0WGMkZwY/xLQUFVUs1ilDnUzX/QMen4UHxh+evvSa8EP/FV9Y3BxwjDvQG?=
 =?us-ascii?Q?O2f9BuK9jfDbhl/oRf+9Q5mDQSzZkH9mzYHuO+OeS4aBGNvO67rY4KYIBwCk?=
 =?us-ascii?Q?3YEgc44e3P4D2dSPmIDkkNJCiOrf/GDeYfbnfp0gCiWVHNyHkGCjPsl2EGlz?=
 =?us-ascii?Q?hDiFAC0Afvh5IKXq397M1jTrEkSXuqlhtT18bW6rJMN0YQjc2gY3RC/g4FYS?=
 =?us-ascii?Q?irDJc5OfsUjoEp8rlaaxwkof5i8u2GozYA=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2c44c973-ba7d-4d91-687c-08dbc61b3188
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 03:20:30.1724
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MMUbCMxxC+h7Un2yLot5dzYblEotCo47rUDWcQf2blekbtkPa/q4Imzg5W6km39qbyRTvUR/i/UcSGDHkK5r9Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6615
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 spamscore=0
 phishscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000
 definitions=main-2310060024
X-Proofpoint-ORIG-GUID: nE8fvQnH6Hbh3Ti7U5BawZ3ypMOM8UWy
X-Proofpoint-GUID: nE8fvQnH6Hbh3Ti7U5BawZ3ypMOM8UWy
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 05 Oct 2023 20:22:40 -0700 (PDT)

When adding hugetlb pages to the pool, we first create a list of the
allocated pages before adding to the pool.  Pass this list of pages to a
new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization.

Due to significant differences in vmemmmap initialization for bootmem
allocated hugetlb pages, a new routine prep_and_add_bootmem_folios
is created.

We also modify the routine vmemmap_should_optimize() to check for pages
that are already optimized.  There are code paths that might request
vmemmap optimization twice and we want to make sure this is not
attempted.

Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
Reviewed-by: Muchun Song <songmuchun@bytedance.com>
---
 mm/hugetlb.c         | 42 ++++++++++++++++++++++++++++++++++--------
 mm/hugetlb_vmemmap.c | 11 +++++++++++
 mm/hugetlb_vmemmap.h |  5 +++++
 3 files changed, 50 insertions(+), 8 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 4ccb54824daa..2df9435afa48 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2249,6 +2249,9 @@ static void prep_and_add_allocated_folios(struct hstate *h,
 {
 	struct folio *folio, *tmp_f;
 
+	/* Send list for bulk vmemmap optimization processing */
+	hugetlb_vmemmap_optimize_folios(h, folio_list);
+
 	/* Add all new pool pages to free lists in one lock cycle */
 	spin_lock_irq(&hugetlb_lock);
 	list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
@@ -3287,6 +3290,34 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
 	prep_compound_head((struct page *)folio, huge_page_order(h));
 }
 
+static void __init prep_and_add_bootmem_folios(struct hstate *h,
+					struct list_head *folio_list)
+{
+	struct folio *folio, *tmp_f;
+
+	/* Send list for bulk vmemmap optimization processing */
+	hugetlb_vmemmap_optimize_folios(h, folio_list);
+
+	/* Add all new pool pages to free lists in one lock cycle */
+	spin_lock_irq(&hugetlb_lock);
+	list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
+		if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
+			/*
+			 * If HVO fails, initialize all tail struct pages
+			 * We do not worry about potential long lock hold
+			 * time as this is early in boot and there should
+			 * be no contention.
+			 */
+			hugetlb_folio_init_tail_vmemmap(folio,
+					HUGETLB_VMEMMAP_RESERVE_PAGES,
+					pages_per_huge_page(h));
+		}
+		__prep_account_new_huge_page(h, folio_nid(folio));
+		enqueue_hugetlb_folio(h, folio);
+	}
+	spin_unlock_irq(&hugetlb_lock);
+}
+
 /*
  * Put bootmem huge pages into the standard lists after mem_map is up.
  * Note: This only applies to gigantic (order > MAX_ORDER) pages.
@@ -3307,7 +3338,7 @@ static void __init gather_bootmem_prealloc(void)
 		 * in this list.  If so, process each size separately.
 		 */
 		if (h != prev_h && prev_h != NULL)
-			prep_and_add_allocated_folios(prev_h, &folio_list);
+			prep_and_add_bootmem_folios(prev_h, &folio_list);
 		prev_h = h;
 
 		VM_BUG_ON(!hstate_is_gigantic(h));
@@ -3315,12 +3346,7 @@ static void __init gather_bootmem_prealloc(void)
 
 		hugetlb_folio_init_vmemmap(folio, h,
 					   HUGETLB_VMEMMAP_RESERVE_PAGES);
-		__prep_new_hugetlb_folio(h, folio);
-		/* If HVO fails, initialize all tail struct pages */
-		if (!HPageVmemmapOptimized(&folio->page))
-			hugetlb_folio_init_tail_vmemmap(folio,
-						HUGETLB_VMEMMAP_RESERVE_PAGES,
-						pages_per_huge_page(h));
+		init_new_hugetlb_folio(h, folio);
 		list_add(&folio->lru, &folio_list);
 
 		/*
@@ -3332,7 +3358,7 @@ static void __init gather_bootmem_prealloc(void)
 		cond_resched();
 	}
 
-	prep_and_add_allocated_folios(h, &folio_list);
+	prep_and_add_bootmem_folios(h, &folio_list);
 }
 
 static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 76682d1d79a7..4558b814ffab 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -483,6 +483,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)
 /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */
 static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head)
 {
+	if (HPageVmemmapOptimized((struct page *)head))
+		return false;
+
 	if (!READ_ONCE(vmemmap_optimize_enabled))
 		return false;
 
@@ -572,6 +575,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head)
 		SetHPageVmemmapOptimized(head);
 }
 
+void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
+{
+	struct folio *folio;
+
+	list_for_each_entry(folio, folio_list, lru)
+		hugetlb_vmemmap_optimize(h, &folio->page);
+}
+
 static struct ctl_table hugetlb_vmemmap_sysctls[] = {
 	{
 		.procname	= "hugetlb_optimize_vmemmap",
diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h
index 4573899855d7..c512e388dbb4 100644
--- a/mm/hugetlb_vmemmap.h
+++ b/mm/hugetlb_vmemmap.h
@@ -20,6 +20,7 @@
 #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
 int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head);
 void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head);
+void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list);
 
 static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h)
 {
@@ -48,6 +49,10 @@ static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page
 {
 }
 
+static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
+{
+}
+
 static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h)
 {
 	return 0;
-- 
2.41.0