Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc:     Muchun Song <songmuchun@bytedance.com>,
        Joao Martins <joao.m.martins@oracle.com>,
        Oscar Salvador <osalvador@suse.de>,
        David Hildenbrand <david@redhat.com>,
        Miaohe Lin <linmiaohe@huawei.com>,
        David Rientjes <rientjes@google.com>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        Barry Song <song.bao.hua@hisilicon.com>,
        Michal Hocko <mhocko@suse.com>,
        Matthew Wilcox <willy@infradead.org>,
        Xiongchun Duan <duanxiongchun@bytedance.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>
Subject: [PATCH v3 07/12] hugetlb: perform vmemmap optimization on a list of pages
Date:   Fri, 15 Sep 2023 15:15:40 -0700
Message-ID: <20230915221548.552084-8-mike.kravetz@oracle.com>
In-Reply-To: <20230915221548.552084-1-mike.kravetz@oracle.com>
References: <20230915221548.552084-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?TwEqCjTIvRg+vOBcMjMtCfh3HLe3aomuU2yTAhJXnrLWWbXAeJmOzf+il7Pg?=
 =?us-ascii?Q?b+cUM4uLkNXRc4/mVrRaDvm+MAOD32YARDCjA9aXX6G8Tyl+30FOe3yTmRyM?=
 =?us-ascii?Q?2t3mmDbtJbLWsh2sWnsgrToyIHflaeHVg8+jJsenyCf1wxXTSRbJtfpd5nGg?=
 =?us-ascii?Q?KtIVNFW5l4BF/Q6rYkNQ7qdKNCyx2b3IX0fqSv8taaIjVRjbxCsfRyk67PJX?=
 =?us-ascii?Q?lwK1NHcgUH9UW/CtfVhzqacsCBPuKSYHdIRD2XC01UlDN02HnZpEtJBNpExT?=
 =?us-ascii?Q?rqJQyKt7fTQgeVcUA6qIoW2q84+8ww8rk0WbE/Mn8LKubuozzXenzTV9+0JW?=
 =?us-ascii?Q?wz6PUjnlGC2k1agygF5vKinBbhWLk6wms6kKNwPf7UzBrqtmS3Ljn0G9FC1q?=
 =?us-ascii?Q?5Wf2pNTmqnHNIFygXXmhF+iR1yzYENc2bJ87vzC3YSGb9xCdT16W7xJ3UKbH?=
 =?us-ascii?Q?2dJDLaB7iI22q41DUqMHshjj7xVlUIIeXNv5PJDSh7mvujjBhe0K6kSfeJVX?=
 =?us-ascii?Q?CWlhCKu03YZz2SZrbGVELqyX5p1S37Dtn8Ld1eNR5u6LBVSxQ92wcyCNm8on?=
 =?us-ascii?Q?B3LYnMGFUNOjQzawdpI25Gr4R/jU5wbQ71I7kUPK8S7HbwGnmKWuUPbfWTqs?=
 =?us-ascii?Q?rSCiUiffxYS0RKiaiIGRTq3v1xHglVCwmrPAFTt6g6TrlBbPnHit6++cU3Sv?=
 =?us-ascii?Q?eZORfMoozHzRy8o0OWKTwlMOxTzMhuEzYNjEfKImtlj4UTH2Q6O60dluJpoV?=
 =?us-ascii?Q?pYdvFlGKHmdYbRm5mMy+vFpdLn7gXHu+afWgStiElUDEGNo4fq58BEsy8QkM?=
 =?us-ascii?Q?lDCUa2IBhcGaBTclV+3eUWv5LY5TyWZtb9Ju7pld2eWokcWmONViJfSI7hsn?=
 =?us-ascii?Q?o+9AW5DtNfSKdP993eNDoGuURdAuKhVr2sDotkIN6fSMMOrS+8SnQoA/gQnQ?=
 =?us-ascii?Q?orYd0qpB/bgIrAskaVcsr4QHaB3qF5X0KV7Y7BFhhphjg47/UmfCqu0FX+v3?=
 =?us-ascii?Q?WEbb78zJX89kPNNosnqiH00C3/i+k2vvG6+MPgxwZ9bWd2xmFxavjAsU01j1?=
 =?us-ascii?Q?MjWLWlTRQLLk3cYH/Q9ITevLsZ0O4OT1utgug2bvDZW9JwAmVbEwqRCqWaoM?=
 =?us-ascii?Q?VmVCb5fpvN7Tc2T00pLfqdB8OQiIEmIAzyzLcecnLBGStLDXG820Cif1XeMB?=
 =?us-ascii?Q?6A9oP+rIzTV9UuZtzDgng2X+13+tu7xSH8e2jHj3aHkspYQfh5PGL3cR12ea?=
 =?us-ascii?Q?YS4cAWZh4XSaKJAo3E4vdrKjA1zTjUDM5mCGiFlerSVpbp4EccbqsG/RpZ3O?=
 =?us-ascii?Q?x6cYAQocZVCvphgpT+1/qaAyRTm26md9BjpBmCjFlVu4Yi8NG14khMPl9yVu?=
 =?us-ascii?Q?gsP5pZjlgBedUc+HTzjzO9h3S+ObPLlZbtKOgIbw/rsMFyeVoO2XS/GtLOtm?=
 =?us-ascii?Q?iliwYkDbQNkkXqStee7Ir2NpzWZMLiBf3/UYpEqfGAxfoYX9n+GhR827ToVW?=
 =?us-ascii?Q?C4jWkrlJQPM8n/SdCnycI/lSnnx2k1c1Qzq2DOsSrSYNellcPXpJjONlvUvk?=
 =?us-ascii?Q?nUr8ELam+bAUCyYB+UMfg/2dyF8m4pEl/U5baryGBjxQE89gO2ZN/WPemlf9?=
 =?us-ascii?Q?tQ=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?1CN/tGvphJwXn3dpFfbR1jgD5CF0IOYxPvcvtYSJWfWs+94pqiu+a5g4xaQy?=
 =?us-ascii?Q?cSd5JkauATGKaZJJN/sH2mkzKapE+sOWjglGIaH7KJr1pPomRQ02H3h6poe4?=
 =?us-ascii?Q?gPeZYTiHDKFvBRu9BsDlc+TzaQSDp1evb9CFx30xCRJPhqKezGBcJnRg2n4Y?=
 =?us-ascii?Q?rvUNuYuuMCqzJkOS9TcNqN3v8IrBahpE/mdZtESqL3xb77zxqn50PFBY5dNq?=
 =?us-ascii?Q?61yht4IXwnXfJfuM5sswA+RPU7TZmbUvU36n46HkfA+8kwqTudQSYh/ZrkCZ?=
 =?us-ascii?Q?kN1iDYXcmaxLqN71x3Q3By9InviTS94E5jxxMVKteu5nN63xAORwPeSgkd7s?=
 =?us-ascii?Q?yKHg7N5Lut3UVCYUIM1miFs1PMfkjyHJybn9D/7REJyvCBJcW8jy3vuIvGYn?=
 =?us-ascii?Q?aYm9AkSf0Dmtn3KsiTekayLxvvHV9/MGpXrnnHhttFVwpDcgSnghfNWrDOU3?=
 =?us-ascii?Q?/0NbBWM3z7ofn0hEUEu/LZ+kTfqwWQSoueCT/4apyLZzvhD6pFiISJEm+LSs?=
 =?us-ascii?Q?bdt2Fco0TmhajrwcW1hDRQPx+z4hV35p1wR9rEljRie/iz54YzZJQg2omSwA?=
 =?us-ascii?Q?SCUilcwFnctSaT+go88ABjxiZsdWnG7TEj078TI1gnJ9MG/LDKkC4dmPy/TM?=
 =?us-ascii?Q?gUfLq4ddg/1zff8QdFiqj/b4NHHDbuvQU38yX66MEjl/RvjoHSow4iMcpa5M?=
 =?us-ascii?Q?h5s+E9XI7by41EBcN63f0grsqN1BJO6P1cj+vAkciKqlu1KV5PT1ty0JOc4A?=
 =?us-ascii?Q?NKLL7weFFGQQLursCzQNvte/DZmZThN5EzobBY39HYvDO23EpTcTmUQYr0ew?=
 =?us-ascii?Q?IsdRkBs1VcXs3tGgkUxE3RO4VCTfpe1hmgQFRdZoLh4ij8TJ2VjR6XmsFb1y?=
 =?us-ascii?Q?jWFqM8FhKGQkzq9plQIy7DTSnHRiKqoeqFgeQACc1d6Fh3fmbTTD5l6tAx8b?=
 =?us-ascii?Q?tEjNKC4SoZMdJK2udb8lEOkoriYlj2dIjn9uIxG2xBRdE9mprM9NZ9rsTgYV?=
 =?us-ascii?Q?W3A1S86WaTaEl5DgQYGJDVTgsJaJJLNbhEc4r4bW4/HJLVcrC9APofWYgyX0?=
 =?us-ascii?Q?fm1aF5Zb2//viap82b6yDlthKrFjVTCqf0OI6urN6NW/fEPOebXotuIGgZ/Z?=
 =?us-ascii?Q?YQZLNbepcL4a?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 25efbe30-169c-4b56-1ab4-08dbb6395e79
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 22:16:11.9181
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: H2SGDHWQr9kMEx/0piHnE+V065w8n32w9Sp487NfEmUqkgRuCUu0BdYODL1/7W5fBjoM1ydpoNL1um7vUBHYZw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5861
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26
 definitions=2023-09-15_19,2023-09-15_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0
 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2308100000 definitions=main-2309150200
X-Proofpoint-ORIG-GUID: 7eoyWQtg0vZ_C6KjTxBNa9nbQIokJEDs
X-Proofpoint-GUID: 7eoyWQtg0vZ_C6KjTxBNa9nbQIokJEDs
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 15 Sep 2023 15:17:54 -0700 (PDT)

When adding hugetlb pages to the pool, we first create a list of the
allocated pages before adding to the pool.  Pass this list of pages to a
new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization.

We also modify the routine vmemmap_should_optimize() to check for pages
that are already optimized.  There are code paths that might request
vmemmap optimization twice and we want to make sure this is not
attempted.

Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
Reviewed-by: Muchun Song <songmuchun@bytedance.com>
---
 mm/hugetlb.c         |  5 +++++
 mm/hugetlb_vmemmap.c | 11 +++++++++++
 mm/hugetlb_vmemmap.h |  5 +++++
 3 files changed, 21 insertions(+)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 77313c9e0fa8..214603898ad0 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2248,6 +2248,11 @@ static void prep_and_add_allocated_folios(struct hstate *h,
 {
 	struct folio *folio, *tmp_f;
 
+	/*
+	 * Send list for bulk vmemmap optimization processing
+	 */
+	hugetlb_vmemmap_optimize_folios(h, folio_list);
+
 	/*
 	 * Add all new pool pages to free lists in one lock cycle
 	 */
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 0bde38626d25..c17784f36dc3 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -483,6 +483,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)
 /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */
 static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head)
 {
+	if (HPageVmemmapOptimized((struct page *)head))
+		return false;
+
 	if (!READ_ONCE(vmemmap_optimize_enabled))
 		return false;
 
@@ -572,6 +575,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head)
 		SetHPageVmemmapOptimized(head);
 }
 
+void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
+{
+	struct folio *folio;
+
+	list_for_each_entry(folio, folio_list, lru)
+		hugetlb_vmemmap_optimize(h, &folio->page);
+}
+
 static struct ctl_table hugetlb_vmemmap_sysctls[] = {
 	{
 		.procname	= "hugetlb_optimize_vmemmap",
diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h
index 25bd0e002431..036494e040ca 100644
--- a/mm/hugetlb_vmemmap.h
+++ b/mm/hugetlb_vmemmap.h
@@ -13,6 +13,7 @@
 #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
 int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head);
 void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head);
+void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list);
 
 /*
  * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See
@@ -47,6 +48,10 @@ static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page
 {
 }
 
+static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
+{
+}
+
 static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h)
 {
 	return 0;
-- 
2.41.0