Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc:     Jiaqi Yan <jiaqiyan@google.com>,
        Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        Muchun Song <songmuchun@bytedance.com>,
        Miaohe Lin <linmiaohe@huawei.com>,
        Axel Rasmussen <axelrasmussen@google.com>,
        James Houghton <jthoughton@google.com>,
        Michal Hocko <mhocko@suse.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>
Subject: [PATCH v2 2/2] hugetlb: optimize update_and_free_pages_bulk to avoid lock cycles
Date:   Mon, 17 Jul 2023 17:49:42 -0700
Message-ID: <20230718004942.113174-3-mike.kravetz@oracle.com>
In-Reply-To: <20230718004942.113174-1-mike.kravetz@oracle.com>
References: <20230718004942.113174-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?bMmnxjIDcqNnb/iYJoQ241b+5iEnY98WoX/AKOg0gPnYQM6U962+6DW644gY?=
 =?us-ascii?Q?BVA/u7KEIsjWR3IgMkH0a96lfNC1Drr4ORuJrMVFnw4dXXJ0ZokgZtavh1Xf?=
 =?us-ascii?Q?Kzei8mAUGzFcfXrO5N/2cohfC2skoihlVBsEt7ipwWkB+L75VGvKdckeqdFB?=
 =?us-ascii?Q?Bogb4JUTmqW976CivQDy3kLKinNnzFjS7tVzLRPabvQ0ZeudQJWtfTiWx+No?=
 =?us-ascii?Q?YjX29QD7CcaUyl0YToXjl20O1/gXWoIPJatDmK4A9uGRCaAURuRS2Neo7gr2?=
 =?us-ascii?Q?7qHdiEyHJs7j/drz1FzhU3IKBsw9FZzAU8bhldrdRp4fTCwZatoto8mlInpc?=
 =?us-ascii?Q?Y15+gJExdDJ/Y54DYtqwPwbnueK7Eg6zI9KOAcpb+0KzIqHMSmedKELSOla7?=
 =?us-ascii?Q?jwUQ6vZjxAfw2MDEpS7M3xG2TcQs074QcIjTUaUtxKpGuHxRySk0G5MWxR1D?=
 =?us-ascii?Q?cFnfuOBWep/oJNdpIxhw7fiy+CUBXL873h4TB5tjewuQUoLQtM/HDoUthrpi?=
 =?us-ascii?Q?XoVpsfED1LrHQZdkoV+TFv4R9XLExz32z16Bd9TJUIxR6O0ujlhDCEOWroz2?=
 =?us-ascii?Q?qh+SpE4BG0pXELa3Dit+zWycs34tJ5BLDKUsrWwmoM/BezqCc5AcGQDIpjau?=
 =?us-ascii?Q?171Z81L2UUwi5UzkShHh+yDnag2DdAjiaA0+D+6n/ZCei36BR9EWVASR0j4U?=
 =?us-ascii?Q?d0s+/STZqp1EJdjD05Nht4kaKqvgydu2QWLc7ZkXKUB/lln4T6wg0/GPVnob?=
 =?us-ascii?Q?HVKE2uYJF0Ps8US9xtduDbFZxTDo/l1A+Y9KloisNZJngw/r6ljQ69oQ63zf?=
 =?us-ascii?Q?pqVLkuXKMxYTiHO5poD18sGOCYYcJDI6Eu4MSRVcQxnjLFRLDWh56m9SkZVv?=
 =?us-ascii?Q?7YnxXZagmhuoLj6KXTT5ML/COLLUowhQUNxzx54XuBd4DlGFU/UMiTkitI0s?=
 =?us-ascii?Q?XteIXQWVwFVv/9wtQQvEaSg+41VP/5ENfG/PvblSl3APAoVf9KzWQtrCP7+M?=
 =?us-ascii?Q?XuSzCbT2rMtV3KDnNMvMtvNGffhKyr1AdfPysBPTf+2fBoNNKA1XzXoYQP3E?=
 =?us-ascii?Q?Cw4F1x6HHyXHdsrT0MR8khvsZtAonQghSuhZuyqyEs8oDyJc7p651Dt3VNas?=
 =?us-ascii?Q?FV9FAElB8ckx5NEj/h4ycDluXfjmGttrfW9bN2buXyUDgNWJyo6UKQRq64Pn?=
 =?us-ascii?Q?sxef7OBz6tLKk0hm0hZJ4lY4KorZRimWvUebFqjD44NvDyaGwpgsI5wIEDca?=
 =?us-ascii?Q?WCt+DUssb21OEjgE0n9FpL8fvS7mssy3rZmxbXcb/2c9gnCpvZapwIhXf0zV?=
 =?us-ascii?Q?MEWKJ2T9ZCb4/rhu84yyGNe2V5x0nKUXQA1uT5WT7MtepC9FEyUF6IQUOq2b?=
 =?us-ascii?Q?LupDhprwgmY2Foo4HZzQKlFA7JZgyhlo/9bftrq9QcOvF47j1oP3fo2kiNt5?=
 =?us-ascii?Q?llZ89KG51/cZzcj5rcvEwY4mdHH+vBo6tRgRJIUOAQDnHbpmClqQj1YzGtDW?=
 =?us-ascii?Q?/ssKAFXlMNuglwjvuJfd1hFBQKtvcLa/NYAdEHQ+V772Ac3DPL2zUVcgzT0l?=
 =?us-ascii?Q?pmm/NLBgVzGY7MWazbWNWV+ZlEZJkIc755zVy++HIrikb9eAguece4gx4vfu?=
 =?us-ascii?Q?4A=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?eXvGaZXsU6P8RGNnAq+SjTN7Ht4qX2IINgOLXa+slLloCtYEKDcCqOkY5yoB?=
 =?us-ascii?Q?AvUEuvEVZQ3oPg67ldsXZq0Ah9XRhmugFLaMbH1dnhehmRVpWUx8hcA76DYi?=
 =?us-ascii?Q?XoKhTq8rmwRMK9freKzDHIsgDFniDVICKYBYAYiGXVd8uCyJIB0nyWqXlM86?=
 =?us-ascii?Q?QmSjlR8Sz7Gmlqo6FCAdnjMd/qPU5FKNk5IJE+h3XFt245mr8gsjewZK0C2t?=
 =?us-ascii?Q?RjQms4kjafFsnmljuGVzyjR/FITt8XEht6y+mXygFijKFKB83DWz7efPg4/n?=
 =?us-ascii?Q?X8lD5Xvd0jWCvwlCKpiBccj2RBLzxxXgZYde49NVWwKVmIgwEiqxiVAmGbus?=
 =?us-ascii?Q?eznPQuYRvJz5riGaMoWJAj+i6Rs8vLZSe6SIjDhqC35/UG/O3htmvM82vjCM?=
 =?us-ascii?Q?El5yuTJYN5G7uj3N0N19qpRT8RJjLXivaQ+n3rEAUQx9M3HVCFKCw9C6VLuv?=
 =?us-ascii?Q?xXs+JkKkrAwMhv3zS5V0YzNs+DKyrfzAgYaynm+zYYfX5EKsTmmJBktaZ2rW?=
 =?us-ascii?Q?Elrx9l02Q3mVrqItVVN2tv/mdQTCENoN+GKz7X6nrFN9EUSlb1nk6mStxBVw?=
 =?us-ascii?Q?TxSczZ02EK+z+WD2N+Pmx9tG5nIFaTmlrpfynutM9zsriYmuS8Nq76eyUQvk?=
 =?us-ascii?Q?KNZzhZrBBdSBe06GhplFZxtLbtFohXrJA8oXTvFIf/nOiWPVhPYOE45SffuR?=
 =?us-ascii?Q?VdCcth13qgccp52SfX8sSISmryjuN8sKkZ6gU40t+3CPVfj1frVJWWRrIgMV?=
 =?us-ascii?Q?zVQtddV3AG9XuJnF35aNOBT+PdzKFVA7WaI9dc7fnbXvsSqqjng3+KOPNsYJ?=
 =?us-ascii?Q?PbPEaEv7tEiRzA13T8oSlXPZe79TCSP+3jjDyzBC0JQuCfbC4FW3WCB+zB+Y?=
 =?us-ascii?Q?2PuG5lR9jeYTBSB4f5NLW3OIMDciGcpQIZQ8AfYbnf2/uN61vn+Y+3EwyZoy?=
 =?us-ascii?Q?LZftOMOcgtTcm88wgStA+Q1DKvy+L0bSB2E3pftqyNjxTEP71n0PvKLBSy1D?=
 =?us-ascii?Q?X2LM?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5b0a0528-001c-4cd6-4f73-08db8728e5e5
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2023 00:49:52.9624
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CMJuAxBUdTfDB1y7JACY4im6hyYsSnJm9J9SCnBJfCl+3NrHOo2Xx8c/GaRYxHLKIB8VXMNfSymnH7KbzQfs7g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB6193
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-07-17_15,2023-07-13_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0
 spamscore=0 phishscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000
 definitions=main-2307180006
X-Proofpoint-GUID: -sYHlFC7LOPP7X9cvRVh3iPq52MmGGLK
X-Proofpoint-ORIG-GUID: -sYHlFC7LOPP7X9cvRVh3iPq52MmGGLK
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

update_and_free_pages_bulk is designed to free a list of hugetlb pages
back to their associated lower level allocators.  This may require
allocating vmemmmap pages associated with each hugetlb page.  The
hugetlb page destructor must be changed before pages are freed to lower
level allocators.  However, the destructor must be changed under the
hugetlb lock.  This means there is potentially one lock cycle per page.

Minimize the number of lock cycles in update_and_free_pages_bulk by:
1) allocating necessary vmemmap for all hugetlb pages on the list
2) take hugetlb lock and clear destructor for all pages on the list
3) free all pages on list back to low level allocators

Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
 mm/hugetlb.c | 38 ++++++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 4a910121a647..e6b780291539 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1856,13 +1856,43 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio,
 static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list)
 {
 	struct page *page, *t_page;
-	struct folio *folio;
+	bool clear_dtor = false;
 
+	/*
+	 * First allocate required vmemmmap for all pages on list.  If vmemmap
+	 * can not be allocated, we can not free page to lower level allocator,
+	 * so add back as hugetlb surplus page.
+	 */
 	list_for_each_entry_safe(page, t_page, list, lru) {
-		folio = page_folio(page);
-		update_and_free_hugetlb_folio(h, folio, false);
-		cond_resched();
+		if (HPageVmemmapOptimized(page)) {
+			if (hugetlb_vmemmap_restore(h, page)) {
+				spin_lock_irq(&hugetlb_lock);
+				add_hugetlb_folio(h, page_folio(page), true);
+				spin_unlock_irq(&hugetlb_lock);
+			} else
+				clear_dtor = true;
+			cond_resched();
+		}
+	}
+
+	/*
+	 * If vmemmmap allocation performed above, then take lock to clear
+	 * destructor of all pages on list.
+	 */
+	if (clear_dtor) {
+		spin_lock_irq(&hugetlb_lock);
+		list_for_each_entry(page, list, lru)
+			__clear_hugetlb_destructor(h, page_folio(page));
+		spin_unlock_irq(&hugetlb_lock);
 	}
+
+	/*
+	 * Free pages back to low level allocators.  vmemmap and destructors
+	 * were taken care of above, so update_and_free_hugetlb_folio will
+	 * not need to take hugetlb lock.
+	 */
+	list_for_each_entry_safe(page, t_page, list, lru)
+		update_and_free_hugetlb_folio(h, page_folio(page), false);
 }
 
 struct hstate *size_to_hstate(unsigned long size)
-- 
2.41.0