Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc:     Muchun Song <songmuchun@bytedance.com>,
        Joao Martins <joao.m.martins@oracle.com>,
        Oscar Salvador <osalvador@suse.de>,
        David Hildenbrand <david@redhat.com>,
        Miaohe Lin <linmiaohe@huawei.com>,
        David Rientjes <rientjes@google.com>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        Barry Song <21cnbao@gmail.com>, Michal Hocko <mhocko@suse.com>,
        Matthew Wilcox <willy@infradead.org>,
        Xiongchun Duan <duanxiongchun@bytedance.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>
Subject: [PATCH v5 8/8] hugetlb: batch TLB flushes when restoring vmemmap
Date:   Sun, 24 Sep 2023 17:39:52 -0700
Message-ID: <20230925003953.142620-9-mike.kravetz@oracle.com>
In-Reply-To: <20230925003953.142620-1-mike.kravetz@oracle.com>
References: <20230925003953.142620-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tDKvKEUfyS3vqsfjPMflm7+M/+Ms9NnLXCWz2jMAhavmz2C8sJNuujTtWU9o?=
 =?us-ascii?Q?1V/rtn9W+BjZqth8/QoPn75AYLQadsrGHizpRlSgT9MB8Ux3YhDB/hy1W4HR?=
 =?us-ascii?Q?HRPbEd1xKVDD7OTFuyYcptE6rXD14s7n6ksJdx1F0vS7BTG3hg7mitzxtCRD?=
 =?us-ascii?Q?1ZoJGfNobr4iAVNPl7AqlBzHmdQF5epco7MTR6YnFAzXJB5jj3AMgBD6fWqc?=
 =?us-ascii?Q?ne6za5zj/tCsNI2HFDlr2dDaFrT1dNzJKVgm4xB9OsxjnunrQRli9Nv6SBiy?=
 =?us-ascii?Q?FUB6+/CONno29ZOO/KqjKEFkD0PyYqily6hy2/aF8GKCLGrt//Z1bbXKi1c5?=
 =?us-ascii?Q?hP4mWC+1tWCs7okUjVOAc+jMVmqTMVC6/xA0e78DWHTZxFg5tOYqDvFXPuKB?=
 =?us-ascii?Q?97Jb4LyJVY2bcuUzL7PZF/+oTdTWUNidBN2qSxtgu7EcCzNKjxcF1hPQnrCh?=
 =?us-ascii?Q?D/YIHMG9CsIfJ3PJgbATugxIOzwpo9x66oupilQ5JboAYmbSy1IW/OtqGwAJ?=
 =?us-ascii?Q?cd+mHUPoYGTShR0l5p/dblLO7Bjx+XV+IbNPnK1rGI0pNOkYDD2I5taNWwVE?=
 =?us-ascii?Q?HP1h57+hCiddBf4xb2xHEQSgP6O7+9+TzxmDEoT+xxQBrgzUGV2UDHTeKCZL?=
 =?us-ascii?Q?/+1bbGSP61kDrKS9FQN+AARSXvhubhbpdgsyqEXvBGZ2Y7xQL+Rw/SlH6qk4?=
 =?us-ascii?Q?sCfy9fJmX1d07hTUYkJdjVL1WTUMDZaK94/iCVqihjiAcNgbjySSxdSkorpg?=
 =?us-ascii?Q?EIL48TO9CMADdsWezlUr5fzFqwTeAXSexXlFR0/Zo7utzB4uz8GRoMFSuZ3N?=
 =?us-ascii?Q?r9KS6ZQt0boFDtafUMgBWDVMP5NY0y43TXdViPliDeSVz+iYT24mTTV5X8V5?=
 =?us-ascii?Q?GV6LFI/jSJ/0cZxQbaSJpxmzAxYmXQZM8/VzrdxGpLhSVoYgXzl3asC54H9B?=
 =?us-ascii?Q?qRbLCqEecZv7vftHV0DnbRBc0n0PlZbFtu/aefYuk3iZqnx2T/pgZ3oMdCFG?=
 =?us-ascii?Q?L+QopBReYkl1K5Z3CW/Az1SLw2i3FjVOEPTYvLjOPVlOo2UfIO4LWwKvby5u?=
 =?us-ascii?Q?qPE7WmH3MBfVq0/WGKl4EkFUOPCdAqMgEZD4/TqjEtC4Te87wL/dhXsuwRZU?=
 =?us-ascii?Q?4x9ohkJbrSrW8uKGuE7iH6pWGDMrTzSlk3u0CDC0jDraJrO3f+lF0LfCBmJ1?=
 =?us-ascii?Q?wLYHNQnWvdns2vHJn98k/9NmpcUvLPbVpkR2rce6DVJyC5m0jxu7aflqusY7?=
 =?us-ascii?Q?1eLBJFNL53SJEYzdR9xXSzqUIpugFWZn9bMhnrlxz1VyGTofMynppzw15301?=
 =?us-ascii?Q?ips/c2GMZemqYqqp8LihCQWeDN1o0E+4BrBfWM6Mgzv5bbsTQSXv9zxOx3Sa?=
 =?us-ascii?Q?bv6grtr+LVtvYqJQ9iJI9mUWasNXQqGhzmluXrBC44NB/GH5Hl7Z1gqL0evy?=
 =?us-ascii?Q?xERr5VryNUdmT22Jqtwc/fa2JtR0QkNG9kNuKBwLFnijaevG6qaQ517bgUuh?=
 =?us-ascii?Q?+LM1BHeE1SpytQGI1d7voNc7E5cien3bNaPNyMLIGUa2ekUPzekmshJZXp+O?=
 =?us-ascii?Q?K5vZbxwPUAddQ54DDV6qSdqXDd1mCcEkN4Dvs9KS?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?YHyqp6+/8bjXPej6JaTFF8SsdO4nCxAGxBsgEbJkTDlTb4bcoQXGC0XpcQHr?=
 =?us-ascii?Q?KRSsmJvpYBhYtPB6cAOe3UhhbIDonpAaBP6di9NCZiSyRISqnNS0I7aRYhCE?=
 =?us-ascii?Q?y1X7xtSeZHIOnM/ooGIyVj20DljBptkgrLizz1iLutuTq+I6assVMnYY8LMC?=
 =?us-ascii?Q?okBTck2T58MlgePPQLxC/NTohGPMNfbXkllCuV+TRvEKm9i6Y28NPo8n1kCU?=
 =?us-ascii?Q?fY607gWl92AhsgVtSbjPZ/fez7ySzMInILf1nkdHvdIX5jcCSJWg9b6PJjyb?=
 =?us-ascii?Q?iiq2mBD4AyRkDCGJlV4BQwmNNn3fr2srCD/NMcOfS9AhfA/kNDC7GY/O5xfU?=
 =?us-ascii?Q?MayDM0Z5XARkx1bFW8Stn8ZZjoMnNsTaPobRhIDmUfw92QGD+u6xX+3HuWjd?=
 =?us-ascii?Q?wVG0D3nWR0jfAw9atSlHxlCGW2sEWZZddiM/FBaCSHvC09nDyMXnO34MsjTu?=
 =?us-ascii?Q?togDGPgEXesX6zcqBHXbDoXqQSCL1p+FnfSecyQdJhNf7SeCk13Bq4qvTwpx?=
 =?us-ascii?Q?dgdvY4TVo4YkauczzYelqA/sptOnpHCd/4Z8Cg7G5Rj2MiPlFBa+LLzyDwDr?=
 =?us-ascii?Q?wAdvU3Rfv10ERCz5XvgnE/wLKcarYtiVhH7m0PqV82FAfnjmx4jOVJ3Zx9XM?=
 =?us-ascii?Q?eqsCgYf6Mn6gWyHFSgT3LxVozyhhbWH1joSJ7vk7JVbjwrrVZnSV97DJi3iM?=
 =?us-ascii?Q?EfvqVEBQ5VyLJAHBUoRyJ6yyHrmWXNoO9a+mfAlUEW0Dfrh5yKSThObU37t1?=
 =?us-ascii?Q?rdVh3NsDqeifR2aJ28thhxPBVc3cRnX0f10RxNjgy5qkibA8VQOmtWAMgHJf?=
 =?us-ascii?Q?sc8sfmNcHQpCh7xfsCvIKFiaHxuhHuwXLZXZJuOXopGUwjfXk0MKnEF3+Szg?=
 =?us-ascii?Q?fgNe1tvjxHuqgzgOF/Th18oZ7l4LOW4RMMZYDNRPiLJjCGgVxEHscOpRsyjh?=
 =?us-ascii?Q?jj03NVPauyQ0ijI23mwigUcMktd8cP8zjszNLQCWzDui0SdTx5Agv1P/qWL0?=
 =?us-ascii?Q?FLAPVjZIoGp1PHDFRXGFzCQCD4DOqNia6i1Horzmui0I/cfu7gIcHt8gY+NT?=
 =?us-ascii?Q?Lxqi64S3RGKm3343l2bGLx82ZpVJF9NYMJkAdPGjVm1FfgyaNfIgLjFrSMVa?=
 =?us-ascii?Q?17RU6nZfdcXU?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 850d5942-369b-42eb-e66d-08dbbd6007a3
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 00:40:34.6878
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3paTb7dxTRgo759+XdEXmKnAP2aiveNoFdqXOPU52e19++MtE64aHco9zRP3MyTub9YFp8Uki6hy9XYIOZ0rQg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB7174
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-09-24_21,2023-09-21_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 mlxscore=0
 mlxlogscore=999 suspectscore=0 phishscore=0 adultscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000
 definitions=main-2309250000
X-Proofpoint-ORIG-GUID: pSbux-Z7blrrWnBynn8BvvvvUoxkzByW
X-Proofpoint-GUID: pSbux-Z7blrrWnBynn8BvvvvUoxkzByW
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Sun, 24 Sep 2023 17:41:49 -0700 (PDT)

Update the internal hugetlb restore vmemmap code path such that TLB
flushing can be batched.  Use the existing mechanism of passing the
VMEMMAP_REMAP_NO_TLB_FLUSH flag to indicate flushing should not be
performed for individual pages.  The routine hugetlb_vmemmap_restore_folios
is the only user of this new mechanism, and it will perform a global
flush after all vmemmap is restored.

Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
 mm/hugetlb_vmemmap.c | 39 ++++++++++++++++++++++++---------------
 1 file changed, 24 insertions(+), 15 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 9df350372046..d2999c303031 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -461,18 +461,19 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end,
  * @end:	end address of the vmemmap virtual address range that we want to
  *		remap.
  * @reuse:	reuse address.
+ * @flags:	modifications to vmemmap_remap_walk flags
  *
  * Return: %0 on success, negative error code otherwise.
  */
 static int vmemmap_remap_alloc(unsigned long start, unsigned long end,
-			       unsigned long reuse)
+			       unsigned long reuse, unsigned long flags)
 {
 	LIST_HEAD(vmemmap_pages);
 	struct vmemmap_remap_walk walk = {
 		.remap_pte	= vmemmap_restore_pte,
 		.reuse_addr	= reuse,
 		.vmemmap_pages	= &vmemmap_pages,
-		.flags		= 0,
+		.flags		= flags,
 	};
 
 	/* See the comment in the vmemmap_remap_free(). */
@@ -494,17 +495,7 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key);
 static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON);
 core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0);
 
-/**
- * hugetlb_vmemmap_restore - restore previously optimized (by
- *			     hugetlb_vmemmap_optimize()) vmemmap pages which
- *			     will be reallocated and remapped.
- * @h:		struct hstate.
- * @head:	the head page whose vmemmap pages will be restored.
- *
- * Return: %0 if @head's vmemmap pages have been reallocated and remapped,
- * negative error code otherwise.
- */
-int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)
+static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, unsigned long flags)
 {
 	int ret;
 	unsigned long vmemmap_start = (unsigned long)head, vmemmap_end;
@@ -525,7 +516,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)
 	 * When a HugeTLB page is freed to the buddy allocator, previously
 	 * discarded vmemmap pages must be allocated and remapping.
 	 */
-	ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse);
+	ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags);
 	if (!ret) {
 		ClearHPageVmemmapOptimized(head);
 		static_branch_dec(&hugetlb_optimize_vmemmap_key);
@@ -534,6 +525,21 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)
 	return ret;
 }
 
+/**
+ * hugetlb_vmemmap_restore - restore previously optimized (by
+ *				hugetlb_vmemmap_optimize()) vmemmap pages which
+ *				will be reallocated and remapped.
+ * @h:		struct hstate.
+ * @head:	the head page whose vmemmap pages will be restored.
+ *
+ * Return: %0 if @head's vmemmap pages have been reallocated and remapped,
+ * negative error code otherwise.
+ */
+int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head)
+{
+	return __hugetlb_vmemmap_restore(h, head, 0);
+}
+
 /**
  * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list.
  * @h:			hstate.
@@ -557,7 +563,8 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h,
 
 	list_for_each_entry_safe(folio, t_folio, folio_list, lru) {
 		if (folio_test_hugetlb_vmemmap_optimized(folio)) {
-			ret = hugetlb_vmemmap_restore(h, &folio->page);
+			ret = __hugetlb_vmemmap_restore(h, &folio->page,
+						VMEMMAP_REMAP_NO_TLB_FLUSH);
 			if (ret)
 				break;
 			restored++;
@@ -567,6 +574,8 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h,
 		list_move(&folio->lru, non_hvo_folios);
 	}
 
+	if (restored)
+		flush_tlb_all();
 	if (!ret)
 		ret = restored;
 	return ret;
-- 
2.41.0