Received-SPF: pass (google.com: domain of linux-kernel+bounces-210390-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
From: "Liam R. Howlett" <Liam.Howlett@oracle.com>
To: linux-mm@kvack.org, Andrew Morton <akpm@linux-foundation.org>
Cc: linux-kernel@vger.kernel.org, Vlastimil Babka <vbabka@suse.cz>,
        Lorenzo Stoakes <lstoakes@gmail.com>,
        Suren Baghdasaryan <surenb@google.com>,
        "Liam R. Howlett" <Liam.Howlett@oracle.com>
Subject: [PATCH 4/8] mm/mmap: Extract the gathering of vmas from do_vmi_align_munmap()
Date: Tue, 11 Jun 2024 14:01:56 -0400
Message-ID: <20240611180200.711239-5-Liam.Howlett@oracle.com>
In-Reply-To: <20240611180200.711239-1-Liam.Howlett@oracle.com>
References: <20240611180200.711239-1-Liam.Howlett@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?uWq+BxStc9h3bb2E1yZxRsMi7XYeG0puVjjmIcT7bNcL4Zev2vhNVokMN4As?=
 =?us-ascii?Q?4f0ZtHYP4Qoy8d2HzRYmU0FKaNS5wRHR+rFl/AFefs+EW+farf15LP64Mzxi?=
 =?us-ascii?Q?XZppo0l3plm20l3TJChfNuOPeb4Z5hgowr8kcZ2hLXHFaxwQdT1FIFVJM6cC?=
 =?us-ascii?Q?huS9hIUH2qaGfDpPXxEY1NkPb/PDVXTqX+g7KBZfFathsxh2rhuX39YYkum/?=
 =?us-ascii?Q?cWXwF0P6kIeEhGMcj6PIXUT1/pK20JkFNc+DiXawZWoDNOzPJ8yyc6rQioPc?=
 =?us-ascii?Q?J//vYlbGkuT3IZWxEcKTgK8kC8ciEdoiAx4v1CMc73MgFpdVxBMel8Kgt+0X?=
 =?us-ascii?Q?W7Zh5aTFquVP+Zn+0AN/3RTLXSBb7VIiCZQkKsJGIE3TmU3NhwC9LD4ib1ZJ?=
 =?us-ascii?Q?3clf+6nP1pzEZihyv7503cqdanJcO43oN/9AUWcIVyaIeK/jGiFVsq7Rcggi?=
 =?us-ascii?Q?3YJhAjsWVDt6gUMP4Z8UkvLZnAe57406PCuiqWIoBVMhDErs54B81FS9/QqQ?=
 =?us-ascii?Q?Ppo26qq/Pdc2tnBHQ5spIe7hNowMHmicPbgjlpIGZun1W0jP95di5lgpLTTI?=
 =?us-ascii?Q?sfR9LBU1r+BUY7og8/kx1IbUXRIZ0/Omt/8W4PVGht8EBejoZL1LdgXAJbpf?=
 =?us-ascii?Q?hj3vUZHbkNwBn7qNR4IytpkkpeSSGszZ7tlwN6KEbt7lJnTCM2nB12XmLnLM?=
 =?us-ascii?Q?AnUFxrMtOp5jTANPR+4JEDdN/W6OQLY6sGdHvBO9RhcDK6CU6w/9BZjhjK4F?=
 =?us-ascii?Q?YcISimkQa5Ou5WhiI/TA6XxR9xlF6eX3F7fsOnTick4CUoWqFT/mLznVfNNJ?=
 =?us-ascii?Q?GaGVTLr7Ca88acb/WDDqHFLIpehyRavT963pQUe3K4zGJ/73xdUlOSulee28?=
 =?us-ascii?Q?oyDxurT7HegI0OX0SQLpZlBFYylRFKgst7NZQui+JfwK8JaHG2UUytKzmyrz?=
 =?us-ascii?Q?tbHq7Kx7RphAJboEkhXF4YFu+k+f5pYIPiTqANsaydVqPHHvCaJrzBLNnflD?=
 =?us-ascii?Q?fTVxT2VjUyTPjLVCFh6ru322828Ka8n2/JSYa8Sagdp5B06OcerrkkR0RJOH?=
 =?us-ascii?Q?SYRIUkkla5CdMmjkDFK5wQYl52FnpEtV/EWhVT7S/u4/S7vnuYXpOfcaC+8O?=
 =?us-ascii?Q?+VL6BLOwn1y1/nSn83vzeTdPgELNFWa3aNNI5N7H5s8FE3V1rP9pDNCzxOsG?=
 =?us-ascii?Q?NXLe11DUnGXMHmbnSk5nMmvrz1JceQbTgdicied/1Iyv8Ejla8CbIr/SX2js?=
 =?us-ascii?Q?+Qgk4AJ1o1eCp4ceWUoJD2IRrKjGa1SOLGnMwM+e2QNYVaQh4NqXJbJ+iR/O?=
 =?us-ascii?Q?wbYqWsch3OlzlhKPmeZNvvn2Ir/EBfBPXNvDfFmp5wmXlzbYG0NCJGnF0yIZ?=
 =?us-ascii?Q?GkFvPhxvAw/2ebrz8KEaJbiuIW/MzDLKKSjfgctfJztD283VKQAKZZ31vCu8?=
 =?us-ascii?Q?pCaHQMVaD3V9f53njILKU+RvTv+3haBUcU+x+fD17Xl4+Icrw8qP+nYXNFeG?=
 =?us-ascii?Q?VvAaJm/TOjfGWRU2oPYylGnP4nx8GYBuOtFdi/mXVcJoIVWIbhk52Ctue9q8?=
 =?us-ascii?Q?kkdCUOV7aTgRj8/IVjRLMX8icMCACGCjtYDvExu0?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	N4O5JWR6vQsjdtjZ4AojFWNGfm7IScEPpd8T0Ieto9kGGAaDRx8zLOgjS2uOc84uxAXAnUTPtFIoWJOfsPlL4ilinZTlUes+nPI9sblr5O0GE1JiDqrJxsD6hLbvH+w9gUWB7kOrNdOZX/27hn2pVXLN8/hiCQ7IXjDIg1EHlzZGr9fyzNE9u8pnpElKji71JeJBH+ckOI/EPAgXMy7mgW/6zNqCOW43MD5repZJ7CJONtE+L4wA4LJEkQpnLat37oex+fcyDjDLpumTrN4EL+oIpAEHkZqqvKUj7sg4RaCdG1yXVubl7WxVd0OLnT0u980yDiI25zx8wEDNi7zAzbNzBWaTDh3TfU5+Zhefxpdf86DU00SsO4eBP9j1SP70fYjt+V4NQA4Hv4NDWBP+H8HljTEX1ktP3lincl+8Xm+owgsNKHtucEA9HU8O9XyDuKhmhngBN/Pe44hlErxoCuFt6FRFRtFr1XRHwAiDuKRFqzXrEa/CFpXLK8AQ6zuzgYPHT8ZRijxzig9/Jq1O/wjAI5qaZS2rvD1aFo/MWeC/qzwYgZPqdkaZB4798sRVpkUUTFZE31nV8tpPb9ZAMZmEtnwXfgSpmVey4cYsXWM=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 24f1402a-2155-4a4b-3041-08dc8a40a291
X-MS-Exchange-CrossTenant-AuthSource: DS0PR10MB7933.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jun 2024 18:02:19.0457
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wHpDA7eJiy7yPJ9KOA2iwWuOjbWz93fvLNr9QgpeYn+bxYv1LnGRmY+dGQgn+jXCo6a/hlok3jkXVBT3XljhLA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6154
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16
 definitions=2024-06-11_09,2024-06-11_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0
 malwarescore=0 spamscore=0 mlxscore=0 phishscore=0 mlxlogscore=999
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2406110123
X-Proofpoint-GUID: M9ZE3rJwkCJL9Z6J70tyD8w-ZqPzaMBe
X-Proofpoint-ORIG-GUID: M9ZE3rJwkCJL9Z6J70tyD8w-ZqPzaMBe

Create vmi_gather_munmap_vmas() to handle the gathering of vmas into a
detached maple tree for removal later.  Part of the gathering is the
splitting of vmas that span the boundary.

Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com>
---
 mm/mmap.c | 82 +++++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 58 insertions(+), 24 deletions(-)

diff --git a/mm/mmap.c b/mm/mmap.c
index 01bef2f7af2e..fbd36a8cb031 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -2615,32 +2615,29 @@ vmi_complete_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma,
 }
 
 /*
- * do_vmi_align_munmap() - munmap the aligned region from @start to @end.
+ * vmi_gather_munmap_vmas() - Put all VMAs within a range into a maple tree
+ * for removal at a later date.  Handles splitting first and last if necessary
+ * and marking the vmas as isolated.
+ *
  * @vmi: The vma iterator
  * @vma: The starting vm_area_struct
  * @mm: The mm_struct
  * @start: The aligned start address to munmap.
  * @end: The aligned end address to munmap.
  * @uf: The userfaultfd list_head
- * @unlock: Set to true to drop the mmap_lock.  unlocking only happens on
- * success.
+ * @mas_detach: The maple state tracking the detached tree
  *
- * Return: 0 on success and drops the lock if so directed, error and leaves the
- * lock held otherwise.
+ * Return: 0 on success
  */
 static int
-do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,
+vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma,
 		    struct mm_struct *mm, unsigned long start,
-		    unsigned long end, struct list_head *uf, bool unlock)
+		    unsigned long end, struct list_head *uf,
+		    struct ma_state *mas_detach, unsigned long *locked_vm)
 {
 	struct vm_area_struct *next = NULL;
-	struct maple_tree mt_detach;
 	int count = 0;
 	int error = -ENOMEM;
-	unsigned long locked_vm = 0;
-	MA_STATE(mas_detach, &mt_detach, 0, 0);
-	mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & MT_FLAGS_LOCK_MASK);
-	mt_on_stack(mt_detach);
 
 	/*
 	 * If we need to split any vma, do it now to save pain later.
@@ -2679,15 +2676,14 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,
 				goto end_split_failed;
 		}
 		vma_start_write(next);
-		mas_set(&mas_detach, count);
-		error = mas_store_gfp(&mas_detach, next, GFP_KERNEL);
+		mas_set(mas_detach, count++);
+		if (next->vm_flags & VM_LOCKED)
+			*locked_vm += vma_pages(next);
+
+		error = mas_store_gfp(mas_detach, next, GFP_KERNEL);
 		if (error)
 			goto munmap_gather_failed;
 		vma_mark_detached(next, true);
-		if (next->vm_flags & VM_LOCKED)
-			locked_vm += vma_pages(next);
-
-		count++;
 		if (unlikely(uf)) {
 			/*
 			 * If userfaultfd_unmap_prep returns an error the vmas
@@ -2712,7 +2708,7 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,
 #if defined(CONFIG_DEBUG_VM_MAPLE_TREE)
 	/* Make sure no VMAs are about to be lost. */
 	{
-		MA_STATE(test, &mt_detach, 0, 0);
+		MA_STATE(test, mas_detach->tree, 0, 0);
 		struct vm_area_struct *vma_mas, *vma_test;
 		int test_count = 0;
 
@@ -2732,6 +2728,48 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	while (vma_iter_addr(vmi) > start)
 		vma_iter_prev_range(vmi);
 
+	return 0;
+
+userfaultfd_error:
+munmap_gather_failed:
+end_split_failed:
+	abort_munmap_vmas(mas_detach);
+start_split_failed:
+map_count_exceeded:
+	return error;
+}
+
+/*
+ * do_vmi_align_munmap() - munmap the aligned region from @start to @end.
+ * @vmi: The vma iterator
+ * @vma: The starting vm_area_struct
+ * @mm: The mm_struct
+ * @start: The aligned start address to munmap.
+ * @end: The aligned end address to munmap.
+ * @uf: The userfaultfd list_head
+ * @unlock: Set to true to drop the mmap_lock.  unlocking only happens on
+ * success.
+ *
+ * Return: 0 on success and drops the lock if so directed, error and leaves the
+ * lock held otherwise.
+ */
+static int
+do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,
+		    struct mm_struct *mm, unsigned long start,
+		    unsigned long end, struct list_head *uf, bool unlock)
+{
+	struct maple_tree mt_detach;
+	MA_STATE(mas_detach, &mt_detach, 0, 0);
+	mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & MT_FLAGS_LOCK_MASK);
+	mt_on_stack(mt_detach);
+	int error;
+	unsigned long locked_vm = 0;
+
+	error = vmi_gather_munmap_vmas(vmi, vma, mm, start, end, uf,
+				       &mas_detach, &locked_vm);
+	if (error)
+		goto gather_failed;
+
 	error = vma_iter_clear_gfp(vmi, start, end, GFP_KERNEL);
 	if (error)
 		goto clear_tree_failed;
@@ -2742,12 +2780,8 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	return 0;
 
 clear_tree_failed:
-userfaultfd_error:
-munmap_gather_failed:
-end_split_failed:
 	abort_munmap_vmas(&mas_detach);
-start_split_failed:
-map_count_exceeded:
+gather_failed:
 	validate_mm(mm);
 	return error;
 }
-- 
2.43.0