Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Liam R. Howlett" <Liam.Howlett@oracle.com>
To:     Linus Torvalds <torvalds@linux-foundation.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        maple-tree@lists.infradead.org, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, Matthew Wilcox <willy@infradead.org>,
        Suren Baghdasaryan <surenb@google.com>
Cc:     "Liam R. Howlett" <Liam.Howlett@Oracle.com>,
        stable@vger.kernel.org,
        "Liam R . Howlett" <Liam.Howlett@oracle.com>,
        syzbot+8d95422d3537159ca390@syzkaller.appspotmail.com
Subject: [PATCH 8/8] mm: enable maple tree RCU mode by default.
Date:   Mon, 27 Mar 2023 14:55:32 -0400
Message-Id: <20230327185532.2354250-9-Liam.Howlett@oracle.com>
In-Reply-To: <20230327185532.2354250-1-Liam.Howlett@oracle.com>
References: <20230327185532.2354250-1-Liam.Howlett@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?O7LXugZrmMHK6qfGDNwkmMBnlCVdpgBHW3OvNDPxh5peACG79WYhghZCDQnZ?=
 =?us-ascii?Q?GtzWtkFbUjZfKmcIfiNeW474bQ+IlQbmT5cWp7KiLrmR6n/LXXu1iw95ATc4?=
 =?us-ascii?Q?h8eVsiutstWptNIPWmweVD1UK3NYzGFRjz74AtKsoXK7t7IH0bNBS3iqrrhn?=
 =?us-ascii?Q?f5Uzz9rJipS09FMckG0ui+mXCvV56CtqwoXiUfePuTDo6XiYRMsOj+Cnkfmq?=
 =?us-ascii?Q?RQ0OvrEWATbKIMxMpOcIgD0XDLR9psF93QSRh2P7wosI7ha5xEPLtZTZjNng?=
 =?us-ascii?Q?Tiq+EW8TSXVgoe4jJjJs35G3rZaGqmWaBq9L40VDqq8vtVK8XyXWIrmLSUHe?=
 =?us-ascii?Q?SpU397vCBDOy01Qfdmvfdtext88ufmbiWDou2WH0ztjoaihBcwi86gJxxuhA?=
 =?us-ascii?Q?wup28ZjdSecgSNTjSUpagw8oLSrUDU6earr9V5u+AONSu4zTdjfT+oDQj20H?=
 =?us-ascii?Q?BLyT+14F0Qg0RCJCtsRW1O7pn9UiVH/ZysHRvjjUwmsf5LEgRjH/+IoI9R52?=
 =?us-ascii?Q?VU8GtXNu74LlHiJ1VjZCSInNZx4dtbEzieF6n39eG0pa1ROR7Hjn24S81lAT?=
 =?us-ascii?Q?koLZoduClCzkNVSzSDMICogHYQzDO39hNKuBjJd4N8kr5RpQ1/6zJ6qgyYxZ?=
 =?us-ascii?Q?hcyUkZZvtLL9iSprKC58X9lBPtapzukSs9RzOJTGAhpO4b1vrFKgyPLSgEDN?=
 =?us-ascii?Q?LTZb/Rx1Y9nrG4NXltDP5byqZs4XyTZ72YxOrGmsH3dXAQuaBhumuz4bU3S8?=
 =?us-ascii?Q?6wQVn8cETZymadhvRE9aO2zZOWdKvz4Gnq0C+ByYLT/S3JiA2qjXP2hlTZ2B?=
 =?us-ascii?Q?uhBTtHOJQ9iK8xHKUh91AcovSOiCZXux2W1Is2viqX0Wzo6E3acevFKwSdYi?=
 =?us-ascii?Q?QULrOnWIxHFXpN9I8jXdEBlNo6ZVkNWXwWp86NmBgQG2yfYpiGcRsrW5Opn0?=
 =?us-ascii?Q?bJZIq2XeeAANngfXYli/F7FN0VsAf0j9ZhyWQ12R33VDPyk3ebiTXo0C7ANO?=
 =?us-ascii?Q?uE9sBJRyXSGoqJY+jaZfZq8S6zabXb5AaMP+B+ErLuEh4idughuIeOh5sYHC?=
 =?us-ascii?Q?EqW9f2JWyb9IOK6WKPpIAIHquo0m/av95bPNUAix+YYp7ZozYx4kcMtI9n87?=
 =?us-ascii?Q?qjLrQcgsQUPhnlw94LM6c5sEB7QPBASASJf+XJFmQW9QJSnudzAWft+ikCa2?=
 =?us-ascii?Q?326H36TlOqi6g+yoNf0YBgn9hX5p3fzL8LbQVTx9H6PsUkPXfVRdISvyJXeq?=
 =?us-ascii?Q?2QvQP6eqzB9DK9HrIdgfSqmHmoxPd0gbdYv8N4XrqF9HePEsOcwyL4ZFqqzf?=
 =?us-ascii?Q?iMw7ubr0/RyjQtFfi6FtgfD0SI1osVYfn3ov34oIJOkOxtO7CnYvv/gOL/5P?=
 =?us-ascii?Q?bsw5mn7EY1VnzVNrKjGpqm0KDFlEwRfhYroZigOPgBw42RSRc8vq99xiBTsb?=
 =?us-ascii?Q?0pozZRPH4Jx3z9OdkzNMajyUGB98TxX33brcgRv/3tolId07S7ok/4zzea6E?=
 =?us-ascii?Q?djxnCSkQAdzUVAYK02PFluW1OimaoG6Bf9bYRw4AnqqJPF9AObMoZg4WM13x?=
 =?us-ascii?Q?5eFsEsBL20RTHaxSV4m207ap7MfpX7wnFyUknibplQbgEYON2hi5XF2FLJ8v?=
 =?us-ascii?Q?Hw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?dGruqNlnqqEx5TK2EwswUyii8dTXcTjaYv0c/QYUrCqVnQs8vi9LDqfO/Htv?=
 =?us-ascii?Q?cJmNFW8YsiZ7ipMoYbDVo28yDHOTHbLucs6MZ135zN4WyHz5xIeF9lEK+nd7?=
 =?us-ascii?Q?Vtb6LYqpYeidMqJcQkGwEIC7qzuVPWfUypswifH4sZaStMnrBqBeGdG0UhUD?=
 =?us-ascii?Q?s7ocJlISwzG3PEZIIq13If0mwLqySNG5qyh3t/p3TdCcqSIUEfVxUnf0Mheq?=
 =?us-ascii?Q?5dZWoFLC7pPd4RQDn48sBByHCG+TRaNJm9lpUXuI/VfMUiZQRvXFFWu923TP?=
 =?us-ascii?Q?1acanrUBLumoZ2zAX5QFkkIgDelJFsBSoSLSBgAGQbhBnghQ+a26Iyy3r4T9?=
 =?us-ascii?Q?aPw6E5mx0qXuIEbewhLT2rtNNNlRpiHGwv49glbpg6uLX9hCo0BloVLRpyz2?=
 =?us-ascii?Q?W+FZnPwrImQqPJMKYmEPbOIbnYe9CWQNcRNEvUTfXUkdyjYVIOZfTKcpkVle?=
 =?us-ascii?Q?yHgcd0S2PyY39kdihClM80kFQ+a35TzndiZqtWKxBeHGC17cSSl7BqKwWXKW?=
 =?us-ascii?Q?B4xBWP3Xb07pRjCWyeDEcVgu8ZifmMhYPp0r1SMjms7OY1eUcA6GLQI5oumI?=
 =?us-ascii?Q?5mFFlJxQLuoK+wErrkXVwkuZeR5eZMNvlYrix55grOUIRVlw4KKf0bKZmojG?=
 =?us-ascii?Q?9gLLVGjrLR1HoJjZxgm90+20Hy21aaJ39t4QvJ68mQc3Ec3X5tVK/TW4G4sY?=
 =?us-ascii?Q?JAXnA7q8l+yNFMth0mtbScuGozXAILoetSRzYXhnd1dQHdF+AXWJDlUT9xk/?=
 =?us-ascii?Q?7Xd3Cssg/jrLGyHBqY0/oBmQ89Lc6F9QPmypkW0dRuJj4y0YXB2d0ES9UgnH?=
 =?us-ascii?Q?PSTSCkaFzBxaUGct4eJqrD0wKIbD+86UvVQsZp92pjsy/FHzUolYu5S/fUaA?=
 =?us-ascii?Q?6FM3xT3V1+8zSEHW9N+y5nTegaNVserD2zKPSNBHSbxdpv3c+cl9Hol3Yhct?=
 =?us-ascii?Q?KT1t2uqclC1G64SxTpMRg9+YQlJQRVSXbJby3HtY+LpfS2C9EGnzsb8h1MTU?=
 =?us-ascii?Q?dABuO6KRUmw05US8PQ+PtP8ddQ=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6608797e-bcf7-47cc-94c2-08db2ef4e4b9
X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2023 18:55:54.9055
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Bsvs+oCyozlh35lJ69iBDHCeM19ykRk3ToUb8sneOtdTcvJz1fspGecjup31rcRyUnFdp9F6+Cv279prOaJPlw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB5722
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-03-24_11,2023-03-27_02,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0
 suspectscore=0 mlxscore=0 mlxlogscore=436 spamscore=0 adultscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2303200000 definitions=main-2303270155
X-Proofpoint-GUID: 3R6cOwk1anQb4C3_qvpqgQRBQ5NbA7Ce
X-Proofpoint-ORIG-GUID: 3R6cOwk1anQb4C3_qvpqgQRBQ5NbA7Ce
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
        DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2,
        SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

From: "Liam R. Howlett" <Liam.Howlett@Oracle.com>

Use the maple tree in RCU mode for VMA tracking.

The maple tree tracks the stack and is able to update the pivot
(lower/upper boundary) in-place to allow the page fault handler to write
to the tree while holding just the mmap read lock.  This is safe as the
writes to the stack have a guard VMA which ensures there will always be
a NULL in the direction of the growth and thus will only update a pivot.

It is possible, but not recommended, to have VMAs that grow up/down
without guard VMAs.  syzbot has constructed a testcase which sets up a
VMA to grow and consume the empty space.  Overwriting the entire NULL
entry causes the tree to be altered in a way that is not safe for
concurrent readers; the readers may see a node being rewritten or one
that does not match the maple state they are using.

Enabling RCU mode allows the concurrent readers to see a stable node and
will return the expected result.

Link: https://lkml.kernel.org/r/20230227173632.3292573-9-surenb@google.com
Link: https://lore.kernel.org/linux-mm/000000000000b0a65805f663ace6@google.com/
Cc: stable@vger.kernel.org
Fixes: d4af56c5c7c6 ("mm: start tracking VMAs with maple tree")
Signed-off-by: Liam R. Howlett <Liam.Howlett@oracle.com>
Reported-by: syzbot+8d95422d3537159ca390@syzkaller.appspotmail.com
---
 include/linux/mm_types.h | 3 ++-
 kernel/fork.c            | 3 +++
 mm/mmap.c                | 3 ++-
 3 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 0722859c3647..a57e6ae78e65 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -774,7 +774,8 @@ struct mm_struct {
 	unsigned long cpu_bitmap[];
 };
 
-#define MM_MT_FLAGS	(MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN)
+#define MM_MT_FLAGS	(MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \
+			 MT_FLAGS_USE_RCU)
 extern struct mm_struct init_mm;
 
 /* Pointer magic because the dynamic array size confuses some compilers. */
diff --git a/kernel/fork.c b/kernel/fork.c
index d8cda4c6de6c..1bf31ba07e85 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 	if (retval)
 		goto out;
 
+	mt_clear_in_rcu(vmi.mas.tree);
 	for_each_vma(old_vmi, mpnt) {
 		struct file *file;
 
@@ -700,6 +701,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
 	retval = arch_dup_mmap(oldmm, mm);
 loop_out:
 	vma_iter_free(&vmi);
+	if (!retval)
+		mt_set_in_rcu(vmi.mas.tree);
 out:
 	mmap_write_unlock(mm);
 	flush_tlb_mm(oldmm);
diff --git a/mm/mmap.c b/mm/mmap.c
index 740b54be3ed4..16cbb83b3ec6 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -2277,7 +2277,7 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	int count = 0;
 	int error = -ENOMEM;
 	MA_STATE(mas_detach, &mt_detach, 0, 0);
-	mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN);
+	mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & MT_FLAGS_LOCK_MASK);
 	mt_set_external_lock(&mt_detach, &mm->mmap_lock);
 
 	/*
@@ -3042,6 +3042,7 @@ void exit_mmap(struct mm_struct *mm)
 	 */
 	set_bit(MMF_OOM_SKIP, &mm->flags);
 	mmap_write_lock(mm);
+	mt_clear_in_rcu(&mm->mm_mt);
 	free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS,
 		      USER_PGTABLES_CEILING);
 	tlb_finish_mmu(&tlb);
-- 
2.39.2