Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc:     Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        David Hildenbrand <david@redhat.com>,
        Axel Rasmussen <axelrasmussen@google.com>,
        Mina Almasry <almasrymina@google.com>,
        Peter Xu <peterx@redhat.com>, Rik van Riel <riel@surriel.com>,
        Vlastimil Babka <vbabka@suse.cz>,
        Matthew Wilcox <willy@infradead.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>,
        Wei Chen <harperchen1110@gmail.com>, stable@vger.kernel.org
Subject: [PATCH v2] hugetlb: don't delete vma_lock in hugetlb MADV_DONTNEED processing
Date:   Sat, 22 Oct 2022 19:50:47 -0700
Message-Id: <20221023025047.470646-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?xIupHzxbEU6pkWC5mzB59JL+yp43keoqcN/41v6PPkJuXDkYOaMIsV3rIxdW?=
 =?us-ascii?Q?OC+8S38G7HwHacoooSLGDla+AqIH5EqDeKQ5nvJrjTncQfnOVBKdFAntfNa0?=
 =?us-ascii?Q?INFnUitECtt9SKqIMvXo6QgFIfuQVPqpiFJlCminAg79xToQWzAei+Z4CNLY?=
 =?us-ascii?Q?i9yCV9OdBTMDRrXz4i34bsvwAGg1cexBQ81j8z2Q9Qpw+8z6aGgZDy9M8vwM?=
 =?us-ascii?Q?gSPwXCvAO4PmJEk7cThKOXL8pyA0VhNcv1qNfe9brV55a8qlBN48LRbM8EJy?=
 =?us-ascii?Q?f0LoVDyiS21wcqbch2DfzYgJbRi82U2ga2MjVBOMCrDMCtQkQX2C26YdyKem?=
 =?us-ascii?Q?e9N3kF8lREi+pCX8UWU7Wjl6eZGQQw5xdTrpkMOdLXqHk7dOTrPQsFms4lHb?=
 =?us-ascii?Q?w9uQkoFKis3Tc/1PiSATYPjMUVcyf4CRtZX3FYcJej7oRC5kv7HcFz8yB9Da?=
 =?us-ascii?Q?GukKMoLeRqjmCp3EqxB04mA0iV8HvBnde5TnVO10MZ7oXN87LeDPv/S78cAa?=
 =?us-ascii?Q?D+omNbkQOI3x6V6SEsM0xebvQa5DlaNAFR+aqWFa2SrzxYGORop1ac4MBoiw?=
 =?us-ascii?Q?2F/BcxlgfhtCOn2yq/gKplOKRH2pmkKaEMzxwKRjfm9p8zUx/dHVq17zQktD?=
 =?us-ascii?Q?foRjFqCyL6JHMNseV7ZCC0wP9xi3GNaiwl0UrKbAF9sj0GLbznL8KHiysr8Z?=
 =?us-ascii?Q?54Mv4i5y4r72J+1spqJhAjRyedeAQRukmB0DexE6xbqOYuF/EZ6Riiing7PU?=
 =?us-ascii?Q?m1wlLyWwrY8WgdJRfs0Gs5P7BEXOaaxTFEOnTt9+cLNsv1R4ZQkIFvls73Te?=
 =?us-ascii?Q?0rkJs6XPZFhYu4s1jJu29KwC3+vqj1BSybD9fZJL4ILdyMIVvWIL2SG0JHTS?=
 =?us-ascii?Q?GP4+e5eWJd49AS+juCkUmLPMOBv937x/kP3w0HsTko76y9Hho+ESn+Ph0uFY?=
 =?us-ascii?Q?3re3czZpuNGG9lw4JbAETBmCN4C6IprujDBtQ6ECyVWTrF648wThLkbfPLEs?=
 =?us-ascii?Q?ndp2bYgMtjU3KoIco89eec0/j+eov/HMa3fJOimw0zbzdHeAJ31mE8DJJz+2?=
 =?us-ascii?Q?6D6Zhm4+l9VNEaH5yCPd1aftHm7Sf9gUpk1MTIicBbMjei9Pv1YeInJHv/Z6?=
 =?us-ascii?Q?LyV7PRjvqxAgFtHLVjfqMf6H3QEAELj4xq3Kl+xbYMph4m3dseyHwjMGUPbV?=
 =?us-ascii?Q?wJTQSs9/Z/CpKjqJbVMwlKPAD7134W72Yk+uUc/xueC4JtnW1/lqBI52Xrys?=
 =?us-ascii?Q?vYDY1kx3Pb+nF4wpMWswxg1324RUz2TXXHPE7ymjvDvykBeA8XTrJ6U2au9f?=
 =?us-ascii?Q?c8IVHwJqT1+K/sUqve8eB4S2FZKwSI64Mp3OqfN5b0JOhgQaKULWFRLeLdT1?=
 =?us-ascii?Q?HS0QywIUKDnjRmpmdRaDYWdmSSeve/qIF1lfw1s4z4BSPbQZDty+kNXt5kcW?=
 =?us-ascii?Q?2mcfcjL6vhp19kYSMg5EGzlKP5DvSNOijvW0qjR/+5M1Q4CF2Dee0IVdYwvO?=
 =?us-ascii?Q?YyVh6qeLaK132LxIbiJQxxTZSqhG5kpyiZsh1ofVNrH39/fJ3D3GvJ+7D61k?=
 =?us-ascii?Q?DOKZF2sH66BI74tey4KgowPKd8PK8mtzrit5JOlSHYB+73cSQN+1DFhEaV7h?=
 =?us-ascii?Q?6A=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7382de3a-56b0-49df-4fac-08dab4a1665a
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2022 02:50:52.8618
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: W+d2Xjww9VPlAJ4FxAe8Mkw5p6DNcO6uvA6tiHNN8F5e7a14tNcbjSQXrxtcw3b2MatQNqmiXFLAefEiexVa7Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB4960
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-10-21_04,2022-10-21_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 phishscore=0
 mlxlogscore=999 adultscore=0 malwarescore=0 spamscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000
 definitions=main-2210230016
X-Proofpoint-GUID: bHqztj2VfC0b6lL5Mdv8-gZndM-eG8pr
X-Proofpoint-ORIG-GUID: bHqztj2VfC0b6lL5Mdv8-gZndM-eG8pr
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

madvise(MADV_DONTNEED) ends up calling zap_page_range() to clear the
page tables associated with the address range.  For hugetlb vmas,
zap_page_range will call __unmap_hugepage_range_final.  However,
__unmap_hugepage_range_final assumes the passed vma is about to be
removed and deletes the vma_lock to prevent pmd sharing as the vma is
on the way out.  In the case of madvise(MADV_DONTNEED) the vma remains,
but the missing vma_lock prevents pmd sharing and could potentially
lead to issues with truncation/fault races.

This issue was originally reported here [1] as a BUG triggered in
page_try_dup_anon_rmap.  Prior to the introduction of the hugetlb
vma_lock, __unmap_hugepage_range_final cleared the VM_MAYSHARE flag to
prevent pmd sharing.  Subsequent faults on this vma were confused as
VM_MAYSHARE indicates a sharable vma, but was not set so page_mapping
was not set in new pages added to the page table.  This resulted in
pages that appeared anonymous in a VM_SHARED vma and triggered the BUG.

Create a new routine clear_hugetlb_page_range() that can be called from
madvise(MADV_DONTNEED) for hugetlb vmas.  It has the same setup as
zap_page_range, but does not delete the vma_lock.

[1] https://lore.kernel.org/lkml/CAO4mrfdLMXsao9RF4fUE8-Wfde8xmjsKrTNMNC9wjUb6JudD0g@mail.gmail.com/

Fixes: 90e7e7f5ef3f ("mm: enable MADV_DONTNEED for hugetlb mappings")
Reported-by: Wei Chen <harperchen1110@gmail.com>
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
Cc: <stable@vger.kernel.org>
---
v2 - Fix build issues associated with !CONFIG_ADVISE_SYSCALLS

 include/linux/hugetlb.h |  7 +++++
 mm/hugetlb.c            | 62 +++++++++++++++++++++++++++++++++--------
 mm/madvise.c            |  5 +++-
 3 files changed, 61 insertions(+), 13 deletions(-)

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index a899bc76d677..0246e77be3a3 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -158,6 +158,8 @@ long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *,
 void unmap_hugepage_range(struct vm_area_struct *,
 			  unsigned long, unsigned long, struct page *,
 			  zap_flags_t);
+void clear_hugetlb_page_range(struct vm_area_struct *vma,
+			unsigned long start, unsigned long end);
 void __unmap_hugepage_range_final(struct mmu_gather *tlb,
 			  struct vm_area_struct *vma,
 			  unsigned long start, unsigned long end,
@@ -426,6 +428,11 @@ static inline void __unmap_hugepage_range_final(struct mmu_gather *tlb,
 	BUG();
 }
 
+static void __maybe_unused clear_hugetlb_page_range(struct vm_area_struct *vma,
+			unsigned long start, unsigned long end)
+{
+}
+
 static inline vm_fault_t hugetlb_fault(struct mm_struct *mm,
 			struct vm_area_struct *vma, unsigned long address,
 			unsigned int flags)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 931789a8f734..807cfd2884fa 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5202,29 +5202,67 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct
 		tlb_flush_mmu_tlbonly(tlb);
 }
 
-void __unmap_hugepage_range_final(struct mmu_gather *tlb,
+static void __unmap_hugepage_range_locking(struct mmu_gather *tlb,
 			  struct vm_area_struct *vma, unsigned long start,
 			  unsigned long end, struct page *ref_page,
-			  zap_flags_t zap_flags)
+			  zap_flags_t zap_flags, bool final)
 {
 	hugetlb_vma_lock_write(vma);
 	i_mmap_lock_write(vma->vm_file->f_mapping);
 
 	__unmap_hugepage_range(tlb, vma, start, end, ref_page, zap_flags);
 
-	/*
-	 * Unlock and free the vma lock before releasing i_mmap_rwsem.  When
-	 * the vma_lock is freed, this makes the vma ineligible for pmd
-	 * sharing.  And, i_mmap_rwsem is required to set up pmd sharing.
-	 * This is important as page tables for this unmapped range will
-	 * be asynchrously deleted.  If the page tables are shared, there
-	 * will be issues when accessed by someone else.
-	 */
-	__hugetlb_vma_unlock_write_free(vma);
+	if (final) {
+		/*
+		 * Unlock and free the vma lock before releasing i_mmap_rwsem.
+		 * When the vma_lock is freed, this makes the vma ineligible
+		 * for pmd sharing.  And, i_mmap_rwsem is required to set up
+		 * pmd sharing.  This is important as page tables for this
+		 * unmapped range will be asynchrously deleted.  If the page
+		 * tables are shared, there will be issues when accessed by
+		 * someone else.
+		 */
+		__hugetlb_vma_unlock_write_free(vma);
+		i_mmap_unlock_write(vma->vm_file->f_mapping);
+	} else {
+		i_mmap_unlock_write(vma->vm_file->f_mapping);
+		hugetlb_vma_unlock_write(vma);
+	}
+}
 
-	i_mmap_unlock_write(vma->vm_file->f_mapping);
+void __unmap_hugepage_range_final(struct mmu_gather *tlb,
+			  struct vm_area_struct *vma, unsigned long start,
+			  unsigned long end, struct page *ref_page,
+			  zap_flags_t zap_flags)
+{
+	__unmap_hugepage_range_locking(tlb, vma, start, end, ref_page,
+					zap_flags, true);
 }
 
+#ifdef CONFIG_ADVISE_SYSCALLS
+/*
+ * Similar setup as in zap_page_range().  madvise(MADV_DONTNEED) can not call
+ * zap_page_range for hugetlb vmas as __unmap_hugepage_range_final will delete
+ * the associated vma_lock.
+ */
+void clear_hugetlb_page_range(struct vm_area_struct *vma, unsigned long start,
+				unsigned long end)
+{
+	struct mmu_notifier_range range;
+	struct mmu_gather tlb;
+
+	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm,
+				start, end);
+	tlb_gather_mmu(&tlb, vma->vm_mm);
+	update_hiwater_rss(vma->vm_mm);
+
+	__unmap_hugepage_range_locking(&tlb, vma, start, end, NULL, 0, false);
+
+	mmu_notifier_invalidate_range_end(&range);
+	tlb_finish_mmu(&tlb);
+}
+#endif
+
 void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start,
 			  unsigned long end, struct page *ref_page,
 			  zap_flags_t zap_flags)
diff --git a/mm/madvise.c b/mm/madvise.c
index 2baa93ca2310..90577a669635 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -790,7 +790,10 @@ static int madvise_free_single_vma(struct vm_area_struct *vma,
 static long madvise_dontneed_single_vma(struct vm_area_struct *vma,
 					unsigned long start, unsigned long end)
 {
-	zap_page_range(vma, start, end - start);
+	if (!is_vm_hugetlb_page(vma))
+		zap_page_range(vma, start, end - start);
+	else
+		clear_hugetlb_page_range(vma, start, end);
 	return 0;
 }
 
-- 
2.37.3