Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc:     Naoya Horiguchi <naoya.horiguchi@linux.dev>,
        David Hildenbrand <david@redhat.com>,
        Axel Rasmussen <axelrasmussen@google.com>,
        Mina Almasry <almasrymina@google.com>,
        Peter Xu <peterx@redhat.com>, Rik van Riel <riel@surriel.com>,
        Vlastimil Babka <vbabka@suse.cz>,
        Matthew Wilcox <willy@infradead.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Mike Kravetz <mike.kravetz@oracle.com>,
        Wei Chen <harperchen1110@gmail.com>, stable@vger.kernel.org
Subject: [PATCH] hugetlb: don't delete vma_lock in hugetlb MADV_DONTNEED processing
Date:   Fri, 21 Oct 2022 16:07:22 -0700
Message-Id: <20221021230722.370587-1-mike.kravetz@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tb7RpENoaWy4KNi4Isw8lCLYhpC07lK8Pgx9uHrlZSNjUarH9fBI4nKTTJiD?=
 =?us-ascii?Q?+1Q0mx7G4lxiPB7O1q4bOzihowFxyxBnfzcWm8yGoBFTLbVdYC4P3+XkxFpT?=
 =?us-ascii?Q?Bg/BgnIVaDbAkCGaQjCJq74pFBHdNdRkzFxQsObTh0m7OIg5J8IMc/s/ciR1?=
 =?us-ascii?Q?EHtV2Tm+myeBlNeXHGwzfdnyYa19haay6duKFSs56bje4AXiNyIS8uBF1QCv?=
 =?us-ascii?Q?ulwBxEtKKbt39fWfvOQChmmmzHaTG2J35ZvCrOBb4XwPS9KVlrVPEfuWYtTf?=
 =?us-ascii?Q?sVIoESa8q0CWvIHObEKIe4m15COYya0qbmyvss/XLcO85quNheDXM8Do1e23?=
 =?us-ascii?Q?iynXY6DbonZOqrdUDR6J3kAWQ9mtv51MR38GA6Mg2FJGsXbTTA6IT1HA8Z+i?=
 =?us-ascii?Q?BY3JYExjoOO3Rj2HhYI1TNwOwoskozQ94HP/JjB5VH/m2gdR0Md9RLeqfNgo?=
 =?us-ascii?Q?u7AntsQI5oftdvA0MVxtfwr6pRC7Zp8yPgV/sKOwmKDuy4zj/h7CkDwN98nV?=
 =?us-ascii?Q?TK8G3LBIF6xVMakMTEqrqqtGH0MLqnL8yEoGvGTdRNsHiJK5iNgZRQxAJOYx?=
 =?us-ascii?Q?KAj+lhw9BTqICllOywCYe2qStfk30c/cE+2zuqZgIEAsL1+zJgfd2TUqVKbN?=
 =?us-ascii?Q?R3Y1nkmUJa6vj7qNyL4jqmcwYszmhlfCFO9CpjDGzrC2RJ0g2SkDXZw+iyz5?=
 =?us-ascii?Q?2MYeTGRH+9BoOflK3/iBZbziIo8dnDBbIcFlsLUTnbxGVoZpTMS3Y+Tmn+gp?=
 =?us-ascii?Q?67+yfK3o0oQrgVHfWe5lElBxZpze2SBlt63D/Y5C1xmOmNL75Y7UmcyH28ZV?=
 =?us-ascii?Q?yjPSKliJCAs0582gZA5IubyY/uTGE72kO2uhwEQDmpuvC5tKaOG8K99VsS3a?=
 =?us-ascii?Q?WvUpVCYRgS+iAG+4Fth03WdHiuFVoFk+kNbpdnCzQ4WyetV71fK0hNanWzBt?=
 =?us-ascii?Q?q+N5MlUTtuqX6z9X1vr73o4/dJeoByuA9ZEi20CRkcJtLAotjs8obkftUlFJ?=
 =?us-ascii?Q?hPUE+tm3uI67SGKqkSH9SuW3FtBlSPCrryqH/cYH1y9tBrJdnSEj4hoUtx6Z?=
 =?us-ascii?Q?10TI/p52+b16qk/3CrvnakSIHtlP9Mso8dNumv6HYF/NI7kUXX0NIanBMhZ8?=
 =?us-ascii?Q?u7i33h4DW/wAbwAViiYiBS0gselZSRGfEDaxNAXF/I864HM9tTqUUseiVLj/?=
 =?us-ascii?Q?rP5Ty2513s2PUdDCEfJJawYuptqBlUW68xUUkcEK/q9ObSC+8qUeqveBNITo?=
 =?us-ascii?Q?IwjN/HS6Q1l1CbfX9tZ09FFfC27bbtNNomRuY15hdpDxSWsRuJXsQ0t73IxY?=
 =?us-ascii?Q?/TwlnrpOxlX60E+75W9dq62J1bVNxLaU0kbk791XcfA1xOh9pHHrd/DDHm7W?=
 =?us-ascii?Q?VwdykzhlWE9QusJPJBaEXBbnHcSFYnI5GYJLhl+zteGh2Ds3xcynQHt9NMyv?=
 =?us-ascii?Q?lVB33sdDh6XEVd9HsPGfrhiISmBzjO+aI+TOpSJcPjc5jmT0ciUKT8HdAcSX?=
 =?us-ascii?Q?hWybqbdNARN+inQU3mgryNFG6JRJVwIeFdwTwChmdV4jF5Blq9BIUJG4NpWp?=
 =?us-ascii?Q?fX05kwlQOp1u9P/IUW+osg2lM2wdoPJ+7veyHeFnER7c65lodzo0iDBVKDtV?=
 =?us-ascii?Q?kg=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0d0e9dea-d08b-4334-21c4-08dab3b9065b
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Oct 2022 23:07:28.4242
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EZ+ymqBOehclu+u5HTIlD1SKJfv0iNZg4mAP3nuj3BM2WXpICU8BUPG7DzkLT5NyF+K5GaBpl8ZFvgmbQI80Ig==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6380
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-10-21_04,2022-10-21_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 mlxscore=0
 suspectscore=0 mlxlogscore=880 phishscore=0 malwarescore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000
 definitions=main-2210210134
X-Proofpoint-ORIG-GUID: 8EfPUnoYRoOABdx_OA0jm594Y5Oyqx0r
X-Proofpoint-GUID: 8EfPUnoYRoOABdx_OA0jm594Y5Oyqx0r
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

madvise(MADV_DONTNEED) ends up calling zap_page_range() to clear the
page tables associated with the address range.  For hugetlb vmas,
zap_page_range will call __unmap_hugepage_range_final.  However,
__unmap_hugepage_range_final assumes the passed vma is about to be
removed and deletes the vma_lock to prevent pmd sharing as the vma is
on the way out.  In the case of madvise(MADV_DONTNEED) the vma remains,
but the missing vma_lock prevents pmd sharing and could potentially
lead to issues with truncation/fault races.

This issue was originally reported here [1] as a BUG triggered in
page_try_dup_anon_rmap.  Prior to the introduction of the hugetlb
vma_lock, __unmap_hugepage_range_final cleared the VM_MAYSHARE flag to
prevent pmd sharing.  Subsequent faults on this vma were confused as
VM_MAYSHARE indicates a sharable vma, but was not set so page_mapping
was not set in new pages added to the page table.  This resulted in
pages that appeared anonymous in a VM_SHARED vma and triggered the BUG.

Create a new routine clear_hugetlb_page_range() that can be called from
madvise(MADV_DONTNEED) for hugetlb vmas.  It has the same setup as
zap_page_range, but does not delete the vma_lock.

[1] https://lore.kernel.org/lkml/CAO4mrfdLMXsao9RF4fUE8-Wfde8xmjsKrTNMNC9wjUb6JudD0g@mail.gmail.com/

Fixes: 90e7e7f5ef3f ("mm: enable MADV_DONTNEED for hugetlb mappings")
Reported-by: Wei Chen <harperchen1110@gmail.com>
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
Cc: <stable@vger.kernel.org>
---
Note: backports to stable will be somewhat different as hugetlb vma_lock
      was added in 6.1.

 include/linux/hugetlb.h |  7 +++++
 mm/hugetlb.c            | 60 ++++++++++++++++++++++++++++++++---------
 mm/madvise.c            |  5 +++-
 3 files changed, 59 insertions(+), 13 deletions(-)

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index a899bc76d677..5cf2028ebad4 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -158,6 +158,8 @@ long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *,
 void unmap_hugepage_range(struct vm_area_struct *,
 			  unsigned long, unsigned long, struct page *,
 			  zap_flags_t);
+void clear_hugetlb_page_range(struct vm_area_struct *vma,
+			unsigned long start, unsigned long end);
 void __unmap_hugepage_range_final(struct mmu_gather *tlb,
 			  struct vm_area_struct *vma,
 			  unsigned long start, unsigned long end,
@@ -426,6 +428,11 @@ static inline void __unmap_hugepage_range_final(struct mmu_gather *tlb,
 	BUG();
 }
 
+static void clear_hugetlb_page_range(struct vm_area_struct *vma,
+			unsigned long start, unsigned long end)
+{
+}
+
 static inline vm_fault_t hugetlb_fault(struct mm_struct *mm,
 			struct vm_area_struct *vma, unsigned long address,
 			unsigned int flags)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 931789a8f734..3de717367e0a 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5202,27 +5202,63 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct
 		tlb_flush_mmu_tlbonly(tlb);
 }
 
-void __unmap_hugepage_range_final(struct mmu_gather *tlb,
+static void __unmap_hugepage_range_locking(struct mmu_gather *tlb,
 			  struct vm_area_struct *vma, unsigned long start,
 			  unsigned long end, struct page *ref_page,
-			  zap_flags_t zap_flags)
+			  zap_flags_t zap_flags, bool final)
 {
 	hugetlb_vma_lock_write(vma);
 	i_mmap_lock_write(vma->vm_file->f_mapping);
 
 	__unmap_hugepage_range(tlb, vma, start, end, ref_page, zap_flags);
 
-	/*
-	 * Unlock and free the vma lock before releasing i_mmap_rwsem.  When
-	 * the vma_lock is freed, this makes the vma ineligible for pmd
-	 * sharing.  And, i_mmap_rwsem is required to set up pmd sharing.
-	 * This is important as page tables for this unmapped range will
-	 * be asynchrously deleted.  If the page tables are shared, there
-	 * will be issues when accessed by someone else.
-	 */
-	__hugetlb_vma_unlock_write_free(vma);
+	if (final) {
+		/*
+		 * Unlock and free the vma lock before releasing i_mmap_rwsem.
+		 * When the vma_lock is freed, this makes the vma ineligible
+		 * for pmd sharing.  And, i_mmap_rwsem is required to set up
+		 * pmd sharing.  This is important as page tables for this
+		 * unmapped range will be asynchrously deleted.  If the page
+		 * tables are shared, there will be issues when accessed by
+		 * someone else.
+		 */
+		__hugetlb_vma_unlock_write_free(vma);
+		i_mmap_unlock_write(vma->vm_file->f_mapping);
+	} else {
+		i_mmap_unlock_write(vma->vm_file->f_mapping);
+		hugetlb_vma_unlock_write(vma);
+	}
+}
 
-	i_mmap_unlock_write(vma->vm_file->f_mapping);
+void __unmap_hugepage_range_final(struct mmu_gather *tlb,
+			  struct vm_area_struct *vma, unsigned long start,
+			  unsigned long end, struct page *ref_page,
+			  zap_flags_t zap_flags)
+{
+	__unmap_hugepage_range_locking(tlb, vma, start, end, ref_page,
+					zap_flags, true);
+}
+
+/*
+ * Similar setup as in zap_page_range().  madvise(MADV_DONTNEED) can not call
+ * zap_page_range for hugetlb vmas as __unmap_hugepage_range_final will delete
+ * the associated vma_lock.
+ */
+void clear_hugetlb_page_range(struct vm_area_struct *vma, unsigned long start,
+				unsigned long end)
+{
+	struct mmu_notifier_range range;
+	struct mmu_gather tlb;
+
+	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm,
+				start, end);
+	tlb_gather_mmu(&tlb, vma->vm_mm);
+	update_hiwater_rss(vma->vm_mm);
+
+	__unmap_hugepage_range_locking(&tlb, vma, start, end, NULL, 0, false);
+
+	mmu_notifier_invalidate_range_end(&range);
+	tlb_finish_mmu(&tlb);
 }
 
 void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start,
diff --git a/mm/madvise.c b/mm/madvise.c
index 2baa93ca2310..90577a669635 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -790,7 +790,10 @@ static int madvise_free_single_vma(struct vm_area_struct *vma,
 static long madvise_dontneed_single_vma(struct vm_area_struct *vma,
 					unsigned long start, unsigned long end)
 {
-	zap_page_range(vma, start, end - start);
+	if (!is_vm_hugetlb_page(vma))
+		zap_page_range(vma, start, end - start);
+	else
+		clear_hugetlb_page_range(vma, start, end);
 	return 0;
 }
 
-- 
2.37.3